5

Duplication, Replication, and Complementarity

INTRODUCTION

The second charge to the committee was to determine any areas of duplication in the research programs of the National Institutes of Health (NIH) and the Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA). 1 The impetus for such concern is that, with two agencies comprising 16 research institutes, federal dollars might be wasted funding duplicative research. The committee used the research in the three case studies as examples for its examination of replication, duplication, and complementarity.

DEFINITIONS

In discussing these terms, it is important to distinguish between research projects and research programs. Projects refers to singular research efforts, such as individual experiments or a set of experiments, that are designed to answer a specific hypothesis and that are funded by a research grant or reported in a journal article. Programs, on the other hand, refers to a much larger constellation of research funded by an institute or performed by an investigator over a period of time. Replication is an integral part of research projects; duplication and complementarity are related to both projects and programs. The committee employed the following definitions for replication, duplication, and complementarity: 2

  • Replication represents the deliberate or conscious repetition of research efforts, intended to confirm or extend previously or simultaneously obtained, but still uncertain, findings.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization 5 Duplication, Replication, and Complementarity INTRODUCTION The second charge to the committee was to determine any areas of duplication in the research programs of the National Institutes of Health (NIH) and the Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA). 1 The impetus for such concern is that, with two agencies comprising 16 research institutes, federal dollars might be wasted funding duplicative research. The committee used the research in the three case studies as examples for its examination of replication, duplication, and complementarity. DEFINITIONS In discussing these terms, it is important to distinguish between research projects and research programs. Projects refers to singular research efforts, such as individual experiments or a set of experiments, that are designed to answer a specific hypothesis and that are funded by a research grant or reported in a journal article. Programs, on the other hand, refers to a much larger constellation of research funded by an institute or performed by an investigator over a period of time. Replication is an integral part of research projects; duplication and complementarity are related to both projects and programs. The committee employed the following definitions for replication, duplication, and complementarity: 2 Replication represents the deliberate or conscious repetition of research efforts, intended to confirm or extend previously or simultaneously obtained, but still uncertain, findings.

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization Duplication represents inadvertent, unconscious, or, more rarely, deliberate repetition of research efforts, thus not serving a need to confirm or otherwise verify conclusions from previous research undertakings. Complementarity represents efforts involving independent approaches or overall strategies to confirm, overturn, or extend particular research findings. Replication is important at the project level, because single experiments must be repeated and hypotheses constantly reevaluated. The scientific process depends in no small part on the ability of independent observers to repeat and confirm one another's findings. In this context, replication is not only legitimate but essential, providing “proof positive” of otherwise uncertain research findings and lending confidence to conclusions drawn from them. Replication also serves to overthrow false hypotheses: if an experiment is repeated and does not confirm the original results, the alternative hypothesis must be seriously considered. Once a research finding or hypothesis is well accepted, however, repeating experiments no longer adds to the knowledge base and becomes duplicative. Duplication, on the other hand, is inherently wasteful and is normally guarded against, although it can also be excused or even encouraged under special circumstances. For example, an investigator can diligently search existing sources to verify that proposed research has not been done before, but the enormity of the scientific literature precludes complete assurance that a project is novel. This inadvertent duplication of research projects is not optimal but excusable. It is not thought to be particularly rampant. Eliminating duplicative research programs is a means of saving public money, but duplication of research projects and programs is acceptable under special circumstances, such as a period of great scientific opportunity or a period of great crisis. For example, the AIDS crisis demanded changes in the normal scientific process: As part of the federal response to the AIDS crisis, certain usually stringent practices for evaluating research proposals were eased somewhat as efforts began rapidly accelerating to formulate and then pursue as many promising leads as possible. . . . When a deadly disease with a high social cost dictated an extraordinarily rapid research program build-up, it made sense to start many similar, potentially duplicative research efforts in parallel. Although some fraction of those parallel efforts inevitably led down blind alleys, others served

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization to confirm new and unexpected findings or to provide insights that gave renewed momentum to projects that may have temporarily stalled. In the aggregate, such a broad-based effort has expedited the development of novel therapies for individuals with AIDS or improved the quality of medical care available to them. 3 Complementarity, by contrast, is an important attribute of research and is to be encouraged. In essence, complementarity depends on reaching the same or very similar conclusions by taking different approaches. Addiction, for example, is a complex problem with far-reaching social consequences, and it is being studied at many levels simultaneously —neurochemical and neurophysiological as well as behavioral and sociological —in an appropriately complementary strategy for identifying and understanding fundamental components of this complex phenomenon. Examples of complementarity within research projects would be to include both in vivo and in vitro effects of a drug, or to compare both rat and mouse responses to a particular manipulation. If the results of comparisons between species or techniques agree, the experiments confirm the findings; if the results differ, they provide new information that spurs new research. Examples of complementary research programs, for example, are the pain research programs in the National Institute of Mental Health (NIMH), the National Institute on Drug Abuse (NIDA), the National Institute of Neurological Diseases and Stroke (NINDS), and the National Institute of Dental Research (NIDR). As indicated in Public Health Service (PHS) agreements on funding and referral guidelines, 4 each institute is interested in unique aspects of pain—anatomy, physiology, pharmacology, etiology, organ systems, behavior, and so forth—and, therefore, the overlapping research programs are, for the most part, complementary. REPLICATION, DUPLICATION, AND COMPLEMENTARITY IN THE CASE STUDIES Information on research grants funded by NIH and ADAMHA in the case study areas was obtained and classified as described in Chapter 3 . Table 5-1 shows the number of grants in each case study area in health status research and health interventions research. (Health status research is defined as that which provides new knowledge regarding health, disease, biological, and behavioral processes. Health interventions research, by contrast, is defined as

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization TABLE 5-1 Results of Classification of Case Study Research Grants Year Alzheimer's Disease Schizophrenia and Parkinson's Disease: Dopamine a Substance-Abusing Pregnant Women Health Status Research     1975 17 14 30 1978 24 26 73 1982 72 b 20 79 b 1985 266 b 34 88 b 1987 308 b 46 119 b 1989 469 b 70 153 b Health Interventions Research c     1975 0 11 1 1978 3 10 1 1982 18 b 7 5 b 1985 49 b 10 6 b 1987 42 b 9 6 b 1989 7 b 21 19 b a The search was on “schizophrenia and dopamine” or “Parkinson's disease and dopamine”; therefore, many grants on schizophrenia research and Parkinson's disease research are not represented in this search. See Figure 3-3 and Figure 3-4 for comparison. b Represents estimates based on sampling as described in Chapter 3 . c Health services research and research demonstrations are not well represented as discussed in Chapter 3 . that which provides new knowledge concerning the modulation of or intervention in health status. These are discussed in more detail in Appendix B.) The assessment of the extent of duplication is based on several approaches and several premises. A major premise in this analysis is that, by definition, grants in separate categories cannot be duplicative. A second premise is that duplication is most likely in situations where there are many grants funded in a particular area, particularly in several institutes. Alzheimer's disease research, for example, was thought most likely to include duplication because of the large number of grants in health status research and the many institutes that support research in this

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization area. As discussed below, however, there did not seem to be much duplication in Alzheimer's disease research. As with AIDS research, duplication can be deliberate and excusable when an overwhelming social and medical problem, such as Alzheimer's disease, demands attention. Although the substance-abusing pregnant women category seems to include quite a large number of grants, it includes research on at least four abused substances. It was only in the fetal alcohol syndrome that the number of health status research grants was large enough to cause concern about duplication. Information gathered for the study suggests that duplication of basic and clinical biomedical research is not a problem in the PHS. The approach to assessing duplication entailed looking at grants within categories, reading titles and abstracts of the grants within a category, and deciding, given the definitions and discussion of replication, duplication, and complementarity, whether grants were duplicative. Additional information used in the following discussion of the case study areas comes from the Division of Research Grants Referral Guidelines and from interviews conducted for the case studies. As described in Chapter 3 , the committee found it extremely difficult to address the question of duplication because of the lack of a standard nomenclature within PHS agencies for classifying research and service programs and projects. The lack of an agreed-upon nomenclature presents a barrier to planning, evaluation, public access to information, and accountability within the PHS. If the Congress and executive agencies have a continued interest in periodic assessments of research, demonstration, service, and dissemination programs and of the relationships among programs, it will be necessary to develop a standard nomenclature that allows for consistent use of terms across PHS agencies. The committee recommends that an interagency task force be formed to develop a standard nomenclature for classifying basic and clinical research, demonstrations, and service development activities across PHS agencies. The committee further recommends that the National Library of Medicine be mandated and given appropriate resources to carry out whatever research is necessary for the development of this standard nomenclature. Alzheimer's Disease Case Study The Alzheimer's disease (AD) research programs in the National Institute on Aging (NIA), NINDS, and NIMH are more complemen-

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization tary than duplicative. 5 For example, as a disease primarily of the aged, with neurological and psychiatric symptoms, AD is clearly of interest to NIMH, NIA, and NINDS, as well as many other federal research organizations. 6 However, each institute's AD program differs in primary emphasis: NIA has primary responsibility for AD and investigates it with an eye to the aging process; NINDS emphasizes the neurological aspects of the disease; and NIMH emphasizes the neuropsychiatric aspects, as well as the problems of caregiving stress. 7 This is not to say that there are no overlaps, or (for example) that some grantees funded by NIA could not have applied for funding for the same project from NINDS. In fact, these margins of overlap are thought to generate healthy competition both between institutes for the best researchers and between researchers for the most generous and stable funding. 8 The particular aspect of AD research that crosses institute boundaries most obviously is the cellular and molecular biology of cells that degenerate in AD or produce the abnormal protein associated with AD symptoms. However, current thinking is that the research opportunities for AD are so promising that there are more questions to be asked than there are dollars to pay: with so little understanding and so many leads, the chances of individual projects being duplicative are small. These projects are replicative or complementary. Not nearly enough is known about the cellular and molecular biology of AD for projects to be duplicative. (Coordinating mechanisms within the PHS and the Department of Health and Human Services [DHHS] to deal with potential duplication and coordination of priorities within AD are discussed later in the chapter.) If in the course of this all-out attack on many fronts of a very difficult scientific problem, some potentially duplicative research projects or programs are funded, it is probably not worth ferreting out or worth the risk of missing a potentially important discovery. 9 , 10 Schizophrenia and Parkinson's Disease: Dopamine Research In the case study of schizophrenia and Parkinson's disease, the area of most likely overlap is the aberrant chemical system common to both diseases—the dopamine system. 11 However, the dopamine systems relevant to schizophrenia differ in anatomy, biochemistry, genetics, pharmacology, and physiology from those relevant to Parkinson's disease. Parkinson's is primarily a disease of degenerating dopamine neurons that result in motor system dysfunction. Schizo-

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization phrenia is primarily a disease of aberrant, but not degenerating, dopamine system functioning that results primarily in cognitive, emotional, and sensory dysfunction. Although both NIMH and NINDS sponsor very fundamental research on, for example, the molecular genetics of the dopamine receptor, recent research has demonstrated separate genes that encode for distinct dopamine receptors. If there are many projects on dopamine receptor genetics and physiology, it is because there are many dopamine receptors to study. These projects in NINDS and NIMH are not duplicative. Neither research projects nor research programs regarding dopamine in NIMH or NINDS are significantly duplicative according to the committee 's analysis. In fact, they are replicative in some cases, but more often they are complementary. Substance-Abusing Pregnant Women The case study of substance-abusing pregnant women revealed the least potential for duplication of health status and health interventions research. 12 So little is known in this area that research could hardly be duplicative. Only in the field of fetal alcohol syndrome did there seem to be a risk of duplication, but inspection of the research grants did not reveal significant duplication even in this area. The research grants were replicative and complementary. This case study illustrates that the concepts of replication, duplication, and complementarity apply to demonstration research as well. Research demonstrations need to be replicated at several sites before being accepted as effective treatment interventions, and demonstrations can also involve complementary approaches to the same problem or population. Demonstration programs and service system development activities for substance-abusing pregnant women are sponsored by NIDA, the Office of Treatment Improvement (OTI), and the Office of Substance Abuse Prevention (OSAP), frequently in the same state and city with no coordination within ADAMHA. Other efforts aimed at pregnant women (sponsored by the Health Resources and Services Administration [HRSA]) and at substance abusers (sponsored by ADAMHA) run the risk of being neither duplicative nor complementary, but simply uncoordinated. Therefore, the case study identified at least one specific example where coordination across agencies at the Assistant Secretary level would be helpful: between programs for primary care for pregnant women (through HRSA) and programs for substance abuse treatment for female addicts (through ADAMHA). In addition, within ADAMHA, there was no evidence of

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization coordination between the block grant programs for substance abuse treatment administered by OTI and treatment demonstrations for pregnant women administered by OSAP. GUARDING AGAINST DUPLICATION In light of the finding that duplication is not a serious problem in the case study research areas, it is important to discuss what mechanisms, characteristics, or procedures serve to decrease duplication. Mechanisms in place to limit unnecessary duplication of research include self-regulation by the scientific community, the peer review process for journal publication and grant applications, and guidelines developed by the PHS for funding and referral. Perhaps the most effective of these measures is the intense competitive pressure that scientists apply to one another and to themselves: The federal biomedical research system works like a market-place, with imaginative ideas and specialized materials being the principally traded commodities, and information exchange as well as peer recognition for one's accomplishments acting as the currency of trade. In such a system, quite naturally, there are penalties for doing purely duplicative work, and a premium is put on being creative and making breakthrough discoveries. . . . Reporting a supposedly new finding that turns out to be previously acknowledged work inevitably carries a severe penalty to one's reputation within the scientific community. Part of the penalty can be an investigator's subsequent difficulty in obtaining federal (or private) support for further research. Thus, although this crucial safeguard against duplication may not work in the initial instance, it can prove devastatingly effective in the long term. 13 A significant means by which federal research administrators guard against duplication occurs through coordinating councils, described in Chapter 3 in the discussion of Alzheimer's disease. The DHHS Alzheimer's Disease Council and the congressionally appointed Advisory Panel on Alzheimer's Disease are a means for the many federal agencies and institutes involved in AD programs to communicate on a regular basis. Interviews conducted for the AD case study suggest that these mechanisms contribute significantly to a comprehensive approach to AD and a decrease in duplicative efforts. 14

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization The process of peer review of scientific journal articles and grant applications also functions to detect and reduce duplicative research. When a researcher submits a grant proposal requesting funds for a set of experiments, or submits an article for publication describing the results of research, the group of scientists who review the proposal (or manuscript) and make recommendations regarding funding (or publication) consider the originality of the work. These scientists, as a group, are well acquainted with a vast proportion of the relevant scientific literature; proposals (or manuscripts) that include a great deal of unoriginal work would not be approved for funding (or publication). Although this monitoring system is generally considered effective, it has not been subject to a rigorous analysis. It is not designed to recognize or stop all duplicative efforts—such a system would be cumbersome to establish and costly to maintain. Another important protective mechanism against duplication occurs through the NIH Division of Research Grants (DRG), whose responsibility it is to receive, review, and assign research grants to the appropriate review committee and funding institute, center, or division (ICD). DRG makes every effort to resolve areas of overlapping scientific interest among ICDs, and to assign research applications to the appropriate ICD, but it also makes liberal dual assignments to ICDs in cases of genuine overlap. 15 ICDs also protect themselves against duplication through coordinating councils such as exist for Alzheimer's disease. 16 Institute and agency administrators meet on a regular basis to discuss their contribution to the cross-cutting effort; such communication helps to decrease the chances that institute research programs seriously duplicate each other; instead, these coordinating mechanisms encourage the development of complementary programs. The scientist–administrators would gain nothing by approving, encouraging, or funding duplicative research; the key to increasing next year's appropriation is to show results from this year's investment. Duplicative research would do little to advance the goals of the researcher, the administrator, or the agency, much less the goals of science. NOTES 1. Anti-Drug Abuse Act of 1988 (P.L. 100-690), section 2073. 2. A task force was convened to discuss the concepts. A document based on the day's discussion was prepared by science writer Jeffrey Fox.

OCR for page 109
Research and Service Programs in the PHS: Challenges in Organization 3. J. Fox, “Report on Scientific Methodology,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va. * 4. Division of Research Grants, NIH, PHS, DHHS, “Referral Guidelines for Funding Components of PHS,” 1990. 5. C. E. Blixen, “Case Study on Alzheimer's Disease,” prepared for the IOM Committee on Co-Administration of Service and Programs of the NIH, ADAMHA, and Related Agencies, available from the National Technical Information Springfield, Va. 6. Justification of Appropriation Estimates for Committee on Appropriations, Public Health Service Supplementary Budget Data (Moyer Material), Part 1, Volume IX, Fiscal Year 1991. 7. Division of Research Grants, NIH, PHS, DHHS, “Referral Guidelines for Funding Components of PHS,” 1990. 8. Blixen, “Case Study on Alzheimer's Disease.” 9. Fox, “Report on Scientific Methodology.” 10. Based on discussions in the task force meeting. 11. K. Stratton, “Parkinson's Disease and Schizophrenia: Dopamine and Beyond,” a case study prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va. 12. L. V. Klerman and M. A. Johnson, “Case Studies of Substance-Abusing Pregnant Women, Their Infants and Children,” prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va. 13. Fox, “Report on Scientific Methodology.” 14. Blixen, “Case Study on Alzheimer's Disease.” 15. Division of Research Grants, NIH, PHS, DHHS. “Referral Guidelines for Funding Components of PHS,” 1990. 16. Blixen, “Case Study on Alzheimer's Disease.” * For readers interested in obtaining copies of these papers, the full address of the National Technical Information Service is 5285 Port Royal Road, Springfield, VA 22161; telephone 703-487-4650.