A Healthy NIH Intramural Program: Structural Change or Administrative Remedies? Report of a Study (1988)

The National Institutes of Health (NIH) intramural research program has made important contributions to basic research, clinical research, the training of young scientists, diffusion of knowledge, and national efforts toward specific health objectives. The record of the many medical scientists responsible for these accomplishments is universally admired. The committee, having determined for itself that the traditional missions of the intramural program are relevant now and will be into the foreseeable future, turned its attention to how well those missions are being accomplished. It examined evidence to determine whether the quality of the intramural program has diminished and whether any existing policies or practices are detrimental to future accomplishments.

Since the concepts of scientific excellence and creativity are difficult to define, let alone measure precisely, the committee looked at a variety of indicators of excellence, including:

• important discoveries by scientists in the NIH intramural research program,

• bibliometric studies,

• the number of NIH-trained or NIH-employed scientists receiving prestigious awards and memberships,

• replenishment of the scientific staff with young talent.

The intramural research program quality control mechanisms, while not direct indicators of excellence, can be used as process measures. Each of these indicators gives some measure of the quality of a scientific program—its influence on other scientists, peer judgments of scientific contributions, the quality of new talent, management processes that evaluate scientific personnel, and programs that guide the use of resources. Singly, these indicators lead to limited conclusions regarding excellence. If, however, they all point in the same direction, we can view conclusions with greater confidence.

To get a concrete sense of recent contributions of the intramural program to the biomedical knowledge base, the committee asked the directors of each institute to identify three outstanding examples of achievements of the intramural program in the last ten years. Appendix C lists these examples. Below is a selection that illustrates the range and importance of the intramural program.

• Development of new vaccines against important bacterial infections of infants and children, including Hemophilus influenzae type B, pertussis, and typhoid.

• Development of a curative therapy for cystinosis, an inborn error of metabolism.

• Discovery of AZT as an effective agent against human immunodeficiency virus (HIV or AIDS virus).

• Discovery of the human immunodeficiency virus (HIV) to be the cause of AIDS.

• Determination of the molecular defects in various types of abnormal lipoprotein metabolism.

• Discovery of an oncogene that led to the identification of the gene for cystic fibrosis and of another oncogene that codes for a growth factor.

• Discovery of the toxic effects of the enzyme, aldose reductase, in diabetes. Such effects probably underlie the complications of diabetes, such as blindness and nerve damage. Inhibitors of the toxic enzyme have been developed and are now in clinical trials.

• Characterization of different types of the protein phospholipase C, important in signal transduction mechanisms in cells, and demonstration that the sub-types of this protein are differentially present in specific cells and tissues.

• Development of recombinant DNA techniques, the first cloning of a mammalian gene.

• First demonstration of the molecular basis of antibody diversity.

• Discovery of interleukin-2, which is produced by a certain immune system cells called T lymphocytes; interleukin-2 also promotes the proliferation of T lynphocytes.

• Identification and cloning of the earliest genes to be expressed during embryo growth in vertebrates and determination of the products of those genes.

• Gene transfer and expression in intact animals using retroviral vectors.

• The Surveillance, Epidemiology, and End Results Program (SEER) is an intramural program which provides a statistical survey capability for tracking cancer incidence, morbidity, and mortality in the nation.

• Extensive program of evaluation of prosthetic heart valves and the mechanisms of their failure.

These examples of excellence illustrate why the committee believes that the intramural program has made, and continues to make, invaluable contributions to our knowledge of basic biologic processes and their dysfunction in disease. To probe more systematically and widely into the intramural contribution, the committee commissioned a bibliometric analysis of the scientific literature.

The term “bibliometrics” refers to the quantitative analysis of papers in research journals and of the citations received by these papers in subsequent journal articles. The extent to which published papers are cited by authors of subsequent papers is arguably the best single estimate of the quality of this published research output over extended time periods. Well designed experiments have yielded very high correlations (~.90) between peer judgments of quality and citation rates (Narin, 1983).

Since 1973, the National Institutes of Health Program Evaluation Branch has worked collaboratively with Computer Horizons, Inc. (CHI) to develop and apply a range of bibliometric tools to the analysis of NIH research and training programs. The NIH-CHI research collaboration and related efforts supported by the National Science Foundation (NSF) have produced extensive evidence of the validity, reliability, and utility of these tools for the assessment of research and training programs.

Caution in the application of bibliometrics is essential. When the number of publications is small, changes in either publication or citation rates may appear statistically significant, though they are of little practical significance. Therefore, it is desirable that data be aggregated into units of time and classification that are large enough to yield useful information. Large aggregates, however, mask such individual cases in which, for example, a paper reporting highly meritorious research might receive few citations simply because it is published in an unpopular journal or discipline, or because the work is so far ahead of its time that its significance is not recognized for years. Negative or critical

citations, also important in the individual case, have virtually no effect in an analysis of large aggregates of papers. It is sometimes argued that the propensity for methodological advances to receive large numbers of citations is another limitation on the utility of citation data.

The fact that publication practices of scientists vary with their disciplines has significance for the analysis of bibliometric data. Citation practices also vary among disciplines, and it is essential that statistical measures be standardized within disciplines so that comparisons among them, and collective statistics across disciplines, may be valid and reliable. These cautions have been observed in the analyses that follow.

Although the number of papers contributed to the biomedical sciences literature by the intramural research program increased during each four-year period between the mid-1970s and mid-1980s, the proportion of papers by intramural scientists declined slightly. This indicates that the intramural program continued to be increasingly productive, but the productivity (as crudely measured by publication counts) of the external world grew at a somewhat faster rate. But, when the focus of concern is on eminence as it is in the present analysis, a measure that permits comparison of performance at the top of the distribution of participants is desirable. For this kind of comparison the top citation decile is used. This measure is based on the frequency with which a specified subset of papers is found among the most highly cited 10 percent of all papers in the set. By this measure, the intramural program excels. Among papers published between 1981–1984, almost 25 percent of the top decile from clinical medical journals and almost 22 percent from those classified as biomedical research, were contributed by the intramural program. This performance has been consistent over the decade studied (Table 2-1). Comparing the average number of citations per paper received by intramural papers with the average number of citations per paper received by papers authored by academic investigators, the ratio has been approximately 2:1 (Table 2-2).

Another measure is the “average influence” of the journals in which papers are published. This is an indication of the relative quality of journals measured by the frequency with which these articles are cited by other journals. As with other quality measures, performance of the NIH intramural program is extraordinarily high. The average journal influence per paper for all U.S. papers during the period 1981–1984 in clinical medicine was 18.1; for intramural papers it was 30.2. The discrepancy is similar for the basic biomedical research journals. The journal influence average for all U.S. papers during the period 1981–1984 for the basic biomedical papers was 45.6; the intramural average was 62.0.

Examination of papers published in journals that are intended to serve the biomedical sciences broadly (e.g., Nature, Proceedings of the National Academy of Sciences, Proceedings of the Royal Society, Science, etc.) indicates that more than seven percent of intramural papers between 1981–1984 appeared in these journals (activity index 3.3). The average journal influence per paper for intramural papers in this category was 100, as against 80 for papers from universities and medical schools. More than 22 percent of the 1981–1984 papers were among the most highly cited 10 percent, even among the authors appearing in these highly influential journals.

Overall performance is, however, a crude measure that conceals a complex network of change over time. This is revealed when the data are subdivided by journal classification into the many separate clinical and basic biomedical disciplines in which NIH intramural scientists publish.

When intramural research activities are viewed from the perspective of individual disciplines, several patterns of change emerge. These changes for 27 clinical and basic biomedical sub-fields are detailed in Appendix D. Eleven sub-fields maintained extremely high levels of quality, with some even increasing in strength. Only six sub-fields lost strength in the top citation decile. On the basis of these measures, only one declined in both productivity and quality.

The committee was aware of the dangers of over-interpreting the results of the bibliometric analysis, but was satisfied that, overall, the intramural program demonstrated a high level of performance when compared with the general academic community. Analyses that compare the intramural program’s mean citations per paper to selected elite research institutions, namely Rockefeller University and Scripps Clinic and Research Foundation, show that the intramural program ranks marginally below the latter and more substantially below the former. But as the author warned, “Comparisons of one institution with another should be undertaken with caution, owing to the diversity of research pursued by each and varying levels of citation among various fields” (The Scientist, 1988). Size of the institution also plays a role, and given the large size and scope of research, it is not surprising that the intramural program ranks below a smaller and more focused center such as Rockefeller University. Importantly, the NIH intramural programs^’ publication record has not deteriorated over time.

During the course of its deliberations, the committee inquired widely to gather comments from professional associations, foundations, and voluntary organizations on the issues being studied. The committee regularly heard of the accomplishments of the intramural program and the valuable contributions made by its scientific staff.

These accomplishments are reflected to some extent by the awards and honors intramural scientists have received, including four Nobel prizes since 1968 and more than a dozen Lasker Foundation awards. Of the approximately 60 members elected each year to the National Academy of Sciences (NAS) in the past decade, at least two to three per year have been NIH scientists (National Academy of Sciences, unpublished data, 1988). Over the past two decades, 28 percent of the NAS scientists in relevant specialties have either worked and/or trained at NIH. The American Society for Clinical Investigation (ASCI), which elects scientists aged late thirties to early forties, fills 5–15 percent of its places with NIH scientists. On average, each year since the 1950s, approximately one-sixth of those elected to ASCI have worked at NIH in some capacity. Roughly one-third, on average, of ASCI members elected annually since 1970 have received their training in the intramural program (Institute of Medicine, unpublished data, 1988). Four of the 9 American Federation for Clinical Research (AFCR) Young Investigator Awards were awarded to intramural physicians; 80 percent of the finalists for this award received their research training in the intramural program (L.Morrison, American Federation for Clinical Research, personal communication, 1988).

Looking at these accolades over time, there are no changes in the frequency with which NIH staff are honored. This information indicates that an elite core of scientists in the intramural program is among the nation’s most highly regarded researchers.

Although the quality of senior-level scientists as indicated by peers remains high, NIH must be concerned about whether the stock of talent is being adequately replenished.

There are approximately 1,300 doctoral scientists completing their training now in various non-tenured positions at NIH (NIH, 1988). Unfortunately, there are no centralized data systems that allow the committee to compare the quality of the present cohort with earlier years.

The committee examined the records of some prestigious research training fellowship programs in an effort to detect trends in the numbers of young scientists taking such fellowships to NIH. It also analyzed data from the NSF’s longitudinal survey of recipients of doctorates to get an indication of how the quality ratings of the graduate schools providing doctorates to NIH compared to those of other settings, and whether this measure of quality indicated any patterns over time at NIH. These crude indicators show little change. However, because relatively small numbers are involved in the samples, the committee was reluctant to place much weight on this evidence which runs counter to some powerful external factors. Chief among these is the growing number of employers, including industry, competing for the same pool of biomedical postdoctoral

candidates. Also, the incentives of the doctor draft are not likely to be recreated to give NIH a large competitive edge as an employer of young physicians.

Lacking are both objective information on the quality of recent cohorts, and a means of measuring the extent to which the intramural program is having difficulty recruiting the best candidates. There is no information on the applicant pool, because prospective fellows tend to negotiate directly with intramural laboratories or with particular researchers. Candidates are often attractive to laboratories because of the particular training, experience, research accomplishments, or interest in an area of work rather than a more generic measure of quality such as grades, test scores, and graduate school reputation. The committee’s interviews with NIH staff did not reveal the sense of a problem in recruiting the fellows when they wish. Senior researchers have been more concerned about full-time equivalent (FTE) limits and their ability to obtain the necessary position than their ability to fill those positions with the right person.

Because the application procedure for the medical staff fellowship (the major source of physician scientists) is centralized, some data about this group are available. There has been a precipitous drop in the number of applications submitted in the past two years, from 294 to 187. Unfortunately, NIH does not maintain records in a way that permits an assessment of the characteristics of the applicants accepted into the program; however, because NIH has continued to accept the same number of fellows from a shrinking number of applicants, one might be tempted to surmise that quality is declining. Anecdotal evidence is equivocal on this score. The president of ASCI has stated that salaries at this level are highly competitive. He believes that this, together with the opportunity to work at the Clinical Center, has enabled the intramural program to maintain its share of talent at the most junior levels. However, some medical educators have informally expressed concern about the lack of interest of some of their best students to pursue training at NIH.

One of the most critical factors in attracting postdoctoral candidates is the opportunity for them to work and learn under world-class senior investigators, as well as under rising young stars. The committee concluded that by focusing on improving the intramural program^’s capacity to continue to attract scientists in these categories, it will help ensure the quality of future postdoctoral cohorts.

The mechanisms used by the intramural program to ensure quality are considered to be particularly important. Moreover, there has been longstanding concern among biomedical scientists that the review process for the intramural research program lacks the rigor of the competitive

peer review process of the extramural program. The committee therefore reviewed the two major quality assurance mechanisms of the intramural program—the review of the intramural research laboratories conducted by the boards of scientific counselors, and the process through which scientists are selected for tenure in the NIH intramural program. A brief description of the current system and its recent history follows.

NIH policy states that “all research conducted intramurally must be reviewed regularly by highly qualified outside scientists.” For this purpose, each institute appoints a board of scientific counselors to review the intramural work of each institute. The boards are composed of scientists with outstanding achievements and expertise in the fields under review. The institute’s scientific director may invite additional experts to supplement the expertise of the board members for specific reviews.

Nominations for board membership are made by scientific directors and are approved by the Deputy Director of NIH and the individual institute director. Members are appointed by the Director of NIH. During these reviews, the board of scientific counselors considers the quality of research accomplished since the last review and its contribution to the institute’s mission. Over a period of time, each laboratory and tenured principal investigator is examined, as are scientists being considered for tenured positions. The board offers advice to the scientific director regarding allocation of personnel positions, funding of specific research areas, and future directions for research.

Prior to the review, members of the board of scientific counselors receive written descriptions of the laboratories’ research, staff qualifications, budget summaries, space allocations, and research support contracts. When the board meets, tenured scientists and junior staff report on present and planned research—board members have an opportunity to question the scientists and to visit laboratories.

Within four months of review, a report of the board’s findings and recommendations is submitted for information to the scientific director, the director of the institute, the NIH Deputy Director for Intramural Research, and the Director of NIH. Following review by the boards of scientific directors of all the institutes, the report is sent to each institute’s national advisory council for information and comment. The scientific director is required to report to the board of scientific counselors, at the earliest practical date, on actions taken on their recommendations (Eberhart, 1982; National Institutes of Health, 1986a).

Feedback to the board of scientific counselors was of particular concern to members of the 1984 Institute of Medicine’s Panel to Study the Current Organizational Structure of the NIH (Institute of Medicine, 1984),

who interviewed members of the Boards. Many board members expressed a desire to be better informed regarding the institutes’ implementation of their recommendations, suggesting that their recommendations were not always given serious consideration.

In 1985, the Director of NIH recruited a group of external reviewers to examine the management of the Clinical Center. As part of the review, 8 experts from a broad spectrum of clinical disciplines evaluated 50 current or recently completed protocols selected by the director and deputy director of the Clinical Center. These protocols were chosen to represent “high quality” examples of research that made intensive use of the Clinical Center resources. The reviewers concluded:

At about this time, Congress became concerned about the intramural review process in the context of deliberating on the 1985 Health Research Extension Act. The conference committee report notes the following:

As a result of congressional concern, the review process has been tightened. The recommendations of the boards of scientific counselors must be answered in writing by the scientific director of the institute at the next meeting of the board (National Institutes of Health, 1986b).

The committee commends NIH for its actions to strengthen the review procedures. It has, however, identified two specific points in the review system that lack mechanisms to ensure objectivity and accountability. First, while few question the stature of those who serve on boards of scientific counselors and as ad hoc consultants, the fact that they are nominated by the scientific director of the institute whose program will be under review compromises the external credibility of their report. Second, despite improved feedback to the board of scientific counselors on the implementation of their recommendations, the scientific director is at no time accountable to anyone outside the institute. Because this process is so often unfavorably compared with the rigorous extramural peer review—particularly by some who believe that intramural funds would be better spent for unfunded extramural projects—it is important that the system has real and visible safeguards.

A rigorous review process is necessary but not sufficient to sustain quality. The scientific director of each institute is key to the success of the research program, providing both intellectual and administrative leadership. Not only do the scientific directors control resources, but, less tangibly, they are responsible for the scientific esprit of the institute. Subsequent chapters include discussion of the problems of recruiting and retaining outstanding individuals for these jobs.

The long-term creativity and productivity of the intramural program depends to a great extent on the quality of the people who became tenured. Not only is their own productivity important, but they make up the pool from which the future leadership is drawn and play a role in attracting postdoctoral fellows to the intramural program. It is therefore important that there exists a rigorous procedure for selecting the 30 to 35 scientists who each year win tenure.

A scientist is usually eligible for tenure after 5 to 10 years as a postdoctoral fellow. The laboratory chief makes the decision about recommending the fellow for tenure. The successful candidate must pass review by the laboratory branch chiefs of his institute, the board of scientific counselors, the scientific directors of all the institutes, who vote by secret ballot, and finally the NIH Deputy Director for Intramural Research. This year-long process requires that the candidate be supported by at least six letters of reference as well as other evidence to support the case for tenure.

The intention of this process is to parallel its counterpart in universities. Whether the best scientists emerge from the process depends to a great extent on the caliber of fellows the laboratory chiefs propose as candidates for tenure.

Because of the NIH environment, where postdoctoral experiences often begin with assignment to work under a mentor, some believe that the time allowed is insufficient for a fellow to establish credentials as an independent investigator.

Lacking formal evaluations of the intramural tenure process, and not being in a position to make its own evaluation, the committee was unable to comment on whether the process is comparable to academic processes or whether it successfully selects the best candidates. In the course of its discussions, the committee heard some criticism of the NIH tenure process, including variability among institutes, and believes an evaluation is needed.

This report has to do with sustaining excellence in the intramural program. As far as the committee was able to determine, using a variety of indirect measures, the intramural program has maintained its scientific stature. However, as is indicated later in this report, there are actions the NIH intramural program can take to improve its performance. There are also administrative problems that are beyond its control that could jeopardize quality in the future. The next chapter will describe these problems.

Eberhart, J.C. 1982. Review and Evaluation of Intramural Research. A Report to the NIH Scientific Directors. Bethesda, MD: National Institutes of Health. Photocopy.

Gee, H.H. 1988. A Report on Publications of NIH Intramural Scientists, 1973–1984. Paper commissioned for Institute of Medicine. Washington, D.C.: IOM.

Health Research Extension Act. 1985. Congressional Conference Report (99–309) on NIH Reauthorization. Washington, D.C.

Institute of Medicine. 1984. Responding to Health Needs and Scientific Opportunity: The Organizational Structure of the National Institutes of Health. Washington, D.C.: National Academy Press.

Narin, F. 1983. Subjective vs. Bibliometric Assessment of Biomedical Research Publications. NIH Program Evaluation Report. Bethesda, MD: NIH.

National Institutes of Health. 1985. Review of NIH Clinical Center Management Issues. Bethesda, MD: NIH. Photocopy.

National Institutes of Health. 1986a. Biennial Report of the Director. Vol. 1. Bethesda, MD: NIH.

National Institutes of Health. 1986b. Manual Transmittal Sheet, 3005: Review and Evaluation of Intramural Research by Boards of Scientific Counselors. Bethesda, MD: NIH. Photocopy.

National Institutes of Health. 1988. The Nation’s Commitment to Health Through Biomedical Research—The NIH, the NIH Intramural Research Program and the Pursuit of Scientific Excellence. Bethesda, MD: NIH. Photocopy.

The Scientist. July 11, 1988. Despite Brain Drain at NIH. The Quality Rises. Page 15.

This page intentionally left blank.