The first charge to the committee was "to assess the factors or criteria used by NIH to determine funding allocations for disease research." The committee defined the charge broadly to include the adequacy of the criteria used to set priorities and allocate resources among all areas of research, some but not all of which are disease specific, in order to fulfill the mission of NIH to improve health through research.

NIH's budget for the current FY (1998) of $13.6 billion makes it the largest civilian research agency in the federal government (accounting for 37 percent of the federal nondefense R&D budget). Nevertheless, there is never as much funding as is needed to address all important health problems and pursue all research opportunities or as much funding as NIH's supporters would like it to have. Choices must be made and priorities must be set.

Priorities are not only driven by a scarcity of resources. They are also affected by health emergencies and epidemics, demographic trends affecting health, such as the aging of the population or changing patterns of tobacco and alcohol use, and new opportunities in science stemming from research advances or better research instruments.

The criteria that guide these choices are essential for public understanding of NIH's activities and for ensuring adequate levels of public support for its mission and budget. These criteria are a key element in the agency's accountability to the public.

How NIH allocates funding among areas of research has become a major issue over the past several years. As noted in Chapter 1, Congress has inquired about priorities among specific diseases or programs on a number of occasions since 1995. Some legislators believe that the distribution of NIH funding does not adequately reflect the number of citizens affected by various diseases or the costs of various diseases to society. Advocates for individuals with various diseases tell Congress that NIH is neglecting research on their focal disease relative to the levels of research on other diseases. Congress has also become concerned about how NIH allocates funding to areas that are not disease specific, for example, clinical research (versus basic research) and public health education.

In response to concerns about how it allocates funding among categories of research, NIH recently published "Setting Research Priorities at the National Institutes of Health," a 15-page booklet describing the criteria and processes that it currently uses (National Institutes of Health, 1997b) (referred to hereafter as Setting Research Priorities). In its FY 1999 budget request, NIH told Congress that the booklet "delineated" NIH's priority-setting principles and mechanisms and that NIH would "continue to consider the full array of relevant criteria in setting priorities for research funding, including the societal and economic costs associated with particular diseases and disorders" (U.S. Department of Health and Human Services, 1998:OD-33).

Setting Research Priorities lists five major general criteria that NIH calls "both influential and continuous" in priority setting (National Institutes of Health, 1997b:4). These criteria are

• public health needs,
• scientific quality of the research,
• potential for scientific progress (the existence of promising pathways and qualified investigators),
• portfolio diversification along the broad and expanding frontiers of research, and
• adequate support of infrastructure (human capital, equipment and instrumentation, and facilities).

According to NIH, these criteria are used to set priorities in a broad and complicated program with multiple goals and numerous levels. They are used in a process that allocates funding among NIH institutes and centers and, within each institute or center, among a hierarchy of programs. These allocation choices affect how much goes to basic versus clinical versus epidemiologic research, to need-driven versus science-driven research, to intramural versus extramural programs, to solicited versus investigator-initiated research projects, and to specific mechanisms, such as grants to investigator-proposed projects, grants solicited by NIH through specific RFAs, contracts, and centers, and so forth. They also affect the allocation of research funding among disease-specific activities on the one hand and more general or crosscutting activities on the other (e.g., research on basic processes that underlie many or all diseases and the research infrastructure).

Decision making—that is, priority setting—is highly decentralized at NIH, as is appropriate for a research organization with diverse forms of specialized expertise. However, the cumulative impact of such decisions needs to be monitored and perhaps adjusted at each level, especially at the level of the institute directors and the level of the director of NIH. That will ensure that priorities that involve higher-level coordination are met, that there are no significant gaps or unnecessary overlaps in research, and that emerging problems and opportunities affecting NIH and its mission are recognized. These process issues are addressed in Chapter 3. This chapter addresses the adequacy and completeness of the criteria per se (listed in National Institutes of Health [1997b:4]).

The NIH has an obligation to respond to public health needs, as judged by the incidence, severity, and cost of specific disorders. Calculating these needs is difficult, and there is not always clear distinction between expense and results (National Institutes of Health, 1997b:4).

Public health needs is clearly an important priority-setting criterion for an agency with the mission of uncovering new knowledge that will lead to better health for everyone. However, the description of the response to public health needs quoted above from Setting Research Priorities as being related to addressing specific disorders is too narrow. It reduces the concept of health to the absence of disease instead of taking a broader view of health as leading a full and high-quality life even in the presence of pathologies, chronic symptoms, and functional limitations (Institute of Medicine, 1997). NIH's actual research portfolio reflects this broader conception of health. Research on the needs of persons with disabling conditions, for example, focuses on treatment but also on how factors in the environment can be modified to enable them to function more fully despite their pathology.

In 1995, at the request of Congress, NIH published a report with a table listing the burdens and costs of 66 diseases and conditions, including the 15 leading causes of death, and the amount of research funding that NIH was devoting to each. The data in that report and a later edition with more recent data (National Institutes of Health, 1997a) have been used by critics of NIH priority setting to argue that there is little correlation between the distribution of disease burdens or costs and NIH funding allocations, for example, by finding much higher research spending per death from relatively rare diseases compared with the leading causes of death (see Table C-7 in Appendix C for an excerpt from the table in the report). NIH for its part notes that the data were derived in different ways and at different times, which limits comparability, that cost-of-illness studies have serious methodological limitations, and that even if the data were improved, many other important factors would have to be taken into account in allocating research dollars, such as the importance of research advances and opportunities and the availability of research tools and trained personnel (National Institutes of Health, 1997a:4-5).

Setting Research Priorities discusses alternative indicators of public health needs in greater detail in a later section on assessing health needs (National Institutes of Health, 1997b:8-9). It notes that U.S. health needs indicators that could be considered in allocating research funds include

• number of people who have a particular disease;
• number of deaths caused by a disease;
• degree of disability produced by a disease;
• degree to which a disease cuts short a normal, productive, comfortable life;
• economic and social costs of a disease; and
• need to act rapidly to control the spread of a disease.

Setting Research Priorities notes that all these criteria are relevant and, conversely, that the use of any one of these criteria exclusively would lead to the unwarranted neglect of some diseases (National Institutes of Health, 1997b). For example, "funding according to the number of deaths would neglect chronic diseases that produce long-term disability and high costs to society (disease such as mental illness and arthritis would be neglected)," but "funding according to the economic cost of illness would under-fund diseases that result in short illness and rapid death (this choice would result in a great deal of funding for Alzheimer's disease and muscular dystrophy and little, or none, for sudden infant death syndrome or certain types of cancer)" (p. 9). Moreover, the booklet says, "any of these criteria used exclusively would . . . underfund research on rare diseases, research that has taught us much about the diseases themselves and a great deal about normal human biology, other diseases, and new approaches to treatment" (p. 9).

NIH's discussion of these indicators reflects the recognition that the selection of measures is an expression of values (National Research Council, 1996), insofar as each will increase the priority placed on some health problems and decrease that placed on others. The booklet does not describe, however, how NIH in practice combines these (or other) health indicators in setting its research priorities or how these indicators are integrated with the other criteria, such as the degree of scientific opportunity, when allocating resources among programs or creating new programs.

Nor does NIH have a systematic process for collecting and analyzing data on the full range of health indicators (e.g., demographic trends and societal changes), whether on its own or through arrangements with agencies that do, such as the Health Care Financing Administration and the National Center for Health Statistics, Centers for Disease Control and Prevention. As a result, it is not in a position to evaluate its success in meeting this criterion. For example, changes in the age structure of the nation's population and the growth in the numbers of people who are members of minority groups and of persons with disabling conditions have important implications for the incidence and prevalence of certain health problems. NIH's current informal arrangements for considering health needs indicators leave it open to charges that it only gives lip service to consideration of the broad spectrum of health needs in setting priorities and allocating resources.

As noted in Chapter 1, NIH's Office of Financial Management has long tracked how much it spends on particular diseases and other activities to be able to respond to inquiries from Congress and other interested groups and individuals (see Table C-4 in Appendix C). Disease-specific interest groups use these statistics to help assess the priority that NIH gives to particular diseases or health problems relative to the priorities that it gives to other diseases or problems. In so doing, these groups are in effect trying to figure out NIH's implicit policy for implementing Criterion 1. However, NIH's position is that little reliance should be placed on these numbers because of variations in definitions and accounting procedures from institute to institute and over time. Indeed, although the Office of Financial Management has provided standard definitions for approximately 30 of the approximately 250 categories for which it compiles data reported by the institutes, it does not have the resources to ensure consistency and quality (the Office of Financial Management staff levels have been cut by two-thirds in recent years). In any case, such revealed preferences analyses (as economists would describe attempts to infer priorities from observed behavior) are a poor substitute for the direct expression of NIH's priorities and documentation and for the evaluation of their implementation. Nor can they capture the contribution of basic biological or behavioral research that could contribute to reducing multiple health problems.

In assessing public health needs as a factor in the process of allocating funds, NIH also must consider the disease-specific research being supported by others, including industry, voluntary health associations, and foundations. The combined research budgets of pharmaceutical companies and biotechnology firms are greater than that of NIH and are largely concentrated on applied research and development of new drugs and other disease-specific products. NIH should not fund research that would be supported by industry.

The NIH applies stringent review for scientific quality on all research proposals in order to return the maximum possible on the public's investment in medical research (National Institutes of Health, 1997b:4).

The text of this criterion reflects NIH's long-standing reliance on peer review of specific research proposals by relevant scientific experts to ensure that it supports the best projects in terms of scientific impact or significance, technical merit and feasibility, researcher qualifications, and facility adequacy. Although scientific peer review of each proposal is an important part of NIH's success, it relies on specialized criteria and processes that are not the main focus of this report. This report looks at the process that determines how funding is allocated among the various research programs, not how specific proposals are chosen for funding within each research program.

The committee is aware that NIH has undertaken a major revamping of the peer-review process. The Division of Research Grants created in 1948 was elevated to the Center for Scientific Review in 1997 and was placed under new leadership. The review criteria and process used by the Center for Scientific Review have been assessed by several groups, most recently, the Peer Review Oversight Group, a panel appointed by the director of NIH. The Peer Review Oversight Group recommended revised criteria, which have been adopted (see Box 2-1).

NIH is also addressing another problem in the peer-review system: the poor success of clinical research proposals competing for approval with basic research proposals within research areas addressed by review groups (called study sections). The problem, which is linked to the decline in the number of clinical investigators in recent years, was identified in several studies, including one by an IOM committee (Institute of Medicine, 1994). The NIH Director's Panel on Clinical Research recommended changes in its December 1997 report (NIH Director's Panel on Clinical Research, 1997), such as increasing the percentage of clinical research proposals to 30 to 50 percent in those sections reviewing them and increasing the number of experienced clinicians on those study sections, increasing support for training and research support programs for clinical investigators, and expanding the General Clinical Research Center program of the National Center for Research Resources (National Institutes of Health, 1997c).

The director of the Center for Scientific Review recently appointed a blue ribbon external panel to address the aspect of peer review that most affects overall priority setting among study sections: the NIH practice of "percentiling," that is, funding the same percentage of proposals in each study section, which means that each section has a certain claim on funding. As a result, the allocation of funding among research topics is affected in part by the structure of the study sections. If the structure does not keep up with changes in science—that is, if study sections are not created and phased out as the frontiers of science advance and scientific fields become obsolete and new ones emerge—then the allocation of the majority of NIH's funding that goes through the Center for Scientific Review is adversely affected. The new blue ribbon panel was directed explicitly to address this problem and recommend ways to update and to keep updated the study section structure.

As an administrator of science, the NIH has learned that many significant advances occur when new findings, often unforeseen, expand experimental possibilities and open new pathways for the imagination. Not all problems are equally approachable, no matter their importance to public health. Pursuit of a rare disease may often have unexpected benefits for more common problems. By the same token, increased spending on a disease is wasteful when there are neither promising pathways to follow nor an adequate number of qualified investigators to fund (National Institutes of Health, 1997b:4).

Setting Research Priorities has an extended discussion of assessing scientific opportunities (National Institutes of Health, 1997b:10-11), which notes that it "is no less complex than evaluating health needs":

It requires expertise in various scientific fields, breadth of vision across many disciplines, and judgment to determine the likely yield from making investments in particular areas of research. It is never known with certainty which scientific areas will produce the greatest returns soonest. At any given time, moreover, some fields are judged to be progressing more rapidly than others and more likely to repay the investment in them by yielding great discoveries that advance knowledge.

This discussion touches on two aspects of scientific discovery. First, at any given time, some areas of research are more promising than others because, for example, recent advances open new lines of inquiry or a new instrument makes it feasible to explore long-standing research questions for the first time. This situation is addressed by the scientific opportunity criterion. Second, other discoveries are serendipitous or turn out to have effects in areas that seemed remotely related at first. This is addressed by the next criterion, portfolio diversification.

NIH has well-established and elaborate procedures, described in Chapter 3, for assessing scientific merit in priority setting beginning with the expertise of its own staff and strongly supplemented by formal and ad hoc mechanisms for receiving advice from outside experts. The committee wants to ensure that, in NIH's decision making on programs and funding of research, NIH has adequate procedures to ensure scientific innovation and to foster support for high-risk research with high payoffs if it is successful, interdisciplinary research that crosses program and study group lines, and unconventional but promising research approaches. Some of the institutes have adopted procedures and programs for ensuring innovative research, and as noted above, innovation has been explicitly added to the list of peer-review criteria.

The NIH's portfolio must be large and diverse. Because we cannot predict discoveries or anticipate the opportunities fresh discoveries will produce, the NIH must support research along a broad—in fact, expanding—frontier (National Institutes of Health, 1997b:4).

Although it may invest differentially in areas of scientific promise, NIH also tries to invest in every health-related area of research to some degree. This prudent strategy is a hedge against the uncertainty of knowing where some if not most advances will actually occur, especially in the long run.

Each of the NIH institutes also funds a continuum of research approaches, from basic inquiries to clinical applications. "By supporting disease-related and basic research projects simultaneously, the NIH can achieve both near-term improvements in the diagnosis, treatment, and prevention of specific diseases as well as long-term discoveries in basic science that in time will produce great advances in our ability to understand, treat, and prevent disease or delay its onset" (National Institutes of Health, 1997b:9).

NIH could put all of its resources into applying current knowledge to health problems. Unfortunately, current knowledge is usually inadequate to lead to complete prevention of or cures for many diseases and conditions, although in many cases it may lead to better diagnosis and treatment or prevention options. Alternatively, NIH could invest all of its resources into long-term basic research to build the knowledge base for complete cures or prevention. This approach would leave those suffering now with no benefit from what knowledge is available, and it would not create the needed bridges from the laboratory to the real world, nor would it allow basic research to be informed systematically by clinical experience.

Although NIH should, and does, invest in the full range of research approaches (e.g., basic, translational, clinical, epidemiologic, and behavioral research), the balance among them should depend on the state of knowledge in each problem area. At its public hearing and in written submissions, the committee received testimony that certain fields of research (e.g., clinical, social, and behavioral research) are not given enough priority within NIH's overall research portfolio.

NIH is aware of this issue. The report and recommendations of the NIH Director's Panel on Clinical Research have already been discussed. The Office of Behavioral and Social Research in the Office of the Director of NIH was established in 1993 to foster greater attention to the social and behavioral aspects of health-related research. In establishing both the panel and the office, however, NIH acted only after receiving explicit directives from Congress.

The NIH must continue to support the human capital and material assets of science. To this end, the NIH's budget supports research training, acquisition of equipment and instruments, some limited construction projects, and grantee institution costs of enabling the research programs (National Institutes of Health, 1997b:4).

Finally, NIH must invest in the long-term integrity of the health research enterprise, not just the costs of current research projects. As an earlier IOM committee concluded, "the most critical and longest-term investment in the research system is the development of career scientists who contribute to the long-term success of the enterprise through both their own research efforts and their training of future generations of scientists" (Institute of Medicine, 1990:5). That committee also concluded that capital investment in adequate facilities and equipment was another necessary long-term investment. These bases of a strong research system must be developed or increased investment in research projects will have less of an impact.

Furthermore, a major change in the ethnic makeup of the nation is taking place. It is essential that NIH invest in training members of growing minority groups, because it directly affects the health of those communities. These communities offer a perspective that would enrich the research enterprise and improve priority setting. Outreach to and involvement of members of minority communities in training programs will also help make those communities part of the biomedical research system.

The testimony that the committee heard at its open meeting on April 3, 1998, or that it received in written form indicates that disease-specific interest groups are generally satisfied with NIH's priority-setting criteria, although they are not necessarily satisfied with how the criteria are applied in practice (an issue addressed in Chapter 3 on priority-setting processes).

In its discussions, the committee concluded that NIH's criteria for priority setting are reasonable and appropriate. Use of these criteria has been one of the most important factors contributing to the great success of NIH; the balanced application of these criteria has led to the accumulation of basic knowledge about human biology that is unparalleled in the history of science. This increased understanding has, in turn, led to better approaches to the diagnosis, prevention, and treatment of disease. However, the conceptualization of public health needs (Criterion 1) should be broadened beyond the medical model implied in the discussion of the criterion in Setting Research Priorities. Improving health through research goes beyond treating diseases to understanding how to promote healthy lifestyles and help those with diseases and disabilities improve their quality of life, and NIH in fact already supports such research. The committee also wants to ensure that the concept of human resources in the infrastructure criterion (Criterion 5) include outreach to underrepresented groups in biomedical and behavioral research careers.

Recommendation 1. The committee generally supports the criteria that NIH uses for priority setting and recommends that NIH continue to use these criteria in a balanced way to cover the full spectrum of research related to human health.

By beginning to elucidate these criteria, NIH has provided a potentially valuable tool for all those concerned with its priority-setting operations. Members of any group may now look to see whether their goals appear in this statement of NIH's philosophy and what weight they are afforded. To the extent that the criteria are made clear and thus understandable to members of the public, citizens can examine NIH's procedures to see if they are consonant with these stated criteria, and they can assess NIH's actions to see how well this philosophy has been realized in practice. For NIH staff, these criteria will provide guidance as they establish and implement procedures as well as when they report on their activities and accomplishments. Over time the criteria could provide clear, predictable goals for their aspirations and standards for their accountability.

Recommendation 2. NIH should make clear its mechanisms for implementing its criteria for setting priorities and should evaluate their use and effectiveness.

NIH's public health needs criterion requires systematic analysis of its portfolio in terms of public health statistics. There may be no simple relationship between health statistics (especially any single measure) and how NIH allocates funding. However, NIH should be able to show that it has considered data on health burdens and costs in a systematic fashion, in addition to considering scientific opportunity, cost differences among different kinds of research, non-NIH investments in health-related research (e.g., investments by industry, other government agencies, philanthropic institutions, and other countries), and other relevant factors. Some institutes conduct regular analyses of disease rates both to adjust future research agendas to changing trends and to assess the impacts of past research advances.^* Such analyses might be done more systematically across NIH. NIH has also compiled statistics in response to specific congressional requests (National Institutes of Health, 1997a). What is called for here is a regular process that is supported by a clear philosophy and that is sensitive to scientific understanding of the meaning of such data and their interpretation (National Research Council, 1996).

Recommendation 3. In setting priorities, NIH should strengthen its analysis and use of health data, such as burdens and costs of diseases, and of data on the impact of research on the health of the public.

Individuals and groups concerned about specific health problems or health research often use NIH-generated data on spending by specific disease or area of research to assess the overall research portfolio. Because NIH officials have said that NIH's own statistics on spending by disease are not very useful, it is incumbent on those officials to improve the quality and analysis of its data on the allocation of NIH funds by disease for planning and priority-setting purposes. NIH could approach the problem by being more systematic about deciding what to track and publish, using consistent definitions, deciding how much "related" as well as "direct" research should be included, keeping the statistics current, and ensuring quality control.

NIH should enhance its efforts at communicating and explaining the data to the public. For example, calculations of spending by disease should include not only all research directly related to the disease but also research projects on fundamental areas indirectly related to that disease. Users of the data should know that such calculations reflect the best estimates of total NIH spending in particular areas and that fundamental science is essential to discovering the etiology and progression of disease and other basic knowledge needed to develop improved means of diagnosis, improved prevention efforts, and improved treatments.

In making this recommendation, the committee is mindful of the limits on the use of such data in research priority setting. Health needs is an important factor, but there is rarely a straightforward one-to-one relationship between health needs and research funding allocations. This is shown in NIH's discussion of the limits to simple measures of health needs, as well as the need to consider other criteria. For example, the scientific opportunities for progress vary greatly across diseases in terms of the knowledge base, promising lines of inquiry, and the availability of sufficient numbers of researchers and facilities. Therefore, the amount of research support that can be linked directly to a specific disease is not by itself an adequate measure of how much or how well NIH is making progress against that disease. Nor does it reflect the potential relevance of basic research to specific diseases (e.g., research on dopamine receptors or transporters are relevant to Parkinson's disease and other diseases, even though the disease may not be mentioned in the title of the grants). As a result, NIH should improve its mechanisms for analyzing such connections to help ensure that the potential relevance of basic research to specific diseases is understood and capitalized upon.

NIH should also collect and analyze data on health research spending by others, such as other federal agencies, industry, nonprofit health organizations that fund research, foundations, or other countries. This should help identify gaps, overlaps, and opportunities for joint efforts and ensure that NIH invests wisely in areas and approaches that no one else is funding, provides the appropriate coordination, and supports the training of personnel and the other infrastructure needed in the national research enterprise.

Recommendation 4. NIH should improve the quality and analysis of its data on funding by disease and should include both direct and related expenditures.

The full and adequate implementation of Recommendations 3 and 4—to improve data collection and analysis both within NIH and in collaboration with other agencies and the private sector—will require more resources and personnel, which should be considered necessary costs, much as the costs of the peer-review process are. The need for additional resources for priority setting is addressed to Congress in Recommendation 12, which addresses current congressional limitations on research management and support budgets.

*For example, see the recent report on trends in cancer rates (Wingo et al., 1998).