Priority setting is a difficult, perennial issue in science policy, made more difficult in times of tightening research budgets. This report responds to a request from the Office of Behavioral and Social Research (BSR) at the National Institute on Aging (NIA) for advice on how best to judge the rates of progress in the research fields the office supports and to evaluate whether and how to shift the balance of research investments across fields. The request partly reflects a concern that traditional expert review processes are too strongly influenced by established disciplines and fields and too conservative in relation to the need to support research that might generate scientific breakthroughs.
In developing our recommendations, we considered available knowledge about how science makes progress, which shows great variety in types of progress and paths to progress, as well as the considerable difficulty of accurately anticipating these paths. Research areas that appear at one time to be “hot” may prove in retrospect to have been fads, and fields that appear unproductive may be stagnant, fallow, or pregnant. Accurate foresight is very difficult to achieve. We considered decision-making strategies that could address the sponsor’s concerns, along with other legitimate science policy concerns about the quality and rationality of the decision process, the accountability of decision making, and the appropriate balance of influence between scientific communities and agency science managers. Our recommendations are addressed to BSR, but we think the decision strategy we propose is appropriate for a wider range of federal science agencies within and beyond the National Institutes of Health (NIH).
Two generic decision strategies are available for assessing scientific progress and setting research priorities: (1) applying analytic techniques, such as benefit-cost analysis, bibliometric analysis, and decision analysis, and (2) using deliberative processes, such as those of expert review panels. Many methods, both analytic and deliberative, can have value for assessment and decision making, but they all also have limitations. Quantitative analytical methods typically have limitations associated with data collection, reliability, validity, cost, timeliness, and acceptability, as well as the lack of knowledge about how best to combine measures of qualitatively different aspects of scientific progress. Qualitative methods of deliberation are of unknown reliability and may be highly dependent on who is involved in the deliberations and how the questions for deliberation are framed.
Because of the uncertainties about the reliability and validity of all existing methods, we recommend a strategy for decision making that relies primarily on processes and secondarily on methods. It uses techniques based on decision research to structure and inform deliberation within groups of scientific advisers and agency decision makers and to make communication between such groups more transparent, for example, by clarifying the sources of any disagreements in judgment between them. Analytic techniques for quantifying scientific progress can provide useful input to decision-making deliberations, but they should not be used as substitutes for the necessary judgment.
We recommend a strategy that combines analysis and deliberation, in which processes of open, explicit dialogue are organized to raise all the major decision-relevant issues, allow for input from all relevant perspectives, and provide for iterative discussion between researchers and science managers and for orderly reconsideration of past decisions. Such dialogue can also provide for improved accountability of decision making.
Three principles should guide BSR practice in setting priorities across research fields:
Explicitness. Judgments about the progress and potential of scientific fields should be based on explicit consideration of them in relation to all the major scientific and societal goals of the BSR Program and all the major processes and inputs supporting progress in each field.
Perspective. Both extramural research scientists and institute program managers should be involved in assessing the progress and potential of the research fields supported by the BSR Program. Both sets of contributors to priority-setting decisions bring valuable knowledge and insights to the process, as well as different, complementary perspectives.
Iteration. Priority-setting exercises should be conducted regularly, and they should include reconsideration of past decisions.
We make the following specific recommendations for implementing these principles:
The staff of the BSR Program, with the help of the program’s scientific advisers, should develop an explicit list of scientific outcome and societal impact goals for the program in line with the strategic program goals of NIA. Information from the staff to advisory groups regarding the progress of program-supported research should reference these goals.
NIA should periodically conduct a general assessment of the BSR Program with respect to its overall adequacy for supporting the program’s scientific outcome and societal impact goals.
Assessments should be conducted approximately every four years and should consider each program goal in relation to each aspect of the BSR Program judged to be important for achieving it (e.g., the different kinds of research activities supported and modes of support).
NIA should periodically conduct an area-based assessment of the BSR Program that includes recommended priorities for new and continued support among the substantive areas of research included in the program. These efforts should explicitly assess and compare the past and potential contributions of research in each area receiving major BSR support with regard to each of BSR’s goals for scientific outcome and societal impact and with respect to the various inputs and processes that contribute to achieving the goals.
These assessments should also be conducted approximately every four years. They should make recommendations as appropriate for each area on issues of portfolio allocation between disciplinary and interdisciplinary research; basic and applied research; high-risk and low-risk research; development of research methods, of data, and of findings; support of research centers, program projects, and individual investigators; and support of research, infrastructure, and human resources development.
The BSR program director should consider the area-based assessments and recommendations carefully in reallocating funds among fields. One year after completion of each area-based assessment, BSR staff should report on decisions reached and actions taken that involve priority setting among research areas and portfolio allocation within areas. The report should explicitly discuss the justification for program decisions that might seem inconsistent with the assessment’s recommendations. The report should be delivered to the NIA director and the NIA advisory council.
There can be good justifications for institute decisions that deviate from the recommendations of a body of scientists. The purpose of the recommended report is to ensure that such justifications are made explicit and thus to provide increased accountability in an institutional sense and a continuing rational dialogue among scientists and program managers, focused on the program’s objectives.
The NIA BSR Program, together with the rest of NIA and NIH, as well as the National Science Foundation and other federal science agencies, should support a coordinated program of research to promote well-informed, high-quality research policy making.
This research program would provide knowledge of broad value to federal science policy and contribute to development of what has been called a “social science of science policy.” It is for this reason that we recommend that a broad range of federal science agencies support this research program. The research should pursue three objectives:
Improve basic understanding of scientific progress and the roles of research funding agencies in promoting it. Research pursuing this objective would examine the nature and paths of progress in science, including the roles of decisions by science agencies. It might include historical analyses of the evolution of scientific fields; advanced bibliometric analyses of the development of research fields over time and the flows of influence among them; studies of the effects of the structure of research fields on their progress; studies of the roles of officials in science agencies in scientific progress; studies of how expert advisory groups, including study sections and advisory councils, make decisions affecting scientific progress; and studies of the effects of the organization of such groups on their success at promoting interdisciplinary and problem-focused scientific activity and ultimately at improving scientific outcomes and societal impacts. In the case of BSR, the research should focus on progress in fields of behavioral and social science related to aging.
Improve understanding of the uses of quantitative decision aids in making research policy decisions. This research should include the development, trial use, and empirical investigation of the use of quantitative measures and decision-analytic techniques as inputs to priority setting. It should not seek techniques that can supplant deliberation, because different areas of science make different kinds of progress and judgment will always be required to assess progress against multiple objectives. The research would aim to identify useful techniques and determine how to use them effectively. The research might include studies to assess the value of providing information developed through specific analytic techniques; studies comparing
indicators of scientific progress with each other and with unaided expert judgment; comparative quantitative studies of fields that are widely judged to differ in rates of progress; tests of ways to combine information from different analytic methods; and studies of the use of qualitative decision analytic techniques for guiding deliberation.
Develop useful techniques for systematic deliberation in advisory and decision-making procedures. This research would explore and assess techniques for structured deliberation, some of them including the use of indicators of scientific progress and potential, for retrospective assessment and for priority setting. It would be used to elaborate and refine deliberative methods now in use and those recommended in this study. It should include studies that apply techniques for structuring deliberation to the research priority-setting tasks facing BSR; studies of trials in which review and advisory panels are instructed or trained to focus their deliberations on how each research field might contribute to specified program objectives or goals, including both those related to scientific quality and to mission relevance; studies of attempts to adapt the NIH Consensus Development Conference model to research priority setting; studies comparing advisory panels of different composition; and studies of the effects of instruction and training of advisory panel members to consider the full range of BSR and NIA objectives.