A Strategy for Assessing Science: Behavioral and Social Research on Aging (2007)

The Behavioral and Social Research (BSR) Program at the National Institute on Aging (NIA) asked the National Research Council to undertake this study to “explore methodologies for assessing the progress and vitality of areas of behavioral and social science research on aging, and to identify the factors that contribute to the likelihood of discoveries in areas of aging research.” The ultimate purpose was to “seek practicable approaches that can help research managers improve their judgments and research portfolios.”

These purposes are perennial in U.S. science policy. They are important not only to BSR, but also to many other science and technology research organizations across the federal government, to scientific communities, and to the science policy community. Our findings and recommendations are intended as a direct response to the questions posed by BSR, but we have considered this response in this larger national context based on the thinking that it may be useful to other federal agencies that provide support for scientific research.

Long-established procedures for determining research priorities and allocating research funds in federal science agencies are increasingly being questioned for several reasons.

• Tighter funding: In an environment of projected static or declining research budgets for other than national defense and homeland security, proposals to open up new areas of scientific inquiry, support currently dynamic fields, and support the increased costs of existing lines of research become

competitive; with increasing force, they imply reallocations of funds, including possible reductions in support for long-established fields.

• Increased demands for accountability and documented performance of all federal agencies, including research agencies, which are increasingly accompanied by calls for the use of quantitative performance measures.

• A belief that existing peer review procedures are unduly conservative in identifying or supporting transformative, interdisciplinary, and translational research that truly presses against the frontiers of science or integrates research findings with clinical applications.

• Science program managers’ efforts to justify their decisions, especially when these involve launching new scientific initiatives with static budgets or adjusting program priorities in ways that may conflict with recommendations from scientific advisory groups.

• Developments in analytical methods, databases, and statistical and data mining techniques that promise better ways to assess the impacts of lines of research on knowledge and agency societal objectives.

These pressures on standard decision processes come from inside and outside federal science agencies. They reflect the desire of agency officials to be more proactive, entrepreneurial, and responsive both to the dynamics of scientific discovery and to external pressures on the agency. They also reflect the concerns of some sectors of the scientific community, including established researchers who are seeking to extend their work into new fields and newer researchers seeking to venture into relatively uncharted domains, about the rigidity and conservatism of established review mechanisms.

Multiple variations on existing procedures are being considered. These include changing the composition of review panels; changing the criteria, methods, and means by which review panels function; and changing the relative decision-making authority of review panels and agency officials. Pervading all considerations about changes to existing peer review procedures is the recurrent concern that any such changes should not detract from the workings of a national system for allocating federal funds for research that has historically been associated with the rise to preeminence of U.S. science and graduate education.

The merit-based, peer review procedures that have become traditional in NIA and many other science agencies reflect a political consensus about the societal utility of allowing the “republic of science” to rule within constraints defined by national priorities set through budgetary and regulatory processes. Merit-based, peer review procedures serve not just as sorting mechanisms to generate the best science; they also provide essential safeguards against the insertion of patronage or ideological factors into the selection of research proposals and research performers. Consideration of

changes to current peer review procedures, whatever may be their merit in terms of opening up selection processes to newer, fresher, more relevant theories and approaches, must not lose sight of the larger institutional and inherently political context of federally sponsored research. One of the key issues is how particular procedural changes might alter the distribution of power and influence between scientists and agency administrators and research managers.

Alternatives to peer review, although increasingly discussed, have to date been tried only by limited implementation of variations on standard practices. Thus, very little is presently known about the likely impacts of such changes on scientific performance or attainment of agency mission objectives. All proposals for change thus should be considered as hypotheses that improved outcomes will follow upon their adoption; logically and necessarily then, implementation of such changes should be accompanied by careful and systematic evaluation to determine whether or not they produce the desired results.

1. The scientific base for conducting valid and accountable assessments of the progress of scientific fields and for supporting research policy decisions is seriously underdeveloped. Despite the existence of a considerable body of historical case research, little systematic knowledge exists about the paths of the development of science, particularly behavioral and social science with applications to health and well-being; about the roles of government agency decisions in that progress; about the possibility of accurately measuring and assessing such progress; about the best ways to use analytic approaches to improve decision making; or about the best ways to structure decision making to take advantage of information from studies of science.

2. No theory exists that can reliably predict which research activities are most likely to lead to scientific advances or to societal benefit. It is for this reason that the case for expert judgment continues to remain persuasive. Evidence of past performance and current vitality, that is, heightened interest among scientists in a topic or line of research, are imperfect predictors of future progress. Thus, any choice to support an emerging research direction is speculative. Scientific managers can best defend such choices by developing an explicit rationale for allocating funds among established and emerging fields and for making choices, particularly involving the latter.

3. Science produces diverse kinds of benefits by diverse mechanisms that are not well understood. Consequently, assessing the potential of scientific fields or lines of research is a complex problem of multiattribute decision making under uncertainty. Investment strategies suitable for uncertain conditions are therefore appropriate for managing the BSR portfolio. Re-

search activities can advance science on several dimensions, which we have summarized under the broad categories of discovery, analysis, explanation, integration, and development, and they can contribute to society on several other dimensions (identifying issues, finding solutions, informing choices, and educating the society). Every research policy choice and every research activity will have its own profile with regard to how much progress it is making across these dimensions, and there is no agreed weighting of the importance of one against another. Consequently, judgment is required to assess the evidence regarding how science is progressing toward each objective, as well as to consider the weight to be given to progress toward each one. The policy implications of a finding that a line of research is or is not currently making much progress on one or more dimensions are not self-evident. Hot areas may prove in retrospect to have been fads. A field that appears unproductive may be stagnant, fallow, or pregnant. Telling which is not easy.

Given BSR’s environment of complexity and uncertainty, the following investment strategies seem appropriate:

1. Portfolio diversification strategies that involve investment in multiple fields and multiple kinds of research. Such strategies can allow research managers to minimize the consequences of overreliance on any single presumption about what kinds of research are likely to be most productive. Diversification is also advisable in terms of the kinds of evidence relied on to make decisions about what to support. For example, when quantitative indicators and informed peer judgment suggest supporting different lines of research, it is worth considering supporting some of each.

2. Investing where the investment is most likely to add value. For example, although directly contributing to major discoveries remains the gold prize of federal science agencies, more indirect methods, such as supporting improvements in databases and analytic techniques, integrating knowledge across fields and levels of analysis, calling attention to underresearched questions, and facilitating the entry of new people to work on old and new research problems, can yield high scientific and societal returns. By promoting scientific analysis, integration, and development, research managers can contribute indirectly to discovery and explanation. Research managers should also consider favoring support to research organizations or in modes that have been shown to have characteristics that are likely to promote progress. And BSR managers may reasonably prefer to support research in fields that need only small investments from NIA to leverage funds from other sources or in which BSR seems the only viable sponsor for the research.

3. Support for issue-focused interdisciplinary research. Interdisciplinary research has significant potential to advance scientific objectives that

research management can promote, such as scientific integration and development and scientists’ attention to societal objectives of science consistent with BSR’s mission. Moreover, BSR has a good track record of promoting these objectives through its support of selected areas of interdisciplinary, issue-focused research.

4. Both working extramural scientists and NIA program managers have essential perspectives to contribute to research priority setting. For example, extramural scientists often have a keener understanding of the theoretical and methodological quality of research in their areas of expertise and of which research problems are tractable given existing data and methods. By contrast, NIA program managers may have a keener understanding of the potential for linking recent developments in disparate fields that are not yet communicating, and of the ways certain lines of research might influence policy decisions in the health sector. It follows that both groups should have roles in priority setting and that an exchange of ideas among their various perspectives can promote enlightened priority setting.

5. None of the available analytical methods of science assessment is sufficiently valid to justify its use for assessing scientific fields or setting priorities among them. Judgment must be applied to interpret the results from these methods and discern their implications for policy choices. This situation seems unlikely to change any time soon. Although analytic techniques aimed at quantifying scientific progress can provide useful input to decision-making deliberations in BSR, they should not be used as substitutes for judgment or deliberation.

Analytical methods for assessing scientific progress and potential are those that use “rigorous, replicable methods developed by experts” (National Research Council, 1996:20). In science priority setting, these include bibliometric analysis and decision-analytic techniques such as benefit-cost analysis. Their inadequacies as aids to decision reflect (a) uncertainties inherent in projecting the future development of scientific fields on the basis of their past performance, (b) uncertainties and unknowns concerning the relationships between measurable scientific activity and the kinds of outcomes sought by the NIA, and (c) the difficulties of comparing research activities that are likely to contribute to different Institute objectives, such as scientific understanding and societal benefit. In addition, quantitative analytical methods typically have limitations associated with data collection, reliability, validity, cost, timeliness, and acceptability, as well as the lack of knowledge about how best to combine measures of qualitatively different aspects of scientific progress.

Resistance from federal science agencies and their advisory groups to the introduction of analytic techniques into decision-making processes partly

reflects a concern that they may be misapplied or applied to the exclusion of good judgment. Indeed, the use of these methods in a reductionist, bureaucratic approach to priority setting and assessment potentially threatens the validity of research assessment and, ultimately, the vitality of the U.S. research system. Resistance to the use of quantitative methods may also reflect the possibility that the use of techniques that can be applied without relying on the judgment of scientists will alter the distribution of authority and influence among such parties as agency officials, program managers, study group panels, and individual reviewers. Acceptance or rejection of specific methods may therefore reflect matters of organizational politics as well as evaluation methodology. These are legitimate concerns.

6. Despite the many limitations of analytic techniques for assessing science, judgments in NIA that involve comparisons among fields of behavioral and social science research can be more systematic, more accountable, and more strongly defensible if they are informed by appropriate use of systematic analytic techniques. Although no analytic technique is sufficiently developed to replace judgment, judgment can be disciplined and enhanced by careful analysis. Analytic techniques should have two main roles: (1) to help structure the deliberations about research priorities by scientific advisory groups to BSR and by the program’s decision-making bodies and (2) to help structure communication between institute officials and their scientific advisers about priority setting (e.g., by clarifying the sources of any disagreements in judgment between them).

We are saying that neither judgment nor any foreseeable analytic technique provides a gold standard for science priority setting. However, we think that wise integration of analysis and judgment may yield better results than either approach unaided by the other. We consider it possible to constitute expert review panels that draw on their own experiences and insights, augmented by quantitative data on the outputs, outcomes, impacts, productivity, or quality of research, to arrive at better informed and more systematically considered expert judgments about the progress and prospects of scientific fields than they could reach without quantitative data. A key to the effective use of analytic techniques is to embed them in deliberative processes that explicitly consider clearly specified decision objectives and that focus deliberation on the expected relationships of particular decision options to particular objectives. Structuring deliberation in this way can encourage more explicit consideration of how particular kinds of research may promote specific goals of the BSR Program and provide a way of thinking carefully about what analytic techniques do and do not offer. Structured deliberation can be usefully employed in advisory groups, in program management settings, and for structuring communication about priority setting between BSR and its advisers.

On the basis of our consideration of the state of knowledge about scientific assessment and priority setting and of the specific programmatic setting of BSR, we have arrived at a set of principles for priority-setting activities. These principles undergird a strategy for decision making that relies primarily on institutionalizing processes for coping with uncertain knowledge and secondarily on analytic techniques intended to reduce uncertainty and aid deliberation on matters of judgment.

The best way to make good, defensible priority-setting decisions is through processes of open, explicit dialogue that are organized to raise all the major decision-relevant issues, allow for input from all relevant perspectives, and provide for iterative discussion between researchers and science managers and for orderly reconsideration of past decisions. Such dialogue can also improve the accountability of decision making. Thus, we recommend adopting processes that can collect the best available information about the progress and prospects of the areas of science being considered; use systematic procedures to consider, interpret, and discuss the import of this information for the decisions; involve extramural scientists, agency officials, and user communities in the decision processes; encourage them to consider the information in the light of all of BSR’s strategic objectives; give advisory group discussions significant weight as input to decisions within the institute; allow science managers to reallocate funds among lines of research in light of advisory group judgments and their own; and allow the institute to learn from the results of its decisions.

Three principles should guide BSR practice in setting priorities across research fields:

1. Explicitness. Judgments about the progress and potential of scientific fields should be based on explicit consideration of them in relation to all the major scientific and societal goals of the BSR Program and all the major processes and inputs supporting progress in each field.

BSR, like many other federal science agencies with both scientific and societal objectives, must consider disparate objectives in assessing scientific progress and setting priorities. Box 2-1 shows the most recent articulation of the research goals of NIA. Different lines of research are likely to have different profiles in terms of how much they contribute to these disparate goals. Priority setting involves consideration, either explicitly or implicitly, of how much each line of research is likely to contribute to each goal, as well as about the relative importance of the goals. Moreover, different lines of research require different inputs, and these needs must also be considered separately for each field if rational priority setting is to occur. Making such considerations more explicit and deliberating about the bases for judgments,

including disagreements about judgments, will contribute to better considered and better justified decisions and recommendations for action.

The principle of explicit consideration is based in part on research on decision making that shows the limitations of unstructured judgments about multiattribute choices and the potential benefits of explicit consideration of all the relevant objectives and the values associated with them. In addition, we think that explicit consideration will make decisions in BSR more accountable and contribute to high-quality communication between researchers and science managers. The principle should be applied to recommendations made by BSR’s advisory council and other groups organized to advise on priority setting. It should also be applied to internal decision making by BSR Program managers.

2. Perspective. Both extramural research scientists and institute program managers should be involved in assessing the progress and potential of the research fields supported by the BSR Program.

Wise priority-setting requires both depth and breadth of view, as well as concern with both the scientific and societal goals of research. For these reasons and others, it requires input both from research scientists and program managers, as well as an appropriate and productive balance of influence and power among them. Both sets of contributors to priority-setting decisions bring valuable knowledge and insights to the process, but they have different perspectives—sometimes conflicting, but nevertheless complementary. Research scientists typically see the value of research in their own fields more easily than they see the value of research in distant fields. Compared with program managers, they typically have a deeper understanding of the science in their fields, but a narrower range of view across fields. They may also give more weight than program managers do to scientific goals relative to other agency mission goals. Program managers are frequently better able to gauge the relevance of a line of research to agency mission objectives. They are typically better positioned to observe converging or intersecting advances or trends in science that fall between or at the outer boundaries of existing disciplines, and they are manifestly better positioned to support embryonic lines of inquiry. They are well positioned to observe the extent to which fields nurture one another: to observe which mainstreams spill over their banks to contribute to the productivity of other fields and which evolve in ways that produce value only to those in the field, with limited contribution either to the broader scientific enterprise or to the BSR mission.

3. Iteration. Priority-setting exercises should be conducted regularly, and they should include reconsideration of past decisions.

Scientific priority-setting decisions are made under conditions of uncertainty: the larger, more discontinuous the scientific advance being sought, the higher the risk. Moreover, because of the likelihood of surprise in the history of science and of change in societal priorities, even the most judiciously considered decisions need to be revised from time to time. The process of reconsideration should include reflection on the value of the analytical information provided about scientific progress to inform those decisions and the adequacy of the methods for producing that information.

We make five recommendations for implementing the above principles to develop a stronger scientific basis for research priority setting and to strengthen the basis of priority setting in BSR both before and after research results come in. We think these recommendations can be adapted for use in other science agencies that, like BSR, have multiple objectives to advance science and benefit society.

1. The staff of the BSR Program, with the help of the program’s scientific advisers, should develop an explicit list of scientific outcome and societal impact goals for the program in line with the strategic program goals of NIA. Information from the staff to advisory groups regarding the progress of program-supported research should reference these goals.

Box 2-1 lists NIA’s research goals; the goals of BSR are likely to be consistent with these, but more specific and perhaps more elaborated. Chapter 5 includes a list of dimensions of scientific progress that may help BSR elaborate its list of scientific outcome goals. It also suggests procedures by which BSR might identify more specific sets of scientific objectives for the program, as well as factors believed to contribute to achieving these objectives. The BSR staff already provides useful information about research progress to the advisory council—for example, in its stories of discovery and reports on science advances. We recommend that staff frame such information at least in part in terms of explicit program goals, thus promoting a more focused discussion of the kinds of scientific progress being made in each of the lines of research BSR supports and the kinds of future advances that might be expected. Explicit consideration of program goals may lead BSR staff to develop additional information beyond what is now provided to the council.

We recognize that efforts to increase accountability by justifying program activities in terms of broader goals can easily degenerate into bureaucratic exercises aimed only at complying with administrative mandates. The intent of this recommendation is not to call for new administrative require-

ments, but rather to guide ongoing science policy deliberations toward more explicit and focused discussion of how particular activities may or may not advance program goals.

2. NIA should periodically conduct a general assessment of the BSR Program with respect to its overall adequacy for supporting the program’s scientific outcome and societal impact goals.

Assessments should be conducted approximately every four years, either as part of the regular quadrennial review of the BSR Program or as a separate activity by an ad hoc advisory group organized by the NIA advisory council. General assessments should explicitly consider each program goal in relation to each aspect of the BSR Program judged to be important for achieving it (e.g., the different kinds of research activities supported and modes of support). The assessments may consider indicators of scientific output (for example, bibliometric measures of publications and citations or evidence of presentations at scientific conferences), but, as they do so, the meaning of output indicators should be discussed explicitly in light of the program’s scientific outcome and societal impact goals. Outputs should not be taken as evidence in themselves of progress toward scientific or societal goals. The assessments should consider the value added by NIA investment vis-à-vis investments in the same fields by other agencies and the private sector. On the basis of these considerations, the group conducting the assessment should advise on ways to alter processes or inputs in the BSR Program if this is needed to advance the goals. Following the perspective principle, these periodic assessments should be based on input from both research scientists and program managers. Following the principle of iteration, the assessments should include reexamination of past assessments.

3. NIA should periodically conduct an area-based assessment of the BSR program that includes recommended priorities for new and continued support among the substantive areas of research included in the program. These efforts should explicitly assess and compare the past and potential contributions of research in each area receiving major BSR support with regard to each of BSR’s goals for scientific outcome and societal impact and with respect to the various inputs and processes that contribute to achieving the goals.

Area-based assessments should be conducted approximately every four years, either as part of the regular NIA review of BSR or as a separate activity organized by the NIA advisory council. These assessments should consider information provided by BSR staff about progress in each area or line of research, referenced to program goals. Indicators of scientific output

in each area should be collected and reviewed. The group conducting the assessment should recommend priorities after considering the progress of each area, BSR’s inputs to the area, and inputs from outside BSR that may affect the value added by BSR contributions. Following the principles of perspective and iteration, it should draw on input from both research scientists and program managers and reflect on and reexamine past assessments. It should consider and make recommendations as appropriate for each area on issues of portfolio allocation between disciplinary and interdisciplinary research; basic and applied research; high-risk and low-risk research; development of research methods, of data, and of findings; support of research centers, program projects, and individual investigators; and support of research, infrastructure, and human resources development.

In conducting area-based assessments, NIA should explicitly consider BSR’s activities in each area against each of the program’s scientific impact and societal outcome goals, perhaps by organizing discussions around specific, stated objectives and explicitly addressing the issue of how to combine objectives in making recommendations. We caution, however, about adopting procedures that add together individual or group judgments of how research contributes to each BSR outcome goal multiplied by a weight for each goal.

Our caution results from the judgment that the appropriate way to integrate multiple goals is not always additive. A line of research that clearly produces desired societal benefits may be highly justified even if it does little to advance basic understanding. The reverse is also true: a line of research that generates a breakthrough in basic understanding may be highly justified even if it produces no current or foreseeable practical benefits for human health and well-being. All of BSR’s program goals should be considered in making area-based assessments, but the group conducting the assessment should have flexibility in how it does that. It should, however, be explicit in its assessment reports about how it addressed the multicriteria aspect of the assessment.

4. The BSR Program director should consider the area-based assessments and recommendations carefully in reallocating funds among fields. One year after completion of each area-based assessment, BSR staff should report on decisions reached and actions taken that involve priority setting among research areas and portfolio allocation within areas. The report should explicitly discuss the justification for program decisions that might seem inconsistent with the assessment’s recommendations. The report should be delivered to the NIA director and the NIA advisory council.

This report should not be construed as a mechanism for tying BSR decisions tightly to the recommendations of the assessment group. There

can be good justifications for institute decisions that deviate from the recommendations of a body of scientists. The purpose of the report is to ensure that such justifications are made explicit and open to question, thus providing increased accountability in an institutional sense and promoting a continuing rational dialogue among scientists and program managers, all of whom want to make the BSR program effective and productive, focused on the program’s objectives. We presume that the NIA advisory council may comment on these staff reports either informally or in subsequent formal assessments, thus maintaining the iterative process of rational deliberation that involves both science managers and extramural scientists.

5. The NIA BSR Program, together with the rest of NIA and the National Institutes of Health, as well as the National Science Foundation and other federal science agencies, should support a coordinated program of research to advance well-informed, high-quality research policy making.

This research would support the need recently articulated for a social science of science policy that would improve the knowledge base for setting priorities for scientific research (Marburger, 2005). A multiagency effort is desirable because it can yield benefits that all science agencies can share, such as general lessons about advancing interdisciplinary and mission-relevant science and improved understanding about which aspects of scientific progress are generic and which are domain- or discipline-specific. The research would aim to achieve three objectives:

1. Improve basic understanding of scientific progress and the roles of research funding agencies in promoting it. Research pursuing this objective would examine the nature and paths of progress in science, including the roles of decisions by science agencies. It might include historical analyses of the evolution of scientific fields that have differed in their productivity; advanced bibliometric analyses that examine scientific outputs in relation to measures of inputs and outcomes, the cross-fertilization of research findings, and the emergence of new fields of knowledge; and studies of how the structure of research fields affects their progress. Research on the roles of science agency decisions in scientific progress might include studies of the role of officials in science agencies as entrepreneurs and stewards of scientific fields; studies of how expert advisory groups, including study sections and advisory councils, make decisions affecting scientific progress; studies of the effects of the organization of advisory groups on their success at promoting interdisciplinary and problem-focused scientific activity, and ultimately at improving scientific outcomes and societal impacts. In the case of BSR, the research should focus on progress in fields of behavioral and social science related to aging.

2. Improve understanding of the uses of quantitative decision aids in making research policy decisions. This research should include the development, trial use, and empirical investigation of the use of quantitative measures and decision-analytic techniques as inputs to priority setting. It should not seek techniques that can supplant deliberation, because different areas of science make different kinds of progress and judgment will always be required to assess progress against multiple objectives. The research would aim to identify useful techniques and determine how to use them effectively. The research might include studies to assess the value of providing information developed through specific analytic techniques (e.g., bibliometric analysis, techniques derived from decision analysis) to the deliberations of expert review groups; studies comparing multiple indicators of research activity, outputs, outcomes, or impacts and indicators of scientific progress with each other and with unaided expert judgment to seek reliable and valid quantitative measures of scientific progress and to determine whether any of them might be useful as leading indicators; comparative quantitative studies of fields that are widely judged to differ in rates of progress to identify quantitative indicators that yield results consistent with expert judgment; tests of ways to combine information from different analytic methods; and studies of the use of techniques of decision structuring and multiattribute evaluation for guiding deliberation about research priority setting.

3. Develop useful techniques for systematic deliberation in advisory and decision-making procedures. Research pursuing this objective should explore and assess techniques for structured deliberation, some of them including the use of indicators of scientific progress and potential, for retrospective assessment and for priority setting. It should include studies to apply techniques for structuring deliberation to the research priority-setting tasks facing BSR, possibly using simulated advisory groups; trials of such techniques in which review and advisory panels are instructed or trained to focus their deliberations on how each research field might contribute to specified program objectives or goals, including both those related to scientific quality and to mission relevance; studies of attempts to adapt the NIH Consensus Development Conference model to research priority setting; comparative studies of advisory panels or simulated advisory panels of different composition to test hypotheses about the limitations of current review panels in recommending research priorities; and studies of the effects on decision making of instruction and training of advisory panel members to consider the full range of specific BSR and NIA objectives.