National Academies Press: OpenBook

Implementing the Government Performance and Results Act for Research: A Status Report (2001)

Chapter: APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act

« Previous: APPENDIX D Summary of Workshop
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 155
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 156
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 157
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 158
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 159
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 160
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 161
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 162
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 163
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 164
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 165
Suggested Citation:"APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2001. Implementing the Government Performance and Results Act for Research: A Status Report. Washington, DC: The National Academies Press. doi: 10.17226/10106.
×
Page 166

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

APPENDIX E Executive Summary of EVALUATING FEDERAL RESEARCH PROGRAMS: Research and the Government Performance and Results Act he Government Performance and Results Act (GPRA), enacted in T 1993, focuses agency and oversight attention on the performance and results of government activities by requiring that all federal agencies measure and report on the results of their activities annually. Agencies are required to develop a strategic plan that sets goals and objectives for at least a 5-year period, an annual perfor- mance plan that translates the goals of the strategic plan into annual targets, and an annual performance report that demonstrates whether the targets are met. The Committee on Science, Engi- neering, and Public Policy (COSEPUP) of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine has addressed the issue of measuring and evaluating research in compliance with the requirements of GPRA. COSEPUP recognizes the opportunities and challenges that GPRA presents for agencies that invest in research. GPRA offers those agencies the opportunity to communicate to policy- makers and the public the rationale for and results of their research programs. At the same time, GPRA presents substantial challenges to the agencies. During the course of this study, COSEPUP held several workshops. In these workshops and in other input to the commit- tee, we have heard two distinct and conflicting viewpoints on approaches to measuring basic research. One is that it should be possible to measure research, including basic research, annually and provide quantitative measures of the useful outcomes of both basic and applied research. The other is that, given the long-range 155

IMPLEMENTING THE GOVERNMENT PERFORMANCE AND RESULTS ACT FOR RESEARCH nature of basic research, there is no sensible way to respond to the GPRA annual measurement requirement and that the best that can be done is to provide measures that appear to respond but in fact are essentially meaningless, such as a list of an agency’s top 100 discoveries of the preceding year. COSEPUP’s view, spelled out in more detail in what follows, is different from both those viewpoints. In essence, our report takes two strong positions. First, the useful outcomes of basic research cannot be measured directly on an annual basis, because the usefulness of new basic knowledge is inherently too unpredictable; so the usefulness of basic research must be mea- sured by historical reviews based on a much longer timeframe. Second, that does not mean that there are no meaningful measures of performance of basic research while the research is in progress; in fact, the committee believes that there are meaningful measures of quality, relevance, and leadership that are good predictors of eventual usefulness, that these measures can be reported regularly, and that they represent a sound way to ensure that the country is getting a good return on its basic research investment. The problem of reporting on applied research is much simpler: it consists of systematically applying methods widely used in industry and in some parts of government. For example, an applied research program usually includes a series of milestones that should be achieved by particular times and a description of the intended final outcomes and their significance. Periodic reporting can indicate progress toward those milestones. The remainder of this executive summary provides a more in-depth description of COSEPUP’s conclusions and recommenda- tions regarding how to evaluate federal research programs relative to GPRA. It also addresses coordination among federal research programs and human-resource issues. COSEPUP concludes that both basic research and applied research programs1 can be mean- ingfully evaluated on a regular basis. For the applied research programs of the mission agencies, specific practical outcomes can be 156

Executive Summary of Evaluating Federal Research Programs documented and progress toward their achievement can be mea- sured annually. For example, if the Department of Energy adopted the goal of producing cheaper solar energy, it could measure the results of research directed toward decreasing the cost of solar cells; this applied research project would be evaluated annually against specific measurable milestones. However, the practical outcomes of basic research in science and engineering can seldom be identi- fied while the research is in progress. Basic research has annual results that can be meaningfully evaluated, but these evaluations often do not give even a hint of ultimate practical outcomes. History tells us unmistakably that by any measure, the benefit to the United States for leadership in basic research is extremely high—lives saved, inventions fostered, and jobs and wealth created. History also shows us how often basic research in science and engineering leads to outcomes that were unexpected or took many years or even decades to emerge. COSEPUP strongly believes that measures of the practical outcomes of basic research usually must be retrospective and historical and that the unpredict- able nature of practical outcomes is an inherent and unalterable feature of basic research. For example, pre-World War II basic research on atomic structure contributed to today’s Global Position- ing System, an outcome of great practical and economic value, but, attempts to evaluate a year’s worth of that early research even if they demonstrated high quality and world leadership, would have contained no hint of this particular outcome. Since we cannot predict the ultimate practical outcomes of basic research, we must find ways to ensure that the basic research programs that the nation funds generate the kinds of knowledge that have given us great practical benefits in the past. To do that, we must find ways to measure the quality of our current research programs, their contributions to our world leadership in the relevant fields, and their relevance to agency goals and intended users. World leadership is an important measure. In an earlier report (COSEPUP, 1993), COSEPUP recommended that, for the 157

IMPLEMENTING THE GOVERNMENT PERFORMANCE AND RESULTS ACT FOR RESEARCH sake of the nation’s well-being, the United States be among the leaders in all major fields of science and pre-eminent in selected fields of national importance. That is because a nation must be performing research at the forefront of a field if it is to understand, appropriate, and capitalize on current advances in that field, no matter where in the world they occur. New knowledge has value to nations where highly educated people performing cutting-edge research in the field of discovery can make use of the new knowl- edge when practical outcomes appear possible. The people best qualified to evaluate basic or applied research are those with the knowledge and experience to under- stand its quality, and, in the case of applied research, its connection to public and agency goals. Evaluating basic research requires substantial scientific or engineering knowledge. Evaluating applied research requires, in addition, the ability to recognize its potential applicability to practical problems. With those considerations in mind, COSEPUP has reached six conclusions and offers six recommendations regarding the evaluation of federally supported research programs. 158

Executive Summary of Evaluating Federal Research Programs For applied research programs, progress toward specified practical outcomes can usually be measured annually by using milestones and other fairly standard approaches common in indus- try and in some parts of the federal government. For basic research, in contrast, progress toward practical outcomes cannot be measured annually, and attempts to measure such progress annually can in fact be harmful. Basic research progress can be reported annually in terms of quality, leadership, and relevance to agency goals, but practical outcomes can be measured only against a far longer historical perspective. In practical terms, because quality, leader- ship, and relevance will usually change slowly, the GPRA annual- reporting requirement can usually be met by minor updating of full evaluations that are done in a more flexible timeframe. There is a much greater chance that important events will take place in subfields, because of either scientific events or funding changes, so subprogram changes should constitute much of the updating. Different expertise is required for measuring the worth of applied research and the worth of basic research. Measuring both requires technical and scientific knowledge, but applied research entails some factors that basic research does not, such as ultimate usability, so the input of potential users is required. That leads to our next conclusion. Expert review is widely applied—used, for example, by congressional committees, by other professions, by industry boards, and throughout the realm of science and engineering—to answer complex questions through consultation with expert advisers. It is 159

IMPLEMENTING THE GOVERNMENT PERFORMANCE AND RESULTS ACT FOR RESEARCH useful in helping an agency answer three kinds of questions of particular relevance to GPRA: • What is the quality of the research program—for ex- ample, how good is current research work compared with other work being conducted in the field?2 This question is best an- swered by reviewers who are sufficiently expert in the field being assessed to perform a quality review. This approach is traditionally called peer review. Peer review is commonly applied to projects, but here we are applying it to programs. The talent, objective judgment, and experience of these experts, or peers, are paramount and should be the criteria for their selection. • Is the research program focused on the subjects most relevant to the agency mission? Another form of expert review is relevance review, in which potential users, joined by experts in related fields, evaluate the relevance of research to agency goals— is the research on subjects in which new understanding could be important in fulfilling the agency’s mission? In reviewing the relevance of a program, a panel would assess the appropriateness of the direction of the research to the agency mission and its potential value to intended users. • Is the research being performed at the forefront of scientific and technological knowledge? This is a relevant question for many programs, but it is particularly important for whole fields and subfields being supported. Evaluations of fields and subfields is best done through international benchmarking by a panel of experts who have sufficient stature and perspective to assess the interna- tional standing of research. For agencies whose missions include a specific responsibil- ity for basic research—such as the National Science Foundation in broad fields of science and engineering, the National Institutes of Health in fields related to health, or the Department of Energy in high-energy physics—world leadership in a field can itself be an 160

Executive Summary of Evaluating Federal Research Programs agency goal. That is equally true for mission agencies, such as Department of Defense (DOD) but in more focused ways. For example, DOD can take as a goal world leadership in basic materi- als research relevant to its mission. Once such a goal is established, the usual measures of quality and leadership should be applied. Federal agencies that support research and exploit its results are able to do so because the education and training pro- grams of the universities, in the course of performing much of that research, and the federal laboratories provide a continuing flow of qualified scientists and engineers. Even though section 1115(a)(3) of GPRA requires agencies to describe the human resources required to meet their performance goals, few agencies describe the importance of human resources or propose ways to ensure their adequacy in their strategic or performance plans. It is common and valuable for agencies to approach similar fields of research from different perspectives. Indeed, this pluralism is a major strength of the U.S. research enterprise. But, better communication among agencies would enhance opportunities for 161

IMPLEMENTING THE GOVERNMENT PERFORMANCE AND RESULTS ACT FOR RESEARCH collaboration, help keep important questions from being over- looked, and reduce instances of inefficient duplication of effort. Present mechanisms need strengthening. The researchers who work in agency, university, and industrial laboratories are the people who perform and best under- stand the research programs funded by the federal government. Many researchers contribute substantial time and effort to review- ing papers submitted for publication, grant applications, and program proposals, yet few of them are aware of GPRA, its objec- tives, and its mandates. Increased contact with and advice from the broader scientific and engineering community regarding the methods of determining and reporting quality and regarding the leadership position of agency research programs and the relevance of research to agency missions can benefit the GPRA process. On the basis of those conclusions, COSEPUP offers the following recommendations: The performance of research is critical to the missions of many federal agencies. Therefore, a full description of an agency’s goals and results, which is a principal objective of GPRA, must contain an evaluation of research activities and their relevance to the agency’s mission. 162

Executive Summary of Evaluating Federal Research Programs Because the evaluation of applied research is directly connected to practical outcomes, whereas the evaluation of basic research is in terms of quality, relevance, and leadership, which ultimately lead to practical outcomes, there might be a tendency to bias an agency’s overall research program toward applied research at the expense of basic research. This should be avoided, and a proper balance should be maintained. The most effective way to evaluate research programs is by expert review. The most commonly used form of expert review of quality is peer review. This operates on the premise that the 163

IMPLEMENTING THE GOVERNMENT PERFORMANCE AND RESULTS ACT FOR RESEARCH people best qualified to judge the quality of research are experts in the field of research. This premise prevails across the research spectrum, from basic research to applied research. A second form of expert review is relevance review, in which potential users and experts in other fields or disciplines related to an agency’s mission or to the potential application of the research evaluate the relevance of research to the agency’s mission. A third form of expert review is benchmarking, in which an international panel of experts compares the level of leadership of a research program relative to research being performed worldwide. In early drafts of strategic and performance plans, agencies have generally omitted discussions of education and training, which are fundamental to the ability of agencies to fulfill their missions. The goal of developing and maintaining adequate human resources in fields critical to their missions should be supported by plans that produce that outcome. The nation cannot benefit from advances in science and technology without a continuing supply of well- educated and well-trained scientists and engineers. In addition, in the absence of such a flow, the capability of an agency to fulfill its mission will be compromised and the knowledge learned and technology developed will be lost. 164

Executive Summary of Evaluating Federal Research Programs It is common and valuable for multiple agencies to ap- proach similar fields of research from different perspectives. Indeed, this pluralism is a major strength of the U.S. research enterprise. However, better communication among agencies would enhance opportunities for collaboration, help to keep important questions from being overlooked, and reduce instances of ineffi- cient duplication of effort. A single agency should be identified to serve as the focal point for each particular field of research so that all significant supported fields are covered. Information regarding support for that field should be provided to all the agencies in- volved in it so that they can adjust their efforts to ensure that the field is appropriately covered. Agencies should use benchmarking, which affords the opportunity to look across fields, in their efforts to understand the status of a particular field of research. The researchers who work in agency, university, and industrial laboratories are the people who perform and best under- stand the research programs funded by the federal government. Many researchers contribute substantial time and effort to review- ing papers submitted for publication, grant applications, and 165

IMPLEMENTING THE GOVERNMENT PERFORMANCE AND RESULTS ACT FOR RESEARCH program proposals, but few of them are aware of GPRA. Their greater involvement in implementing GPRA would be beneficial to the country. Increased contact with and advice from the broader scientific and engineering community regarding both the quality and the leadership position of agency research programs and the relevance of the research to agency missions can benefit the GPRA process. COSEPUP intends to address mechanisms and guidelines for implementing these recommendations in workshops and meetings with representatives from federal agencies, Congress, OMB, and oversight bodies. Given the diverse portfolio of research conducted by federal agencies and the urgency of addressing the question of how basic research can be evaluated in the context of GPRA, the level of detail and specificity needed in designing procedures and guidelines for implementation was beyond the scope of this report. The Government Performance and Results Act provides an opportunity for the research community to ensure the effective use of the nation’s research resources in meeting national needs and to articulate to policy-makers and the public the rationale for and results of research. We believe that our recommendations can assist federal agencies in complying with GPRA. 1. For purposes of this study, program refers to a set of activities focused on a particular area that can include multiple projects with different risks, time horizons, and outcomes. 2. There are at least two aspects of quality—one absolute and one relative. The absolute aspects are related to the quality of the research plan, the methods by which it is being pursued, its role in education when conducted at a university, and the importance of its results to its sponsor, either obtained or expected. The relative aspects pertain to its leadership at the edge of an advancing field. Although the leadership aspect is generally important, the results might in some cases be of great importance to an agency albeit not at the leading edge of a field. 166

Next: APPENDIX F Government Performance and Results Act »
Implementing the Government Performance and Results Act for Research: A Status Report Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $47.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

As requested by Congress and the White House Office of Science and Technology Policy (OSTP), this report assists federal agencies in crafting plans and reports that are responsive to the Government Performance and Results Act (GPRA), OMB Guidance, and agency missions. Using a case study approach, the report identifies best practices used by individual agencies to evaluate the performance and results of their science and technology programs. The report takes into account individual agencies' missions and how science and technology programs and human resource needs are factored into agency GPRA plans. Specific applications of recommendations are included from COSEPUP's earlier report entitled Evaluating Federal Research Programs: Research and the Government Performance and Results Act.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!