National Academies Press: OpenBook

Evaluating Research Efficiency in the U.S. Environmental Protection Agency (2008)

Chapter: Appendix F: Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews

« Previous: Appendix E: Agency and Industry Efficiency Measures
Suggested Citation:"Appendix F: Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page 109
Suggested Citation:"Appendix F: Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page 110
Suggested Citation:"Appendix F: Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page 111
Suggested Citation:"Appendix F: Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page 112

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix F Draft Board of Scientific Counselors Handbook for Subcommittee Chairs: Draft Proposed Charge Questions for BOSC Reviews1 PROGRAM ASSESSMENT (EVALUATE ENTIRE RESEARCH PROGRAM) The responses to the program assessment charge questions below will be in a narrative format, and will capture the performance for the entire research program and all the activities in support of the program’s Long Term Goals (LTGs). The Long term Goals should be consistent with EPA’s Strategic Plan and mutually agreed upon by ORD and OMB. Program Relevance 1. How consistent are the Long Term Goals (LTGs) of the program with achieving the Agency’s strategic plan and ORD’s Multi-Year Plan? 2. How responsive is the program focus to program office and regional re- search needs? 3. How responsive is the program to recommendations from outside advi- sory boards and stakeholders? 4. How clearly evident are the public benefits of the program? Program Structure 1. How clear a logical framework do the LTGs provide for organizing and planning the research and demonstrating outcomes of the program? 1 EPA 2007. 109

110 Evaluating Research Efficiency in EPA 2. How appropriate is the science used to achieve each LTG, i.e., is the program still asking the right questions, or has it been eclipsed by advancements in the field? 3. Does the MYP describe an appropriate flow of work (i.e., the sequenc- ing of related activities) that reasonably reflects the anticipated pace of scientific progress and timing of client needs? 4. Does the program use the MYP to help guide and manage its research? 5. How logical is the program design, with clearly identified priorities? Program Performance 1. How much progress is the program making on each LTG based on clearly stated and appropriate milestones? Program Quality 1. How good is the scientific quality of the program’s research products? 2. What means does the program employ to ensure quality research (in- cluding peer review, competitive funding, etc.? 3. How effective are these processes? Scientific Leadership 1. Please comment on the leadership role the research program and its staff have in contributing to advancing the current state of the science and solv- ing important research problems. Coordination and Communication 1. How effectively does the program engage scientists and managers from ORD and relevant program offices in its planning? 2. How effectively does the program engage outside organizations, both within and outside government, to promote collaboration, obtain input on pro- gram goals and research, and avoid duplication of effort? 3. How effective are the mechanisms that the program uses for communi- cating research results both internally and externally? Outcomes 1. How well-defined are the program’s measures of outcomes? 2. How much are the program results being used by environmental deci- sion makers to inform decisions and achieve results?

Appendix F 111 SUMMARY ASSESSMENT (RATE PROGRAM PERFORMANCE BY LTG) The responses to the three summary assessment charge questions below will rate the performance for each LTG. For each LTG, a qualitative score will be assigned that reflects the quality and significance of the research as well as the extent to which the program is meeting or making measurable progress to- ward the goal—relative to the information and evidence provided to the BOSC. The scores will be given in the form of adjectives that are clearly defined and which are intended to promote consistency among reviews. The adjectives will be used as part of a narrative summary of the review of each LTG so that the context of the rating and the rationale for selecting a particular rating will be transparent. The rating may reflect considerations beyond the summary assess- ment questions, and will be explained in the narrative. The adjectives to describe progress are: • Exceptional: indicates that the program is meeting all and exceeding some of its goals, both in the quality of the science being produced and the speed at which research result tools and methods are being produced. An excep- tional rating also indicates that the program is addressing the right questions to achieve its goals. The review should be specific as to which aspects of the pro- gram’s performance have been exceptional. • Exceeds Expectations: indicates that the program is meeting all of its goals. It addresses the appropriate scientific questions to meet its goals and the science is competent or better. It exceeds expectations for either the high quality of the science or for the speed at which work products are being produced and milestones met. • Meets Expectations: indicates that the program is meeting most of its goals. Satisfactory programs live up to expectations in terms of addressing the appropriate scientific questions to meet its goals, and that work products are being produced and milestones are being reached in a timely manner. The qual- ity of the science being done is competent or better. • Not Satisfactory: indicates that the program is failing to meet a substan- tial fraction of its goals, or if meeting them, that the achievement of milestones is significantly delayed, or that the questions being addressed are inappropriate or insufficient to meet the intended purpose. Questionable science is also a rea- son for rating a program as unsatisfactory for a particular long term goal. The review should be specific as to which aspects of a program’s performance have been inadequate. For each program review, the summary assessment charge questions be- low will be tailored to the specific review and LTG:

112 Evaluating Research Efficiency in EPA 1. How appropriate is the science used to achieve each LTG, i.e., is the program still asking the right questions, or has it been eclipsed by advancements in the field? 2. How good is the scientific quality of the program’s research products? 3. How much are the program results being used by environmental deci- sion makers to inform decisions and achieve results? REFERENCES EPA (U.S. Environmental Protection Agency). 2007. Draft Board of Scientific Counsel- ors Handbook for Subcommittee Chairs. Board of Scientific Counselors, U.S. En- vironmental Protection Agency, Washington, DC.

Next: Appendix G: OMB's Research and Development Program Investment Criteria »
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Get This Book
×
Buy Paperback | $48.00 Buy Ebook | $38.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A new book from the National Research Council recommends changes in how the federal government evaluates the efficiency of research at EPA and other agencies. Assessing efficiency should be considered only one part of gauging a program's quality, relevance, and effectiveness. The efficiency of research processes and that of investments should be evaluated using different approaches. Investment efficiency should examine whether an agency's R&D portfolio, including the budget, is relevant, of high quality, matches the agency's strategic plan. These evaluations require panels of experts. In contrast, process efficiency should focus on "inputs" (the people, funds, and facilities dedicated to research) and "outputs" (the services, grants, publications, monitoring, and new techniques produced by research), as well as their timelines and should be evaluated using quantitative measures. The committee recommends that the efficiency of EPA's research programs be evaluated according to the same standards used at other agencies. To ensure this, OMB should train and oversee its budget examiners so that the PART questionnaire is implemented consistently and equitably across agencies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!