|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. Appendix E
|
||||||||||||||||
 |
 |
Page 95 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 95
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Appendix E
Agency and Industry Efficiency Measures
The table in this appendix (Table E-1) includes efficiency measures developed by agencies and industry. Most of the measures for the agencies were excerpted from the Office of Management and Budget (OMB) Program Assessment Rating Tool (PART) Web site (OMB 2007) and are related to programs of the “research and development” type. These measures either appear in the “Program Performance Measures” section or are cited as agency responses to questions 3.4 or 4.3 on the PART Web site. Question 3.4 is “Does the program have procedures (for example, competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?” Question 4.3 is “Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?” Some of the agency measures listed have been approved for use by OMB as of July 26, 2007.
The table is not an exhaustive list of efficiency measures used by the federal government, but it includes efficiency measures from a variety of agencies, including the Environmental Protection Agency (EPA), the Department of Energy (DOE), the U.S. Department of Agriculture (USDA), the Department of Commerce (DOC), the Department of Defense (DOD), the Department of Health and Human Services (DDHHS), the Department of the Interior (DOI), the Department of Labor (DOL), the Department of Transportation (DOT), the National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA). The industry efficiency measures (and a few agency efficiency measures) were gleaned from presentations during the April 2007 committee meeting.
OCR for page 96
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
TABLE E-1 Agency and Industry Efficiency Measures
Agency or Organization
Program
Year
Efficiency Measure
EPA
Endocrine Disruptors (combined EPAPART)
2004
(OPPTS) Cost per labor hour of contracted validation studies (EPA, unpublished material, April 23, 2007)
EPA
EPA Human Health Research
2005
Average time (in days) to process research-grant proposals from RFA closure to submittal to EPA's Grants Administration Division while maintaining acredible and efficient competitive merit-review system (as evaluated by external expert review) (EPA, unpublished material, April 23, 2007)
EPA
Land Protection and Restoration Research
2006
Average time (in days) for technical support centers to process and respond to requests for technical document review, statistical analysis, and evaluation of characterization and treatability study plans (EPA, unpublished material, April 23, 2007)
EPA
Water Quality Research
2006
Number of peer reviewed publications per FTE (EPA, unpublished material, April 23, 2007)
EPA
Human Health Risk Assessment Program
2006
Average cost to produce Air Quality Criteria/Science Assessment documents (EPA, unpublished material, April 23, 2007)
EPA
EPA Ecological Research
2007
Percentage variance from planned costand schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007)
EPA
Drinking Water Research
2007
Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007)
EPA
PM Research
2007
Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007)
EPA
Global Change Research
2007
Percentage variance from planned costand schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007)
EPA
Pollution Prevention Research
2007
Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007)
OCR for page 97
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
DOD
Defense Basic Research
2002
Long-term measure: portion of funded research chosen on basis of merit review; reduce non-merit-reviewed and determined projects by half in 2 years (from 6.0% to 3.0%) (OMB 2007)
DOE
Advanced Simulation and Computing
2002
Annual average cost per teraflops of delivering, operating, and managing all Stockpile Stewardship Program (SSP) production systems in given fiscal year (OMB 2007)
DOE
Coal Energy Technology
2005
Administrative costs as percentage of total program costs (OMB 2007)
DOE
Advanced Fuel Cycle Initiative
2003
Program direction as percentage of total R&D program funding (OMB 2007)
DOE
Generation IV Nuclear Energy Systems Initiative
2003
Program direction as percentage of total R&D program funding (OMB 2007)
DOE
National Nuclear Security Administration: Nonproliferation and Verification Research and Development
2005
Cumulative percentage of active research projects for which independent R&D peer assessment of project's scientific quality and mission relevance has been completed during second year of effort (and again in each later 3-year period for projects found to be of merit) (OMB 2007)
DOE
Nuclear Power 2010
2003
Program direction as percentage of total R&D program funding (OMB 2007)
DOE
Basic Energy Sciences/ Biological and Environmental Research
2006
Average achieved operation time of scientific user facilities as percentage of total scheduled annual operation time; cost-weighted mean percentage variance from established cost and schedule baselines for major construction, upgrade, or equipment procurement projects (cost variance listed first) (OMB 2007)
DOE
Hydrogen Program
2003
In 2003, EERE Hydrogen Program had about 130 fuel-cell and hydrogen production research projects that were subject to in-progress peer review by independent experts
OCR for page 98
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Agency or Organization
Program
Year
Efficiency Measure
For all reviewed projects, reviewers provided written comments and numerical ratings
on a scale of 1-4, with 4 being highest
with resulting scores ranging of 2.2-3.9
Program used review results to make important decisions to continue or discontinue projects
Research efficiency = 1− [( no. of projects discontinued/(total no. of projects reviewed − no. of projects judged as completed − earmark projects)] (Beschen 2007)
DOI
U.S. Geological Survey – Biological Information Management and Delivery
2005
Average cost per gigabyte of data available through servers under program control (EPA, unpublished material, 2006)
DOI
U.S. Geological Survey – Biological Research & Monitoring
2005
Average cost per sample for selected high-priority environmentally available chemical analyses (EPA, unpublished material, 2006)
DOI
U.S. Geological Survey – Energy Resource Assessments
2007
Average cost of systematic analysis or investigation (dollars in millions) (EPA, unpublished material, 2006)
DOI
U.S. Geological Survey – Mineral Resource Assessment
2003
Average cost of systematic analysis or investigation; average cost per analysis allows comparisons among projects to determine how efficiencies can be achieved (EPA, unpublished material, 2006)
DOI
U.S. Geological Survey – Water Resources Research
2004
Average cost per analytic result, adjusted for inflation, is stable or declining over 5-year period (EPA, unpublished material, 2006)
DOI
U.S. Geological Survey – Water Information Collection and Dissemination
2004
Percentage of daily streamflow measurement sites with data that are converted from provisional to final status within 4 months of day of collection (EPA, unpublished material, 2006)
OCR for page 99
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
DOI
U.S. Geological Survey – Biological Research & Monitoring
2005
Percentage improvement in detectability limits for selected high-priority environmentally available chemical analytes (EPA, unpublished material, 2006)
DOI
U.S. Geological Survey – Geographic Research, Investigations, and Remote Sensing
2003
Percentage of total cost saved through partnering for data collection of high-resolution imagery (EPA, unpublished material, 2006)
DOI
Bureau of Reclamation – Science and Technology Program
2003
Each year, increase in R&D cost-sharing per reclamation R&D program dollar will contribute toward achieving long-term goal of 34% cumulative increase over 6-year period (OMB 2007)
DOT
Highway Research and Development/Intelligent Transportation Systems
2004
Annual percentage of all research projects completed within budget (OMB 2007)
DOT
Highway Research and Development/Intelligent Transportation Systems
2004
Annual percentage of research-project deliverables completed on time (OMB 2007)
DOT
Railroad Research and Development
2004
Organizational Excellence: Percentage of projects completed on time (OMB 2007)
Department of Education
National Assessment for Educational Progress
2003
Timeliness of NAEP data for Reading and Mathematics Assessment in support of President's No Child Left Behind initiative (time from end of data collection to initial public release of results for reading and mathematics assessments) (EPA, unpublished material, 2006)
Department of Education
National Center for Education Statistics
2003
NCES will release information from surveys within specified times; NCES collected baseline information in 2005, examining time-to-release for 31 recent surveys ( National Assessment of Educational Progress releases not included in these figures) (EPA, unpublished material, 2006)
OCR for page 100
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Agency or Organization
Program
Year
Efficiency Measure
DHHS
National Center for Health Statistics
2005
Number of months for release of data as measured by time from end of data collection to data release on Internet (OMB 2007)
DHHS
NIH Extramural Research Programs
By 2013, provide greater functionality and more streamlined processes in grant administration by continuing to develop NIH Electronic Research Administration System (eRA)
(FY 2004) Develop plan to integrate OPDIVs into eRA
(FY 2005) Integrate DHHS 50% of eligible DHHS OPDIVs as eRA users for administration of research grants
(FY 2006) Integrate DHHS 100% of eligible DHHS OPDIVs as eRA users for administration of research grants
Conversion of business processes
(FY 2005) 25% of business processes done electronically
(FY 2006) 40%
(FY 2007) 55%
(FY 2008) 80% ( Duran 2007)
DHHS
NIH Intramural Research Program
2005
Reallocation of laboratory resources based on extramural reviews by Boards of Scientific Counselors (OMB 2007)
DHHS
Bioterrorism: CDC Intramural Research
2006
Decrease annual costs for personnel and materials development with development and continuous improvement of budget and performance integration information system tools (OMB 2007)
DHHS
NIOSH
2004
Percentage of grant award or funding decisions made available to applicants within 9 months of application receipt or deadline date while maintaining credible and efficient two-level peer-review system (OMB 2007)
DHHS
NIOSH
Not used currently
Determine future human capital resources needed to support programmatic strategic goals, focusing on workforce development or training and succession planning (Sinclair 2007)
OCR for page 101
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
DHHS
NIOSH
2007
Percentage of grant award or funding decisions made available to applicants within 9 months of application receipt or deadline date while maintaining credible and efficient two-level peer-review system (Sinclair 2007)
DHHS
Extramural Construction
By 2010, achieve average annual cost savings of managing construction grants by expanding use of electronic project-management tools that enhance oversight and 20-year use monitoring
(Each FY) Achieve average annual cost of managing construction grants (Duran 2007)
DHHS
HIV/AIDS Research
By 2010, use enhanced AIDS Research Information System (ARIS) database to more efficiently conduct portfolio analysis to invest in priority AIDS research
(FY 2005) Improve existing ARIS by converting its mainframe system into Web-based system designed by OAR and IC representatives in consultation with a contractor
(FY 2006, FY 2007, FY 2008) Track, monitor, and budget for trans-NIH AIDS research, using enhanced ARIS database, to more efficiently conduct portfolio analysis of 100% of expiring grants to determine reallocation of resources for priority research (Duran 2007)
DHHS
Research Training Program
2006
By 2012, ensure that 100% of trainee appointment forms are processed electronically, to enhance program management (OMB 2007)
NASA
Human Systems Research and Technology
2005
Time between solicitation and selection in NASA Research Announcements (OMB 2007)
NASA
Solar System Exploration
2006
Percentage of budget for research projects allocated through open peer-reviewed competition (OMB 2007)
NASA
Solar System Exploration
2006
Number of days within which NASA Research Announcement research grants for program are awarded, from proposal due date to selection, with goal of 130 days (OMB 2007)
OCR for page 102
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Agency or Organization
Program
Year
Efficiency Measure
NASA
Original Uniform Measures
Complete all development projects within 110% of cost and schedule baseline
Peer-review and competitively award at least 80%, by budget, of research projects
Reduce time within which 80% of NRA research grants are awarded, from proposal due date to selection, by 5% per year, with goal of 130 days
Deliver at least 90% of scheduled operating hours for all operations and research facilities (Pollitt 2007)
NASA
2007
Year-to-year reduction in Space Shuttle sustaining engineering workforce for flight hardware and software while maintaining safe flight
Reduction in ground operations cost (through 2012) of Constellation Systems based on comparison with Space Shuttle Program
Number of financial processing steps and time to perform year-end closing
Number of hours required for NASA personnel to collect, combine, and reconcile data of contract-management type for external agency reporting purposes (Pollitt 2007)
NASA
2007
On-time availability and operation of Aeronautics Test Program ground test facilities in support of research, development, test, and engineering milestones of NASA and DOD programs from both schedule and cost perspectives
Operational cost per minute of Space Network support of missions
Ratio of Launch Services Program cost per mission to total spacecraft cost
Number of people reached via e-education technologies per dollar invested (Pollitt 2007)
OCR for page 103
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
NOAA
Climate Program
2004
Volume of data taken in annually and placed into archive (terabytes) (EPA, unpublished material, 2006)
NOAA
Ecosystem Research
2005
Cost per site characterization (OMB 2007)
NOAA
Ecosystem Research
2005
Percentage of grants awarded on time (OMB 2007)
NSF
Fundamental Science and Engineering Research
2005
Percentage of award decisions made available to applicants within 6 months of proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007)
NSF
Research on Biocomplexity in the Environment
2004
Percentage of award decisions made available to applicants within 6 months of proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007)
NSF
Construction and Operations of Research Facilities
2003
Percentage of construction acquisition and upgrade projects with negative cost and schedule variances of less than 10% of approved project plan (EPA, unpublished material, 2006)
NSF
Polar Research Tools, Facilities and Logistics
2004
Percentage of construction cost and schedule variances of major projects as monitored by earned-value management (OMB 2007)
NSF
Support for Research Institutions
2004
Percentage of award decisions made available to applicants within 6 months of proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007)
NSF
Support for Small Research Collaborations
2004
Percentage of award decisions made available to applicants within 6 months of proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007)
NSF
Construction and Operations of Research Facilities
2003
Percentage of operational facilities that keep scheduled operating time lost to less than 10% (OMB 2007)
NSF
Federally Funded Research and Development Centers
2005
Percentage of operational facilities that keep scheduled operating time lost to less than 10% (OMB 2007)
OCR for page 104
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Agency or Organization
Program
Year
Efficiency Measure
NSF
Information Technology Research
Qualitative assessment by external experts that there have been significant research contributions to software design and quality, scalable information infrastructure, high-end computing, workforce, and socioeconomic impacts of IT (EPA, unpublished material, 2006)
NSF
Polar Research Tools, Facilities and Logistics
Percentage of person-days planned for Antarctic research for which program is able to provide necessary research support (EPA, unpublished material, 2006)
NSF
Polar Research Facilities and Support
Research facilities: keep construction cost and schedule variances of major polar facilities projects as monitored by earned-value management at 8% or less Research support: provide necessary research support for Antarctic researchers at least 90% of time (OMB 2007)
NSF
Support for Individual Researchers
External validation of "significant achievement" in attracting and preparing U.S. students to be highly qualified members of global S&E workforce (EPA, unpublished material, 2006)
NSF
Science and Engineering Centers Program
2006
Percentage of decisions on preproposals that are merit-reviewed and available to Centers Program applicants within 5 months of preproposal receipt or deadline date (OMB 2007)
NSF
Time to decision for proposals: for 70% of proposals submitted to National Science Foundation, inform applicants about funding decisions within 6 months of proposal receipt or deadline date or target date, whichever is later (Tsuchitani 2007)
NSF
Facilities cost, schedule, and operations: keep negative cost and schedule variances at less than 10% of approved project plan for 90% of facilities; keep loss of operating time due to unscheduled downtime to less than 10% of total scheduled operating time for 90% of operational facilities (Tsuchitani 2007)
OCR for page 105
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
USDA
USDA Research: Economic Opportunities for Producers
Percentage of construction acquisition and upgrade projects with negative cost variance of less than 10% of approved project plan (EPA, unpublished material, 2006)
USDA
Economic Opportunities for Producers
2004
Cumulative dollars saved for grant review (OMB 2007)
USDA
Economic Opportunities for Producers
2004
Proposal review time in days (OMB 2007)
USDA
Research on Protection and Safety of Agricultural Food Supply
2005
Additional research funds leveraged from external sources (OMB 2007)
USDA
Economic Research Service
2005
Index of ERS product releases per staff year (OMB 2007)
USDA
Grants for Economic Opportunities and Quality of Life for Rural America
2006
Cumulative dollars saved for grant review: dollars saved reflect average salary saved by calculating number of calendar days saved annually between receipt of proposal and date funding awarded for competitively reviewed proposals, then multiplied by average daily salary for CSREES employees (OMB 2007)
USDA
In-House Research for Natural Resource Base and Environment
2006
Relative increase in peer-reviewed publications (OMB 2007)
USDA
In-House Research for Nutrition and Health
2006
Relative increase in peer-reviewed publications (OMB 2007)
OCR for page 106
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Agency or Organization
Program
Year
Efficiency Measure
Alcoa
Return-on-investment calculation:
(FY 2005) Improve existing ARIS by converting its mainframe system into Web-based system designed by OAR and IC representatives in consultation with contractor
Variable cost improvement
Margin impact from organic growth
Capital avoidance
Cost avoidance
Annual impact of these four metrics over 5-year period becomes numerator; denominator is total R&D budget
Metric is used most often to evaluate overall value of R&D program and current budget focus (Atkins 2007)
Alcoa
Time (Atkins 2007)
Alcoa
Cost (Atkins 2007)
Alcoa
Customer demand (Atkins 2007)
Alcoa
Risk (Atkins 2007)
Alcoa
Impact on business (Atkins 2007)
Alcoa
Impact on customers (Atkins 2007)
Alcoa
Location (Atkins 2007)
Alcoa
Intellectual property (Atkins 2007)
Alcoa
Aggregate R&D expenditures by laboratory group or by identifiable programs and publish value capture or “success rate” for each on annual basis (Atkins 2007)
Alcoa
ROI on R&D spending; success rate of launched products (Atkins 2007)
OCR for page 107
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Dow Chemical
Publications; participation and leadership in scientific community (collaborative research efforts; trade associations; ILSI-HESI; external workshops; adjunct faculty positions, journal or book editors, professional societies) (Bus 2007)
IBM
ROI on Summer Internship Program and Graduate Fellowship Program: what percentage return as regular IBM research employees?
IBM
“Bureaucracy Busters” Initiative to reduce bureaucracy in laboratory support, information-technology support, HR processes, and business processes (Kenney 2007)
IBM
Tracking of patent-evaluation process (Kenney 2007)
IBM
Customer-satisfaction surveys for support functions to evaluate effect of service reductions (Kenney 2007)
IBM
Measurement of response time and turnaround for external contracts (Kenney 2007)
IBM
Measurement of span of responsibility for secretarial support (Kenney 2007)
Procter & Gamble
Time saved in product development (Daston 2007)
Procter & Gamble
Increased confidence about safety (Daston 2007)
Procter & Gamble
External relations benefits (although not quantifiable) (Daston 2007)
OCR for page 108
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
REFERENCES
Atkins, P. 2007. Alcoa Research and Development. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Beschen, D. 2007. Federal Research Agencies' Overview. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Bus, J.S. 2007. Research Efficiency: Industry Perspective. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Daston, G. 2007. Evaluating Efficiency of P&G Central Product Safety Research. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Duran, D. 2007. Measuring Efficiency in Science. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Kenney, J. 2007. Evaluating R&D Efficiency IBM’s Perspective-Within and Without IBM. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
OMB (Office of Management and Budget). 2007. ExpectMore.gov. Office of Management and Budget [online]. Available: http://www.whitehouse.gov/omb/expectmore/ [accessed April 5, 2007].
Pollitt, J.A. 2007. Evaluating the Efficiency of NASA’s R&D Programs. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Sinclair, R. 2007. PART Efficiency Measures at NIOSH. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.
Tsuchitani, P. 2007. PART Efficiency Measures at NIOSH. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.