Thinking Strategically The Appropriate Use of Metrics for the Climate Change Science Program (2005) / Chapter Skim
Currently Skimming:
2 Lessons Learned From Developing Metrics
2 Lessons Learned From Developing Metrics
Pages 19-46
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 19... ...
and National Research Council, 1999, Industrial Environmental Performance Metrics: Challenges and Opportunities, National Academy Press, Washington, D.C., 252 pp.
|
From page 20... ...
See reports on AIChE collaborative projects, focus area on sustainable development at |
From page 21... ...
Although such goals, stated as process metrics, might not be reachable, they serve to drive research in a desirable direction. Ultimately, most R&D metrics fell from favor because of the long period between measurement and analysis of the result of R&D and the need for expert judgment in evaluating the quality of the item being measured.
|
From page 22... ...
22 THINKING STRATEGICALLY TABLE 2.1 Example of the Stage-Gate Steps and Metrics for R&D in a Traditional Advanced Materials Chemical Industry Stage 1 Stage 2 Stage 3 Metric Theme Feasibility Confirmation Commercialization Sustainable · Customer needs · New discovery · Customer product have been analyzed has led to an alternatives to the · Improved established patent use of the new properties, position product have been identified through · Manufacturing analyzed analysis of or marketing customer needs, strengths of the such as increased new discovery strength and have been analyzed corrosion or stain resistance, have been demonstrated Economics · Product or · Materials cost, · Sustained pilot process concept process yield, operation has been has been proven, catalyst life, achieved even though capital intensity, · Impurities and economic and competitors recycle streams have practicality has not have been analyzed been analyzed been established Customer · Target customers · Plan exists for · Partnerships and acceptance have been partnerships and access to market identified for access to have been established market · Customer reaction to prototype has been satisfactory Safety and · Alternative · Inherently safe · Design exists for environment materials have and "green" "fail-safe" operation been considered concepts have been and "zero" emissions · Radical demonstrated processing concept · Toxicology tests has been have been considered completed · Safety in use has been analyzed
|
From page 23... ...
Standardization of data collection, quality assessment, and verification are necessary to produce broadly credible results. 5Blaustein, M.A., 2003, Managing a breakthrough research portfolio for commercial success, Presentation to the American Chemical Society, March 25, 2003; Carberry, J., 2004, Managing research programs via numerical metrics and/or a "stage gate" process," Presentation to the NAE Committee on Global Climate Change R&D, March 3, 2004.
|
From page 24... ...
Finally, a stage-gate process might be used to help CCSP agencies plan how to move a program emphasis from the discovery phase that precedes Stage 1 (feasibility of using basic research results to improve decision making) , to Stage 2 (developing and testing decision-making tools)
|
From page 25... ...
in the four CCSP core approaches, based on hypothetical data in Table 2.2. From these data, program managers would decide if the distribution of effort is appropriate or if adjustments are needed.
|
From page 26... ...
An illustration of that scoring system follows: R&D Metric: Quality of the Internal or External Review Process for This Task 1 = Poor -- no review plan in place; no reviews, even ad hoc 2 = Fair -- no review plan in place; infrequent, ad hoc reviews; unreliable follow-up 3 = Average -- review plan exists; irregularly followed; unreliable follow-up 4 = Good -- plan exists; regularly followed; spotty follow-up 5 = Excellent -- plan exists; regularly followed; excellent follow-up TABLE 2.3 Hypothetical Example of 11 R&D Process Metrics Applied to Six Research Projects Metric Project 1 Project 2 Project 3 Project 4 Project 5 Project 6 Average Quality of the 4 5 3 2 2 4 3.3 internal or external peer review process for this task Statement of 4 5 3 3 3 3 3.5 the task is sufficiently focused and specific to be evaluated by the peer review process Quality of the 4 5 2 1 3 2 2.8 selection and definition of long-term goals Quality of the 3 3 2 1 1 2 2.0 selection and definition of milestones
|
From page 27... ...
LESSONS LEARNED FROM DEVELOPING METRICS 27 TABLE 2.3 Continued Metric Project 1 Project 2 Project 3 Project 4 Project 5 Project 6 Average Progress in 3 3 1 1 1 2 1.8 achieving milestones Communication 1 4 1 2 1 3 2.0 of the work Projected cost 2 4 1 3 3 3 2.7 to completion in relation to relative importance of the subject and total funds that might be available Usefulness of 4 5 1 2 3 3 3.0 the results in meeting the overall goal Feasibility of 4 4 2 2 4 3 3.2 completing the work in a time frame useful for the overall study What is the 4 5 3 4 2 3 3.5 assessment of the scientific quality of the work? What is the 4 4 2 3 3 3 3.2 assessment of the performance versus the technical specification?
|
From page 28... ...
, and it usually takes the form of internal committees that both review the person's work and take account of outside letters of evaluation from experts in fields relevant to the particular candidate. These evaluations require a good deal of personal judgment-about qualities of mind, the influence of particular ideas or writings, and the person's promise for future contributions -- but usually these subjective judgments are bolstered by metrics of research performance.
|
From page 29... ...
Ideally, each member of the evaluation group would invest significant time developing at least a basic understanding of the other relevant fields. However, this is a luxury that peer review committees rarely, if ever, have.
|
From page 30... ...
. Nevertheless, peer review committees, supported by output and impact metrics, have proven to be the most successful means of evaluating research progress in most academic disciplines.8 Applicability to the CCSP Peer review, supplemented by output and impact metrics, has generally been successful in evaluating academic research progress.
|
From page 31... ...
11Army Research Laboratory, 1996, Applying the Principles of the Government Performance and Results Act to the Research and Development Function: A Case Study Submitted to the Office of Management and Budget, 27 pp., |
From page 32... ...
Tight deadlines (court-ordered, administrative, political) can deter peer review planning and shorten or abort scheduled peer reviews.
|
From page 33... ...
A 2004 General Accounting Office report found that R&D man 13Army Research Laboratory, 1996, Applying the Principles of the Government Performance and Results Act to the Research and Development Function: A Case Study Submitted to the Office of Management and Budget, 27 pp., |
From page 34... ...
their K-12 underrepresented engineering groups teaching, of and career and fields Act formal engineering, knowledge and education from and developing prepare providing appreciation and knowledge provide and science and research and and learning, to forefront science Results the between on the learning discoveries engineering in including technology, at engineering individuals and underrepresented research attract boundaries and of bridges capability and for identifying selected that force, research work between competitive science, in in diversity partnerships cross-disciplinary understanding build continuous science work mathematics who science and and nation's perspectives Measure and for to and collaborative international progress Performance greater participation programs S&E public innovative the opportunities across improving people leadership and and connections high-quality, and for global integrative skills Promote Support Promote Support Develop Enable Encourage Foster Increase Provide Accelerate Performance · increased · the collaborations, · mathematics · basis · opportunities engineering, · contributions · sectors, · · conduct · within · new new Government of Goal for of for of related related goal goal Types a people ideas will will majority following the majority following the Annual Performance NSF demonstrate significant achievement the the performance indicators to outcome NSF demonstrate significant achievement the the performance indicators to outcome Foundation Contrasting and and and 2.4 force frontier to Goal Science diverse, and society A engaged Discovery the to work scientists, science TABLE Strategic National People: competitive, globally U.S. of engineers, technologists, well-prepared citizens Ideas: across of engineering, connected learning, innovation, service
|
From page 35... ...
the insignificant and construction, computation policy major of fish skill tools have increased of databases, lowered lowered estimates research cyberinfrastructure inform "unknown" (annual/cumulative) 22 Atlantic researchers, instrument that contiguous analysis year to an overfished with with tools, large stocks to education per true U.S.
|
From page 36... ...
Annual percentage hazard 2005 Measure (increased 69%) lead forecast to (threat lead Congres FY time coastal to Lead Increased Hurricane Accuracy Increased Cumulative Performance · (decreased · floods · · · · reduce Request Administration of Budget Goal accuracy and 2005 information FY timeliness Atmospheric Annual Performance Improve and weather water and Foundation Continued Oceanic 2.4 weather Science Goal society's for water TABLE Strategic Serve needs and information National National a b NOAA05APP.pdf>.
|
From page 37... ...
per ozone depletion-potential-ton phase-out targets NASA · As validated by external review, and quantitatively where appropriate, demonstrate the ability of NASA developed data sets, technologies, and models to enhance understanding of the Earth system, leading to improved predictive capability in each of the six science focus area roadmaps · Continue to develop and deploy advanced observing capabilities and acquire new observations to help resolve key [Earth system] science questions; progress and prioritization validated periodically by external review · Progress in understanding solar variability's impact on space climate or global change in Earth's atmosphere · Progress in developing the capability to predict solar activity and the evolution of solar disturbances as they propagate in the heliosphere and affect the Earth NOAA · U.S.
|
From page 38... ...
agers in particular continue to have difficulty establishing meaningful outcome measures, collecting timely and useful performance information, and distinguishing between results produced by the government and results caused by external factors or players such as grant recipients.16 The report also found that issues within the purview of many agencies (e.g., the environment) are not being addressed in the GPRA context.
|
From page 39... ...
As a result, the aggregate of agency measures does not address the full scope of the CCSP.18 Nevertheless, approaches that agencies have taken to develop performance measures may be useful to the CCSP. Performance measures developed for climate change programs in the agencies provide a starting point for developing CCSP-wide metrics, and OMB guidelines and the Washington Research Evaluation Network (WREN)
|
From page 40... ...
Below is a review of the stratospheric ozone program of the 1970s and 1980s, which offers an opportunity to determine what factors made this multiagency program successful. Lessons Learned from Stratospheric Ozone Depletion Research The existence of ozone at high altitude and its role in absorbing incoming ultraviolet (UV)
|
From page 41... ...
As a result, the magnitude of the overall effects of CFCs on stratospheric ozone predicted by the models changed. In fact, in an NRC report issued in 1984, just prior to the discovery of the Antarctic ozone hole, even the sign of ozone change was in doubt.30 25For example, see Stolarski, R.S., and R.J.
|
From page 42... ...
Today, the response of governments to regulate stratospheric ozone depletion is viewed as a policy success, and concentrations of CFCs have leveled off or begun to decline, although it will be many decades before the Antarctic ozone hole is expected to disappear.33 Applicability to the CCSP A number of lessons can be drawn from the ozone example above: 1. The unpredictable nature of science.
|
From page 43... ...
A dramatic loss of ozone in the lower Antarctic stratosphere was first noticed by a research group from the British Antarctic Survey that was monitoring the atmosphere using a ground-based network of instruments.35 The same decline was famously missed by satellite observations at first because "anomalously low" values for total column ozone were flagged as potentially unreliable, and the satellite team's foremost concern at the time was its ability to accurately measure column ozone with the instrument. Subsequent reanalysis of the satellite data corroborated the existence of the Antarctic ozone hole.
|
From page 44... ...
Similar progress in understanding basic physical and chemical properties of the Earth system is required before credible climate change predictions can be made. However, the scope of needed advances is vast because most greenhouse gases have important sources and sinks in the biosphere and hydrosphere, and the controls on these fluxes feed back to atmospheric composition and climate.
|
From page 45... ...
For example, increased storage of carbon in fertilized agricultural fields may be offset by increased release of nitrous oxide. CONCLUSIONS Although industry, academia, and federal agencies have not had to develop metrics for programs as complex as global change, their experience can provide useful guidance to the CCSP.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.