5

Assessing Impact

Measuring the impact of R&D activities is the most subjective aspect of an assessment. An insightful definition of impact was posed by William Banholzer in his presentation at the National Research Council’s workshop on best practices in assessment of research and development organizations: “What would not have happened if you did not exist, and how much would society have missed?”1

An assessment of impact may be designed by considering the following questions:

1. Does a survey of program outcomes for the time period being assessed (near term, midterm, and far term) indicate expected levels of success in the context of the assigned program objectives?

a. Was the stakeholder, sponsor, or customer satisfied with the output delivered?

b. Did the outcome advance the field significantly?

c. Was there a sustainable advantage associated with the outcome?

d. Was there external recognition of the outcome as fundamentally differentiating in the field?

2. Does a survey of programs launched within the relevant time period indicate an ability to consistently lead in the definition of the next steps required to continue to advance the field within the scope of the mission—for example, what is the track record of the organization in providing forecasts to the sponsor, stakeholder, or customer?

3. Has the organization consistently identified foundational discontinuities in the trajectory of the field of interest, opening unanticipated new fields of study and opportunities?

Providing answers to these questions has been challenging for most organizations. The answers are best sought primarily through analysis of completed R&D, sometimes by examining the recent past and at other times by conducting retrospective analyses of more distant history.

R&D organizations are part of a system or process that leads to products that an organization’s management, stakeholders, and customers will ultimately use as the basis for judgments about the worth of the R&D. However, the links between the R&D and the final impact, perhaps well understood by those performing the R&D, are often not understood by some stakeholders and customers, who may not be technical experts. It is the task of the R&D organization to make the case about those links between R&D and impact, and various strategies have been used. For convenience, a single word, transition, is used here to characterize the process and the consequence of delivering the product of any accomplishment of the R&D organization to any one of its stakeholders and customers.

For basic research, the claim is often made that the distance between the research and the end-item product is so great that one should judge the research as an entity on its own. To

____________________________

1 National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations—Summary of a Workshop. The National Academies Press, Washington, D.C., p. 10.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 38
5 Assessing Impact Measuring the impact of R&D activities is the most subjective aspect of an assessment. An insightful definition of impact was posed by William Banholzer in his presentation at the National Research Council's workshop on best practices in assessment of research and development organizations: "What would not have happened if you did not exist, and how much would society have missed?"1 An assessment of impact may be designed by considering the following questions: 1. Does a survey of program outcomes for the time period being assessed (near term, midterm, and far term) indicate expected levels of success in the context of the assigned program objectives? a. Was the stakeholder, sponsor, or customer satisfied with the output delivered? b. Did the outcome advance the field significantly? c. Was there a sustainable advantage associated with the outcome? d. Was there external recognition of the outcome as fundamentally differentiating in the field? 2. Does a survey of programs launched within the relevant time period indicate an ability to consistently lead in the definition of the next steps required to continue to advance the field within the scope of the mission--for example, what is the track record of the organization in providing forecasts to the sponsor, stakeholder, or customer? 3. Has the organization consistently identified foundational discontinuities in the trajectory of the field of interest, opening unanticipated new fields of study and opportunities? Providing answers to these questions has been challenging for most organizations. The answers are best sought primarily through analysis of completed R&D, sometimes by examining the recent past and at other times by conducting retrospective analyses of more distant history. R&D organizations are part of a system or process that leads to products that an organization's management, stakeholders, and customers will ultimately use as the basis for judgments about the worth of the R&D. However, the links between the R&D and the final impact, perhaps well understood by those performing the R&D, are often not understood by some stakeholders and customers, who may not be technical experts. It is the task of the R&D organization to make the case about those links between R&D and impact, and various strategies have been used. For convenience, a single word, transition, is used here to characterize the process and the consequence of delivering the product of any accomplishment of the R&D organization to any one of its stakeholders and customers. For basic research, the claim is often made that the distance between the research and the end-item product is so great that one should judge the research as an entity on its own. To 1 National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations-- Summary of a Workshop. The National Academies Press, Washington, D.C., p. 10. 38

OCR for page 38
identify transitions from basic research, measures are collected on papers published, citations by others, invitations to speak about the work, patents received, and awards conferred. These metrics for assessing research quality are useful for assessing the quality of R&D, as discussed in Chapter 4, but they are often of little interest to those stakeholders and customers whom the organization wishes to satisfy. Organizations that conduct R&D recognize this, and so their reports to stakeholders frequently include anecdotal stories about how research funded many years earlier has found its way into application today. In many R&D organizations transitions from basic to applied research are frequent, and are often the subject of management decisions. Within an R&D organization additional indicators for examining transitions from basic research are available, but they are less frequently tracked than the obvious outputs in papers and technical talks. Some related questions are the following: Has the technical advance in a particular project opened the door for more applied work? Does the applied work justify additional funding and possibly new hires? These questions may apply whether the basic work was done in-house or extramurally. These questions are regularly asked and decisions are made, but organized records of such transitions are not commonly compiled or recognized as appropriate indicators for judging the basic work. With respect to assessing the impact of applied research or product development, the situation becomes far more complex. Industry will appropriately focus on its bottom line, and predicted return on investment for the R&D investment can be calibrated against actual sales. Feedback from both failures and successes may be communicated to stakeholders and used to modify future investments. Government organizations rarely have such a direct metric and must search for more information and a structure in order to communicate to their myriad stakeholders. Four general types of R&D organizations are identified above in Chapter 2: mission- specific, industrial and contract organizations, product-driven organizations (e.g., national laboratories), and universities. The question of impact is different for each type of organization. Mission-specific organizations in the government and industrial organizations with clearly defined missions are generally considered the underpinning of an organization that includes centers for development, testing and evaluation, and maintenance. Each organization includes management processes designed for transitioning the product of its efforts to the next stage of development and/or application. In many instances these process include ensuring that adequate resources are available so that a transition can indeed occur. Formal agreements often ensure handoffs of responsibility upon completion of the organization's contributions. These processes offer excellent opportunities for examining the recent past as well as for aiding in the scholarly task of examining the more distant past. In the NIST laboratories, for example, there are frequently tangible consequences of the R&D work that can be monitored, including standard reference materials sold, data used, and standards developed and promulgated by standards-making bodies. In addition, NIST participates, as do other government labs, in cooperative research and development agreements (CRADAs) that may be seen as a potential database of relevant transitions. Within such CRADAs, the R&D organization and its partners (other government organizations, universities, and industries) carry out collaborative efforts designed explicitly to transition the results of more basic research toward application. The challenge with respect to providing information supporting assessments is to identify these transitions in a manner that reveals the implications 39

OCR for page 38
for future decisions and to make the data accessible and comprehensive enough to allow comparisons over time and within the organization. For product-driven national laboratories, the impact of R&D activities has more variation in its characteristics. An organization typically has carved out one or more core mission spaces, and the impact of those is measured by the degree to which these missions are being carried out. In emerging mission areas, the impact is measured by the growth of external investment, and in future mission areas impact is measured by the successful establishment of the workforce and competencies that will be needed to address these areas. In its simplest form, an industrial organization could measure its impact in terms of return on investment. This form of assessment is a best practice, but not a universal one, and involves caveats--many things affect earnings, such as price, volume, and cost of the existing product, and the time lag between investment and earning can be as long as a decade.2 For academic research groups, a common best practice is to assess the quality of the organization by benchmarking it against other laboratories generally recognized to be successful. This is a qualitative but widespread practice. TIMESCALES OF IMPACT In attempts to project the future impact of current or proposed programs, the R&D organization is hampered by the fact that it may take many years or even decades before the full impacts of current programs are realized. When that eventually does come to pass, there have usually been so many different organizations involved in developing, engineering, producing, and fielding the end item that its identity with the research organization is often lost. Regardless of how the story is formulated, the stakeholders' confidence in the organization and its management will be bolstered by demonstration that the decision making and processes of the present are comparable to, or better than, those of the past that led to measurable impacts. To tell this story properly, many organizations have had recourse to looking backward and tracing the consequences of R&D events long past. An example of this approach for learning about impact is Project Hindsight.3 Carried out by the Department of Defense in the mid- to late 1960s, Project Hindsight was a study of the development of 22 different weapons systems drawn from across the military services. It involved more than 200 personnel over a period of approximately 6 years. For years afterward the observations and conclusions of Project Hindsight guided military R&D planning and decision making. In 2004, recognizing that much had changed in the intervening years, the U.S. Army commissioned a new study, Project Hindsight Revisited.4 The Air Force Research Laboratory (AFRL) identified research transitions over a 50-year period at the AFRL,5 and Berry 2 W. Banholzer and L. Vosejpka, 2011. Risk taking and effective R&D management. Annual Review of Chemical and Biomolecular Engineering 2:8.1-8.16. 3 Office of the Director of Defense Research and Engineering (DDRE), 1969. Project Hindsight: Final Report. Office of the DDRE, Washington, D.C. 4 J. Lyons, R. Chait, and D. Long, 2006. Critical Technology Events in the Development of Selected Army Weapons Systems: Project Hindsight Revisited. National Defense University, Washington, D.C. 5 Air Force Office of Scientific Research, 2002. AFOSR at 50: Five decades of research that helped change the world. Research Highlights, March/April. 40

OCR for page 38
and Loeb identified breakthrough Air Force capabilities spawned by basic research.6 Coffey and coauthors describe methods and challenges in tracing the development of selected technologies from basic research through significant exploitation of the research.7 The Department of Energy (DOE) utilized a similar retrospective analysis, with the assistance of the NRC, examining the impacts on energy-producing and energy-using industries of R&D programs executed by the DOE laboratories over the time period 1978-2000. The report summarizing the findings of the assessment makes the case in economic terms for a return on investment that by itself could justify funding the research, while recognizing that societal impacts are far more difficult to measure and are not readily quantifiable.8 An exemplary example of after-the-fact analysis of impact is the economic impact analyses performed by NIST. Stimulated in part by the concerns of many in Congress that a program such as the NIST Advanced Technology Program (ATP) was inappropriate for government and that the R&D carried out under its aegis would be better left to industry, NIST began the ATP program in the late 1980s with a coordinated plan to measure economic impact. Using outside expertise and a well-articulated process, this impact analysis was applied to all funded activities and its record made available to all interested parties. A summary of the first 10 years of ATP funding, published in 2003, provides methodologies and results gained from their application to many industrial sectors. NIST has gone on to apply such organized studies to many of its laboratory-based efforts, and the historic record of accomplishment provides a strong justification for future work in the areas whose impacts have been assessed.9 SUMMARY OF FINDINGS Measuring the impact of R&D activities is the most subjective aspect of assessment and is ill-suited to quantitative measures. Transitions are measurable at every level of R&D (see Figure 2-1). The impact of research can only be measured after the fact. Near-term impacts, including many transitions during R&D phases, require looking back at the recent past but can be monitored for most of the R&D effort. Long-term impacts require deeper historical probes and are more likely to be assessed for only a few notable examples. In both cases, organized processes for gathering and analyzing the data require management attention and designated leadership. Developing the data and presenting them in a manner that makes the data useful to the intended audience is a job for professionals. The R&D organization will benefit from having an appropriately supported historian and internal report requirements to ensure the utility of the process. 6 W. Berry and C. Loeb, 2007. Breakthrough Air Force Capabilities Spawned by Basic Research. National Defense University, Washington, D.C. 7 T. Coffey, J. Dahlburg, and E. Zimet, 2005. The S&T Innovation Conundrum. National Defense University, Washington, D.C. 8 National Research Council, 2001. Energy Research at DOE: Was It Worth It? Energy Efficiency and Fossil Energy Research 1978 to 2000. National Academy Press, Washington, D.C. 9 R. Ruegg and I. Feller, 2003. Evaluating Public R&D Investment Models, Methods, and Findings from ATP's First Decade. National Institute of Standards and Technology, Gaithersburg, Md. 41