for future decisions and to make the data accessible and comprehensive enough to allow comparisons over time and within the organization.
For product-driven national laboratories, the impact of R&D activities has more variation in its characteristics. An organization typically has carved out one or more core mission spaces, and the impact of those is measured by the degree to which these missions are being carried out. In emerging mission areas, the impact is measured by the growth of external investment, and in future mission areas impact is measured by the successful establishment of the workforce and competencies that will be needed to address these areas.
In its simplest form, an industrial organization could measure its impact in terms of return on investment. This form of assessment is a best practice, but not a universal one, and involves caveats—many things affect earnings, such as price, volume, and cost of the existing product, and the time lag between investment and earning can be as long as a decade.2
For academic research groups, a common best practice is to assess the quality of the organization by benchmarking it against other laboratories generally recognized to be successful. This is a qualitative but widespread practice.
In attempts to project the future impact of current or proposed programs, the R&D organization is hampered by the fact that it may take many years or even decades before the full impacts of current programs are realized. When that eventually does come to pass, there have usually been so many different organizations involved in developing, engineering, producing, and fielding the end item that its identity with the research organization is often lost. Regardless of how the story is formulated, the stakeholders’ confidence in the organization and its management will be bolstered by demonstration that the decision making and processes of the present are comparable to, or better than, those of the past that led to measurable impacts. To tell this story properly, many organizations have had recourse to looking backward and tracing the consequences of R&D events long past.
An example of this approach for learning about impact is Project Hindsight.3 Carried out by the Department of Defense in the mid- to late 1960s, Project Hindsight was a study of the development of 22 different weapons systems drawn from across the military services. It involved more than 200 personnel over a period of approximately 6 years. For years afterward the observations and conclusions of Project Hindsight guided military R&D planning and decision making. In 2004, recognizing that much had changed in the intervening years, the U.S. Army commissioned a new study, Project Hindsight Revisited.4 The Air Force Research Laboratory (AFRL) identified research transitions over a 50-year period at the AFRL,5 and Berry
2 W. Banholzer and L. Vosejpka, 2011. Risk taking and effective R&D management. Annual Review of Chemical and Biomolecular Engineering 2:8.1-8.16.
3 Office of the Director of Defense Research and Engineering (DDRE), 1969. Project Hindsight: Final Report. Office of the DDRE, Washington, D.C.
4J. Lyons, R. Chait, and D. Long, 2006. Critical Technology Events in the Development of Selected Army Weapons Systems: Project Hindsight Revisited. National Defense University, Washington, D.C.
5 Air Force Office of Scientific Research, 2002. AFOSR at 50: Five decades of research that helped change the world. Research Highlights, March/April.