1. NRC “tire tracks” reports.1 For software, the path often connects the research results to the DoD through the development of commercial capabilities, where private investment takes a promising research idea and matures it to the point that it can be adopted by development teams. This adoption could be by software development teams in defense contractors or it could be by development teams creating commercial products or services. For example, the reliability of DoD desktop computers undeniably was improved, quite dramatically, as a result of the improvements made by Microsoft to the process of development and evaluation for device driver code enabled by the SLAM tool (described elsewhere in this chapter), which in turn were enabled by research sponsorship from DARPA and NSF. In addition to defining the impact, there is value in understanding not only those stakeholders who will benefit, but also those who may be disrupted in other ways.

  2. What are the risks and the payoffs? This is not only an accounting of the familiar “risk/reward” model, but also an indication of what are the principal uncertainties, how (and when) they might be mitigated, and what are the rewards for success in resolving those uncertainties.2

  3. How much will it cost? How long will it take? An important question is whether there are specific cost thresholds. For certain physics experiments, for example, either the apparatus can be built, or not. But for other kinds of research there may be more of a “gentle slope” of payoff as a function of level of effort. The answer to the questions of cost and schedule, therefore, should not only be specific numbers, but also, in many cases, should provide a description of a function that maps resources to results.

  4. What are the midterm and final “exams” to assess progress? It is essential that there be ways to assess progress, not only at the end of a project, but also at milestones along the way. (This is analogous to the idea of “early validation” of requirements, architecture, design, etc., as a way to reduce engineering risk in software.) In many research areas, quantitative measures of progress are lacking or, indeed, their formulation is itself the subject of research. For this reason, in some challenging research areas the identification


1 See National Research Council (NRC), 1995, Evolving the High Performance Computing and Communications Initiative, Washington, DC: National Academy Press; and NRC, 2003, Innovation in Information Technology, Washington, DC: National Academies Press.


2 An inventory of “engineering” risks related to research program management is in the NRC report on E-Government National Research Council, 2002, Information Technology Research, Innovation, and E-Government, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=10355. Last accessed August 3, 2010.

research community. In a market economy, with internal ROI cases prerequisite for R&D investment inside firms, this is a role most appropriate to universities and similar institutions—of course firms often carry out or sponsor such innovation for a variety of reasons, but it is not their core purpose. For IT in particular, such R&D is essential to national competitiveness and to increases in market-wide value. Although the openness of university research is sometimes considered a negative factor with respect to the advancement of technology for national security, it is also the case that universities have unique incentives, unlike industry, to advance the discipline even when the hard-won results are non-appropriable or difficult to fully appropriate. As noted above, it is evident from the history of the field that the advancement of IT and software producibility disproportionately depends on this kind of technology advance. Of course, universities also create an enormous body of appropriable intellectual property that has the potential to be transitioned into practice.

Finding 5-1: Academic research and development continues to be the principal means for developing the most highly skilled members of the software workforce, including those who will train the next generation of leaders, and for stimulating the entrepreneurial activity that leads to disruptive innovation in the information technology industry. Both academic and industry labs are creating

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement