BOX 5.1

Heilmeier Criteria

When George Heilmeier was DARPA director in the mid 1970s he developed a set of pithy questions to ask research program managers when they proposed new program ideas. That set of questions has persisted, and it continues to be applied in various forms by research managers everywhere. Here is a composite rendering of the questions, along with some commentary regarding research program management.

  1. What are you trying to do? Explain objectives using no jargon. The scope of the project must be defined, as well as the key stakeholders in the outcome. The purpose of “no jargon,” in part, is to assure that the scope and value can be described in ways that individuals outside the immediate field can appreciate the context and value of what is proposed.

  2. How is it done today? What are the limits of current practice? This is an accounting of the baseline state, the value it delivers, the limits on what can be done in the present configuration, and, to some extent, the pain experience as a consequence of those limits.

  3. What’s new in your approach? Why do you think it will be successful? Often the novelty is less in the form of a dramatically “new idea,” but rather in the convergence of existing ideas and new developments elsewhere in the field. A cynical view of “cloud computing,” for example, is that it is a delivery on the dream of “utility computing” articulated in the early 1960s at the dawn of the era of timesharing. Cloud, of course, takes this idea many steps forward in scalability, capability, and other ways. In other words, it is less important that the idea be “novel,” but rather timely, potentially game changing, and feasible. Feasibility, in this context, does not mean free of risk, but rather that the dependencies on infrastructure and other elements of the package are realistic. Feasibility also means that there are potential research performers who have the means and motive to engage on the topic. For academic research, this means the ability to build a capable team of PhD students, engineering staff as required, potential transition partners, collaborators at other institutions, etc.

  4. If you’re successful, what difference will it make? To whom? This is an identification of stakeholders, and in addition an indication of potential pathways from research results to impact. For many research projects related to computing and software, those pathways can be complex. These complexities are discussed in the

undiminished for several decades and perhaps is accelerating because of a now-global involvement in advancing IT. Given the rapid change intrinsic to IT, the research community (in academia and in industry, especially start-up companies) serves not only as a source of solutions to the hardest problems, a source of new concepts and ideas, and a source of trained people with high levels of expertise, but also as a bellwether, in the sense that it anticipates and provides early warning of important technological changes. For software, the potential for surprise is heightened by a combination of the rapid growth of globalization, the concurrent movement up the value chain of places to which R&D has been outsourced, and the explicit investments from national governments and the European Union in advancing national technological capability. Given the role of externalities in IT economics, it is not unreasonable to expect the innovation center of gravity to change rapidly in many key areas, which could shift control in critical areas of the technology ecosystems described above. This is already happening in several areas of IT infrastructure, such as chip manufacturing. In this sense, the research community has a critical role in defense-critical areas that are experiencing rapid change. A consequence of this role is the availability of top talent to address critical software-related defense problems as they arise.

The fourth component of the academic R&D value proposition is non-appropriable invention, as described in Chapter 1. This is one of the several forms of innovation carried out by the university

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement