can very often lead to unhappy surprises—late-breaking negative feedback regarding early design commitments that, when learned at a late stage in the process, can be very costly to revise. That is, what appears to be a “safe,” conservative decision to follow the most basic process is in fact a dangerous decision that can drastically increase programmatic risk—and the possibility of total project failure.2 The key features of a well-managed incremental process for innovative systems are (1) measurements that are informative and relevant, and (2) process feedback loops that are relatively short, with potential major reductions in programmatic risk.

For many of the innovative systems at the heart of the DoD’s software producibility challenge, the details of the future requirements are not—and in many cases cannot be—fully understood. Thus the need to innovate is made more challenging by the simultaneous need to be agile as requirements necessarily evolve over time.


There are attempts to manage innovative software development following process patterns more appropriate to precedented systems and to established predictable engineering disciplines. One consequence is that linear development processes are inappropriately used despite the presence of high engineering risk (and requirements risk also), with the consequence that those engineering risks are unnecessarily transformed into increasing project risks. A second consequence is that there is unjustified emphasis on achieving excessive precision at the outset regarding functionality desired by the user, choices of infrastructure platforms, and possibly also economic tradeoffs in various complex dimensions of quality. This drive for excessive precision in these areas can yield a surfeit of specifications and other early design artifacts, which may in fact give only false comfort—and lead to downstream scrap and rework.

This is because these process patterns do not account for the engineering risks and uncertainties inherent in developing innovative software, where there are no laws of physics and materials to constrain solutions to particular structural patterns. In precedented software, the structural patterns derive from established software ecosystems and from the body of precedent. In innovative SIDRE systems, these patterns are lacking, which is both advantageous, in that opportunity is afforded for innovation and creativity, and also disadvantageous, in that greater levels of uncertainty must be addressed.

Modern governance approaches for larger systems must account for the management of uncertainty. At scale, they must exploit collaboration among distributed teams and in rich supply chains for which there is a continuous negotiation of scope, quality, and resources to balance the opportunities in delivering more value with the uncertainties inherent in software development cost and scope targets. That is:

It is important to treat scope, plans and resources as variables (not frozen baselines) and explicitly manage the variances in these variables until they converge on acceptable levels to commit a project/product to full scale production.

Fortunately, recent DoD and NRC studies3 have resulted in some very initial steps, as evidenced in


Some program managers sarcastically refer to an inappropriately used linear (waterfall) process model as the “requirements, delay, surprise” process model. Fred Brooks’s recent book, The Design of Design (Boston: Addison-Wesley, 2010), succinctly concludes, “The Waterfall Model is wrong and harmful; we must outgrow it.” This point was also made in Fred P. Brooks, 1987, “No Silver Bullet—Essence and Accidents of Software Engineering,” Information Processing 20(4):10-19.


Assessment Panel of the Defense Acquisition Performance Assessment Project, 2006, Defense Acquisition Performance Assessment; see also NRC, Richard W. Pew and Anne S. Mavor, eds., 2007, Human-System Integration in the System Development Process: A New Look, Washington, DC: National Academies Press. Available online at Last accessed August 20, 2010; and National Research Council (NRC), 2008, Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition, Washington, DC: National Academies Press. Available online at Last accessed August 20, 2010.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement