4. failure to first establish and then to carry along event-based strategies—instead employing schedule-based strategies—and failure to use strict pass/fail criteria for each phase of development;
5. failure to carry out continuous, independent assessments of the effectiveness and suitability of defense systems in development from initial development through the various stages of testing and production, extending to early introduction to the field; and
6. failure to use feedback loops to inform the broad acquisition community as to when acquisition methods have worked and when they have failed so that all can learn from others’ experiences.
We discuss several of these factors throughout this report.
The following actions, some of which are discussed in the report, can help ameliorate these problems:
• Competitive prototype development and testing should be a strict prerequisite for any new development program prior to entry into engineering and manufacturing development.
• Emphasis should be on an event-based strategy, rather than a schedule-based strategy, with meaningful and realistic pass/fail criteria for each stage of development. In particular, systems should not be allowed to proceed to operational testing unless that step is supported by developmental test performance that strongly anticipates that the system will pass; such a determination can be greatly aided through the conduct of a rigorous operational test readiness review.
• Use of continuous and independent evaluation tracking of each program through the stages of development, testing, and production should be required. These assessments should rely heavily on empirical tests and should focus on those capabilities that were the basis for program approval.
Problems with suitability performance of defense systems are just as widespread, and the Defense Science Board (2008) made the following recommendations for remedying them:
1. Identify reliability, availability, and maintainability (RAM) requirements during the joint capabilities integration development system process, and “incorporate them in the request for proposal as a mandatory contractual requirement” (p. 6).
2. When evaluating proposals, evaluate the bidder’s approaches to satisfying RAM requirements.