Recommendation 3.3. Test with users in their actual work or field environment (sometimes referred to as a beta deployment).

It is essential to put capabilities into the hands of users early and to measure performance and otherwise obtain feedback. Engaging expected users of the system early enough makes it possible to provide meaningful feedback to developers and influence the small-r requirements. One especially useful approach is to engage users through pilot projects deployed in the field, possibly operating in parallel with production systems.

Recommendation 3.4. Accept certification and functional IT system component test results across organizational boundaries.

The DOD has broadly adopted a set of networking capabilities that are integral to every IT system. As a matter of DOD policy, such capabilities should not be required to undergo separate revalidation or recertification during operational testing. Examples include such capabilities as Internet Protocol and Domain Name System software modules. As more of the technology stack becomes commoditized or provided as a service, the set of associated capabilities not requiring revalidation should likewise grow. Current examples include public key infrastructure and on-demand computing and storage. The same policy should apply to DOD-deployed and DOD-certified software modules that are reused across DOD programs. The testing of the composite system product that is evaluated in a realistic environment for operational effectiveness and suitability is sufficient.

Recommendation 3.5. Use virtual test environments to support both continuous feedback and certification of IT program increments.

A variety of opportunities to establish virtual test environments already exist, as identified in Chapter 4. The definition of test environment could, over time, be significantly extended to include the use of beta testing in actual operational environments, the use of commensurate commercial data and/or operating environments, and the extension of virtual test environments to include operations monitoring as a source of continuous test feedback. The DOD should codify this approach and encourage the use of virtual environments. Operational realism is important but must be balanced against pragmatic considerations.


The current DOD approach to IT acquisition has not been broadly successful in delivering needed capability in a timely manner. To lever-

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement