The increasing role that software-based components are playing in complicated defense systems and the evidence that a substantial percentage of cost increases, schedule delays, and performance problems are due to software problems require greater access to expertise in software engineering and software testing methods. Although a number of extremely effective relevant procedures have been developed in the past decade, there is little evidence that these techniques have been applied to defense systems (see, for example, National Research Council, 2003).
The greater role that modeling and simulation will play in testing and evaluation of increasingly complex systems-of-systems will also require greater access to expertise in the use of physics-based modeling and simulation and modeling at the operational level.
A test and field data archive is absolutely necessary to provide facilitated access to information on the test designs and outcomes of tests from previous stages of development. Methods of combining information clearly require information from previous stages of development. By a test (and field) data archive, we mean the following (as discussed in National Research Council, 2004:59-60):
a rich set of variables to adequately represent the test environment, the system under test, and the performance of the system….
In order to accurately represent system performance, including the appearance of various failure modes and their associated failure frequencies, the circumstances of the test must be understood well enough that the test, training exercise, or field use can be effectively replicated, including the environment of use (e.g., weather, terrain, foliage, and time of day) and type of use (e.g., mission, intensity, and threat).
While a system is under development, the system design is often under constant modification. Given the need to be able to replicate a test event in the database, it is crucial to represent with fidelity the system that was in operation during the event so that proper inference is possible.