system, there are major difficulties in obtaining definitive test cost data for any major acquisition program. These difficulties arise because relevant cost information does not reside within a single repository but is divided between project management offices and test and evaluation agencies. The authors also cite variations in cost accounting across test and evaluation agencies and developmental testing organizations.
Hodges identified two institutional problems contributing to poor data utilization within DoD. First, no one has responsibility for accumulating test data in one place and ensuring proper utilization. Second, there are competing objectives among the chief players—defense contractors, the services, and the Office of the Secretary of Defense —that complicate the collection and interchange of data. A technical problem is that some of the needed statistical methods—for example, Bayesian methods for incorporating prior information—are not trivial, which complicates their routine implementation. Hodges suggested, therefore, that DoD needs to develop a collection of methods for sensibly using an accumulated body of data.
Hodges and others believe that DoD should identify an agency to be the designated repository of data and the advocate of data collection and use. Such an agency could be a new office, an existing office, or a consortium of existing offices. In order to function effectively as a data advocate, this agency would need broad access to data that are being collected, some role in deciding what data will be collected, and some role in how data are used. In closing remarks at the workshop, Donald Rubin echoed Hodges's call for a data advocate within DoD, and Samaniego emphasized the need for long-term data management.