Improved Operational Testing and Evaluation and Methods of Combining Test Information for the Stryker Family of Vehicles and Related Army Systems Phase II Report (2004) / Chapter Skim
Currently Skimming:
4. Prerequisites for Combining Information
4. Prerequisites for Combining Information
Pages 53-68
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 53... ...
But the use of methods for combining information can be made easier if the appropriate methodological and logistic frameworks are in place. This chapter discusses several key steps that should be taken to establish these frameworks: broader definitions of data so that nontest data (e.g., expert judgment and computer models)
|
From page 54... ...
The use of nontest data for evaluation can be contentious, although it is done routinely. Military, engineering, and statistical judgments are required to design test plans and interpret data; and computer modeling and simulation are applied to test data collected under certain scenarios to extrapolate the scope of their validity to other scenarios or to larger fighting units.
|
From page 56... ...
to formally elicit and quantify engineering judgment for inclusion in statistical calculations, and there is a growing body of literature by statisticians, decision analysts, social scientists, and cognitive psychologists, developed over the past two decades, describing methods for eliciting and using expert judgment. Using information based on expert judgment requires considerable care, explicit documentation, and careful sensitivity analysis.
|
From page 57... ...
This is true even for previous development stages of a system. Once a system has been fielded, the absence of rigorous information on system performance greatly limits the effectiveness offeedback loops relating performance in the field to performance during testing, feedback that could be very useful for improving system designs, the system development process, and operational and developmental test design.
|
From page 58... ...
Data archiving can also contribute to improvement of defense system assessment by providing a means to better understand the differences between failure modes and failure frequencies in moving from developmental to operational testing and from operational testing to field use; understand the sources of system deficiencies identified in the field, which can then be used to guide design improvements; improve both developmental and operational testing and evaluation, e.g., by understanding how deficiencies identified in the field escaped detection in the developmental and operational tests; and estimate system and component residual lifetimes and life cycle costs. The current lack of priority for data archiving, given the above advantages, suggests that the primary purpose of test data is to evaluate a system for promotion to the next stage of the milestone process of defense system development.
|
From page 59... ...
Averaged over all defense systems in development, the cost of such an archive would be extremely small, but its value, as has been discovered in many industrial settings, could be substantial. A test data archive would need to contain a rich set of variables to adequately represent the test environment, the system under test, and the performance of the system.
|
From page 60... ...
Finally, as mentioned in Chapter 5 for the Future Combat System, systems developed using evolutionary acquisition provide an additional argument for the establishment and use of a test (and field) data archive, since it is vital to link the performance of the system as it proceeds through the various stages of development.
|
From page 61... ...
For large, complex systems with heterogeneous data sources, representations of a system have several advantages: they set out a common language that all communities can use to discuss the problem; heterogeneous data sources can be explicitly located in the representation; and the representation provides an explicit mapping from the problem to the data to the metrics of interest. If the system and its assessment are to be put in a decision contexttor example, an overall assessment of system effectiveness and suitability supporting an acquisition decision the fault trees and block diagrams may need to be embedded in a representation that supports these broader goals and connects the disparate and heterogeneous sources of data.
|
From page 62... ...
, which are a formal graphical language for representing logical relationships; they are used extensively in the artificial intelligence, information technology, and computer modeling communities. Similar to the less formal scratch nets, conceptual graphs use labeled nodes (which represent any entity, attribute, action, state, or event that can be described in natural language)
|
From page 63... ...
63 1 ~ 1 o W o o o tr In in ~ g o ~ In lo ~ 2 ~ Cot ~ ~ o O A Q ~ ~ O m ~ _~ ,~\ L~',~ be Cat o o be o Fag H
|
From page 64... ...
Bayesian networks, in particular, are a flexible class of statistical graphical models that capture causal relationships Jensen, 1996) in a way that meshes well with conceptual graphs; they are considered flexible because standard reliability diagrams (like block diagrams and fault trees)
|
From page 65... ...
Complex systems tend to have complex problems, which usually exhibit one or more of the following characteristics (Booker and McNamara, 20031: a poorly defined or understood system or process, such as high cycle fatigue effects on a turbine engine; a process characterized by multiple exogenous factors whose impacts are not fully understood, such as the effects on a new system of changing combat missions; an engineered system in the very early stages of design, such as a new concept design for a fuel cell; a system, process, or problem that involves experts from different disciplinary backgrounds, who work in different geographical locations, and/or whose problem-solving tools vary widely (as is the case in the work involved to ensure the reliability of a manned mission to Mars) ; and any new groups of experts in novel configurations brought together for its solution.
|
From page 66... ...
The more sophisticated category contains the rich collection of hierarchical and random effects models that have enjoyed recent, very rapid development and that have been successfully applied to a large number of new situations. Flexible, public-use software currently exists for a rich set of models, greatly simplifying software development.
|
From page 67... ...
It is also likely that use of these complicated methods will provide tangible benefits in operational evaluation. With respect to both hierarchical and random effects modeling, while there are some standard models that have been repeatedly applied and that may be useful for some defense applications, it is very likely that procedures at the leading edge of research will often be needed for high-quality operational test designs and evaluations.
|
From page 68... ...
In addition, ATEC should consider making available all sources and types of information for a candidate defense system to a selected group of qualified statisticians in industry and academia as a case study to understand the potential advantages of combining information for operational evaluation. Recommendation: ATEC should examine how to increase statistical capabilities to support future use of techniques for combining information.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.