Click for next page ( 54


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 53
4 Prerequisites for Combining Information The development and implementation of techniques for combining information, whether in design or in evaluation, are often sophis- ticated activities. What may conceptually seem to be relatively straightforward applications often require original thought, nontrivial modification of existing techniques, and software development. But the use of methods for combining information can be made easier if the appro- priate methodological and logistic frameworks are in place. This chapter discusses several key steps that should be taken to establish these frame- works: broader definitions of data so that nontest data (e.g., expert judg- ment and computer models) can be formally and correctly included in analyses; development of test data archives so that what is learned about a system continues to be of use to future evaluators; use of graphical repre- sentations of complex systems to aid in the understanding of overall reli- ability and performance; and use of formal statistical methods for informa- . . . . tlon combination. ~ . . . . ~ . His chapter also identifies the statistical capabilities required to imple- ment such strategies. There is no clear evidence that the service test agen- cies have these capabilities in place today, and so if they find the advantages presented here compelling, it will be necessary for them, with help from higher-level officials within the services, to acquire the capabilities described in this chapter. 53

OCR for page 53
54 IMPROVED OPERATIONAL TESTING AND EVALUATION NEED FOR A BROADER DEFINITION OF DATA When performing an assessment of a complex system, the most com- monly used data are test data, whether from operational, developmental, or contractor tests. Other sources of information about the system include training exercises, field use, computer models and simulation, and military and engineering judgment. Figure 4-1 is a schematic diagram of a system and its available data sources. If resources were available, it would be desirable to collect test data on every part of the system and to perform system tests under a variety of conditions. For large and complex systems, however, that is seldom pos- sible, and so the assessment often resembles that shown in Figure 4-1, where some parts are not tested at all, some have computer modeling and simula- tion data, some have historical data, others have test data, and some have multiple sources of data. The challenges in methods for combining information are to (1) repre- sent the system under test in a way that all of the stakeholders can under- stand (in Figure 4-1, a fault tree is used, one of many useful representa- tional schemes); (2) collect data (broadly defined) to assess the system and map them onto the representation; and (3) perform appropriate statistical analyses to combine the available information into estimates of the metrics of interest. All of these steps are performed in some way by ATEC's current operational evaluation; this chapter provides suggestions for additional ca- pabilities. For example, the graphical representation of Figure 4-1 could be used to facilitate understanding of the system evaluation plan data source matrix, to suggest areas where data are (or will be) missing and where data combination is possible, and to provide a structure for test planning. It is important to acknowledge and account for possible weaknesses in different kinds of data. The use of nontest data for evaluation can be con- tentious, although it is done routinely. Military, engineering, and statistical judgments are required to design test plans and interpret data; and com- puter modeling and simulation are applied to test data collected under certain scenarios to extrapolate the scope of their validity to other scenarios or to larger fighting units. Methodological contention arises when attempts are made to use military judgment or computer modeling and simulation results formally as data, instead of using them only to inform design, mod- . . . e. .lng, or interpretation. The use of expert judgments, in particular, is especially vulnerable to inappropriate application due to procedural or cognitive biases. For

OCR for page 53
55 C) o ~ . Q ~ o ~ in + C) . _ IL C' a I c`i in ~ ~ ~ C) ~ S ~ o Z + i , :. in c) c) A_ c) c) TIC .= ~ C) .' =._ ~ ~ in Q .m in o be o .; Fig H

OCR for page 53
56 IMPROVED OPERATIONAL TESTING AND EVALUATION example, suppose that test data are not available for a particular component but that engineering judgment considers the system design "unreliable" (Figure 4-11. Methods have been proposed (Meyer and Booker, 2001) to formally elicit and quantify engineering judgment for inclusion in statisti- cal calculations, and there is a growing body of literature by statisticians, decision analysts, social scientists, and cognitive psychologists, developed over the past two decades, describing methods for eliciting and using expert judgment. Using information based on expert judgment requires consider- able care, explicit documentation, and careful sensitivity analysis. With the recognition that all statistical analyses depend, to some degree, on subjec- tive judgment (Berger and Berry, 1988) comes the obligation to ensure that such judgments are made in a rational and defensible manner. It is well established that the major barrier to successful elicitation is the presence of biases inherent in the process used to evoke expert responses (see, for example, the pioneering work of Tversky, Slovic, and Kahneman, 19851. These biases are often characterized as cognitive or motivational, and attributed to a variety of sources, including: intrinsic cognitive failures, the instrument used to elicit responses, the social or institutional setting within which the expert operates, and the response mode. Cognitive biases are evident in effects such as anchoring, the tendency not to adjust from a first response even after receiving information contrary to the position; availability, the elicitation of event probabilities or other values based on what readily comes to mind; conservatism, a reluctance or inability to draw inferences agreeing with those that would be obtained using Bayes' rule; and underestimation, an understatement of the uncer- tainty of an assessment. Motivational biases include group think, whereby experts tend to slant their assessments to what they perceive to be a consen- sus; and misinterpretation, in which the method or instrument of elicitation affects the expert's responses (as when, for example, the framing of a ques- tion cues the expert to provide a preferred response). The test and evaluation environment contains strong institutional in- centives and is therefore possibly subject to equally pervasive motivation biases on the part of experts asked to provide their judgment. These experts can be specifically trained in methods to avoid or mitigate an array of cog- nitive biases, and the elicitations themselves can be structured to minimize the effects of bias. A growing literature of methods addresses these issues. In particular, Meyer and Booker (2001) and Booker and McNamara (2003) provide exemplary guides to such ameliorative methods as indirect prob-

OCR for page 53
PREREQUISITES FOR COMBINING INFORMATION 57 ability assessment, use of documented processes for elicitation, expert iden- tification, motivation and training, modes of communication, and appro- priate framing. If expert judgment is used in methods for combining infor- mation, it is extremely important that these or similar techniques be used, especially when arriving at prior distributions for critical parameters, such as failure rates. Some industrial organizations have become comfortable using such techniques, while being aware of and adjusting for potential biases, in high- profile, politically sensitive analyses. For example, General Motors Corpo- ration reports on its ability to assess technical success probabilities in Bordley (1998) and has used panels of over 40 experts to develop cumula- tive prior probability distributions for the improvement of fuel economy by using a novel powertrain concept. Computer modeling and simulation, which can be thought of as com- bining the original data with the knowledge incorporated in the model, can also provide a cost-effective way of expanding the use of the data. The appropriate use of computer modeling and simulation methods depends crucially on the trustworthiness of the models in transporting data to other scenarios. Although simulation can generate a large amount of new data, it is a serious mistake to combine these generated data directly with the origi- nal data to increase the sample size. Instead, more sophisticated statistical methods (e.g., as described by Reese et al., 2000) should be employed. NEED FOR A TEST DATA ARCHIVE Given the wide variety of data sources available when performing a system assessment, a mechanism should be developed to archive the data and make them available for current and future assessments. At present, such data are not saved in a readily accessible database along with contex- tual information. This is true even for previous development stages of a system. Once a system has been fielded, the absence of rigorous informa- tion on system performance greatly limits the effectiveness offeedback loops relating performance in the field to performance during testing, feedback that could be very useful for improving system designs, the system develop- ment process, and operational and developmental test design. A data archive of military system performance could be put to several uses that would assist in test design and system evaluation. In support of test design, data archiving can be used to:

OCR for page 53
58 IMPROVED OPERATIONAL TESTING AND EVALUATION help set the requirements for the test design and develop the opera- tional mission summary/mission profile (OMS/MP); determine the set of conditions and miniscenarios to be included in the developmental and operational tests; identify scenarios in which the new system is expected to perform better than previous systems (e.g., by providing information on how other systems performed in similar scenarios); similarly, identify scenarios in which the new system may perform poorly; identify factors that have an important impact on system pertor- . mance; understand the factor levels that stress the system weakly, moder- ately, and severely; and determine adequate sample sizes through power calculations. In support of system evaluation, data archiving can be used to provide information to support analysis of the validity of computer models and simulations used in test evaluation; support identification of appropriate statistical models for use in system evaluation; and support pooling and other forms of information combining. With the increasing development of statistical methods and models for information combining this last rea- son has become increasingly more compelling. Data archiving can also contribute to improvement of defense system assessment by providing a means to better understand the differences be- tween failure modes and failure frequencies in moving from developmental to operational testing and from operational testing to field use; understand the sources of system deficiencies identified in the field, which can then be used to guide design improvements; improve both developmental and op- erational testing and evaluation, e.g., by understanding how deficiencies identified in the field escaped detection in the developmental and opera- tional tests; and estimate system and component residual lifetimes and life cycle costs. The current lack of priority for data archiving, given the above advan- tages, suggests that the primary purpose of test data is to evaluate a system for promotion to the next stage of the milestone process of defense system development. Processes and techniques for combining data across acquisi- tion stages either for a given system or across systems are not currently envisioned or well supported. However, such data, often acquired at enor-

OCR for page 53
PREREQUISITES FOR COMBINING INFORMATION 59 mous cost (e.g., operational tests can cost many millions of dollars), could and should be stored in an accessible form that would facilitate the above uses. Averaged over all defense systems in development, the cost of such an archive would be extremely small, but its value, as has been discovered in many industrial settings, could be substantial. A test data archive would need to contain a rich set of variables to adequately represent the test environment, the system under test, and the performance of the system. Failure to initially include such a comprehen- sive set of variables should not be used as an argument for not getting started, since many of the potential benefits from such an archive could be derived from a subset of what is described here, with increasing detail added over time. In order to accurately represent system performance, including the ap- pearance of various failure modes and their associated failure frequencies, the circumstances of the test must be understood well enough that the test, training exercise, or field use can be effectively replicated, including the environment of use (e.g., weather, terrain, foliage, and time of day) and type of use (e.g., mission, intensity, and threat). This information is not easy to collect in controlled settings such as operational testing, and is con- siderably more difficult to collect in less controlled types of use, such as training exercises or field use. However, much in this direction can be ac- complished. In addition, contextual information that might be relevant for an operational test might have little relevance in the developmental test, because often only particular components are under test. While a system is under development, the system design is often under constant modification. Given the need, stated above, to be able to replicate a test event in the database, it is crucial to represent with fidelity the system that was in operation during the event so that proper inference is possible. Since modifications can and do occur during late-stage operational testing and after fielding, this is not only a concern for the developmental test. Even for systems produced at the same stage of development, knowledge of the order and location of manufacture can be useful to understanding why some prototype systems perform differently from others. In addition to storing the length of time between system failures, it is also important to identify which hardware or software component mal- functioned; the maintenance (including repair) record of the system; the time of previous failures; the number of cycles of use between failures; the degree offailure; and any other variables that indicate the stresses and strains

OCR for page 53
60 IMPROVED OPERATIONAL TESTING AND EVALUATION to which the system was subjected, such as speed and payload. It is also useful to include the environments and stresses to which individual system prototypes have been exposed historically (e.g., in transport, storage, and repeated on/off cycling), in order to support comprehensive failure mode analysis, especially if an apparent declining trend in system reliability ap- pears. This sort of information is difficult to collect in less controlled set- tings; however, in many industries sensors have been attached to systems to collect much of the information automatically. The information stored should be both quantitative and qualitative. The latter is important to include because the contextual information needed to help recreate the environment of use often includes qualitative information. To facilitate use across services, such an archive should make use of terminology common across services and, in its design and accessi- bility, should address classification issues. With respect to the structure and function of the database, it should be able to track failures over time and identify systems that, while considerably different, have similar components. These needs argue for a database in which these linkages are facilitated. An analysis of similar data archives in industry would enable the DoD to build on existing processes and tech- niques. The panel is pleased to note that there are defense databases that satisfy some of the above needs; the ATEC Distributed Data Archive and Re- trieval System and several servicowide reliability or failure reporting data- bases are leading examples. However, those that the panel has seen support only a few of the potential benefits listed above, rather than the breadth, structure, and accessibility that we envision. The marginal costs of data collection, input, and maintenance could be easily met through routine allocation of a small percentage of the devel- opment funds from every ACAT I program. The initial fixed costs for the Army might be funded by the Army Materiel Command and other related groups. Finally, as mentioned in Chapter 5 for the Future Combat System, systems developed using evolutionary acquisition provide an additional ar- gument for the establishment and use of a test (and field) data archive, since it is vital to link the performance of the system as it proceeds through the various stages of development. This test and field data archive could (1) assist in operational test design for the various stages of system develop- ment, (2) help in diagnosing sources of failure modes, and (3) assist in operational evaluation.

OCR for page 53
PREREQUISITES FOR COMBINING INFORMATION 61 Recommendation: The Department of Defense should provide the funds to establish a test data archive that will be a prerequisite for combining information for test and evaluation of future systems. REPRESENTATIONS The fault tree represented in Figure 4-1 captures logically how the parts of the system under study interact. The same can be conveyed in reliability block diagrams. These and other classes of representations can be quite useful when assessing systems as large and complex as those evaluated by ATEC. For large, complex systems with heterogeneous data sources, representations of a system have several advantages: they set out a common language that all communities can use to discuss the problem; heteroge- neous data sources can be explicitly located in the representation; and the representation provides an explicit mapping from the problem to the data to the metrics of interest. If the system and its assessment are to be put in a decision context- tor example, an overall assessment of system effectiveness and suitability supporting an acquisition decision the fault trees and block diagrams may need to be embedded in a representation that supports these broader goals and connects the disparate and heterogeneous sources of data. Within the data archive one can use representations of the test environments to under- stand and compare variables such as the environment of use (weather, ter- rain, foliage, and time of day) and type of use (e.g., mission, intensity, and threat) across multiple tests. It is important to develop a set of higher-level representations of the system under evaluation for use both within the data archive and more broadly in the system assessment. These representations, of necessity, change over time as the system and the context of the evaluation change. Standard reliability assessment methods focus on individual parts or simple groups of parts within a system. Assessing the overall reliability and performance of a complex system, however, involves understanding and integrating the reliabilities associated with the subsystems and parts, and this understand- ing and integrating are not always straightforward. Multiple and heteroge- neous data types may exist, and the wider community that owns the system may not understand all the features and relationships that can affect system reliability. One way to illustrate all of the factors that characterize and im- pinge upon system reliability is by building qualitative graphical systems

OCR for page 53
62 IMPROVED OPERATIONAL TESTING AND EVALUATION representations that can be migrated to graphical statistical models to assess reliability. For this reason, it is important that the information stored in a data archive be both quantitative and qualitative, as noted above. Most groups developing complex systems do develop compartmental- ized graphical representations of reliability. These representations may in- clude reliability block diagrams; timelines, process diagrams, or Gantt/ PERT charts dealing with mission schedule and risk; and engineering sche- matics of physical systems and subsystems. But none of these disparate representations capture all aspects or concerns of the integrated complex system. Moreover, since the system is likely under development with users, procurers, planners, managers, designers, manufacturers, testers, and evalu- ators spanning multiple organizations, geographical locations, and fields of expertise, these numerous, specialized, and compartmentalized representa- tions foil attempts for the multiple groups to meaningfully discuss (or even understand) total system reliability or performance. There are sets of methods and graphical representations that capture the full range of features and relationships that affect system reliability. For example, Leishman and McNamara (2002) employ ethnographic methods to elicit a model structure from the pertinent communities of experts in- volved in developing the system. The information on system reliability can initially be captured using "scratch nets" (Meyer and Paton, 2002) (i.e., simple diagrams that sketch out the important features of the system and its decision frame), which also allow a preliminary mapping of the key relationships between features. These scratch nets form the basis for more formalized representations called conceptual graphs (Sowa, 1984), which are a formal graphical language for representing logical relationships; they are used extensively in the artificial intelligence, information technology, and computer modeling communities. Similar to the less formal scratch nets, conceptual graphs use labeled nodes (which represent any entity, at- tribute, action, state, or event that can be described in natural language) and arcs (relationships) to map out logical relationships in a domain of knowledge. The example in Figure 4-2 is a typical use of a conceptual graph to convey the meaning of natural language propositions within the context of a complex system. Generally, representations of complex systems are used to capture higher-level concepts, but the grounding of conceptual graphs in both natural language and formal logic also allows them to be used for expert judgment elicitation (and even potentially for text mining) to build

OCR for page 53
63 1 ~ 1 o W o o o tr In in ~ g o ~ In lo ~ 2 ~ Cot ~ ~ o O A Q ~ ~ O m ~ _~ ,~\ L~',~ be - Cat o o be o Fag H

OCR for page 53
64 IMPROVED OPERATIONAL TESTING AND EVALUATION formal logic models that can then become formal mathematical and statis- tical models. From the initial scratch nets, conceptual graphs are used to create an ontology (i.e., a representation of high-level concepts and main ideas relat- ing to a particular problem domain) for the system. This ontology repre- sents the major areas of the system and its decision frame, such that any pertinent detail that needs to be added to the representation can be added hierarchically under one of the existing nodes. The ontology is also used as a boundary object (i.e., an information object that facilitates discussion and interaction between divergent communities that share common inter- ests but have different perspectives on those interests) so that the diverse stakeholders involved with the project can understand and agree on the features that must be taken into account when assessing system reliability and performance. Building on the ontology, important features and relationships from . . . . . . . . . .. the various existing representations e.g., engineering ( diagrams, tlmellne and process diagrams) are integrated in a conceptual graph (or series of graphs). One of the strengths of conceptual graphs is that they are an effec- tive common format to capture diverse concepts and relationships and thus provide an effective structure for combining information. If the concept or relationship can be described in natural language, it can be represented logically and eventually mathematically (the process works backward as well). The graphical statistical models developed in this process can be eas- ily explained to stakeholder communities because they are representations of natural language in which relationships can be understood without hav- ing to explain the underlying mathematical and statistical notation. Unlike reliability block diagrams and fault trees, conceptual graphs do not correspond directly to a particular statistical model. There must be a translation from the qualitative conceptual graph model to a quantitative model. Bayesian networks, in particular, are a flexible class of statistical graphical models that capture causal relationships Jensen, 1996) in a way that meshes well with conceptual graphs; they are considered flexible be- cause standard reliability diagrams (like block diagrams and fault trees) can easily be represented as Bayesian networks (Almond, 19951. The Bayesian network can also be used to model the conditional dependence and inde- pendence relationships important for specifying the more complex Baye- sian, hierarchical, and random effects models mentioned earlier in this re- port. (For an example of the development of representations and the subsequent use of a Bayesian network for analysis, see Appendix C.)

OCR for page 53
PREREQUISITES FOR COMBINING INFORMATION 65 Not every performance and reliability assessment requires the careful development of an integrated series of representations. However, these kinds of representations do help accomplish the goals of a system evaluation plan by making explicit the relationships among parts of the system and the analysis and by providing an explicit mapping of the evaluation to the data sources and the metrics of interest. The conceptual graph representations are flexible enough to achieve these goals within a framework that can change dynamically as the system and evaluation goals develop. COMBINING INFORMATION FOR COMPLEX SYSTEMS One of the steps required in an industrial reliability assurance program is the use of failure modes and effects analysis (FMEA) and reliability block diagrams to quantify the relationships between a system's subsystems, com- ponents, interfaces, and potential environmental effects. (As noted in the previous section, other representational methods can also be employed to create a unified picture of the system and decision space under consider- ation.) These representations result in a reliability model. For large, complex, changing systems, however, developing and quan- tifying a reliability or performance model can be an extremely challenging problem. Complex systems tend to have complex problems, which usually exhibit one or more of the following characteristics (Booker and McNamara, 20031: a poorly defined or understood system or process, such as high cycle fatigue effects on a turbine engine; a process characterized by multiple exogenous factors whose impacts are not fully understood, such as the effects on a new system of changing combat missions; an engineered system in the very early stages of design, such as a new concept design for a fuel cell; a system, process, or problem that involves experts from different disciplinary backgrounds, who work in different geographical locations, and/or whose problem-solving tools vary widely (as is the case in the work involved to ensure the reliability of a manned mission to Mars); and any new groups of experts in novel configurations brought together for its solu- tion. Any time these sorts of complexities are involved, stakeholders may have difficulties coming to a common understanding of the problem to be addressed. As discussed previously, experts are always involved in the devel- opment and justification of assumptions used for modeling and analyses. In complex systems, one approach to dealing with the difficulties of formu-

OCR for page 53
66 IMPROVED OPERATIONAL TESTING AND EVALUATION rating the model is to formalize the involvement of the experts. Briefly, the stages of expert involvement include the following: 1. Identifying the problem: What is the system under consideration? What are the primary metrics that must be evaluated? Who are the relevant stakeholders and what are their needs and expectations? 2. Operationalizing the problem: What are the operational definitions of the metrics? How will the metrics be evaluated? What are the constraints on the evaluation? 3. Developing the model: What are the core concepts that structure this problem? How are they related to one another? What classes of qualita- tive and quantitative models best fit the problem as currently structured? 4. Integrating and analyzing information: What data sources can be used to characterize this problem? Who owns those data sources? Can the postulated model answer the questions identified previously? 5. Statistical analysis: What are the appropriate techniques to combine the information from the available data sources? What are the appropriate graphical displays of information? What predictions or inferences must be made to support decisions? NEED FOR ADDITIONAL STATISTICAL CAPABILITIES The procedures, both informal and formal, for combining informa- tion comprise a broad set of techniques ranging from those that are ex- tremely easy to apply (being robust to the precise circumstances of the application and with solutions in a simple, closed form) to very sophisti- cated models that are targeted to a specific application, require great imagi- nation and technical expertise to identify and construct, and often require additional technical expertise to implement, possibly involving software development.] The more sophisticated category contains the rich collec- tion of hierarchical and random effects models that have enjoyed recent, very rapid development and that have been successfully applied to a large number of new situations. Flexible, public-use software currently exists for a rich set of models, greatly simplify- ing software development. For example, R (http://cran.r-project.org/), BUGS (Bayesian In- ference Using Gibbs Sampling), and WinBUGS (http://www.mrc-bsu.cam.ac.uk/bugs/ winbugs/contents.shtml) are widely used in a variety of fields of application and are available at no cost on the Internet.

OCR for page 53
PREREQUISITES FOR COMBINING INFORMATION 67 This report has described the potential applicability of various infor- mation-combining techniques to operational test design and evaluation for defense systems. Given the great complexity of ACAT I defense systems, and the many important facets of their development and evaluation, it is likely that many of these systems will require technically complicated meth- ods to support their evaluation. It is also likely that use of these compli- cated methods will provide tangible benefits in operational evaluation. With respect to both hierarchical and random effects modeling, while there are some standard models that have been repeatedly applied and that may be useful for some defense applications, it is very likely that procedures at the leading edge of research will often be needed for high-quality operational test designs and evaluations. Operational test evaluation is carried out under fairly substantial time pressures, in circumstances where errors can have extremely serious conse- quences. Experts with demonstrated proficiency in the use of combining information methods are required to ensure their fast, correct application. Furthermore, so that the ultimate decision makers can fully understand the findings based on these techniques, careful articulation of the methods and findings, including the important contribution of sensitivity analyses of divergence from assumptions, is very important, and also argues for the involvement of individuals with a complete understanding of the methods and their strengths and weaknesses. Though there are exceptions, these sophisticated techniques are ordi- narily not fully understood and correctly applied by those with a master's degree in statistics. A doctorate in statistics or a closely related discipline is generally required. This raises the question as to how ATEC can gain access to such expertise. The 1998 NRC report (National Research Council, 1998) mentioned available expertise at the Naval Postgraduate School, the Insti- tute for Defense Analyses, RAND, and other federally funded research and development centers, as well as academia. This panel generally supports the recommendations contained in that report. One complication with the use of statisticians either on staff or, even more crucially, as consultants, is that more than statistical expertise will be required. Statisticians working on the methodology for system evaluation will need to work in close collaboration with experts in defense acquisition, military operations, and the system under test. Knowledge of physics and engineering would also be extremely useful. Given the need for collaboration and acquired expertise, having appro- priately qualified on-site staff is clearly the best option. Another option that .. . . . .

OCR for page 53
68 IMPROVED OPERATIONAL TESTING AND EVALUATION should be considered, especially in the short term, is to offer sabbaticals and other temporary arrangements to experts in industry and academia. The panel suggests that one approach, which would institutionalize the use of sabbatical arrangements but that would require cooperation of the ser- vices, would be for each service to create an Interagency Personnel Act (PIA) position for a statistical expert for test and evaluation, reporting to the head of each service's operational test agency. DOT&E could also cre- ate a similar position. These statistical experts would work both as indi- vidual resident experts in each of the service test agencies, and would also be available to work jointly on the most challenging test and evaluation issues in the DoD. These temporary positions would rotate every three years and would have sufficient salary and prestige to attract leading statis- ticians from academia and industry. In addition, ATEC should consider making available all sources and types of information for a candidate defense system to a selected group of qualified statisticians in industry and academia as a case study to under- stand the potential advantages of combining information for operational evaluation. Recommendation: ATEC should examine how to increase statistical capabilities to support future use of techniques for combining infor- mation. As a first step, ATEC should consider providing all sources of information for a candidate defense system to a group of qualified statisticians in industry anti academia as a case stutly to untlerstantl the potential advantages of combining information for operational evalua- t~on.