Improved Operational Testing and Evaluation and Methods of Combining Test Information for the Stryker Family of Vehicles and Related Army Systems Phase II Report (2004) / Chapter Skim
Currently Skimming:
2. Examples of Combining Information
2. Examples of Combining Information
Pages 17-39
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 17... ...
We then discuss its use more generally in planning the test, selecting the experimental design, and selecting sample sizes for testing. Following this, we explore a variety of techniques in which prior information can be used in combination with test data to provide assessments of system performance.
|
From page 18... ...
:1 mission X company pair Since prior knowledge suggests that company pair is not likely to interact with any of the factors, the 8-run fractional factorial design presented in Table 2-1 can be used to safely estimate the three two-factor interactions of interest: mission X intensity, mission X terrain, and terrain X intensity. This achieves reduction of the total number of possible test combinations by half, saving costs and time during the operational testing phase.
|
From page 19... ...
Given the high stakes, it is critical that operational testing be planned and executed carefully and systematically and that as much relevant prior information as possible be taken into account in designing efficient test plans. It is difficult, and in some cases impossible, to generate useful information from a poorly designed test plan.
|
From page 20... ...
Some industrial organizations make use of predesign master guide sheets (see, e.g., Coleman and Montgomery, 1993) that query the test designers to specify the objectives of the test, any relevant background issues, response variables, control variables, factors to be held constant, nuisance factors, strong interactions, any further restrictions on test inputs, design preferences, analysis and presentation techniques, and responsibility for coordination.
|
From page 21... ...
Related discussions of Bayesian optimal designs examine formal incorporation of prior information about model parameters (Chaloner, l 9841. Selecting Sample Sizes Selection of sample sizes is dependent on the objective of the operational test.
|
From page 22... ...
. The variance of the test statistic is not directly measured prior to carrying out the operational test; however, it can often be indirectly estimated through use of development test information, pilot studies, or variances estimated for similar systems and adjusted through the use of engineering judgment.
|
From page 23... ...
For example, one might have collected times to first failure for several systems in developmental testing and for an additional, smaller number of systems in operational testing. If all samples are pooled into one large sample regardless of where they came from, the required assumption is that the origin of each sample has no impact on the distribution of sample values.
|
From page 24... ...
. Such tests are rank tests and are sensitive to a wide range of differences in the individual empirical distribution functions, in contrast to the analysis of variance Ftest for equality of means (assuming common variances and normality)
|
From page 25... ...
This pooling, as usual, depends on the validity of the various assumptions, and diagnostic checks including residual analyses need to be made before building on them. Pooling using regression is a special case of a more general approach, including generalized linear models and various nonparametric fitting techniques, which can be applied to normal, count, and other forms of data.
|
From page 26... ...
Given this prior distribution and a statistical model for the data, Bayes Theorem produces the posterior distribution that represents the subjective probabilities for different values of p based on both the observed data and the prior information.2 In this case it is natural to assume for the statistical model that the observed number of failures y is distributed as a binomial random variable with 20 trials, each having failure probability p. The resulting posterior distribution can provide an estimate of p and a probabilistic upper bound, or any other summary of uncertainty about p, based on the data and prior information.
|
From page 27... ...
The table gives a point estimate for the median and the 95th percentile of the posterior distribution. For purposes of comparison, the table also shows the uncombined maximum likelihood point estimate for p and upper confidence limits based on the binomial model and operational test data alone.
|
From page 29... ...
In this situation, the prior information seems to have been inappropriate, and the process by which it was generated should be examined. This short example demonstrates a way to quantify and combine expert opinion with observed data in a relatively simple setting.
|
From page 30... ...
In row 4, there were 125 units in service for around 300 hours. Figure 2-3 presents a Weibull probability plot of the same bearing cage data, showing the maximum likelihood estimate of fraction failing, the reliability target, and approximate confidence limits.
|
From page 31... ...
In the figure, the dots represent the bearing cage observed data, straight line (a) represents the maximum likelihood estimate of fraction failing, intersection (b)
|
From page 32... ...
also suggests using given values for the Weibull shape parameter ~ when there are few failures in censored life data, but strongly encourages using sensitivity analysis to assess the effect of the uncertainty in the Weibull shape parameter because the value is never in practice known with certainty. The range of evaluation can be determined from past experience with the same failure mode in similar materials or components.
|
From page 33... ...
In this example the engineers responsible for the reliability of the bearing cage have useful prior information on the Weibull shape parameter, which they quantify with a lognormal distribution with lower and upper 99 percent limits (1.5, 31. For the B10 parameter itself there is little prior information, so a diffuse prior distribution is used by specifying a loguniform distribution with lower and upper limits (500, 20,0001.
|
From page 34... ...
Five types of developmental testing have been carried out on five sets of motors. For each set of motors, the means and standard deviations of the failure times were observed as follows: Mean Standard Deviation Test 1 87.0 5.0 Test 2 83.0 3.5 Test3 67.0 3.0 Test 4 77.0 4.0 Tests 70.0 5.0 Classically, these various sources of information would be joined using a linear combination of the separate estimates weighted inversely propor
|
From page 35... ...
. To build the likelihood from the experiments, we assume that the failure times have mean ~ and standard deviations that we will estimate using the data (though combining information approaches to determine the standard deviations could also be used if there were relevant prior information)
|
From page 36... ...
(In a less simplistic situation, one would, of course, be concerned with failure modes appearing in operational testing that did not appear in the developmental test.) The operational test data can then be used to update the estimated probabilities of these situations.
|
From page 37... ...
We can update a prior distribution about the mean distance to failure, using operational test data, to arrive at a posterior distribution. This posterior distribution will itself have a mean, the expected mean distance to failure, and a standard deviation.
|
From page 38... ...
Although this approach to combining information from developmental and operational testing is a tempting means to increase the efficiency of operational test results, a number of potential difficulties remain. To the extent that an analyst must speculate about possible situations that have not been realized, an assessment of their probabilities may be more vulnerable to cognitive biases than the better understood assessment of distribu
|
From page 39... ...
Moreover, inferences made about performance measures are couched in language appropriate for decision making. In summary, inferences about the number of failure modes that have been fixed prior to OT, the number of new failure modes that OT has introduced, and related problems can be addressed using combining information techniques.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.