6


First Article Testing Protocols for Resistance to Penetration:
Statistical Considerations and Evaluation of DoD Test Plans

6.0 SUMMARY

The test protocols for Army helmets were originally based on a requirement of zero penetrations in 20 shots (five shots on each of four helmets). The Director, Operational Test and Evaluation (DOT&E) protocol replaced this legacy plan with a requirement of 17 or fewer penetrations in 240 shots (five shots on each of 48 helmets). The helmets spanned four sizes and were tested in four different environments. The 0-out-of-20 (0, 20) plan and DOT&E’s 17-out-of-240 (17, 240) plan have comparable performance if the probability of penetrating a helmet shell on a single shot is around 0.10. As noted in Chapter 5, available data indicate that penetration probabilities are around 0.005 or less. Near this value of penetration probability, both plans have a 90 percent or higher chance of passing the test, so the manufacturer’s risk is small, as it should be. However, if there is a 10-fold increase in the penetration probability from the current level of 0.005 to 0.05, DOT&E’s (17, 240) plan still has a 95 percent chance of acceptance. This may not provide sufficient incentive for the manufacturer to sustain current penetration-probability levels. Thus, the (17, 240) plan may have the unintended effect of leading to a reduction in helmet penetration resistance. In the absence of a link between penetration probability and human injury, there is no scientific basis for setting a limit on the penetration probability. In such a circumstance, the committee’s view is that the objective of a new test plan should be to provide assurance that newly submitted helmets are at least as penetration-resistant as current helmets. This chapter proposes appropriate criteria for selecting test protocols and illustrates their use through several plans.

6.1 INTRODUCTION

The primary goal of this chapter is to evaluate DOT&E’s protocol for testing a helmet’s resistance to penetration (RTP). The committee compares its performance with that of the Army’s legacy plan and a modified version of the DOT&E plan that has recently been adopted by the Army. A modification of the current protocol for the enhanced combat helmet (ECH) is also examined. These discussions are directly relevant to the issues raised in the correspondence between U.S. Representative Slaughter and the Department of Defense. To provide adequate background, the chapter begins with an overview of the statistical considerations in the design of test protocols for RTP. The chapter ends with a discussion of several topics: (1) robustness of the operating characteristic (OC) curves when the penetration probabilities vary across different test conditions; (2) examination of possible protocols for testing by helmet sizes; (3) post-test analysis of the RTP data to determine the achieved penetration probabilities of the tested helmets; and (4) a proposal to base future protocols with the helmets as the test unit rather than shots.

6.2 STATISTICAL CONSIDERATIONS IN DESIGNING TEST PLANS FOR RESISTANCE TO PENETRATION

As described in Chapter 4, the RTP test protocol specifies that helmets of different sizes be conditioned in selected environments and that shots be taken at different locations on the helmet. However, in this section, the committee starts with a simple setup—a single helmet size, a single shot location on the helmet, and a single environment—so that the test deals with a homogeneous population of units and a single test environment. (To be specific, one can think of a medium helmet, top location on the helmet, at ambient temperature.) It is then reasonable to view the penetration outcomes when n helmets are tested in this manner as being independent and identically distributed binary (pass/fail) random variables with constant penetration probability θ. Thus, the probability distribution of X, the (random) number of penetrations in n shots, is a binomial distribution with parameters (n, θ). The statistical properties of a test plan can be derived from this distribution.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 39
6 First Article Testing Protocols for Resistance to Penetration: Statistical Considerations and Evaluation of DoD Test Plans 6.0  SUMMARY DOT&E plan that has recently been adopted by the Army. A modification of the current protocol for the enhanced com- The test protocols for Army helmets were originally based bat helmet (ECH) is also examined. These discussions are on a requirement of zero penetrations in 20 shots (five shots directly relevant to the issues raised in the correspondence on each of four helmets). The Director, Operational Test between U.S. Representative Slaughter and the Department and Evaluation (DOT&E) protocol replaced this legacy plan of Defense. To provide adequate background, the chapter with a requirement of 17 or fewer penetrations in 240 shots begins with an overview of the statistical considerations in (five shots on each of 48 helmets). The helmets spanned four the design of test protocols for RTP. The chapter ends with a sizes and were tested in four different environments. The discussion of several topics: (1) robustness of the operating 0-out-of-20 (0, 20) plan and DOT&E’s 17-out-of-240 (17, characteristic (OC) curves when the penetration probabili- 240) plan have comparable performance if the probability of ties vary across different test conditions; (2) examination of penetrating a helmet shell on a single shot is around 0.10. As possible protocols for testing by helmet sizes; (3) post-test noted in Chapter 5, available data indicate that penetration analysis of the RTP data to determine the achieved penetra- probabilities are around 0.005 or less. Near this value of pen- tion probabilities of the tested helmets; and (4) a proposal to etration probability, both plans have a 90 percent or higher base future protocols with the helmets as the test unit rather chance of passing the test, so the manufacturer’s risk is small, than shots. as it should be. However, if there is a 10-fold increase in the penetration probability from the current level of 0.005 to 0.05, DOT&E’s (17, 240) plan still has a 95 percent chance 6.2  STATISTICAL CONSIDERATIONS IN DESIGNING of acceptance. This may not provide sufficient incentive for TEST PLANS FOR RESISTANCE TO PENETRATION the manufacturer to sustain current penetration-probability As described in Chapter 4, the RTP test protocol speci- l ­evels. Thus, the (17, 240) plan may have the unintended fies that helmets of different sizes be conditioned in selected effect of leading to a reduction in helmet penetration resis- environments and that shots be taken at different locations tance. In the absence of a link between penetration probabil- on the helmet. However, in this section, the committee starts ity and human injury, there is no scientific basis for setting a with a simple setup—a single helmet size, a single shot loca- limit on the penetration probability. In such a circumstance, tion on the helmet, and a single environment—so that the test the committee’s view is that the objective of a new test plan deals with a homogeneous population of units and a single should be to provide assurance that newly submitted helmets test environment. (To be specific, one can think of a medium are at least as penetration-resistant as current helmets. This helmet, top location on the helmet, at ambient temperature.) chapter proposes appropriate criteria for selecting test proto- It is then reasonable to view the penetration outcomes when cols and illustrates their use through several plans. n helmets are tested in this manner as being independent and identically distributed binary (pass/fail) random variables 6.1  INTRODUCTION with constant penetration probability θ. Thus, the probability distribution of X, the (random) number of penetrations in n The primary goal of this chapter is to evaluate DOT&E’s shots, is a binomial distribution with parameters (n, θ). The protocol for testing a helmet’s resistance to penetration statistical properties of a test plan can be derived from this (RTP). The committee compares its performance with that distribution. of the Army’s legacy plan and a modified version of the 39

OCR for page 39
40 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS c-out-of-n Test Plans The y axis in Figure 6-1 shows the probability that a (c = 1, n = 40) test will be successful as a function of the The test plans used by DOT&E for RTP are of the fol- underlying penetration probability θ. These acceptance lowing form: take n shots, and if c or fewer penetrations are probabilities are given by the cumulative distribution, P(X ≤ observed, the first article testing (FAT) passes; otherwise, it 1| θ), where X has a binomial distribution with n = 40 and fails. In this study, the committee refers to such tests as (c, n)- penetration probability equal to θ. For example, if θ, the plans. They are also called binomial reliability demonstra- underlying (unknown) penetration probability, equals 0.02 tions plans or acceptance-sampling plans for attribute data. (green line), the probability of acceptance is 0.8 (80 percent The plan is defined by the value of two constants: c and n. chance of passing). If θ = 0.10 (red line), the probability Once these are specified, the protocol’s properties are deter- of acceptance is approximately 0.10. Conversely, in order mined and can be studied through its operating characteristic to have a probability of acceptance of 0.6 (black line), the (OC) curve. An OC curve is a plot of the probability (P) of true penetration probability needs to be about 0.38. So the acceptance (y axis) against the underlying failure (penetra- OC curve describes the relationship between the acceptance tion) probability of the items under test (x axis). Figure 6-1 probabilities and the underlying penetration probability as θ shows the OC curve for a (c = 1, n = 40) test plan; i.e., the ranges across values of interest. FAT is successful if there are one or fewer penetrations in Suppose the decision maker examined the OC curve for 40 shots. the 1-out-of-40 (1, 40) plan in Figure 6-1 and decided that In Figure 6-1 and subsequent plots of OC curves in this the acceptance probability of 0.10 when θ = 0.10 is too high. report, the x axis is the true (but unknown) penetration There are two options for reducing this value: decreasing c probability θ. This format is different from the OC curves or increasing n. that are currently used by the Army and DOT&E that plot Figure 6-2 provides a comparison with two alternatives: the probability of nonpenetration in the x axis. One should 0-out-of-40 (0, 40) and 1-out-of-70 (1, 70) plans. For both focus on the penetration probability, because it is easier to (c = 0, n = 40) and (c = 1, n = 70) plans, the acceptance interpret the curve as the penetration probability changes. probabilities are close to zero for θ = 0.10. This may be For example, an increase in θ from 0.005 to 0.05 is easy acceptable to the decision maker who is the purchaser in to interpret as a 10-fold increase in penetration probability; this situation. But one cannot discriminate between the two it is hard to interpret this change in terms of 1 – θ, which plans at this value of θ. decreases from 0.995 to 0.95. Consider the case where the target penetration probability is θ = 0.01. Figure 6-2 shows that, at this level, the (0, 40) Recommendation 6-1. The operating characteristic curves plan has an acceptance probability of about 0.63, while the used by the Department of Defense should display penetra- 6-1 (1, 70) plan has an acceptance probability of about 0.83. tion probabilities rather than non-penetration probabilities Since this is the target penetration probability, the decision on the x axes. maker will want to accept helmets with a high probability and will choose the (1, 70) plan or another plan that provides an even higher acceptance probability at θ = 0.01. 6-1 1.0 0.8 Probability of Acceptance n c 1.0 40 1 40 0 70 1 0.6 0.8 Probability of Acceptance n sample size c acceptance number 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Probability of Penetration Probability of Penetration FIGURE 6-1 Operating characteristic curve for (c = 1, n = 40) FIGURE 6-2 Operating characteristic curves comparing 1-out- test plan. The green and red lines show the probabilities of accep- of-40 test plan with 0-out-of-40 and 1-out-of-70 test plans. The blue tance for the plan when the true probabilities of penetrations are, lines show the probabilities of acceptance for the two plans when respectively, 0.02 and 0.10. The black line shows that, if we want the true probability of penetration is 0.1; the green lines show the the probability of acceptance to be 0.6, the true penetration prob- corresponding acceptance probabilities when the true penetration ability has to be 0.38. probability is 0.005.

OCR for page 39
FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 41 Because manufacturers want to have a high probability P( X ≤ c | n, θ = θH) ≤ β Equation 6.2 of passing the test, their helmet design and manufacturing process should attain a penetration probability that achieves In quality control terminology, θL is the “acceptable quality this goal. For example, to have a 90 percent chance of pass- level” for the plan, and θH is the “rejectable quality level.” ing under the (0, 40) plan, the penetration probability will There are two kinds of errors that can occur in the (c, n) need to be about 0.003. To pass the (1, 70) test, penetration accept-reject decision. The first error is to reject the helmet probability will need to be about 0.008, which is not as (fail the acceptance test) when the underlying penetration stringent a target as is set by the (0, 40) plan. These are the probability is at the low (or desired) value (i.e., θ ≤ θL); this kinds of considerations and trade-offs that go into selecting is often referred to as producer’s or manufacturer’s risk. The a test plan. The next subsection provides a discussion of test term manufacturer’s risk is used in this report. Equation 6.1 designs that are derived by specifying two points on a plan’s limits the probability of this error to at most α. The second OC curve. error is to accept helmets when the penetration probability is A few additional remarks on Figure 6-2: too high (i.e., for values of θ ≥ θH). These are usually called consumer’s or customer’s risk. The committee refers to this • The OC curve for the (0, 40) plan is always below that risk as government’s risk in this report. As shown by Equa- of the (1, 40) plan. This is intuitively clear because tion 6.2, the probability of this error is at most β. These are the (0, 40) plan is more stringent (it has the same the Type I and Type II error probabilities in the correspond- sample size but accepts fewer failures), so the prob- ing statistical hypothesis testing formulation of the problem. ability of passing the test is lower. Equations 6.1 and 6.2 specify the cumulative binomial • The OC curve for the (1, 70) plan is always below that acceptance probabilities at two points. By setting the inequal- of the (1, 40) plan. This is also obvious because the ities as equalities, one can solve them to get the values of test (1, 70) has a larger sample size but allows the same size, n, and acceptance limit, c, that satisfy these equations. number of failures as the (1, 40) plan. Because the binomial distribution is discrete, one typically • More generally, consider two plans that have OC cannot achieve the equalities for α and β exactly. (There are curves that cross, such as the (0, 40) and (1, 70) plans catalogs of test plans and software that can be readily used to in Figure 6-2. The two plans cross at a penetration obtain the values of c and n to meet particular risks.) probability of 0.05. To the left of that point, the (1, As a concrete example, suppose the test should be 70) plan has the higher acceptance probability. To the designed to ensure that helmets with an underlying penetra- right, the (0, 40) plan has the higher probability of tion probability of θ = 0.005 have at least a 90 percent chance acceptance (although the differences are quite small). of passing the test. So θL = 0.005 and (1 – α) = 0.90, or α = 0.10. Further, suppose it was decided that if the penetra- The different perspectives of manufacturer and purchaser could lead them to prefer different plans. Different plans 6-3 could be considered and evaluated and a compromise plan could be negotiated. Alternatively, as described in the next subsection, plans can be derived from specifications of manu- facturer’s and purchaser’s risks. 1.0 0.8 Probability of Acceptance Statistical Approaches to Selecting (c, n)-Test Plans 0.6 The conventional statistical approach for choosing a test plan is to specify two points on the OC curve: (1) a 0.4 low penetration-probability, θL, at which a high acceptance probability, denoted by (1 – α), is desired (a manufacturing 0.2 process that produces good helmets has a high probability of being accepted), and (2) a high penetration-probability, 0.0 θH, at which a low acceptance probability β is desired (a 0.00 0.02 0.04 0.06 0.08 0.10 manufacturing process that produces poor helmets has a high Probability of Penetration probability of being rejected). Expressing these objectives algebraically leads to the following two equations: FIGURE 6-3 Operating characteristic curves of (c = 1, n = 77) plan with the desired risks. The black line shows the probability P( X ≤ c | n, θ = θL ) ≥ (1 – α) Equation 6.1 of acceptance for the plan when the true probability of penetration is 0.1; the green and red lines show the corresponding acceptance and probabilities when the true penetration probabilities are, respec- tively, 0.005 and 0.02.

OCR for page 39
42 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS tion probability is θ = 0.05, which is an order of magnitude The committee performed numerical investigations to higher, there must be at most a 10 percent chance of passing examine the differences between the true OC curves and the the test. So, θH = 0.05, and β = 0.10. Therefore, the test is OC curves obtained by assuming that the penetration prob- designed to discriminate between helmets with penetration abilities are the same across all shots. It examined a range of probabilities of 0.005 and 0.05. In this example, both α and deviations for the penetration probabilities. Further, it took β are the same, but they do not have to be. These two risks the constant penetration probability for comparison to be the are specified by the decision maker. average of the varying probabilities. The study shows that the Figure 6-3 shows the OC curve for the 1-out-of-77 (1, differences in the OC curves are negligible for the range of 77) test plan that meets the above requirements. It has the penetration probabilities and deviations that are relevant to desired properties at the specified penetration probabilities the helmet situation. of 0.005 and 0.05. In practice, however, after a plan has been obtained, one should also examine its OC curve at other Finding 6-1. RTP data aggregated over helmet sizes, values of θ to see if it has reasonable (not too low or not too environments, and shot locations may not have a constant high) acceptance probabilities. In this case, if θ = 0.02 (a underlying penetration probability. An evaluation of operat- four-fold increase from the desired penetration probability), ing characteristics for modest departures from this situation the acceptance probability is about 0.55. One may decide indicates that the actual acceptance probabilities are negli- that this is too high and look for a more stringent plan—say gibly different from those calculated assuming a constant one with c = 1 but a larger value of n. That change, however, underlying penetration probability. This means that the OC would increase the manufacturer’s risk and decrease the gov- curves computed under the assumption of constant prob- ernment’s risk. The OC curve of an acceptance plan conveys ability provide very good approximations. a variety of incentives and disincentives to stakeholders in the acceptance decision. 6.3  STATISTICAL EVALUATION OF DOD PROTOCOLS FOR RESISTANCE TO PENETRATION Zero-Failure Plans “Legacy” Protocol for the Advanced Combat Helmet A common class of test protocols is based on zero-failures (i.e., c = 0). One reason is that the lower the value of c, The legacy protocol, first specified by the program man- the smaller the number of units to be tested, n, in order to ager for the Advanced Combat Helmet (DoD IG, 2013), was achieve a particular level of government’s risk. However, a (0, 20) test plan. It involved testing four helmets, one each there may be a false perception associated with zero-failure at four test environments (ambient, hot, and cold tempera- plans: Because it does not allow any failures, the quality of tures and seawater). Only large-size helmets were tested. For the products must be, in general, considerably higher than each helmet, the protocol required shooting a 9-mm bullet at the government’s threshold quality. It is clear but worth five different locations, for a total of 20 shots. The five shots reiterating that a zero-failure plan does not imply that the on each helmet were in a fixed shot sequence and pattern. No penetration probability is zero! For example, if the penetra- penetrations were allowed (i.e., it was a zero-failure plan). tion probability is 0.03, the probability of zero penetrations in 20 shots is 0.54. This means that, even though there is a 3 percent chance of penetration, the 0-out-of-20 failure plan will pass the test more than half of the time. Therefore, an outcome of 0/20 does not imply zero penetration probability. 0.005 0.1 1.0 Robustness to Deviations from the Binomial Distribution 0.9 Probability of Acceptance 0.8 The preceding subsection was based on a framework 0.6 in which the penetration probability θ was constant across all shots. This assumption does not strictly hold in helmet 0.4 testing: the helmets are of different sizes, they are tested at 0.2 different environmental conditions, and the shots are taken 0.12 at multiple locations on the helmet. It is possible that the 0.0 penetration probability is different at different helmet loca- 0.00 0.05 0.10 0.15 0.20 penetration probability tions. When the penetration probabilities vary across shots, the number of penetrations, X, in n shots would not have a FIGURE 6-4 Operating characteristic curve for the legacy (0, 20) binomial distribution. Therefore, the OC curves computed test plan. The darker dashed lines show the probabilities of ac- under this model would not apply exactly. The question of ceptance for the plan when the true penetration probabilities are interest is whether the binomial calculations are still useful. 0.10 and 0.005.

OCR for page 39
FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 43 2007 to take over the responsibility to prescribe policy and 1.0 n c procedures for the conduct of live-fire test and evaluation of 20 0 240 17 body armor and helmets (DoD IG, 2013). DOT&E decided to increase the number of helmets n sample size 0.8 c acceptance number Probability of Acceptance tested to 48 in order to cover a range of conditions and to 0.6 have adequate precision in comparing any differences in penetration probability, or BFD, due to environment, helmet 0.4 size, and shot location. The new protocol called for testing 48 helmets, 12 each for Small, Medium, Large, and Extra 0.2 Large sizes. Three helmets of each size were conditioned in the four environments before testing. There were five shots 0.0 at different helmet locations, leading to a total of 240 shots. 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Probability of Penetration There are good statistical reasons to justify DOT&E’s increase in the number of helmets tested to 48 helmets from FIGURE 6-5 Comparison of the operating characteristic curves for the Army’s 5. One gets more precise estimates of the pen- (0, 20) and (17, 240) plans. The blue lines show the probabilities etration probability from 240 shots than from 20 shots. In of acceptance for the two plans when the true probability of pen- addition, DOT&E’s plan allows better statistical comparison etration is 0.1; the purple and green lines show the corresponding of possible differences between helmet sizes and environ- acceptance probabilities when the true penetration probabilities are, mental conditions. respectively, 0.005 and 0.05. To examine the properties of the (c = 17, n = 240)-plan, recall that if n is specified, one can control only one point on the OC curve, or one of the two risks, by the choice of This legacy test plan was adapted from prior helmet c. With n chosen, the DOT&E approach was to specify that, protocols and was not designed to meet specified statistical for penetration probability of 0.10, the probability of accep- risks. Nevertheless, one can study its properties through its tance (the government’s risk) should be no more than 10 OC curve in Figure 6-4. The acceptance probability is about percent. This is referred to as the 90/90 plan (corresponding 0.12 when the penetration probability is 0.10. In other words, to a rejection probability of at least 0.90 at nonpenetration if the underlying shot penetration probability is 0.10, the hel- probability of 0.90). To summarize, DOT&E’s (17, 240) plan mets will fail the demonstration test 88 percent of the time. was chosen by first increasing the sample size n to be 240 Consider the behavior of the curve to the left of θ = 0.10 for statistical reasons. Then, the 90/90 standard was applied and the implications for manufacturers. If a manufacturer to get the maximum number of acceptable failures to be 17. wants to have a 90 percent chance or higher of passing the Thus, there is a direct relationship between the 90/90 stan- (0, 20) test, the helmet design and production process would dard and the (17, 240) plan. have to achieve a penetration probability of θ = 0.005 or less. However, there is no scientific or empirical basis for Note that the manufacturer has to achieve a penetration specifying 0.10 as the acceptable limit for a helmet’s pen- probability considerably less than the government’s standard etration probability. It appears that the 90/90 standard was of θ = 0.10 to have a good chance of passing the (0, 20) test. chosen because of its use in body armor protocols1 and also While the government, by its specification of θ = 0.10 as its because the legacy protocol approximately had this property. limit on penetration probability, may be willing to purchase That specification led to the (c = 17, n = 240) test plan. The helmets with, say, θ = 0.075, the manufacturer would not aim committee does not know if there was any attempt to control at that target because the chance of passing the (0, 20) test is the manufacturer’s risk. too low for comfort—about 0.20 in Figure 6-5. Figure 6-5 provides a comparison of the OC curves for As noted earlier, the government’s risk at θ = 0.10 was the (0, 20) and (17, 240) plans. The two OC curves cross 0.12. So, this plan does not strictly satisfy the 90/90 property at about θ = 0.092. The (0, 20) plan has higher acceptance (at most 10 percent government’s risk at penetration prob- probabilities to the right of this penetration probability and ability 0.10 or, equivalently, at least 90 percent chance of has lower acceptance probabilities to the left. The two plans failing the test if the nonpenetration probability is 0.90.) One have about the same acceptance probabilities (government needs a 0-out-of-22 (0, 22) plan to satisfy this requirement. risks), in the neighborhood of θ = 0.10, as intended. The 90/90 criterion was explicitly adopted by DOT&E in its When θ = 0.005, near the region where the manufactur- subsequent protocols. ers are currently operating (see Chapter 5), the acceptance probability of the (0, 20) plan is about 0.9, while that of the DOT&E’s (c = 17, n = 240) Protocol (17, 240) plan is essentially 1.0. Thus, the (17, 240) plan has In response to a Senate and House Armed Services Com- mittee’s request, the Secretary of Defense asked DOT&E in 1Personal communication between Christopher Moosmann, DOT&E, and Nancy Schulte, NRC, via e-mail on May 14, 2013.

OCR for page 39
44 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS lower manufacturer’s risk. Director Gilmore’s letter to Rep. there is a 10-fold increase in the current penetration prob- Slaughter (see Appendix A) recognized that the DOT&E ability (from 0.005 to 0.05), this may provide a disincentive protocol would lessen the burden on manufacturers to pass to maintain current levels of penetration resistance. In this the test with helmets with an underlying penetration prob- sense, the (17, 240) plan is not as good as the legacy plan ability less than the “standard” of 0.10. However, this is not of (0, 20). necessarily an advantage. Consider a comparison of the two plans when the penetra- It is likely that manufacturers are more motivated by tion probability equals 0.05, which is a 10-fold increase in having a high probability of passing the test than they are the penetration probability from the currently achieved level in avoiding a penetration probability at the current DOT&E of around 0.005. For this value of θ = 0.05, the acceptance “standard” of 0.10, a value nearly two orders of magnitude probability is about 0.38 for the (0, 20) plan, while it is about higher than what current data indicate for a helmet penetra- 0.95 for the (17, 240) plan. Thus, even if there is a 10-fold tion probability. If manufacturers have a very high probabil- degradation in the penetration resistance of helmets, there ity of passing the test, even if there is a substantial increase is a 95 percent chance of accepting the helmets under the in the penetration probability, the (17, 240) plan may have DOT&E protocol. Similar comparisons can be made at other the unintended effect of leading to a reduction in helmet values of θ to the left of the point where the two curves cross. penetration resistance. For example, for any values of penetration probability of θ ≤ 0.04—a five-fold increase—the helmets will almost certainly Recommendation 6-2. If there is a scientific basis to link be accepted. To the right of the crossing point, however, the brain injury with performance metrics (such as penetra- (0, 20) plan has a higher acceptance probability (and hence tion frequency and backface deformation), the Director of poorer performance in terms of screening out helmets with Operational Test and Evaluation (DOT&E) should use this high penetration probabilities, but still less than a 12 percent information to set the appropriate standard for performance chance of acceptance). metrics in the test protocols. In the absence of such a sci- A decision on which of the two plans is better comes down entific basis, DOT&E should develop a plan that provides to deciding what is the relevant range of values of the pen- assurance that it leads to the production of helmets that are etration probability. DOT&E’s (17, 240) plan focuses around at least as penetration-resistant as currently fielded helmets. θ = 0.10, and its main objective is to prevent helmets with a 0.10 penetration probability or more from being accepted. Enhanced Combat Helmet Protocol: Modified DOT&E The (17, 240) plan has comparable performance to the (0, Protocol 20) plan at this point and has lower acceptance probabilities for θ ≥ 0.10. So if this is the region of interest, then the (17, The ECH protocol, a modification of the DOT&E pro- 240) plan is superior to the (0, 20) plan. However, if the tocol, is a 5-out-of-96 (5, 96) plan that involves taking two objective of the plan is to provide an incentive for manufac- shots each at 48 helmets. The acceptance limit of c = 5 is turers to produce helmets at least as good as current helmets based on the 90/90 criterion. Figure 6-6 provides a compari- (θ ≤ 0.005), the (0, 20) plan is better in that it has a lower son of its OC curve with that of the (0, 20) plan. It shows that, probability of acceptance for helmets that are not as good if the penetration probability is 0.035, the manufacturer’s risk as current helmets up to a penetration probability of 0.10. To evaluate a plan, one needs to consider the whole OC curve, not just one point that may have been used to specify the plan. The DOT&E plan focuses on the point at which θ = 0.10. Its main objective is to prevent helmets with a 0.10 1.0 n c 20 0 penetration probability or more from being accepted. Avail- 96 5 n sample size able data show that the Department of Defense’s design and 0.8 Probability of Acceptance c acceptance number production specifications have led to helmets with a much 0.6 lower penetration probability. The committee considers it appropriate to replace the current (17, 240) plan, in light of 0.4 the available RTP data, with a plan that has the objective of providing an incentive for manufacturers to produce helmets 0.2 at least as penetration resistant as current helmets (θ ≤ 0.005). 0.0 The (17, 240) plan does not have that property. 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Probability of Penetration Finding 6-2. Helmet manufacturers are currently produc- ing helmets with a penetration probability near θ = 0.005, conservatively. If, as is the case for the (17, 240) plan, the FIGURE 6-6 Comparison of the operating characteristic curves for manufacturers have a low risk of failing the test even when (0, 20) and (5, 96) plans.

OCR for page 39
FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 45 plan of (0, 20). It is intuitively clear that the OC curve of 1.0 0.005 the hybrid plan should be below that of its two component 0.9 0.9 plans—(0, 22) and (17, 218)—because it is more stringent Plan 0.8 P(0/20) than either one. Figure 6-8 confirms that this is indeed the Probability of Acceptance P(0/22) 0.7 P(17/218) P(0/22;17/218) case. The plan’s government risk when θ = 0.005 is around 0.6 0.10 (i.e., there is a 90 percent chance that helmets with 0.5 penetration probability of 0.005 will be accepted). This is 0.4 comparable to the (0, 20) legacy plan and also the first-stage 0.3 (0, 22) plan. The government’s risk when θ = 0.10 is close 0.2 to zero and much lower than the other three plans being 0.1 0.1 compared. 0.0 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 Because of the first stage, the modified protocol maintains Probability of Penetration essentially the same incentive for a manufacturer to achieve 6-5 a penetration probability in the 0.001 to 0.005 neighborhood, FIGURE 6-7 Operating characteristic curves for the hybrid plan in order to have a high probability of passing the acceptance and comparison to others. test. Further, thanks to the (0, 22) first-stage threshold, the protocol is considerably more stringent in rejecting submit- ted product with underlying penetration probability in the 0.05 to 0.10 range than is the (17, 240) plan in Figure 6-5. n c 1.0 60 60 0 1 The (17, 218) criterion for Stage 2 would, by itself, give the 60 2 impression that a penetration probability as high as 17/218 = 0.8 8 percent is acceptable, which is quite different from Stage 1 n sample size Probability of Acceptance c acceptance number of the plan. Fortunately, if a product was submitted that had 0.6 an underlying 0.08 probability of penetration, that helmet is 0.4 unlikely to pass the (0, 22) first stage test.2 With this hybrid protocol, the Army has actually made 0.2 this hybrid test plan more stringent than the earlier (0, 20) plan, particularly for penetration probabilities in the range 0.0 of 0.05 to 0.12. 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Probability of Penetration Finding 6-3. The Army’s modified plan satisfies the crite- rion that it will provide an incentive for manufacturers to FIGURE 6-8 Operating characteristic curves for three plans with produce helmets that are at least as penetration resistant as n = 60. The blue lines show the probabilities of acceptance for the current helmets. three plans when the true probability of penetration is 0.05; the purple lines show the corresponding acceptance probabilities when the true penetration probability is 0.05. 6.4  EXAMINATION OF SEPARATE TEST PLANS BY HELMET SIZE The committee made a recommendation in Chapter 5 is about 0.10 (i.e., there is a 90 percent probability of accep- related to testing by separate helmet sizes (Recommendation tance). Again, this is about an order of magnitude greater 5-3). It is neither the committee’s intention nor its charge to than the penetration probability that available data indicate. recommend a specific alternative. Instead, the committee The above findings and recommendations pertaining to the discusses the properties of several plans to indicate the con- full DOT&E protocol also apply here. siderations that DOT&E should take into account in making its decision. If the current practice of 240 total shots is continued, Army’s Modification of the DOT&E Protocol there would be 60 9-mm shots for each helmet size. Figure In 2012, with DOT&E’s approval, the Army modified the 6-8 compares some possible acceptance plans. It shows that (17, 240) plan to a hybrid (two-stage) protocol (U.S. Army, at the current operating level of around θ = 0.005, the three 2012). The two stages involve conducting a (0, 22) plan in plans have acceptance probabilities of about 0.76, 0.95, and the first stage; if the lot passes this test, then a second 17-out- almost 1, respectively, for c = 0, 1, and 2. One could decide of-218 (17, 218) plan is used, for a total of 240 shots. Figure 6-7 provides a comparison of the OC curves of 2The Army’s hybrid plan essentially separates the procurement decision the hybrid plan with its component plans and also the legacy from the characterization analysis that is made possible by the complete set of 240 shots.

OCR for page 39
46 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS that the manufacturer’s risk of 1 – 0.76 = 0.24 for the c = 0 plan is too stringent. One can compare the two remaining plans at θ = 0.05, which represents a 10-fold increase in penetration probability. The c = 2 plan has a 40 percent 1.0 chance of acceptance, while the c = 1 plan has about a 19 0.9 0.8 percent chance of acceptance. One can then conclude that 0.7 a 40 percent chance of accepting helmets with penetration Phmt-acc(1/16) 0.6 probability of 0.05 is too high, in which case the c = 2 plan Prob(acc) 0.5 is not desirable. If the 19 percent is at an acceptable level, 0.4 then one can go with the 1-out-of-60 (1, 60) plan. 0.3 An alternative approach to determining a plan for each 0.2 0.1 helmet size is to specify the manufacturer’s and govern- 0.0 ment’s risks and derive both the sample size and acceptance 0 0.02 0.04 0.06 0.08 0.1 limit that would meet those criteria. Earlier in this chapter p-shot the committee derived a (1, 77) plan that had a 90 percent chance of acceptance probability at θ = 0.005 and a 10 per- FIGURE 6-9 Comparison of helmet-level and shot-level test proto- cent chance of acceptance probability at θ = 0.05. This plan cols. Blue line corresponds to a helmet-level plan; and dashed red provided an incentive for manufacturers to achieve helmets line corresponds to the (1,77) shot-level plan. with a penetration resistance that is at least as good as cur- rent helmets and protected against the acceptance of helmets that are 10 times worse than current helmets. By increasing Recommendation 6-3. The government’s risk should be the number of helmets tested in each environment to 4, the controlled at much lower penetration levels than the 0.10 number of tests for each helmet size would be 80. A 1-out- value specified by the 90/90 standard. of-80 (1, 80) plan would have an OC curve with comparable (slightly lower) acceptance probabilities as the (1, 77) plan. 6.6  FUTURE TEST PROTOCOLS: HELMET AS THE UNIT OF TEST 6.5  POST-TEST ANALYSIS The current FAT protocols are based on a shot as the It is important that the Army and DOT&E compute the unit of test: The (17, 240) plan takes 240 shots, and FAT is upper confidence bounds for the penetration probability after successful if there are 17 or fewer penetrations. However, the test is conducted. This confidence bound will provide the basic unit of production is a helmet, not a shot location additional information on the quality level of the helmets on a helmet. While it is important to test RTP at different being tested. locations, it seems desirable to make accept/reject decisions As an example, consider the (17, 240) test plan. Suppose based on a helmet as the test unit. For example, observing the test is conducted, and the result was one penetration. The five penetrations on a single helmet is quite different from estimated penetration probability of 1/240 = 0.004. The 90 a single penetration at the same location on five different percent upper confidence bound for the underlying penetra- helmets. A helmet-level test, one that scores a helmet as a tion probability based on these data is 0.016. On the other failure if there is at least one penetration, would distinguish hand, if there were 10 penetrations, and the estimated pen- between these two cases: one failure in the former case, and etration probability is 0.04, an order of magnitude higher, the five failures in the latter. upper 90 percent confidence limit would be 0.06. The upper This section studies the properties of FAT plans defined at 95 percent confidence limit is exactly equal to the designed the helmet level. This option with respect to lot acceptance value of 0.10 only if there are 17 penetrations. In other words, testing is discussed in Chapter 8. the 90/90 conclusion is pertinent only if the maximum num- Consider the rule where a helmet is scored a failure if ber of acceptable penetrations is observed during the test. there is at least one penetration among the five shots on that In these three examples, the observed number of failures helmet.3 Let the penetration probabilities for the five loca- differs substantially, so the data provide additional informa- tions be denoted by θ1, θ2, θ3, θ4, and θ5. Further, for the sake tion on the underlying penetration probability and, hence, of illustration, suppose the penetrations at different locations the quality of the helmets that will be manufactured. The only exception is with zero-failure plans where the observed number of failures is fixed up front and only a single outcome 3In practice, one might declare a helmet failure at the first penetration and (zero failures) is allowed for a successful outcome. not complete the five shots, and thus reduce the cost of testing. However, for the sake of further characterization analyses, the protocol might require that each suite of five shots might be completed. Note that this is part of the test protocol to evaluate helmet performance. There is no assumption that this test plan represents a situation in which a soldier takes five helmet hits.

OCR for page 39
FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 47 are independent events. Let θ(helmet) denote the probability The two plans have virtually identical OC curves. This is of a helmet failure. Then, not surprising. Two or more penetrations on any one helmet has a small probability for the range of θ values considered. 1 – θ(helmet) = (1– θ1) × (1– θ2) × (1– θ3) × (1– θ4) × (1– θ5) So, one failure in 16 helmets means most likely that only one penetration occurred among the 80 shots in the 16 helmet Suppose one wants a helmet-level test plan with the prop- tests. A (1, 80) plan is not much different from one of (1, 77). erties that the probability of acceptance is at least 0.90 when θ(helmet) = 0.025 and at most 0.10 when θ(helmet) = 0.25. Finding 6-4. Test plans with a helmet as the unit of test are The blue solid line in Figure 6-9 shows the OC curve for this more desirable and interpretable than those based on shots as 1-out-of-16 (1, 16) plan: test n = 16 helmets, and the FAT is the unit. When the penetration probability of a shot is small, successful if no more than one helmet fails. the helmet-level test plans and the shot-level test plans will One can compare this helmet-level plan with a plan based require about the same number of shots. on shots as the unit of test. When the θi’s are all small, θ(helmet) can be approximated as the sum of the θi’s, the Recommendation 6-4. The Department of Defense should individual shot-location probabilities. For illustrative pur- consider developing and using protocols with helmets as the poses, it is assumed that all the θi’s are the same and equal unit of test for future generations of helmets. θ. Then, if θ(helmet) = 0.025, θ approximately equals 0.005; further, if θ(helmet) = 0.25, θ approximately equals 0.05. 6.7  REFERENCES Earlier in this chapter, it was shown that a shot-level plan that satisfied these properties was a (1, 77) plan, shown in DoD IG (Department of Defense Inspector General). 2013. Advanced Combat Helmet Technical Assessment. DODIG-2013-079. Department Figure 6-3. This OC curve is superimposed in Figure 6-9 as of Defense, Washington, D.C. the dashed red line. U.S. Army. 2012. Advanced Combat Helmet (ACH) Purchase Description, Rev A with Change 4. AR/PD 10-02. Soldier Equipment, Program Executive Office—Soldier, Fort Belvoir, Va.