The test protocols for Army helmets were originally based on a requirement of zero penetrations in 20 shots (five shots on each of four helmets). The Director, Operational Test and Evaluation (DOT&E) protocol replaced this legacy plan with a requirement of 17 or fewer penetrations in 240 shots (five shots on each of 48 helmets). The helmets spanned four sizes and were tested in four different environments. The 0-out-of-20 (0, 20) plan and DOT&E’s 17-out-of-240 (17, 240) plan have comparable performance if the probability of penetrating a helmet shell on a single shot is around 0.10. As noted in Chapter 5, available data indicate that penetration probabilities are around 0.005 or less. Near this value of penetration probability, both plans have a 90 percent or higher chance of passing the test, so the manufacturer’s risk is small, as it should be. However, if there is a 10-fold increase in the penetration probability from the current level of 0.005 to 0.05, DOT&E’s (17, 240) plan still has a 95 percent chance of acceptance. This may not provide sufficient incentive for the manufacturer to sustain current penetration-probability levels. Thus, the (17, 240) plan may have the unintended effect of leading to a reduction in helmet penetration resistance. In the absence of a link between penetration probability and human injury, there is no scientific basis for setting a limit on the penetration probability. In such a circumstance, the committee’s view is that the objective of a new test plan should be to provide assurance that newly submitted helmets are at least as penetration-resistant as current helmets. This chapter proposes appropriate criteria for selecting test protocols and illustrates their use through several plans.

The primary goal of this chapter is to evaluate DOT&E’s protocol for testing a helmet’s resistance to penetration (RTP). The committee compares its performance with that of the Army’s legacy plan and a modified version of the DOT&E plan that has recently been adopted by the Army. A modification of the current protocol for the enhanced combat helmet (ECH) is also examined. These discussions are directly relevant to the issues raised in the correspondence between U.S. Representative Slaughter and the Department of Defense. To provide adequate background, the chapter begins with an overview of the statistical considerations in the design of test protocols for RTP. The chapter ends with a discussion of several topics: (1) robustness of the operating characteristic (OC) curves when the penetration probabilities vary across different test conditions; (2) examination of possible protocols for testing by helmet sizes; (3) post-test analysis of the RTP data to determine the achieved penetration probabilities of the tested helmets; and (4) a proposal to base future protocols with the helmets as the test unit rather than shots.

**6.2 STATISTICAL CONSIDERATIONS IN DESIGNING TEST PLANS FOR RESISTANCE TO PENETRATION**

As described in Chapter 4, the RTP test protocol specifies that helmets of different sizes be conditioned in selected environments and that shots be taken at different locations on the helmet. However, in this section, the committee starts with a simple setup—a single helmet size, a single shot location on the helmet, and a single environment—so that the test deals with a homogeneous population of units and a single test environment. (To be specific, one can think of a medium helmet, top location on the helmet, at ambient temperature.) It is then reasonable to view the penetration outcomes when *n* helmets are tested in this manner as being independent and identically distributed binary (pass/fail) random variables with *constant penetration probability* θ. Thus, the probability distribution of *X*, the (random) number of penetrations in *n* shots, is a binomial distribution with parameters (*n*, θ). The statistical properties of a test plan can be derived from this distribution.

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.

Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 39

6
First Article Testing Protocols for Resistance to Penetration:
Statistical Considerations and Evaluation of DoD Test Plans
6.0 SUMMARY DOT&E plan that has recently been adopted by the Army. A
modification of the current protocol for the enhanced com-
The test protocols for Army helmets were originally based
bat helmet (ECH) is also examined. These discussions are
on a requirement of zero penetrations in 20 shots (five shots
directly relevant to the issues raised in the correspondence
on each of four helmets). The Director, Operational Test
between U.S. Representative Slaughter and the Department
and Evaluation (DOT&E) protocol replaced this legacy plan
of Defense. To provide adequate background, the chapter
with a requirement of 17 or fewer penetrations in 240 shots
begins with an overview of the statistical considerations in
(five shots on each of 48 helmets). The helmets spanned four
the design of test protocols for RTP. The chapter ends with a
sizes and were tested in four different environments. The
discussion of several topics: (1) robustness of the operating
0-out-of-20 (0, 20) plan and DOT&E’s 17-out-of-240 (17,
characteristic (OC) curves when the penetration probabili-
240) plan have comparable performance if the probability of
ties vary across different test conditions; (2) examination of
penetrating a helmet shell on a single shot is around 0.10. As
possible protocols for testing by helmet sizes; (3) post-test
noted in Chapter 5, available data indicate that penetration
analysis of the RTP data to determine the achieved penetra-
probabilities are around 0.005 or less. Near this value of pen-
tion probabilities of the tested helmets; and (4) a proposal to
etration probability, both plans have a 90 percent or higher
base future protocols with the helmets as the test unit rather
chance of passing the test, so the manufacturer’s risk is small,
than shots.
as it should be. However, if there is a 10-fold increase in the
penetration probability from the current level of 0.005 to
0.05, DOT&E’s (17, 240) plan still has a 95 percent chance 6.2 STATISTICAL CONSIDERATIONS IN DESIGNING
of acceptance. This may not provide sufficient incentive for TEST PLANS FOR RESISTANCE TO PENETRATION
the manufacturer to sustain current penetration-probability
As described in Chapter 4, the RTP test protocol speci-
l
evels. Thus, the (17, 240) plan may have the unintended
fies that helmets of different sizes be conditioned in selected
effect of leading to a reduction in helmet penetration resis-
environments and that shots be taken at different locations
tance. In the absence of a link between penetration probabil-
on the helmet. However, in this section, the committee starts
ity and human injury, there is no scientific basis for setting a
with a simple setup—a single helmet size, a single shot loca-
limit on the penetration probability. In such a circumstance,
tion on the helmet, and a single environment—so that the test
the committee’s view is that the objective of a new test plan
deals with a homogeneous population of units and a single
should be to provide assurance that newly submitted helmets
test environment. (To be specific, one can think of a medium
are at least as penetration-resistant as current helmets. This
helmet, top location on the helmet, at ambient temperature.)
chapter proposes appropriate criteria for selecting test proto-
It is then reasonable to view the penetration outcomes when
cols and illustrates their use through several plans.
n helmets are tested in this manner as being independent and
identically distributed binary (pass/fail) random variables
6.1 INTRODUCTION with constant penetration probability θ. Thus, the probability
distribution of X, the (random) number of penetrations in n
The primary goal of this chapter is to evaluate DOT&E’s
shots, is a binomial distribution with parameters (n, θ). The
protocol for testing a helmet’s resistance to penetration
statistical properties of a test plan can be derived from this
(RTP). The committee compares its performance with that
distribution.
of the Army’s legacy plan and a modified version of the
39

OCR for page 39

40 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS
c-out-of-n Test Plans The y axis in Figure 6-1 shows the probability that a
(c = 1, n = 40) test will be successful as a function of the
The test plans used by DOT&E for RTP are of the fol-
underlying penetration probability θ. These acceptance
lowing form: take n shots, and if c or fewer penetrations are
probabilities are given by the cumulative distribution, P(X ≤
observed, the first article testing (FAT) passes; otherwise, it
1| θ), where X has a binomial distribution with n = 40 and
fails. In this study, the committee refers to such tests as (c, n)-
penetration probability equal to θ. For example, if θ, the
plans. They are also called binomial reliability demonstra-
underlying (unknown) penetration probability, equals 0.02
tions plans or acceptance-sampling plans for attribute data.
(green line), the probability of acceptance is 0.8 (80 percent
The plan is defined by the value of two constants: c and n.
chance of passing). If θ = 0.10 (red line), the probability
Once these are specified, the protocol’s properties are deter-
of acceptance is approximately 0.10. Conversely, in order
mined and can be studied through its operating characteristic
to have a probability of acceptance of 0.6 (black line), the
(OC) curve. An OC curve is a plot of the probability (P) of
true penetration probability needs to be about 0.38. So the
acceptance (y axis) against the underlying failure (penetra-
OC curve describes the relationship between the acceptance
tion) probability of the items under test (x axis). Figure 6-1
probabilities and the underlying penetration probability as θ
shows the OC curve for a (c = 1, n = 40) test plan; i.e., the
ranges across values of interest.
FAT is successful if there are one or fewer penetrations in
Suppose the decision maker examined the OC curve for
40 shots.
the 1-out-of-40 (1, 40) plan in Figure 6-1 and decided that
In Figure 6-1 and subsequent plots of OC curves in this
the acceptance probability of 0.10 when θ = 0.10 is too high.
report, the x axis is the true (but unknown) penetration
There are two options for reducing this value: decreasing c
probability θ. This format is different from the OC curves
or increasing n.
that are currently used by the Army and DOT&E that plot
Figure 6-2 provides a comparison with two alternatives:
the probability of nonpenetration in the x axis. One should
0-out-of-40 (0, 40) and 1-out-of-70 (1, 70) plans. For both
focus on the penetration probability, because it is easier to
(c = 0, n = 40) and (c = 1, n = 70) plans, the acceptance
interpret the curve as the penetration probability changes.
probabilities are close to zero for θ = 0.10. This may be
For example, an increase in θ from 0.005 to 0.05 is easy
acceptable to the decision maker who is the purchaser in
to interpret as a 10-fold increase in penetration probability;
this situation. But one cannot discriminate between the two
it is hard to interpret this change in terms of 1 – θ, which
plans at this value of θ.
decreases from 0.995 to 0.95.
Consider the case where the target penetration probability
is θ = 0.01. Figure 6-2 shows that, at this level, the (0, 40)
Recommendation 6-1. The operating characteristic curves
plan has an acceptance probability of about 0.63, while the
used by the Department of Defense should display penetra-
6-1 (1, 70) plan has an acceptance probability of about 0.83.
tion probabilities rather than non-penetration probabilities
Since this is the target penetration probability, the decision
on the x axes.
maker will want to accept helmets with a high probability
and will choose the (1, 70) plan or another plan that provides
an even higher acceptance probability at θ = 0.01.
6-1
1.0
0.8
Probability of Acceptance
n c
1.0
40 1
40 0
70 1
0.6 0.8
Probability of Acceptance
n sample size
c acceptance number
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18
Probability of Penetration Probability of Penetration
FIGURE 6-1 Operating characteristic curve for (c = 1, n = 40) FIGURE 6-2 Operating characteristic curves comparing 1-out-
test plan. The green and red lines show the probabilities of accep- of-40 test plan with 0-out-of-40 and 1-out-of-70 test plans. The blue
tance for the plan when the true probabilities of penetrations are, lines show the probabilities of acceptance for the two plans when
respectively, 0.02 and 0.10. The black line shows that, if we want the true probability of penetration is 0.1; the green lines show the
the probability of acceptance to be 0.6, the true penetration prob- corresponding acceptance probabilities when the true penetration
ability has to be 0.38. probability is 0.005.

OCR for page 39

FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 41
Because manufacturers want to have a high probability P( X ≤ c | n, θ = θH) ≤ β Equation 6.2
of passing the test, their helmet design and manufacturing
process should attain a penetration probability that achieves In quality control terminology, θL is the “acceptable quality
this goal. For example, to have a 90 percent chance of pass- level” for the plan, and θH is the “rejectable quality level.”
ing under the (0, 40) plan, the penetration probability will There are two kinds of errors that can occur in the (c, n)
need to be about 0.003. To pass the (1, 70) test, penetration accept-reject decision. The first error is to reject the helmet
probability will need to be about 0.008, which is not as (fail the acceptance test) when the underlying penetration
stringent a target as is set by the (0, 40) plan. These are the probability is at the low (or desired) value (i.e., θ ≤ θL); this
kinds of considerations and trade-offs that go into selecting is often referred to as producer’s or manufacturer’s risk. The
a test plan. The next subsection provides a discussion of test term manufacturer’s risk is used in this report. Equation 6.1
designs that are derived by specifying two points on a plan’s limits the probability of this error to at most α. The second
OC curve. error is to accept helmets when the penetration probability is
A few additional remarks on Figure 6-2: too high (i.e., for values of θ ≥ θH). These are usually called
consumer’s or customer’s risk. The committee refers to this
• The OC curve for the (0, 40) plan is always below that risk as government’s risk in this report. As shown by Equa-
of the (1, 40) plan. This is intuitively clear because tion 6.2, the probability of this error is at most β. These are
the (0, 40) plan is more stringent (it has the same the Type I and Type II error probabilities in the correspond-
sample size but accepts fewer failures), so the prob- ing statistical hypothesis testing formulation of the problem.
ability of passing the test is lower. Equations 6.1 and 6.2 specify the cumulative binomial
• The OC curve for the (1, 70) plan is always below that acceptance probabilities at two points. By setting the inequal-
of the (1, 40) plan. This is also obvious because the ities as equalities, one can solve them to get the values of test
(1, 70) has a larger sample size but allows the same size, n, and acceptance limit, c, that satisfy these equations.
number of failures as the (1, 40) plan. Because the binomial distribution is discrete, one typically
• More generally, consider two plans that have OC cannot achieve the equalities for α and β exactly. (There are
curves that cross, such as the (0, 40) and (1, 70) plans catalogs of test plans and software that can be readily used to
in Figure 6-2. The two plans cross at a penetration obtain the values of c and n to meet particular risks.)
probability of 0.05. To the left of that point, the (1, As a concrete example, suppose the test should be
70) plan has the higher acceptance probability. To the designed to ensure that helmets with an underlying penetra-
right, the (0, 40) plan has the higher probability of tion probability of θ = 0.005 have at least a 90 percent chance
acceptance (although the differences are quite small). of passing the test. So θL = 0.005 and (1 – α) = 0.90, or α
= 0.10. Further, suppose it was decided that if the penetra-
The different perspectives of manufacturer and purchaser
could lead them to prefer different plans. Different plans 6-3
could be considered and evaluated and a compromise plan
could be negotiated. Alternatively, as described in the next
subsection, plans can be derived from specifications of manu-
facturer’s and purchaser’s risks. 1.0
0.8
Probability of Acceptance
Statistical Approaches to Selecting (c, n)-Test Plans
0.6
The conventional statistical approach for choosing a
test plan is to specify two points on the OC curve: (1) a
0.4
low penetration-probability, θL, at which a high acceptance
probability, denoted by (1 – α), is desired (a manufacturing 0.2
process that produces good helmets has a high probability
of being accepted), and (2) a high penetration-probability, 0.0
θH, at which a low acceptance probability β is desired (a 0.00 0.02 0.04 0.06 0.08 0.10
manufacturing process that produces poor helmets has a high Probability of Penetration
probability of being rejected). Expressing these objectives
algebraically leads to the following two equations: FIGURE 6-3 Operating characteristic curves of (c = 1, n = 77)
plan with the desired risks. The black line shows the probability
P( X ≤ c | n, θ = θL ) ≥ (1 – α) Equation 6.1 of acceptance for the plan when the true probability of penetration
is 0.1; the green and red lines show the corresponding acceptance
and probabilities when the true penetration probabilities are, respec-
tively, 0.005 and 0.02.

OCR for page 39

42 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS
tion probability is θ = 0.05, which is an order of magnitude The committee performed numerical investigations to
higher, there must be at most a 10 percent chance of passing examine the differences between the true OC curves and the
the test. So, θH = 0.05, and β = 0.10. Therefore, the test is OC curves obtained by assuming that the penetration prob-
designed to discriminate between helmets with penetration abilities are the same across all shots. It examined a range of
probabilities of 0.005 and 0.05. In this example, both α and deviations for the penetration probabilities. Further, it took
β are the same, but they do not have to be. These two risks the constant penetration probability for comparison to be the
are specified by the decision maker. average of the varying probabilities. The study shows that the
Figure 6-3 shows the OC curve for the 1-out-of-77 (1, differences in the OC curves are negligible for the range of
77) test plan that meets the above requirements. It has the penetration probabilities and deviations that are relevant to
desired properties at the specified penetration probabilities the helmet situation.
of 0.005 and 0.05. In practice, however, after a plan has been
obtained, one should also examine its OC curve at other Finding 6-1. RTP data aggregated over helmet sizes,
values of θ to see if it has reasonable (not too low or not too environments, and shot locations may not have a constant
high) acceptance probabilities. In this case, if θ = 0.02 (a underlying penetration probability. An evaluation of operat-
four-fold increase from the desired penetration probability), ing characteristics for modest departures from this situation
the acceptance probability is about 0.55. One may decide indicates that the actual acceptance probabilities are negli-
that this is too high and look for a more stringent plan—say gibly different from those calculated assuming a constant
one with c = 1 but a larger value of n. That change, however, underlying penetration probability. This means that the OC
would increase the manufacturer’s risk and decrease the gov- curves computed under the assumption of constant prob-
ernment’s risk. The OC curve of an acceptance plan conveys ability provide very good approximations.
a variety of incentives and disincentives to stakeholders in
the acceptance decision. 6.3 STATISTICAL EVALUATION OF DOD PROTOCOLS
FOR RESISTANCE TO PENETRATION
Zero-Failure Plans
“Legacy” Protocol for the Advanced Combat Helmet
A common class of test protocols is based on zero-failures
(i.e., c = 0). One reason is that the lower the value of c, The legacy protocol, first specified by the program man-
the smaller the number of units to be tested, n, in order to ager for the Advanced Combat Helmet (DoD IG, 2013), was
achieve a particular level of government’s risk. However, a (0, 20) test plan. It involved testing four helmets, one each
there may be a false perception associated with zero-failure at four test environments (ambient, hot, and cold tempera-
plans: Because it does not allow any failures, the quality of tures and seawater). Only large-size helmets were tested. For
the products must be, in general, considerably higher than each helmet, the protocol required shooting a 9-mm bullet at
the government’s threshold quality. It is clear but worth five different locations, for a total of 20 shots. The five shots
reiterating that a zero-failure plan does not imply that the on each helmet were in a fixed shot sequence and pattern. No
penetration probability is zero! For example, if the penetra- penetrations were allowed (i.e., it was a zero-failure plan).
tion probability is 0.03, the probability of zero penetrations
in 20 shots is 0.54. This means that, even though there is a
3 percent chance of penetration, the 0-out-of-20 failure plan
will pass the test more than half of the time. Therefore, an
outcome of 0/20 does not imply zero penetration probability.
0.005 0.1
1.0
Robustness to Deviations from the Binomial Distribution 0.9
Probability of Acceptance
0.8
The preceding subsection was based on a framework
0.6
in which the penetration probability θ was constant across
all shots. This assumption does not strictly hold in helmet 0.4
testing: the helmets are of different sizes, they are tested at
0.2
different environmental conditions, and the shots are taken 0.12
at multiple locations on the helmet. It is possible that the 0.0
penetration probability is different at different helmet loca- 0.00 0.05 0.10 0.15 0.20
penetration probability
tions. When the penetration probabilities vary across shots,
the number of penetrations, X, in n shots would not have a
FIGURE 6-4 Operating characteristic curve for the legacy (0, 20)
binomial distribution. Therefore, the OC curves computed
test plan. The darker dashed lines show the probabilities of ac-
under this model would not apply exactly. The question of ceptance for the plan when the true penetration probabilities are
interest is whether the binomial calculations are still useful. 0.10 and 0.005.

OCR for page 39

FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 43
2007 to take over the responsibility to prescribe policy and
1.0
n c procedures for the conduct of live-fire test and evaluation of
20 0
240 17 body armor and helmets (DoD IG, 2013).
DOT&E decided to increase the number of helmets
n sample size
0.8 c acceptance number
Probability of Acceptance
tested to 48 in order to cover a range of conditions and to
0.6 have adequate precision in comparing any differences in
penetration probability, or BFD, due to environment, helmet
0.4 size, and shot location. The new protocol called for testing
48 helmets, 12 each for Small, Medium, Large, and Extra
0.2
Large sizes. Three helmets of each size were conditioned in
the four environments before testing. There were five shots
0.0
at different helmet locations, leading to a total of 240 shots.
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18
Probability of Penetration There are good statistical reasons to justify DOT&E’s
increase in the number of helmets tested to 48 helmets from
FIGURE 6-5 Comparison of the operating characteristic curves for the Army’s 5. One gets more precise estimates of the pen-
(0, 20) and (17, 240) plans. The blue lines show the probabilities etration probability from 240 shots than from 20 shots. In
of acceptance for the two plans when the true probability of pen- addition, DOT&E’s plan allows better statistical comparison
etration is 0.1; the purple and green lines show the corresponding of possible differences between helmet sizes and environ-
acceptance probabilities when the true penetration probabilities are, mental conditions.
respectively, 0.005 and 0.05. To examine the properties of the (c = 17, n = 240)-plan,
recall that if n is specified, one can control only one point
on the OC curve, or one of the two risks, by the choice of
This legacy test plan was adapted from prior helmet
c. With n chosen, the DOT&E approach was to specify that,
protocols and was not designed to meet specified statistical
for penetration probability of 0.10, the probability of accep-
risks. Nevertheless, one can study its properties through its
tance (the government’s risk) should be no more than 10
OC curve in Figure 6-4. The acceptance probability is about
percent. This is referred to as the 90/90 plan (corresponding
0.12 when the penetration probability is 0.10. In other words,
to a rejection probability of at least 0.90 at nonpenetration
if the underlying shot penetration probability is 0.10, the hel-
probability of 0.90). To summarize, DOT&E’s (17, 240) plan
mets will fail the demonstration test 88 percent of the time.
was chosen by first increasing the sample size n to be 240
Consider the behavior of the curve to the left of θ = 0.10
for statistical reasons. Then, the 90/90 standard was applied
and the implications for manufacturers. If a manufacturer
to get the maximum number of acceptable failures to be 17.
wants to have a 90 percent chance or higher of passing the
Thus, there is a direct relationship between the 90/90 stan-
(0, 20) test, the helmet design and production process would
dard and the (17, 240) plan.
have to achieve a penetration probability of θ = 0.005 or less.
However, there is no scientific or empirical basis for
Note that the manufacturer has to achieve a penetration
specifying 0.10 as the acceptable limit for a helmet’s pen-
probability considerably less than the government’s standard
etration probability. It appears that the 90/90 standard was
of θ = 0.10 to have a good chance of passing the (0, 20) test.
chosen because of its use in body armor protocols1 and also
While the government, by its specification of θ = 0.10 as its
because the legacy protocol approximately had this property.
limit on penetration probability, may be willing to purchase
That specification led to the (c = 17, n = 240) test plan. The
helmets with, say, θ = 0.075, the manufacturer would not aim
committee does not know if there was any attempt to control
at that target because the chance of passing the (0, 20) test is
the manufacturer’s risk.
too low for comfort—about 0.20 in Figure 6-5.
Figure 6-5 provides a comparison of the OC curves for
As noted earlier, the government’s risk at θ = 0.10 was
the (0, 20) and (17, 240) plans. The two OC curves cross
0.12. So, this plan does not strictly satisfy the 90/90 property
at about θ = 0.092. The (0, 20) plan has higher acceptance
(at most 10 percent government’s risk at penetration prob-
probabilities to the right of this penetration probability and
ability 0.10 or, equivalently, at least 90 percent chance of
has lower acceptance probabilities to the left. The two plans
failing the test if the nonpenetration probability is 0.90.) One
have about the same acceptance probabilities (government
needs a 0-out-of-22 (0, 22) plan to satisfy this requirement.
risks), in the neighborhood of θ = 0.10, as intended.
The 90/90 criterion was explicitly adopted by DOT&E in its
When θ = 0.005, near the region where the manufactur-
subsequent protocols.
ers are currently operating (see Chapter 5), the acceptance
probability of the (0, 20) plan is about 0.9, while that of the
DOT&E’s (c = 17, n = 240) Protocol (17, 240) plan is essentially 1.0. Thus, the (17, 240) plan has
In response to a Senate and House Armed Services Com-
mittee’s request, the Secretary of Defense asked DOT&E in 1Personal communication between Christopher Moosmann, DOT&E, and
Nancy Schulte, NRC, via e-mail on May 14, 2013.

OCR for page 39

44 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS
lower manufacturer’s risk. Director Gilmore’s letter to Rep. there is a 10-fold increase in the current penetration prob-
Slaughter (see Appendix A) recognized that the DOT&E ability (from 0.005 to 0.05), this may provide a disincentive
protocol would lessen the burden on manufacturers to pass to maintain current levels of penetration resistance. In this
the test with helmets with an underlying penetration prob- sense, the (17, 240) plan is not as good as the legacy plan
ability less than the “standard” of 0.10. However, this is not of (0, 20).
necessarily an advantage.
Consider a comparison of the two plans when the penetra- It is likely that manufacturers are more motivated by
tion probability equals 0.05, which is a 10-fold increase in having a high probability of passing the test than they are
the penetration probability from the currently achieved level in avoiding a penetration probability at the current DOT&E
of around 0.005. For this value of θ = 0.05, the acceptance “standard” of 0.10, a value nearly two orders of magnitude
probability is about 0.38 for the (0, 20) plan, while it is about higher than what current data indicate for a helmet penetra-
0.95 for the (17, 240) plan. Thus, even if there is a 10-fold tion probability. If manufacturers have a very high probabil-
degradation in the penetration resistance of helmets, there ity of passing the test, even if there is a substantial increase
is a 95 percent chance of accepting the helmets under the in the penetration probability, the (17, 240) plan may have
DOT&E protocol. Similar comparisons can be made at other the unintended effect of leading to a reduction in helmet
values of θ to the left of the point where the two curves cross. penetration resistance.
For example, for any values of penetration probability of θ ≤
0.04—a five-fold increase—the helmets will almost certainly Recommendation 6-2. If there is a scientific basis to link
be accepted. To the right of the crossing point, however, the brain injury with performance metrics (such as penetra-
(0, 20) plan has a higher acceptance probability (and hence tion frequency and backface deformation), the Director of
poorer performance in terms of screening out helmets with Operational Test and Evaluation (DOT&E) should use this
high penetration probabilities, but still less than a 12 percent information to set the appropriate standard for performance
chance of acceptance). metrics in the test protocols. In the absence of such a sci-
A decision on which of the two plans is better comes down entific basis, DOT&E should develop a plan that provides
to deciding what is the relevant range of values of the pen- assurance that it leads to the production of helmets that are
etration probability. DOT&E’s (17, 240) plan focuses around at least as penetration-resistant as currently fielded helmets.
θ = 0.10, and its main objective is to prevent helmets with
a 0.10 penetration probability or more from being accepted.
Enhanced Combat Helmet Protocol: Modified DOT&E
The (17, 240) plan has comparable performance to the (0,
Protocol
20) plan at this point and has lower acceptance probabilities
for θ ≥ 0.10. So if this is the region of interest, then the (17, The ECH protocol, a modification of the DOT&E pro-
240) plan is superior to the (0, 20) plan. However, if the tocol, is a 5-out-of-96 (5, 96) plan that involves taking two
objective of the plan is to provide an incentive for manufac- shots each at 48 helmets. The acceptance limit of c = 5 is
turers to produce helmets at least as good as current helmets based on the 90/90 criterion. Figure 6-6 provides a compari-
(θ ≤ 0.005), the (0, 20) plan is better in that it has a lower son of its OC curve with that of the (0, 20) plan. It shows that,
probability of acceptance for helmets that are not as good if the penetration probability is 0.035, the manufacturer’s risk
as current helmets up to a penetration probability of 0.10.
To evaluate a plan, one needs to consider the whole OC
curve, not just one point that may have been used to specify
the plan. The DOT&E plan focuses on the point at which θ
= 0.10. Its main objective is to prevent helmets with a 0.10 1.0
n c
20 0
penetration probability or more from being accepted. Avail- 96 5
n sample size
able data show that the Department of Defense’s design and 0.8
Probability of Acceptance
c acceptance number
production specifications have led to helmets with a much
0.6
lower penetration probability. The committee considers it
appropriate to replace the current (17, 240) plan, in light of 0.4
the available RTP data, with a plan that has the objective of
providing an incentive for manufacturers to produce helmets 0.2
at least as penetration resistant as current helmets (θ ≤ 0.005).
0.0
The (17, 240) plan does not have that property.
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18
Probability of Penetration
Finding 6-2. Helmet manufacturers are currently produc-
ing helmets with a penetration probability near θ = 0.005,
conservatively. If, as is the case for the (17, 240) plan, the
FIGURE 6-6 Comparison of the operating characteristic curves for
manufacturers have a low risk of failing the test even when (0, 20) and (5, 96) plans.

OCR for page 39

FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 45
plan of (0, 20). It is intuitively clear that the OC curve of
1.0
0.005
the hybrid plan should be below that of its two component
0.9 0.9 plans—(0, 22) and (17, 218)—because it is more stringent
Plan
0.8 P(0/20) than either one. Figure 6-8 confirms that this is indeed the
Probability of Acceptance
P(0/22)
0.7 P(17/218)
P(0/22;17/218)
case. The plan’s government risk when θ = 0.005 is around
0.6 0.10 (i.e., there is a 90 percent chance that helmets with
0.5
penetration probability of 0.005 will be accepted). This is
0.4
comparable to the (0, 20) legacy plan and also the first-stage
0.3
(0, 22) plan. The government’s risk when θ = 0.10 is close
0.2
to zero and much lower than the other three plans being
0.1 0.1
compared.
0.0
0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 Because of the first stage, the modified protocol maintains
Probability of Penetration
essentially the same incentive for a manufacturer to achieve
6-5 a penetration probability in the 0.001 to 0.005 neighborhood,
FIGURE 6-7 Operating characteristic curves for the hybrid plan in order to have a high probability of passing the acceptance
and comparison to others. test. Further, thanks to the (0, 22) first-stage threshold, the
protocol is considerably more stringent in rejecting submit-
ted product with underlying penetration probability in the
0.05 to 0.10 range than is the (17, 240) plan in Figure 6-5.
n c
1.0 60
60
0
1
The (17, 218) criterion for Stage 2 would, by itself, give the
60 2 impression that a penetration probability as high as 17/218 =
0.8
8 percent is acceptable, which is quite different from Stage 1
n sample size
Probability of Acceptance
c acceptance number
of the plan. Fortunately, if a product was submitted that had
0.6
an underlying 0.08 probability of penetration, that helmet is
0.4
unlikely to pass the (0, 22) first stage test.2
With this hybrid protocol, the Army has actually made
0.2 this hybrid test plan more stringent than the earlier (0, 20)
plan, particularly for penetration probabilities in the range
0.0 of 0.05 to 0.12.
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18
Probability of Penetration
Finding 6-3. The Army’s modified plan satisfies the crite-
rion that it will provide an incentive for manufacturers to
FIGURE 6-8 Operating characteristic curves for three plans with produce helmets that are at least as penetration resistant as
n = 60. The blue lines show the probabilities of acceptance for the current helmets.
three plans when the true probability of penetration is 0.05; the
purple lines show the corresponding acceptance probabilities when
the true penetration probability is 0.05. 6.4 EXAMINATION OF SEPARATE TEST PLANS BY
HELMET SIZE
The committee made a recommendation in Chapter 5
is about 0.10 (i.e., there is a 90 percent probability of accep- related to testing by separate helmet sizes (Recommendation
tance). Again, this is about an order of magnitude greater 5-3). It is neither the committee’s intention nor its charge to
than the penetration probability that available data indicate. recommend a specific alternative. Instead, the committee
The above findings and recommendations pertaining to the discusses the properties of several plans to indicate the con-
full DOT&E protocol also apply here. siderations that DOT&E should take into account in making
its decision.
If the current practice of 240 total shots is continued,
Army’s Modification of the DOT&E Protocol there would be 60 9-mm shots for each helmet size. Figure
In 2012, with DOT&E’s approval, the Army modified the 6-8 compares some possible acceptance plans. It shows that
(17, 240) plan to a hybrid (two-stage) protocol (U.S. Army, at the current operating level of around θ = 0.005, the three
2012). The two stages involve conducting a (0, 22) plan in plans have acceptance probabilities of about 0.76, 0.95, and
the first stage; if the lot passes this test, then a second 17-out- almost 1, respectively, for c = 0, 1, and 2. One could decide
of-218 (17, 218) plan is used, for a total of 240 shots.
Figure 6-7 provides a comparison of the OC curves of 2The Army’s hybrid plan essentially separates the procurement decision
the hybrid plan with its component plans and also the legacy from the characterization analysis that is made possible by the complete
set of 240 shots.

OCR for page 39

46 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS
that the manufacturer’s risk of 1 – 0.76 = 0.24 for the c = 0
plan is too stringent. One can compare the two remaining
plans at θ = 0.05, which represents a 10-fold increase in
penetration probability. The c = 2 plan has a 40 percent 1.0
chance of acceptance, while the c = 1 plan has about a 19 0.9
0.8
percent chance of acceptance. One can then conclude that 0.7
a 40 percent chance of accepting helmets with penetration Phmt-acc(1/16)
0.6
probability of 0.05 is too high, in which case the c = 2 plan Prob(acc) 0.5
is not desirable. If the 19 percent is at an acceptable level, 0.4
then one can go with the 1-out-of-60 (1, 60) plan. 0.3
An alternative approach to determining a plan for each 0.2
0.1
helmet size is to specify the manufacturer’s and govern-
0.0
ment’s risks and derive both the sample size and acceptance 0 0.02 0.04 0.06 0.08 0.1
limit that would meet those criteria. Earlier in this chapter p-shot
the committee derived a (1, 77) plan that had a 90 percent
chance of acceptance probability at θ = 0.005 and a 10 per- FIGURE 6-9 Comparison of helmet-level and shot-level test proto-
cent chance of acceptance probability at θ = 0.05. This plan cols. Blue line corresponds to a helmet-level plan; and dashed red
provided an incentive for manufacturers to achieve helmets line corresponds to the (1,77) shot-level plan.
with a penetration resistance that is at least as good as cur-
rent helmets and protected against the acceptance of helmets
that are 10 times worse than current helmets. By increasing
Recommendation 6-3. The government’s risk should be
the number of helmets tested in each environment to 4, the
controlled at much lower penetration levels than the 0.10
number of tests for each helmet size would be 80. A 1-out-
value specified by the 90/90 standard.
of-80 (1, 80) plan would have an OC curve with comparable
(slightly lower) acceptance probabilities as the (1, 77) plan.
6.6 FUTURE TEST PROTOCOLS: HELMET AS THE
UNIT OF TEST
6.5 POST-TEST ANALYSIS
The current FAT protocols are based on a shot as the
It is important that the Army and DOT&E compute the
unit of test: The (17, 240) plan takes 240 shots, and FAT is
upper confidence bounds for the penetration probability after
successful if there are 17 or fewer penetrations. However,
the test is conducted. This confidence bound will provide
the basic unit of production is a helmet, not a shot location
additional information on the quality level of the helmets
on a helmet. While it is important to test RTP at different
being tested.
locations, it seems desirable to make accept/reject decisions
As an example, consider the (17, 240) test plan. Suppose
based on a helmet as the test unit. For example, observing
the test is conducted, and the result was one penetration. The
five penetrations on a single helmet is quite different from
estimated penetration probability of 1/240 = 0.004. The 90
a single penetration at the same location on five different
percent upper confidence bound for the underlying penetra-
helmets. A helmet-level test, one that scores a helmet as a
tion probability based on these data is 0.016. On the other
failure if there is at least one penetration, would distinguish
hand, if there were 10 penetrations, and the estimated pen-
between these two cases: one failure in the former case, and
etration probability is 0.04, an order of magnitude higher, the
five failures in the latter.
upper 90 percent confidence limit would be 0.06. The upper
This section studies the properties of FAT plans defined at
95 percent confidence limit is exactly equal to the designed
the helmet level. This option with respect to lot acceptance
value of 0.10 only if there are 17 penetrations. In other words,
testing is discussed in Chapter 8.
the 90/90 conclusion is pertinent only if the maximum num-
Consider the rule where a helmet is scored a failure if
ber of acceptable penetrations is observed during the test.
there is at least one penetration among the five shots on that
In these three examples, the observed number of failures
helmet.3 Let the penetration probabilities for the five loca-
differs substantially, so the data provide additional informa-
tions be denoted by θ1, θ2, θ3, θ4, and θ5. Further, for the sake
tion on the underlying penetration probability and, hence,
of illustration, suppose the penetrations at different locations
the quality of the helmets that will be manufactured. The
only exception is with zero-failure plans where the observed
number of failures is fixed up front and only a single outcome 3In practice, one might declare a helmet failure at the first penetration and
(zero failures) is allowed for a successful outcome. not complete the five shots, and thus reduce the cost of testing. However,
for the sake of further characterization analyses, the protocol might require
that each suite of five shots might be completed. Note that this is part of the
test protocol to evaluate helmet performance. There is no assumption that
this test plan represents a situation in which a soldier takes five helmet hits.

OCR for page 39

FIRST ARTICLE TESTING PROTOCOLS FOR RESISTANCE TO PENETRATION 47
are independent events. Let θ(helmet) denote the probability The two plans have virtually identical OC curves. This is
of a helmet failure. Then, not surprising. Two or more penetrations on any one helmet
has a small probability for the range of θ values considered.
1 – θ(helmet) = (1– θ1) × (1– θ2) × (1– θ3) × (1– θ4) × (1– θ5) So, one failure in 16 helmets means most likely that only one
penetration occurred among the 80 shots in the 16 helmet
Suppose one wants a helmet-level test plan with the prop- tests. A (1, 80) plan is not much different from one of (1, 77).
erties that the probability of acceptance is at least 0.90 when
θ(helmet) = 0.025 and at most 0.10 when θ(helmet) = 0.25. Finding 6-4. Test plans with a helmet as the unit of test are
The blue solid line in Figure 6-9 shows the OC curve for this more desirable and interpretable than those based on shots as
1-out-of-16 (1, 16) plan: test n = 16 helmets, and the FAT is the unit. When the penetration probability of a shot is small,
successful if no more than one helmet fails. the helmet-level test plans and the shot-level test plans will
One can compare this helmet-level plan with a plan based require about the same number of shots.
on shots as the unit of test. When the θi’s are all small,
θ(helmet) can be approximated as the sum of the θi’s, the Recommendation 6-4. The Department of Defense should
individual shot-location probabilities. For illustrative pur- consider developing and using protocols with helmets as the
poses, it is assumed that all the θi’s are the same and equal unit of test for future generations of helmets.
θ. Then, if θ(helmet) = 0.025, θ approximately equals 0.005;
further, if θ(helmet) = 0.25, θ approximately equals 0.05.
6.7 REFERENCES
Earlier in this chapter, it was shown that a shot-level plan
that satisfied these properties was a (1, 77) plan, shown in DoD IG (Department of Defense Inspector General). 2013. Advanced
Combat Helmet Technical Assessment. DODIG-2013-079. Department
Figure 6-3. This OC curve is superimposed in Figure 6-9 as
of Defense, Washington, D.C.
the dashed red line. U.S. Army. 2012. Advanced Combat Helmet (ACH) Purchase Description,
Rev A with Change 4. AR/PD 10-02. Soldier Equipment, Program
Executive Office—Soldier, Fort Belvoir, Va.