National Academies Press: OpenBook
« Previous: 7 Test Protocols for Backface Deformation: Statistical Considerations and Assessment
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

8


Lot Acceptance Testing

8.0 SUMMARY

Lot acceptance testing (LAT) is used to ensure that manufacturers continue to produce helmets that conform to contract specifications. A random sample of helmets is selected from the production lot, and the helmet shells as well as hardware are tested according to the LAT protocol. The number of helmets in the protocols is determined from an American National Standards Institute (ANSI) standard, and they vary by lot size. This chapter examines the operating characteristic (OC) curves for the Director, Operational Test and Evaluation’s (DOT&E’s) LAT plans and compares them with first article testing (FAT) protocols in the Army’s legacy plans and DOT&E’s plans. The OC curves for the LAT plans for the different lot sizes can vary a lot, indicating that the manufacturer’s and government’s risks can be quite different across lot sizes. This is primarily due to the different sample sizes (number of helmets and number of shots) as determined from ANSI standard. Further, DOT&E’s FAT protocols are considerably less stringent (higher probabilities of acceptance for the OC curves) than their corresponding LAT protocols. This is counter to the philosophy that it should be more difficult for manufacturers to pass FAT than LAT. This issue can be addressed if DOT&E makes changes to the (17, 240) FAT protocol as discussed in Chapters 6 and 7. This chapter also proposes using binary data for backface deformation (BFD) LAT protocols, to make them consistent with the recommendations for FAT. Finally, the committee examines the properties of LAT protocols based on helmets as the unit of testing.

8.1 INTRODUCTION

After a helmet manufacturer has passed FAT and begins production, LAT is used to ensure that the helmets continue to meet contract specifications. This chapter describes the DOT&E’s LAT protocol, which is based on the ANSI standard ASQ Z1.4-20081 for selecting lot sample sizes and acceptance limits (ASQ, 2008). The performance of the DOT&E’s LAT protocol is compared to the Army’s original FAT protocol and DOT&E’s FAT protocol, both in terms of resistance to penetration (RTP) and BFD. This chapter also examines the feasibility of helmet-based LAT protocols.

8.2 LOT ACCEPTANCE TESTING PROTOCOLS

The Army’s Original Lot Acceptance Testing Protocol

Table 8-1 shows the Army’s original LAT protocol for RTP (DoD IG, 2013, p. 6). Note that the number of helmets, and thus the resulting number of shots, is small.

TABLE 8-1 Sample Sizes for the Army’s Historical Lot Acceptance Testing Protocol for a 9-mm RTP Shell

Lot Size

Sample Size

Accept

Reject

4-150

5 shots, 1 helmet

0

1

151-1,200

5 shots, 1 helmet

0

1

1,201-3,200

10 shots, 2 helmets

0

1

SOURCE: DoD IG (2013).

DOT&E’s Lot Acceptance Testing Protocol

For DOT&E’s LAT, the sample sizes (numbers of helmets to be tested) are derived from the ANSI standard ASQ Z1.4-2008 (ASQ, 2008). Table 8-2 is the helmet LAT matrix from Appendix A of the DOT&E LAT protocol.2 It provides the requirements in terms of the number of helmets to be tested,

 

_________________

1The committee notes that the DOT&E protocol does not mention or explicitly reference the ANSI standard. The Army purchase description does specify the ANSI standard (U.S. Army, 2012).

2The current DOT&E LAT and FAT protocols are found in Appendix B of this report.

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

TABLE 8-2 Helmet Lot Acceptance Testing Matrix

Lot Size

Sub-Test

Shots

Helmets

RTP Accept

RTP Reject

91-150

9-mm Hardware RTP

  6

  3

  0

  1

9-mm Shell RTP/BTD

25

  5

  0

  1

151-500

9-mm Hardware RTP

10

  5

  0

  1

9-mm Shell RTP/BTD

40

  8

  1

  2

501-1,200

9-mm Hardware RTP

10

  5

  0

  1

9-mm Shell RTP/BTD

65

13

  1

  2

1,201-3,200

9-mm Hardware RTP

16

  8

  1

  2

9-mm Shell RTP/BTD

65

13

  1

  2

NOTE: BTD, ballistic transient deformation (synonymous with the term BFD used in this report); RTP, resistance to penetration.
SOURCE: DOT&E (2012).

 

the total number of shots, and the accept/reject criteria by lot size. The test plan in Table 8-2 involves a finer division of lot sizes and a larger number of helmets and shots than the Army’s legacy protocol (Table 8-1).

The other aspects of DOT&E’s LAT are similar to its FAT protocol, including range setup, the use of clay as a backing material and its calibration, the definitions of complete and partial penetrations, and the metrics (RTP and BFD). However, unlike FAT, all tests are conducted only under ambient conditions.

Note that the sample sizes for LAT are smaller than FAT sample sizes. Further, the protocol varies substantially by lot sizes: from a sample size of 5 helmet shells (and a total of 25 shots) for the smallest lot to a sample size of 13 helmet shells (and a total of 65 shots) for the largest lot. Similarly, for hardware testing, the sample sizes vary from 3 helmets (and 6 shots) to 8 helmets (and 16 shots).

As with FAT, the DOT&E LAT protocol specifies a helmet test matrix that defines the shot order for each helmet in the test sequence (Table 8-3).

The DOT&E LAT protocol makes no mention of helmet size. If lots consist of only one helmet size, then it is clear how to implement the protocol in Table 8-3. However, for situations where there are helmets of multiple sizes in a lot, Table 8-2 does not specify the order in which the different-sized helmets should be tested.

Finding 8-1. The DOT&E LAT protocol does not specify helmet size, while the FAT protocol specifies testing of four different helmet sizes.

The 1996 report DoD Preferred Methods for Acceptance of Product, MIL-STD-1916, states:

The product shall be assembled into identifiable lots, sublots, or batches, or in such other manner as may be prescribed. Each lot or batch shall, as far as practicable, consist of units of product of a single type, grade, class, size [emphasis added], and composition, manufactured under essentially the same conditions, and at essentially the same time. (DoD, 1996, p. 9).

Recommendation 8-1. The protocol established by the Director, Operational Test and Evaluation, should be revised

TABLE 8-3 Helmet Shot Order Test Matrix for Aramid 9-mm

Helmet

Order

LAT Helmet #1

B

L

Cr

F

R

LAT Helmet #2

Cr

R

B

L

F

LAT Helmet #3

R

B

Cr

L

F

LAT Helmet #4

B

F

L

R

Cr

LAT Helmet #5

B

R

F

L

Cr

LAT Helmet #6

Cr

B

L

F

R

LAT Helmet #7

L

B

Cr

F

R

LAT Helmet #8

Cr

B

R

F

L

LAT Helmet #9

L

F

R

B

Cr

LAT Helmet #10

F

Cr

B

L

R

LAT Helmet #11

Cr

L

R

B

F

LAT Helmet #12

R

F

B

L

Cr

LAT Helmet #13

Cr

F

L

B

R

NOTE: B, back; CR, crown; F, front; L, left; R, right; LAT, lot acceptance testing.
SOURCE: DOT&E, 2012.

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

to explicitly state that: (1) it will be applied separately to each helmet size; and (2) if the lot contains helmets of multiple sizes, the test requirements will be applied according to the number of helmets of each size in the lot.

The Army’s Hybrid Protocols

As with FAT, the Army has recently introduced modified LAT protocols. For penetration, it is a hybrid of the Army’s historical LAT protocol and DOT&E’s LAT protocol (DOT&E, 2012).

•   In Stage 1, either 5 or 10 shots are taken, depending on the lot size (as specified in Table 8-3). If there is any complete penetration, the test terminates in a failure. If there are no complete penetrations, the test continues to Stage 2.

•   In Stage 2, passing the LAT RTP requirement is based on the accept/reject criterion specified in the DOT&E protocol (Table 8-2). As described in the DOT&E protocol, if a penetration is observed, then a new helmet is substituted and tested, and the data from both helmets are counted toward the final accept/reject determination.

Hardware testing is conducted strictly in accordance with the DOT&E protocol (DOT&E, 2012).

For BFD, the Army’s LAT hybrid protocol is based on the same hybrid test for penetration (DOT&E, 2012). If the test continues as a result of successful completion of the first stage RTP test described above, then passing the LAT BFD requirement is based on all of the data collected and the accept/reject criterion specified for the lot size. As before, if a penetration is observed during the test, a new helmet is substituted and tested, and the BFD data from both helmets (excluding the shot that resulted in a penetration) are used in the BFD calculations. Thus, the Army’s lightweight advanced combat helmet protocol is virtually the same as the DOT&E protocol. The only difference is that the lightweight protocol does not specify a two-stage procedure for lot sizes of 91 to 150 helmets; instead, it simply requires a 75 percent upper tolerance limit (UTL) at 90 percent confidence (DOT&E, 2012).

The committee does not study the properties of these hybrid protocols in this chapter because their properties are complex. Moreover, as noted in Chapters 6 and 7, the committee proposes that the DOT&E protocols be modified rather than addressing the issues through modified two-stage protocols.

8.3 EVALUATING PERFORMANCE: COMPARISON OF OPERATING CHARACTERISTIC CURVES

Resistance to Penetration

This section compares the OC curves of DOT&E’s LAT protocol with DOT&E’s FAT protocol and the Army’s original FAT protocol. In comparing LAT and FAT, it is important to keep in mind that the manufacturer has already demonstrated the ability to meet specification requirements via FAT. The goal of LAT is to assess whether the manufacturer’s helmets continue to conform, and thus the government is expected to assume greater risk at this stage.

Figure 8-1 shows the OC curves for the DOT&E LAT protocols for the three different lot sizes: 91 to 150 (black), 151 to 500 (red), and 501 to 3,200 (green). The interpretation of an OC curve here is the same as that in Chapter 6: It is a plot of the probability of acceptance (passing LAT in this case) on the y axis versus the true penetration probability on the x axis. In Figure 8-1, the OC curves for the different lot sizes vary considerably and hence can have quite different manufacturer’s and government’s risks. For example, the blue line corresponds to a penetration probability of 0.005 (current levels where manufacturers are operating), and the probabilities of acceptance for the three curves range from about 0.88 to about 0.99. Thus, the manufacturer’s risks (which equal 1 – probability of acceptance) range from 0.01 to 0.12. Consider now the case where the probability of penetration is around 0.05—which is an order of magnitude higher. The purple lines indicate that the probabilities of acceptance, or government’s risk, vary from about 0.18 to 0.4.

It is difficult to match the OC curves very closely if one wishes to vary the sample sizes for different lot sizes and, in particular, fix the sample sizes using the ANSI standard.

image

FIGURE 8-1 Operating characteristic curves for resistance to penetration for the three Director, Operational Test and Evaluation, protocols by lot sizes: 91 to 150 (black), 151 to 500 (red), and 501 to 3,200 (green).

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

image

FIGURE 8-2 Comparison of operating characteristic curves for the three Director, Operational Test and Evaluation (DOT&E) lot acceptance testing protocols (black, red, and green) with the Army’s Legacy first article testing (FAT) protocol (blue) and DOT&E’s FAT protocol (orange).

Figure 8-2 provides a comparison of the DOT&E LAT protocols (black, red, and green OC curves) with the Army’s legacy FAT protocol (blue) and DOT&E’s FAT protocol (orange). The OC curve for the Army’s legacy FAT protocol is within the range of the curves for DOT&E’s LAT protocols. However, DOT&E’s FAT protocol (17-out-of-240 penetrations) has a much higher probability of acceptance than the LAT protocols in the left end of Figure 8-2. This region corresponds to penetration probabilities of 0.08 or less, covering the current region where manufacturers operate as well as penetration levels more than an order of magnitude higher. So, the manufacturer’s risk for the LAT protocols is higher than that for the DOT&E FAT protocol. This is counter to the philosophy that LAT should be easier for manufacturers to pass than FAT.

Finding 8-2. Some of the DOT&E LATs for penetration are more difficult for manufacturer’s to pass than the FAT plans. This is contrary to the philosophy that LAT is intended to assess whether the manufacturers helmets continue to conform to specifications, and so it should be less stringent than FAT.

As discussed in Chapter 6, the problem illustrated in Figure 8-2 is with DOT&E’s (17, 240) FAT protocol. For illustrative purposes, consider the situation in which the DOT&E FAT is changed to a 1-out-of-60 (1, 60) plan. Figure 8-3 shows a comparison of the OC curve of this plan with those of the current LAT OC curves. The blue curve corresponds to the (1, 60) FAT plan and, as to be expected, it is very close to the 1-out-of-65 (1, 65) LAT plan that corresponds to the largest lot size. If one wanted to insist that LAT plans be less stringent than the corresponding FAT plans, one could restrict the number of shots for the LAT plans to be 60 or fewer, rather than its current value of 65.

image

FIGURE 8-3 Comparison of operating characteristic curves for the three DOT&E lot acceptance testing protocols (black, red, and green) with an illustrative (1, 60) first article testing protocol (red).

The committee emphasizes that these are just illustrative discussions and that the committee is not endorsing a particular FAT plan for RTP.

Backface Deformation

The Army’s historical LAT BFD protocol was also based on the sample sizes in Table 8-1. For each of the shots, the BFD was measured and compared to a threshold: 25.4 mm for front and back shots and 16 mm for side and crown shots. If any of the BFDs exceeded its associated standard, then the lot failed. In other words, the BFD LAT protocol, like the BFD FAT protocol, was based on binary outcomes—whether the BFD measurement exceeded the threshold or not.

DOT&E’s LAT protocol, like its FAT protocol, assesses helmet BFD performance using statistical tolerance limits (discussed in Chapter 7). The LAT procedures continue to fix the confidence levels at 90 percent. However, unlike FAT where the UTL was also fixed at 90 percent, the UTLs for LAT vary with lot size (and hence with sample size): 80 percent UTL for lot sizes of 501 to 3,200 helmets, 75 percent UTL for lot sizes of 151 to 500 helmets, and a more complicated two-stage procedure for lot sizes of 91 to 150 helmets.

The DOT&E LAT protocol states that the “UTL (at 90 percent confidence) will be calculated by combining the right and left shot locations if the data from the qualifying First Article Test indicates the data from the side locations can be combined for analysis.”3 This procedure is different from the DOT&E FAT protocol in which back and front

 

_________________

3DOT&E, 2012, pp. 5-6; reprinted in Appendix B

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

image

FIGURE 8-4 Backface deformation (BFD) operating characteristic curves for the Director, Operational Test and Evaluation (DOT&E) first article testing (FAT) protocol in blue, the original Army FAT protocol in black, and the DOT&E lot acceptance testing (LAT) protocols in red. NOTE: N is the lot size; UTL, upper tolerance limit.

 

are grouped into one category and left, right, and crown are grouped into another.

Figure 8-4 compares the performance of the various DOT&E LAT protocols (one for each lot size) against the Army’s original (0, 20) FAT protocol and DOT&E’s FAT protocol for BFD. These results are based on a simulation study conducted under the following scenario:

•   The BFD measurements are normally distributed.

•   The sample size is held constant in accordance with the lot size requirements of Table 8-1 (which occurs if there are no penetrations).

•   The standard deviations are fixed as follows: 2.02-mm for the front and back locations and 1.58-mm for the side and crown locations. (These values were derived from actual BFD data).

•   The means are varied. The x axis of Figure 8-4 shows the true mean in terms of standardized distance from the respective UTL thresholds. The standardized distance (true mean minus BFD*) is divided by the standard deviation. BFD* is the UTL threshold: 25.4-mm for front and back shots and 16-mm for side and crown shots. For example, if the true mean for the front location is set at 23.38-mm, the standardized distance on the x axis in Figure 8-4 will be (23.38-mm – 25.4-mm)/2.02-mm = –1.

Figure 8-4 shows the OC curves for the original Army FAT protocol (in black), the DOT&E FAT protocol (blue), and the three DOT&E LAT protocols (solid and dashed reds). As was the case with penetration, the curves for the three BFD LAT protocols vary considerably, indicating that they can have quite different manufacturer’s and government’s risks. In particular, the OC curves for the large two lot sizes (dashed reds) have much higher probabilities of acceptance (OC curves to the right) than that of the small lot size. Thus, it is easier to pass the LATs for the larger lot sizes.

Turning to a comparison with the FAT protocols (black and blue curves), one sees that the Army’s legacy FAT protocol has a very similar performance to that of the LAT curve for the small lot size of 91 to 150. On the other hand, the OC curve for the DOT&E FAT protocol (blue curve) is much further to the right than the other curves, indicating that the FAT protocol for BFD is much easier to pass than the LAT protocols. This conclusion is similar to the one that can be made from Figure 8-2 for penetration.

Finding 8-3. The OC curves of the DOT&E LATs for BFD vary considerably, indicating that the protocols for the different lot sizes can have quite different manufacturer’s and government’s risks. The protocol for the small lot size is more stringent than the ones for the medium and large lot sizes.

Finding 8-4. DOT&E’s LAT protocols for BFD are more difficult for manufacturers to pass than its FAT. This mirrors a similar finding for penetration. This result is contrary to the philosophy that LAT should be less stringent than FAT.

Backface Deformation Lot Acceptance Testing Protocols Based on Binary Data

As noted in Chapter 7, there are many difficulties with the use of tolerance limits for the BFD protocols. If DOT&E were to implement Recommendation 7-1 to revert to the use of binary data for BFD for FAT protocols, a similar change should necessarily be made to LAT protocols. This would simplify many of the additional complexities associated with LAT protocols and combine them across shot locations. It would also have the added advantage of using the same LAT protocols for penetration and BFD and make the BFD protocols easier to understand and more transparent to nonstatisticians.

8.4 ANSI STANDARD AND THE ACCEPTANCE QUALITY LIMIT

Comparison to the ANSI Standard

DOT&E’S LAT protocol attempts to be consistent with ANSI standard because it designates the helmet shell as both the unit of sampling and the unit of testing and analysis. However, the protocol also says:

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

TABLE 8-4 Subtest Acceptance Quality Limits (Approximate)

Lot Size

Subtest

Sample Size (Number of Helmets)

Accept/Reject Criteria (Number of Helmets)

Subtest AQL(%)

91-150

Hardware

  3

0/1

4.0

Shell

  5

0/1

2.5

151-500

Hardware

  5

0/1

2.5

Shell

  8

1/2

6.5

501-1,200

Hardware

  5

0/1

2.5

Shell

13

1/2

1.0

1,201-3,200

Hardware

  8

1/2

6.5

Shell

13

1/2

4.0

If a perforation [complete penetration] occurs, no additional shots will be taken on the perforated helmet. The perforated helmet will count against the accept/reject criteria in Appendix A. To complete the test matrix,4 a new (untested) helmet will be tested using the full 9mm V0 shot sequence for the helmet that was perforated. Valid penetration and BTD data from both helmets will be used for analysis (DOT&E, 2012, p. 5).5

The result of this requirement is that, if a penetration occurs, the number of helmets sampled will not match the sample size in Table 8-1 or the ANSI standard. Substituting for the penetrated helmet is a conservative approach, in the sense that additional data are collected when a perforation is observed. However, it introduces an additional level of complexity into the test, and it makes it difficult to quantify and compare test protocol performance in terms of OC curves.

A testing regime strictly implemented per the ANSI standard would simply fail any helmet that experienced a single penetration (out of five shots to the helmet). No additional helmets would be substituted in order to complete the total number of shots indicated in Table 8-1. Under this testing protocol, the helmet is the unit of testing and analysis. As such, the helmet is subject to a multi-shot test, and it either passes if no penetrations are observed, or it fails as soon as one penetration is observed. (Note that this is similar to the helmet-level test for FAT that was proposed at the end of Chapter 6.)

Finding 8-5. The DOT&E LAT protocol does not precisely follow the ANSI/ASQ Z1.4-2008 testing protocol that calls for sampling a fixed number of items out of a lot. It requires testing of additional helmets when penetrations occur. Further, the shot is the actual unit of testing, despite the fact that sample sizes are stated in terms of helmets.

This lack of consistency in the current protocol—whether a shot or a helmet is the actual unit of test—makes it challenging to understand and interpret its properties. Further, as described below, it is difficult to connect the test sample sizes to the ANSI standard quality metrics.

Determining the Acceptance Quality Limit

The helmet sample sizes in Table 8-1 are derived from the ANSI standard special inspection level6 S-2 for the hardware and special inspection level S-3 for the shell.7,8 The DOT&E protocol alludes to this indirectly by saying, “Helmet testing is unique in that [it requires] two to three disparate destructive tests…. The total number of helmets allocated to … tests closely reflects the quantities required for … sampling at either the S-2 or S-3 levels” (DOT&E, 2012, p. 5). Table 8-4 provides the acceptance quality limit (AQL) for each of the sub-tests assuming the tested helmets are not perforated. As such, they are approximations of the actual AQLs for the LAT protocol.9

The DOT&E protocol goes on to say that the helmet sample sizes are based on a “4% acceptable quality level” or AQL10 (DOT&E, 2012, p. 6), where “the total number of helmets allocated to penetration and BTD tests closely reflects the quantities required for the S-4 sampling level” (DOT&E, 2012, p. 5) of ANSI/ASQ Z1.4-2008 (ASQ, 2008). This is not correct, in the sense that the quality of shells in the

 

_________________

4Here the term “test matrix” does not refer to Table 8-1. Rather it refers to a second matrix that specifies the shot order for each helmet.

5The committee notes that the DOT&E FAT protocol is silent on what should be done in the event that a helmet perforation occurs during testing. However, the lightweight ACH purchase description matches the DOT&E LAT requirement both to substitute a new helmet if a perforation occurs and to use all of the data (U.S. Army, 2012).

6Per the ANSI standard, special inspection levels “may be used where relatively small sample sizes are necessary and large sampling risks can or must be tolerated” (ASQ, 2008, p. 5).

7Using Table II-A of ANSI/ASQ Z1.4-2008, convert the helmet shell sample sizes in Table 8-1 to the sample size code letters and then use Table I to see that the lot size and letter combinations correspond to the S-2 and S-2 inspection levels.

8The “Shots” sample sizes in Table 8-1 do not correspond to any of the single sampling plan sample sizes in ANSI/ASQ Z1.4-2008. For example, see Table II-A in ASQ (2008).

9These AQLs are approximate because they are derived from the ANSI standard that assumes a fixed sample size, unlike the DOT&E protocol in which the sample size can vary if a perforation is observed.

10Note that ANSI/ASQ Z1.4-2008 defines AQL as the “Acceptance Quality Limit.” It explicitly states, “the use of the abbreviation AQL to mean Acceptable Quality Level is no longer recommended” (ASQ, 2008, p. 8).

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

TABLE 8-5 Sample Sizes per ANSI Standard ASQ Z1.4-2008 to Achieve an AQL of 0.4 Percent

General Inspection Level

Lot Size

Sample Size (Number of Helmets)

Accept/Reject Criteria (Number of Helmets)

S-4

1,201-3,200

32

0/1

S-4

3,201-10,000

32

0/1

S-3

35,001-150,000

32

0/1

S-3

150,001-500,000

32

0/1

S-4

500,001+

125

1/2

SOURCE: Adapted from ASQ (2008).

 

helmets tested for hardware is unknown, and the hardware quality of the helmets whose shells are tested is unknown. Thus, while it is clear that for any lot the subtest AQLs are approximately those given in Table 8-4, the AQL of the helmets can be anywhere between the largest subtest AQL (because different types of defects tend to occur within the same helmets) and the sum of the AQLs for all the subtests (because different types of defects tend to occur on different helmets).

Finding 8-6. The AQL at the helmet level is unknown, despite the current DOT&E protocol that suggests helmets are being tested to a 4 percent AQL. Although the AQL for the helmet shell and hardware can be specified (see Table 8-4), it is not clear how these subsystem AQLs combine at the helmet level, and, further, the AQL associated with helmet BFD performance is not assessed.

The 2013 DoD Inspector General report Advanced Combat Helmet Technical Assessment found, “In selecting the LAT RTP requirement of 4 percent AQL … DOT&E did not consider selecting an AQL that was based on the safety criticality of the helmet” (DoD IG, 2013, p. 13). The report further notes that the Defense Contract Management Agency (DCMA) uses a 0.4 AQL for personal protective equipment and that manufacturers are currently working to a 0.4 percent AQL (DoD IG, 2013).

Table 8-5 provides the sample sizes necessary to achieve an AQL level of 0.4 percent. However, during presentations to the committee on June 17, 2013, DCMA stated that it would defer to Program Executive Office Soldier and DOT&E for setting the appropriate AQL for combat helmets.11

Finding 8-7. As Table 8-5 shows, the required sample size (in terms of helmet shells) to achieve an AQL of 0.4 is roughly three to six times larger than what is specified in the current DOT&E protocol. However, the sample size of 32 helmets for lots up to 500,000 helmets is generally smaller than the total number of helmets required for all the LAT tests as specified in the lightweight helmet purchase description (see the table on p. 76 of U.S. Army [2012], reproduced in Table 8-6 below). These values range from 28 for a lot of 500 helmets or less to 44 for lots of 1,201 to 3,200 helmets.

8.5 USING THE HELMET AS THE UNIT OF TESTING

Helmet-Based Lot Acceptance Testing Protocols

Chapter 6 (Section 6.6) proposed that protocols for future helmet designs be based on helmets as the units of test rather than shots. Such a test design has the advantage of following the ANSI standard more closely. In this section, the committee pursues this topic in the context of LAT.

Table 8-6 shows the number of lightweight Advanced Combat Helmets required for LAT under the current purchase description. Note that the total, including the contingency, is close to (or more than) the 32 helmets required for a 0.4 AQL test (cf. Table 8-4). Thus, if the various tests can be appropriately combined, then a helmet-based test at 0.4 AQL is feasible within the current contract requirements. Similarly, if two shots were required per helmet (say, consisting of a combination of two shell shots or one shell shot and one hardware shot)—rather than five shots per helmet shell and two per hardware test—then the total number of shots is 64, which is less than the combined number of shell and hardware shots currently required for lots greater than 500 helmets. This suggests that a helmet-based test is feasible within current resources.

To illustrate the concept, the committee studied the properties of a helmet-based LAT using simulation. The framework for the simulation study was as follows:

•   32 helmets are shot at three random locations, two of the standard five locations (front, back, right, and left sides, and the crown) and one on hardware.

•   Each non-hardware shot is evaluated for whether it perforates and whether the resulting BFD is less than the required threshold and the hardware test is evaluated for perforation.

•   Hence, in this illustrative test, each helmet is subject to five binary-outcome tests, and each helmet is scored as a pass if all five tests are passed or as a fail otherwise.

Making the BFD test a binary pass/fail is consistent with Recommendation 7-1 and consistent with past Army testing practice.

Figure 8-5 shows the OC curves for this illustrative helmet-based LAT protocol (red) compared to the DOT&E LAT protocol (blue). To do the comparison, the committee calcu-

 

_________________

11CIayton Maddio, Soldier Systems Sector Integrator, DCMA Operations Directorate, noted during an informal discussion with the committee on June, 17, 2013, that, while DCMA Critical Safety Items (CSI) policy is stated with an AQL of 0.4 percent, DCMA policy permits the customer to decide the AQL for CSI items, thus overriding DCMA Policy.

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

TABLE 8-6 Lot Acceptance Testing Helmet Sampling Rate as Specified in the Lightweight Advanced Combat Helmet Purchase Description

Lot Size

Lot Acceptance Testing (Number of Helmets Required)

9-mm RTP/ BTD (Shell)

9-mm RTP (Hardware)

17-grain FSP V50

Blunt Impact

Edging Adhesion

Paint Adhesion

Static Pull Test (Ref. System)

Pad Water Absorbancy

Barcode Label/ Marking

Contingency

Total

500

8

5

2

2

1

2

1

~

~

7

28

501-1,200

13

5

2

3

2

3

1

~

~

8

37

1,201-3,200

13

8

3

3

3

4

1

~

~

9

44

NOTE: BTD, ballistic transient deformation; FSP, fragment simulating projectile; RTP, resistance to penetration.
SOURCE: U.S. Army (2012).

 

lated a combined OC curve for the DOT&E LAT tests. This was accomplished by simulating the appropriate number of shell and hardware shots, each at the same probability of penetration, and also simulating the BFDs associated with the shell impacts. A helmet passed the LAT if the number of shell penetrations did not exceed their accept/reject requirements and the hardware penetrations did not exceed their accept/reject requirements and all of the BFD upper tolerance limits were within requirements. For example, for a manufacturer to pass the DOT&E Combined LAT protocol for lot sizes 1,201 to 3,200, there could be no more than 1 penetration out of 65 shots on 13 helmets and no hardware failures in 16 shots on 8 helmets, and the 80 percent upper tolerance limits in each of the five locations (where the assumption was made that the side shots could not be combined) had to be less than the required thresholds with 90 percent confidence. It is important to note that these combined OC curves are based on the assumption that, if there is a change in the x axis, that change is reflected in the probability of test failure across all tests in the LAT.

image

FIGURE 8-5 Operating characteristic (OC) curves for the illustrative helmet-based lot acceptance testing (LAT) protocol in red compared to the OC curve for the combined resistance to penetration and backface deformation for the Director, Operational Test and Evaluation (DOT&E) LAT protocol in blue. NOTE: AQL, acceptance quality limit.

The main points of Figure 8-5 are that (1) the curves for the illustrative helmet-based test are similar to the current DOT&E LAT in many respects, and (2) varying the AQL allows for tailoring the performance of the helmet-based test.

Finding 8-8. Implementing a helmet-based LAT in place of the current DOT&E protocol is feasible from the perspective of the required testing resources, and such a test can be appropriately tailored by setting the AQL.

Adding Switching Rules

According to ANSI/ASQ Z1.4-2008, “AQL is the quality level that is the worst tolerable process average when a continuing series of lots is submitted for acceptance sampling” (ASQ, 2008, p. 2). The standard goes on to say,

The purpose of this standard is, through the economic and psychological pressure of lot non-acceptance, to induce a supplier to maintain a process average at least as good as the specified AQL while at the same time providing an upper limit on the consideration of the [government’s] risk of accepting occasional poor lots. The standard is not intended as a procedure for estimating lot quality or for segregating lots (p. 3).

Further, it is important to note that the ANSI standard specifically says,

The concept of AQL only applies when an acceptance sampling scheme with rules for switching between normal, tightened and reduced inspection and discontinuance of sampling inspection is used. These rules are designed to encourage suppliers to have process averages consistently better than the AQL. If suppliers fail to do so, there is a

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

image

FIGURE 8-6 Switching rules from ANSI/ASQ Z1.4-2008. SOURCE: ASQ (2008).

 

high probability of being switched from normal inspection to tightened inspection where lot acceptance becomes more difficult. Once on tightened inspection, unless corrective action is taken to improve product quality, it is very likely that the rule requiring discontinuance of sampling inspection will be invoked (p. 2).

Figure 8-6 illustrates how the switching rules work. A manufacturer starts under the normal regime. Should the manufacturer fail one or two of five consecutive lots, then it is switched to tightened rules, which make it more difficult to pass the LAT. If five consecutive lots are accepted under the tightened rules, then the manufacturer is switched back to the normal regime. On the other hand, if five consecutive lots are not accepted under the tightened regime, then the manufacturer must re-qualify via FAT.

If a manufacturer under the normal regime has 10 consecutive lots accepted, then it is switched to reduced rules that make it easier to pass the LAT. However, as soon as it fails a lot while under the reduced rules, the manufacturer is switched back to the normal regime.

For example, Table 8-7 shows the switching rules for lot sizes of 1,200 to 3,200 with an AQL of 0.4.

Finding 8-9. The DOT&E LAT protocol does not specify the use of switching procedures. Further, the lightweight ACH purchase description explicitly states that switching procedures will not be used (DOT&E, 2012). As a result, the motivation inherent in the ANSI standard for manufacturers to maintain a process average at least as good as the specified AQL is not incorporated into current LAT procedures.

With the current DOT&E LAT protocol, it is difficult to implement switching rules because they must be applied at the subtest level, which introduces a level of complexity in terms of record keeping that may be burdensome. However,

TABLE 8-7 Switching Rules for Lot Sizes of 1,200 to 3,200 with Acceptance Quality Limit of 0.4

Switching Rule

Lot Size

Sample Size (Number of Helmets)

Accept/Reject Criteria (Number of Helmets)

Normal

1,201-3,200

32

0/1

Tightened

3,201-10,000

50

0/1

Reduced

35,001-150,000

13

0/1

SOURCE: Adapted from ASQ (2008).

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×

 

with the application of a helmet-based test, the implementation of switching rules is more feasible.

Recommendation 8-2. If the Director of Operational Test and Evaluation implements a helmet-based protocol, it should specify the use of switching procedures so that manufacturers are motivated to maintain a process average at least as good as the specified acceptance quality limit.

8.6 REFERENCES

ASQ (American Society for Quality). 2008. American National Standard Sampling Procedures and Tables for Inspection by Attributes. ANSI/ASQ Z1.4-2008. American Society for Quality, Milwaukee, Wisc.

DoD (Department of Defense). 1996. Department of Defense Test Method Standard: DoD Preferred Methods for Acceptance of Product. MIL-STD-1916. Department of Defense, Washington, D.C.

DoD IG (Department of Defense Inspector General). 2013. Advanced Combat Helmet Technical Assessment. DODIG-2013-079. Department of Defense, Washington, D.C.

DOT&E. 2012. Standard for Lot Acceptance Ballistic Testing of Military Combat Helmets. Memorandum from J. Michael Gilmore, Director. May 4, 2012. Office of the Secretary of Defense, Washington, D.C. [reprinted in Appendix B]

U.S. Army. 2012. Advanced Combat Helmet (ACH) Purchase Description, Rev A with Change 4. AR/PD 10-02. Soldier Equipment, Program Executive Office—Soldier, Fort Belvoir, Va.

Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 54
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 55
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 56
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 57
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 58
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 59
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 60
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 61
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 62
Suggested Citation:"8 Lot Acceptance Testing." National Research Council. 2014. Review of Department of Defense Test Protocols for Combat Helmets. Washington, DC: The National Academies Press. doi: 10.17226/18621.
×
Page 63
Next: 9 Characterization Tests for the Advanced Combat Helmet and Future Helmets »
Review of Department of Defense Test Protocols for Combat Helmets Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Combat helmets have evolved considerably over the years from those used in World War I to today's Advanced Combat Helmet. One of the key advances was the development of aramid fibers in the 1960s, which led to today's Kevlar-based helmets. The Department of Defense is continuing to invest in research to improve helmet performance, through better design and materials as well as better manufacturing processes.

Review of the Department of Defense Test Protocols for Combat Helmets considers the technical issues relating to test protocols for military combat helmets. At the request of the DOD Director of Operational Test and Evaluation, this report evaluates the adequacy of the Advanced Combat Helmet test protocol for both first article testing and lot acceptance testing, including its use of the metrics of probability of no penetration and the upper tolerance limit (used to evaluate backface deformation). The report evaluates appropriate use of statistical techniques in gathering data; adequacy of current helmet testing procedures; procedures for the conduct of additional analysis of penetration and backface deformation data; and scope of characterization testing relative to the benefit of the information obtained.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!