8


Lot Acceptance Testing

8.0 SUMMARY

Lot acceptance testing (LAT) is used to ensure that manufacturers continue to produce helmets that conform to contract specifications. A random sample of helmets is selected from the production lot, and the helmet shells as well as hardware are tested according to the LAT protocol. The number of helmets in the protocols is determined from an American National Standards Institute (ANSI) standard, and they vary by lot size. This chapter examines the operating characteristic (OC) curves for the Director, Operational Test and Evaluation’s (DOT&E’s) LAT plans and compares them with first article testing (FAT) protocols in the Army’s legacy plans and DOT&E’s plans. The OC curves for the LAT plans for the different lot sizes can vary a lot, indicating that the manufacturer’s and government’s risks can be quite different across lot sizes. This is primarily due to the different sample sizes (number of helmets and number of shots) as determined from ANSI standard. Further, DOT&E’s FAT protocols are considerably less stringent (higher probabilities of acceptance for the OC curves) than their corresponding LAT protocols. This is counter to the philosophy that it should be more difficult for manufacturers to pass FAT than LAT. This issue can be addressed if DOT&E makes changes to the (17, 240) FAT protocol as discussed in Chapters 6 and 7. This chapter also proposes using binary data for backface deformation (BFD) LAT protocols, to make them consistent with the recommendations for FAT. Finally, the committee examines the properties of LAT protocols based on helmets as the unit of testing.

8.1 INTRODUCTION

After a helmet manufacturer has passed FAT and begins production, LAT is used to ensure that the helmets continue to meet contract specifications. This chapter describes the DOT&E’s LAT protocol, which is based on the ANSI standard ASQ Z1.4-20081 for selecting lot sample sizes and acceptance limits (ASQ, 2008). The performance of the DOT&E’s LAT protocol is compared to the Army’s original FAT protocol and DOT&E’s FAT protocol, both in terms of resistance to penetration (RTP) and BFD. This chapter also examines the feasibility of helmet-based LAT protocols.

8.2 LOT ACCEPTANCE TESTING PROTOCOLS

The Army’s Original Lot Acceptance Testing Protocol

Table 8-1 shows the Army’s original LAT protocol for RTP (DoD IG, 2013, p. 6). Note that the number of helmets, and thus the resulting number of shots, is small.

TABLE 8-1 Sample Sizes for the Army’s Historical Lot Acceptance Testing Protocol for a 9-mm RTP Shell

Lot Size

Sample Size

Accept

Reject

4-150

5 shots, 1 helmet

0

1

151-1,200

5 shots, 1 helmet

0

1

1,201-3,200

10 shots, 2 helmets

0

1

SOURCE: DoD IG (2013).

DOT&E’s Lot Acceptance Testing Protocol

For DOT&E’s LAT, the sample sizes (numbers of helmets to be tested) are derived from the ANSI standard ASQ Z1.4-2008 (ASQ, 2008). Table 8-2 is the helmet LAT matrix from Appendix A of the DOT&E LAT protocol.2 It provides the requirements in terms of the number of helmets to be tested,

 

_________________

1The committee notes that the DOT&E protocol does not mention or explicitly reference the ANSI standard. The Army purchase description does specify the ANSI standard (U.S. Army, 2012).

2The current DOT&E LAT and FAT protocols are found in Appendix B of this report.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 54
8 Lot Acceptance Testing 8.0  SUMMARY dard ASQ Z1.4-20081 for selecting lot sample sizes and acceptance limits (ASQ, 2008). The performance of the Lot acceptance testing (LAT) is used to ensure that DOT&E’s LAT protocol is compared to the Army’s original manufacturers continue to produce helmets that conform FAT protocol and DOT&E’s FAT protocol, both in terms of to contract specifications. A random sample of helmets is resistance to penetration (RTP) and BFD. This chapter also selected from the production lot, and the helmet shells as examines the feasibility of helmet-based LAT protocols. well as hardware are tested according to the LAT protocol. The number of helmets in the protocols is determined from an American National Standards Institute (ANSI) standard, 8.2  LOT ACCEPTANCE TESTING PROTOCOLS and they vary by lot size. This chapter examines the operat- ing characteristic (OC) curves for the Director, Operational The Army’s Original Lot Acceptance Testing Protocol Test and Evaluation’s (DOT&E’s) LAT plans and compares Table 8-1 shows the Army’s original LAT protocol for them with first article testing (FAT) protocols in the Army’s RTP (DoD IG, 2013, p. 6). Note that the number of helmets, legacy plans and DOT&E’s plans. The OC curves for the and thus the resulting number of shots, is small. LAT plans for the different lot sizes can vary a lot, indicating that the manufacturer’s and government’s risks can be quite TABLE 8-1 Sample Sizes for the Army’s Historical Lot different across lot sizes. This is primarily due to the differ- Acceptance Testing Protocol for a 9-mm RTP Shell ent sample sizes (number of helmets and number of shots) as determined from ANSI standard. Further, DOT&E’s FAT Lot Size Sample Size Accept Reject protocols are considerably less stringent (higher probabilities 4-150 5 shots, 1 helmet 0 1 of acceptance for the OC curves) than their corresponding 151-1,200 5 shots, 1 helmet 0 1 LAT protocols. This is counter to the philosophy that it 1,201-3,200 10 shots, 2 helmets 0 1 should be more difficult for manufacturers to pass FAT than LAT. This issue can be addressed if DOT&E makes changes SOURCE: DoD IG (2013). to the (17, 240) FAT protocol as discussed in Chapters 6 and 7. This chapter also proposes using binary data for backface deformation (BFD) LAT protocols, to make them consistent DOT&E’s Lot Acceptance Testing Protocol with the recommendations for FAT. Finally, the committee For DOT&E’s LAT, the sample sizes (numbers of helmets examines the properties of LAT protocols based on helmets to be tested) are derived from the ANSI standard ASQ Z1.4- as the unit of testing. 2008 (ASQ, 2008). Table 8-2 is the helmet LAT matrix from Appendix A of the DOT&E LAT protocol.2 It provides the 8.1  INTRODUCTION requirements in terms of the number of helmets to be tested, After a helmet manufacturer has passed FAT and begins production, LAT is used to ensure that the helmets continue 1The committee notes that the DOT&E protocol does not mention or to meet contract specifications. This chapter describes the explicitly reference the ANSI standard. The Army purchase description does DOT&E’s LAT protocol, which is based on the ANSI stan- specify the ANSI standard (U.S. Army, 2012). 2The current DOT&E LAT and FAT protocols are found in Appendix B of this report. 54

OCR for page 54
LOT ACCEPTANCE TESTING 55 TABLE 8-2 Helmet Lot Acceptance Testing Matrix Lot Size Sub-Test Shots Helmets RTP Accept RTP Reject 91-150 9-mm Hardware RTP  6  3 0 1 9-mm Shell RTP/BTD 25  5 0 1 151-500 9-mm Hardware RTP 10  5 0 1 9-mm Shell RTP/BTD 40  8 1 2 501-1,200 9-mm Hardware RTP 10  5 0 1 9-mm Shell RTP/BTD 65 13 1 2 1,201-3,200 9-mm Hardware RTP 16  8 1 2 9-mm Shell RTP/BTD 65 13 1 2 NOTE: BTD, ballistic transient deformation (synonymous with the term BFD used in this report); RTP, resistance to penetration. SOURCE: DOT&E (2012). the total number of shots, and the accept/reject criteria by lot how to implement the protocol in Table 8-3. However, for size. The test plan in Table 8-2 involves a finer division of situations where there are helmets of multiple sizes in a lot, lot sizes and a larger number of helmets and shots than the Table 8-2 does not specify the order in which the different- Army’s legacy protocol (Table 8-1). sized helmets should be tested. The other aspects of DOT&E’s LAT are similar to its FAT protocol, including range setup, the use of clay as a backing Finding 8-1. The DOT&E LAT protocol does not specify material and its calibration, the definitions of complete and helmet size, while the FAT protocol specifies testing of four partial penetrations, and the metrics (RTP and BFD). How- different helmet sizes. ever, unlike FAT, all tests are conducted only under ambient conditions. The 1996 report DoD Preferred Methods for Acceptance Note that the sample sizes for LAT are smaller than FAT of Product, MIL-STD-1916, states: sample sizes. Further, the protocol varies substantially by lot sizes: from a sample size of 5 helmet shells (and a total of The product shall be assembled into identifiable lots, sublots, 25 shots) for the smallest lot to a sample size of 13 helmet or batches, or in such other manner as may be prescribed. shells (and a total of 65 shots) for the largest lot. Similarly, Each lot or batch shall, as far as practicable, consist of units for hardware testing, the sample sizes vary from 3 helmets of product of a single type, grade, class, size [emphasis added], and composition, manufactured under essentially (and 6 shots) to 8 helmets (and 16 shots). the same conditions, and at essentially the same time. (DoD, As with FAT, the DOT&E LAT protocol specifies a helmet 1996, p. 9). test matrix that defines the shot order for each helmet in the test sequence (Table 8-3). Recommendation 8-1. The protocol established by the The DOT&E LAT protocol makes no mention of helmet Director, Operational Test and Evaluation, should be revised size. If lots consist of only one helmet size, then it is clear TABLE 8-3 Helmet Shot Order Test Matrix for Aramid 9-mm Helmet Order LAT Helmet #1 B L Cr F R LAT Helmet #2 Cr R B L F LAT Helmet #3 R B Cr L F LAT Helmet #4 B F L R Cr LAT Helmet #5 B R F L Cr LAT Helmet #6 Cr B L F R LAT Helmet #7 L B Cr F R LAT Helmet #8 Cr B R F L LAT Helmet #9 L F R B Cr LAT Helmet #10 F Cr B L R LAT Helmet #11 Cr L R B F LAT Helmet #12 R F B L Cr LAT Helmet #13 Cr F L B R NOTE: B, back; CR, crown; F, front; L, left; R, right; LAT, lot acceptance testing. SOURCE: DOT&E, 2012.

OCR for page 54
56 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS to explicitly state that: (1) it will be applied separately to each 8.3  EVALUATING PERFORMANCE: COMPARISON OF helmet size; and (2) if the lot contains helmets of multiple OPERATING CHARACTERISTIC CURVES sizes, the test requirements will be applied according to the number of helmets of each size in the lot. Resistance to Penetration This section compares the OC curves of DOT&E’s LAT The Army’s Hybrid Protocols protocol with DOT&E’s FAT protocol and the Army’s origi- nal FAT protocol. In comparing LAT and FAT, it is important As with FAT, the Army has recently introduced modi- to keep in mind that the manufacturer has already demon- fied LAT protocols. For penetration, it is a hybrid of the strated the ability to meet specification requirements via FAT. Army’s historical LAT protocol and DOT&E’s LAT protocol The goal of LAT is to assess whether the manufacturer’s (DOT&E, 2012). helmets continue to conform, and thus the government is expected to assume greater risk at this stage. • In Stage 1, either 5 or 10 shots are taken, depending Figure 8-1 shows the OC curves for the DOT&E LAT pro- on the lot size (as specified in Table 8-3). If there is tocols for the three different lot sizes: 91 to 150 (black), 151 any complete penetration, the test terminates in a to 500 (red), and 501 to 3,200 (green). The interpretation of failure. If there are no complete penetrations, the test an OC curve here is the same as that in Chapter 6: It is a plot continues to Stage 2. of the probability of acceptance (passing LAT in this case) • In Stage 2, passing the LAT RTP requirement is based on the y axis versus the true penetration probability on the x on the accept/reject criterion specified in the DOT&E axis. In Figure 8-1, the OC curves for the different lot sizes protocol (Table 8-2). As described in the DOT&E vary considerably and hence can have quite different manu- protocol, if a penetration is observed, then a new facturer’s and government’s risks. For example, the blue line helmet is substituted and tested, and the data from corresponds to a penetration probability of 0.005 (current both helmets are counted toward the final accept/ levels where manufacturers are operating), and the prob- reject determination. abilities of acceptance for the three curves range from about 0.88 to about 0.99. Thus, the manufacturer’s risks (which Hardware testing is conducted strictly in accordance with the equal 1 – probability of acceptance) range from 0.01 to 0.12. DOT&E protocol (DOT&E, 2012). Consider now the case where the probability of penetration For BFD, the Army’s LAT hybrid protocol is based on is around 0.05—which is an order of magnitude higher. The the same hybrid test for penetration (DOT&E, 2012). If the purple lines indicate that the probabilities of acceptance, or test continues as a result of successful completion of the government’s risk, vary from about 0.18 to 0.4. first stage RTP test described above, then passing the LAT It is difficult to match the OC curves very closely if one BFD requirement is based on all of the data collected and the wishes to vary the sample sizes for different lot sizes and, accept/reject criterion specified for the lot size. As before, in particular, fix the sample sizes using the ANSI standard. 8-1 if a penetration is observed during the test, a new helmet is substituted and tested, and the BFD data from both helmets (excluding the shot that resulted in a penetration) are used in the BFD calculations. Thus, the Army’s lightweight advanced combat helmet protocol is virtually the same as the DOT&E protocol. The only difference is that the light- 1.0 n 25 c 0 weight protocol does not specify a two-stage procedure for 40 65 1 1 lot sizes of 91 to 150 helmets; instead, it simply requires a 75 0.8 n sample size Probability of Acceptance c acceptance number percent upper tolerance limit (UTL) at 90 percent confidence (DOT&E, 2012). 0.6 The committee does not study the properties of these hybrid protocols in this chapter because their properties 0.4 are complex. Moreover, as noted in Chapters 6 and 7, the 0.2 committee proposes that the DOT&E protocols be modified rather than addressing the issues through modified two-stage 0.0 protocols. 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Probability of Penentration FIGURE 8-1 Operating characteristic curves for resistance to pen- etration for the three Director, Operational Test and Evaluation, protocols by lot sizes: 91 to 150 (black), 151 to 500 (red), and 501 to 3,200 (green).

OCR for page 54
LOT ACCEPTANCE TESTING 57 n c n c 1.0 25 0 1.0 25 0 40 1 40 1 65 1 65 1 0.8 20 0 0.8 60 1 Probability of Acceptance Probability of Acceptance 240 17 n sample size n sample size c acceptance number c acceptance number 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Lot Proportion Defective Probability of Penetration FIGURE 8-2 Comparison of operating characteristic curves for the FIGURE 8-3 Comparison of operating characteristic curves for three Director, Operational Test and Evaluation (DOT&E) lot ac- the three DOT&E lot acceptance testing protocols (black, red, and ceptance testing protocols (black, red, and green) with the Army’s green) with an illustrative (1, 60) first article testing protocol (red). Legacy first article testing (FAT) protocol (blue) and DOT&E’s FAT protocol (orange). Figure 8-2 provides a comparison of the DOT&E LAT restrict the number of shots for the LAT plans to be 60 or protocols (black, red, and green OC curves) with the Army’s fewer, rather than its current value of 65. legacy FAT protocol (blue) and DOT&E’s FAT protocol The committee emphasizes that these are just illustrative (orange). The OC curve for the Army’s legacy FAT protocol discussions and that the committee is not endorsing a par- is within the range of the curves for DOT&E’s LAT proto- ticular FAT plan for RTP. cols. However, DOT&E’s FAT protocol (17-out-of-240 pen- etrations) has a much higher probability of acceptance than Backface Deformation the LAT protocols in the left end of Figure 8-2. This region corresponds to penetration probabilities of 0.08 or less, cov- The Army’s historical LAT BFD protocol was also based ering the current region where manufacturers operate as well on the sample sizes in Table 8-1. For each of the shots, the as penetration levels more than an order of magnitude higher. BFD was measured and compared to a threshold: 25.4 mm So, the manufacturer’s risk for the LAT protocols is higher for front and back shots and 16 mm for side and crown shots. than that for the DOT&E FAT protocol. This is counter to If any of the BFDs exceeded its associated standard, then the the philosophy that LAT should be easier for manufacturers lot failed. In other words, the BFD LAT protocol, like the to pass than FAT. BFD FAT protocol, was based on binary outcomes—whether the BFD measurement exceeded the threshold or not. Finding 8-2. Some of the DOT&E LATs for penetration DOT&E’s LAT protocol, like its FAT protocol, assesses are more difficult for manufacturer’s to pass than the FAT helmet BFD performance using statistical tolerance limits plans. This is contrary to the philosophy that LAT is intended (discussed in Chapter 7). The LAT procedures continue to to assess whether the manufacturers helmets continue to fix the confidence levels at 90 percent. However, unlike FAT conform to specifications, and so it should be less stringent where the UTL was also fixed at 90 percent, the UTLs for than FAT. LAT vary with lot size (and hence with sample size): 80 percent UTL for lot sizes of 501 to 3,200 helmets, 75 percent As discussed in Chapter 6, the problem illustrated in Fig- UTL for lot sizes of 151 to 500 helmets, and a more compli- ure 8-2 is with DOT&E’s (17, 240) FAT protocol. For illus- cated two-stage procedure for lot sizes of 91 to 150 helmets. trative purposes, consider the situation in which the DOT&E The DOT&E LAT protocol states that the “UTL (at 90 FAT is changed to a 1-out-of-60 (1, 60) plan. Figure 8-3 percent confidence) will be calculated by combining the shows a comparison of the OC curve of this plan with those right and left shot locations if the data from the qualifying of the current LAT OC curves. The blue curve corresponds First Article Test indicates the data from the side locations to the (1, 60) FAT plan and, as to be expected, it is very close can be combined for analysis.”3 This procedure is different to the 1-out-of-65 (1, 65) LAT plan that corresponds to the from the DOT&E FAT protocol in which back and front largest lot size. If one wanted to insist that LAT plans be less stringent than the corresponding FAT plans, one could 3DOT&E, 2012, pp. 5-6; reprinted in Appendix B

OCR for page 54
58 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS 1.0 and the three DOT&E LAT protocols (solid and dashed reds). As was the case with penetration, the curves for the three BFD LAT protocols vary considerably, indicating that they can have quite different manufacturer’s and government’s 0.8 risks. In particular, the OC curves for the large two lot sizes (dashed reds) have much higher probabilities of acceptance Pr[Pass BFD Test(s)] (OC curves to the right) than that of the small lot size. Thus, 0.6 it is easier to pass the LATs for the larger lot sizes. Turning to a comparison with the FAT protocols (black and blue curves), one sees that the Army’s legacy FAT proto- 0.4 col has a very similar performance to that of the LAT curve for the small lot size of 91 to 150. On the other hand, the 0.2 Original Army FAT Protocol (0/20) Current DOT&E FAT Protocol (17/240) OC curve for the DOT&E FAT protocol (blue curve) is much DOT&E LAT Protocol, N=91-150 DOT&E LAT Protocol, N=151-500 further to the right than the other curves, indicating that the DOT&E LAT Protocol, N=501-3200 FAT protocol for BFD is much easier to pass than the LAT 0.0 protocols. This conclusion is similar to the one that can be -5 -4 -3 -2 -1 0 made from Figure 8-2 for penetration. True mean BFD (in standard deviations from UTL criteria) Finding 8-3. The OC curves of the DOT&E LATs for BFD FIGURE 8-4 Backface deformation (BFD) operating characteristic vary considerably, indicating that the protocols for the dif- curves for the Director, Operational Test and Evaluation (DOT&E) ferent lot sizes can have quite different manufacturer’s and first article testing (FAT) protocol in blue, the original Army FAT government’s risks. The protocol for the small lot size is protocol in black, and the DOT&E lot acceptance testing (LAT) pro- more stringent than the ones for the medium and large lot tocols in red. NOTE: N is the lot size; UTL, upper tolerance limit. sizes. Finding 8-4. DOT&E’s LAT protocols for BFD are more are grouped into one category and left, right, and crown are difficult for manufacturers to pass than its FAT. This mirrors grouped into another. a similar finding for penetration. This result is contrary to Figure 8-4 compares the performance of the various the philosophy that LAT should be less stringent than FAT. DOT&E LAT protocols (one for each lot size) against the Army’s original (0, 20) FAT protocol and DOT&E’s FAT Backface Deformation Lot Acceptance Testing Protocols protocol for BFD. These results are based on a simulation Based on Binary Data study conducted under the following scenario: As noted in Chapter 7, there are many difficulties with • The BFD measurements are normally distributed. the use of tolerance limits for the BFD protocols. If DOT&E • The sample size is held constant in accordance with were to implement Recommendation 7-1 to revert to the use the lot size requirements of Table 8-1 (which occurs of binary data for BFD for FAT protocols, a similar change if there are no penetrations). should necessarily be made to LAT protocols. This would • The standard deviations are fixed as follows: 2.02- simplify many of the additional complexities associated mm for the front and back locations and 1.58-mm with LAT protocols and combine them across shot loca- for the side and crown locations. (These values were tions. It would also have the added advantage of using the derived from actual BFD data). same LAT protocols for penetration and BFD and make the • The means are varied. The x axis of Figure 8-4 shows BFD protocols easier to understand and more transparent to the true mean in terms of standardized distance from nonstatisticians. the respective UTL thresholds. The standardized distance (true mean minus BFD*) is divided by the 8.4  ANSI STANDARD AND THE ACCEPTANCE standard deviation. BFD* is the UTL threshold: 25.4- QUALITY LIMIT mm for front and back shots and 16-mm for side and crown shots. For example, if the true mean for the Comparison to the ANSI Standard front location is set at 23.38-mm, the standardized distance on the x axis in Figure 8-4 will be (23.38- DOT&E’S LAT protocol attempts to be consistent with mm – 25.4-mm)/2.02-mm = −1. ANSI standard because it designates the helmet shell as both the unit of sampling and the unit of testing and analysis. Figure 8-4 shows the OC curves for the original Army However, the protocol also says: FAT protocol (in black), the DOT&E FAT protocol (blue),

OCR for page 54
LOT ACCEPTANCE TESTING 59 TABLE 8-4 Subtest Acceptance Quality Limits (Approximate) Sample Size Accept/Reject Criteria Lot Size Subtest (Number of Helmets) (Number of Helmets) Subtest AQL(%) 91-150 Hardware  3 0/1 4.0 Shell  5 0/1 2.5 151-500 Hardware  5 0/1 2.5 Shell  8 1/2 6.5 501-1,200 Hardware  5 0/1 2.5 Shell 13 1/2 1.0 1,201-3,200 Hardware  8 1/2 6.5 Shell 13 1/2 4.0 If a perforation [complete penetration] occurs, no additional This lack of consistency in the current protocol—whether shots will be taken on the perforated helmet. The perforated a shot or a helmet is the actual unit of test—makes it chal- helmet will count against the accept/reject criteria in Ap- lenging to understand and interpret its properties. Further, pendix A. To complete the test matrix,4 a new (untested) as described below, it is difficult to connect the test sample helmet will be tested using the full 9mm V0 shot sequence for sizes to the ANSI standard quality metrics. the helmet that was perforated. Valid penetration and BTD data from both helmets will be used for analysis (DOT&E, 2012, p. 5).5 Determining the Acceptance Quality Limit The result of this requirement is that, if a penetration occurs, The helmet sample sizes in Table 8-1 are derived from the the number of helmets sampled will not match the sample ANSI standard special inspection level6 S-2 for the hardware size in Table 8-1 or the ANSI standard. Substituting for the and special inspection level S-3 for the shell.7,8 The DOT&E penetrated helmet is a conservative approach, in the sense protocol alludes to this indirectly by saying, “Helmet testing that additional data are collected when a perforation is is unique in that [it requires] two to three disparate destruc- observed. However, it introduces an additional level of com- tive tests. . . . The total number of helmets allocated to . . . plexity into the test, and it makes it difficult to quantify and tests closely reflects the quantities required for . . . sampling compare test protocol performance in terms of OC curves. at either the S-2 or S-3 levels” (DOT&E, 2012, p. 5). Table A testing regime strictly implemented per the ANSI stan- 8-4 provides the acceptance quality limit (AQL) for each of dard would simply fail any helmet that experienced a single the sub-tests assuming the tested helmets are not perforated. penetration (out of five shots to the helmet). No additional As such, they are approximations of the actual AQLs for the helmets would be substituted in order to complete the total LAT protocol.9 number of shots indicated in Table 8-1. Under this testing The DOT&E protocol goes on to say that the helmet protocol, the helmet is the unit of testing and analysis. As sample sizes are based on a “4% acceptable quality level” such, the helmet is subject to a multi-shot test, and it either or AQL10 (DOT&E, 2012, p. 6), where “the total number passes if no penetrations are observed, or it fails as soon as of helmets allocated to penetration and BTD tests closely one penetration is observed. (Note that this is similar to the reflects the quantities required for the S-4 sampling level” helmet-level test for FAT that was proposed at the end of (DOT&E, 2012, p. 5) of ANSI/ASQ Z1.4-2008 (ASQ, 2008). Chapter 6.) This is not correct, in the sense that the quality of shells in the 6Per Finding 8-5. The DOT&E LAT protocol does not precisely the ANSI standard, special inspection levels “may be used where relatively small sample sizes are necessary and large sampling risks can or follow the ANSI/ASQ Z1.4-2008 testing protocol that calls must be tolerated” (ASQ, 2008, p. 5). for sampling a fixed number of items out of a lot. It requires 7Using Table II-A of ANSI/ASQ Z1.4-2008, convert the helmet shell testing of additional helmets when penetrations occur. Fur- sample sizes in Table 8-1 to the sample size code letters and then use Table ther, the shot is the actual unit of testing, despite the fact that I to see that the lot size and letter combinations correspond to the S-2 and sample sizes are stated in terms of helmets. S-2 inspection levels. 8The “Shots” sample sizes in Table 8-1 do not correspond to any of the single sampling plan sample sizes in ANSI/ASQ Z1.4-2008. For example, 4Here the term “test matrix” does not refer to Table 8-1. Rather it refers see Table II-A in ASQ (2008). to a second matrix that specifies the shot order for each helmet. 9These AQLs are approximate because they are derived from the ANSI 5The committee notes that the DOT&E FAT protocol is silent on what standard that assumes a fixed sample size, unlike the DOT&E protocol in should be done in the event that a helmet perforation occurs during testing. which the sample size can vary if a perforation is observed. However, the lightweight ACH purchase description matches the DOT&E 10Note that ANSI/ASQ Z1.4-2008 defines AQL as the “Acceptance Qual- LAT requirement both to substitute a new helmet if a perforation occurs and ity Limit.” It explicitly states, “the use of the abbreviation AQL to mean to use all of the data (U.S. Army, 2012). Acceptable Quality Level is no longer recommended” (ASQ, 2008, p. 8).

OCR for page 54
60 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS TABLE 8-5 Sample Sizes per ANSI Standard ASQ Z1.4- roughly three to six times larger than what is specified in the 2008 to Achieve an AQL of 0.4 Percent current DOT&E protocol. However, the sample size of 32 Accept/Reject helmets for lots up to 500,000 helmets is generally smaller General Sample Size Criteria than the total number of helmets required for all the LAT tests Inspection (Number of (Number of as specified in the lightweight helmet purchase description Level Lot Size Helmets) Helmets) (see the table on p. 76 of U.S. Army [2012], reproduced in S-4 1,201-3,200 32 0/1 Table 8-6 below). These values range from 28 for a lot of S-4 3,201-10,000 32 0/1 500 helmets or less to 44 for lots of 1,201 to 3,200 helmets. S-3 35,001-150,000 32 0/1 S-3 150,001-500,000 32 0/1 8.5  USING THE HELMET AS THE UNIT OF TESTING S-4 500,001+ 125 1/2 Helmet-Based Lot Acceptance Testing Protocols SOURCE: Adapted from ASQ (2008). Chapter 6 (Section 6.6) proposed that protocols for future helmet designs be based on helmets as the units of test rather helmets tested for hardware is unknown, and the hardware than shots. Such a test design has the advantage of following quality of the helmets whose shells are tested is unknown. the ANSI standard more closely. In this section, the commit- Thus, while it is clear that for any lot the subtest AQLs are tee pursues this topic in the context of LAT. approximately those given in Table 8-4, the AQL of the Table 8-6 shows the number of lightweight Advanced helmets can be anywhere between the largest subtest AQL Combat Helmets required for LAT under the current pur- (because different types of defects tend to occur within the chase description. Note that the total, including the contin- same helmets) and the sum of the AQLs for all the subtests gency, is close to (or more than) the 32 helmets required for (because different types of defects tend to occur on different a 0.4 AQL test (cf. Table 8-4). Thus, if the various tests can helmets). be appropriately combined, then a helmet-based test at 0.4 AQL is feasible within the current contract requirements. Finding 8-6. The AQL at the helmet level is unknown, Similarly, if two shots were required per helmet (say, consist- despite the current DOT&E protocol that suggests helmets ing of a combination of two shell shots or one shell shot and are being tested to a 4 percent AQL. Although the AQL for one hardware shot)—rather than five shots per helmet shell the helmet shell and hardware can be specified (see Table and two per hardware test—then the total number of shots 8-4), it is not clear how these subsystem AQLs combine at the is 64, which is less than the combined number of shell and helmet level, and, further, the AQL associated with helmet hardware shots currently required for lots greater than 500 BFD performance is not assessed. helmets. This suggests that a helmet-based test is feasible within current resources. The 2013 DoD Inspector General report Advanced Com- To illustrate the concept, the committee studied the prop- bat Helmet Technical Assessment found, “In selecting the erties of a helmet-based LAT using simulation. The frame- LAT RTP requirement of 4 percent AQL . . . DOT&E did work for the simulation study was as follows: not consider selecting an AQL that was based on the safety criticality of the helmet” (DoD IG, 2013, p. 13). The report • 32 helmets are shot at three random locations, two further notes that the Defense Contract Management Agency of the standard five locations (front, back, right, and (DCMA) uses a 0.4 AQL for personal protective equipment left sides, and the crown) and one on hardware. and that manufacturers are currently working to a 0.4 percent • Each non-hardware shot is evaluated for whether it AQL (DoD IG, 2013). perforates and whether the resulting BFD is less than Table 8-5 provides the sample sizes necessary to achieve the required threshold and the hardware test is evalu- an AQL level of 0.4 percent. However, during presentations to ated for perforation. the committee on June 17, 2013, DCMA stated that it would • Hence, in this illustrative test, each helmet is subject defer to Program Executive Office Soldier and DOT&E for to five binary-outcome tests, and each helmet is setting the appropriate AQL for combat helmets.11 scored as a pass if all five tests are passed or as a fail otherwise. Finding 8-7. As Table 8-5 shows, the required sample size (in terms of helmet shells) to achieve an AQL of 0.4 is Making the BFD test a binary pass/fail is consistent with Recommendation 7-1 and consistent with past Army testing 11CIayton Maddio, Soldier Systems Sector Integrator, DCMA Opera- practice. tions Directorate, noted during an informal discussion with the committee Figure 8-5 shows the OC curves for this illustrative hel- on June, 17, 2013, that, while DCMA Critical Safety Items (CSI) policy is met-based LAT protocol (red) compared to the DOT&E LAT stated with an AQL of 0.4 percent, DCMA policy permits the customer to decide the AQL for CSI items, thus overriding DCMA Policy. protocol (blue). To do the comparison, the committee calcu-

OCR for page 54
LOT ACCEPTANCE TESTING 61 TABLE 8-6 Lot Acceptance Testing Helmet Sampling Rate as Specified in the Lightweight Advanced Combat Helmet Purchase Description Lot Acceptance Testing (Number of Helmets Required) Static Pull Barcode 9-mm RTP/ 9-mm RTP 17-grain Blunt Edging Paint Test (Ref. Pad Water Label/ Lot Size BTD (Shell) (Hardware) FSP V50 Impact Adhesion Adhesion System) Absorbancy Marking Contingency Total 500  8 5 2 2 1 2 1 ~ ~ 7 28 501-1,200 13 5 2 3 2 3 1 ~ ~ 8 37 1,201-3,200 13 8 3 3 3 4 1 ~ ~ 9 44 NOTE: BTD, ballistic transient deformation; FSP, fragment simulating projectile; RTP, resistance to penetration. SOURCE: U.S. Army (2012). lated a combined OC curve for the DOT&E LAT tests. This limits in each of the five locations (where the assumption was was accomplished by simulating the appropriate number of made that the side shots could not be combined) had to be shell and hardware shots, each at the same probability of pen- less than the required thresholds with 90 percent confidence. etration, and also simulating the BFDs associated with the It is important to note that these combined OC curves are shell impacts. A helmet passed the LAT if the number of shell based on the assumption that, if there is a change in the x penetrations did not exceed their accept/reject requirements axis, that change is reflected in the probability of test failure and the hardware penetrations did not exceed their accept/ across all tests in the LAT. reject requirements and all of the BFD upper tolerance limits The main points of Figure 8-5 are that (1) the curves for were within requirements. For example, for a manufacturer the illustrative helmet-based test are similar to the current to pass the DOT&E Combined LAT protocol for lot sizes DOT&E LAT in many respects, and (2) varying the AQL 1,201 to 3,200, there could be no more than 1 penetration allows for tailoring the performance of the helmet-based test. out of 65 shots on 13 helmets and no hardware failures in 16 shots on 8 helmets, and the 80 percent upper tolerance Finding 8-8. Implementing a helmet-based LAT in place of the current DOT&E protocol is feasible from the perspec- tive of the required testing resources, and such a test can be appropriately tailored by setting the AQL. 1.0 Helmet-level LAT Protocol (AQL 0.4): n=32, a/r=0/1 Helmet-level LAT Protocol (AQL 1.0): n=13, a/r=0/1 Helmet-level LAT Protocol (AQL 2.5): n=5, a/r=0/1 DOT&E Combined LAT Protocols: N=91-150 Adding Switching Rules DOT&E Combined LAT Protocols: N=151-500 According to ANSI/ASQ Z1.4-2008, “AQL is the quality 0.8 DOT&E Combined LAT Protocols: N=501-1200 DOT&E Combined LAT Protocols: N=1201-3200 level that is the worst tolerable process average when a con- tinuing series of lots is submitted for acceptance sampling” 0.6 (ASQ, 2008, p. 2). The standard goes on to say, Pr(Pass LAT) The purpose of this standard is, through the economic and 0.4 psychological pressure of lot non-acceptance, to induce a supplier to maintain a process average at least as good as the specified AQL while at the same time providing an up- per limit on the consideration of the [government’s] risk of 0.2 accepting occasional poor lots. The standard is not intended as a procedure for estimating lot quality or for segregating lots (p. 3). 0.0 0.00 0.02 0.04 0.06 0.08 0.10 Further, it is important to note that the ANSI standard True Pr(individual test fails) specifically says, FIGURE 8-5 Operating characteristic (OC) curves for the illustra- The concept of AQL only applies when an acceptance tive helmet-based lot acceptance testing (LAT) protocol in red sampling scheme with rules for switching between normal, compared to the OC curve for the combined resistance to penetra- tightened and reduced inspection and discontinuance of tion and backface deformation for the Director, Operational Test sampling inspection is used. These rules are designed to and Evaluation (DOT&E) LAT protocol in blue. NOTE: AQL, encourage suppliers to have process averages consistently acceptance quality limit. better than the AQL. If suppliers fail to do so, there is a

OCR for page 54
62 REVIEW OF DEPARTMENT OF DEFENSE TEST PROTOCOLS FOR COMBAT HELMETS FIGURE 8-6 Switching rules from ANSI/ASQ Z1.4-2008. SOURCE: ASQ (2008). high probability of being switched from normal inspection Finding 8-9. The DOT&E LAT protocol does not specify to tightened inspection where lot acceptance becomes more the use of switching procedures. Further, the lightweight difficult. Once 8-6, fixed image Figuer on tightened inspection, unless corrective ACH purchase description explicitly states that switching action is taken to improve product quality, it is very likely procedures will not be used (DOT&E, 2012). As a result, the that the rule requiring discontinuance of sampling inspection motivation inherent in the ANSI standard for manufacturers will be invoked (p. 2). to maintain a process average at least as good as the speci- fied AQL is not incorporated into current LAT procedures. Figure 8-6 illustrates how the switching rules work. A manufacturer starts under the normal regime. Should the With the current DOT&E LAT protocol, it is difficult to manufacturer fail one or two of five consecutive lots, then it implement switching rules because they must be applied at is switched to tightened rules, which make it more difficult the subtest level, which introduces a level of complexity in to pass the LAT. If five consecutive lots are accepted under terms of record keeping that may be burdensome. However, the tightened rules, then the manufacturer is switched back to the normal regime. On the other hand, if five consecutive lots are not accepted under the tightened regime, then the manufacturer must re-qualify via FAT. TABLE 8-7 Switching Rules for Lot Sizes of 1,200 to If a manufacturer under the normal regime has 10 con- 3,200 with Acceptance Quality Limit of 0.4 secutive lots accepted, then it is switched to reduced rules Accept/Reject that make it easier to pass the LAT. However, as soon as it Sample Size Criteria (Number of (Number of fails a lot while under the reduced rules, the manufacturer is Switching Rule Lot Size Helmets) Helmets) switched back to the normal regime. For example, Table 8-7 shows the switching rules for lot Normal 1,201-3,200 32 0/1 sizes of 1,200 to 3,200 with an AQL of 0.4. Tightened 3,201-10,000 50 0/1 Reduced 35,001-150,000 13 0/1 SOURCE: Adapted from ASQ (2008).

OCR for page 54
LOT ACCEPTANCE TESTING 63 with the application of a helmet-based test, the implementa- DoD (Department of Defense). 1996. Department of Defense Test Method Standard: DoD Preferred Methods for Acceptance of Product. MIL- tion of switching rules is more feasible. STD-1916. Department of Defense, Washington, D.C. DoD IG (Department of Defense Inspector General). 2013. Advanced Recommendation 8-2. If the Director of Operational Test Combat Helmet Technical Assessment. DODIG-2013-079. Department and Evaluation implements a helmet-based protocol, it of Defense, Washington, D.C. should specify the use of switching procedures so that manu- DOT&E. 2012. Standard for Lot Acceptance Ballistic Testing of Military Combat Helmets. Memorandum from J. Michael Gilmore, Director. facturers are motivated to maintain a process average at least May 4, 2012. Office of the Secretary of Defense, Washington, D.C. as good as the specified acceptance quality limit. [reprinted in Appendix B] U.S. Army. 2012. Advanced Combat Helmet (ACH) Purchase Description, Rev A with Change 4. AR/PD 10-02. Soldier Equipment, Program 8.6  REFERENCES Executive Office—Soldier, Fort Belvoir, Va. ASQ (American Society for Quality). 2008. American National Standard Sampling Procedures and Tables for Inspection by Attributes. ANSI/ ASQ Z1.4-2008. American Society for Quality, Milwaukee, Wisc.