5

Instrumentation and Procedures for Measuring an Indent in a Clay Backing Material

The committee was tasked to provide findings on the best instrumentation and procedures for measuring backface deformation (BFD) in clay. Accordingly, this chapter discusses relevant criteria for test instrumentation and procedures, including fixed and variable costs, precision and accuracy, and human operator considerations.

CONCEPTUAL STEPS TOWARD IMPROVEMENTS IN THE MEASUREMENT OF BFD

It is informative to review past events to learn how the instrumentation and measurements of BFD relate to the overall methodology of the original animal tests, clay selection, selection of performance specifications, and instrumentation measurements. These conceptual steps give some direction for improvements of testing procedures. It may be noted that in most experimental studies and scientific measurements, there are several conceptual and scientific stages that need to be considered and followed: (1) conceptualization of which phenomena or parameters must be measured, (2) the validity and completeness of the design of the experiment or measurement protocol to ensure a complete and accurate data set that eliminates outside variables, and (3) statistics associated with the measurements, including instrumentation accuracy vs. required accuracy. The six conceptual steps that follow trace the development of BFD measurement specifications.

Step One

Animal experiments with goats indicated that indents of 40 to 50-mm were the maximum acceptable value. Specifically, from Chapter 3 in this report, 44-mm deformation in the goats was correlated with ~10 percent probability of death from the impactor, similar to the initial program requirement of less than or equal to 10 percent lethality. This deformation level in the goats was below the lowest value for which any of the goats died in the much lower velocity impactor tests. As such, this depth was selected as the injury reference value in the clay.

This selection implies that a 44-mm goat impactor deformation is similar to a 56-mm deformation in clay or gelatin. Conversely, a 44-mm deformation in the clay is similar to 34-mm deformation in the goat for this hard impactor, as shown in Figure 3-5. (In Figure 3.5, the fatality risk at 34 mm of deformation in



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION 5 Instrumentation and Procedures for Measuring an Indent in a Clay Backing Material The committee was tasked to provide findings on the best instrumentation and procedures for measuring backface deformation (BFD) in clay. Accordingly, this chapter discusses relevant criteria for test instrumentation and procedures, including fixed and variable costs, precision and accuracy, and human operator considerations. CONCEPTUAL STEPS TOWARD IMPROVEMENTS IN THE MEASUREMENT OF BFD It is informative to review past events to learn how the instrumentation and measurements of BFD relate to the overall methodology of the original animal tests, clay selection, selection of performance specifications, and instrumentation measurements. These conceptual steps give some direction for improvements of testing procedures. It may be noted that in most experimental studies and scientific measurements, there are several conceptual and scientific stages that need to be considered and followed: (1) conceptualization of which phenomena or parameters must be measured, (2) the validity and completeness of the design of the experiment or measurement protocol to ensure a complete and accurate data set that eliminates outside variables, and (3) statistics associated with the measurements, including instrumentation accuracy vs. required accuracy. The six conceptual steps that follow trace the development of BFD measurement specifications. Step One Animal experiments with goats indicated that indents of 40 to 50-mm were the maximum acceptable value. Specifically, from Chapter 3 in this report, 44-mm deformation in the goats was correlated with ~10 percent probability of death from the impactor, similar to the initial program requirement of less than or equal to 10 percent lethality. This deformation level in the goats was below the lowest value for which any of the goats died in the much lower velocity impactor tests. As such, this depth was selected as the injury reference value in the clay. This selection implies that a 44-mm goat impactor deformation is similar to a 56-mm deformation in clay or gelatin. Conversely, a 44-mm deformation in the clay is similar to 34-mm deformation in the goat for this hard impactor, as shown in Figure 3-5. (In Figure 3.5, the fatality risk at 34 mm of deformation in -92-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION goats, or 44 mm of deformation in clay, is approximately 4 percent.) It was reasonable to assume that use of the blunt impactor tests to model the injury behavior from behind-armor body trauma was “likely” conservative for this impact velocity range. However, outside the velocity range typical of handgun rounds (i.e., 240 m/sec) into soft body armor, the relationship between injury and deformation response in the clay is much less certain. More recent measurements in sheep using velocities of about 800 m/sec showed significant lethality even with 34-mm deep indents (cf. Chapter 8 and Gryth et al., 2007). Step Two Clay BFDs were tailored to mimic such an indent. From Chapter 4 in this report, and as introduced in Chapters 2 and 3, the Roma Plastilina #1 (RP #1) modeling clay backing material used in armor testing has two important purposes. The first is to simulate the tissue response beneath the point of impact so that ballistic data generated in laboratory tests can be correlated to effects seen on the human body. The second purpose is to denote the extent of BFD during ballistic testing (Prather et al., 1977). Multiple materials are available to simulate a body; in fact, at the time it was introduced, modeling clay was recognized to only approximate tissue response, and empirical correlations were needed to develop a probability for lethality or injury. The chief advantage of modeling clay over other materials available at the time was that it better served the function of recording BFDs; that is, when impacted, modeling clay deforms plastically, and a permanent cavity (also termed “indent,” “impression,” or “crater”) is developed under the point of impact. Correlations were developed between the geometry of the cavity and the probability of lethal injury. Step Three The U.S. Army Aberdeen Test Center (ATC) has set the maximum acceptable BFD value at 44 mm for body armor plates tested using clay. This value appeared reasonable based upon the past measurements. As noted in Chapter 3, the Army does not have the medical outcomes to know whether 44 mm is a conservative value. Step Four Measurement instruments were used to verify the test results, as directed by the procurement specifications. Digital calipers and then laser-based instruments were used to better measure the BFD under nonideal conditions (i.e., offset and side/edge indents) (Walton et al., 2008). However, different instruments may give different BFD readings due to each instrument having -93-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION different measurement precision and different measurement accuracy that is dependent upon the measurement scenario. Steps Five and Six Two steps have taken place since the National Research Council (NRC) was tasked to study BFD measurement techniques: Step Five: The office of the Director, Operational Test & Evaluation (DOT&E) developed a statistically based protocol and test processes, including measurement techniques, for first article and lot acceptance testing of hard body armor (DOT&E, 2010). (See Chapter 6.) Step Six: The NRC committee examined data related to the precision and accuracy of different measurement instruments, under different measurement scenarios, to gain insights into which instruments or usage might meet or exceed the precision and accuracy required to measure BFD. The different instruments and different measurement scenarios are covered in this chapter, and statistical considerations are presented in Appendixes G and M. It is informative to keep in mind that the above conceptual steps were made to address four overall questions: (1) How well do the testing procedure and measurements of the BFD quantify the probability of lethality being measured? (2) Is the measurement a complete and scientifically valid measurement set that eliminates outside variables—e.g., the design of experiment or measurement procedure? (3) How well must the BFD be measured? (4) What are the statistical accuracy and precision of the measurements? INSTRUMENTATION PERFORMANCE BASED ON STATISTICAL ANALYSIS When discussing body armor testing and, particularly, the equipment required in the conduct of adequate tests, the question arises: How well must the BFD be measured? Put another way, what are the limits of acceptable error in BFD measurement? To answer this question, it is first necessary to define a number of terms. The Phase I report (NRC, 2009) discussed the difference between the accuracy and the precision of a measuring device. Although the two terms are often used interchangeably and considered synonymous in colloquial use, they have quite different technical meanings. Accuracy is the closeness of a measured quantity to its actual (true) value. Precision is the closeness of agreement between -94-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION measured values obtained by replicate measurements under specified conditions (NRC, 2009). A measurement device is valid if it is both accurate and precise. However, a device can be precise but not accurate, accurate but not precise, or neither accurate nor precise. If a measurement device has a systematic error, then increasing the number of samples (the sample size) increases precision but does not improve accuracy. On the other hand, eliminating the systematic error improves accuracy but does not change precision. We often quantify precision in terms of either the standard deviation or expanded uncertainty (twice the standard deviation of the repeated values) of the measurement device. Accuracy or bias is estimated as the difference between the average of a large number of repeated values under a specific set of conditions and the true value. Beginning with bias, the ideal measurement device for BFD should have no bias. That said, a biased measuring device may be acceptable in armor testing if it is consistently biased across all possible plate and test configurations. Under these conditions, the BFD test standard can simply be increased or decreased to account for the bias. However, determining that a device is consistently biased is likely to be difficult at best and unlikely to be true in practice. Appendix G demonstrates that there are diminishing returns (and probably increasing costs) in the pursuit of ever more precise measuring devices. This result follows from the fact that the necessary level of measurement precision is a function of the overall variation in the testing process, where, for example, highly precise measurements add little value to a testing process that is itself inherently highly variable. Conversely, in any testing process, there should be a precision threshold that any measurement device must meet—again based on the overall variation of the testing process—to ensure that the measurement process itself does not add to the variability arising from the test. For the current clay-based test methodology, the results from the Appendix G analysis suggest that a precision threshold of 0.5 mm (i.e., a standard deviation) is necessary to ensure that the measurement device does not add any appreciable variation to the body armor testing process. This value is consistent with subject matter experts who expressed to the committee their intuition that measurement precision on the order of 0.5 mm is sufficient for the current testing process. It is also consistent with injury “effect size” calculations done by the committee. It is somewhat larger than the heuristic suggested in the Phase I report (NRC, 2009) that the measurement system variance required for a test should better by a factor of 10 or more than the total measured variation (McNeese and Klein, 1991). The original procurement specifications for body armor plates state that the BFD shall be measured with an instrument that has an accuracy of ±0.1 mm. However, the detailed analysis presented in Appendix G indicates that this value (and the accompanying wording) was probably too stringent (and somewhat ambiguous in its use of the term "accuracy"), and that a more reasonable value for an instrumentation precision of about 0.5 mm would be sufficient to correctly test and detect statistically significant effects. -95-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Finding: Given the current clay variation, a measurement precision (standard deviation) of 0.5 mm is sufficient; instruments featuring greater precision add little practical value to the testing process. Future improvements in the inherent variability of the backing material will require instruments that are correspondingly more precise. OVERVIEW OF CURRENT INSTRUMENTATION AND MEASUREMENTS This section covers instrumentation that has been and is being used for BFD measurements. In particular, three instruments have been used: the Co- ordinate Measuring Machine (CMM), used as a reference instrument; the digital caliper; and a Faro laser scanning probe system. A CMM costs about $500,000; the Faro system, about $150,000; and the digital caliper, about $200. The three systems were used in extensive measurements and tests as reported in the Walton et al. (2008) report. Coordinate Measuring Machine The CMM is a Wenzel X-orbit Bridge type with both digital point probes and LDI laser scanner (Model SLP250); this system is has a precision of ±0.35 mm (0.00035 mm) and an accuracy of 10 mm (0.01 mm). It was considered by ATC to be highly precise and accurate and yields measurement results that can serve as a “true” value (Walton et al., 2008). The two systems that have been used to measure BFD indents made in clay during body armor testing are the digital caliper and the Faro laser. Digital Caliper The digital caliper (referred to as the “caliper”) was used for many years as a low cost, point-to-point instrument to measure BFD depths. This was accomplished by using a manually operated depth probe integrated with an electronic digital display and paired with a bridge gauge to provide a stable base for measurement. The operator affixes a bridge gauge to the side of the box that holds the body armor and underlying clay recording medium. A baseline preshot measurement is made with a digital caliper to the point of aim where the test bullet is expected to strike the armor. The bridge gauge and caliper are removed. The test firing is conducted. The bridge gauge is replaced on the box and a measurement of the postshot BFD is made with the digital caliper. The deepest point in the BFD crater is visually located by the operator, and one reading with the caliper is taken at that point; in this case, an operator with experience and judgment is required for an accurate and consistent measurement. The two types of caliper instruments used are shown in Figure 5-1. -96-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION FIGURE 5-1 Digital calipers used in armor testing. The ATC standard caliper with the small end (3 mm) is shown at the top. The caliper used by commercial testers (H.P. White Laboratory and Chesapeake Testing) with a large 19-mm tip is shown at bottom. The dimension of the wide tip (19 mm) was measured by Chesapeake Testing at the request of the committee. (The center caliper is not used.) SOURCE: Courtesy H.P. White Laboratories. Faro System The Office of the Director, Operational Test and Evaluation (DOT&E) has designated the Faro Quantum Laser Scan Arm and Geomagic Qualify software for hard and soft body armor (referred to as “the Faro”) as the device to measure BFDs (DOT&E, 2010). Laser profilometry, as used by the Faro scanning laser instrument, employs the commonly used principle of optical triangulation. A laser generates a collimated beam, which is then focused and projected onto a target surface. A lens reimages the laser spot formed on the surface of the target onto a charge-coupled device, which generates a signal that is indicative of the spot’s position on the detector. As the height of the target surface changes, the image of -97-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION the laser spot shifts owing to parallax. To generate a three-dimensional image of the specimen’s surface, the sensor scans in two dimensions, generating a set of noncontact measurements that represent the surface topography of the specimen under inspection. The data are then used to compute the three-dimensional geometrical profile of the surface, with readings essentially continuous over the scanned region. Thus, the laser scanner produces a series of measurements over the whole surface of the clay, as opposed to the single reading obtained with the digital caliper. In addition, a laser scanning system has the ability to acquire substantial quantities of inspection data. Figure 5-2 shows the Faro Quantum Laser Scan Arm. FIGURE 5-2 Faro Quantum laser scan arm. SOURCE: © 2012, FARO. http://www.faro.com/FaroArm/Home.htm. Accessed on March 8, 2011. -98-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION BFD MEASURING PROCEDURES The precision and accuracy of instruments depend to a great degree on the associated operating procedures and on the skill of human operators. This section describes the “art of measurement” in testing procedures as observed by the committee. Human Operator Considerations A number of practical human operator considerations have an impact on the measurement differences and variations associated with all measuring systems. These include subjective differences in human handling, process transparency, and the selection and settings of software. Operator-to-Operator (Human Handling) Variability The operator-to-operator (human handling) variability associated with both measuring devices appears to be generally different. However, Walton et al. (2008) suggest that operator variability for the Faro is 0.041 mm and operator/caliper variability for the digital caliper is an order of magnitude greater, at 0.471 mm (Walton et al., 2008). Members of the committee interviewed operators at ATC and at two commercial testing companies.26 They learned a couple of things. The caliper end makes contact with the clay. The operator must use judgment to determine the deepest point in what may be a complex BFD and then carefully and manually push the caliper arm down so it just touches the surface of the clay but does not dent the clay. Operators state that making precise measurements is an art. Variation among operators can be 0.1-0.3 mm when measuring the same BFD in the center of an armor plate. This variation was actually observed at ATC when three different operators measured the same BFD using a digital caliper. The Faro is a noncontact device. The operator must use judgment as to where and how fast to “paint” the armor on a prefire event (to digitally capture the surface of the armor) and similarly how fast to paint the BFD area in a postfire event. The computer program compares the pre- and postfire digits and calculates the maximum BFD. According to experienced operators, variation due to these judgmental factors can result in measurement differences among Faro users of 0.1-0.2 mm for the same BFD. Similar differences in results were observed during a demonstration to the committee when three different operators measured the same BFD with the Faro.27 26 Site visits to the ATC, H.P. White Laboratory, and Chesapeake Testing by members of the committee on August 30 and 31, 2010. 27 Variations were reported by Faro operators at the ATC and commercial testing sites. Variations were then observed by the committee during demonstration at the ATC, August 31, 2010. -99-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Testing protocols should anticipate that anomalous data can occur for any number of reasons and should include procedures to ensure data quality. These protocol procedures can provide a means for operators to quickly confirm that a measurement outside a predetermined upper and lower bound is not due to a major equipment or software malfunction. The committee notes that great caution is warranted if this idea is implemented because it has the potential to lead test operators to focus on measurement differences that are the result of noise and not actual differences. Software Variability There is a software variability associated with the Faro resulting from a software selection that allows for smoothing the raw digital data captured by the Faro. For example, the committee was shown that a change from one level of smoothing to another resulted in a 1-mm difference in the BFD measurement for the same cavity.28 Two software settings are in use, one for 0.7 mm and one for 1.5 mm spatial resolution. An ATC manager stated that ATC testers tended to use the most conservative setting (i.e., higher spatial resolution of 0.7 mm), which will result in the largest BFD measurement to ultimately protect soldiers.29 Manufacturers, on the other hand, feel that their armor may be unfairly penalized due to judgment decisions that depend on the smoothing setting the operator chooses.30 In statistical and testing terms, the choice of the smoothing setting directly affects the accuracy of the Faro. That is, the choice of smoothing settings can introduce a systematic bias into the measurements, a bias that can make the test either harder or easier to pass depending on whether the bias results in systematically larger or smaller BFDs. As discussed in Appendix G (Key Point 4), an overly conservative setting on the Faro laser resulting in high spatial resolution may result in a design penalty that is roughly five times larger than the design space improvement achieved via better measurement precision. Thus, unless care is taken to understand the effect of the software smoothing algorithms on the indent measurement, any gains in precision achieved by using the Faro could be more than offset by a systematic bias. This result might not only make the test harder for manufacturers to pass but also might result in heavier armor if manufacturers are driven to make plates heavier to compensate for a measurement bias. The committee considers the National Institute of Standards and Technology (NIST) to be an excellent third-party source of expertise on measurement instruments and standards, because the NIST has provided significant support to both DOT&E and the Army Program Executive Officer Soldier on both body armor testing and body armor design issues in the past. 28 As observed by committee members during their site visit to the ATC, August 31, 2010. 29 Discussion with Irene Johnson, ATC, during site visit, August 31, 2010. 30 David Reed, President, North American Operations, Ceradyne, Inc. , “Pragmatics of Body Armor Testing—Manufacturer’s View,” presentation to the committee, August 9, 2010. -100-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Recommendation 5-1: An organization such as the National Institute of Standards and Technology should conduct a controlled study to determine the most reasonable and consistent Faro smoothing settings to be used while measuring backface deformations (BFDs) in body armor testing. Similarly, any other software selections that could cause relevant changes to BFD measurements should be studied. Corresponding values for the precision and accuracy of each software setting will need to be quantified. Compensating for Offset between the Point of Aim and the Deepest Indent Sometimes the deepest penetration in the clay and the initial bullet aim point are offset by a small distance. This affects both accuracy and precision of the instrumentation measurements. Operators of the caliper calibrate their instrument on the aim point but move the caliper to measure the deepest point of impact when the aim point and deepest penetration do not align, which happens frequently. The caliper measurement procedure disregards the curvature of the plate and tends to overestimate the depth of the BFD. This correction and offset value can be large and is the result of having only one preshot reading for the plate surface. As a mathematical correction for the offset, an operator referred the committee to an equation for offset contained on an ATC test instruction sheet.31 Government and commercial operators alike felt that the equation was imprecise and would likely lead to an underestimation of BFD. Also, the equation does not make provision for the offset being changed to a positive number if the deepest point is on the upside of the aim point on an edge shot. In comparison to the caliper, the Faro takes into account the curvature of the plate, calculates the geometry, and reduces offset errors. This computational capability allows the Faro to measure and calculate an offset value with high precision and leads to a more accurate measurement of the maximum indent. Variability (Noise) in the Overall Testing Process As described in Chapter 4, there is significant variability in the RP #1 modeling clay that has been used for decades as the backing material in the testing process.  The RP #1 modeling clay was and continues to be designed for artists and not for the ballistics testing community. As a result of requests from artists for a certain feel and other characteristics the formulation has changed over time. From the standpoint of the ballistics testing community, the clay over time has been allowing less deformation than the original RP #1. 31 The equation used is Offset = −0.25 × BFD. The test instruction sheet was shown to members of the committee during site visit to the ATC, August 31, 2010. -101-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION  To compensate for the change in formulation, the testing community has had to warm the clay in ovens to achieve the calibration numbers required by National Institute of Justice (NIJ) standards. Heat introduces significant variation.  The amount of manual working that is performed using mallets to initially pound the clay bricks into the testing box or recondition the clay after a test shot introduces additional variability in clay deformation. As described in Chapter 4, some studies indicate that this human dynamic of working might introduce even more variability in the deformation of clay measurements than changes in temperature.  It has been observed over the years of body armor testing that clay in a box used for testing has a limited useful life. Since old clay may result in unreliable deformations during testing, both government and commercial testers routinely discard clay before it is a year old. Although variability due to aging appears to be less than variability introduced by temperature and working, it is one more indicator of the significant variability that is inherent in RP #1. Owing to the above and other considerations, the NIJ standard allows for significant modeling clay variation. Specifically, to determine if the modeling clay is ready for testing it must calibrate to a specification of 25 mm ± 3 mm. In other words, 6 mm of overall clay variability is accepted as, and perhaps understates, the noise in the testing process related to clay. A great deal of variation is introduced into the measurement system by RP#1. As discussed in Chapter 4, there is much merit in reducing the variability in the recording medium, because with less variability in the recording medium testers can more fairly state that the differences in test results are related to plate behavior. One battlefield payoff will be greater confidence that the plates will function successfully in combat. Another is the possibility that lighter weight plates will be able to pass the tests, ultimately reducing the weight burden on soldiers. Some additional variation occurs as a result of the ammunition that is used during testing. If, for example, a tester was to replicate the real-world threats that a soldier might face, that tester would use ammunition procured from third-world countries. Such ammunition may not have been manufactured to specifications that result in consistent velocity and bullet mass from round to round. Variation in velocity will cause some variation among BFDs that are created during testing, because the energy is proportional to the velocity squared. Bullet velocity measurements are part of ATC testing procedures and should be part of all live- fire tests. Within one batch, manufactured small arms have a velocity variation leading to a 12 percent difference in deposited energy. Variability can also result from human handling or subjective software selection within the measurement systems. As discussed previously, such variability can result in different measurements for the same BFD. -102-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION NEED FOR A STAND-ALONE BFD ARTIFACT OR STANDARD MODEL FOR INTERORGANIZATION VERIFICATION An important issue that should be addressed is the importance of having a measurement standard to determine the ability of any given device (caliper, Faro, etc.) to precisely measure a representative BFD regardless of the organization, measurement instrument, software version, operator, and so forth. In the development of methods to measure BFDs, virtually no inter- laboratory testing has been carried out to date to determine sources of inter- laboratory errors as a consequence of test procedure differences or differences in the setup of the test equipment or of differences in the operation of equipment. Interlaboratory errors are often systematic, resulting in a constant statistical difference in BFD measurement from one measurement laboratory to the next. These measurement errors can lead to undue acceptance or rejection of lots of ceramic armor, which is undesirable. Interlaboratory errors can be rooted out by having several laboratories run the same test with the same or equivalent instrumentation. The source of the error can be identified and eliminated by a change in experimental procedure or equipment. A physical artifact would replicate a standard BFD cavity and perform a gauge block function for noncontact instruments. That said, the BFD standard artifact should be more than just a gauge block. Rather it should represent a physical model of the complete BFD measurement process. It should allow operators to follow a four-step process: 1. Measure a representative preshot surface; 2. Measure a representative postshot BFD crater; 3. Subtract the two numbers; and Compare the number from the previous step with the artifact’s standard 4. depth. The result would quickly determine if the device as used was sufficiently accurate for this application. For example, a complete artifact system could be made that mimics the preshot surface with a flap that covers a replicate BFD crater. Such a model could be made of hard plastic, or, a softer coating could be applied. While the thickness of the flap would affect the absolute readings, the relative readings between organizations and operators would not be affected. A single artifact system, upon acceptance by NIST and the testing community, would become the one national standard for quickly confirming a device’s precision and accuracy for measuring a BFD. Previous work by NIST has established the usefulness of such a standard (NIST, 2010). The committee supports turning this idea into a practical solution for the entire body armor testing community. In addition to the test standard just described, evaluation of interlaboratory test variation is important for establishing test reliability. Hence, interlaboratory tests should be run in order to establish the accuracy of a test as well as its precision. -103-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Recommendation 5-2: An organization such as the National Institute of Standards and Technology should develop a standard backface deformation artifact system and procedures to allow operators to ensure that different measurement devices at different locations are able to meet specified levels of accuracy and precision. CHARACTERISTICS OF A “BEST UTILITY” MEASURING INSTRUMENT Based on the preceding discussion of the instruments and procedures for measuring BFD, the committee developed criteria for a measuring device that would provide the “best utility” for the body armor testing applications. A best utility measurement device must meet the following criteria:  Meet or exceed precision and accuracy requirements for measuring BFD;  Achieve the lowest practical fixed and variable costs; and  Minimize human judgment and error.32 In addition, it would be advantageous if the instrument could also  Be versatile enough to measure indents behind both plates and helmets33 and  Be widely available and supportable here and abroad. An example of an instrument that will have potential in the future is being tested at the Army Research Laboratory.34 The Microscribe Model G2LX is a digital arm/mechanical scribe instrument that is being used by the Army Research Laboratory in research on the BFD cavities formed in the head forms used to test helmets. The G2LX, which costs approximately $8,500,35 has an advertised precision of 0.012 in. (0.3 mm). The system is connected to a computer that can capture measurements made by the operator based on a three-dimensional x, y, and z coordinate system. It also has an automated database that captures 32 Capturing measurement readings in an automated database is helpful. Expert testing operators who spoke with the committee agreed that manually capturing readings can lead to transposition and other errors. There are commercially available automated database interfaces for both contact and noncontact instruments. 33 See Chapter 7 for a description of the helmet testing process. The di fferences between armor plate testing and helmet testing are considerable, and all operators interviewed agreed that a laser-based measuring tool was generally preferred for helmet testing due to the complex curves of the head form, on which the helmet BFD measurements must be made. 34 Committee site visit to the ATC, August 31, 2010; Rob Kinzler, Army Research Laboratory, “Improvements in Helmet Measurement,” presentation, to the committee, October 13, 2010. 35 Source: http://www.3d-microscribe.com. -104-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION measurements made by the user. The user can activate a finger or foot switch to notify the computer to enter the current measurement into the database. During a demonstration to the committee, the time required to measure a clay indent appeared to be less than that required by a caliper since there is no need to set up a bridge gauge. The G2LX system is a basic one-point contact measurement system, which means it suffers from the same inability to compensate for offset as does the caliper.36 The system combines a fairly inexpensive robotic arm capability, similar to that of the Faro system, with an inexpensive hard-mount caliper scribe end. The MicroScribe system could be used for testing both body armor plate and helmet BFD measurements (although a finer grid pattern is needed for helmet testing) and could significantly reduce the offset error currently seen with the caliper, which uses one preshot measurement. MicroScribe offers a more sophisticated arm advertised to achieve a precision of 0.003 in. (0.0762 mm) on the upgraded MLX model; it costs approximately $22,000. The robotic arm can be outfitted with a laser scanner similar to that used on the Faro for an additional $15,000 or so. The MicroScribe system is just one example of instruments that are available in the commercial sector. The committee believes there are also others that have “best utility” characteristics and are readily available. Finding: The data available to the committee were not obtained through a formal gauge or “artifact standard” repeatability and reproducibility study by an independent agency. Thus, the committee can draw no quantitatively reliable conclusions about the precision and accuracy (potential biases) of the measurement systems it examined. Late in the course of the committee’s final deliberations as it prepared this report, it received additional test results data that had not been available to it earlier in the effort (see Appendix M). Considering all available data, the committee recognized (1) an insufficiency of sample sizes for all the data examined; (2) inconsistencies in the direction and magnitude of biases; and (3) presumed large differences in offset magnitudes between the data in Hosto and Miser (2008) and the more recent live-fire test experiences. The committee determined from its analysis of the available data that remedial procedures for properly designed evaluations are needed to determine the magnitudes of accuracy and precision of current or proposed instruments in the measurement of body armor BFD before definitive conclusions can be drawn regarding best utility. Recommendation 5-3: In anticipation of future test measurement requirements, the Office of the Director, Operational Test and Evaluation and/or the Army 36 The offset measurement problem could be overcome by having the operator enter several point measurements on the surface of the clay near the aim point prior to the test round being fired. The extent of the grid pattern (e.g., 3 × 3 vs. 4 × 4 grid) would depend on the accuracy of the BFD measurement that was needed. -105-

OCR for page 92
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION should charter an organization such as the National Institute of Standards and Technology to conduct an analysis of available candidate commercial instruments with inputs from vest users, manufacturers, testers, policy makers, and others. The goal is to identify one or more devices meeting the characteristics of “best utility” measuring instruments as defined in this study to the government, industry, and private testing labs. The list of best utility instruments should be shared with NIJ, international allies, and others, as appropriate, to promote measuring instrument standardization for body armor testing nationally and internationally. A formal gauge repeatability and reproducibility study is required to quantify accuracy and precision as inputs to the best utility analysis. REFERENCES DOT&E. 2010. Memo on Standardization of Hard Body Armor Testing, 27 April 2010, Director, Operational Test and Evaluation. Gryth, D., D. Rocksen, J. Persson, U. Arborelius, D. Drobin, J. Bursell, L. Olsson, and T. Kjellstrom. 2007. Severe Lung Contusion and Death After High-Velocity Behind-Armor Blunt Trauma: Relation to Protection Level. Military Medicine 172(10): 1110-1116. Hosto, J. and C. Miser. 2008. Quantum FARO Arm Laser Scanning Body Armor Back Face Deformation. Report No. 08-MS-25. Aberdeen, Md.: U.S. Army Aberdeen Test Center Warfighter Directorate, Applied Science Test Division, Materials and Standards Testing Team. McNeese, W., and R. Klein. 1991. Measurement systems, sampling, and process capability. Quality Engineering 4(1): 21-39. NIST (National Institutes of Standards and Technology). 2010. Dimensional Metrology Issues of Army Body Armor Testing. Gaithersburg, Md.: NIST. NRC (National Research Council). 2009. Phase I Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army: Letter Report. Washington, D.C.: National Academies Press. Prather, R., C. Swann, and C. Hawkins. 1977. Backface Signatures of Soft Body Armors and the Associated Trauma Effects. ARCSL-TR-77055. Aberdeen Proving Ground, Md.: U.S. Army Armament Research and Development Command Technology Center. Walton, S., A. Fournier, B. Gillich, J. Hosto, W. Boughers, C. Andres, C. Miser, J. Huber, and M. Swearingen. 2008. Summary Report of Laser Scanning Method Certification Study for Body Armor Backface Deformation Measurements. Aberdeen Proving Ground, Md.: Aberdeen Test Center. -106-