Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 274
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Appendix F Committee Responses to the Government Accountability Office Report A primary motivation for this study was the Government Accountability Office (GAO) report GAO-10-119 (GAO, 2009). In the report, the GAO recommended that “the Army should provide for an independent ballistics evaluation of the First Article Testing results,” that “the Army should assess the need to change its procedures based on the outcome of the independent experts’ review and document these and all other key decisions made to clarify or change the testing protocols,” and that “the Army provide for an independent external peer review of Aberdeen Test Center’s body armor testing protocol, facilities and instrumentation” (GAO, 2009, p. ii). The committee has addressed questions raised by the GAO in its report, and its responses to seven specific items that were recommended for evaluation and action (GAO, 2009, pp. 37-38.) are summarized here. The committee took its charge to conduct an independent assessment very seriously. The GAO report was thoughtful and pointed out a number of important issues. There was give and take between GAO, DoD, and the Army. On some issues, DoD and the Army agreed with GAO. On others, there was disagreement. The committee has listened to arguments on many sides of this debate. Its findings and recommendations are based on discussions not only with government experts but also with body armor manufacturers and commercial testers certified by the National Institute of Justice (NIJ). The committee was precluded from using classified data. As a result, specific first article testing data based on real-world threats were not available to it. However, it spent many hours reading background information and being briefed by experts throughout the body armor community and believes that its findings and recommendations inform most of the important issues raised in the GAO report. The committee’s overarching analysis of the report is that it was generally on target. Over time the Director, Operational Test and Evaluation (DOT&E) and the Army have adopted a number of the GAO recommendations. The committee has studied the report closely, discussed the issues with experts from throughout the body armor community, and agreed with several recommendations; it has taken a number of the issues to additional detailed levels with its findings and recommendations. -274-

OCR for page 274
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION SUMMARY OF RESPONSES TO EVALUATION ITEMS AND ACTION ITEMS RECOMMENDED FOR INDEPENDENT ASSESSMENT Evaluation Item 1: The rounding of back-face deformation measurements The Phase I report (NRC, 2009, p. 17) suggested that the DOT&E and the Army adopt a common standard for rounding and indicating the appropriate number of significant digits. The specific observation was that the Army should consider using the American Society of Testing and Materials (ASTM) standard ASTM E29-08, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications (ASTM, 2008). During briefings of the Phase I report, both DOT&E and the Army said they agreed with the recommendation. Evaluation Item 2: Not scoring penetrations of material through the plate as a compete penetration unless broken fibers are observed in the Kevlar backing behind the plate The Phase I committee was briefed by the COL Jeffrey Holt, Commander of the Aberdeen Test Center (ATC). He said that the command wanted a definition for a “complete penetration” that minimized subjectivity on part of the tester and manufacturer. As a result, ATC adopted a convention that any test round that penetrated the armor plate and continued on to completely break any Kevlar fiber on the back of the shoot pack would be considered a complete penetration. The committee felt this provided a consistent description for the body armor community and virtually eliminated subjectivity when scoring a plate as having been completely penetrated. As defined in the new DOT&E protocol, “a complete penetration of the test plate sample occurs on any fair record test shot impact in which the projectile, any fragment of the projectile, or any fragment of the armor material is ejected from the rear of the plate and passes into the first ply (minimum of one complete yarn broken) of the soft armor (ballistic package) located behind the test plate sample when it is placed into the soft armor test panel. The first ply of the soft armor (ballistic package) shall serve as a witness plate.” (DOT&E, 2010, p. 4) Evaluation Item 3: The use of the laser scanner to measure back-face deformations without a full evaluation of its accuracy as it was actually used during testing, to include the use of software modifications and operation under actual test conditions The Phase I committee was provided a copy of ATC Internal Operating Procedure No. 001: Measurement of Backface Deformation (BFD) Using Faro® Quantum Laser Scan Arm and Geomagic® Qualify® for Hard and Soft Body -275-

OCR for page 274
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Armor, and found that it adequately describes the appropriate and consistent use of the laser scanning system (Huber, 2009; NRC, 2009). Additionally, the Phase I report stated that the “laser scanning system (including the testing protocols, facilities and instrumentation) as currently implemented by the Army (or for similar equipment) if used in accordance with the Army’s procedures, is a valid approach for determining the contours of an indent in a nontransparent clay material at a level of precision adequate for the Army’s current ballistic testing of body armor.” (NRC, 2009, p. 13) During Phase III the committee looked for improvements to procedures that could result in more consistent measurements made with laser scanners. There are several findings in the Phase III report that if implemented by DOT&E and the Army, could improve consistency in laser scanning. For example, there is software that allows for smoothing the raw digital data captured by the Faro. The software has settings that allow for various levels of resolution. The committee was shown that just by switching from one level of resolution to another resulted in a 1 mm difference in measurement for the same backface deformation (BFD). Manufacturers can be burdened, in this example, by a 1 mm penalty simply because an operator has selected a particular setting. The committee finds that a control study should be conducted to determine the most reasonable and consistent Faro smoothing settings to be used while measuring BFDs in body armor testing. Similarly, any software selections or system changes that could cause a relevant change to BFD measurements should be studied. Participation in this study should include, at a minimum, ATC, NIJ-certified private testing labs, and body armor manufacturers. Another finding in the Phase III report that relates to this issue is that there needs to be a single measurement standard to determine the ability of any system to measure a representative BFD regardless of different labs, measurement instruments, and operators. Specifically, a standard model or a BFD artifact (as initially recommended by the National Institute of Standards and Technology) should be developed and used as part of this analysis. Previous work in this area by ATC and by the National Institute of Standards and Technology had established the importance of using an artifact to mimic the measurement process. The committee believes that because the BFD measurement process is a two-step process (measurement of the preshot surface and measurement of the postshot BFD), both of these steps should be made with a single national standard artifact. It is suggested that an artifact be made that mimics the preshot surface with a flap that covers a multiple-BFD imprinted plate, or a flap on a helmet surface that covers a model BFD crater. Such a model could be made of hard plastic, or, a softer coating could be applied. While the thickness of the flap would affect the absolute readings, the relative readings between labs and operators would not be affected. This artifact would be used to confirm that any change (e.g., hardware, software, or operator) still resulted in a consistent measurement. -276-

OCR for page 274
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Evaluation Item 4: The exposure of the clay backing material to rain and other outside environmental conditions as well as the effect of high oven temperatures during storage and conditioning ATC demonstrated to the committee that the ovens used to condition the clay prior to testing had been moved inside the testing range building to preclude exposure of clay to rain and other outside weather. The issue of high oven temperatures is more complicated. Back in the late 1970s, Roma Plastilina #1, a modeling clay developed for artists, calibrated at room temperature when used for testing applications. Subsequently, the composition of the clay changed over time to meet artists’ needs. As the composition slowly changed, experimentation has shown the only way that the calibration standard (25 mm ± 3 mm) could be achieved was by heating the clay. As a rule of thumb, experienced test operators at NIJ -certified commercial testing laboratories, who have decades of experience with clay, reported to the committee that the clay has had to be heated about an additional 1°F each year to achieve calibration standards. Currently, there is an informal experimental effort at both ATC and the commercial labs to attempt to restrict the temperature of the clay boxes to about 105°F.68 (Interestingly, for the Phase I body armor testing demonstrations at ATC in December 2009, the temperature goal was approximately 104°F.) However, if the clay formulation continues to change to meet artist expectations, it is likely that the temperature of the clay will have to be increased or decreased to allow the clay to properly calibrate. Accordingly, the committee found that DOT&E and the Army should urgently develop a standard formulation for ballistics backing material that allows calibration at room temperature, as was the case when body armor testing using clay began. Additionally, as a medium-term goal, the committee found that DOT&E and the Army should research an alternative to the current Roma Plastalina #1 that not only calibrates at room temperature but also minimizes the effect of human working (thixotropy). By standardizing a ballistics backing material formulation that minimizes the variation caused by temperature and thixotropy the committee feels that the body armor community will have eliminated a significant amount of the variation in the body armor testing process. Historically (and incorrectly) all variation in the testing process has been assumed to be due to body armor plates inconsistencies. The practical result of a standard ballistics backing material formulation as just described could be the production of lighter body armor that still provides the same level of soldier protection. 68 Confirmed during personal communication between Larry Lehowicz, Committee chair, and Travis Humiston, ATC, October 28, 2010. -277-

OCR for page 274
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Evaluation Item 5: The use of an additional series of clay calibration drops when the first series of clay calibration drops does not pass required specifications ATC test operators demonstrated the clay calibration process during each phase of the study. Concerning the actions to repair and recondition clay boxes, which include using additional clay calibration drops when the first series of clay calibration drops did not pass specifications, the Phase I committee agreed that test operators were following “procedures that are consistent with standard practice by artists and others for filling space without entrapping air. That is, small additions are made sequentially and each is heavily sheared by hand to express any entrapped air. This procedure represents good practice.” (NRC, 2009, p. 15) Rheologists on the committee appreciate that there are many variables and unknowns in clay calibration that need to be investigated by DOT&E and the Army. The Phase II report (NRC, 2010) contains several findings and recommendations pertaining to calibration and the drop test. Some of the committee’s calibration-related findings are these:  Until a standard ballistics backing material that calibrates at room temperature is developed, the body armor testing community should conduct a posttest calibration drop to ensure that the clay has remained in calibration.  Evaluate measurement differences when calibrations are made on the side of the box and when they are made at the center of the box.  Conduct calibration experiments with a gas gun to better replicate the projectile velocity and penetration depth that create a BFD.  Ensure pre- and postshot calibration consistency.  Investigate the effects of box size and shape on calibration results. Action Item 1: Determine whether those practices that deviated from established testing protocols during First Article Testing will be continued during future testing and change the established testing protocols to reflect those revised practices The committee chair was told by senior Army staff members that the Army would appoint a knowledgeable person to conduct an independent assessment of the practices mentioned in the GAO report. -278-

OCR for page 274
PREPUBLICATION DRAFT—SUBJECT TO EDITORIAL CORRECTION Action Item 2: Evaluate and re-certify the accuracy of the laser scanner to the correct standard with all software modifications incorporated and include in this analysis a side-by-side comparison of the laser measurements of actual back-face deformations with those taken by digital caliper to determine whether laser measurements can meet the standard of the testing protocols The committee spent considerable time with ATC and NIJ-certified commercial testing labs to better understand the advantages and disadvantages of the digital caliper and the laser scanner. See Chapter 5 and Appendix M. REFERENCES ASTM (American Society for Testing and Materials). 2008. ASTM E29-08 Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications. West Conshohocken, Pa.: ASTM International. DOT&E (Director, Operational Test and Evaluation). 2010. Standardization of Hard Body Armor Testing. Memorandum dated April 27, 2010. Arlington, Va.: Director, Operational Test and Evaluation. GAO (United States Government Accountability Office). 2009. GAO-10-119. Independent Expert Assessment of Army Body Armor Test Results and Procedures Needed Before Fielding. Washington, D.C.: Government Accountability Office. Huber, J. 2009. Internal Operating Procedure No. 001: Measurement of Backface Deformation [BFD] Using Faro® Quantum Laser Scan Arm and Geomagic® Qualify® for Hard and Soft Body Armor. Aberdeen Proving Ground, Md.: Aberdeen Test Center. NRC (National Research Council). 2009. Phase I Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army: Letter Report. Washington, D.C.: National Academies Press. NRC. 2010. Phase II Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army. Washington, D.C.: National Academies Press. -279-