THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

Division on Engineering and Physical Sciences

Board on Army Science and Technology Mailing Address:

500 Fifth Street, NW Washington, DC 20001 www.nationalacademies.org

April 22, 2010

J. Michael Gilmore

Director,

Operational Test and Evaluation Department of Defense

1700 Defense Pentagon Washington, DC 20301-1700

RE: Phase II Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army

Dear Dr. Gilmore:

At your request, the National Research Council (NRC) of the National Academies established the Committee to Review the Testing of Body Armor Materials for Use by the U.S. Army to assess the methodologies used for body armor testing. The committee provided its Phase I report to you on January 4, 2010. What follows is the evaluation developed in satisfaction of the Phase II component of the statement of task (see Attachment A):

In Phase II, the committee will consider in greater detail [than in Phase I] the validity of using the column drop performance test described by the Army for assessing the part-to-part consistency of a clay body within the level of precision that is identified by the Army test procedures.


The committee will prepare a letter report documenting the findings from its Phase II considerations.

This Phase II report is focused on the behavior of ballistic clay and on other issues relating to the test process that were raised in Phase I of the study. More detailed evaluations of the array of issues surrounding body armor testing, both present and future, will be presented in the final Phase III report.

The recommendations in this letter report are based on the information that the committee received from the Army and on discussions and observations during a single 4-day meeting that included a site visit to the Aberdeen Test Center (ATC) at the Aberdeen Proving Ground, Maryland. At this meeting, the committee received briefings on specific issues raised in Phase I and by the Phase II statement of task that were of interest to the sponsor. The committee reviewed documentation on the Army’s body armor testing program in general and on its tasks for Phase II in particular.

During the site visit, the committee members observed how ATC tests body armor using consistent methodologies for the handling and calibration of the clay (the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Division on Engineering and Physical Sciences Board on Army Science and Technology Mailing Address: 500 Fifth Street, NW Washington, DC 20001 www.nationalacademies.org April 22, 2010 J. Michael Gilmore Director, Operational Test and Evaluation Department of Defense 1700 Defense Pentagon Washington, DC 20301-1700 RE: Phase II Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army Dear Dr. Gilmore: At your request, the National Research Council (NRC) of the National Academies established the Committee to Review the Testing of Body Armor Materials for Use by the U.S. Army to assess the methodologies used for body armor testing. The committee provided its Phase I report to you on January 4, 2010. What follows is the evaluation developed in satisfaction of the Phase II component of the statement of task (see Attachment A): In Phase II, the committee will consider in greater detail [than in Phase I] the validity of using the column drop performance test described by the Army for assessing the part-to-part consistency of a clay body within the level of precision that is identified by the Army test procedures. The committee will prepare a letter report documenting the findings from its Phase II considerations. This Phase II report is focused on the behavior of ballistic clay and on other issues relating to the test process that were raised in Phase I of the study. More detailed evaluations of the array of issues surrounding body armor testing, both present and future, will be presented in the final Phase III report. The recommendations in this letter report are based on the information that the committee received from the Army and on discussions and observations during a single 4-day meeting that included a site visit to the Aberdeen Test Center (ATC) at the Aberdeen Proving Ground, Maryland. At this meeting, the committee received briefings on specific issues raised in Phase I and by the Phase II statement of task that were of interest to the sponsor. The committee reviewed documentation on the Army’s body armor testing program in general and on its tasks for Phase II in particular. During the site visit, the committee members observed how ATC tests body armor using consistent methodologies for the handling and calibration of the clay (the

OCR for page 1
Phase II Report on Review of the Testing of Body 2 Armor Materials for Use by the U.S. Army column drop performance test) and for the measurement of the backface deformation, including procedures for assessing the part-to-part consistency of the clay. In addition, the committee reviewed the statistical basis for the testing and analyzed proposed revisions to the statistical protocols used. The Phase II Committee was greatly appreciative of the dedication, qualifications, and openness of the ATC staff. Clearly they seek to achieve the highest standards possible for armor testing and are pursuing refinements in established techniques and advances in technology to provide the very best armor performance for our soldiers. As described in the pages that follow, adequate resources are required to achieve such a goal. The committee’s analysis of the Phase II issues resulted in the development of 19 recommendations that are summarized in Box S-1 on page 3. These actions are urgently needed to achieve greater part-to-part consistency in the ballistic clay, to analyze BFD dynamics, to determine possible replacements for modeling clay, to achieve a national clay standard for testing body armor, and to implement statistically based protocols. The overarching recommendation is as follows: Overarching Recommendation: The committee applauds DOT&E for assuming a national-level leadership role in bringing the body armor test community together. The committee recommends that the DOT&E (1) work with Congress, DoD, the military services, and other organizations to find the resources necessary to implement the recommendations described in this report and summarized in Box 1 and (2) oversee, review, track, and assist the designated action organizations with implementing these recommendations. This approach should result in more consistent test results that will provide equally survivable but lighter-weight body armor to our military service members and civilian police forces. Sincerely, MG (ret.) Larry G. Lehowicz, Chair Committee to Review the Testing of Body Armor Materials for Use by the U.S. Army Attachments A Statement of Task B Committee to Review the Testing of Body Armor Materials for Use by the U.S. Army – Phase II C Acknowledgment of Reviewers D Acronyms

OCR for page 1
Phase II Report on Review of the Testing of Body 3 Armor Materials for Use by the U.S. Army Phase II Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army The committee’s analysis of the Phase II issues resulted in the development of 19 recommendations that are summarized in Box S-1 and discussed in detail in the following sections of the report. Box S-1 Phase II Recommendations to Improve Body Armor Testing Achieving Greater Part-to-Part Consistency in Clay 1. Quantify the Medical Results of Blunt Force Trauma on Tissue and Incorporate Results into the BFD Methodology 2. Determine Short-Term Standard Clay Specification 3. Conduct Rheological and Thermogravimetric Measurements 4. Procure and Experiment with a Clay Compounding Machine 5. Examine Technologies for “In Box” Mechanical Clay Working 6. Modify TOP 10-2-210 Procedures to Add a Post-calibration Drop (ATC, 2008) 7. Experiment with Various Clay Box Sizes and Shapes 8. Develop and Experiment with a Gas Gun Calibrator or Equivalent Device Analyzing Backface Deformation Dynamics 9. Analyze the Signal-to-Noise of Flash X-Ray Cineradiography 10. Experiment with Microscopic Temperature and Displacement Sensors in Clay 11. Experiment with the High-Speed Photographic Analysis of BFD Creation in Ballistic Gelatin Determining Possible Replacements for Modeling Clay 12. Study Ballistic Gelatin as a Mid-Term Alternative to Modeling Clay 13. Study Microcrystalline Waxes as a Long-Term Alternative to Modeling Clay or Ballistic Gelatin. Achieving a Single National Clay Standard for Body Armor Testing 14. Empower and Resource the Ad Hoc Clay Working Group 15. Convene a Nationally Recognized Group to Establish a Single National Standard for Handling and Validating Clay Implementing Statistically Based Protocols 16. Compare the Proposed Statistically Based Protocol with the Existing USSOCOM Protocol 17. Quantify the Variation in the Body Armor Test Process and Incorporate in the Protocol 18. Develop a Statistically Based LAT Protocol 19. Conduct Due Diligence Before Implementing and Formally Adopting a Set of Statistically Based Protocols

OCR for page 1
Phase II Report on Review of the Testing of Body 4 Armor Materials for Use by the U.S. Army INTRODUCTION This section describes the expertise of the Phase II Committee membership, the ceramic armor plates being tested, the testing process, the layout of the testing range, and relationships between medical studies and use of modeling clay in body armor testing. Phase II Committee Expertise At the conclusion of Phase I, the Phase I Committee felt that greater consistency in the oil-based modeling clay could reduce variability in the body armor test and give more consistent and precise results. More precise results, in turn, could allow certifying with a high degree of confidence lighter weight armor plates that achieve the same survivability for a soldier. As a consequence, the membership of the Phase II Committee included additional experts on clay, who could address the statement of task requirement to “assess the part-to-part consistency of clay” in more detail. The sponsor appreciated the Phase I Committee’s support for the development of a statistically based protocol to determine test sample sizes and other aspects of testing. The Phase II Committee was asked to continue that work in Phase II, and an additional statistician was appointed to the Phase II Committee. Ceramic Armor Ceramic materials have been used successfully in personal armor systems to defeat small-arms threats. They are preferred for personal armor systems because they are lighter than more traditional armor made of metallic alloys. Properties that contribute to the performance of ceramic armor include superior hardness, low density, favorable elastic constants, and high compressive strength. However, as stand-alone items, ceramics would not be particularly good because of their low tensile strength, brittle response, and sensitivity to small mechanical defects such as pores and cracks. Hence, ceramics are used in combination with other materials, such as polymers and metals, to form laminar composites that provide excellent properties for body protection. A typical insert (also referred to as a “plate”) of body armor consists of a layer of dense boron carbide or silicon carbide backed by a layer of metal or polymer composite; The entire plate is wrapped in tightly woven ballistic fabric. The ceramic layer breaks up an incoming projectile and dissipates its kinetic energy. The layer of polymer composite and/or metallic alloy provides ductility and structural integrity and spreads the forces resulting from the impact of a projectile over a larger area. All hard body armor systems currently add a significant burden of weight on the soldier. Armor testing therefore has implicit goals of ensuring that body armor meets survivability standards while allowing sufficient soldier mobility and flexibility. To provide soldiers with more weight than necessary to defeat a specified threat can lead to unintended consequences such as premature exhaustion and restricted ability to rapidly move and react in life threatening situations.

OCR for page 1
Phase II Report on Review of the Testing of Body 5 Armor Materials for Use by the U.S. Army Current Army Body Armor Testing Process As described in the Phase I report (NRC, 2009), the Army’s procedures for testing hard body armor using a clay backing for the measurement of deformations in the clay from ballistic impacts are documented in “Test Operations Procedure (TOP) 10-2-210: Ballistic Testing of Hard Body Armor Using Clay Backing,” dated October 1, 2008 (ATC, 2008). The approach may be summarized in four paragraphs: A clay box 1 and clay chest plate appliqué 2 (Figure 1) are assembled, appropriately calibrated for part-to-part consistency using the column-drop performance test, and placed upright in the test holder. Independently, a “shoot pack” is prepared. To create a shoot pack, the armor plate is placed in a fabric envelope together with multiple layers of Kevlar to replicate the vest worn by the soldier. The dimensions of the armor plate depend on the size of the vest and can range from 18 cm × 29 cm to 28 cm × 36 cm, with a thickness of approximately 2 cm. The vest has a significant nonconstant radius of curvature. Once assembled, the shoot pack is pressed firmly into the surface of the appliqué to ensure conformance. The shoot pack is then removed and the laser scanning system scans the surface of the appliqué to provide a reference surface relative to which subsequent deformations caused by the firing of the projectiles can be compared. The laser scanning system is moved out of the way, the shoot pack is repositioned onto the surface of the clay, with care taken not to disturb the reference surface, and the shoot pack is secured. The projectile being tested is then fired into the shoot pack, after which the shoot pack is removed from the clay and inspected for penetration. The displacement or indent in the clay made by the deformation of the armor is thereby exposed. The velocity of the projectile was measured using Oehler Model 57 Ballistic Screens to verify that it was within the desired range. 1 A plywood-backed aluminum frame (~61 cm × 61cm × 14 cm) filled with modeling clay is subsequently referred to in this report as a “clay box” or as a “part” when discussing part-to-part variations. 2 The appliqué is an additional layer of clay that has been molded to the shape of the specific armor plate to be tested.

OCR for page 1
Phase II Report on Review of the Testing of Body 6 Armor Materials for Use by the U.S. Army FIGURE 1 The clay appliqué applied to the clay box. SOURCE: Richard Sayre, Deputy Director, and Tracy Sheppard, Executive Officer and Staff Specialist, Office of the Secretary of Defense, Director of Operational Test and Evaluation (OSD DOT&E) Live Fire Test and Evaluation, “DoD in brief to the National Research Council study team,” Presentation to the committee, on November 30, 2009. The deformation is measured with the laser scanning system. The data are collected and used to compute the profile (depth distribution) indent. The deformation is analyzed and serves as an indication of the survivability of a soldier subjected to a similar shot and protected by a similar plate in a protective vest. 3 A representative deformation is shown in Figure 2. The nominal design specification is that the maximum depth in the clay relative to the original surface be less than 43 mm. That is, a backface deformation (BFD) with a maximum depth of less than 43 mm is considered to indicate acceptable performance of body armor in service. Experimental data collected by the Army indicate that under nominally identical conditions the standard 3 As shown in the Prather et al. (1977) study, there is a correlation between the depths of penetration as a function of time into various media, including the modeling clay Roma Plastilina #1, and the probability of lethality when the same penetrator enters a human surrogate (goat) (Prather et al., 1977). (The study did not address volume of the indentation.)

OCR for page 1
Phase II Report on Review of the Testing of Body 7 Armor Materials for Use by the U.S. Army deviation for the maximum depth of the BFD (hard armor) is in the range of 2.5 to 4 mm. 4 The BFD measurements in combination with the penetration data are used to evaluate the armor. ≤43mm Aim point / line of sight depth Deepest point depth Undisturbed surface curvature FIGURE 2 Surface of the BFD as measured by a laser scanning system. SOURCE: Richard Sayre, Deputy Director and Tracy Sheppard, Executive Officer and Staff Specialist, Office of the Secretary of Defense, Director of Operational Test and Evaluation (OSD DOT&E) Live Fire Test and Evaluation, “DoD in brief to the National Research Council study team,” Presentation to the committee, on November 30, 2009. 4 James Zheng, Chief Scientist, Program Executive Office–Soldier, “Ballistic protection for warfighters,” Presentation to the committee, on November 30, 2009.

OCR for page 1
Phase II Report on Review of the Testing of Body 8 Armor Materials for Use by the U.S. Army Body Armor Testing Range A typical firing range used to test body armor consists of: a rifle-like device to fire a projectile; an instrument to measure the velocity of the projectile; the armor plate being tested, which is affixed to an oil-based clay backing of modeling (clay this backing becomes indicated in response to the kinetic forces created on the plate); and a laser system to measure the geometry of the indentation in the clay. A photograph of an indoor range set up for testing body armor at the Aberdeen Test Center (ATC) is shown in Figure 3. FIGURE 3 The body armor test range at ATC. SOURCE: John Wallace, Technical Director, ATC, “Body armor test capabilities,” Presentation to the committee, on March 10, 2010. The highest priority of the Phase I report was to examine the validity of the laser profilometry system to determine the contours of the indentation in the oil-based modeling clay, the BFD, at the level of precision established by Army procedures. The committee found the Army’s laser system 5 used in accordance with its ATC Internal Operating Procedure No. 001 provides a valid approach for measuring the BFD indentation at the appropriate level of precision. The Phase I report also asked the committee to address the oil-based modeling clay medium in which the BFD is formed. Specifically, the committee was asked to provide interim observations on the Army’s column-drop performance test used to 5 Faro® Quantum Laser Scan Arm and Geomagic® Qualify® for Hard and Soft Body Armor.

OCR for page 1
Phase II Report on Review of the Testing of Body 9 Armor Materials for Use by the U.S. Army determine that the clay-filled boxes in the test are consistent from box to box. This is referred to as “part-to-part consistency.” The committee’s Phase I report found that the column-drop performance test (including testing protocols, facilities, and digital caliper instrumentation) is a valid method for assessing the part-to-part consistency of the clay boxes used for testing body armor. Medical Study Basis for Use of Modeling Clay in Testing Body Armor The use of clay as a recording medium for body armor testing dates from a 1977 study that correlated the depth that a 200-g, 80-mm hemispherical missile, impacting at approximately 55 m per second (Prather et al., 1977), penetrated live animal tissue and other media. The goal of the study was to develop a simple, readily available backing material for characterizing both the penetration and deformation effects of ballistic impacts on body armor materials and to relate this information to the injury potential of nonpenetrating ballistic impacts. The depth of penetration into various media as a function of time was compared to the probability of lethality for the same penetrator entered into a live animal model (in this study goats were used as models) (Clare et al., 1975). The study observed strong correlations between lethality probability and penetration into ballistic gelatin 6 and also into modeling clay Roma Plastilina #1. The ballistic gel required the use of high-speed photography to record BFDs, because the gel was elastic and returned to its original shape after the projectile firing. To avoid the necessity of using expensive high-speed photography, an alternative material was sought that would retain its deformation The first conclusion of the Prather et al. (1977) report had a profound effect on testing over the next 30 years. It reads as follows (Prather et al., 1977, p. 11): A readily available, easy-to-use backing material, Roma Plastilina 1, has been found which can be correlated to tissue response for use in characterizing both the penetration and deformation effects of ballistic impacts on soft body armor materials. Roma Plastilina #1 has since been adopted as a recording medium to assess the likelihood of injury or death from ballistics, and its use has been extended from assessing soft armor such as Kevlar vests to assessing hard armor plates, knife wounds, industrial injuries to a drop-forge operator, and nonlethal projectiles (Lyon, 1997; Chadwick et al., 1999; O’Callaghan et al., 2001; Vaughan, 2001; and Karahan, 2008). Roma Plastilina #1 appears to have become an industry standard despite being an imperfect simulant of the human body. The procedures for the use of this clay have evolved with time. In part, this is because the behavior of the material has changed over time. The manufacturer confirmed 6 Ballistic gelatin is a clear or yellowish gelatin that is the standard medium for seeing and evaluating what happens to bullets on impact with soft tissue.

OCR for page 1
Phase II Report on Review of the Testing of Body 10 Armor Materials for Use by the U.S. Army that the formulation sold as Roma Plastilina #1 has changed as the sources of raw material have changed and in response to the needs of artists. Modeling clay provides an approximation of the actual BFD. It does not record maximum displacement since the clay may exhibit some elastic recovery, nor does it record the rate of deformation. Both of these dynamic events may be important in predicting the magnitude of injury to a person. To address these issues DoD conducted a medical research program, the Body Armor Blunt Trauma Assessment (BABTA) Project, from 2002 to 2006. 7 The BABTA project developed an anthropomorphic test module (ATM) onto which body armor plates could be placed and firing tests performed. The ATM was equipped with sensors that directly measured the spatial and temporal distribution of the forces and motions caused by a bullet impacting armor. The pressure sensors in the ATM are on the backside of an approximately 25 mm thick layer of Dragon Skin, a high- performance silicone polymer that simulates the mechanical properties of the body. The layer of Dragon Skin is necessary to avoid damaging the sensors during the test. The spatial and temporal distribution of the forces at the surface was inferred using a finite- element calculation. Based on this, a blunt projectile was developed that when fired from an air-gun impactor at the ATM without an armor plate resulted in a distribution of forces and motions believed to accurately simulate those produced by the bullet impacting armor, assuming that the armor was not completely penetrated. The blunt projectile was a ~60 mm hemisphere mounted on the end of a smaller cylinder. Using this blunt projectile impactor, the project team experimented with highly instrumented and anesthetized human surrogates (in this case, pigs). From the response of the human surrogates, including in some cases postmortem analysis, a large database was generated that related the temporal and spatial distribution of forces to the injury. The BABTA project and numerous others indicate that depth of indentation alone is an inadequate indicator of injury probability (Cannon, 2001; Bass et al., 2006). The BABTA study suggests that a means of easily and economically measuring the temporal and spatial distribution of the forces during the BFD would enable a more accurate assessment of armor efficacy and could lead to lighter armor that provides equal protection. The BABTA project established the need to develop testing methodologies that determine not just the BFD but also the dynamic forces that result from the impact of the projectile on body armor. During presentations to the committee on March 9, 2010, Dr. Prather and Dr. Legierri agreed that the initial conclusions on BFD in the 1977 Prather study are very conservative. That is, humans may be capable of more easily surviving forces that correspond to those that make a BFD larger than the 43 mm BFD that is currently accepted by the body armor community. The committee applauds the efforts of the medical and testing communities to better quantify and correlate laboratory-generated mechanical impacts with the blunt force trauma caused in surrogate human beings. During Phase III the committee hopes to be able to further investigate (1) the relationships between temporal and spatial forces that cause blunt force trauma in the laboratory and injuries experienced by soldiers on the battlefield and (2) ways to more 7 Michael Leggieri, Director, DoD Blast Injury Research Program Coordinating Office, U.S. Army Medical Research and Materiel Command, “DoD medical research perspective on the clay-based body armor performance testing methodology,” Presentation to the committee, on March 9, 2010.

OCR for page 1
Phase II Report on Review of the Testing of Body 11 Armor Materials for Use by the U.S. Army accurately correlate blunt force trauma impacts experienced by soldiers with the signatures that similar forces cause in body armor media that are not restricted to oil- based modeling clay or similar approaches. Recommendation 1: The Army’s medical and testing communities should be adequately funded to expedite the research necessary both to quantify the medical results of blunt force trauma on tissue and to use those results as the updated mathematical underpinnings of the back face deformation (BFD) body armor testing methodology. Regardless of the current imperfect correlation between existing medical data and the BFD approach, the committee believes that the current methodology for testing body armor should be continued, mainly because this approach has allowed the Army to send body armor with adequate survivability characteristics to soldiers in combat. Importantly, the committee was informed earlier by the Program Executive Office–Soldier that no soldier deaths are known to be attributable to a failure of the issued ceramic body armor. 8 , 9 , 10 The committee agrees with a number of briefers that additional study in these areas could lead to insights that that current body armor, which provides an adequate level of survivability, may be unnecessarily heavy for a given threat. CLAY PROPERTIES AND TESTING METHODOLOGY This section provides brief descriptions of clay properties and behavior, clay in testing methodology, and short-term development of a standard clay formulation. It also discusses the procedures for preparing and working clay, calibrating clay, and analyzing BFD dynamics. It concludes with recommendations on possible mid-term and long-term replacements for modeling clay in the testing of body armor. 8 Question-and-answer session between Debi Dawson, Director, Strategic Communications, Program Executive Office–Soldier, and the Body Armor Phase I committee, December 1, 2009. 9 Personal communication between LTC Jon Rickey, Product Manager (PM), Personnel Survivability Equipment, Program Executive Office–Soldier, and Larry G. Lehowicz, Chair, December 21, 2009. 10 Personal communication between James Zheng, Chief Scientist, Program Executive Office– Soldier, and Larry Lehowicz, Chair, December 29, 2009. According to LTC Rickey and Dr. Zheng, in no case has it been determined that an issued enhanced small arms protective insert (ESAPI) or enhanced side ballistic insert (ESBI) armor plate failed to prevent an armor piercing (AP) by small arms projectiles of 7.62 mm × 63 mm or less. However, in some instances a casualty may become separated from issued body armor. In these situations it may not be possible to track the armor back to the original casualty. As a result the Army chose the word “known” to qualify the statement “no known deaths.” For a nonmilitary, nonexpert audience it is noted that soldiers wearing body armor may suffer casualties when their ceramic armor is defeated by rounds of caliber larger than 7.62 mm × 63 mm when projectiles or shrapnel strike a portion of the body not protected by body armor, when the blast comes from improvised explosive devices (IEDs) or other explosives, and so forth.

OCR for page 1
Phase II Report on Review of the Testing of Body 37 Armor Materials for Use by the U.S. Army (i.e., of no penetration) of 0.98 stands an 11.9 percent chance that at least three or more of the 60 plates will fail the test, so that manufacturer will fail the test. Conversely, under these test conditions, the government has a 5 percent chance that a manufacturer's marginally-performing plates that have a probability of no penetration of 0.90 will pass the test. As the table shows, for a sample size of 60, a manufacturer must produce hard body armor that has a true probability of no penetration substantially higher than 0.9 to have a reasonable chance of passing the test. From a soldier-safety perspective, this is appropriate. This kind of analysis can be helpful for ensuring that the test design does not lead to overdesign. The first three lines of the Table 4 demonstrate that reducing the sample size from 60 shifts the risk to the manufacturer. For a sample size of 15 it is not possible to pass the test because the sample size is too small to demonstrate a 90 percent requirement with high (90 percent) confidence. The last two lines of Table 4 show the sharp increases in required sample size when the requirement is increased beyond 0.9 and the risks are held roughly constant. TABLE 4 Risk Comparisons for Probability of Complete Penetration Allowable True P (No Government Manufacturer Sample Size Failures Penetration) Requirement Risk Risk 15 0 0.98 0.86 0.206 0.261 22 0 0.98 0.90 0.098 0.359 40 1 0.98 0.90 0.080 0.190 60 2 0.98 0.90 0.053 0.119 60 2 0.99 0.90 0.053 0.022 60 2 0.92 0.90 0.053 0.868 300 9 0.98 0.95 0.000 0.082 6,000 134 0.98 0.975 0.000 0.092 Recommendation 16: Before adopting the proposed statistically based protocol, the Department of Defense Director, Operational Test & Evaluation, (DOT&E) should explicitly compare the risks of the proposed protocol and those of the existing Army and U.S. Special Operations Command (USSOCOM) protocols, in order to establish which test plan increases soldier safety while balancing the manufacturer’s risk and incentives to overdesign. The committee notes that the USSOCOM first article test protocol may not be intended as a comprehensive technical test, and clarifying this issue would also help in the comparison of the protocols. Because of the issues discussed in earlier sections of this report, it is difficult to tell if the observed variation in BFD for hard body armor is attributable mainly to the variation in plates, to the variation in the test process, or to both. As a result, all observed variation is being attributed to the plates. While this is clearly incorrect, without a better understanding of the specific sources of variation, it is impossible to do otherwise. This probably results in overdesign and/or overmanufacture of the plate to ensure a high probability of passing FAT and LAT.

OCR for page 1
Phase II Report on Review of the Testing of Body 38 Armor Materials for Use by the U.S. Army If manufacturer’s average BFD = If manufacturer’s average BFD = 38 mm, probability of failing 40 mm, probability of failing testing for various process and testing for various process and testing standard deviations testing standard deviations 1.0 1.0 100% variance 90% variance 50% variance 0.8 0.8 Probability of failing test Probability of failing test 10% variance 0.6 0.6 0.4 0.4 100% variance 90% variance 50% variance a) b) 10% variance 0.2 0.2 0.0 0.0 3.0 3.5 3.0 4.5 5.0 3.0 3.5 3.0 4.5 5.0 Standard deviation of hard body armor Standard deviation of hard body armor FIGURE 6 Risk comparisons for BFD assume that the manufacturer’s true mean BFD is (a) 38 mm and (b) is 40 mm. The associated fraction of variation is shown on the x-axis. The plots show that decreasing variability in BFD, by means of a more consistent manufacturing process or more repeatable testing measures, reduces the manufacturer’s chance of failing testing (given that the manufacturer’s plates do meet standards and holding everything else constant). Figure 6 illustrates the potential impact on manufacturers by simulating the effects of the BFD test on the probability of a manufacturer failing FAT under various conditions. In Figure 6 (a), the assumption is that the plates resulting from a manufacturer's process have a mean BFD of 38 mm. The solid line (100% variance) shows the results when all observed variation is attributed to the plates. The amount of variation is shown on the x-axis in terms of standard deviations, and the probability of failing to meet the BFD criterion is shown on the y-axis. The plot shows that the probability of failure ranges from zero for standard deviations just above 3 to nearly 1 for standard deviations just less than 5. The dashed curves show the impact of attributing less of the observed variation to the plates. Notice that the percentage attributed to the plates decreases as the probability of passing the test increases. Figure 6(b) shows a similar result for a mean BFD of 40 mm. The plots show that decreasing variability in BFD, by means of a more consistent manufacturing processes or as a result of more repeatable testing measures, lowers the manufacturer’s chance of failing testing (given that the manufacturer’s plates do meet standards and that all other factors are constant). At issue is the current impossibility of estimating what fraction of the variation in BFD is attributable to variation in the plates and what fraction is attributable to the testing methods. As the experimentation recommended in the Clay section of this report (Recommendations 2-8) is completed, a better estimate of the test process variation may become apparent. As discussed in earlier sections, there are known but not well quantified issues that relate to variations in the thermal and stress properties of the clay medium itself, variations caused by different individuals hand working the clay as it is prepared for testing, variations in calibration, and other variation causes. Information on how the existing process performs will

OCR for page 1
Phase II Report on Review of the Testing of Body 39 Armor Materials for Use by the U.S. Army facilitate improving the process (minimizing excess variation, should it exist.) Importantly, this information may require refining the proposed standard. Recommendation 17: The committee recommends that testers and statisticians continue to work together as a team (1) to quantify in a statistically rigorous manner the amount of variation in backface deformation attributable to the testing process and that attributable to the plates, and (2) to ensure these results are appropriately reflected in an updated protocol. In particular, the statisticians involved with developing and implementing the statistically based protocol should be involved with the experimentation recommended in Recommendations 2-8. It would be helpful for statisticians to be part of the process of understanding and quantifying test system variation. Lot Acceptance Testing Once a manufacturer has passed FAT and begins production, LAT is used to ensure that body armor continues to conform to contract requirements. LAT samples are typically smaller than FAT samples and often vary by the size of the lot. For example, the current ESAPI/XSAPI Army contract specifies a sample of 3 for lots of 26-150 plates, of 5 for lots of 151-1,200 plates, of 8 for lots of 1,201-3,200, and of 13 for lots of 3,200 plates or more. As with the FAT, a penalty point system is used to determine acceptance, rejection, or additional testing of the lot. However, as described in MIL-STD-1916, “sampling inspection alone does not control or improve quality” (DoD, 1996, p. 8). Owing to the critical nature of safety when it comes to body armor, continued LAT testing is both desirable and necessary, but the committee also recognizes that modern quality control calls for manufacturing processes to be improved to eliminate as much variation as possible. As the committee has previously shown, elimination of variation can provide a number of benefits, including the reduction of risks to both the manufacturer and the DoD. In addition, to the extent that such reductions in variation lead to more predictability in plate performance and testing outcomes, these reductions might lead to innovations in plate design that allow reductions in plate weight while maintaining ballistic protection. The committee also notes that MIL-STD-1916 describes “switching procedures” by which the quantity of items inspected during LAT can vary depending on how the manufacturer has performed on previous LATs (DoD, 1996). For example, a manufacturer that has demonstrated consistently good LAT performance can have the number of items tested in future LATs reduced. Conversely, a manufacturer that has demonstrated poor past performance can have the number of items tested in future LATs increased for tighter scrutiny. Recommendation 18: The Department of Defense should develop standard statistically based body armor Lot Acceptance Testing (LAT) protocols that incorporate aspects of MIL-STD-1916, particularly those related to quality control and improvement and switching procedures. Adopting and incorporating modern statistical process control methods into the manufacturing processes is specifically recommended so that plate

OCR for page 1
Phase II Report on Review of the Testing of Body 40 Armor Materials for Use by the U.S. Army quality can be managed and assessed prior to lot acceptance testing. This could potentially reduce testing effort and costs. Note that while MIL-STD-1916 states that the “sampling plans and procedures of this standard are not intended for use with destructive tests,” these aspects of the military standard are relevant to body armor LAT testing. Using a statistically based approach that quantifies the risks inherent in test design enables decision makers to explicitly address the trade-offs and is both commendable and desirable. While further research and coordination are necessary to finalize the design and continuing review will be needed as test and manufacturing conditions change over time, a statistically sound protocol will ensure the quality of the body armor that is so critical to our soldiers. Conducting Due Diligence Before Formally Adopting the Statistically Based Protocol The DoD IG report DoD Testing Requirements for Body Armor recommended that DOT&E use “quantitative methods to develop a test sample size for testing that limits the number of possible failures” (DoD, 2009, p. 34). A statistically based test protocol is critical because it is the only way to quantitatively characterize body armor performance under a variety of threat conditions and operating environments to better inform DoD decisions. Because there is variation in manufactured body armor, testing alone cannot ensure that body armor is 100 percent effective. One can, however, develop higher confidence in the effectiveness of the body armor by using a statistically rigorous assessment with sufficient sample size. The committee feels that DOT&E is making a good faith effort to follow the DoD IG guidance in its development of the proposed statistically based protocol. Many thousands of body armor plates have been produced and sent to soldiers on the battlefield. No soldiers are known to have been killed from projectiles that the plates were designed to defeat. The acquisition process that has successfully fielded body armor with superior ballistic protection sets a high threshold when making changes to armor design, manufacture, and testing, but at the same time it should not hinder the development of even better body armor for fielding to soldiers. For example, current test procedures induce an unknown amount of variation into the measurement of BFD, adding risk for the manufacturer and perhaps resulting in heavier-than-necessary plates. Thus, any and all changes in design, manufacturing, or testing processes should be made with deliberate caution to ensure that plate performance is maintained while also ensuring that the best science and engineering are brought to bear on testing and improving body armor. The DoD has a responsibility to set performance requirements and to establish the protocols that verify that they are met. Any test protocol involves some risk that bad body armor will pass the test and good body armor will fail. The DoD has a responsibility to be explicit about these risks and to design a test protocol that balances cost, performance, ability to execute, fairness to the manufacturer, and risk to the soldier. Trade-offs can be made to result in protocols that are both statistically rigorous and practical in application. This conceptual approach is supported by the development of the current DOT&E

OCR for page 1
Phase II Report on Review of the Testing of Body 41 Armor Materials for Use by the U.S. Army proposed protocol. The committee believes DoD should address these issues with all stakeholders, DoD (military service Program Executive Officers, testers, users, and others) and non-DoD (NIJ, NIST, certified private testing laboratories, manufacturers, and others). The committee also believes that DOT&E should use the input from discussions with stakeholders to reconsider (and possibly revise) the proposed statistically based FAT protocol, and it believes that the following three considerations should be explicitly addressed. First, issues in the proposed statistically based protocol that add to the manufacturer’s risk and to the armor’s weight must be openly discussed. It is important to reach consensus on what constitutes a BFD failure and how such failure relates to soldier injury or death. Dr. Prather said that his 1977 study was very conservative, and he felt it was highly likely that a human could survive a blunt trauma force resulting from a BFD somewhat in excess of 47 mm. However, The Army’s body armor testing Test Operations Procedure (TOP) states that 43 mm constitutes BFD failure while also allowing a sliding BFD point scale between 44 and 47 mm. The proposed statistically based protocol changes the BFD upper limit for failure to 44 mm and requires that this limit be demonstrated with 90 per cent confidence for 90 per cent of the population of the plates. NIJ Standard 0101.06 states that “the armor model shall be deemed to meet these requirements [that is, to constitute a failure] if no BFD depth measurement due to a fair hit exceeds 50 mm (1.97 in.)” (NIJ, 2008, p. 49). Thus, at the outset of the report, Recommendation 1 highlights the need to conduct the research to quantify the medical results of blunt force trauma on tissue and to use those results to underpin a BFD standard. Second, the committee wishes to avoid the misperception that the proposed statistically based protocol accepts a higher death rate for soldiers wearing body armor. In particular, there must be no inadvertent public misperception that U.S. body armor is less survivable under the proposed protocol. For example, Table 3 of this report shows that the proposed requirement for a first-shot resistance to penetration is “90 percent no penetration with 90 percent confidence.” Taken out of context, a nontechnical, nonstatistical audience with no understanding of statistics could interpret this requirement to mean that the DoD will accept some penetration of the plates and some deaths on the battlefield. Adding to the confusion is the wording in NIJ Standard 0101.06, which is used extensively by testers for civilian law enforcement organizations: “Each panel must withstand the appropriate number of fair hits and may not experience any perforations” (NIJ, 2008, p. 49). One unintended consequence of the proposed protocol is that nontechnical audiences could incorrectly conclude that soldiers are knowingly being placed at greater risk than is currently the case in combat or in civilian law enforcement. It is possible that DoD could develop zero-failure FAT protocols that would achieve the appropriate levels of risk and that would eliminate such misperceptions.

OCR for page 1
Phase II Report on Review of the Testing of Body 42 Armor Materials for Use by the U.S. Army Third, it is important that the proposed statistically based protocol be seen not just as another in a long line of standards but as an improvement that incorporates input from all of the stakeholders and that embodies the best science. Recommendation 15 calls for the DOT&E and NIJ to take the lead in developing a single national body armor standard. It is particularly important to develop broad-based support for a future statistically based protocol and ensure that its adoption will not undo many years of successful engineering, or significantly increase manufacturer costs, or take too much time. For example, vendors of body armor plates are likely to view a new protocol more favorably if the protocol is founded on: (1) discussions on the basis and application of statistically based protocols; (2) feedback and debriefings from statistical experiments; (3) the post-test availability of armor for inspection and further technical analysis; and (4) shared knowledge on how armor standards and test protocols influence armor development. The committee unequivocally supports the concept of a statistically based test protocol that explicitly and scientifically acknowledges and addresses the testing risks described in this report. However, it also appreciates that due diligence and deliberate caution are warranted to ensure that the change in test protocol does not result in unintended effects on body armor manufacture or performance. If these considerations are addressed in a straight forward and transparent manner with the body-armor stakeholders, the proposed statistically based protocol is likely to be well accepted. Recommendation 19: The Department of Defense (DoD) Director, Operational Test & Evaluation (DOT&E) should provide briefings to and receive feedback from all stakeholders in DoD (military service Program Executive Officers, testers, users) and non-DoD organizations (National Institute of Justice, National Institute of Standards and Technology, certified private testing laboratories, vendors) concerning the statistically based protocol. This feedback, as well as the results of the experiments and analyses proposed in this report, should be used as due diligence to carefully and completely assess the effects, large and small, of the proposed statistically based protocol before it is formally adopted across the body armor testing community. DOT&E should act on feedback from the community to improve the proposed protocol as necessary, to ensure that testing terms and concepts make sense to a nontechnical audience, and it should promote the use of statistically based protocols in future national standards for body armor testing, as appropriate. OVERARCHING RECOMMENDATION FOR PHASE II Over the course of both the Phase I and Phase II studies, committee members were impressed with the ongoing work at ATC but were dismayed by the lack of standardization in body armor test criteria exhibited by DoD and other organizations concerned with body-armor testing. They considered their recommendations for action to be fully justified by the urgency and criticality of body armor on the battlefield. Each of the 19 recommendations in the Phase II study contains an express or implied action to achieve greater part-to-part consistency in ballistic clay, analyze BFD dynamics, determine possible replacements for modeling clay, achieve a national clay

OCR for page 1
Phase II Report on Review of the Testing of Body 43 Armor Materials for Use by the U.S. Army standard for body-armor testing, or implement a statistically based protocol. The importance of body armor testing, as well as the life-or-death nature of body armor itself, require that such a large number of recommendations be acted on in a concerted fashion. Both DOT&E and the Army will recognize the issues addressed by the recommendations and have initiated or proposed several projects that move in the same or similar directions toward essential improvements. Many, if not most, of these projects are unfunded and will require the resources of DoD, the military services, specific service laboratories and testing organizations, as well as non-DoD agencies, such as NIJ, NIST or a Federally Funded Research and Development Center/University Affiliated Research Center, to implement. Overarching Recommendation: The committee applauds DOT&E for assuming a national-level leadership role in bringing the body armor test community together. The committee recommends that the DOT&E (1) work with Congress, DoD, the military services, and other organizations to find the resources necessary to implement the recommendations described in this report and summarized in Box 1 and (2) oversee, review, track, and assist designated action organizations with implementing these recommendations. This approach should result in more consistent test results that will provide equally survivable but lighter-weight body armor to our military service members and civilian police forces.

OCR for page 1
Phase II Report on Review of the Testing of Body 44 Armor Materials for Use by the U.S. Army Box 1 Phase II Recommendations to Improve Body Armor Testing Achieving Greater Part-to-Part Consistency in Clay 1. Quantify the Medical Results of Blunt Force Trauma on Tissue and Incorporate Results into the BFD Methodology 2. Determine Short-Term Standard Clay Specification 3. Conduct Rheological and Thermogravimetric Measurements 4. Procure and Experiment with a Clay Compounding Machine 5. Examine Technologies for “In Box” Mechanical Clay Working 6. Modify TOP 10-2-210 Procedures to Add a Post-calibration Drop (ATC, 2008) 7. Experiment with Various Clay Box Sizes and Shapes 8. Develop and Experiment with a Gas Gun Calibrator or Equivalent Device Analyzing Backface Deformation Dynamics 9. Analyze the Signal-to-Noise of Flash X-Ray Cineradiography 10. Experiment with Microscopic Temperature and Displacement Sensors in Clay 11. Experiment with the High-Speed Photographic Analysis of BFD Creation in Ballistic Gelatin Determining Possible Replacements for Modeling Clay 12. Study Ballistic Gelatin as a Mid-Term Alternative to Modeling Clay 13. Study Microcrystalline Waxes as a Long-Term Alternative to Modeling Clay or Ballistic Gelatin. Achieving a Single National Clay Standard for Body Armor Testing 14. Empower and Resource the Ad Hoc Clay Working Group 15. Convene a Nationally Recognized Group to Establish a Single National Standard for Handling and Validating Clay Implementing Statistically Based Protocols 16. Compare the Proposed Statistically Based Protocol with the Existing USSOCOM Protocol 17. Quantify the Variation in the Body Armor Test Process and Incorporate in the Protocol 18. Develop a Statistically-Based LAT Protocol 19. Conduct Due Diligence Before Implementing and Formally Adopting a Set of Statistically Based Protocols

OCR for page 1
Phase II Report on Review of the Testing of Body 45 Armor Materials for Use by the U.S. Army REFERENCES Aerospace Corporation. 1974. Equipment Systems Improvement Program–Protective Armor Development Program–Final Report, v 3. ATR-75(7906)-1. El Segundo, Calif.: Aerospace Corporation. Al-Tabbaa, A., and C. Evans. 2003. Deep soil mixing in the UK: Geoenvironmental research and recent applications. Land Contamination & Reclamation 11(1): 1-14. ATC (U.S. Army Aberdeen Test Center). 2008. Test Operations Procedure (TOP) 10-2- 210 Ballistic Testing of Hard Body Armor Using Clay Backing. Aberdeen Proving Ground, Md.: Aberdeen Test Center. Bass, C., R. Salzar, S. Lucas, M. Davis, L. Donnellan, B. Folk, E. Sanderson, and S. Waclawik. 2006. Injury risk in behind armor blunt thoracic trauma. International Journal of Occupational Safety and Ergonomics 12(4):429-442. Bonn, D., and M. Denn. 2009. Yield stress fluids slowly yield to analysis. Science 324(5933): 1401-1402. Cannon, L. 2001. Behind armour blunt trauma—An emerging problem. Journal of the Royal Army Medical Corps 147: 87-96. Chadwick, E., A. Nicol, J. Lane, and T. Gray. 1999. Biomechanics of knife stab attacks. Forensic Science International 105(1): 35-44. Clare, V., J. Lewis, A. Mickiewicz, and L. Sturdivan. 1975. Blunt Trauma Data Correlation. EB-TR-75016. Aberdeen Proving Ground, Md.: Edgewood Arsenal. DoD (U.S. Department of Defense). 1996. DoD Preferred Methods for Acceptance of Product. MIL-STD-1916. Arlington, Va.: Department of Defense. DoD. 2008. DoD Test Method Standard for Performance Requirements and Testing of Body Armor. MIL-STD-3027. Arlington, Va.: Department of Defense. DoD (Department of Defense). 2009. DoD Testing Requirements for Body Armor. Report Number D-2009-047. Arlington, Va.: Department of Defense Inspector General. Dunn, N. 2010. ATEC Proposed Army Protocol and Background for NAS Statisticians. Aberdeen Proving Ground, Md.: Army Testing and Evaluation Center. Huang, Z., M. Lucas, and M. Adams. 2002. A numerical and experimental study of the indentation mechanics of plasticine. Journal of Strain Analysis for Engineering Design 37(2): 141-150. Ji, H.., E. Robin, and T. Rouxel. 2009. Compressive creep and indentation behavior of plasticine between 103 and 353 K. Mechanics of Materials 41(3): 199-209. Karahan, M. 2008. Comparison of ballistic performance and energy absorption capabilities of woven and unidirectional aramid fabrics. Textile Research Journal 78(8): 718-730.

OCR for page 1
Phase II Report on Review of the Testing of Body 46 Armor Materials for Use by the U.S. Army Liechty, B., and B. Webb. 2007. The use of plasticine as an analog to explore material flow in friction stir welding. Journal of Materials Processing Technology 184(1-3): 240-250. Lyon, D. 1997. Development of a 40-mm Nonlethal Cartridge. ARL-TR-1465. Defense Technology Federal Laboratories Research Journal. Metker, L., N. Prather, and E. Johnson. 1975. A Method for Determining Backface Signatures of Soft Body Armors. EB-TR-75029. Aberdeen Proving Ground, Md.: U.S. Army Armament Research and Development Command. Nicholas, N., and J. Welsch. 2004. Ballistic Gelatin. Available online at . Accessed March 19, 2010. NIJ (National Institute of Justice). 1987. Ballistic Resistance of Police Body Armor. NIJ Standard 0101.03. Washington, D.C.: National Institute of Justice. NIJ. 2008. Ballistic Resistance of Body Armor. NIJ Standard 0101.06. Washington, D.C.: National Institute of Justice. NIST (National Institute of Standards and Technology). 1994. Memorandum: Rheology of Clays. December 12, 1994. Gaithersburg, Md.: National Institute for Standards and Technology. NRC (National Research Council). 2009. Phase I Report on Review of the Testing of Body Armor Materials for Use by the U.S. Army. Washington, D.C.: The National Academies Press. O’Callaghan, P., M. Jones, D. James, S. Leadbeatter, S. Evans, and L. Nokes. 2001. A biomechanical reconstruction of a wound caused by a glass shard—A case report. Forensic Science International 117(3): 221-231. Pena, L., B. Lee, and J. Stearns. 1994. Structural rheology of a model ointment. Pharmaceutical Research 11(6): 875-881. Prather, R., C. Swann, and C. Hawkins. 1977. Backface Signatures of Soft Body Armors and the Associated Trauma Effects. Aberdeen Proving Ground, Md.: Chemical Systems Laboratory. Raftenberg, M. 2006. Modeling thoracic blunt trauma: Towards a finite-element-based design methodology for body armor. Pp. 219-226 in Selected Topics in Electronics and Systems, Volume 42: Transformational Science and Technology for the Current and Future Force, Proceedings of the 24th US Army Science Conference, Orlando, Fla. J.A. Parmentola, A.M. Rajendran, W. Bryzik, B.J. Walker, J.W. McCauley, J. Reifman, and N.M. Nasrabadi, editors. Hackensack, N.J.: World Scientific Publishing Company, Incorporated. RDECOM (U.S. Army Research, Development and Engineering Command). 2009. Amendment of Solicitation/Modification of Contract W91CRB-09-D-0001/P00004. Aberdeen Proving Ground, Md.: RDECOM.

OCR for page 1
Phase II Report on Review of the Testing of Body 47 Armor Materials for Use by the U.S. Army Vaughan, N. 2001. Assessment of Aprons for Protection Against Drop Forging Projectiles. Contract Research Report 395/2001. Sheffield, U.K.: Health and Safety Laboratory. Weber, D. 2000. Measuring Impact Velocities of Non-Lethal Weapons. Available online at . Accessed March 19, 2010. OTA (Office of Technology Assessment). 1992. Police Body Armor Standards and Testing, Volume II: Appendices, OTA-ISC-535. Washington, D.C.: Office of Technology Assessment.