Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 54
2009-2010 Assessment of the Army Research Laboratory 5 Survivability and Lethality Analysis Directorate INTRODUCTION The Survivability and Lethality Analysis Directorate (SLAD) was reviewed by the Panel on Survivability and Lethality Analysis of the Army Research Laboratory Technical Assessment Board (ARLTAB) during June 22-25, 2009, at White Sands Missile Range in New Mexico and during July 25-28, 2010, at Aberdeen Proving Ground, Maryland. SLAD’s mission is to provide expert assessment and evaluation support with respect to the survivability, lethality, and vulnerability (SLV) of Army equipment and soldier systems in the context of full-spectrum battlefield environments, with a goal of ensuring that soldiers and systems can survive and function reliably to be lethal to enemy forces. SLAD performs this mission through the development and application of well-founded testing and evaluation (T&E) methodologies and facilities; deterministic and probabilistic tools comprising physically motivated, empirically based damage-assessment models; experimentation; simulation; and theoretical analysis, all integrated with a solid understanding of battlefield functional requirements. The SLV assessments are performed throughout the entire life cycle of Army systems, from development through acquisition, deployment, and operation. SLAD’s mission supports Army Headquarters, program executive officers, program managers, system developers, contractors, and other defense-oriented laboratories. SLAD is composed of two divisions: the Information and Electronic Protection Division (IEPD) and the Ballistics and Nuclear, Biological, Chemical Division (BND). The SLAD portfolio consists of many small T&E programs directly related to the assessment of specific Army components, with several larger programs aimed at developing tools necessary to enhance T&E efforts and to broaden the scope of the SLAD contribution to Army system life-cycle assessment. In each 2-year review cycle, the panel is exposed to about 90 percent of the SLAD programs at some level, but to less than 20 percent of the portfolio at a level sufficient to assess in depth the technical quality of the work. The panel focused on
OCR for page 55
2009-2010 Assessment of the Army Research Laboratory those programs with the strongest continuity of effort, with the addition of a crosscut of new and mature programs to provide the context of the full spectrum of work performed within SLAD. Table A.1 in Appendix A characterizes the staffing profile for SLAD. CHANGES SINCE THE PREVIOUS REVIEW The SLAD portfolio and evaluation structure are based strongly in the directorate’s long history of critically important and successful ballistics-based vulnerability assessment, which has supported the Army testing and evaluation activities. SLAD has continued to broaden its program base to include the assessment of the vulnerability of communications, networks, and information processing on the battlefield. SLAD has taken the Mission and Means Framework (MMF) for analysis and incorporated it into a larger analysis methodology called Mission-Based Test and Evaluation (MBT&E). SLAD legacy tools only evaluate performance but not the effect on mission. The objective of SLAD’s MBT&E methodology development is to add the capability of assessing how the loss of performance affects mission requirements. MMF, a methodology described in the previous ARLTAB review,1 defines the “mission capability requirements” from the top down, starting with strategic national, to strategic theater, to operational, to tactical, to task. Thus, all of the tasks required to perform a specific mission are defined. “Capability” is defined from the ground up, starting with deployed forces, up through the subsystem, to the platform capability. Top down, a set of mission-required tasks is defined; bottom up, a set of mission capabilities is defined. The MBT&E analyses then match requirements with capabilities and the effects of threats on both, resulting in a “mission performance degradation assessment.” SLAD now has, conceptually, the capability to tie together all capability and performance metric assessment groups and/or tools into a mission framework, including those of the Human Research and Engineering Directorate (HRED); the Tank Automotive Research, Development and Engineering Center (TARDEC); and survivability, lethality, and vulnerability assessment (SLVA). MBT&E is intended to provide the linkage between any functionality or degradation and mission capability. Three pilot programs have been identified for the application of this new methodology. One of the pilot programs will utilize S4 software (System of Systems Survivability Simulation) and MUVES 3 (Modular UNIX-based Vulnerability Estimation Suite) as tools within the MBT&E framework. It is notable that after trying the MBT&E process, the Director of the Army Evaluation Center (AEC) decided that it would be used for all AEC projects. The systems capabilities analytic process demonstrated meaningful results that SLAD customers want. Using it, they can directly correlate testing and modeling results to mission task success. According to SLAD, the directorate is the only organization actively developing a process to link quantitatively from a system’s capabilities to a system’s components. This is an excellent example of an opportunity for the expansion of a successful methodology to application with other activities. SLAD has now imposed a more formal structure for program management. The directorate has an internal steering committee (including both SLAD division chiefs) that provides management oversight of the system-of-systems analysis (SoSA) program manager and subordinate SoSA operating teams. In addition, the SLAD Director has stood up a project management office within his staff as a resource on project management processes, practices, and procedures for all programs within SLAD, including SoSA. SoSA and MUVES 3 are the two places where SLAD has been the most successful with 1 National Research Council. 2009. 2007-2008 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press.
OCR for page 56
2009-2010 Assessment of the Army Research Laboratory implementing its project management strategies, but these projects were also the ones most in need of such support. SLAD has also instituted a matrix structure that appears to be a creative way to balance the culture of innovation and prototype development with program support. This approach should be promoted, but a critical concern is whether the leaders who are equivalent to program and/or project managers have the requisite control over resources necessary to be responsive to sponsors’ needs, or whether they are simply internal coordinators. Perhaps the largest change that occurred during this review cycle was the cancellation of the Army’s Future Combat Systems (FCS) Program. Much of the focus of SLAD’s analytical tool development had been on developing the capability to assess systems being considered under the umbrella of FCS. The SLV assessment tool methodology focus has now shifted to assessing the Ground Combat Vehicle (GCV) Program. It is not yet clear how the change in focus will affect the SLAD tool development process. ACCOMPLISHMENTS AND ADVANCEMENTS SLAD management is aware of the importance of hiring creative, energetic, and innovative professionals, as well as competent and enthusiastic early-career engineers and interns. Management also recognizes the value of professional development and provides several methods for this: pursuit of advanced degrees, certifications, personal leadership opportunities, developmental assignments, conferences, and collaborations. More emphasis should be placed on continuing education, to enhance the knowledge and experience base in the principles of basic research and to broaden the scope of collaboration outside the Army Research Laboratory (ARL). Response on Improvised Explosive Devices In wartime, rapid response to soldier-in-the-field challenges is imperative. SLAD personnel have provided exemplary service to their country in this time of war with their dedication and “can-do” attitude. This highly motivated, mission-aligned, enthusiastic group of engineers and technicians has made significant contributions to the Army and the Department of Defense (DoD) through creative problem solving and solid engineering know-how. An excellent example of this effectiveness is the program in radio-frequency (RF) countermeasures. This program is very impressive. This group has provided effective response to Army needs and support of Army personnel, demonstrating a high-energy, creative approach to solving problems quickly while maintaining cost-consciousness. This support work for the Joint Improvised Explosive Device Defeat Organization (JIEDDO) is commendable and an impressive help to Army field personnel. The modular approach used in this program ensures extensibility of the methodology to enable quick turnaround on future problems. This program also shows a good use of existing software for analysis. Target Interaction Lethality/Vulnerability Program The Target Interaction Lethality/Vulnerability (TILV) Program is an excellent example of an SLAD strategic program aimed at improving SLAD’s vulnerability analysis capabilities. A typical TILV project is 3 to 4 years in duration and is designed to address a significant methodology shortfall. Some examples of recent projects include the work on underbody blast effects and the work on ballistic impacts on helmets. The development of underbody blast methodology is well thought out and is commendable for filling a critical need in this area. The methodology combines appropriate physics and necessary codes to
OCR for page 57
2009-2010 Assessment of the Army Research Laboratory assess damage to vehicles. There is a good understanding of the limitations of the various elements that are interconnected to form the overall methodology. This is a notable example of research that could have an impact on the next generation of vehicles. The work is being presented at appropriate conferences and would benefit from peer review for publication in archival journals. The work on ballistic impacts on helmets is a good example of solid engineering using well-established principles to improve test standards. This program provides an excellent opportunity to collaborate with outside experts to define and conduct further research efforts in this area. TILV projects demonstrate SLAD’s collaboration efforts with the Army Research Laboratory’s Weapons and Materials Research Directorate as well as with the U.S. Air Force Research Laboratory (AFRL). There was also commendable collaboration in international data-collection efforts using facilities such as the test ranges at Aberdeen Proving Ground, Maryland; facilities at Adelphi, Maryland; and Eglin Air Force Base, Florida; as well as at ARL’s computational facilities. Past TILV projects have contributed to the development of the end-to-end active protection system (APS) model and to improvements in the models used to predict the blast field created by munition detonations. The TILV Program is a very good example of effective project management. Each project has clearly defined objectives and is a customer-driven, short-term project to address perceived, or customer-defined, issues (modeling and testing shortfalls). Each project has an “end” metric that includes a time limit. These termination time limits avoid any pursuit of activities that are beyond what is reasonable and practical. Software Development Prior ARLTAB assessment reports suggested that SLAD management needs to achieve a better balance between tasks that are more beneficially performed in-house and those that are best outsourced. There were several programs in the current review cycle in which commercial-off-the-shelf (COTS) software was effectively utilized to enhance SLAD capability, demonstrating an admirable change to the in-house paradigm. The Optical Augmentation Micro/Macro Modeling Program is one example of the effective use of COTS technology and of intelligent outsourcing. By hiring an outside vendor to integrate existing COTS tools, SLAD personnel and budget were freed up for what they are uniquely qualified to do: that is, to understand the military requirement, locate the tools and methods to address it, and apply the software for SLAD-specific uses. The Virtual Shot Line (VSL) Program approach provided another example of an application of commercial computer-aided design (CAD) tools and formats: these were used in the VSL Program for the successful generation of a program that works in real time to evaluate and identify which components of a vehicle are affected by a projectile or fragment following a given shot line and what effect the shot will have on vehicle subsystems and system performance. The program exploits work previously done in ray tracing for radio propagation. The use of standard geometry formats and commercial CAD tools reduced the programming effort by a large factor and also reduced computation time. SLAD staff and its customers are equally enthusiastic about this new capability. The VSL Program demonstrated many programmatic aspects found missing in previous program reviews, from both a technical and a managerial perspective. First, on a technical level, the use of graphics-processing-unit programming technology makes supercomputer performance possible on a desktop for certain highly parallel tasks. Architecture decisions for SLAD’s next-generation ballistic vulnerability/lethality model, MUVES 3, should be reviewed to take advantage of this new capability if possible. The VSL Program has admirably demonstrated its ability to display interactions between subsystems and the system that they support and to identify critical nodes whose failure can have catastrophic impact on the system if they are not better protected. This approach is applicable far beyond the fighting vehicle, and other systems can benefit from future application. The way in which the VSL
OCR for page 58
2009-2010 Assessment of the Army Research Laboratory project was executed—sending an SLAD analyst on a development assignment with a major external research organization—represents a significant departure from SLAD’s traditional way of doing business, as well as a possible new paradigm for how SLAD can use external contacts and collaborations as a force-multiplier for its limited resources. SLAD management’s seizing of this opportunity to commit to this experiment is commendable, and new and even more exciting moves in this direction are to be expected in the future. Internal SLAD reviews for the funding of future projects require a serious consideration of opportunities for in-depth external collaboration, and periodic reviews should be made to ensure that those opportunities have been exploited. Laboratory Facilities The laboratory tours during this review period clearly demonstrated the high level of enthusiasm of the personnel who have access to the numerous SLAD state-of-the-art facilities. Many problems critical to the SLAD mission are being addressed, including that of protective armor for personnel. The collection of programs on weapons effects on personnel is commendable and should serve as an excellent database for comparison and evaluation. The cooperation with various federal agencies and industry is exemplary. The gel block methodology that is being developed to study the effectiveness of body armor is a promising direction that could lead to fundamental insights for future designs. The helmet testing utilizes advanced measurement methods and is currently used as a screening tool. The recently established Electro Magnetic Vulnerability Assessment Facility (EMVAF) is a state-of-the-art facility with the goal of evaluating and investigating antenna performance, vehicle effects (jamming and communication), and multipath effects. The facility is impressive, well designed and managed, and well staffed by dedicated and energetic personnel. The management should be commended for conceiving, designing, and developing this facility, which serves many critical missions and needs of the Army. SLAD has increased its ability to maintain a leading edge for this sort of analysis. OPPORTUNITIES AND CHALLENGES The SLAD organization appears to be evolving from what was once primarily a testing and evaluation service organization to an organization that now also includes methods and process development functions for analysis and evaluation. This mission drift seems quite appropriate, given the nature of the tasks and uniqueness of the products and systems being analyzed. The development of the underbody blast methodology initiative is a good example of where it makes sense for SLAD to be active in research to support the vulnerability function. However, in order for SLAD to transition to this extended mission, the focus of the organization needs to include research as part of that mission. Communications The battlefield is fast becoming “integrated,” whether the communications involve interoperability with other services, situational awareness, tactics, collateral damage, fratricide, rules of engagement, mobility, survivability, or lethality. SLAD should be looking to expand its tools and expertise in order to help its Army customers address survivability and lethality in an integrated fashion in this complex network of systems. Areas in which SLAD is already working and that merit further, in-depth involvement are described below. SLAD should conduct more work in interoperability. SLAD has good knowledge of interference with Army radios, but it seems to have done very little to share this knowledge with the other services.
OCR for page 59
2009-2010 Assessment of the Army Research Laboratory As joint operations become the norm, interoperability will become critical for effective operations. Because the Army will be “at the business end” of many joint operations, it is imperative that the Army take the lead in this area. SLAD needs to recognize that this is a critical element of operational T&E, and it needs to take appropriate measures to incorporate interoperability into its test programs. Combat Systems Research As an organization, SLAD seems focused on survivability—for example, collecting vulnerability data and developing vulnerability algorithms—far more than on lethality. SLAD seems to address survivability as an entity separate and isolated from lethality. Collateral damage was not mentioned in any of the programs reviewed, but this is a critical consideration when evaluating lethality. It is becoming an issue even with survivability (e.g., active protection systems). Survivability is strongly tied to tactics, visual/infrared/electro-optic/radio-frequency/audio signatures, survivability systems (APS), and host-platform vulnerability. In most cases SLAD is aware of these other considerations but is not addressing them in an integrated fashion. SLAD should broaden its perspective by starting to consider what kinds of analysis tools, expertise, and products will be needed for future Army programs, and then it should start preparing roadmaps for the development of those capabilities. For instance, unmanned aerial vehicles (UAVs) were shown as part of the integrated FCS environment. Much work on UAVs is being done by the other services (and associated laboratories). SLAD effort to leverage these activities was not in evidence. Combat systems research at SLAD has migrated toward mine-resistant ambush-protected (MRAP) modeling and live-fire analysis as the need has emerged. This migration is very responsive and critical to informing system acquisition. The SLAD desire to extend models to be predictive with respect to vehicle and personal damage and/or injuries should be supported. Just as in ARLTAB reviews in previous years, a major bottleneck in the flow of analysis is CAD data conversion and the staff time required to review and validate input data. The Ballistic Research Laboratory (BRL)-CAD project should receive support to continue its progress in the area of CAD data conversion. The major technical gap in this area, with the largest potential payoff for support to current operations, is in the predictive modeling of underbody blast. This is a nontrivial technical problem; a multiyear program is underway to deal with it. The development of underbody blast methodology is well thought out and fills a critical need in the assessment of underbody blast vulnerability. It is a good example of research that could have an impact on the next generation of vehicles, or even on refinements of current production vehicles. A number of detailed research problems, including the effect of geometry on the coupling between blast and structure, need to be undertaken to improve the fidelity of the codes. Simulation Tool Development The Information and Electronic Protection Division is extending SLV capabilities into earlier stages of program development. This is commendable and should be encouraged. SLAD has many capabilities and much expertise that lend themselves to early product development. IEPD is already doing this on selected programs, but it could be extended throughout all of the division’s technical areas. The goal of simulation-tool development should be to enhance the simulation and predictive capability so that less laboratory and field experimentation is required (only enough to validate predictions). The use or development of a modular code would help accelerate the simulation-tool development process. Also important here is the need to do regression testing of deployed technology against new or modified threats. SLAD needs to develop a test methodology that includes full regression testing against all variants of the products that it has fielded and a communication and update mechanism for the users of those products.
OCR for page 60
2009-2010 Assessment of the Army Research Laboratory Countermeasure Capabilities SLAD has a very capable missile electro-optic countermeasure (EOCM) simulation capability, testing methodology, and set of experienced personnel. SLAD personnel have strong contacts with other DoD organizations involved in missile EOCM testing, and they are recognized within the defense industry as a key player in the field. SLAD’s activities and knowledge base are focused on Army EOCM system testing for established programs, which is appropriate, because such work is within SLAD’s basic charter. Even so, SLAD has contacts within industry and is currently involved in a cooperative test program with a defense contractor investigating an advanced EOCM concept for a ground-based threat response system. This effort is commendable, and it should be taken further. SLAD has the opportunity to develop additional knowledge and expertise by expanding its awareness of non-Army programs, capabilities, and advanced concepts. For example, SLAD was not aware of an AFRL program called the Laser IRCM (infrared countermeasures) Flyout Experiment that developed and tested a prototype of a closed-loop infrared countermeasures capability for large, fixed-wing aircraft. This technology could have application for Army platforms and should be considered. All of this information has been openly discussed at conferences and is readily available on the Internet, yet SLAD personnel were not aware of it. SLAD would benefit from becoming more involved and aware of non-Army EOCM programs and the lessons learned from those programs. Attending conferences is an obvious means of expanding this awareness. SLAD should also seek out information on advanced concepts and roadmaps from its DoD colleagues. SLAD should place additional emphasis on the field of directed energy (DE). There is some awareness of this field within SLAD leadership (for example, in the Counter-electronic High Power Microwave Advanced Missile project and other high-powered-microwave and high-energy-laser efforts), but the awareness needs to be spread to lower levels of the organization. There are not many U.S. programs in DE at present, but the awareness is only a matter of time before DE capabilities transition to the field and DE threats proliferate. Now is the perfect time for IEPD to build up capabilities and expertise, develop professional relationships, and start thinking about possible collaborations. This would be a perfect area of involvement for some of the division’s early-career personnel. There are dedicated DE conferences sponsored by the Directed Energy Professional Society (www.deps.org) that IEPD personnel can attend. Collaboration and coordination of EMVAF programs with other services (the Air Force and the Navy) should be continued, cultivated, and encouraged for the development and evaluation of capabilities. Modeling and simulation tool development should be encouraged. Successful development could help in the validation of data and lead to predictive capabilities, which would make this facility an unrivaled resource in DoD. Validated modeling and simulation tools could also eventually be incorporated into SoSA (S4), providing enhanced capabilities for analysis and decision making. Warfighter Survivability The Experimental Techniques for Helmet Testing Program provides an excellent opportunity for SLAD to collaborate with outside experts to define and conduct further research to characterize fully all of the variables that might be important in the analysis of test results. There is a need to define more analysis parameters for a better understanding of the standard test method relative to field data. There may be parameters other than those defined thus far that would make more sense for comparisons. There are a number of research opportunities that could result in a markedly improved understanding of the protection of personnel. These include the development of scaling laws for gel blocks in terms of energy absorption, the dispersion of momentum through the study of wave propagation using high-speed pho-
OCR for page 61
2009-2010 Assessment of the Army Research Laboratory tography, and the validation of penetration models used in simulations by cross-comparison. The work on helmets can be expanded by including the characterization of materials at strain rates relevant to the application, a systematic examination of the role of composite structure on energy absorption, and an assessment of damage using nondestructive evaluation (NDE) techniques. It would be useful to provide measures for the uncertainty quantification of the parameters that are being measured. In armor testing, SLAD does not characterize the samples to gain an understanding of the lot-to-lot variability and the effect on scatter in the data. It would be useful to see the development of some sort of NDE methodology for this as well. Both the helmet testing and armor testing are very solid efforts, but there is much more room for defining research programs around these projects. Active Protection Technology The Army and its contractors are developing active protection system technologies to augment the armor protection on combat vehicles. To support the assessment of APS technology, the U.S. Army has developed an end-to-end model for APS systems. SLAD has defined its role as being a major partner and contributor in the APS effort, working to assess the following: the effectiveness of APSs against all of the threats in the class that the system is designed to defeat, the ability of the systems to function against potential threat countermeasures, and the potential vulnerabilities introduced to a host platform by an integral APS. However, the SLAD presentation of this program did not support many of these claims and, in fact, conveyed the distinct impression that SLAD is not yet fully engaged in the end-to-end analysis of active protection systems. It seems clear that SLAD is addressing gaps in the APS modeling capability, which is something that SLAD does particularly well and in which it is likely to make significant contributions. Although SLAD has expertise that can be brought to bear in certain technical aspects of the modeling effort, it is not at all clear that SLAD has the necessary knowledge or experience to do a thorough, system-level analysis. This is a key shortcoming that SLAD should attend to sooner rather than later. System-level analyses are becoming more important to Army programs, and SLAD needs to develop this capability soon or risk losing the pace required for understanding this emerging element of survivability. Information Operations SLAD does not seem to have ready access to classified information or data in the area of information operations, which prevents collaborations with the appropriate contacts. Interaction with the intelligence world is paramount for ensuring that SLAD has access to the classified information needed to develop the state-of-the-art vulnerability models for current and anticipated threats. It was obvious from many of its presentations that SLAD does not have clear inroads into the world of classified information. Remedying this need must be driven from the top down. The resistance of the Deputy Chief of Staff, G-2, U.S. Army, whose area of responsibility is intelligence, and of the Chief Information Officer, G-6, whose area of responsibility is information management, to unconventional threat concepts was identified during this review period as a problem, as was the fact that SLAD cannot talk directly to the all-source intelligence community. It also appears that SLAD may lack a sufficiently high degree of access to classified facilities and communications, which is a serious concern, given that all credible sources of intelligence and live operational planning are Top Secret-Sensitive Compartmental Information and are thus compartmented. SLAD seems not to know how to engage the intelligence community. SLAD should follow up on and look for opportunities to interact with the U.S. Strategic Command, the unified major command at Offutt Air Force Base, Omaha, Nebraska; and also with the U.S. Cyber Command, located
OCR for page 62
2009-2010 Assessment of the Army Research Laboratory at Fort Meade, Maryland (by moving up through the Army G-6, which is in the process of developing an Army Cyber Warfare Doctrine). If SLAD finds it difficult to form direct relationships with these organizations, then it should seek partnerships with laboratory organizations that have those relationships, such as the Department of Energy laboratories. SLAD’s superior chain of command eventually intersects with authorities capable of forging needed relationships, but SLAD will have to initiate and justify requests for such interaction. SLAD has an established methodology for SLV analysis that has worked very well in the past, but this methodology may not be fast enough for information operations (IO). The IO threat can change rapidly, and the methodology requires constant updates to remain effective. SLAD should increase its visibility in the cyberthreat community. It should look to initiate partnerships with organizations that have appropriate knowledge (e.g., the Strategic Command) and the specific networks as they are fielded. These relationships may take substantial time and effort to build, but doing so may prove exceedingly worthy goals. System-of-Systems Analysis and the System of Systems Survivability Simulation Tool SLAD is very much analysis-based and fulfils an important role for ARL and the Army in studying, testing, and evaluating weapons systems for vulnerability and survivability. By comparison, the system-of-systems (SoS) concept, as SLAD conceives it, is at a higher conceptual level, more focused on a larger-scale combat-modeling framework. This is very different from the modeling performed by the current SLAD organization and will require additional investment in personnel resources and time to gain the level of expertise necessary to be on par with existing capabilities within the directorate. The S4 initiative is aimed at providing extended support for analysis and evaluations of larger-scale integrated systems that incorporate more behavioral system interactions and dynamics into the decision making. There are serious issues in SoSA and S4 to be clarified and justified. The problem context for SoSA has not been clearly presented. The program description gives the impression that SLAD is taking on the SoS analysis problem in general, rather than tactical engagement problems at the Army brigade level and below, and perhaps even more narrowly, utilization of Future Combat Systems. SLAD should first present examples of SoS problems that ARL seeks to address, and then move into the analytic approaches that are considered effective for this particular domain and class of systems. From a modeling and analysis point of view, SLAD’s program scope as it is today provides SLV analysis and evaluation support for a range of equipment, processes, and weapons systems. Three issues motivate SLVA: (1) identify system weaknesses, (2) identify adversaries’ capabilities to exploit, and (3) determine the relative impact or consequences of exploitation. These issues are direct parallels with the widely accepted Failure Mode, Effects, and Criticality Analysis in product and system reliability engineering design and analysis. This is reasonable and seems to be consistent with the current methods employed in SLAD. The SoSA initiative is aimed at providing extended support for comparable analysis and evaluations of larger-scale integrated systems that incorporate more behavioral system interactions and dynamics into the decision making. There is a concern that the direction for the SoS framework and philosophy is to develop a platform for the analysis and evaluation of SLV without a specific plan. The concept is a good philosophical approach for describing the integral features of a complex system in terms of the physical elements, cognitive processes and/or domain, and context realization. What are lacking are the details that address the objectives, requirements, and operational level for SoSA model development:
OCR for page 63
2009-2010 Assessment of the Army Research Laboratory What is the purpose and value of the SoSA concept? What does this concept provide that is at present not available for carrying out the SLAD mission for analyzing and evaluating survivability and vulnerability? What is the plan for incorporating or transitioning into this new broader framework, and how will this improve the results of the analyses and evaluations for vulnerability? During this review period the SLAD presentation of its approach to SoSA provided a description of the management oversight process, but issues and concerns still apply: What are the survivability questions to be addressed by the modeling effort? How much granularity is necessary to address the issues? What assumptions can be accepted, verified, and/or confirmed? What level of confidence is permissible for the SLV decisions? There is need for a concise plan that includes the rationale for the use of SoSA in SLAD; the plan should indicate how the use of SoSA will contribute to the SLAD mission and identify the technical-personnel support necessary to sustain the function. This plan needs to define how SoSA tools will be established and implemented and to identify one or more specific applications that will prove the worthiness of this tool relative to the SLAD portfolio. It seems that SLAD is taking a broad and general approach to developing this SoSA tool without a plan for how it will be used in such a way that results can be interpreted meaningfully. This initiative can be viewed as a large experiment, with many factors (independent variables) that can be varied purposefully or randomly and with many dependent variables that can be measured and used variously as objective gauges. SLAD has presented no evidence that it shares this view or that it is concerned about designing experiments so that results can be statistically evaluated. To do so is technically and computationally quite challenging, but it is essential if SLAD is to claim any result of scientific consequence. In developing such a complex tool, it is imperative to identify near-term and long-range goals and to establish a well-defined set of success metrics against which to compare progress. SLAD mentioned that a proof-of-concept analysis is being established. This is a very sound idea; SLAD should rethink this program and take a “Hudson’s Bay Start” (finding and fixing problems early through smaller first steps) to fully assess the feasibility of its approach. SLAD should demonstrate the feasibility of its tool-development methodology on a small, well-defined problem or case study for which the variables and uncertainties are mostly well understood, with a clear, concise description of exactly what information needs to be provided as input and what is expected as output. This requires a sound definition of the customer and of the problem to be addressed. In this manner SLAD can develop confidence in its tool-development methodology before committing vast resources to this effort. Once SLAD has successfully demonstrated the feasibility of its methodology, it can work toward expanding the tool set to perform analyses of a broader range of systems, subsystems, and threats. As a start-up in a nontraditional area for SLAD, this project must have three goals: (1) to make contacts and establish credibility inside the Army, (2) to achieve recognition in the wider scientific community, and (3) and to obtain well-understood, useful, statistically justified, trustworthy technical results. A short-duration, well-defined, proof-of-concept analysis accomplishes these goals without impeding progress toward achievement of the long-term goals. There is a concern that individual SLAD analysis codes are implemented within S4 mainly because they are available and the S4 staff has the time to do this work. SLAD should apply a more disciplined approach, wherein a new code is integrated when it addresses a known accuracy bottleneck, and its performance is tuned to the level of accuracy needed by S4.
OCR for page 64
2009-2010 Assessment of the Army Research Laboratory Specific concerns regarding SoSA and S4 include the following: Adaptation and reliance on SoS as a methodological framework and reliance on red team analysis for threat modeling. The SLAD approach to survivability analysis with regard to information operations is fundamentally flawed. It relies almost exclusively on red team analysis. There are two problems with this approach as practiced. First, the red teams should be used as an adjunct to formal threat-modeling and remediation techniques. In theory, a red team exercise will discover threats that were not sufficiently modeled using other formal methodologies. SLAD should adopt one of the several generally used threat-modeling methodologies. Second, there is insufficient input from the various intelligence organizations on the nature of current threats for creating an adequate basis for either threat modeling or red team operations. IO is a very fast-moving environment, and the nature (source and method) of attacks changes quickly. Everything that SLAD is working on in this arena is almost 2 years old. SLAD should create partnerships with the National Security Agency (Network Tactical Operations Center, Technical Access Operations, Advanced Network Operations) and the Central Intelligence Agency. Acknowledging the difficult and time-consuming nature of the task, it is worthwhile for SLAD to initiate these partnerships. Acceptance and understanding of the use of and limitations to agent-based modeling. SLAD’s current implementation plan relies heavily on agent-based simulation, originally developed at the Santa Fe Institute. In professional hands, agent-based modeling can explain the “what” of some situations—that is, it can sometimes reproduce observed results—but that can be of limited value, because it does not explain the “why.” Agent-based models in inexperienced hands have produced embarrassing failures. Proponents sometimes make much of the fact that such simulations can cosmetically exhibit large-scale behavior that is complex, but of what value is that if the model does not explain why the complex behavior is happening? As a general observation, agent-based modeling can be valuable even when it yields only qualitative results. For example, agent-based models in behavioral economics have been used successfully to choose between proposed structures for markets and regulation of markets, even though these simulations cannot be validated quantitatively. Information assurance is not treated strongly enough in the current cyberthreat environment. There is concern that insufficient priority is being given to SLAD’s information assurance practice, given the current cyberthreat environment. There is a potential to have broad foreign intelligence penetration of SLAD-delivered solutions if the information assurance methodology is not robust enough. SLAD needs to adopt, as soon as possible, a security development life-cycle model for the development of its code and systems. Historically, DoD self-created systems have proven very insecure. SLAD should also have a relationship with the National Visual Analytics Center at the Pacific Northwest National Laboratory. Lack of collaboration and tie-in with the intelligence community. SLAD creates a relatively small set of stable threat effects and models those reliably in the SoS. Although this is a valid approach, it should always be used with caution because of the problems associated with using models describing the vulnerabilities of individual components in order to predict the behavior of whole systems. The approach taken elevates the analysis to get around this problem by looking only at the effect of the attack on a component, not the method of attack at all. The risk that SLAD runs, and it is a recurring theme, is that it does not have actual intelligence data on which to base its analysis. Current silent Trojans and supply-side attacks against network interface cards and routers come to mind as examples of attacks needing analysis for which SLAD does not have access to intelligence data.
OCR for page 65
2009-2010 Assessment of the Army Research Laboratory Lack of collaboration and relationships with the complex-systems-analysis community. SLAD should establish a partnering relationship with the Naval Postgraduate School and other organizations leading the charge in complex systems analysis. Uncertainty analysis not well defined. So-called systems of systems generally contain enormous amounts of variability. Such systems can be made to do many different things by adjusting the individual components differently, and these systems are so complex that the connections between individual components (e.g., agents) and overall system behavior are not at all clear. SLAD needs to detail how uncertainties are incorporated and accounted for within each model. The phrase “managing uncertainty” as used in this review must surely include dealing directly with uncertainty issues, but there is no evidence that the S4 project has done so yet. S4 is a complex simulation, with multiple levels of granularity and complex interactions between different types and levels of simulation. A serious effort must be made to track how errors propagate and accumulate in the system. Verification and validation studies and sensitivity studies not well defined. There appear to be scores, if not hundreds, of control variables that need to be systematically explored before analysts make any judgments about model results. This exploration is an essential precursor to any V&V effort. Unfortunately, without hard basic thinking about just what information the SoS model will provide, how sensitive it is to input assumptions, how to interpret its outputs, and other basic issues, SoSA demonstrations are likely to be largely meaningless. Sensitivity to inputs is a substantial issue, and experimental design can contribute to an understanding of the model’s sensitivity. A major hurdle that the SoSA project must cross to be successful and relevant is to demonstrate an ability to extract useful information from large numbers of Monte Carlo simulations of time histories. This is a challenging problem, and if the S4 staff can make good progress in this direction, it will make a major contribution to the field. The families of metrics that have been defined are clearly intended to be a start in this direction, as are the graphical techniques that highlight these metrics and leverage human pattern recognition capabilities to sort through the S4 output, but it is not yet clear that these are the right metrics, or that effective means of understanding their distributions and correlation structures have been found. SoSA is driven by the analysis question being asked, and it is inherently multidisciplined. SLAD wants to develop a distributed, collaborative environment. If these points had been made upfront and put into the Army tactical engagement context, they would have served to guide the SLAD staff better in their SoSA characterizations and in focusing their work. They are useful points to keep in mind as one develops an SLAD SoSA approach, but that approach either does not yet exist or was never articulated. Systems engineering practices should be a foundation of SLAD’s institutional approach as a complement to a well-defined design for SoSA modeling tailored to the Army’s modern tactical engagement problems. This may not be feasible within SLAD’s resources. There is need for an organization within the directorate that supports the modeling, methods, and analytical needs and which interacts with other ARL directorates for new initiatives, but such an organization would look very different from the current SoSA effort. The SoSA project is the largest research area in which SLAD tries to interact with academia or with outside researchers to gain information on how best to do things. SoSA is SLAD’s largest tool. SLAD is still relying solely on the Physical Science Laboratory at the New Mexico State University to develop the System of Systems Survivability Simulation (S4) tool, but it has started the process of developing collaborations both within and outside the Army (e.g., with the Navy and academia). SLAD should try
OCR for page 66
2009-2010 Assessment of the Army Research Laboratory to make use of interactions with the Defense Advanced Research Projects Agency (DARPA) panel on complex systems, although it is recognized that interactions with DARPA can be difficult. There are also many academic programs being initiated in complex systems with which SLAD can interact. SLAD should consider trying to attract postdoctoral students or visiting professors who are expert in the various fields of interest to SLAD. It is acknowledged that it may be challenging to attract experts of this type, because the constrained topic area of survivability/lethality is not a widely addressed research topic and because much of the SLAD work may be classified and therefore difficult to publish in conventional journals. The Board has requested that SLAD address the questions and suggestions above in a special meeting scheduled for January 2011 and in the next review meeting, scheduled for August 2011. It is expected that these meetings will include discussion of ways in which SLAD’s core competence in component modeling can best be merged with and support system-of-systems modeling, and where the bounds of SLAD’s activities in system-of-systems analysis should be set. MUVES 3 MUVES 3 was presented to the panel as the vehicle by which SLAD’s MBT&E analyses are going to be conducted. Analyses start with BRL-CAD drawings of all of the vehicle components of interest: a library of penetration models, a library of damage models, and then an assessment of a “kill” is made of component, soldier location, or other element of interest. This code presents an admirable capability for using a high-resolution, high-fidelity simulation to support a reduction in the number of live-fire tests required to fully bound component damage due to various threats. This program has made great strides in the past several years, but there are still several areas of concern. This is a large model that combines many components with no clear summary on how the various pieces are linked in order to produce an accurate representation of system behavior and outcomes. SLAD personnel have developed a set of well-defined case studies with a given set of input information and expected outcomes, but there is no clear plan for conducting sensitivity studies or accounting for variable and model uncertainties. Will importance sampling be used to reduce the required number of code runs to assess the sensitivity to the various parameters? SLAD has defined S4 and SoSA as a user of this code, but no plan was apparent for how MUVES 3 will be fed into these force-on-force models. Will there be other users of this model? If so, how will user requirements be defined and implemented into the model? These considerations need to be included in the plan for moving forward with MUVES 3 development. MUVES 3 would be better presented as a cohesive initiative rather than as a number of components and notions for their future integration. MUVES 3 would also be well served by formal organization of its contributing professionals, with a single leader knowledgeable about all of its aspects. For an initiative of such ambition, a star technical leader would be invaluable for leading and coordinating a wide variety of supporting technical work and for representing the program to Army clients, leadership, and external reviews such as the one performed by this panel. OVERALL TECHNICAL QUALITY OF THE WORK The charge of the Army Research Laboratory Technical Assessment Board includes assessing the technical merit of the work performed in each of the ARL directorates against that of leading federal, university, and/or industrial laboratories both nationally and internationally. In contrast with the program portfolio of many of the other ARL directorates, the SLAD program portfolio includes relatively
OCR for page 67
2009-2010 Assessment of the Army Research Laboratory few applied research programs and no basic research program. The vast majority of SLAD programs are funded at much later stages in the DoD research, development, testing, and evaluation chain supporting acquisition and deployment programs. The SLAD portfolio and evaluation structure are based strongly in the directorate’s long history of ballistics-based vulnerability assessment. It is commendable that SLAD has broadened its program base to include assessment of communication, network, and information-processing vulnerability on the battlefield; nonetheless, the efficacy of the SLAD tool development methodology may not be sufficient to identify and stay ahead of the rapidly emerging threats to network-centric warfare in an irregular battlefield environment. The panel was asked to evaluate the performance of SLAD relative to how its people, infrastructure, and technical programs match up with the greater science and technology community. In past reviews as in the present assessment, this task has been and continues to be a source of angst, since SLAD seems rooted in a solid engineering paradigm. SLAD is excellent at understanding and applying sound scientific principles in developing engineering solutions to applied problems, but in order to remain effective in assessing vulnerabilities due to rapidly emerging threats in network-centric warfare, SLAD should work toward leading the way as the recognized innovator in the field in tool-development methodology. This requires a well-defined research program that focuses on expanding basic scientific principles into sound, comprehensive, flexible tools, and it includes full participation in the global research community through collaborations, interactions at conferences, documentation and publication, and staffing that includes people well trained in the scientific principles needed to advance the state of the art in tool development. It is clear from presentations to the panel that SLAD personnel are much more confident in the technical engineering programs than in those dealing with conceptual work or system modeling. The reason may be their comfort with using well-understood principles to deduce how physical systems will behave in specific situations, as opposed to their seeming discomfort in areas such as systems modeling, in which such well-understood scientific principles do not yet exist. The focus on research (applied and fundamental) within SLAD is relatively low. This is exacerbated by a low percentage of Ph.D.’s on the staff and relatively little collaboration with the external community, including academia and other research laboratories. The age pattern of the SLAD workforce is also a concern. SLAD should look toward postdoctoral, sabbatical, internship, and visiting scholar programs to provide healthy intellectual input and invigoration. In the short term, SLAD should expand its postdoctoral program to bring in personnel trained in areas of need. This might best be accomplished through the National Research Council program of which ARL is a member. More personnel from SLAD should be encouraged and incentivized to pursue advanced degrees (M.S., Ph.D., and post-degree sabbaticals) in relevant areas. Partnering with the Army Research Office to fund programs including 6.1 research and Multidisciplinary University Research Initiatives in areas of interest to SLAD would help in training and recruiting personnel. Establishing centers of excellence in universities in the vicinity of SLAD can also help in these areas. Collaboration and participation with the greater technical community are essential components of any research program. Collaboration enables the leveraging of SLAD facilities and expertise to gain diversity and expanded capabilities. There still does not seem to be significant collaboration with outside researchers. It seems that SLAD does not understand the depth of the panel’s comments on insularity; it is not just about attending conferences and working with other groups that do similar activities, which at times SLAD does well. Collaboration is also working closely with industry on joint projects, exploring new and innovative ideas by means of exploratory research projects with academia, or developing methods for enhancing survivability and lethality in joint projects with national laboratories. Telephone conference calls and briefings are a very weak form of collaboration, and many of the reported collaborative efforts sounded more like customer-client relationships. Although there were several indications of
OCR for page 68
2009-2010 Assessment of the Army Research Laboratory improvement in collaboration and partnering with other organizations, SLAD needs to provide a clear demonstration of the sharing of resources and assignments between parties of a defined collaboration. SLAD personnel need to document their work much better, including their innovations and test methodologies, instead of framing their documentation as internal progress reports. To become the leader in any field, they need to generalize their work enough to publish it, especially lessons learned! They should be keeping their innovations and test methodologies in some sort of ongoing documentation. “Publication” has been widely defined by SLAD to include a few open-literature publications, some technical reports of unknown provenance in regard to critical technical review, and many internal documents and presentations. There is a lack of scientific or technical peer review and thus scant traditional independent evidence of the rigor, efficacy, currency, correctness, and technical quality of the work. Entry into certain communities can be very difficult unless one is contributing to the research. Small Business Innovation Research programs provide opportunities for collaborating with industry to do unique research with relatively little effort. SLAD personnel and programs should be taking advantage of University Affiliated Research Centers, including those in biotechnology and in soldier survivability. SLAD should approach the Department of Homeland Security; they both should be able to collaborate in the area of survivability and lethality, possibly through other national laboratories. SLAD should work with its contractors to define DARPA-funded survivability programs. The Survivability and Lethality Analysis Directorate provides sound support of the assessment and evaluation of the survivability, lethality, and vulnerability of Army equipment and soldier systems to ensure that soldiers and systems can survive and function reliably in the full spectrum of battlefield environments. The directorate is well staffed with bright, creative professionals enthusiastic about their mission, and its facilities include state-of-the-art laboratories. SLAD has many opportunities to expand its testing and evaluation base through select program expansion aimed at developing tools and methodologies to broaden its analysis capabilities and to define and maintain the competitive edge required to be the Army’s primary source for SLV assessment. The SoSA and MUVES 3 programs continue to be of concern, but with appropriate focus these programs can grow to become foundations of the SLAD analysis capabilities.