Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 57
2007-2008 Assessment of the Army Research Laboratory 5 Survivability and Lethality Analysis Directorate INTRODUCTION The Survivability and Lethality Analysis Directorate (SLAD) was reviewed by the Panel on Survivability and Lethality Analysis of the Army Research Laboratory Technical Assessment Board (ARLTAB) during July 10-12, 2007, at White Sands Missile Range, New Mexico, and New Mexico State University (Physical Sciences Laboratory), and during July 22-24, 2008, at the Army Research Laboratory (ARL), Aberdeen Proving Ground, Maryland. SLAD is the U.S. Army’s primary source of survivability, lethality, and vulnerability (SLV) analysis and evaluation support with regard to major Army systems. SLAD’s general objective is to ensure that soldiers and systems can survive and function on the battlefield and to assess the degree to which Army systems are reliably lethal to enemy forces. Its mission includes SLV analysis and assessment through the entire life cycle of major Army systems, from development through acquisition to deployment and operation, in the context of a full spectrum of battlespace environments and threat forces, tactics, and systems. SLAD further provides advice to Army Headquarters, program executive officers, and subordinate program managers, as well as an array of other evaluators, system developers, and Army contractors, and other defense-oriented laboratories. Finally, SLAD is tasked with supporting special studies and inquiries motivated by and affecting current military operations. The SLAD portfolio comprises a very large number of relatively small tasks. It has been difficult for the Board to understand, until this cycle, exactly what proportion of the portfolio it is exposed to. Based on a work breakdown structure provided at the Board’s request, it appears that the Board is able to examine something less than 20 percent of the portfolio per year at a level of technical detail sufficient to assess the technical quality of the work. The Board’s emphasis has been on those tasks that show the strongest continuity of effort, that have the broadest influence on SLAD’s overall performance, and that most closely fit the charter of the Board.
OCR for page 58
2007-2008 Assessment of the Army Research Laboratory In contrast to most other directorates at ARL, SLAD’s portfolio includes relatively little applied research funding and no basic research funding. The overwhelming majority of SLAD funding is later in the Department of Defense (DoD) research, development, test and evaluation (RDT&E) chain, either provided by acquisition programs for SLV support or by RDT&E management support funding organic to ARL. The small fraction of applied research funding supporting SLAD is devoted to the development of tools, techniques, and methodologies required to undertake SLV analysis and assessment. This portfolio of funding reflects a relatively long period of stable SLV techniques, emphasizing ballistic survivability of armored systems and lethality of U.S. weapons systems against armored systems. SLAD is now necessarily supporting SLV analysis in a much broader and more rapidly evolving context, in which communications, networking, and information processing, rather than weapons systems per se, are believed to be the essential and sustaining advantage of U.S. military forces. A central concern of the Board remains the issue of whether the directorate’s funding portfolio will result in tools, techniques, and methodologies capable of providing the Army with the assessment capability needed under the emerging paradigm of network-centric warfare in an irregular battlespace. Tables A.1 and A.2 in Appendix A respectively show the funding profile and the staffing profile for SLAD. CHANGES SINCE THE PREVIOUS REVIEW The proportion of SLAD’s efforts comprising special studies and inquiries motivated by current operations has increased substantially since the advent of Operation Enduring Freedom and Operation Iraqi Freedom. The Board has been exposed to many of these efforts over the recent assessment cycle, necessarily at the expense of the rest of the portfolio. Most of these efforts are of relatively short duration, and although the theme of special operationally oriented studies is clear, continuity of effort and methodological progress are not something that is easily visible to the Board. SLAD’s contributions to the war effort have been competently performed, often under very serious time and resource constraints, and have apparently been significant influences on current operations and supporting acquisition. The Board has been exposed to a large number of these efforts and has consistently been impressed with the dedication and ingenuity of the staff involved. However, this work is not research per se and hence is difficult to evaluate in the context of industrial laboratory or academic research. In particular, it is extremely difficult to understand whether highly responsive and time-constrained analysis and engineering work is among the best in its field, since the field of comparison is necessarily limited. As noted previously, the SLAD portfolio is very granular, and the Board can sample a relatively small fraction of the individual tasks supported by SLAD. SLAD management has tended to emphasize the operationally oriented tasks and special studies in developing agendas for the assessment meetings. For the future, SLAD should consider leaving the assessment of this work primarily to SLAD’s operational customers, who are obviously in the best position to assess its impact and relevance, and to refocus the Board’s attention primarily on methodological progress, tool and infrastructure development, and the development of necessary new capabilities. To the extent that the Board continues to be exposed to the operationally oriented studies, it will focus primarily on the degree to which SLAD is engaging the external technical community to leverage previous work (both academic and industrial) to enhance the technical credibility and quality of its products. Beyond the change described above, SLAD has experienced other significant changes over the 2-year assessment period. One of the most significant of these was the establishment of Warfighter Survivability Branch within the Ballistics and Nuclear, Biological, Chemical Division. This branch was established to provide an organizational focus within SLAD for the soldier-focused portion of the survivability
OCR for page 59
2007-2008 Assessment of the Army Research Laboratory mission. The creation of the new branch brings together many of the disparate tasks supporting current operations; more importantly, it provides the potential for a future focus on soldier survivability in the context of longer-term acquisition programs. ARL and SLAD management expect SLAD funding to decrease in coming fiscal years. SLAD has continued to work on methodologies aimed at assessing the effectiveness of system of systems (SoS), which has been a continuing recommendation of the Board. However, the methodological development has focused increasingly on the System of Systems Survivability Simulation (S4), a fine-grained, event-driven simulation whose development is focused on human decision-making processes. Methodological development focusing on the Mission and Means Framework (MMF) has essentially stopped; the MMF is an approach to decomposing missions and systems for analytically identifying links between subsystems and mission performance. Finally, in 2006, ARL management indicated that it was willing to consider a Strategic Technology Initiative (STI) in the area of SoS methodology under SLAD leadership. The Board strongly recommended that SLAD avail itself of this opportunity and, in particular, that SLAD add a third leg to its platform of SoS methodologies. This third methodology should provide enough fidelity to enable a meaningful study of scenarios for identifying major system-level impacts of, for example, communications bandwidth; intelligence, surveillance, and reconnaissance (ISR); and precision weaponry, without modeling fine-grain entities such as packet-level communications or details of terrain. Developing this methodology in collaboration with an extramural team other than the team that has been developing the S4 tool would stimulate needed fresh perspectives in SoS analysis. In early 2008, a group consisting of members from the Panel on Survivability and Lethality Analysis and the Soldier Systems Panel met with ARL management on an STI proposal jointly prepared by SLAD and the Human Research and Engineering Directorate (HRED), and relying on S4 as the centerpiece of the approach. Qualified support was provided by the panel members for this proposal, which was not crisply defined. Progress on the STI was not assessed during this evaluation cycle, but the Board notes its disappointment that SLAD management declined to follow the Board’s recommendation to develop an additional approach to SoS analysis. ACCOMPLISHMENTS AND ADVANCEMENTS Response on Improvised Explosive Devices One continuing theme since early in Operation Iraqi Freedom has been SLAD’s support of counter-IED (improvised explosive device) operations and acquisition. Beginning with the IED counter electronic (ICE) device system, SLAD has collected an extensive data set and developed significant capabilities that are operationally relevant. The demonstration of the follow-on system, DICE, in 2007, and especially the influence that SLAD had on the rapid acquisition of mine-resistant ambush-protected (MRAP) vehicles in 2007 and 2008, are impressive. The technical quality of the work in this area is good, especially considering the time and resource constraints under which it takes place. As noted in the previous section, however, the evaluation of this work is most appropriately done in the context of operations, not research and development per se. In the MRAP work, many engineering assumptions were made to simplify analysis (consistent with the time constraints), and while some validation was performed using live-fire data, there was no presentation of statistical analysis of data scatter or error budget analysis to address remaining sources of uncertainty.
OCR for page 60
2007-2008 Assessment of the Army Research Laboratory Communications System Support SLAD has a long history of providing excellent vulnerability support for the most widely deployed radios used by ground forces—the single channel ground and airborne radio system, or SINCGARS.1 The breadth of the effort—starting from a focus on a very important issue and continuing with the depth of the analysis, the results, product improvements, the support for implementation, and rapid deployment—has been exemplary. SLAD and the SINCGARS program manager have been working together to determine the performance of improved SINCGARS against electronic warfare (EW) threats and to ensure that the improvements enhance antijam performance through comparisons with previous versions of SINCGARS. SLAD should be commended for the development of state-of-the-art laboratory tools emulating current and emerging EW threat systems (CSAL), and state-of-the-art automatic generation of radio performance curves (CEWIS). Laboratory investigations of radios without technical publications require fairly extensive investigation with levels of reverse engineering. Today’s technology enables fast and sophisticated EW threat systems; components are available off the shelf at very affordable cost. SLAD work on SINCGARS provides an excellent basis for the directorate’s becoming the leader in vulnerability assessment of the next generation of radios, that is, the Joint Tactical Radio System (JTRS). SLAD now has an encouraging agreement with the JTRS Joint Program Executive Office that it will get actual JTRS code (the Board expects that it will be for both soldier radio waveform [SRW] and wideband networking waveform [WNW]) and that it will not be continuing with surrogates or pre–Engineering Development Model JTRS code (e.g., SLICE). SLAD has also tried to use 802.11g as a surrogate; however, the orthogonal frequency division multiplexing (OFDM) in 802.11g differs from the OFDM approach in JTRS (e.g., for the WNW). This is especially important since results to date have shown that the issues previously identified and solved for SINCGARS have not been addressed in the pre-JTRS units that have been tested. If there are any further delays in getting actual code, SLAD can leverage other relationships to press for quick delivery and testing of actual JTRS code. Furthermore, considering the complexity and much broader application of JTRS as an Internet Protocol networked radio, the Board expects that there are many more vulnerability issues that need to be addressed than in SINCGARS or even SLICE. Information Operations SLAD achieved something of a breakthrough in communicating to the Board its support for Army information operations and assurance efforts during this assessment cycle. SLAD participates in ongoing experimentation and demonstration with respect to command, control, communications, computers, intelligence, surveillance, and reconnaissance (C4ISR) on-the-move capabilities. It supported information assurance (IA) testing and determination of compliance with Army regulations and policies, and it was tasked with the analysis and identification of critical system and/or network vulnerabilities that may be exploited by an adversary, as well as the development of mitigation strategies for all system and/or network vulnerabilities. SLAD employed hacker methodology and conducted penetration testing. It has developed an impressive tool set in INVA/DE.2 However, the 1 SINCGARS is a combat network radio currently used by U.S. and allied military forces. The radios, which handle voice and data communications, are designed to be reliable, secure, and easily maintained. Vehicle-mount, backpack, airborne, and handheld form factors are available. 2 INVA/DE is a tool set for performing audit and compliance testing against computer network attacks and computer network exploitation. INVA/DE consists of an integrated collection of public domain and SLAD-developed exploits/utilities resident on a
OCR for page 61
2007-2008 Assessment of the Army Research Laboratory specific results or lessons learned and the extent of the vulnerability assessment effort for this large-scale experiment were not clear, since they were not explained. This effort presents another potential opening to establish a position to drive issues rather than to react to them. The experience gained should be leveraged to develop an overall network vulnerability assessment methodology and to define specific metrics to evaluate performance in this area. Another area presented was SLAD’s black core analysis. The black core concept is fundamental to building the Core Backbone Department of Defense (DoD) network. It represents the design of multiple logical networks and associated interfaces and includes encryption and agreements on specific information needed for quality of service and routing across the boundary to enable the transport of these logical networks over a single physical network. In January 2006, the Army Chief Information Officer (CIO/G-6) selected ARL-SLAD as the most qualified Army IA organization and designated ARL-SLAD as the technical lead organization for system-of-systems network vulnerability assessment of the next generation of networks for ground forces. FY 2007 activities included modeling and simulation activities and vulnerability assessment of the Army’s Future Combat Systems (FCS) middleware (SOSCOE)3 source code analysis as well as an assessment of the FCS-proposed tactical public key infrastructure (PKI) and firewall analysis and risk assessment. However, the work that was specifically discussed was really an interface analysis to confirm or dispute other results (including FCS program manager and National Security Agency results), just barely impinging on the black core. This effort was not commensurate with the role assigned by the CIO/G-6. The methodology appeared to be quite simplistic, possibly because the black core design is not complete and the level of detail used in the modeling was necessarily coarse. While it is certainly useful to do a security analysis of an incomplete design, questions arise. In the opinion of the analyst, how much confidence should one place in the current analysis? Can the analysis be used incrementally as the basis for a more complete analysis when the full protocol is available? Although SLAD describes the risk of the design as low, what implementation details might cause the risk to increase? SLAD should use this effort and participation in the 3Star Network Vulnerability Analysis meetings as an opening to address the overall security architecture, focusing on obtaining funding for the needed level of resources. SLAD should develop work plans that are broader in scope and commensurate with the role that the CIO/G-6 assigned to SLAD. A highlight of the panel’s experience during this evaluation cycle was the live demonstration of the information operations exploitation laboratory at White Sands Missile Range, which drove home the quality of the overall effort. A few recommendations might be useful. The main one (repeated from a previous evaluation cycle) is to consider the testbed interface software developed at the University of Utah’s Emulab. That software is being used for a large, secure testbed at the University of Southern California’s Information Sciences Institute under the name Deter Lab. The interface and associated tools are powerful and easy to use, and Deter Lab would be interested in working with ARL on spreading the technology. This might even obviate physically separating the exploitation laboratory from the testing laboratory. Another suggestion is to do more outreach on preparing materials that might be useful to red (adversarial) teams under contract to DoD. The Defense Advanced Research Projects Agency (DARPA) laptop outfitted with ethernet and wireless network interfaces (Bluetooth, Wi-Fi, ZigBee, WiMax) with distributed capabilities. 3 SOSCOE is the foundation for FCS networked software including vehicle management systems, C4ISR, and soldier and unmanned air and ground systems. Just as an operating system on a computer allowing one to interact with resources and other computers, SOSCOE allows battlefield systems to communicate and interact with the unit of action.
OCR for page 62
2007-2008 Assessment of the Army Research Laboratory spends quite heavily on red teams, and establishing quality standards for red teams might be an area where ARL could help make a difference. Another recommendation that the Board has made in the past is that SLAD keep informed by attending research conferences such as the following: the IEEE Security and Privacy Conference, the Advanced Computing Systems Association (USENIX) Security Conference, the Internet Society’s Network and Distributed System Security Symposium, the ACM4 Computer and Communications Security Conference, the Applied Computer Security Applications Conference (ACSAC), and others. Attendance at such conferences would keep SLAD aware of the trends in defense and detection. SLAD should also present a research paper at one of the conferences, and the Board recommends ACSAC as a likely venue. System of Systems Survivability Simulation (S4) The System of Systems Survivability Simulation (S4) has made very significant progress since the previous evaluation cycle. The development of additional tools should significantly enhance the productivity of analysis using S4; the increased focus on more concrete productivity enhancements and the de-emphasis of highly abstract and difficult-to-communicate approaches such as formal method analysis are commendable. The appropriation of the Menard graphical summary of force structure is a key example; this approach will be much more appealing and intuitive to the Army operational audience who will need to understand the S4 results. Similarly, the scenario-building capability will greatly increase flexibility in analyzing diverse scenarios (as will be required in an increasingly irregular context). Several further developments will be required in order for S4 to fulfill its potential in supporting real decisions. These include validation of results, characterization of the fidelity of the platform and network information embodied in S4, and, perhaps most important, the development of design reference missions that are supported by Army leadership (probably embodied in the Training and Doctrine Command [TRADOC]). Validation and verification of S4 results are still at the stage of an operational expert rationalizing the observed simulation behavior. At a minimum, it is appropriate to expand the community of operators involved to include a more diverse experience base, particularly in the context of irregular operations. This may also be an appropriate point at which to formally engage TRADOC, which would facilitate closure on digital rights management (DRM). The Board recommended in 2007 that it was also time to consider the implications of much more powerful computers than have been used to run S4. Although SLAD noted to the Board that S4 had been implemented on supercomputer hardware as of July 2008, it seemed that this development may have been a box-checking exercise. The Board emphasizes that the use of high-performance computing may afford significant opportunities to exploit the S4 tool. In other fields, established communities of computational experts have found high-performance computing to be a disruptive development, opening new frontiers that had not been appreciated because existing computational platforms were adequate to perform conventional analyses. Finally, the Board notes that it previously urged SLAD not to commit exclusively to S4 as its principal SoS simulation tool, particularly if given the opportunity to apply additional resources from an ARL STI. First, although S4 has clearly advanced in utility and flexibility since previous exposures, there are still questions as to the self-consistency of its architecture, its robustness with respect to widely varying temporal and spatial granularity, and the extent to which sufficient runs can be both executed and analyzed to address broad SoS issues (particularly without the use of high-performance comput- 4 ACM (Association for Computing Machinery) is an educational and scientific society uniting the world’s computing educators, researchers, and professionals to inspire dialogue, share resources, and address the field’s challenges.
OCR for page 63
2007-2008 Assessment of the Army Research Laboratory ing). Second, SLAD’s collaboration footprint has been a primary concern of this Board for many years. Developing another significant collaboration, along different lines of analysis and with collaborators other than those at New Mexico State University, could essentially double that footprint. That opportunity seems to have been lost. This is particularly troublesome now that organic SLAD resources are being used to support the S4 effort, instead of its being funded through the congressionally directed program under which it began. Warfighter Survivability Branch As noted above, the new Warfighter Survivability Branch provides an organizational focus for soldier survivability. The branch is actively engaged in the counter-IED efforts discussed above. It is primarily responsible for the Operational Requirements-based Casualty Assessment (ORCA) program, designed to develop a tool set to characterize soldier injuries and estimate casualties and performance degradation produced by enemy munitions. Finally, the branch is integrally involved in the Joint Trauma Analysis and Prevention of Injury in Combat (JTAPIC) program. The JTAPIC program is led by the Army Medical Research and Materiel Command, with participation by the Armed Forces Medical Examiner, the Naval Health Research Center, the Institute of Surgical Research, the Aeromedical Research Laboratory, the National Ground Intelligence Center (NGIC), ARL, and the Program Manager for Soldier Equipment. The program combines medical, materiel, and operational intelligence data to improve the understanding of events that have caused casualties and to develop solutions that will mitigate future blast-related injuries. The SLAD role in the program is to re-create the reported casualty-generating event by modeling vehicle configuration and crew positions, to analyze and model threat characteristics through reverse engineering, to compare predicted injuries and platform vulnerabilities with the actual data, and to examine potential mitigation techniques. In spite of uncertainties in the reported events and difficulties caused by different terminologies used in the diverse communities, it has proven possible in many cases to apply such SLAD tools as Modular UNIX-based Vulnerability Estimation Suite (MUVES)-S2 and ORCA to model actual events with sufficient accuracy to explain the observed injuries and damage. The expertise of ARL in processing and analyzing fragments recovered in the field is an important contributor to the program. The JTAPIC program has accomplished impressive results under difficult time constraints. One case was described in which the feedback of analytical results was rapid enough to affect an ongoing operation, locating and neutralizing a specific threat. This performance was made possible by the close relationship of ARL to NGIC, and of NGIC to the operational forces. In another case, where damage to a vehicle led to casualties, the application of MUVES-S2 (including BRL-CAD5 and ORCA) permitted ARL to show that the use of curtains would reduce the spread of behind-armor debris and improve survivability in similar future incidents. Analysis of locally devised armor, applied to some vehicles in the field, showed an actual decrease in survivability, leading to recommendations to avoid this practice. The JTAPIC program is a clear demonstration that ARL, with its survivability tools and techniques, can affect not only future procurements of Army materiel but can provide rapid and valuable feedback to forces in the field. Compared with the situation encountered in the first Gulf War there has been a vast improvement in data gathering and subsequent processing, and ARL has been a key contributor. 5 BRL-CAD is a constructive solid geometry solid modeling computer-aided design (CAD) system. (The acronym “BRL,” for “Ballistic Research Laboratory,” refers to the former name of SLAD.) It includes an interactive geometry editor, ray-tracing support for graphics rendering and geometric analysis, computer network distributed framebuffer support, image-processing, and signal-processing tools.
OCR for page 64
2007-2008 Assessment of the Army Research Laboratory The analysis of data on all too many field incidents provides a unique opportunity for ARL to validate and refine its models. The success of this program, and SLAD’s contributions to it, offer a prime example of the benefits obtained by teaming between organizations with complementary areas of expertise. This is a case in which SLAD overcame its insular tendencies with significant results for its customers and similarly significant professional development of its staff. Extending the domain of collaboration may yield additional insight and data that can be used to improve the models in ORCA. There is a large biomechanical community including members within HRED and NASA and in sports medicine and orthopedics that could potentially be leveraged with respect to stress and trauma. BRL-CAD The presentation of the BRL-CAD effort was a model in terms of technical depth, external engagement with the broader scientific community, and articulation of the essence of BRL-CAD with a clarity that finally dispelled the uncertainties that many of the panel members have had for years. The BRL-CAD program overall is a 20-year success story, which has grown into a tool that is carefully tailored to the needs of many of SLAD’s analyses. The current plan has chosen its targets well. On the one hand, the move to incorporate non-uniform rational B-spline surfaces (NURB)-based boundary representation solid models will place BRL-CAD in the mainstream of modern CAD practice. It will greatly facilitate the development of a STEP translator,6 which in turn allows much easier access to CAD models from platform vendors, eliminating one major bottleneck in the analysis process. On the other hand, enhancements required to support MUVES-3, especially with regard to moving parts and dynamic geometry, are necessary if that program is to reach its goals. The technical approaches proposed are at or beyond the current state of the art. For example, BRL-CAD will implement a new and innovative surface-surface intersection algorithm from the literature, which promises the accuracy needed for building water-tight solid models for ray tracing. If successful, it will be a leapfrog technology. The program is the best model that the Board has seen at SLAD of two-way interaction with the external technical environment. It actively participates in the open-source community and is used and contributed to by first-rate academic researchers. The program sports an excellent Web site with a Wiki, as well as a good Wikipedia article (an idea that other programs could adopt as well). It seems clear that BRL-CAD is of interest to the graphics and CAD research communities, that it rates very highly in terms of performance, and that it has gained wide acceptance by a worldwide community. BRL-CAD uses a good mix between “making” where SLAD can add value (e.g., surface-surface intersection) and “buying” where it cannot (e.g., using an off-the-shelf STEP file parser from the National Institute of Standards and Technology), which acts as an effective force multiplier. However, some of its goals are technically challenging. For example, the revolutionary new surface-surface intersector may not work as advertised, or BRL-CAD’s ideas for geometry comparison algorithms to support versioning of dynamic geometry may not pan out. These have been recognized as hard problems in the CAD community for many years. Risk mitigation plans are essential going forward. One technical issue arose at the interface between MUVES-3 and BRL-CAD that the BRL-CAD group should pursue. The MUVES-3 presentation mentioned that the BRL-CAD ray tracing runs better 6 A number of different file formats can be used to transfer data between mechanical CAD systems (e.g., STEP, IGES, DXF, etc.).
OCR for page 65
2007-2008 Assessment of the Army Research Laboratory when single-threaded. This was such a peculiar result that the BRL-CAD group should be interested, since ray tracing is embarrassingly parallel. Another area that demands careful attention to risk is the open-source software approach. The opensource project is a great innovation because the software is tested by a huge community and because contributions can come from a large number of developers. The open-source approach is one that has helped many small startup companies; these companies make their money on managed services built on their free software, which gets continual improvements from a spectrum of developers. Open source does bring some risk, of course. It is imperative to have a security review of all software updates generated by the world external to ARL and to have acceptance criteria for the BRL-CAD application itself before using it on government computer networks. Assuming that ARL has constant monitoring of its networks, the BRL-CAD developers should develop monitoring rules for intrusion or exfiltration detection systems that are specific to BRL-CAD, such as well-known ports and message format validation. Further assuming that ARL has rules for using applications on classified networks, those rules should be reviewed specifically with respect to BRL-CAD. OPPORTUNITIES AND CHALLENGES MUVES-3 The Board was astonished, during its 2007 meeting at White Sands Missile Range, to learn that the initial operational capability for MUVES-3, SLAD’s primary integrative software environment and interface between its many engineering models, had been deferred by several years (leaving the existing MUVES-S2 as the primary production tool for at least 5 years), and that the defined scope for the project had been significantly redesigned. The Board viewed this as a classic software disaster and recommended in the strongest possible terms that SLAD management undertake a detailed project audit and either kill the program or replan it at the earliest opportunity. The Board did note that the correct metric had been identified for the success of this project, namely, the overall flow-time of an entire analysis project. Since this flow-time is typically dominated by setup costs, it is appropriate for the project to concentrate on streamlining and assisting setup rather than on minimizing execution time for the actual analysis. It was evident that SLAD management took the Board’s concerns to heart, making both organizational and programmatic changes to focus on the issues evident in the MUVES-3 effort. As a result, the project is in much better shape as the Board concludes this evaluation cycle, but the chasm still looms. The initial plan had a number of obvious risks. The programming language was not one that the developers were familiar with, the distributed architecture was new to the group, the existing system had to be maintained and enhanced while the new one was developed, there were few clear intermediate goals, and the overall objectives were somewhat murky. A possible interpretation of what had transpired might be described as a fundamental underestimation of the amount of necessary knowledge and effort. This caused the managers to underestimate the amount of project management effort that was necessary. As the risks resulted in frustrating performance problems and schedule impacts, managers could not see what was wrong. Ultimately, the team learned the programming environment, brought specialized system experts to the forefront, corrected their mistakes by using more appropriate software packages, put better project oversight processes into place, and brought the skeleton prototype up to an acceptable level of performance. The major performance problems seem to have been addressed in the current architecture. The benchmark data provide the validation of the performance goals, and the team’s confidence in the architecture
OCR for page 66
2007-2008 Assessment of the Army Research Laboratory seems well founded. However, it is disconcerting to hear that the performance has been deemed good enough (with no particular justification) and that the most basic measure of performance comparison of MUVES-S2 to MUVES-3 (single-processor problem-solving speed) is buried in more complicated scenarios. The Board understands that MUVES-S2 lacked the important scalability property of being able to distribute the processing over multiple central processing units and multiple machines, but that does not mean that the basic speed comparison is entirely irrelevant. The parts that have been changed seem to be well-known risk factors for distributed architectures and/or high-performance systems, and the development team and its management should take time to understand why the risks were underestimated and why it took so long to address them. It may be necessary to make changes in parts of the architecture that have yet to be realized, so it is important that the team be prepared to deal with problems quickly and effectively. One problem might have been that the team relied on many Java services that are attractive in their power and ease of use without understanding their performance aspects. Because there was (apparently) no detailed breakdown of the design into implied performance requirements (such as the time for a remote procedure invocation), it was not possible to determine if a service would be suitable for MUVES-3 without building a prototype system. A detailed performance model would have allowed developers simply to measure the remote procedure invocation overhead and have an immediate decision about suitability. So, it seems that the team had to build one prototype system after another, searching for a mix of Java packages that finally meshed. In a positive sense it can be said that everyone probably learned a lot, but this is a far cry from the way that R&D works in industry. The architecture now relies on two important pieces of open-source software for the Java environment: Rio (for the distributed processing) and Java Spaces (for the run-time data). Overall, it is good to see SLAD taking advantage of the strengths of the large open-source projects and that it is a project contributor (to Rio). This must be done with careful thought as to the certification of the resulting product for use in sensitive or classified environments and to the security of ARL computing resources. The Board was not provided any supporting data about how the design changes, such as using the peer-to-peer system or batching the BRL-CAD requests, improved performance over the master-worker scheme. The Board heard in passing that BRL-CAD runs better when single-threaded; that seemed interesting and worth investigating. Is there anything in the Iteration 3 investigations that yields insight into how to allocate resources for analyses that test the performance boundaries with respect to resources? Can a systems expert help an analyst determine how many machines of what type are necessary for large problems? Are there internal parameters that can be tweaked by an expert to improve the run-time of an analysis (note, the tweaking might be done at the inception of the analysis; the developers seemed to feel that changes to parameters could not be made during run-time, but the issue of pre-run-time configuration tuning was not addressed)? The demonstration left an uncertain definition of what the Geometry Service in the architectural diagrams entails. The notes on the presentation slides say that it is the ray-tracing engine (BRL-CAD). However, the Geometry Service was later described by SLAD as “to be determined.” When the panel asked how the demonstration was done absent the Geometry Service, the answer was that it just read a file. If the panel saw a demonstration that did not actually invoke the BRL-CAD ray-tracer, the panel should have been informed up-front. The panel members remain unsure on this question and think that the Geometry Service might simply be a service that takes an object name and finds its geometric description in a database. There are other ways to approach the architecture issues, and it was not made clear by SLAD why distributed computing by way of commodity servers and workstations was selected as the computing base. Was that a deliberate decision or happenstance? High-performance scientific databases for scientific
OCR for page 67
2007-2008 Assessment of the Army Research Laboratory computation run quite well on cluster machines, for example. Did the team do a cost-benefit analysis of the hardware base? Would SLAD have been better served by simpler software on more expensive machines? Although it was encouraging to hear that the developers consider their architecture stable (again), it would have been helpful to provide a better understanding of the risks involved if it is necessary to tweak things (again). The Board suspects that SLAD does not have much current experience developing complex software systems. BRL-CAD was developed partly as the result of one dynamic and charismatic individual, and it has been a good foundation for further development. MUVES seems to lack a similar kind of star technical leader. There are many ways to approach the project management problem for a system like MUVES-3, and SLAD could have tried a small tiger-team approach for 9 months as a possible alternative. Or, if it had clear performance and functional criteria, it could have contracted out the initial architecture design and prototyping. Perhaps the design team was too fragmented initially, working part time on MUVES-S2, part time on MUVES-3; if so, the organizational unification of the two teams should help. Perhaps the problem was inherently more complicated than originally realized—could more frequent reviews and benchmarks have delineated the complications earlier? Some years ago the project manager of MUVES told the Board that Java was the language of choice in large part because recent college graduates had a strong preference for Java over C or C++. The Board wonders if the commitment to Java resulted in the hiring and retaining of recent graduates with computer science degrees. Does MUVES-3 have any need for security features such as authentication, access control, and encrypted data (either in a repository or on-the-wire)? SLAD reported that authentication was being considered, but are there elucidated security requirements that can be mapped to an architecture? It is encouraging that the management is working with a software development risk-mitigation consultant and there is hope that this work leads to a deeper understanding of how to control the development cycle for complex software projects in the future. The Management Review Board seems like a good idea, but it remains to be seen if it can handle the fundamental design problems that still lie ahead. An important question is whether SLAD learned any lessons that give it confidence that it can approach similarly complex software systems in the future. Correspondingly, SLAD should document the lessons learned that can inform future significant efforts. S4, SLAD’s other major software development effort, is likely to provide many further opportunities to apply the lessons learned in the MUVES-3 project. The methods and results of the MUVES-3 performance analysis and design decisions would probably make an interesting paper for a distribution systems applied technology conference. Could the developers set a goal of producing at least one paper for publication each year? Finally, the future is not clear with regard to the performance improvement in overall analysis time that currently motivates the MUVES-3 effort. This subject was addressed in the 2008 MUVES-3 presentation to the panel in which MUVES-S2 data were presented, indicating that less than 15 percent of the total analysis process time (measured as effort in person-weeks) was represented by run-time and deliverables.7 The average percentage of preparation effort and the MUVES-3 features aimed at reduction in preparation time for five process phases in seven types of analysis were identified (slide 82), as shown in Table 5.1. No attempt has been made in the figures shown in Table 5.1 to weight the percentages by the importance or relative numbers of analysis types, which range from air vehicle survivability to spare 7 Mark Burdeshaw, Army Research Laboratory, “MUVES 3 Overview,” presentation to the Army Research Laboratory Technical Assessment Board, Aberdeen Proving Ground, Maryland, July 23, 2008, Slides 78 to 90.
OCR for page 68
2007-2008 Assessment of the Army Research Laboratory TABLE 5.1 Average MUVES-S2 Preparation Effort, by Process Phase, and MUVES-3 Features Aimed at Reduction in Preparation Time Process % Time MUVES-3 Feature Target description 24.4 BRL-CAD Geometry Service Criticality analysis 22.4 GUI: Fault Tree Editor Behind-armor debris 2.4 Collaborative Environment Input data 13.4 Collaborative Environment Pre-run review 22.2 Collaborative Environment, MRB, Test Total preparation 84.9 SOURCE: Based on Mark Burdeshaw, Army Research Laboratory, “MUVES 3 Overview,” presentation to the Army Research Laboratory Technical Assessment Board, Aberdeen Proving Ground, Maryland, July 23, 2008, Slides 78 to 90. parts analysis. It is clear, however, that preparation for the run requires the major fraction of effort in the MUVES-S2 system. What is not clear in SLAD’s presentation is the nature of the corrective action to be incorporated in the MUVES-3 program. Except for the improvement in providing a more user-friendly BRL-CAD input system, steps to be taken and the economies to be realized in the MUVES-3 preparation process remain unclear. The Fault Tree Editor and Collaborative Environment have not been defined to the point where a huge reduction in workload can be predicted with confidence. Active Protection Systems The active protection systems (APSs) will become a critically important element of ground platform survivability in the coming years. In fact, an argument can be made that the U.S. Army is late in this area. The threat environment of future battlefields is expected to be even more complex and more lethal than it is today. The ability of Army acquisition programs to evaluate operational requirements relating to APSs and to assess candidate system performance and suitability is an important aspect of fielding APS capabilities. SLAD has the expertise to enhance APS modeling and simulation capabilities in several key areas. SLAD personnel presented a methodological framework for evaluating APS protection of ground vehicles. SLAD’s recognizing that APS modeling needs to be approached in an end-to-end fashion instead of piecemeal is commendable. SLAD’s using the existing Army Research, Development, and Engineering Command (RDECOM) APS model instead of building a new model is a cost-effective approach that will provide needed modeling and simulation capabilities sooner is an appropriate choice. The Board was presented a threat analysis, including the operational aspects of multiple, simultaneous threats and threat signature modeling. SLAD also reviewed work being done to model threat warning systems with respect to evaluating potential false alarms, an analysis of threat characteristics versus other battlefield signature sources, and a possible approach to discriminating threats from battlefield clutter. However, the Board was not shown a system-level analysis. So, whereas SLAD is making incremental augmentation of the RDECOM model with improved representation of phenomenology, the Board was not able to assess whether all of the critical elements affecting APS performance have been identified and if they were being adequately addressed. In fact, there seems to be at least one critical aspect of APS performance that is not being addressed—the overall time line associated with the threat engagement sequence. The HRED IMPRINT model may be of use in estimating task times in this sequence. APS performance will largely be determined by its ability to launch active countermeasures
OCR for page 69
2007-2008 Assessment of the Army Research Laboratory in a timely manner while maintaining a very low false-alarm rate. SLAD should focus on the APS response time line issue. SLAD appears to be behind the leading edge in mid-infrared threat warning sensor phenomenology. This may be an area where SLAD could benefit from collaboration with other government organizations and industry. There is considerable work being done on airborne missile warning systems that should be directly applicable. In addition, the lack of actual algorithms used by prime contractors represents a major obstacle to SLAD’s focusing efforts in the highest-impact areas. The APS project appears to be a candidate for application of the SoS concept, which would allow the individual elements of the threat-sensor-response process to be addressed in a comprehensive way, bringing in the depth of SLAD knowledge and modeling of the battlefield environment. The APS would provide an example of how this approach, presented by SLAD as a rather abstract concept, could be brought to bear on a critical problem that until now has been attacked piecemeal by various contractors and RDECOM, with only peripheral participation by ARL. Mission and Means Framework The Mission and Means Framework has been applied successfully in a number of different instances since its inception. Although the panel members are not unanimous in their support of MMF as a methodological advance, use within the SLAD customer set is a positive indicator. However, SLAD does not appear to be devoting additional effort toward developing the methodology and addressing previously noted technical shortfalls such as an explicit approach to accommodating stochasticity. Indeed, the applications of MMF shown to the Board were predominantly performed by Dynamics Research Corporation (DRC), an independent defense contractor. The use of DRC for production while SLAD continued development of methodology would be a defensible approach, given limited human resources. However, SLAD’s abandonment of an immature methodology to an independent contractor holds reputation risks for ARL. If the contractor applied the methodology inappropriately, would it take responsibility for errors, or would it note that it was just deploying an SLAD-developed tool? In any case, if SLAD discontinues MMF development, it will then be completely dependent on a single tool, S4, for SoS methodology. OVERALL TECHNICAL QUALITY OF THE WORK As noted previously, the work performed by SLAD is technically competent, especially in the context of rapid response to operational needs where time and resources are severely constrained. The SLAD staff has extensive experience with this type of analysis and possesses great domain knowledge concerning specialized systems; they are qualified for the work. In terms of significant, multiyear efforts requiring significant development of models, tools, and analysis methodologies, the verdict is mixed. BRL-CAD is proof that SLAD is capable of significant, even state-of-the-art software development embodying complicated technical underpinnings. MUVES-S2 demonstrates that SLAD is capable of integrating many disparate tools to create a metatool of fairly broad applicability. However, the principal lesson of the MUVES-3 project is that SLAD failed to develop a management construct, including requirements definition, milestones, reviews, and diagnostics, up to the task of modernizing its existing software environment. This shortfall in systems engineering is far from unique in the national security technical community, and it is encouraging that SLAD management is taking the lessons learned seriously.
OCR for page 70
2007-2008 Assessment of the Army Research Laboratory Another area of concern to the Board is whether the extremely granular nature of the SLAD portfolio, compounded by the many rapid-response tasks generated in the course of the war effort, will ever permit SLAD staff to develop the skills necessary to perform larger-scale, longer-term development or analysis efforts. This is particularly bothersome in the context of the Army’s need for SoS analysis methodology for network-centric operations on an irregular battlefield. SLAD is currently depending primarily on a single team of academic collaborators at New Mexico State University to provide the tool set needed for that transformational and essential task. This should be matter of significant concern to ARL management and Army leadership. In contrast, the Board has been frustrated for years about the insularity of SLAD staff with regard to the larger scientific community and the reticence of SLAD to professionally engage with that community. SLAD staff are clearly frustrated with this message, and to the extent that ARL resource management serves as a constraint (in providing travel and conference funds), justifiably so. However, for SLAD staff to develop professionally as their counterparts in other federal, university, and industrial laboratories do nationally and internationally, there is really no alternative to more extensive and prolonged professional interaction. SLAD has new mission and vision statements, notably including the aspiration for the SLAD staff to be unsurpassed in dedication and willing to do whatever it takes to support the warfighters. The Board has long been impressed by the dedication of SLAD staff in supporting our nation’s warriors, especially when measured in terms of long hours, personal risk, and making do with available resources. The Board exhorts SLAD to be similarly dedicated and willing to do whatever it takes to better engage the external scientific community to leverage its findings and results in furthering the SLAD mission. The lesson of the excellent results obtained through SLAD partnership in JTAPIC should help motivate a much broader engagement than currently exists (significant counterexamples such as BRL-CAD notwithstanding). That the SLAD portfolio does not lend itself as readily as those of the other directorates within ARL to external collaboration, publication, and conference participation should not serve as an excuse to an appropriately motivated staff. SLAD insularity significantly compromises the directorate’s ability to leverage academic and commercial developments, especially in such areas as computer and network security, biomechanics, and software development, where investment outside the Army dwarfs organic resources and capabilities. Academic collaboration is also a key to strategic workforce development, since the exposure of graduate and undergraduate students to highly relevant applied research and development may enhance SLAD’s recruiting pool. SLAD staff has shown increasing involvement in conferences and professional societies in recent years. Funding constraints and demands for support of current military operations appear to have recently blunted this improving trend over the current assessment period. SLAD and ARL management should resist the temptation to allow current short-term pressures to cause a relaxation into a more insular posture. Strategic workforce development, as well as longer-term Army needs, demand that SLAD staff seek professional enrichment and involvement in the broader technical community.