8

Analysis and Assessment

The Panel on Assessment and Analysis at the Army Research Laboratory (ARL) conducted a review on July 11-13, 2017, at the Aberdeen Proving Ground (APG) in Maryland. The review was on three of ARL’s Analysis and Assessment Campaign core campaign enablers (CCEs)—ballistics survivability, vulnerability, and lethality (BSVL); personnel survivability (PS); and human systems integration (HSI). This chapter provides an evaluation of that work.

ARL’s Analysis and Assessment (A&A) Campaign provides tools that increase awareness of material capabilities, assesses the survivability and lethality of Army systems, and both improves and simplifies the Army’s decision making. The work in the BSVL program provides the analysis and assessment capability to develop efficient means to understand and influence the factors that reduce vulnerability and increase lethality of Army ground and air combat systems. The work in the PS program develops and applies methodologies and tools to model, analyze, and predict effects of weapons against personnel and soldier protective systems. The work in the HSI program provides methodologies for the assessment of cognitive and physical human performance trade-offs and workload in support of human systems integration.

The Panel on Assessment and Analysis at the Army Research Laboratory conducted a review on July 25-27, 2018, at the U.S. Army’s White Sands Missile Range (WSMR) in New Mexico. This review was on three of ARL’s A&A Campaign CCEs—electronic warfare (EW) survivability, lethality, and vulnerability (SLV); cyber SLV; and complex adaptive systems analysis (CASA). The panel was positively impressed with the changes that were made in response to the recommendations in the previous assessment. The A&A Campaign is commended for its openness and willingness to implement recommendations given by the previous panel.

ARL’s A&A Campaign provides guidance for the development of tools, techniques, and methodologies that increase awareness of material capabilities, assess the survivability and lethality of Army systems, and both improve and simplify the Army’s decision making. The work in the EW SLV CCE

provides the analysis and assessment capability to operate in an increasingly complex, heterogeneous, and contested electromagnetic environment (EME). The work in the Cyber SLV CCE provides analyses of Army systems that are in acquisition, or that are currently operational, in order to mitigate systems vulnerabilities and prevent future susceptibilities. The work in the CASA CCE is aimed at assessing multiple systems interactions, operational contexts, and networks to help the Army make informed decisions regarding the survivability of complex adaptive systems, including humans and autonomous systems.

BALLISTICS SURVIVABILITY, VULNERABILITY, AND LETHALITY

Accomplishments and Advancements

The BSVL team is a high-performing group with a wealth of experience and capabilities. The team is a key contributor to the Army’s system analysis, acquisition, and test and evaluation communities. It has a large set of responsibilities that it performs to a high level within the limits of the resources provided. Specific accomplishments that stood out in the work were: (1) improvements in visualization and augmented reality; (2) computing efficiency efforts; (3) underbody blast advancements; and (4) automation of manual data collection.

BSVL studies generate a large set of data when assessing various threats, aspect angles, threat ranges, threat aim points, response variability, and so on. Previously, study results had been presented in difficult to understand and explain formats. By using the increased computing power and advances in data display tools, the BSVL team has developed visualization and augmented reality methods that can more effectively support Army decisions. The demonstration of augmented reality in a ballistic vulnerability analysis is a good example of applying existing technology to this area to make the outputs more easily understood by analysts, customers, and decision makers.

The work on profiling the modular UNIX-based vulnerability estimation suite (MUVES) and distributing the code over parallel processors to gain a factor of 2-8 decrease in computational time is a significant advancement, particularly as more complexity will have to be added to this code. Two areas demonstrated significant efficiencies in preparing inputs for the MUVES model—scanning of the systems to be analyzed in order to automate portions of the building of the geometric target descriptions, and the tablet-based range-data acquisition computer toolkit. These both represent significant reduction in human labor and make the process more efficient.

BSVL studies are computationally intensive, so historically it typically took weeks or even months to complete a set of BSVL estimates for an Army customer. Calculations are beginning to take advantage of the Army high-performance computing capability, which together with advances in parallel computing, will substantially reduce the time required to perform BSVL studies. For example, high-resolution finite-element modeling of underbody blast is taking advantage of the high-performance computing resources so that it can provide a useful tool for designing underbody protection. Once these advances are fully implemented in MUVES, Army efforts can be completed more quickly and more efficiently. This advance, once fully implemented, will be a significant improvement, since it will increase the range of possible outcomes that can be considered in modeling and simulation studies.

The underbody blast modeling effort is a significant advancement. Underbody blasts from improvised explosive devices (IEDs) are a major concern, and the BSVL team is developing tools to predict damage and casualities due to underbody blasts. Without a validated and verified tool to predict the effects of underbody blasts, the Army has been forced to conduct very many underbody blast tests in an attempt to improve underbody blast protection and reduce casualties. This is a slow and costly process.

Once the emerging tools are verified and validated, they can be put into production. The Army will then be able to more quickly, economically, and effectively address underbody blast damage.

The work on assessing technologies in development on helicopter blades and technologies that depend on Global Positioning System (GPS) data in a denied environment demonstrated the payoff of conducting these analyses earlier in the acquisition process—that is, “moving left.” Because these had very different impacts to Army systems, the A&A Campaign needs to work closely with the Assistant Secretary of the Army for Acquisition, Logistics, and Technology on prioritization of the programs in order to have the widest impact for the Army.

BSVL studies have relied on wooden witness plates for arena tests and crude plywood mannequins representing personnel to absorb impacts of fragments, spall, projectiles, and so on for lethality and vulnerability assessments. The damage in these wood witness damage collectors is measured by hand for location and by probes for depth, angle of entry, fragment mass, and so on. These measurements are recorded by hand on data sheets and subsequently entered into computer databases, again by hand, for documentation of the results and further BSVL studies. The review team was shown efforts to utilize a digital “wand” to probe and automatically record data. This offers the possibility to improve the accuracy and precision of recording results, to reduce the time required, and to minimize the potential for manual data handling errors. This is a worthwhile effort that ought to be implemented as quickly as possible and expanded to automate as much of the current manual effort as possible.

Challenges and Opportunities

Recognizing technologies that require new modeling methodologies and having the tools developed in time to support evaluation is an increasing problem, and the A&A Campaign is falling further behind in keeping up with the technical and environmental complexities. For example, the work on multihit and underbody protection modeling, while very worthwhile efforts, were too late to impact current and near-future designs. A validated model of underbody blast is important to have as soon as possible to avoid future designs that are very vulnerable to underbody blasts. As another example where the A&A Campaign is behind, in order to lessen combat vehicle weight, ceramic armors have been proposed as a replacement for metal-based armors, but ceramic armor is very sensitive to multihit damage. The A&A Campaign is just now initiating efforts to develop methodologies and tools to address multihit damage to ceramic armor. These armor technologies were developed for the Future Concepts Systems more than 10 years ago.

There is a need for the A&A Campaign to move to multithreat analyses. For example, an active protection system can have targeting and tracking sensors on a vehicle that is being protected. These are vulnerable to electronic warfare and cyberattack. Hence, an analysis of the effectiveness of the system requires interdependent analysis of electronics, cyber, and ballistic interactions. Other technologies where electronic and ballistic interactions are not independent include robotics, manned and unmanned teaming systems, and smart munitions. It will be important to analyze these as a coupled system.

Not only are the technologies increasing in complexity but also the environment in which the Army is fighting and may fight in the future includes complex urban environments with a crowded electromagnetic environment, variable atmospheric conditions, and close proximity between the enemy and noncombatants. The electromagnetic environment can be generated only in computer simulations, as there is no comparable environment where the United States can do open-air tests. These added complexities will need to drive changes in the methodology used to produce ballistic survivability and lethality information, and multithreat analyses. There is a need to include multithreats (ballistic, electromagnetic, and cyber) and complex environments in ballistic survivability and lethality analyses. Currently, such

information is averaged too early to be most useful in subsequent studies. The increases in complexity will drive a need to take full advantage of the Department of Defense (DoD) high-performance computing (HPC) resources.

At present, there is a lack of active duty military personnel in the BSVL team. Military experience and expertise are vital in developing study plans and understanding results in a military combat context so that efforts better relate to the military environment and the impact of results on the military situation can be more readily understood by the Army community. This lack of military personnel impacts the value of the BSVL team’s efforts for the Army.

The BSVL team has been reduced in personnel by one-third from the peak of a few years ago. Nevertheless, there continues to be a significant volume of study requests by the Army community to be addressed, as well as significant tool improvement and development required on current and emerging systems, technologies, threats, and so on.

While the efforts being introduced to address critical technology, environment, and threat changes in areas such as manned-unmanned teaming, active protection systems, GPS-denied environments, multidomain integration, behind-armor blunt trauma (BABT), multihit damage/effects, and so on are a positive development, most (if not all) of these concerns have been known for substantial periods of time. The BSVL team needs to stay current with ongoing changes and address emerging and new technologies and threats with new and modified tools to address emerging challenges. The efforts in the past have not always been successful in keeping up as required to support the Army community needs.

The use of HPC resources has improved the BSVL team efforts. However, there are opportunities to make even more extensive use of HPC resources to improve BSVL tools, productivity, and overall contributions to the Army community they support. Using HPC resources to a much greater extent would have a multiplying effect on the overall contributions to the Army community and on how BSVL tools are utilized.

While the efforts on data acquisition are an important first step, methods to fully automate this process and eliminate touch-labor are important and needed. There may be high-resolution imaging techniques to gain all the information required.

While the review team noted some accomplishments and advancements in computing for displaying data and increasing speed of results, no mention was made of any effort to address a basic factor in the whole BSVL approach. Shot lines are modeled as mathematical nondimensional straight line rays, whereas physical projectiles, fragments, jets, and so on obviously have three-dimensional (3D) shapes that interact with armor, components, fluids, and crew. While it is not fully understood what the impact this simplifying assumption might have on BSVL studies, this deviation of the model from reality warrants some thought, as it is a long-standing concern about a basic tenet of the BSVL tools.

PERSONNEL SURVIVABILITY

Accomplishments and Advancements

In the ARL A&A activities in personnel survivability, there was a strong focus on performing analyses at the earliest suitable time during the life cycle of programs, early enough that positive or adverse findings and analysis results could be used to inform and to improve decision making. There was clear understanding that this “move to the left” is a trade-off between informing programs at an early stage and multiple, potentially costly analyses for programs prior to major decision points. The implication is that “move to the left” does not necessarily imply the need for early-stage analysis to be performed for every program. The choice depends on cost and program importance.

Statistical analyses underpin most or all of ARL’s A&A missions. In light of the importance of statistics, the presentations showed a laudable emphasis on statistical concepts beyond simple means that are often used in military programs, and the need to understand confidence intervals and variance in the analysis of programs. As a principal part of statistical analyses, the presentations especially emphasized proper selection of appropriate analyses. This is of significance, since ordinal data has been misused as integer-continuous data in engineering statistical analyses. The ARL team clearly recognizes this issue (Abbreviated Injury Scale [AIS] 2 is in no sense twice as severe as an AIS 1) and specifically underlined the use of the rich field of nonparametric statistics for ordinal data.

BABT and behind-helmet blunt trauma injury risk assessments are long-standing and urgently needed capabilities both for fundamental research and in analysis and assessment missions. This is of crucial importance for informing weight trade-offs with deforming passive momentum defeat mechanism personal protection in both the head and torso. The current focus on BABT injuries from the helmet is commendable and appears to be strongly supported. The team has early-career researchers who are engaged and are knowledgeable about previous limited work in the field. ARL has taken a good approach for the physiological experimental work, combining high-speed X-ray imaging with state-of-the-art and exploratory sensor technologies. This is an example of outstanding work in an area where there has been an urgent need for over a decade. This work ought to continue and be extended to thoracic BABT, for midchest (mediastinal), abdominal, and spinal impacts.

An additional area of commendable progress that supports fundamental analysis of personnel is ARL’s development of capabilities for imaging and ongoing generation of human model data, particularly a range of anthropometry data from medical imaging. Promising aspects include acquisition of imaging data from living people. In addition, presentations outlined the development of dynamic surface imaging for occupational and personnel survivability assessments inside vehicles, using commercial 3D imaging systems. Full development and use of this capability can provide additional valuable information in assessing dynamic positioning for both use assessments and realistic dynamic locations for vulnerability assessments. There is a continuing effort to strengthen and develop high-speed flash X-ray imaging for dynamic blast and ballistic assessments; this is an important tool for assessment of injury from physical models to physiological models, providing valuable feedback on the high-rate dynamic response of rapidly deforming personal protection.

The ongoing development of enabling technology for detailed and realistic finite element models of humans for personnel vulnerability assessments is important. These efforts include good use of high-performance computing support from the core supercomputing facilities of the army. These efforts include support for fundamental human tissue property characterization building on previous work both at ARL and in academia. Efforts in this direction need to be actively encouraged for both midterm and near-term research supporting A&A efforts. Such efforts are enabling for future generations of high-fidelity models, which may support substantial improvements in both vehicle and personal protection.

For biomedical research, the personal survivability group has good, modern facilities. These include live-fire ranges for explosives, fragmenting weapons, ballistics, and available medical assessments of various types. The facilities include a mobile clinical computed tomography (CT) scanner, and available computational facilities include substantial Army supercomputer resources. In recent years, ARL has developed the ability to perform needed biological experimentation using various physiological and cadaveric models. In parallel with the near- and midterm development of analysis tools using these models, ARL continues to refine simple physical models such as efforts to replace existing plywood dummies with more anthropomorphic and physically responding manikins. Progress in this area needs to be maintained and improved with core efforts and collaborative efforts among potential research partners to support A&A efforts.

Challenges and Opportunities

The principal challenges and opportunities arise with the essential tools of ARL’s A&A Campaign—MUVES and operational-requirements-based casualty assessment (ORCA). Both are indispensable, but both need to be further developed and validated in several areas over a near-term to long-term time frame. For MUVES, validation and extension efforts are clear and critical; no other organization will develop a robust analysis tool central to the ARL mission. For ORCA, although long-needed efforts have begun to improve basic aspects of components of the framework, principally increasing resolution for the ComputerMan component, planning and core funding for additional improvements and validation are urgently needed. It is important that the development of these tools does not occur solely on an ad hoc basis when responding to immediate A&A tasks. The scope of these tools for Army development and planning is so broad that development needs to occur on a deliberate and well-funded basis. Long-term tool development of both MUVES and ORCA have similar issues, the priorities are not clear on any time frame beyond current development. The challenge is to develop an organic plan for long-term development of both tools that incorporates current threats and addresses risk assessments for threats to personnel in the intermediate and long term. Desirable developments include increases in computational efficiency, especially for parallel use. Much of the use of ORCA and MUVES are essentially fully parallel assessments. Some of this development has started. In addition, the provision for closer and more organic integration of the individual tools within ORCA would be highly desirable. It would also be extremely valuable to add BABT capabilities for personal protective equipment (PPE), particularly helmet and thoracic body armor. It is essential that ARL provide consistent long-term planning and funding for this effort.

Another key area of challenges and opportunities for personnel vulnerability A&A is in the medical arena. Building on a substantially improved collection of battlefield data by Joint Trauma Analysis and Prevention of Injury in Combat (JTAPIC) and the Joint Trauma Program, ARL analysts have the potential for supporting substantial near-term, midterm, and long-term improvements in survivability and injury risk assessments. However, substantial challenges impede the most efficient use of this data. Key among these challenges is the difficulty of obtaining detailed medical data (principally from JTAPIC and U.S. Army Medical Research and Materiel Command [MRMC]) beyond the coded injury descriptions provided in redacted form, apparently based on the incorrect assumption by the medical community that detailed medical information is not useful for engineering survivability risk assessments. This is an essential limitation (or domain conflict), since effective personnel risk assessments cannot be performed without granular knowledge of injuries well beyond AIS coding categories. Indeed, such detailed medical information offers the opportunity to inform and develop more effective risk models based on battlefield functional capacity, rather than simple risk of injury or fatality reflected in the AIS scores alone.

A related challenge for ARL in developing appropriately granular personal vulnerability A&A for military programs is the continued use of injury coding schemes such as AIS to classify injuries. In this system, injuries are ranked using an ordinal scale from 1 (minor injuries) to 6 (maximum). This scaling, developed principally as a threat to life scale in the automobile biomedical community, has been extended to military injuries. However, this extension cannot fully remediate the problem with its use in the military domain; threat to life is one potentially important aspect in program A&A, but it is not necessarily paramount. For example, a survivable midshaft femur fracture (AIS 3) may be as militarily significant as a likely fatal injury from an aortic laceration (AIS 5), depending on the needed functional capacity for the mission. For future efforts, it is important to both collect and assess injury data with higher granularity and with an eye toward the use of military injury data as an assessment of the mission-oriented functional capacity on the battlefield. The benefits of using additional available

battlefield injury data for various threats spans both the development of research tools for A&A and their use in various operational domains.

One long-standing challenge is in the development of models for BABT, especially for the head and torso. This development includes the refinement or replacement of existing physical models for ballistic BABT with various fragments and ballistic threats. This subject has been addressed by several previous National Academies studies, and the essential recommendations have not been addressed even after the passage of a number of years. These previous National Academies studies emphasize the limitations of both the physical models and the underlying injury biomechanics. For example, the expedient “Prather model” for ballistic BABT injury applied to soft VIP body armor for handgun threats in the 1970s was never intended to apply to rifle-round ballistic threats behind hard body armor. As noted earlier, laudable investigations of helmet BABT in physiological models has begun, but the development of such models and physical surrogates has been needed from the beginning of the development of aramid composites for ballistic helmets in the 1980s and the development of hard body armor in the 1990s. Owing to the importance of such models to the Army, addressing this challenge needs to take a central position in the development of personal vulnerability models. It is important for these models to include a plausible set of injury assessments with various battlefield functional capacities and universal joint tasks, not just lethality and severe incapacitation models.

Principal issues in the development of such models include “closing the loop” to validate structure/physiology results derived in physiological models in humans. This effort needs to be two-pronged—both computational models of humans and surrogates and field epidemiology play a role. For computational models, the challenge of improving finite element models can be addressed, in part, by attempting to add models that contractors develop for military programs, paid for by the Army, to the mix of available tools for the broad development of useful BABT injury risk models. For field epidemiology, collaborations with Program Executive Office Soldier, MRMC, the Armed Forces Medical Examiner, the intelligence community, operational commands, and others need to be formalized to obtain as much information from existing personal protection, including vehicles and armor to assist in developing and validating risk models. For example, the PPE needs to be sent back to ARL from the theater both in instances where it has worked successfully and in instances where the soldier has sustained injury in spite of (or because of) the PPE.

The ARL A&A Campaign has focused on using and developing tools based on appropriate statistical analyses, including the assessment of variance and distributions in calculations rather than the use of simple means. Essentially, every aspect of the A&A mission involves statistical distribution rather than a fixed single exemplar. For instance, threats have a distribution of energies and effects. Human anthropometry arises in military operations as a distribution. Response of vehicles and personal protective equipment can be characterized by a distribution. Even material properties of constituents of vehicles, protective equipment, and people are a distribution. The challenge of using appropriate statistics in assessments is twofold. Often the distribution needed for a given analysis is unknown and is not feasible to measure or obtain. Therefore, approximations need to be made to make the assessment tractable. The other major challenge is determining the appropriate analysis for significance. In the medical and biomedical communities, it has recently become even clearer that the often-used statistical significance value (the “p value”) may not be a desirable or an effective measure of the import of a statistical difference. That is, the simple statistical significance does not tell one whether the difference is “important” or not. If the variance of a particular measurement is small, even unimportant differences can be statistically significant. When assessing the significance of an effect, it is important to consider not only the statistical significance but also the size of the effect and whether a difference is a clinically or programmatically

meaningful difference. Evolution of this philosophy continues in the biomedical field to emphasize important differences in treatments, not simply differences with a statistically discernable mean.

There appears to be insufficient core funding to advance key areas in BABT risk modeling, and the existing efforts appear to be currently funded on a somewhat ad hoc basis.

HUMAN SYSTEMS INTEGRATION

Accomplishments and Advancements

The ARL HSI team has beneficially applied the Improved Performance Research Integration Tool (IMPRINT) and digital human modeling to Army system concepts prior to contracting with industry to fully develop new weapon systems. This has led to early design improvements and has avoided costly redesigns later in the development cycle. Following pre-Milestone B applications, these models have been provided to system development contractors to continue iterative HSI analysis as system concepts and designs continued to evolve. The HSI team also developed a much-needed Manpower Requirements Criteria (MARC) toolset that enables designers to trade off candidate designs to cost-effectively optimize soldier accommodation.

The ARL Human Research and Engineering Directorate (HRED) has recruited and hired early-career scientists and engineers and is mentoring them to fill in for more experienced personnel who will be retiring in the near future. This contributes positively to ARL’s ability to provide continuous support to the development and maintenance of their HSI tools, models, techniques, and methods.

There has been much work, over many years, to improve and extend the capabilities of the digital human models and the accompanying soldier clothing and equipment models. This has allowed comprehensive static evaluation analysis and assessment of male and female soldier accommodation in ground combat vehicles and other systems. ARL supports contractors in the use of these models during design development to ensure their proper employment. The extension of modeling analysis techniques to include dynamic conditions is applauded, as it could significantly improve the ability to analyze vehicle safety.

The ARL HSI team developed, improved, and used the IMPRINT model to predict soldier performance and workload in Army systems for nearly two decades. IMPRINT has been employed by hundreds of contractors to assess system concepts, preliminary designs, baseline designs and to propose design changes. This has resulted in significant cost avoidance over many years.

The Job Assessment Software System (JASS) tool is being developed to compare the skills and aptitudes required for Army jobs against existing military occupational specialty (MOS) requirements. This tool has potential to assist contractors in assessing soldier interface characteristics and job demands against the capabilities of the personnel identified in the target audience description (TAD) for operational, maintenance, support, and training positions. This tool, if appropriately verified and validated, could help to identify areas where jobs have been designed that do not match the capabilities of the soldiers identified for those jobs. It then may also allow contractors to correct user interfaces, modify job or training requirements, or propose changes to personnel requirements prior to establishing the system baseline at the stage of preliminary design review (PDR).

Challenges and Opportunities

Opportunities to involve military personnel as subject-matter experts (SMEs), subjects, and assistance with HSI tool/model validation ought to be identified. More direct interaction with warfighters

is essential to provide high confidence that real Army problems are being successfully addressed in a proactive, timely, and efficient manner. The mechanism for acquiring soldier assistance needs to be more formalized and not inconsistent and ad hoc, as it currently appears to be. Without good ARL situation awareness (SA) with respect to the real needs of the soldier and a continuous focus and feedback loop to ensure that the solutions are usable, there is a high risk that the mission will not be properly supported; there will be much resulting loss of life and material and a failure to achieve objectives. As an absolute minimum, ARL HSI principal investigators (PIs) need to take advantage of opportunities (and be encouraged) to spend time with soldiers in the field to gain an appreciation for their tasking, operational environments, risks, hardships, and the trade-offs that need to be made on a daily basis.

An overarching framework for HSI analysis and assessments ought to be defined and implemented. Needs from potential users (including the Army, contractors, federally funded research and development centers [FFRDCs], academia, etc.) ought to be solicited to identify gaps, prioritize them, and use the results to guide future analysis and assessment investments. This would provide a clear rationale for each A&A tool, technique, and method in terms of Army customer needs and provide the information necessary for a coherent evidence-based prioritization for application of resources. It is not clear why A&A tools that have been emphasized and used on past Army acquisition programs (such as goal-directed task analysis [GDTA], IMPRINT, and SA analysis) are not being aggressively maintained and applied. Other tools (e.g., Preventative Maintenance Checks and Services [PMCS+]), while reducing task completion time and errors, do not appear to add new analysis or assessment capabilities. An overarching framework that includes all HSI domains and relates the A&A techniques to Army needs is needed to identify areas where stakeholders are being underserved.

It is very important for the ORCA model to accommodate various sizes and shapes of female and male soldiers to provide more useful results to guide program-level decisions. The current version of ORCA uses a 50th percentile male digital human model for all calculations. Creation of several different female and male models of varying anthropometry would significantly enhance the ability of ORCA to provide accurate analytical results.

Rigorous verification and validation (V&V) ought to be viewed as an essential step in HSI tool and model development. Warfighter participation is needed to validate the effectiveness of HSI A&A tools, models, and techniques because warfighters are intimately familiar with the combat environment in which soldiers must perform their mission tasks. Failure to rigorously validate A&A tools in a timely manner throughout the entire cycle increases the likelihood of inefficient use and loss of personnel and material resources to successfully accomplish mission objectives.

More emphasis needs to be given to the transition of tools, operator manuals, and training, accompanying approaches and methods to industry. Many of the models, tools, techniques, and methods developed by HSI could be cost-effectively applied by system development contractors as they iteratively define, refine, and baseline system designs. To facilitate this, ARL could provide contractors with HSI models and tools, operating manuals, and training. ARL HSI scientists need to also establish a professional relationship with contractor personnel operating these tools, models, and so on to gather lessons learned and ideas for tool and model enhancement.

ARL ought to focus more on using the results of analyses and assessments to improve human-system integration requirements for future acquisition programs. This could take the form of improved contract requirements or upgraded HSI domain standards. The current approach reflected in the human modeling area is to evaluate contractor designs post-Milestone B. Delaying evaluation until post-Milestone B may be too late in the acquisition process to make cost-effective design changes. It is suggested that improved accommodation or soldier “space claim” requirements could be developed for future contracts,

based on ARL’s 10-plus years of Jack modeling experience. Improved requirements could lead to earlier problem detection and resolution.

It is important for ARL to develop a capability for analyzing and assessing HSI (e.g., human factors engineering [HFE], soldier interface and training) technologies for their potential in improving human-system integration and performance. One of the areas of emphasis for the A&A Campaign is titled “A&A on Technologies.” Currently, this area is limited to “SLV A&A of Technologies” and “Technology Trade Space A&A.” “Human Systems Integration Technologies” need to be added as a third subarea. Emerging soldier interface technologies need to be analyzed to determine if they hold potential for application to future Army systems. Additionally, formal usability assessment tools ought to be developed for analysis of soldier interface technologies.

JASS may hold potential for broader applications. Contractor HFE and user interface designers usually do not have knowledge of the soldier occupational specialties and grades that will be operating their systems to accomplish their required mission tasks. Without knowledge of the skills, aptitudes, and knowledge of the users, contractors depend upon retired military or surrogate soldiers to analyze and assess their designs. With modification, JASS might be used as a tool to evaluate soldier jobs and user interfaces against the MOS requirements for that position as established by the TAD. This could result in earlier detection of mismatches between soldier capabilities and job requirements.

It is important that anthropometric models that deal with more realistic scenarios and dynamic conditions for soldier protection be developed soon. ARL presented plans to develop digital human models capable of analyzing varied scenarios and conditions that cannot be analyzed today. In addition, plans were put forward to extend accommodation analysis to include the dynamic conditions that vehicle occupants may experience—for example, rough terrain and improvised explosive device (IED) detonations. Including realistic, dynamic conditions in physical accommodation analysis could dramatically improve the validity of results and allow a higher probability of overall mission success; it could also materially improve soldier safety in the operational environment.

Some popular and important tools and models (e.g., IMPRINT) are not being supported as strongly as in the past. Improvement plans for other A&A capabilities (e.g., digital human modeling, dynamic accommodation) seem to extend over an excessive time span. A capability to perform dynamic soldier accommodation modeling, for example, is long overdue and compromises the ability to appropriately design vehicles and execute missions in an optimal manner.

The number and scope of analyses and assessments need to be expanded to address all domains of HSI. The analyses and assessments presented did not address all of the HSI domains. ARL ought to proactively collaborate between domains to ensure that all aspects of soldier integration are addressed. Currently, the ARL A&A scope emphasizes physical and cognitive soldier accommodation. To extend this, ARL/HRED can “reach out” to other Army organizations (e.g., Survivability and Lethality Analysis Directorate [SLAD] for force protection and survivability and ARL Orlando for training) and agencies responsible for the other HSI domains (i.e., habitability, safety and occupational health) and include their A&A tools, techniques and methods in the overall HSI toolset.

ELECTRONIC WARFARE SURVIVABILITY, LETHALITY, AND VULNERABILITY

Accomplishments and Advancements

The EW SLV activity has a highly capable workforce that is focused on existing and emerging EW threats to the U.S. Army by foreign adversaries. The EW SLV team demonstrated an understanding of the complexities of this evolving area of warfare that includes traditional EW, cyber effects, and the integration of available technologies. The team recognizes that a trustworthy system is only as good as the information that it can reliably and accurately process. The team is focused on current and future EW threats and exploitable vulnerabilities in the areas of electronic attack, electronic support, and electronic protection. The newest recognized threats and vulnerabilities are emerging from the use of information technology systems that are mostly based on commercial off-the-shelf (COTS) hardware and software that includes emerging software-defined radios, radars, operating systems, network systems, and network protocols (wired and wireless).

The EW SLV team is working to combine methodologies for EW and cyber. The EW team is well educated and trained to carry out analysis and assessments for current and future Army systems and technology in the areas of radio frequency (RF)-based communications, radar, and electro-optical systems. New, evolving threats require complex analysis methods, test equipment, sensors, and a well-prepared staff to perform analysis and assessments in all areas of EW and cyber operations, including offensive and defensive cyber capabilities. The EW SLV team is focused on appropriate areas of EW research, to support the development of more robust offensive and defensive technologies and systems.

The EW SLV team has access to state-of-the-art facilities and instrumentation, such as anechoic chambers and advanced EW transceivers like the optimized modular EW network (OMEN). OMEN is a software-controllable RF signal generation system that can sense and generate EW signals via network control. The OMEN system includes a state-of-the-art signal modification, and rapid retransmission (if desired) to disrupt radar or communications networks. OMEN’s functionality enables ARL to sense and respond to signals of interest as required by localized channel conditions. Expanding the capability OMEN by hosting subsets of OMEN on mobile platforms such as unmanned aircraft systems (UASs) is a significant accomplishment.

Two modern COTS GPS simulators for position, navigation, and timing (PNT) analysis and assessments were obtained. These modern simulators have the capability to emulate the GPS constellation of satellites in a controlled environment to support EW analysis and assessments. The simulators are also capable of emulating a wide range of global navigation satellite systems (GNSSs) for further analysis and assessments. A good demonstration of a commercial wireless attack against commercial technology was provided using various synchronized and specialized RF exploitation tools. The attack demonstrated that ARL can identify existing tools, develop its own tools, and integrate various cyber and EW tools, to cause desired effects on some commercial technologies that are relevant to the Army.

The EW SLV team demonstrated the capability to understand commercial RF technologies and successfully apply cyber and EW techniques against those technologies, thereby informing the Army of the risks associated with using commercial technologies in an operational environment. This research included commercial WiFi and 77 GHz (76-81 GHz band) automotive radar systems. There are plans to include research into semiconductor devices, electro-optic devices, and ultraviolet solid-state devices. This work is focused on a good suite of devices that offer wideband electromagnetic operations in the RF spectrum.

The EW SLV team continues to investigate directed energy technology that includes electro-optical systems, eye-safe lasers, disruption techniques, and high-energy laser systems. This research is an appropriate focus area with potential benefit to the Army.

The cyber and electromagnetic activities (CEMA) laboratory integrates EW and cyber to increase the effectiveness of electronic attack and is an impressive capability that enhances full-spectrum analysis and assessments across EW and cyber. As an example, the use of a reprogrammable and reconfigurable UAS avionics platform as an emulator for hardware-in-the-loop simulation provides improved insight into analysis and assessments of combined EW and cyber effects on specific hardware and software systems. Furthermore, this particular hardware-in-the-loop methodology better supports assessing UASs and counter-UAS systems using cyber and EW, because the effects are directly observable and measurable.

The EW SLV team has developed partnerships with universities such as Texas A&M; the University of Texas, El Paso (UTEP); New Mexico State University (NMSU); and the University of Texas, San Antonio. As a result of these collaborations, new people, teams, concepts, and techniques have been incorporated into the EW SLV portfolio. Researchers were also recruited as a consequence of these collaborative efforts. The new talent brings radar, RF systems engineering, and digital technology expertise into the EW SLV activity.

Challenges and Opportunities

The preliminary research on commercial vehicle radar sensors at 77 GHz demonstrated good use of laboratory equipment, and relevant EW technical work is being performed. Additional work in this important area is encouraged; such as inclusion of clutter and interference into the research once the ideal scenarios (without clutter and interference) are well understood. However, the work with automotive 77 GHz radars appears to be lagging commercial automotive technology. It was also noteworthy that the receiver operating characteristic was not fully understood by the EW team. Understanding the receiver operating characteristic of the radar supports better analysis and assessments, because comparative analysis can be performed and understood relative to the receiver operating characteristic. The COTS vehicle radar sensors studied for ground vehicles need to have a stronger mission context in support of sensor evaluation. There was little evidence that the experimental setup significantly contributes to a validation scenario at this stage of research. There was no indication that the experimental setup will be used to validate a particular concept of employment. A physics-based model/simulation is needed to provide a better understanding of expected results. If not already planned, it would be beneficial to create a physics-based model to ensure that the theory and technology are thoroughly understood.

No evidence was presented that the anechoic chamber is optimized and calibrated for 77 GHz measurements. Additionally, to perform high-quality research in this emergent area, a calibrated anechoic chamber is required with sufficient quantities of modern test equipment to support millimeter wave characterization, test, evaluation, analysis, and assessments. Very limited modern test equipment was observed to support research in the millimeter wave bands of interest. Additional millimeter wave equipment is required to perform detailed analysis and assessments in the future.

The PNT experimental activity needs to be capable of analysis and assessment of non-GPS-based systems. Miniaturized inertial navigation systems and optical-based systems (e.g., lidar) are emerging to augment PNT capabilities when RF-based systems are unavailable.

While the CEMA laboratory is a big step forward, a focus on better-designed and secured systems (National Security Agency [NSA]-grade military systems) are needed. Well-architected, well-designed, and well-implemented communication systems that are robust against simple attacks were demonstrated, and thus more sophisticated efforts are required. Attacks from the inside the network are difficult to

egress out of a properly secured network. Attacks from outside the secured boundary, which is usually at the physical and datalink layers, have a very low probability of success of penetration. Making progress against these types of communication systems would be state of the art, whereas commercial systems represent a much lower bar for penetration and exploitation. In addition, CEMA was mentioned only in the context of a few of the research topics, yet the integration of modeling and experimental/test research activities are crucial to future convergence successes in EW assessments with cyber integration. There is also an opportunity to improve activities related to the use of drones. The CEMA laboratory has made progress investigating UAS (i.e., drone) technology, but the UAS assessment activity needs further context and clarity. It was not clear whether the effort was used to assess Army counter-UAS capabilities.

Machine learning and artificial intelligence are very important technologies for use in EW. While these are emerging technologies as applied to EW, they are important because correctly trained machines make optimized decisions faster than humans. These very important technology areas were not addressed, and there is an opportunity to apply machine learning and artificial intelligence in CEMA.

There is an opportunity to expand the jamming capability to include multinode mobile ad hoc networks (MANETs) with 20 nodes or greater. The solider radio waveform (SRW) is a MANET with the ability to route data around interference. Therefore, jamming effects on a MANET are more complex than jamming a single node of a network unless it is a critical node that bridges two parts of the MANET. SRW also uses modern forward error correction (FEC) to mitigate channel errors and enhance data reliability. The capability to characterize and understand these networks exists within the EW SLV team and its facilities. To increase EW impacts and understanding of these networks, trained MANET staff and laboratory equipment adequate to characterize per-node EW impacts are required. Large-scale MANETs exist (like SRW), and they need to be adequately laboratory tested in sufficient quantities, and with proper equipment. Further, modeling and simulation of large-scale MANETs (greater or equal to 20 nodes) that incorporate EW and cyber effects support large-scale analysis and assessments of relevant MANETs’ robustness.

Although the laboratories were generally well-equipped, some older, less than state-of-the-art instrumentation was present. Good work in a complex RF environment requires modern laboratory equipment. Although progress is being made in upgrading equipment, more progress is needed to perform high-quality work.

The EW effort continues to be challenged by balancing available staff to workload. The EW effort needs to build on the current experienced staff to meet mission demands, while recruiting highly educated personnel to carry out the highest professional level of work. The EW team needs to develop and include more advanced EW techniques beyond simple broadband noise jamming methods at the physical layer. While the EW SLV team is very effective using digital radio frequency memory (DRFM) for electronic attack, which is a more sophisticated EW technique beyond noise jamming, the team would benefit from developing technologies that identify and characterize signals of interest and then developing and using more specific EW techniques that are better targeted toward receivers of interest. The EW team would also benefit from training that includes radar and modern RF communications networks that employ advanced signal processing techniques, such as FEC coding and direct sequence spread spectrum (pulse compression for radar).

There are opportunities to expand and enhance EW-related activities through alliances. For example, there is an opportunity to expand EW assessments in urban environments, such as the Muscatatuck Urban Training Center in Indiana, which offers an opportunity to assess EW technologies and systems in an operationally realistic urban environment. Outreach to EW programs such as those at Georgia Tech, the University of Iowa, and Wright State University, offers an opportunity to recruit EW graduate students and collaborate on advanced EW topics. Other alliances could be formed through participation

in the Association of Old Crow, and participation in the Armed Forces Communications and Electronics Association. A mutually beneficial relationship can also be strengthened with the U.S. Navy Naval Air Systems command (NAVAIR) Point Mugu Code 4.5 community, which focuses on airborne EW offensive and defensive systems engineering and EW technique generation and implementation. There are other Do-related EW efforts that can be mutually beneficial to ARL.

CYBER SURVIVABILITY, LETHALITY, AND VULNERABILITY

Accomplishments and Advancements

The Cyber SLV team at ARL functions as both a “red team” that discovers vulnerabilities in Army systems and a “blue team” that helps Army organizations design and create secure systems and remediate vulnerabilities that the red team function discovers. The Cyber SLV team’s scope includes both Army systems under development and operational Army systems. The red team primarily discovers instances of previously known vulnerabilities in operational systems and zero-day (previously unknown) vulnerabilities in systems under development. The Cyber SLV team employs approximately 70 people (including 9 contractors), representing growth of approximately 20 percent since the previous review in 2016.

The Cyber SLV team has responded to the recommendations of the previous review. At least some team members are sent to top-tier vulnerability research conferences (i.e., Black Hat and DEFCON). The team has continued to apply the cyber table-top methodology to analyze the security of emerging designs and reports success. The team has also extended its capabilities to include security analysis of embedded systems that incorporate CanBus networks for internal communications.

The Cyber SLV team’s services are in very high demand as the Army in-house systems red team. While the team has grown considerably since the prior review, demand for their services has continued to increase; many of the team’s personnel are taking upward of 30 trips per year to participate in security assessments. The team has begun developing capabilities that will permit personnel to remain at WSMR to conduct their analyses, which will relieve some travel demands but will not reduce the requests for the team’s services. The demand for the Cyber SLV team’s services will continue to increase.

The Cyber SLV team has taken the initiative to create an in-house workforce development process for team personnel. New hires enter at an apprentice level and progress through journeyman to master level. This process helps ARL bring on board new Cyber SLV staff members and make them effective contributors.

ARL Cyber SLV personnel are presently working to engage with local universities (UTEP and NMSU) as well as regional universities (through ARL South). The collaboration with UTEP is established as the Center for Cyber Analysis and Assessment. A staff member taught a reverse-engineering class as a visiting instructor at UTEP, and the center has worked with ARL on initiatives in education, collaborative research, and tool development. These kinds of outreach activities have important short-term recruiting benefits, exposing students to the work that ARL does and leading to a number of successful hires. As a direct result of these activities, ARL has hired 10 staff personnel and 3 interns in the past year, and 38 total staff members in the past 4 years.

The integrated network vulnerability assessment discovery and exploitation (INVADE) tool is apparently well designed for extensibility to permit constant upgrading to cover a broader spectrum of vulnerabilities discovered during system analysis. This tool incorporates modern industry standard tools integrated into one package that can scale the analyses of Cyber SLV team personnel broadly across Army and DoD systems. Unfortunately, the INVADE tool was not demonstrated, and so it was not possible to

evaluate its capabilities for finding zero-day vulnerabilities, nor its user interfaces that would indicate its convenience and how well it informs operators of its analytical outcomes.

Challenges and Opportunities

Many of the challenges and opportunities facing the Cyber SLV team result from its importance and success in accomplishing its mission: the ever-growing demand for cybersecurity assessments creates considerable pressure on the Cyber SLV workforce. The demands on personnel’s time limit their ability to conduct applied research, to develop new tools, to make current and new tools more effective, and to remain current with emerging cybersecurity trends. The heavy workload and the restriction of internal development programs to current ARL personnel make it difficult for current team members to advance to higher stages of capability and to acquire new skills. Even though the team has grown considerably, additional personnel are needed to reduce this pressure and improve both the quality and quantity of the Cyber SLV team’s efforts and outcomes.

The Cyber SLV team has established effective partnerships with UTEP and NMSU. There is an opportunity for ARL to build on these successes by expanding these efforts to universities throughout Texas and New Mexico (e.g., University of Houston, Texas A&M, University of New Mexico), leveraging their affiliations with ARL South, which is based at the University of Texas, Austin, campus. Given the plan to locate the Army Futures Command in Austin, building ARL Cyber SLV programs in collaboration with ARL South could pay dividends both in program effectiveness and in visibility. ARL personnel could be encouraged and supported to do one-year tours at affiliated universities, as has been done at UTEP, both to teach classes and to join cybersecurity research efforts being conducted at these universities. These tours would establish deeper collaborations, making it easier for faculty to send their own students as interns to ARL and making it easier for ARL personnel to pursue advanced degrees.

One innovative way to leverage ARL South would be through the development of shared curricular materials across its affiliated universities. For example, ARL personnel could create educational modules related to relevant cyberattacks (videos, slides, code samples, network traces, cyber range experimental environments) and host a cross-university capture the flag exercise. While ARL’s location, remote from much U.S. cybersecurity activity, makes recruiting outside the local area challenging, exposing a broad range of university students to ARL’s people and programs may pay dividends in attracting new and talented people.

While the Cyber SLV team has made progress in early engagement in the design of Army systems (referred to as “moving to the left” on the system development timeline), the level of engagement still presents opportunities for improvement. The Cyber SLV team’s work may benefit by adding techniques such as threat modeling and the flaw hypothesis method to its design analysis practices. Recognizing the importance of cybersecurity to today’s systems, the Army may want to consider mandating that all system development programs engage ARL to conduct design-level security analysis or consulting early in the system life cycle. That option would carry with it a need for additional Cyber SLV personnel at ARL, including senior personnel, but it would be an effective and important way to improve the security of systems when they are fielded.

The very nature of cybersecurity, and especially of vulnerability analysis, requires that the team constantly refresh and update its knowledge base of known vulnerabilities, and entirely new classes of vulnerabilities, not just zero-day variants. Software security is one important aspect of system evaluation, and the Cyber SLV team has demonstrated competency comparable to industry practices. However, there is rapid emergence of new attacks that exploit the hardware-software boundary for which the team

must prepare. These attacks are routinely reported in the academic and hacker research communities.¹ Recent examples include several unforeseen vulnerabilities in ubiquitous hardware designs that leak information that can be exploited by hackers.

The Cyber SLV team strives to sustain a constant diligent effort to learn about new classes of security vulnerabilities that affect either deployed Army systems or the designs of new systems coming online. This effort will be aided if team personnel routinely attend the major industry and academic security conferences and meetings where these classes of vulnerabilities are presented to the public after disclosure to vendors of the vulnerable systems. The team’s limited attendance at Black Hat and DEFCON is a valuable start but is still too limited given the importance of staying up to date and the fast-moving nature of vulnerability research and discovery.

Ongoing research in the software engineering community has demonstrated the application of machine learning applied to large repositories of software to discover common bugs found in widely used software.² These bugs often are security vulnerabilities. The software systems that are deployed in Army systems are a natural focus for this type of advanced analysis, and the Cyber SLV team needs to develop internal competency in application of these machine learning techniques. The opportunity for the Cyber SLV team to utilize these techniques is very likely of high utility in developing a more intelligent INVADE tool that can generalize from detected vulnerabilities and find cases that appear in other Army systems.

The National Cyber Range is a test facility operated by the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation. Its core mission is to provide facilities and knowledgeable staff to test and evaluate cybersecurity systems. The single national resource for the DoD is under great demand and unlikely to provide sufficient capability on a timely basis for all the Services. ARL is well positioned with its facilities, toolsets, and staff expertise to build a core Army Cyber Range. An Army Cyber Range devoted to Army-specific systems will provide efficiencies that are hard to accomplish with a single provider that serves multiple DoD Services and agencies.

Every military branch has personnel with cybersecurity expertise, and many of them face common issues that are more pressing in the military world. While private sector researchers are studying vulnerability defense and exploitation in embedded and Internet of Things (IoT) devices that can be found in a home or automobile, the IoT devices embedded in battlefield weapons and communications systems are likely to differ in significant ways specific to the military. As other branches of the military grapple with these issues, they will inevitably develop expertise and tooling, often stovepiped within their particular areas. ARL Cyber SLV personnel would benefit from engaging with their counterparts in the other branches, allowing the entire U.S. military to advance the state of the art and practice.

___________________

¹ See, for example, A. Tang, S. Sethumadhavan, and S. Stolfo, 2017, CLKSCREW: Exposing the Perils of Security-Oblivious Energy Management, http://ids.cs.columbia.edu/sites/default/files/usenix17_clkscrew_atang.pdf.

² See F. Yamaguchi, F. Lindner, and K. Rieck, 2011, Vulnerability extrapolation: assisted discovery of vulnerabilities using machine learning, WOOT ’11 Proceedings of the 5th USENIX Conference on Offensive Technologies, https://dl.acm.org/citation.cfm?id=2028065; X. Xu, C. Liu, Q. Feng, H. Yin, L. Song, and D. Song, 2017, Neural network-based graph embedding for cross-platform binary code similarity detection, last revised July 17, 2018, https://arxiv.org/abs/1708.06525; R. Paletov, V. Raychev, P. Tsankov, and M. Vechev, 2018, Inferring crypto API rules from code changes, https://files.sri.inf.ethz.ch/website/papers/diffcode-pldi2018.pdf; Y. Tian and B. Ray, 2017, Automatically diagnosing and repairing error handling bugs in C, http://rayb.info/uploads/fse2017-errdoc.pdf.

COMPLEX ADAPTIVE SYSTEMS ANALYSIS

Accomplishments and Advancements

The complex adaptive systems analysis (CASA) team within the SLAD is currently developing the modeling and simulation tools to create a multidomain analysis environment (MDAE) for warfare analysis and assessment. The interaction of EW and cybersecurity and their interactive effects (as addressed in CEMA) on military communication networks as well as unmanned systems may create complex interactions to change the outcome of military engagements. The CASA team also performs studies of evolving technologies to understand their impacts to system behavior. Examples of these evolving technologies include the integration of unmanned systems into the battlespace and complex unmanned systems swarming behaviors. CASA is also developing a Bayesian framework to formalize complexity and emergence theory to create analyses and assessment tools.

Warfare is a complex activity. Within the Army, multiple systems are interacting with one another in order to meet a common goal. The Army’s CASA team performs analyses and assessments to gather insights into how these complex interactions may affect military operational outcomes. The emergence of EW and cybersecurity, as well as unmanned systems, when integrated with manned platforms, may produce unexpected outcomes and emergent behaviors that need to be understood. The CASA team is currently focused on developing modeling and simulation tools to capture these complex and emerging behaviors with test bed modeling and simulation tools for the MDAE.

The CASA team has addressed all of the recommendations from the 2016 ARLTAB assessment. It has moved its base simulation platform to one semi-automated forces (OneSAF), which is the Army’s accredited disaggregated entity-level or agent-based simulation platform tool. It is integrating its validated RF communication propagation model (called SAGE) into OneSAF to better understand the impacts of EW to military networks and communications between simulation entity military platforms. This integration will model the impacts of EW much better than existing methods in a force-on-force simulation environment. The team is integrating cybersecurity effects into the OneSAF simulation. The team has also expanded the expertise of its staff and is reaching out to collaborate with universities, Army commands, simulation specialists at MITRE, and other military branches. It has also initiated the employment of complexity and emergence theory to identify game-changing impacts of technology on Army force-on-force engagement outcomes.

Challenges and Opportunities

The complexity of integrating cyber and EW activities into a physical-world, ballistics-modeling, force-on-force simulation engagement application (i.e., OneSAF) requires major developments to enhance OneSAF. The resources available to support this development within the CASA team are limiting. As a result, the scope of development may need to be narrowed to support a realistic and validated slice of a battle scenario. Given the time and resource limitations, the CASA management recognizes that it needs to quickly demonstrate utility of the integrated OneSAF with expanded capabilities of CEMA and its impacts to the outcome of a simulated battle.

There are challenges in the development of the MDAE. First, OneSAF currently includes neither cyberattack modeling to simulate entity platform components nor wireless networking functionality between simulation entities. It is important, for example, that the simulation reveals whether a cyber exploit takes down the fire control system or some other platform subsystem. Also, as a result of a cyberattack, the behaviors of the entity platforms must be modified to generate the correct response to

a cybersecurity attack. The CASA team may need to look outside to other Army commands to develop these decision tables of behavior for the agent-based platform models. Commands to consider are the training centers such as the U.S. Army Training and Doctrine Command (TRADOC) or the TRADOC Analysis Center (TRAC).

Additionally, OneSAF does not simulate, with any degree of physical modeling, the effects of jamming in an electronic attack against forces. Here, the CASA team is integrating its SAGE RF modeling tool to determine how a jammer in the simulated physical space with a discreet position can affect each platform entity’s communications abilities at other defined locations in the simulated operational environment. This will provide a much better emulation of real-world radio jamming, because specific platforms may be jammed, while others are less compromised. This is a departure from the current OneSAF implementation of jamming, which reduces the probability of successful communications for messaging between all agents with the same probability across the board. In this new approach by CASA, each entity platform will have different probabilities of communication depending upon relative locations of each platform to one another and the jammer. As a result, different simulation outcomes may occur with new complex emergent behaviors depending on how network communications nodes are deployed over a given force structure and formation.

Next, the best software architecture to implement cyber and EW into OneSAF has substantial risk. These CEMA effects must be related with a verified and validated set of causes and responses for each platform and entity to model survivability, lethality, and vulnerability correctly. Otherwise, the effort will not provide any useful insight. Furthermore, since these effects will be stochastic, the simulation itself will need to run in a non-real-time mode to support Monte Carlo operation. This again comes back to software architecture, where the simulation must operate repeatedly with sufficient variation of performance to determine scenarios or configurations that cause unexpected outcomes or generate new complex emergent behaviors.

The scope of the development to modify OneSAF with new physical models and phenomena requires access to people with expertise in OneSAF and the various OneSAF simulation components. More resources may be required to develop the simulation to demonstrate near-term utility. The ability for the SLAD and CASA team to access this outside help is limited due to the few contract vehicles and relationships they have with outside of SLAD contractors, universities, analysts, and developers.

In addition to personnel resources, there are also hardware resource limitations. The eventual tools will require significant computing resources. The CASA personnel have identified cloud computing as a solution to provide computing resources as required to free their development from expensive hardware infrastructure capital expenses. CASA also has access to the Army’s high-performance computing modernization program resources.

In addition to creating the multiple domain battle analysis environments, CASA is also exploring addressing Army key campaign initiatives (KCIs) with small studies. These activities were presented to the ARLTAB in poster sessions. It is noted that these studies were small scale and limited in scope. But the issue with these initiatives is that the study areas required more experienced personnel in the fields of the KCIs to take the studies deeper and develop results that reflect the state of the art.

OVERALL QUALITY OF THE WORK

The A&A Campaign is different from the other campaigns because this is more of an analytically focused, crosscutting activity rather than being research focused. As a result, the criteria for assessing the A&A Campaign are different from those of a research-focused campaign. Nevertheless, the work needs to have technical depth, and the staff need to present material that exhibits this technical depth.

However, limited technical detail was provided, making it difficult to fully assess in detail the overall technical quality of the work.

Judging by the material that was presented, the quality of the work and the technical staff appeared generally outstanding. Nevertheless, current A&A efforts are falling further behind in incorporating the complexities of the technologies and environments needed for A&A. This could be due to lack of personnel resources. In several areas, the team is only one deep, so there is a lack of personnel. Resources need to be made available to address the requirements to analyze and assess complex technologies in complex environments.

The modeling work seemed generally of high quality, but rigorous verification and validation is not always included in the model and tool development; this is especially the case in HSI.

Of the core technical BSVL efforts reviewed, the projects on underbody blast modeling and the collection of data that shows the impact of multiple hits on armor panels were of very high technical quality, and the teams were highly educated and skilled to conduct these efforts. However, limited personnel resources resulted in important effects being modeled well after this information was needed. A lack of validated models has led to the need to carry out a very costly experimental program, illustrating that it is important to stay ahead of the need.

Three of the KCIs—Framework for Complex Multidomain Analysis, Analysis and Assessment of Congested and Contested Operational Environments, and CEMA Analysis and Assessment Methodology for Congested and Contested Environments—have significant overlap, are well outside the current mission space of ARL, and are so broad that the outcomes cannot be clearly seen in the 15-year time frame. More definition and work a needed on these initiatives. The fourth KCI, Virtual Interactive Simulation Analysis and Assessment, is a long-term follow-on effort from the immersive demonstration that served as a proof-of-principle for these visualization techniques. These KCI efforts would all benefit from definition of near- and long-term deliverables.

Overall, the EW SLV team is very resourceful, making do with limited resources and leveraging tasking to further technology improvements that support analysis and assessment. However, more technical depth of analysis is needed in a few key areas such as EW effects on MANETs and EW technique development beyond broadband noise jamming, radar clutter generation, and inclusion in simulations and experiments. There are impressive laboratory facilities that, in some cases, do not exist elsewhere—for example, the extremely large anechoic chamber is a national asset. There are some deficiencies in the laboratory equipment available—in particular, the technical quality of work could be improved with more modern RF equipment, such as programmable attenuators, wideband spectrum analyzers, and arbitrary waveform generators.

The Cyber SLV team has a clear understanding of its mission and is committed to improving the quality of its personnel, tools, and end products. The INVADE framework and a variety of commercial and public domain tools are being used effectively to discover vulnerabilities in Army systems, although there was only limited evidence that the team is developing innovative vulnerability discovery tools. Although limited technical detail was provided, based on the presentations it appears that the cybersecurity assessment work is of good quality and consistent with that of private sector vulnerability assessment consultants.

The CASA team is developing a unique technical approach to examine the MDAE. It is using the best in class, entity-level, and agent-based simulation modeling with OneSAF. However, it is possible that the OneSAF tool may not be appropriate for this purpose, because it was developed as a simulation training tool. On the other hand, a higher level constructive military simulation cannot capture these complex emergent behaviors at the base agent and lower unit levels. These constructive higher echelon models do not have the resolution to generate these complex behaviors.

The CASA team is employing state-of-the-art and validated RF modeling tools to be integrated with OneSAF. They are integrating cybersecurity and networking into the OneSAF modeling and simulation platform; this will allow analysts to examine EW and cyber domain effects into physical world effects of casualties and destroyed weapon platforms. The CASA team is aware of the complexity, risks, and trade-offs to develop this MDAE tool. The CASA team has access to the high-performance computing modernization project at the U.S. Army Armament Research, Development, and Engineering Center (ARDEC) for state-of-the-art computing power and is utilizing state-of-the-art software development practices for their agile software development environment.

CONCLUSIONS AND RECOMMENDATIONS

The area of analysis and assessment is very important to the Army. The analyses and assessments prepared by ARL support Army decisions at all levels of the Army. The products of these assessments and analyses are used by the Army Evaluation Command in preparing recommendations to Army leadership up to and at the Secretariat level. A&A is an important activity that is very under-resourced and falling further behind in meeting mission goals.

Military experience and expertise is vital in developing study plans and understanding results in a military combat context so that efforts can relate better to the military environment and the impact of results on the military situation can be more readily understood by the Army community.

There is a potential problem when a contractor or 5-year term employee is the only employee possessing a critical skill for a large project of major importance for the Army.

In some cases, government employees need to use contractors and contractor facilities to get work done that they cannot do because of the work environment and lack of adequate computational support.

• ARL should develop and articulate clear long-term priorities.
• ARL should provide long-term sustainment and high-priority improvements to core analysis tools (MUVES, ORCA).
• MUVES and ORCA should be examined to determine whether they are structured adequately to handle multihit, multithreat, and complex environment interactions.
• ARL should include plans for verification and validation as an integral activity throughout the tool development process; this is especially the case for human system integration.
• ARL should consider more sophisticated frameworks with a focus on creating the right team that includes industrial and academic partners.
• ARL should consider a red team/blue team approach for analyses/assessments.

• ARL should increase involvement of military personnel in the prioritization process and in tool and model verification and validation.
• ARL should involve project managers at an early stage of the prioritization and development process and in the contracting language for analysis and assessment support.

• Engagement with other services, the North Atlantic Treaty Organization, the Technical Cooperation Program, and other military partners should be further improved.

• ARL should speed up the approval process for conferences and equipment purchases and have these take place a lower level.
• Information technology support for A&A staff should be improved.

The EW team is very resourceful and creative, and it is making good progress. There are a few areas for improvement. Understanding the receiver operating characteristic (ROC) curves of radars is needed for better analysis and assessments of systems that use radar. When an ROC curve is known, comparative analysis can be performed and understood. There was little evidence that the experimental radar setup used significantly contributes to a validated scenario at this stage of research.

• Commercial-off-the-shelf vehicle radar sensors studied for ground vehicles should have a stronger mission context for sensor evaluation.
• A physics-based model/simulation should be constructed to better understand expected results for radar sensors. Clutter and expected interference should be included in the evaluation.
• The ongoing ultraviolet communications system studies conducted should be encouraged. A vision of how higher frequencies (50 GHz to 1 THz) might benefit the warfighter should be developed.
• The position, navigation, and timing (PNT) experimental capability should be capable of analysis and assessment of non-GPS-based PNT systems. Alternative PNT concepts should be investigated to include other global navigation satellite systems and non-radio frequency-based PNT solutions.
• Larger MANET networks (greater than 20 nodes) should be investigated to better understand how they respond as a system to electronic warfare and cyber effects.

The major factor limiting the Cyber SLV team’s ability to make investments and improvements is the day-to-day workload that it faces. The demand for cybersecurity assessments creates considerable pressure on the cybersecurity workforce and in particular limits its ability to develop generally useful

tools and techniques that could improve its efforts for ARL and the Army. There are a variety of options that would enable the Cyber SLV team to improve the training and capabilities of its staff and develop new and valuable tools and techniques, but all depend on achieving a better balance between workload and resources.

It is critically important to the Army to perform security analysis at system design time rather than merely conduct security tests after a system is largely completed and near deployment. The ARL Cyber SLV team conducts some design-time analyses, but these appear to occur when a program office asks rather than as a routine or regular matter. If these analyses become a regular matter, they will place significant demand on the Cyber SLV team to deploy more resources and to deploy more senior talent.

While ARL has responded to the previous review’s recommendation that Cyber SLV team members be allowed to attend industry conferences such as Black Hat and DEFCON, attendance has been restricted to a small number of team members. In addition, other technical conferences have emerged, including defense-oriented conferences that focus on issues directly relevant to the work of the ARL cybersecurity team. Research and experimental results in cybersecurity are reported most promptly, and often only reported, in conference presentations, and it is important for ARL technical staff to be aware of those results.

The effort to develop the MDAE is in its early stages and the goals are ambitious.