1

Introduction

To protect itself from attacks by foreign forces, the United States relies on its armed services, which in turn rely on weapons and other systems to provide the tools needed to successfully neutralize adversary combat capabilities. Maintaining the armed services’ warfighting advantage requires a steady stream of new and improved weapons and technologies. A crucial step in acquiring and using these assets is testing their effectiveness and suitability on Department of Defense (DoD) ranges. DoD has testing ranges that span the globe where new military technologies are tested based on real threats, tasks, and environments to ensure their combat readiness. These ranges are a vital aspect of the nation’s defense, but will they be able to adequately test the increasingly complex military technologies of the future, at the pace required?

Former Director of DoD Operational Test and Evaluation (OT&E) Robert Behler has noted that the U.S. test range system dates to the years during and after World War II, with the most significant updates having been carried out during the Cold War.¹ The committee recognizes that there have been further upgrades and modernization in the 30 years following the end of the Cold War. However, Raymond O’Toole, the current acting director of OT&E, asserted at the committee’s January 2021 public workshop that the ranges have not kept pace with testing demands, technology development, or the capabilities of adversaries

___________________

¹ From remarks delivered at December 4, 2020, committee meeting; recording available at https://www.nationalacademies.org/our-work/assessing-the-physical-and-technical-suitability-of-dod-test-and-evaluation-ranges-and-infrastructure.

(NASEM, 2021, p. 2). Consider just some of the major environmental shifts that have affected weapon system development and testing since that time period:

• The United States now faces not one, but two peer military adversaries in China and Russia, and China is an economic powerhouse that has broadened the realm of great power competition far beyond avenues of military conflict.
• Other adversaries such as North Korea and Iran have rapidly developed missile, cyber, and nuclear weapon technology that can now threaten the U.S. homeland. Furthermore, rival nations have made a concerted effort to outpace the United States in weapons technologies of the future, including artificial intelligence, hypersonics, and space systems.
• Commercially available internet technology, mobile communications, cloud computing, and ubiquitous software have fundamentally reshaped the architecture and testing of weapon and support systems. Moreover, commercial demand and development are creating robust competition for the physical geography and radio frequency spectrum that OT&E has relied upon for decades.

The stewardship and use of DoD test ranges relies on multiple stakeholders to effectively test the nation’s defense systems. The developmental test and evaluation (DT&E) community is responsible for funding and executing the upgrades on which these future capabilities depend and therefore must be fully included in policy, resourcing, and allocation discussions moving forward. It is only through the full participation of all the stakeholders that lasting change can be realized. Urgent and substantial changes to the modernization, sustainment, operation and resourcing of the range enterprise are required to support the scale and diversity of weapon system testing and to meet the challenges posed by rapid insertion of new technology over the next 10–15 years.

The consequences of inaction will be severe, and recovery will be difficult. Technologies such as artificial intelligence, hypersonics, cyber weapons, and directed energy are creating new test capability requirements for DoD ranges. Rapidly improving threats, particularly from peer adversaries such as China and Russia, make the need to test a system’s survivability just as important as testing its lethality. Peer and non-peer adversaries, including actors like North Korea, increasingly employ asymmetric capabilities such as cyberattacks. These trends, when extrapolated to 2035, demand a new approach to modernizing DoD’s ranges’ technical and physical attributes. This new approach must not only preserve the current core capabilities, but also take a more holistic look at the aggregate

capabilities needed to address the software-intensive nature of future weapon systems, as opposed to the too-often piecemealed upgrades that occur today.

STUDY CHARGE

Against this backdrop, DoD’s Office of the Director, OT&E requested the Board on Army Research and Development of the National Academies of Sciences, Engineering, and Medicine to perform a study assessing the physical and technical suitability of DoD test and evaluation ranges, infrastructure, and tools for determining the operational effectiveness, suitability, survivability, and lethality of military systems (see Box 1.1, Statement of Task). While the ranges’ staffing and organizational structures are clearly crucial to their success, the statement of task specified that the study should focus not on the personnel and staffing but on the facilities themselves.

This study is the first of two studies that were requested by DoD’s Director of OT&E. It is an unclassified review that was designed as a stand-alone study but that will also provide the foundational elements for a follow-on classified study that is scheduled to begin before the official publication of this report. The purposes of conducting this study at the unclassified level were to make it possible to include on the committee as wide a range as possible of members from the science and engineering community as well as to build maximum awareness of the serious challenges to be addressed today. The follow-on study, which will be informed by the contents of this report, will assess how well the ranges are able to simulate the threats, threat countermeasures, and capabilities of near-peer adversaries and to test DoD systems in future operational scenarios. The goal of carrying out the two studies is that, by taking advantage of the full range of the nation’s science and technology community in the unclassified study while also having access to complete information on adversaries’ capabilities in the classified study, the two together will offer a comprehensive assessment of DoD’s testing and evaluation ranges and infrastructure.

The Committee’s Approach

To carry out the statement of task and evaluate the nation’s military ranges, the National Academies Board on Army Research and Development (BOARD) assembled a study committee composed of experts from the military, industry, academia, and government. The committee, assisted by BOARD staff members, assembled a broad collection of written and graphic information related to the ranges and OT&E, including many

previous reports authored by various components of DoD, the National Academies, the Congressional Research Service, RAND Corporation, and other groups. This literature formed the foundation on which the committee based its judgments. It was supplemented by presentations, typically held via Zoom, by multiple military officials, both active and retired, as well as other experts familiar with military ranges and the challenges of OT&E in the current environment.

On January 28–29, 2021, the committee held a public virtual workshop, Assessing the Physical and Technical Suitability of DoD Test and Evaluation Ranges and Infrastructure. Over the course of 2 days, the committee heard presentations from representatives of the Test Resource Management Center, the individual services’ testing and evaluation departments, various other service groups involved in the development and testing of military systems, national laboratories, universities, and industry. In April 2021, Key Challenges for Effective Testing and Evaluation Across Department of Defense Ranges: Proceedings of a Workshop—in Brief was published to summarize the workshop’s presentations and what the committee learned from them (NASEM, 2021).

Over the course of the study, committee members and BOARD staff carried out seven site visits, some in person and some virtually, at a diverse selection of ranges that were as representative as possible of the wide array of challenges facing OT&E over the coming decade and a half. A summary of the site visits is provided in Appendix B. During those visits the committee and staff heard from a wide range of military representatives with intimate knowledge of the day-to-day workings of the ranges and OT&E, and the information gleaned from these visits, combined with the knowledge and insights that the individual committee members brought to the process, formed much of the foundations for the deliberations that resulted in this report.

For those deliberations the committee members met regularly, both in full committee and in subcommittees, from December 2020 through July 2021. Depending on their expertise and interest, different committee members contributed to different parts of this report, but all writing, from the narrative to the findings and recommendations, was reviewed by and agreed on by the entire committee. This report is the result of that process.

The extraordinary diversity of DoD missions and test environments, and the large quantity of range locations and installations, precluded an exhaustive evaluation of all range capabilities and gaps in relation to the future OT&E landscape. DoD’s test and training ranges number over 500 in total, including the 23 major facilities in the Major Range and Test Facility Base (MRTFB). Additionally, DoD does not currently have standardized and comprehensive reporting on test ranges and facilities.

To assess the current physical and technical state of the test ranges, the committee selected representative ranges spanning all domains (land, sea, air, space, and cyberspace) to provide insights on the aggregate challenges with operational testing unique to each domain. This strategy enabled the committee to report on concerns and conditions that were articulated by multiple ranges, services, and agencies. The committee further recognizes that each of DoD’s test ranges will face specific challenges and opportunities unique to the individual facility or organization that are not addressed in this report.

This unclassified study addresses certain key challenges and solutions at a general level due to the sensitive nature of many U.S. military capabilities and the intelligence gathered on current and future threats posed by U.S. adversaries. Other topics in the statement of task are not readily addressed without referencing controlled unclassified information (CUI). The second, classified phase of this study will provide important additional detail and context regarding the test and evaluation requirements for the ranges posed by new weapons capabilities and threat characteristics. Appendix A includes a matrix mapping the committee’s work against the statement of task (SOT), with the disposition of task areas not addressed or partially addressed in this report.

MILITARY RANGES PAST, PRESENT, AND FUTURE

DoD operates a large number of ranges, spanning all of the services, which are used to test and evaluate the effectiveness of military systems and train operators in every domain: land, sea, air, space, and cyberspace. These ranges and their infrastructure and associated tools (and personnel) are a critical component of the DoD acquisition community and its systems development process, and they play critical research, experimentation, development, test, and training roles in the never-ending modernization efforts aimed at ensuring that the country’s warfighters are provided with the operational superiority its citizens expect if they are to deal effectively with the nation’s adversaries. Among the technologies that must be capable of being tested at the nation’s ranges are directed energy weapons, hypersonic platforms, autonomous systems, artificial intelligence, space systems, 6th generation aircraft, long-range munitions, acoustic and non-acoustic technologies for undersea warfare, advanced electronic warfare/cyber capabilities, chemical and biological defense, and hard and buried target countermeasures.

The performance and credibility of military weapon systems against threats and adversaries are foundational to U.S. deterrent capability and battlefield advantage. While this fact has not changed, the fundamental nature of warfare has shifted as a result of the information revolution,

the emergence of linked battle networks, and changes to the concepts of operation resulting from what has been termed previously a “revolution in military affairs” (Mowthorpe, 2005; Murray, 1997). The concept of networked warfare—linking sensors, command and control, and precision weapons across platforms—became visible to the world in the first Gulf War. Adversaries, including China and Russia, observed and reacted, developing their own integrated battle networks that can hold U.S. forces at risk.

Future U.S. deterrent and combat capability will depend on the ability to close the kill chains and dismantle adversary kill chains. The fight will span all domains of conflict—air, land, sea, space, and cyberspace—and extend to competition before the active phase of conflict. However, performance and credibility against emerging threats can only be demonstrated through the testing of production-representative systems in realistic operational conditions, against realistic representations of adversarial capabilities. Here, speed is critical for all activities that lead to the fielding of combat capability—including test and evaluation. As adversaries rapidly develop and deploy their own advanced weapon systems in an iterative fashion, it is critical that the range enterprise and infrastructure support testing of the latest systems and technologies in order to keep up with and stay ahead of the most current—and anticipated—threats. The variety of adversary weapons in development, and the speed with which they are being tested and deployed, is at a pace and scale that exceeds anything the United States has seen in a generation or more. The U.S. Air Force Chief of Staff, General Charles Q. Brown, states this bluntly in the title of his August 2020 report “We Must Accelerate Change or Lose (ACOL)” (Brown, 2020).

Today’s information technologies and digital infrastructure create a fundamentally new dynamic for the practice of OT&E, and the tools, approaches, infrastructure, and skill sets must keep pace. As Marc Andreesen commented in 2011, “Software is eating the world” (Andreesen, 2011). His essay described how software-centric products and services were taking over large segments of the economy while fundamentally disrupting value chains across the physical world. Just as no industry, system, or product is immune, OT&E and DoD’s range infrastructure must address the interrelated sets of challenges and opportunities. A major theme of the Defense Innovation Board’s report on DoD software acquisition is that “software is different than hardware”—software intensive systems are the core of U.S. offensive and defensive fighting capabilities, and software is never “done” (Defense Innovation Board, 2019). In some cases, the science and approaches behind the testing of new software-driven technologies such as artificial intelligence (AI) and machine learning are still being developed. While these facts are daunting, the same set

of information and digital technologies, when applied appropriately and at scale, will provide the solution to the challenges if the necessary steps are taken to modernize DoD’s range infrastructure and test methods.

FUNDAMENTAL THEMES

Given this situation and in view of the testimony of experts from across DoD, technology development, and commercial enterprise combined with data collected from site visits, the committee structured the report around the following three fundamental themes, which are further expanded in the paragraphs below:

1. Future combat will demand connected kill chains in a joint all-domain operations environment.
2. Digital technologies are dramatically reshaping the nature, practice, and infrastructure of testing.
3. Speed-to-field is today’s measure of operational relevance, which is in turn a continuously moving target.

Future Combat Will Demand Connected Kill Chains in a Joint All-Domain Operations Environment

As described in the book The Kill Chain, platforms and weapons are the tools of the military, but ultimately “the ability to prevail in war, and thereby prevent it, comes down to one thing: the kill chain. . . . It involves three steps: The first is gaining understanding about what is happening. The second is deciding about what to do. And the third is taking action that creates an effect to achieve an objective” (Brose, 2020, p. xviii). This concept is not new, and it is even documented as a “mission thread with a kinetic outcome” in the DoD Mission Engineering Guide (DoD, 2020, p. 36). However, the networked and interconnected nature of today’s kill chains requires more from the DoD test enterprise. At the committee’s January 2021 workshop, Col. Jason Eckberg, DoD’s deputy director of electromagnetic spectrum dominance, describes the shift required from one-on-one tests that focus on a platform’s lethality and survivability to “tests with multiple components that assess overall force effectiveness.” (NASEM, 2021, p. 7)

As stated in the 2018 National Defense Strategy, “We face an ever more lethal and disruptive battlefield, combined across domains, and conducted at increasing speed and reach—from close combat, throughout overseas theaters, and reaching to our homeland” (DoD, 2018, p. 3). In order to test for the future fight, with composable kill chains across domains, platforms, networks, and command and control systems, the

integration between different platforms and systems will be increasingly tested. Furthermore, test approaches and range capabilities will have to be as agile and adaptable as future weapon systems, as those systems and threats evolve and thus change warfighting tactics, techniques, and procedures. Because kill chains will span many or all warfighting domains, from undersea to space and everything in between, test approaches and test ranges will require the ability to stitch together multiple ranges alongside virtual and constructive models of both “blue” and “red” forces. As Marc Bernstein, the chief scientist for the office overseeing all Air Force acquisition, said at the committee’s January 2021 workshop, it will be necessary to combine many, if not all, of the nation’s test ranges into a very complex “range of ranges” (NASEM, 2021, p. 6).

Digital Technologies Are Dramatically Reshaping the Nature, Practice, and Infrastructure of Test

The weapon systems of today and of the future are defined as much by software as hardware, as are the adversary threats U.S. forces face. Battle networks are central to current and future kill chains, and information technology is at the heart of cyber and electronic warfare. However, as noted at the committee’s January 2021 workshop by David Tremper, director of electronic warfare in the Office of the Secretary of Defense, the ranges lack the software-defined agile threat systems that would allow testing against more representative threats (NASEM, 2021, p. 5). Artificial intelligence and machine learning create novel challenges that require the development of underlying test science to address learning systems that adapt and respond to their environments and the systems deployed against them. Bernstein warns that it will be AI against AI, and test and evaluation (T&E) must model those threats (NASEM, 2021, p. 5). Moreover, as noted by Joshua Marcuse, head of strategy and innovation at Google and formerly executive director of the Defense Innovation Board, some military ranges seem to have barely entered the digital age at all, while facing challenges in exponentially increasing requirements for data and the accompanying instrumentation, collection, telemetry, communication, storage, processing and analytics (NASEM, 2021, p. 9).

Some of the same technologies also create opportunities to reinvent the test ranges for this digital world. Digital engineering techniques and tools offer the promise of co-developing test systems in parallel with the weapon systems they will support and rapidly updating test capabilities to keep pace with the evolution of weapons and the threats they face. “TestDevOps” approaches can mirror the “DevSecOps” agile development processes and platforms increasingly used in system development, enabling comparable responsiveness through automation and continuous

integration/continuous delivery. The expanded integration of modeling and simulation (M&S) with real-world testing in live–virtual–constructive environments will enable the creation of cutting-edge test environments simulating realistic threat densities as well as the adaptability of threat systems. M&S will also support new ways of testing integrated kill chains and enable the evaluation of holistic unit actions and the training of extended forces on how to use new weapon systems. Other M&S benefits include the replication of threats and capabilities that are too sensitive or too dangerous to be reproduced in the real world and the potential to carry out far more tests over wider sets of conditions than are practical on physical test ranges. For all these benefits, digital capabilities create their own unique challenges, such as cybersecurity, the need for rigorous model validation, and limitations in the ability to adequately replicate real-world uncertainty that may constrain applicability for AI systems. Moreover, these digital environments for testing must be both timely and continually refreshed. As described in the Government Accountability Office (GAO) report on the F-35 program’s Joint Simulation Environment (JSE), technical problems with the simulator have put necessary test capabilities years behind schedule, delaying completion of OT&E and the next production milestone decision (GAO, 2021). This example demonstrates how critical digital infrastructure has become in proving operational suitability and the impact to U.S. forces if that infrastructure does not meet the required pace. It also illustrates the many technical and programmatic challenges inherent in development of the complex, high-fidelity, validated M&S environments required for the testing of the most advanced weapon systems that must be understood and addressed to realize the full promise of M&S for test and training.

Speed-to-Field Is Today’s Measure of Operational Relevance, Which Is in Turn a Continuously Moving Target

The vice chairman of the Joint Chiefs of Staff, General John Hyten, stated that “inserting speed into everything the Defense Department does is a priority.” The reason: “when you look at our competitors, large and small, one of the things that you find that they have in common is they’re moving very, very fast. And we are not” (Cronk, 2020). While testing is only a part of the sequence of events in fielding a weapon system, it is generally the step that allows a declaration of operating capability or the approval for rate production. Neither the operational test system nor the ranges that support testing are optimized to increase speed of capabilities to the field. As Behler noted at the January 2021 workshop, “You could update the F-35 plane as fast as an iPhone app . . . and you wouldn’t actually be any faster relative to your adversary because you would

still need a year for me to test it” (NASEM, 2021, p. 2). Pacing functions include readiness of the systems under test and the required test infrastructure, which drive the ability and schedule of those systems to undergo and pass operational testing. Our ability to replicate current operational threats is likewise painfully slow. At the public workshop, Ed Greer, the former deputy assistant secretary of defense for developmental test and evaluation, shared how it takes an average of 3 to 5 years from the time that intelligence is collected on threats to the time those threats are instantiated into testing, during which time adversaries can build new systems faster than intelligence centers can build models.² As a result of this, test ranges and programs must anticipate “pop up” testing requirements driven by new tactics and techniques, or emerging threats, will be the norm in the future, rather than the exception.

There are many factors that slow DoD’s ability to test the latest systems against the latest threats, and often its range enterprise creates limiting factors. Some of these bottlenecks are related to the capacity of the ranges, driven by dated, limited, or unreliable sensing and communications systems; the constrained availability of unique test facilities; the need to link multiple ranges to support a test; and limitations on the ability to conduct tests simultaneously or around the clock. Others, as noted above, tie to the ability to update software-intensive test support systems at the pace required or information security systems and processes that prevent rapid switching between security levels. At the January 2021 workshop, Marcuse noted that despite the digital revolution of the past several decades, testing remains optimized for hardware (NASEM, 2021, p. 4), and does not take advantage of tools, processes, and automation that enable speed and responsiveness in test infrastructure, test execution, and test problem remediation.

Perhaps the greatest challenge to speed and responsiveness for the test ranges, however, is the multi-year process by which funding is allocated to sustain, operate, and modernize the range enterprise. Funds come from a multiplicity of centralized and distributed sources, with different rules for the application and timing of expenditures, which cannot be practically combined or redistributed to meet the most pressing priorities, and often tie the hands of the teams charged to ensure the readiness of the ranges. Beyond the inefficiency and uncertainty of the budget process, in many cases the range investments are inadequate to the meet the capability needs for the weapon systems on the schedules they must support. As shared by Behler at the committee’s December 2020 meeting,

___________________

² From remarks delivered on January 29, 2021, at the public workshop; recording available at https://www.nationalacademies.org/event/01-28-2021/assessing-the-physical-and-technical-suitability-of-dod-test-and-evaluation-ranges-and-infrastructure-meeting-2-and-workshop.

a rule of thumb in the test community is that approximately 1 percent of acquisition spending should be allocated to test and evaluation infrastructure. For critical domains, such as space, the planned investment levels fall far below that level.³

FIVE CATEGORIES OF SOLUTIONS

The study committee identified five sets of solutions that reflect the actions required to address the key cross-cutting themes. Within these categories there are complementary individual recommendations for addressing the challenges and opportunities described in the chapters of this report. Taken together, these committee judgments will enable DoD to develop, implement, and sustain the range enterprise capabilities needed to meet the challenges posed by the three major themes (connected kill chains, digital technology, and speed-to-field).

Across the three main themes the committee’s judgments fall into 5 categories of solution sets:

1. Develop the “range of the future” to test complete kill chains in joint all-domain environments.
2. Restructure the range capability requirements and process for continuous modernization and sustainment.
3. Bootstrap a new range operating system for ubiquitous M&S throughout the weapon system development and test life cycle.
4. Create the “TestDevOps” digital infrastructure for future operational test and seamless range enterprise interoperability.
5. Reinvent the range enterprise funding model for responsiveness, effectiveness, and flexibility.

Develop the “Range of the Future” to Test Complete Kill Chains in JADO Environments

In his opening remarks to the committee in December 2020, Behler emphasized that the ranges must be able to integrate systems and domains to enable the promise of combined arms for decisively closing blue force kill chains in the future fight.⁴ However, OT&E and the range enterprise have focused on the test of single programs and systems against their individual operational requirements, and collaborative effects have not

___________________

³ From remarks delivered at December 4, 2020, committee meeting; recording available at https://www.nationalacademies.org/our-work/assessing-the-physical-and-technical-suitability-of-dod-test-and-evaluation-ranges-and-infrastructure.

⁴ Ibid.

typically informed major test requirements. Large-scale exercises, such as Black Dart, Emerald Flag, and Orange Flag, have been conducted in recent years to investigate the feasibility of seamless operations across services and domains, and these exercises also illuminated the challenges in performing rigorous and repeatable operational testing with this scale and complexity. With the new centrality of integration and all-domain warfighting in the national defense strategy, the infrastructure of the range of the future must instantiate the ability to bring together systems-of-systems across warfighting domains, including land, air, sea, space, and cyberspace, and to measure the effectiveness of end-to-end kill chains performing against threats across those domains.

The range enterprise must be able to connect the ranges together with speed and agility and to perform efficient and effective command and control of tests across this distributed range-of-ranges while maintaining a high level of safety and security. This means that all performing and supporting organizations must have a clear and common understanding of multi-domain operations concepts and definitions and of the requirements for effective testing of today’s diverse cyber-physical systems. Connections via secure, high-bandwidth lines of communication governed by common data standards, processes, and procedures will enable the ranges to collect, share, store, manage, and analyze the massive volumes of test data. A new organizational construct embodied in a joint program office with supporting policy and doctrine changes is recommended to manage the framework for testing of kill chains across systems and technologies, starting from use cases and concepts of operation, continuing through capability development and evolution, enabling integrated kill chain testing, and ultimately providing feedback to both the operational and acquisition communities for informed operations and future developments.

Growing encroachment also poses particular threats for integrated kill chain testing, whether in the physical, radiofrequency, or cyber domains, as tests increasingly span geographical regions and make use of extensive spectrum resources for sensing, communications, and weapons effects. The committee explores the potential mitigations for loss of space or spectrum, with recommendations on managing internal encroachment within DoD’s span of control as well as on managing external encroachment via U.S. government action or technological solutions and workarounds.

Restructure the Range Requirements and Resourcing Process for Continuous Modernization and Sustainment

Range modernization requirements are primarily determined by program test requirements, which are established by programs in the acquisition phase. While establishment of test requirements is intended

to occur via test and evaluation master plans (TEMPs) developed early in the acquisition process, often the understanding of test approaches and resulting range needs is immature. Focus and priority for the ranges and Test Resource Management Center (TRMC) is on test requirements over the next 3 to 5 years, so active preparation for the technologies, techniques, and range infrastructure needed in the next 10 to 15 years receives less attention and resources. Furthermore, test requirements are difficult to modify once testing needs and priorities are established, limiting the flexibility and agility of the range enterprise as the ranges are faced with addressing rapidly evolving weapons technologies and continuously advancing threats. The committee finds that recapitalization and modernization for broader or longer-term use beyond individual test program requirements is not incentivized. Greater attention to range enterprise modernization needs early in the acquisition process by the Joint Requirements Oversight Council (JROC), including tracking of current and projected range capability gaps, is recommended to address the observed range requirements shortfalls.

In addition to range modernization requirements driven by individual program test needs, the ranges must now react to a new set of capability and resource challenges and gaps driven by kill chain testing of integrated systems in representative Joint All-Domain Operations (JADO) environments. The current piecemeal, program-driven requirements process results in many projects to develop individual range capabilities, while structures, accountability, and processes to link and integrate these capabilities are immature or ad-hoc. A new joint program office is recommended to develop, maintain, and update cross-service and cross-domain mission threads, JADO test approaches and an integrated systems test requirements framework, and resulting range and infrastructure demands to test for the future fight.

Encroachment on the nation’s test ranges, including both physical intrusions and limitations, and reduced access to electromagnetic spectrum resources, is also driving test range requirements for the future. Actions are recommended to increase DoD’s oversight and prevention of internal range encroachment actions, while strengthening DoD’s role in U.S. government in regulation and allocation of adjoining geographical regions and military-relevant radio frequency (RF) bands. Additional steps to identify potential Allied open-air range resources, and determine how emerging commercial communications capabilities can be best used to address high-bandwidth range requirements, will also help to mitigate encroachment concerns for the future.

Bootstrap a New Range Operating System for Ubiquitous M&S Throughout the Weapon System Development and Test Life Cycle

M&S are rapidly increasing in importance for test and evaluation of weapon systems. Realistic physical replication of the quantity and diversity of adversary threats that will be faced by U.S. systems in combat is becoming impractical on U.S. test ranges. Often, testing of sensitive capabilities in open-air venues is inadvisable due to increasing vulnerability to adversary monitoring and surveillance. Moreover, the logistical complexity and cost for extensive physical testing of integrated systems across full operational envelopes for multiple use cases is prohibitive.

M&S provide necessary and useful capabilities that can address many of these challenges. Digital engineering and model-based engineering approaches may provide an integrated virtual representation of weapon systems, their interfaces, the operational environment, and diverse threats across the full system life cycle from requirements development and architecture through design, manufacturing, integration, test and evaluation, and sustainment. Increasing availability and reduced costs of high-performance computing and validated functional and physics-based models, coupled with “big data” analytic capabilities, and the application of machine learning expand the opportunity to complement physical testing with large numbers of virtual tests conducted in parallel rather than sequentially. Statistical techniques can be applied to optimize test approaches combining physical and virtual testing, increasing the value of each physical test while minimizing expensive and time-consuming range testing and potentially reducing the quantity and variety of required test assets. Modern, agile, and iterative software development paradigms, such as DevSecOps (for Development-Security-Operations) can increase the pace of test development and execution in the virtual environment, allowing more rapid adaptation to changing operational concepts and threat scenarios.

However, M&S are not a “silver bullet.” Fundamental challenges with integration, physical system equivalence, validation, and realistic development schedules and costs remain to be addressed (Wolfe, 2021). While reducing the projected increase of demands on the range enterprise, and offering approaches that may be more cost-effective and practical than physical testing alone, it is unlikely that M&S will reduce the overall amount of testing on the ranges. At a minimum, testing will be required to validate models, quantify uncertainties, and understand the limits of simulations. To achieve the greatest benefits from M&S for the T&E and the ranges, resources must be applied to create a centralized, persistent and accessible M&S environment and library of validated models that can be shared across the T&E enterprise, coupled with education and training to achieve effective utilization. M&S must be applied from the earliest

phases of concept development, and sustained throughout the life cycle of the programs and systems addressed. Robust and efficient multi-level security approaches must be developed to manage diverse classification guidance and protect sensitive information while moving at the pace required to stay ahead of adversary threats.

Create the “TestDevOps” Digital Infrastructure for Future Operational Test and Seamless Range Enterprise Interoperability

DevSecOps—the combination of agile development methods, continuous integration and continuous delivery (CI/CD) of constantly increasing and evolving capabilities, automation in testing and verification, and secure practices throughout the development life cycle—has transformed commercial software development approaches and are increasingly being adopted across the national security community. Similar concepts are needed for rapid and continuous modernization of the range enterprise, to assure connectivity and security, allow integration of capabilities from different ranges into a composable “range of ranges,” make productive use of the exponentially increasing amounts of data generated and collected in testing, and effectively integrate M&S with physical testing for all-domain kill chain T&E.

However, the range enterprise and T&E functions lack a comprehensive, flexible, and scalable data strategy, resulting in the inefficient use of data collected currently, and the failure to collect some of the most important data that can be used to inform and optimize operational testing. Ranges often lack the resources, infrastructure, tools, and processes to handle the scope and scale of required data and computational operations, and are typically not operating in a seamless end-to-end digital thread from requirement definition through verification and validation (V&V). Data communication and integration challenges, including limited bandwidth for collection and timely transmission, and incompatible standards and formats for sharing and combining data across ranges and test systems, result in extensive manual efforts and delays in analysis when performing tests across multiple ranges or operational domains. Challenges often beyond the control of the ranges further burden the T&E system, particularly slow, laborious, and manual security approval processes coupled with an absence of distributed, multi-level security information systems.

DoD and the range enterprise must develop a data strategy that emphasizes speed and interoperability, and must further define, adopt and promulgate modern interoperable approaches for seamless, fast and secure collection, transmission, sharing, and analysis of very large data sets. Concurrently, the Director of Operational Test and Evaluation (DOT&E) and the ranges need to incorporate modern software

development approaches, enabling a “TestDevOps” construct (for Test-Development-Operations, based on DevSecOps concepts) that leverages digital engineering, permits rapid capability upgrades for software-intensive test systems, and enables effective testing of future systems that will incorporate new software- and data-driven technologies like artificial intelligence and machine learning (AI/ML).

Reinvent the Range Enterprise Funding Model for Responsiveness, Effectiveness, and Flexibility

Given the pace of technological change in U.S. weapon systems, and the relentless and rapid advances in adversary systems across all domains of conflict, speed in testing and learning is critically important for ranges as they seek to modernize their capabilities while sustaining test assets that are often decades old, single-string, and unreliable. A revitalized approach to early definition of range capability requirements is required at both the program and enterprise levels, but that change alone will not address the resource challenges imposed on the range enterprise by today’s complex, uncertain, and inflexible approach to funding for operations, sustainment, and modernization.

Today, range capabilities are funded by a variety of streams from individual programs, the military services and DoD agencies, military construction (MILCON), and central pools such as TRMC resources. Funding levels are subject to annual appropriations and vulnerable to out-prioritization by other emerging DoD needs, resulting in great unpredictability from year to year. Much of the funding is limited to specific purposes and cannot be reallocated based on greatest need, while funding often arrives late in the fiscal year and may have unrealistic requirements for timing of obligations and expenditures for test systems that require years of development and construction. Downward trends in MILCON for T&E collide with increasing demands for test time and modernized capabilities.

Based on this funding landscape, some immediate steps are needed to improve the flexibility in timing of spending for range infrastructure, and also increase the ceilings for flexible use of research, development, test, and evaluation (RDT&E) or operations and maintenance (O&M) funding for minor military construction to enhance the valuable authorities already provided by the Congress. More sustained impact will be enabled through the pilot of additional changes to DoD’s range funding mechanisms, including establishment of a Range Working Capital Fund to stabilize funding for modernization and sustainment, and demonstration of these changes to prioritize and correct capabilities gaps needed for timely T&E of new systems with multi-domain test requirements.

STRUCTURE OF THE REPORT

The remaining chapters of this report are structured into three broad pieces: an examination of what the future of warfare could look like and the implications of that envisioned future for operational testing and evaluation; three chapters each devoted to the one of the three broad themes of this report (kill chains, digital technologies, and speed-to-field); and a chapter summarizing the report’s findings and recommendations. The structure is designed to enable the reader to understand the environmental forces, the state of DoD testing, and the projected future requirements that resulted in the key themes and lines of effort summarized above. The individual chapters provide detail and supporting examples on the themes, the lines of effort, and the recommendations they encompass.

More specifically, Chapter 2, on the envisioned future of warfighting and OT&E, addresses several questions: What is the future of warfare, and what will be required for OT&E? How will testing need to change? What are the implications for DoD’s test and training range enterprise? Finally, what does “good” look like?

Chapter 3, on kill chains, multi-domain operations, and the associated future testing needs, begins with an examination of how kill chains and multi-domain operations work. Next it discusses the challenges to testing of systems in operationally relevant ways that arise in tests involving kill chains or carried out over multiple domains, or both. Finally, the chapter examines what the ranges require for carrying out a kill chain approach to operational testing (OT).

Chapter 4, on digital technologies—mainly focusing on modeling and simulation and on the ranges’ digital infrastructure—discusses two broad categories of challenges. The first is testing issues related to the increasing role of digital engineering and modeling and simulation in the development of weapon systems; the second is challenges related to the ranges’ digital infrastructure, such as the sharing of data among ranges and the corresponding requirements for increased connectivity, interoperability, and security. Among the questions addressed by the chapter are: What does digital engineering mean for test, and why does M&S matter for OT? What needs to change in the range enterprise to take full advantage of digital engineering and M&S? What is needed in the range enterprise for effective OT of software-centric systems? Finally, what range digital infrastructure is required for connectivity and effectiveness?

Chapter 5, on speed-to-field, begins with a discussion of why speed is so critical for OT in today’s world. It examines the range enterprise factors that contribute to delays in fielding systems and kill chains and asks what needs to change in the way that OT and range requirements are established. Finally, it surveys the resource challenges that affect OT speed and efficiency and recommended remedies.

Chapter 6 wraps up the main portion of the report by bringing together all of the committee’s key findings and recommendations in one place, with the recommendations by stakeholders.

The appendices provide additional detail on the background, context, study approach, and sources of information that informed the committee’s work and report outcomes. Appendix A includes the study statement of task. Appendix B provides a summary of the site visits the committee conducted. Appendix C provides biographical sketches of the committee members. Appendix D is the disclosure of unavoidable conflicts of interest. Appendix E lists the abbreviations and acronyms used in this report.

REFERENCES