National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 5
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 6
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 7
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 8
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 9
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 10
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 11
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 12
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 13
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 14
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 15
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 16
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2021. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight. Washington, DC: The National Academies Press. doi: 10.17226/26181.
×
Page 17

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction To protect itself from attacks by foreign forces, the United States relies upon its armed services, which in turn rely upon weapons and other systems to provide them with the tools they need to successfully neutralize adversary combat capabilities. Maintaining the armed services’ warfighting advantage requires a steady stream of new and improved weapons and technologies. A crucial step in acquiring and using these assets is testing their effectiveness and suitability on Department of Defense (DoD) ranges. DoD has testing ranges that span the globe where new military technologies are tested based on real threats, tasks, and environments to ensure their combat readiness. These ranges are a vital aspect of the nation’s defense, but will they be able to adequately test the increasingly complex military technologies of the future, at the pace required? Former Director of DoD Operational Test and Evaluation (OT&E) Robert Behler has noted that the U.S. test range system dates to the years during and after World War II, with the most significant updates having been carried out during the Cold War.1 The committee recognizes that there have been further upgrades and modernization in the 30 years following the end of the Cold War. However, Raymond O’Toole, the current acting director of OT&E, asserted at the committee’s January 2021 public workshop that the ranges have not kept pace with testing demands, technology development, or the capabilities of adversaries (NASEM, 2021, p. 2). Consider just some of the major environmental shifts that have affected weapon system development and testing since that time period:  The United States now faces not one, but two peer military adversaries in China and Russia, and China is an economic powerhouse that has broadened the realm of great power competition far beyond avenues of military conflict.  Other adversaries such as North Korea and Iran have rapidly developed missile, cyber, and nuclear weapon technology that can now threaten the U.S. homeland. Furthermore, rival nations have made a concerted effort to outpace the United States in weapons technologies of the future, including artificial intelligence, hypersonics, and space systems.  Commercially available internet technology, mobile communications, cloud computing, and ubiquitous software have fundamentally reshaped the architecture and testing of weapon and support systems. Moreover, Commercial demand and development are creating robust competition for the physical geography and radio frequency spectrum that OT&E has relied upon for decades. The stewardship and use of DoD test ranges relies on multiple stakeholders to effectively test the nations’ defense systems. The Developmental Test and Evaluation (DT&E) community is responsible for funding and executing the upgrades upon which these future capabilities depend and therefore must be fully 1 From remarks delivered at December 4, 2020, committee meeting. Recording available at https://www.nationalacademies.org/our-work/assessing-the-physical-and-technical-suitability-of-dod-test-and- evaluation-ranges-and-infrastructure. Accessed May 17, 2021. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 5

included in policy, resourcing, and allocation discussions moving forward. It is only through the full participation of all the stakeholders that lasting change can be realized. Urgent and substantial changes to the modernization, sustainment, operation and resourcing of the range enterprise are required to support the scale and diversity of weapon system testing and to meet the challenges posed by rapid insertion of new technology over the next 10–15 years. The consequences of inaction will be severe, and recovery will be difficult. Technologies such as artificial intelligence, hypersonics, cyber weapons, and directed energy are creating new test capability requirements for DoD ranges. Rapidly improving threats, particularly from peer adversaries such as China and Russia, make the need to test a system’s survivability just as important as testing its lethality. Peer and non-peer adversaries, including actors like North Korea, increasingly employ asymmetric capabilities such as cyber-attacks. These trends, when extrapolated to 2035, demand a new approach to modernizing DoD’s ranges’ technical and physical attributes. This new approach must not only preserve the current core capabilities, but also take a more holistic look at the aggregate capabilities needed to address the software- intensive nature of future weapon systems, as opposed to the too-often piecemealed upgrades that occur today. STUDY CHARGE Against this backdrop, DoD’s Office of the Director, Operational Test and Evaluation (OT&E) requested the Board on Army Research and Development of the National Academies of Sciences, Engineering, and Medicine to perform a study assessing the physical and technical suitability of DoD test and evaluation ranges, infrastructure, and tools for determining the operational effectiveness, suitability, survivability, and lethality of military systems (see Box 1-1, Statement of Task). While the ranges’ staffing and organizational structures are clearly crucial to their success, the statement of task specified that the study should focus not on the personnel and staffing but on the facilities themselves. This study is the first of two studies that were requested by DoD’s Director of Operational Test and Evaluation. It is an unclassified review that was designed as a standalone study but that will also provide the foundational elements for a follow-on classified study that is scheduled to begin before the official publication of this report. The purposes of conducting this study at the unclassified level were to make it possible to include on the committee as wide a range as possible of members from the science and engineering community as well as to build maximum awareness of the serious challenges to be addressed today. The follow-on study, which will be informed by the contents of this report, will assess how well the ranges are able to simulate the threats, threat countermeasures, and capabilities of near-peer adversaries and to test DoD systems in future operational scenarios. The goal of carrying out the two studies is that, by taking advantage of the full range of the nation’s science and technology community in the unclassified study while also having access to complete information on adversaries’ capabilities in the classified study, the two together will offer a comprehensive assessment of DoD’s testing and evaluation ranges and infrastructure. The Committee’s Approach To carry out the statement of task and evaluate the nation’s military ranges, the National Academies Board on Army Research and Development (BOARD) assembled a study committee composed of experts from the military, industry, academia, and government. The committee, assisted by BOARD staff members, assembled a broad collection of written and graphic information related to the ranges and OT&E, including many previous reports authored by various components of DoD, the National Academies, the Congressional Research Service, RAND Corporation, and other groups. This literature formed the foundation upon which the committee based its judgments. It was supplemented by presentations, typically held via Zoom, by PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 6

multiple military officials, both active and retired, as well as other experts familiar with military ranges and the challenges of OT&E in the current environment. BOX 1-1 Statement of Task The National Academies of Sciences, Engineering, and Medicine will convene an ad hoc committee to assess the physical and technical suitability of the Department of Defense’s (DoD) ranges, infrastructures, and tools used for test and evaluation (T&E) of military systems’ operational effectiveness, suitability, survivability, and lethality across all domains (land, sea, air, space, and cyberspace). Specifically, the committee will: 1. Assess the aggregate physical suitability of DoD’s ranges to include their testing capacity, the condition of their infrastructure, security measures, and encroachment challenges. 2. Assess the technical suitability of ranges to include spectrum management, instrumentation, cyber and analytics tools, and their modeling and simulation capacity. 3. Evaluate the following attributes for each range: o Physical Attributes of Range: Do ranges allow for full exercise of tested systems in the manner they will be used to achieve their mission? o Electromagnetic Attributes of Range: Can the system under test, and emulated threats to the system, access and utilize spectrum as designed and needed? o Range Infrastructure: Can range instrumentation properly and fully assess system performance and record test data (as well as training data that could be applied to T&E requirements)? Can range tools adequately process and transmit test data and efficiently provide test results? o Test Infrastructure Security: How secure are ranges, infrastructure and test capabilities against physical and cyber intrusion that could lead to exploitation of weapon systems performance data by an adversary? o Encroachment Threats and Impacts: What are the existing and potential future encroachment threats and impacts (physical space, spectrum, alternative/competing DoD uses)? 4. The committee will recommend how the DoD can address and/or mitigate any existing or anticipated deficiencies, and test and evaluate future technologies anticipated to arrive between now and 2035, including discussion of planning and resource allocation for the overall test range enterprise. These technologies include, but are not limited to: o Directed energy, hypersonic systems, autonomous systems, artificial intelligence, space systems and threats, 6th generation aircraft, advanced acoustic and non-acoustic technologies for undersea warfare, and advanced active electronic warfare/cyber capabilities. On January 28–29, 2021, the committee held a public virtual workshop, Assessing the Physical and Technical Suitability of DoD Test and Evaluation Ranges and Infrastructure. Over the course of 2 days, the committee heard presentations from representatives of the Test Resource Management Center, the individual services’ testing and evaluation departments, various other service groups involved in the development and testing of military systems, national laboratories, universities, and industry. In April 2021, Key Challenges for Effective Testing and Evaluation Across Department of Defense Ranges: Proceedings PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 7

of a Workshop—in Brief was published to summarize the workshop’s presentations and what the committee learned from them (NASEM, 2021). Over the course of the study, committee members and BOARD staff carried out seven site visits, some in person and some virtually, at a diverse selection of ranges that were as representative as possible of the wide array of challenges facing OT&E over the coming decade and a half. A summary of the site visits is provided in Appendix C. During those visits the committee and staff heard from a wide range of military representatives with intimate knowledge of the day-to-day workings of the ranges and OT&E, and the information gleaned from these visits, combined with the knowledge and insights that the individual committee members brought to the process, formed much of the foundations for the deliberations that resulted in this report. For those deliberations the committee members met regularly, both in full committee and in subcommittees, from December 2020 through July 2021. Depending on their expertise and interest, different committee members contributed to different parts of this report, but all writing, from the narrative to the findings and recommendations, was reviewed by and agreed upon by the entire committee. This report is the result of that process. The extraordinary diversity of DoD missions and test environments, and the large quantity of range locations and installations, precluded an exhaustive evaluation of all range capabilities and gaps in relation to the future OT&E landscape. DoD’s test and training ranges number over 500 in total, including the 23 major facilities in the MRTFB. Additionally, DoD does not currently have standardized and comprehensive reporting on test ranges and facilities. To assess the current physical and technical state of the test ranges, the committee selected representative ranges spanning all domains (land, sea, air, space, and cyberspace) to provide insights on the aggregate challenges with operational testing unique to each domain. This strategy enabled the committee to report on concerns and conditions that were articulated by multiple ranges, services, and agencies. The committee further recognizes that each of DoD’s test ranges will face specific challenges and opportunities unique to the individual facility or organization that are not addressed in this report. This unclassified study addresses certain key challenges and solutions at a general level due to the sensitive nature of many U.S. military capabilities and the intelligence gathered on current and future threats posed by U.S. adversaries. Other topics in the statement of task are not readily addressed without referencing controlled unclassified information (CUI). The second, classified phase of this study will provide important additional detail and context regarding the test and evaluation requirements for the ranges posed by new weapons capabilities and threat characteristics. Appendix A includes a matrix mapping the committee’s work against the statement of task (SOT), with the disposition of task areas not addressed or partially addressed in this report. MILITARY RANGES PAST, PRESENT, AND FUTURE DoD operates a large number of ranges, spanning all of the services, which are used to test and evaluate the effectiveness of military systems and train operators in every domain: land, sea, air, space, and cyberspace. These ranges and their infrastructure and associated tools (and personnel) are a critical component of the DoD acquisition community and its systems development process, and they play critical research, experimentation, development, test and training roles in the never-ending modernization efforts aimed at ensuring that the country’s warfighters are provided with the operational superiority its citizens expect if they are to deal effectively with the nation’s adversaries. Among the technologies that must be capable of being tested at the nation’s ranges are directed energy weapons, hypersonic platforms, autonomous systems, artificial intelligence, space systems, sixth-generation aircraft, long-range munitions, acoustic and non-acoustic technologies for undersea warfare, advanced electronic warfare/cyber capabilities, chemical and biological defense, and hard and buried target countermeasures. The performance and credibility of military weapon systems against threats and adversaries are foundational to U.S. deterrent capability and battlefield advantage. While this fact has not changed, the PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 8

fundamental nature of warfare has shifted as a result of the information revolution, the emergence of linked battle networks, and changes to the concepts of operation resulting from what has been termed previously a “revolution in military affairs” (Mowthorpe, 2005; Murray, 1997). The concept of networked warfare— linking sensors, command and control, and precision weapons across platforms—became visible to the world in the first Gulf War. Adversaries, including China and Russia, observed and reacted, developing their own integrated battle networks that can hold U.S. forces at risk. Future U.S. deterrent and combat capability will depend on the ability to close the kill chains and dismantle adversary kill chains. The fight will span all domains of conflict—air, land, sea, space, cyber, and the electromagnetic spectrum—and extend to competition before the active phase of conflict. However, performance and credibility against emerging threats can only be demonstrated through the testing of production-representative systems in realistic operational conditions, against realistic representations of adversarial capabilities. Here, speed is critical for all activities that lead to the fielding of combat capability—including test and evaluation. As adversaries rapidly develop and deploy their own advanced weapon systems in an iterative fashion, it is critical that the range enterprise and infrastructure support testing of the latest systems and technologies in order to keep up with and stay ahead of the most current— and anticipated—threats. The variety of adversary weapons in development, and the speed with which they are being tested and deployed, is at a pace and scale that exceeds anything the United States has seen in a generation or more. The U.S. Air Force Chief of Staff, General Charles Q. Brown, states this bluntly in the title of his August 2020 report “We Must Accelerate Change or Lose (ACOL)” (Brown, 2020). Today’s information technologies and digital infrastructure create a fundamentally new dynamic for the practice of OT&E, and the tools, approaches, infrastructure and skillsets must keep pace. As Marc Andreesen commented in 2011, “Software is eating the world” (Andreesen, 2011). His essay described how software-centric products and services were taking over large segments of the economy while fundamentally disrupting value chains across the physical world. Just as no industry, system, or product is immune, OT&E and DoD’s range infrastructure must address the interrelated sets of challenges and opportunities. A major theme of the Defense Innovation Board’s report on DoD software acquisition is that “software is different than hardware”—software intensive systems are the core of U.S. offensive and defensive fighting capabilities, and software is never “done” (Defense Innovation Board, 2019). In some cases, the science and approaches behind the testing of new software-driven technologies such as artificial intelligence (AI) and machine learning are still being developed. While these facts are daunting, the same set of information and digital technologies, when applied appropriately and at scale, will provide the solution to the challenges if the necessary steps are taken to modernize DoD’s range infrastructure and test methods. FUNDAMENTAL THEMES Given this situation and in view of the testimony of experts from across DoD, technology development, and commercial enterprise combined with data collected from site visits, the committee structured the report around the following three fundamental themes, which are further expanded in the paragraphs below: 1. Future combat will demand connected kill chains in a joint all-domain operations environment. 2. Digital technologies are dramatically reshaping the nature, practice, and infrastructure of testing. 3. Speed-to-field is today’s measure of operational relevance, which is in turn a continuously moving target. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 9

Future Combat Will Demand Connected Kill Chains in a Joint All-Domain Operations Environment As described in the book The Kill Chain, platforms and weapons are the tools of the military, but ultimately “the ability to prevail in war, and thereby prevent it, comes down to one thing: the kill chain . . . . It involves three steps: The first is gaining understanding about what is happening. The second is deciding about what to do. And the third is taking action that creates an effect to achieve an objective” (Brose, 2020, p. xviii). This concept is not new, and it is even documented as a “Mission thread with a kinetic outcome” in the DoD Mission Engineering Guide (DoD, 2020, p. 36). However, the networked and interconnected nature of today’s kill chains requires more from the DoD test enterprise. At the committee’s January 2021 workshop, Col. Jason Eckberg, DoD’s deputy director of electromagnetic spectrum dominance, describes the shift required from one-on-one tests that focus on a platform’s lethality and survivability to “tests with multiple components that assess overall force effectiveness.” (NASEM, 2021, p. 7) As stated in the 2018 National Defense Strategy, “We face an ever more lethal and disruptive battlefield, combined across domains, and conducted at increasing speed and reach—from close combat, throughout overseas theaters, and reaching to our homeland” (DoD, 2018, p. 3). In order to test for the future fight, with composable kill chains across domains, platforms, networks, and command and control systems, the integration between different platforms and systems will be increasingly tested. Furthermore, test approaches and range capabilities will have to be as agile and adaptable as future weapon systems, as those systems and threats evolve and thus change warfighting tactics, techniques, and procedures. Because kill chains will span many or all warfighting domains, from undersea to space and everything in between, test approaches and test ranges will require the ability to stitch together multiple ranges alongside virtual and constructive models of both “blue” and “red” forces. As Marc Bernstein, the chief scientist for the office overseeing all Air Force acquisition, said at the committee’s January 2021 workshop, it will be necessary to combine many, if not all, of the nation’s test ranges into a very complex “range of ranges” (NASEM, 2021, p. 6). Digital Technologies are Dramatically Reshaping the Nature, Practice and Infrastructure of Test The weapon systems of today and of the future are defined as much by software as hardware, as are the adversary threats U.S. forces face. Battle networks are central to current and future kill chains, and information technology is at the heart of cyber and electronic warfare. However, as noted at the committee’s January 2021 workshop by David Tremper, the director of electronic warfare in the Office of the Secretary of Defense, the ranges lack the software-defined agile threat systems that would allow testing against more representative threats (NASEM, 2021, p. 5). Artificial intelligence and machine learning create novel challenges that require the development of underlying test science to address learning systems that adapt and respond to their environments and the systems deployed against them. Bernstein warns that it will be AI against AI, and test and evaluation (T&E) must model those threats (NASEM, 2021, p. 5). Moreover, as noted by Joshua Marcuse, the head of strategy and innovation at Google and formerly the executive director of the Defense Innovation Board, some military ranges seem to have barely entered the digital age at all, while facing challenges in exponentially increasing requirements for data and the accompanying instrumentation, collection, telemetry, communication, storage, processing and analytics (NASEM, 2021, p. 9). Some of the same technologies also create opportunities to reinvent the test ranges for this digital world. Digital engineering techniques and tools offer the promise of co-developing test systems in parallel with the weapon systems they will support and rapidly updating test capabilities to keep pace with the evolution of weapons and the threats they face. “TestDevOps” approaches can mirror the “DevSecOps” agile development processes and platforms increasingly used in system development, enabling comparable responsiveness through automation and continuous integration/continuous delivery. The expanded PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 10

integration of modeling and simulation (M&S) with real-world testing in live–virtual–constructive environments will enable the creation of cutting-edge test environments simulating realistic threat densities as well as the adaptability of threat systems. M&S will also support new ways of testing integrated kill chains and enable the evaluation of holistic unit actions and the training of extended forces on how to use new weapon systems. Other M&S benefits include the replication of threats and capabilities that are too sensitive or too dangerous to be reproduced in the real world and the potential to carry out far more tests over wider sets of conditions than are practical on physical test ranges. For all these benefits, digital capabilities create their own unique challenges, such as cybersecurity, the need for rigorous model validation, and limitations in the ability to adequately replicate real-world uncertainty that may constrain applicability for AI systems. Moreover, these digital environments for testing must be both timely and continually refreshed. As described in the Government Accountability Office (GAO) report on the F-35 programs Joint Simulation Environment (JSE), technical problems with the simulator have put necessary test capabilities years behind schedule, delaying completion of OT&E and the next production milestone decision (GAO 2021). This example demonstrates how critical digital infrastructure has become in proving operational suitability and the impact to U.S. forces if that infrastructure does not meet the required pace. It also illustrates the many technical and programmatic challenges inherent in development of the complex, high-fidelity, validated M&S environments required for the testing of the most advanced weapon systems that must be understood and addressed to realize the full promise of M&S for test and training. Speed-to-Field Is Today’s Measure of Operational Relevance, Which Is in Turn a Continuously Moving Target The vice chairman of the Joint Chiefs of Staff, General John Hyten, stated that “inserting speed into everything the Defense Department does is a priority.” The reason: “when you look at our competitors, large and small, one of the things that you find that they have in common is they’re moving very, very fast. And we are not” (Cronk, 2020). While testing is only a part of the sequence of events in fielding a weapon system, it is generally the step that allows a declaration of operating capability or the approval for rate production. Neither the operational test system nor the ranges that support testing are optimized to increase speed of capabilities to the field. As Mr. Behler noted at the January 2021 workshop, “You could update the F-35 plane as fast as an iPhone app . . . and you wouldn’t actually be any faster relative to your adversary because you would still need a year for me to test it” (NASEM, 2021, p. 2). Pacing functions include readiness of the systems under test and the required test infrastructure, which drive the ability and schedule of those systems to undergo and pass operational testing. Our ability to replicate current operational threats is likewise painfully slow. At the public workshop, Ed Greer, the former deputy assistant secretary of defense for developmental test and evaluation, shared how it takes an average of three to five years from the time that intelligence is collected on threats to the time those threats are instantiated into testing, during which time adversaries can build new systems faster than intelligence centers can build models.2 As a result of this, test ranges and programs must anticipate “pop up” testing requirements driven by new tactics and techniques, or emerging threats, will be the norm in the future, rather than the exception. There are many factors that slow DoD’s ability to test the latest systems against the latest threats, and often its range enterprise creates limiting factors. Some of these bottlenecks are related to the capacity of the ranges, driven by dated, limited, or unreliable sensing and communications systems; the constrained availability of unique test facilities; the need to link multiple ranges to support a test; and limitations on the ability to conduct tests simultaneously or around the clock. Others, as noted above, tie to the ability to update software-intensive test support systems at the pace required or information security systems and processes that prevent rapid switching between security levels. At the January 2021 workshop Marcuse 2 From remarks delivered on 29 January 2021 at the public workshop. Recording available at https://www.nationalacademies.org/event/01-28-2021/assessing-the-physical-and-technical-suitability-of-dod-test- and-evaluation-ranges-and-infrastructure-meeting-2-and-workshop. Accessed 27 July 2021. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 11

notes that despite the digital revolution of the past several decades, testing remains optimized for hardware (NASEM, 2021, p. 4), and does not take advantage of tools, processes and automation that enable speed and responsiveness in test infrastructure, test execution, and test problem remediation. Perhaps the greatest challenge to speed and responsiveness for the test ranges, however, is the multi-year process by which funding is allocated to sustain, operate, and modernize the range enterprise. Funds come from a multiplicity of centralized and distributed sources, with different rules for the application and timing of expenditures, which cannot be practically combined or redistributed to meet the most pressing priorities, and often tie the hands of the teams charged to ensure the readiness of the ranges. Beyond the inefficiency and uncertainty of the budget process, in many cases the range investments are inadequate to the meet the capability needs for the weapon systems on the schedules they must support. As shared by former Director of OT&E Hon. Robert Behler at the committee’s December 2020 meeting, a rule of thumb in the test community is that approximately one percent of acquisition spending should be allocated to test and evaluation infrastructure. For critical domains, such as space, the planned investment levels fall far below that level.3 FIVE CATEGORIES OF SOLUTIONS The study committee identified five sets of solutions which reflect the actions required to address the key cross-cutting themes. Within these categories there are complementary individual recommendations for addressing the challenges and opportunities described in the chapters of this report. Taken together, these committee judgments will enable DoD to develop, implement, and sustain the range enterprise capabilities needed to meet the challenges posed by the three major themes (connected kill chains, digital technology, and speed to field). Across the three main themes the committee’s judgments fall into 5 categories of solution sets: 1. Develop the “range of the future” to test complete kill chains in joint all-domain environments. 2. Restructure the range capability requirements and process for continuous modernization and sustainment. 3. Bootstrap a new range operating system for ubiquitous M&S throughout the weapon system development and test life-cycle. 4. Create the “TestDevOps” digital infrastructure for future operational test and seamless range enterprise interoperability. 5. Reinvent the range enterprise funding model for responsiveness, effectiveness, and flexibility. Develop the “Range of The Future” to Test Complete Kill Chains in JADO Environments In his opening remarks to the committee in December 2020, former Director of OT&E Hon. Robert Behler emphasized that the ranges must be able to integrate systems and domains to enable the promise of combined arms for decisively closing blue force kill chains in the future fight.4 However, OT&E and the range enterprise have focused on the test of single programs and systems against their individual operational 3 From remarks delivered at 4 December 2020 committee meeting. Recording available at https://www.nationalacademies.org/our-work/assessing-the-physical-and-technical-suitability-of-dod-test-and- evaluation-ranges-and-infrastructure. Accessed 17 May 2021. 4 From remarks delivered at 4 December 2020 committee meeting. Recording available at https://www.nationalacademies.org/our-work/assessing-the-physical-and-technical-suitability-of-dod-test-and- evaluation-ranges-and-infrastructure. Accessed 17 May 2021. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 12

requirements, and collaborative effects have not typically informed major test requirements. Large-scale exercises, such as Black Dart, Emerald Flag, and Orange Flag, have been conducted in recent years to investigate the feasibility of seamless operations across services and domains, and these exercises also illuminated the challenges in performing rigorous and repeatable operational testing with this scale and complexity. With the new centrality of integration and all-domain warfighting in the national defense strategy, the infrastructure of the range of the future must instantiate the ability to bring together systems- of-systems across warfighting domains, including maritime, air, space, land, cyberspace, and the radiofrequency spectrum, and to measure the effectiveness of end-to-end kill chains performing against threats across those domains. The range enterprise must be able to connect the ranges together with speed and agility and to perform efficient and effective command and control of tests across this distributed range-of-ranges while maintaining a high level of safety and security. This means that all performing and supporting organizations must have a clear and common understanding of multi-domain operations concepts and definitions and of the requirements for effective testing of today’s diverse cyber-physical systems. Connections via secure, high-bandwidth lines of communication governed by common data standards, processes, and procedures will enable the ranges to collect, share, store, manage, and analyze the massive volumes of test data. A new organizational construct embodied in a joint program office with supporting policy and doctrine changes is recommended to manage the framework for testing of kill chains across systems and technologies, starting from use cases and concepts of operation, continuing through capability development and evolution, enabling integrated kill chain testing, and ultimately providing feedback to both the operational and acquisition communities for informed operations and future developments. Growing encroachment also poses particular threats for integrated kill chain testing, whether in the physical, radiofrequency, or cyber domains, as tests increasingly span geographical regions and make use of extensive spectrum resources for sensing, communications, and weapons effects. The committee explores the potential mitigations for loss of space or spectrum, with recommendations on managing internal encroachment within DoD’s span of control as well as on managing external encroachment via U.S. government action or technological solutions and workarounds. Restructure the Range Requirements and Resourcing Process for Continuous Modernization and Sustainment Range modernization requirements are primarily determined by program test requirements, which are established by programs in the acquisition phase. While establishment of test requirements is intended to occur via Test and Evaluation Master Plans (TEMPs) developed early in the acquisition process, often the understanding of test approaches and resulting range needs is immature. Focus and priority for the ranges and TRMC is on test requirements over the next three to five years, so active preparation for the technologies, techniques, and range infrastructure needed in the next ten to fifteen years receives less attention and resources. Furthermore, test requirements are difficult to modify once testing needs and priorities are established, limiting the flexibility and agility of the range enterprise as the ranges are faced with addressing rapidly evolving weapons technologies and continuously advancing threats. The committee finds that recapitalization and modernization for broader or longer-term use beyond individual test program requirements is not incentivized. Greater attention to range enterprise modernization needs early in the acquisition process by the Joint Requirements Oversight Council (JROC), including tracking of current and projected range capability gaps, is recommended to address the observed range requirements shortfalls. In addition to range modernization requirements driven by individual program test needs, the ranges must now react to a new set of capability and resource challenges and gaps driven by kill chain testing of integrated systems in representative Joint All-Domain Operations environments. The current piecemeal, program-driven requirements process results in many projects to develop individual range capabilities, while structures, accountability and processes to link and integrate these capabilities are immature or ad-hoc. A new joint program office is recommended to develop, maintain and update cross- PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 13

service and cross-domain mission threads, JADO test approaches and an integrated systems test requirements framework, and resulting range and infrastructure demands to test for the future fight. Encroachment on the nation’s test ranges, including both physical intrusions and limitations, and reduced access to electromagnetic spectrum resources, is also driving test range requirements for the future. Actions are recommended to increase DoD’s oversight and prevention of internal range encroachment actions, while strengthening DoD’s role in U.S. government in regulation and allocation of adjoining geographical regions and military-relevant radio frequency (RF) bands. Additional steps to identify potential Allied open-air range resources, and determine how emerging commercial communications capabilities can be best used to address high-bandwidth range requirements, will also help to mitigate encroachment concerns for the future. Bootstrap a New Range Operating System for Ubiquitous M&S Throughout the Weapon System Development and Test Life Cycle Modeling and simulation (M&S) are rapidly increasing in importance for test and evaluation of weapon systems. Realistic physical replication of the quantity and diversity of adversary threats that will be faced by U.S. systems in combat is becoming impractical on U.S. test ranges. Often, testing of sensitive capabilities in open-air venues is inadvisable due to increasing vulnerability to adversary monitoring and surveillance. Moreover, the logistical complexity and cost for extensive physical testing of integrated systems across full operational envelopes for multiple use cases is prohibitive. M&S provide necessary and useful capabilities that can address many of these challenges. Digital engineering and model-based engineering approaches may provide an integrated virtual representation of weapon systems, their interfaces, the operational environment, and diverse threats across the full system life cycle from requirements development and architecture through design, manufacturing, integration, test and evaluation, and sustainment. Increasing availability and reduced costs of high-performance computing and validated functional and physics-based models, coupled with “big data” analytic capabilities, and the application of machine learning expand the opportunity to complement physical testing with large numbers of virtual tests conducted in parallel rather than sequentially. Statistical techniques can be applied to optimize test approaches combining physical and virtual testing, increasing the value of each physical test while minimizing expensive and time-consuming range testing and potentially reducing the quantity and variety of required test assets. Modern, agile, and iterative software development paradigms, such as DevSecOps (for Development-Security-Operations) can increase the pace of test development and execution in the virtual environment, allowing more rapid adaptation to changing operational concepts and threat scenarios. However, M&S are not a “silver bullet”. Fundamental challenges with integration, physical system equivalence, validation, and realistic development schedules and costs remain to be addressed (Wolfe, 2021). While reducing the projected increase of demands on the range enterprise, and offering approaches that may be more cost-effective and practical than physical testing alone, it is unlikely that M&S will reduce the overall amount of testing on the ranges. At a minimum, testing will be required to validate models, quantify uncertainties, and understand the limits of simulations. To achieve the greatest benefits from M&S for the T&E and the ranges, resources must be applied to create a centralized, persistent and accessible M&S environment and library of validated models that can be shared across the T&E enterprise, coupled with education and training to achieve effective utilization. M&S must be applied from the earliest phases of concept development, and sustained throughout the life cycle of the programs and systems addressed. Robust and efficient multi-level security approaches must be developed to manage diverse classification guidance and protect sensitive information while moving at the pace required to stay ahead of adversary threats. Create the “TestDevOps” Digital Infrastructure for PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 14

Future Operational Test and Seamless Range Enterprise Interoperability DevSecOps—the combination of agile development methods, continuous integration and continuous delivery (CI/CD) of constantly increasing and evolving capabilities, automation in testing and verification, and secure practices throughout the development life cycle—has transformed commercial software development approaches and are increasingly being adopted across the national security community. Similar concepts are needed for rapid and continuous modernization of the range enterprise, to assure connectivity and security, allow integration of capabilities from different ranges into a composable “range of ranges”, make productive use of the exponentially increasing amounts of data generated and collected in testing, and effectively integrate M&S with physical testing for all-domain kill chain T&E. However, the range enterprise and T&E functions lack a comprehensive, flexible, and scalable data strategy, resulting in the inefficient use of data collected currently, and the failure to collect some of the most important data that can be used to inform and optimize operational test. Ranges often lack the resources, infrastructure, tools and processes to handle the scope and scale of required data and computational operations, and are typically not operating in a seamless end-to-end digital thread from requirement definition through verification and validation (V&V). Data communication and integration challenges, including limited bandwidth for collection and timely transmission, and incompatible standards and formats for sharing and combining data across ranges and test systems, result in extensive manual efforts and delays in analysis when performing tests across multiple ranges or operational domains. Challenges often beyond the control of the ranges further burden the T&E system, particularly slow, laborious and manual security approval processes coupled with an absence of distributed, multi-level security information systems. DoD and the range enterprise must develop a data strategy that emphasizes speed and interoperability, and must further define, adopt and promulgate modern interoperable approaches for seamless, fast and secure collection, transmission, sharing, and analysis of very large data sets. Concurrently, the DOT&E and the ranges need to incorporate modern software development approaches, enabling a “TestDevOps” construct (for Test-Development-Operations, based on DevSecOps concepts) that leverages digital engineering, permits rapid capability upgrades for software-intensive test systems, and enables effective testing of future systems that will incorporate new software- and data-driven technologies like artificial intelligence and machine learning (AI/ML). Reinvent the Range Enterprise Funding Model for Responsiveness, Effectiveness and Flexibility Given the pace of technological change in U.S. weapon systems, and the relentless and rapid advances in adversary systems across all domains of conflict, speed in testing and learning is critically important for ranges as they seek to modernize their capabilities while sustaining test assets that are often decades old, single-string, and unreliable. A revitalized approach to early definition of range capability requirements is required at both the program and enterprise levels, but that change alone will not address the resource challenges imposed on the range enterprise by today’s complex, uncertain, and inflexible approach to funding for operations, sustainment, and modernization. Today, range capabilities are funded by a variety of streams from individual programs, the military services and DoD agencies, military construction (MILCON), and central pools such as TRMC resources. Funding levels are subject to annual appropriations and vulnerable to out-prioritization by other emerging DoD needs, resulting in great unpredictability from year to year. Much of the funding is limited to specific purposes and cannot be reallocated based on greatest need, while funding often arrives late in the fiscal year and may have unrealistic requirements for timing of obligations and expenditures for test systems that require years of development and construction. Downward trends in MILCON for T&E collide with increasing demands for test time and modernized capabilities. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 15

Based on this funding landscape, some immediate steps are needed to improve the flexibility in timing of spending for range infrastructure, and also increase the ceilings for flexible use of RDT&E or O&M funding for minor military construction to enhance the valuable authorities already provided by the Congress. More sustained impact will be enabled through the pilot of additional changes to DoD’s range funding mechanisms, including establishment of a Range Working Capital Fund to stabilize funding for modernization and sustainment, and demonstration of these changes to prioritize and correct capabilities gaps needed for timely T&E of new systems with multi-domain test requirements. STRUCTURE OF THE REPORT The remaining chapters of this report are structured into three broad pieces: an examination of what the future of warfare could look like and the implications of that envisioned future for operational testing and evaluation; three chapters each devoted to the one of the three broad themes of this report (kill chains, digital technologies, and speed-to-field); and a chapter summarizing the report’s findings and recommendations. The structure is designed to enable the reader to understand the environmental forces, the state of DoD testing, and the projected future requirements that resulted in the key themes and lines of effort summarized above. The individual chapters provide detail and supporting examples on the themes, the lines of effort, and the recommendations they encompass. More specifically, Chapter 2, on the envisioned future of warfighting and OT&E, addresses several questions: What is the future of warfare, and what will be required for OT&E? How will testing need to change? What are the implications for DoD's test and training range enterprise? Finally, what does “good” look like? Chapter 3, on kill chains, multi-domain operations, and the associated future testing needs, begins with an examination of how kill chains and multi-domain operations work. Next it discusses the challenges to testing of systems in operationally relevant ways that arise in tests involving kill chains or carried out over multiple domains, or both. Finally, the chapter examines what the ranges require for carrying out a kill chain approach to OT. Chapter 4, on digital technologies—mainly focusing on modeling and simulation and on the ranges’ digital infrastructure—discusses two broad categories of challenges. The first is testing issues related to the increasing role of digital engineering and modeling and simulation in the development of weapon systems; the second is challenges related to the ranges’ digital infrastructure, such as the sharing of data among ranges and the corresponding requirements for increased connectivity, interoperability, and security. Among the questions addressed by the chapter are: What does digital engineering mean for test, and why does M&S matter for OT? What needs to change in the range enterprise to take full advantage of digital engineering and M&S? What is needed in the range enterprise for effective OT of software-centric systems? Finally, what range digital infrastructure is required for connectivity and effectiveness? Chapter 5, on speed-to-field, begins with a discussion of why speed is so critical for OT in today’s world. It examines the range enterprise factors that contribute to delays in fielding systems and kill chains and asks what needs to change in the way that OT and range requirements are established. Finally, it surveys the resource challenges that affect OT speed and efficiency and recommended remedies. Chapter 6 wraps up the main portion of the report by bringing together all of the committee’s key findings and recommendations in one place, with the recommendations by stakeholder. The appendices provide additional detail on the background, context, study approach, and sources of information that informed the committee’s work and report outcomes. Appendix A includes the study statement of task. Appendix B provides a summary of the site visits the committee conducted. Appendix C provides biographical sketches of the committee members. Appendix D lists the abbreviations and acronyms used in this report. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 16

REFERENCES Andreesen, M. 2011. “Why software is eating the world.” Wall Street Journal, August 20. https://www.wsj.com/articles/SB10001424053111903480904576512250915629460. Brose, C. 2020. The Kill Chain: Defending America in the Future of High-Tech Warfare. New York: Hachette Books. Brown, C.Q. 2020. Accelerate Change or Lose. US Air Force Chief of Staff. https://www.af.mil/Portals/1/documents/csaf/CSAF_22/CSAF_22_Strategic_Approach_Accelerate_C hange_or_Lose_31_Aug_2020.pdf. Accessed June 16, 2021. Cronk, T.M. 2020. Speed Must Be Put Back Into DoD, Hyten Says. U.S. Department of Defense – News, January 17. https://www.defense.gov/Explore/News/Article/Article/2060538/speed-must-be-put- back-into-dod-hyten-says/. Accessed June 16, 2021. Defense Innovation Board. 2019. Software is Never Done: Refactoring the Acquisition Code for Competitive Advantage. https://media.defense.gov/2019/Apr/30/2002124828/-1/- 1/0/SOFTWAREISNEVERDONE_REFACTORINGTHEACQUISITIONCODEFORCOMPETITIV EADVANTAGE_FINAL.SWAP.REPORT.PDF. Accessed July 26, 2021. DoD (Department of Defense). 2018. Summary of the 2018 National Defense Strategy of the United States of America: Sharpening the American military’s competitive edge. https://dod.defense.gov/Portals/1/Documents/pubs/2018-National-Defense-Strategy-Summary.pdf. Accessed June 3, 2021. DoD. 2020. Mission Engineering Guide. Washington, DC. https://ac.cto.mil/wp- content/uploads/2020/12/MEG-v40_20201130_shm.pdf. Accessed June 21, 2021. GAO (Government Accountability Office). 2021. F-35 Joint Strike Fighter: DoD Needs to Update Modernization Schedule and Improve Data on Software Development. Washington, DC. https://www.gao.gov/assets/gao-21-226.pdf. Accessed August 5, 2021. Mowthorpe, M. 2005. The Revolution in Military Affairs (RMA): The United States, Russian and Chinese Views. The Journal of Social, Political, and Economic Studies 30, no. 2 (Summer): 137-153. Murray, W. 1997. Thinking About Revolutions in Military Affairs. Joint Forces Quarterly. https://apps.dtic.mil/sti/pdfs/ADA354177.pdf. Accessed August 2, 2021. NASEM (National Academies of Sciences, Engineering, and Medicine). 2021. Key Challenges for Effective Testing and Evaluation Across Department of Defense Ranges: Proceedings of a Workshop—In Brief. Washington, DC: The National Academies Press. Wolfe, F. 2021. “F-35 Joint Program Office Expects to Receive Analysis on Feasibility of Joint Simulation Environment.” Defense Daily, April 22. https://www.defensedaily.com/f-35-joint- program-office-expects-receive-analysis-feasibility-joint-simulation-environment/air-force/. Accessed August 5, 2021. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION 17

Next: 2 An Envisioned Future of Operational Test and Evaluation »
Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight Get This Book
×
Buy Paperback | $40.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Rigorous operational testing (OT) of weapon systems procured by the U.S. Department of Defense (DoD) is fundamental to ensuring that these sophisticated systems not only meet their stated requirements, but also perform under realistic operational conditions when faced by determined adversaries employing their own highly capable offensive and defensive weaponry. DoD's test and training range enterprise provides the geography, infrastructure, technology, expertise, processes, and management that make safe, secure, and comprehensive OT possible. The challenges facing the nation's range infrastructure are both increasing and accelerating. Limited test capacity in physical resources and workforce, the age of test infrastructure, the capability to test advanced technologies, and encroachment impact the ability to inform system performance, integrated system performance and the overall pace of testing.

Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems assesses the physical and technical suitability of DoD test and evaluation ranges, infrastructure, and tools for determining the operational effectiveness, suitability, survivability, and lethality of military systems. This report explores modernization, sustainment, operations, and resource challenges for test and evaluation ranges, and makes recommendations to put the DoD range enterprise on a modernization trajectory to meet the needs of OT in the years ahead.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!