Executive Summary
Rigorous operational testing (OT) of weapon systems procured by the U.S. Department of Defense (DoD) is fundamental to ensuring that these sophisticated systems not only meet their stated requirements, but also perform under realistic operational conditions when faced by determined adversaries employing their own highly capable offensive and defensive weaponry. Without adequate OT, operational commanders would be unable to make the most effective use of their capability and warfighters would lack confidence in the weapons they bring to the fight or, worse, may inadvertently put themselves in harm’s way because they do not have a fundamental understanding of their weapons’ capabilities and limitations. DoD’s test and training range enterprise provides the geography, infrastructure, technology, expertise, processes, and management that make safe, secure, and comprehensive OT possible. However, the range enterprise, along with the talented and committed range workforce that makes the system function, is under great stress. Unless prompt action is taken to address both longstanding and emerging challenges, including test capacity, modernization, digital infrastructure, encroachment, and resources, DoD’s ranges will be unable to support timely or adequate OT in the future.
The challenges facing the nation’s range infrastructure are both increasing and accelerating. Limited test capacity in physical resources and workforce, the age of test infrastructure, the capability to test advanced technologies, and encroachment impact the ability to inform system performance, integrated system performance, and the overall pace of testing. Investments in the U.S. test infrastructure and changes in test
and evaluation (T&E) methodologies and handling of data are necessary to inform the delivery of lethal, survivable, reliable, and affordable weapon systems to the field at a speed that is relevant to the operational need. This study draws on testimony from senior military officers and officials from operational, acquisition, and test backgrounds as well as on test and training experts, leading technologists, leaders from relevant commercial enterprises, and individuals with deep experience in DoD and congressional budget processes. The study committee conducted virtual and physical site visits to a representative sample of test ranges; collected test range inputs on modernization, sustainment, operations, and resource challenges; and reviewed prior studies and reports from the office of the Director of Operational Test and Evaluation (DOT&E), the military service test organizations, and the Test Resource Management Center (TRMC). This report makes a set of interdependent recommendations that the committee believes will put the DoD range enterprise on a modernization trajectory to meet the needs of OT in the years ahead.
The report emphasizes the following three fundamental themes:
- Future combat will demand connected kill chains in a Joint All-Domain Operations (JADO) environment. It is critical that DoD architects, specifies, develops, and tests systems to ensure that they are highly effective when fielded in this new reality. DoD acquisition processes, organizational stovepipes, test methods, and range infrastructure that were optimized for the testing of individual weapon systems in single domains will be inadequate to test future integrated weapon systems in the way that they will be operated in machine-speed warfare that crosses all combat domains, including land, sea, air, space, and cyberspace.
- Digital technologies are dramatically reshaping the nature, practice, and infrastructure of test. The weapon systems of today and tomorrow are fundamentally enabled by data and software, and DoD test ranges will be no different. The rapidly increasing importance of autonomy, artificial intelligence (AI), and machine learning across defense systems is creating novel challenges for OT. Furthermore, the advent of digital twins and high-performance modeling and simulation (M&S) is enabling new ways of testing, even as combinations of new domains and operational constraints increasingly make virtual testing the only practical approach for certain applications.
- Speed-to-field is today’s measure of operational relevance, which is in turn a continuously moving target. Enabled by the global proliferation of many digital, software, and communications-based technologies, U.S. adversaries are rapidly and continuously
deploying new generations of weapons designed to negate U.S. warfighting advantages. At the same time, new weapon systems are employing never-fielded technologies, which are also evolving at a pace enabled by Moore’s Law. Usable weapon systems are fielded promptly, but there is a need for continuous testing and assessment.
To address the challenges tied to these themes, the committee developed conclusions and recommendations that fall into the following five broad categories:
- Develop the “range of the future” to test complete kill chains in JADO environments. It is essential that the range enterprise accommodate new concepts of operation and new test approaches for realistic operational testing, which includes enabling infrastructure for system-of-systems integrated testing and the interoperability of multiple ranges across diverse domains. [Recommendation 3-1]
- Restructure the range capability requirements process for continuous modernization and sustainment. Enabling speed-to-field while maintaining the rigor of operational test and evaluation will require rapid range modernization for new weapons technologies and new threats. At the same time, key capabilities need to be sustained and even augmented to ensure required test capacity and throughput, while mitigating the issues caused by encroachment both in the physical and radio frequency environments. [Recommendations 3-2, 3-3, 3-4, 3-5]
- Bootstrap a new range operating system for ubiquitous M&S throughout the weapon system development and test life cycle. Many of today’s DoD programs cannot be tested effectively in live testing alone. High-fidelity virtual testing can improve readiness and the likelihood of success for actual hardware testing and may be the only environment to do certain types of tests. However, widespread and standardized use of M&S for operational test will depend on a new M&S infrastructure, significant cultural changes within the test community, and new approaches to the validation of M&S in an ever-changing threat and technology environment. [Recommendation 4-1]
- Create the “TestDevOps” digital infrastructure for future operational test and seamless range enterprise interoperability. Redefine the enterprise-supported core digital standards and capabilities for TRMC and test ranges to take advantage of DoD’s scale for software, data, networks, AI, cybersecurity, and M&S.
- Reinvent the range enterprise funding model for responsiveness, effectiveness, and flexibility. The resource needs of today and tomorrow reflect the reality of rapidly changing technology and threats; sustained capital investment for creation, upgrade, and maintenance of long-life range systems; and increasing demands for cross-domain system-of-systems testing with seamlessly integrated M&S. Including DOT&E earlier and continuously in the requirements development and acquisition processes will better establish and certify the timeliness and adequacy of range investments. [Recommendations 5-1, 5-2; Conclusions 5-1, 5-2]
Make model-based engineering, the unbroken digital thread, and continuous integration/continuous delivery software practices the foundation for range agility, rapid test evolution, and speed-to-field. Ultra-high-bandwidth information flows must become frictionless, on-demand, and secure. [Recommendations 4-2, 4-3]
Table ES.1 maps the report recommendations and key conclusions to the themes laid out above.
TABLE ES.1 Report Recommendations and Key Conclusions Mapped to Themes
Theme | Recommendations and Key Conclusions |
---|---|
Develop the “range of the future” to test complete kill chains in JADO environments | Conclusion 3-1: The lack of a Department of Defense or joint publication set of definitions for multi-domain operations and cyber-physical systems can result in different operational use cases. Conclusion 3-2: Testing ranges are not optimized for testing end-to-end kill chains; they were not designed for collaborations with other ranges, and they lack the framework and infrastructure to test concurrent and connected kill chains. Recommendation 3-1: To enable a range of the future that is capable of testing kill chains and multi-domain operations that can integrate effects across National Defense Strategy modernization areas, the Secretary of Defense should address the need to enable Department of Defense ranges to provide regular venues to “test as we fight” for acquisition and prototyping programs in a joint multi-domain battlespace of integrated systems. |
Theme | Recommendations and Key Conclusions |
---|---|
Restructure the range capability requirements process for continuous modernization and sustainment | Conclusion 3-3: Encroachment leads to the inability to demonstrate mission capability and identify deficiencies due to lack of access to the physical and electromagnetic spectrum space with which to conduct test and evaluation. This creates operational risk as DoD will have to field weapon systems that have not been tested against certain threats. Recommendation 3-2: To ensure the ability to validate the survivability of Department of Defense (DoD) weapon systems against a realistic operational threat environment across air, sea, land, space, and cyberspace domains, DoD should identify and prioritize bands that cover U.S. military operational and test requirements which should be protected from sell-off to preserve these capabilities. Recommendation 3-3: The Test Resource Management Center (TRMC) should assess current and projected commercial radio frequency communications technologies and spectrum allocations for secure, agile, high-bandwidth operational test needs. In addition, TRMC should determine the feasibility of developing new large-scale enclosed testing facilities combined with expanded modeling and simulation to support electromagnetic spectrum activities not suitable for open-air testing. Recommendation 3-4: The Department of Defense should broaden the authority of the Test Resource Management Center to address issues of internal encroachment by reviewing internal range policies and actions to ensure that the test groups retain adequate mission space and prevent the placement of equipment or infrastructure that could potentially interfere with test operations. The Director of Defense Research and Engineering for Advanced Capabilities should be granted the authority to mitigate disputes arising over internal encroachment concerns and provided additional funding to manage internal encroachment. Recommendation 3-5: The Test Resource Management Center should develop a strategy that assesses the use of and potential investment in suitable allied resources for open-air testing. This strategy should include criteria for the usage of allied resources and areas of potential investment to include range space available, data collection, security risks, and support facilities. |
Theme | Recommendations and Key Conclusions |
---|---|
Bootstrap a new range operating system for ubiquitous M&S throughout the weapon system development and test life cycle | Recommendation 4-1: A Department of Defense joint program office should establish a shared, accessible, and secure modeling and simulation (M&S) and data ecosystem to drive development and testing across the life cycles of multiple supporting programs. M&S should be planned from early concept development to support the entire life cycle of the system, from requirements generation, through design development, integration and test, and sustainment. Uncertainty quantification should be employed to identify the primary sources of uncertainty in the understanding of the system being developed and to define an integrated testing and simulation activity to reduce those uncertainties to an acceptable level. |
Create the “TestDevOps” digital infrastructure for future operational test and seamless range enterprise interoperability | Recommendation 4-2: A Department of Defense joint program office should adopt and promulgate modern approaches for standardization, architectural design, and security efforts to address data interoperability, sharing, and transmission challenges posed by the complexity of next-generation systems. The joint mission office should determine how to develop and maintain a protected data analysis tool and model repository for testing, increase the interconnectivity of test ranges, and ensure the development of data protocols for the real-time transfer of data at multiple classification levels. Recommendation 4-3: The Test Resource Management Center should continue monitoring and supporting the Assured Development and Operation of Autonomous Systems (ADAS) Project, and prioritize efforts to develop a common set of standards, measurement approaches, and operational scenarios from which to evaluate the performance of artificial intelligence (AI) and autonomous systems, while recognizing that testing approaches may differ between AI and autonomous systems. |
Theme | Recommendations and Key Conclusions |
---|---|
Reinvent the range enterprise funding model for responsiveness, effectiveness, and flexibility | Recommendation 5-1: The Joint Requirements Oversight Council (JROC) should consult regularly with the Director of Operational Test and Evaluation (who is an advisor to the JROC) about the test requirements for systems considered by the JROC. This consultation should include an evaluation of current testing capabilities, facilities shortfalls, and plans to address these shortfalls. Recommendation 5-2: The Office of the Secretary of Defense should either allow an exemption or set shallower expenditure benchmarks for the first 2 years of test modernization programs. This will reflect realistic expense curves for the technologies and projects needed to test next-generation programs and complex integration. Conclusion 5-1: New mechanisms and funding limits for applying minor military construction are necessary for responsive test and evaluation activities. Conclusion 5-2: There exists a need for the Department of Defense to pilot new process and authorities for funding ranges and infrastructure to make them simpler, more responsive, and more effective. |