Demonstration and Validation
The Totally Integrated Munitions Enterprise (TIME) program is attempting to develop and beginning to implement a highly complex, integrated system that will become one of the cornerstones of U.S. national defense. Although the Army would not necessarily be precluded from reverting back to today’s “manual” methods of producing munitions if the system failed, significant benefits of the integrated enterprise, especially faster response at lower cost in the event of national need, would likely result in increasing national reliance on its capabilities. Therefore, the integrated enterprise must be robust and sufficiently validated under a variety of conditions, so that there is a high probability that the system will perform properly when needed. Assurance that the enterprise will work under wartime conditions, which could result in damage to U.S. infrastructure or system impairment due to cyber warfare, makes validation more difficult.
Thus, the TIME program must (1) strive to create a loosely coupled integrated system with sufficient parallelism to make it robust, and (2) extensively validate the system over time, as the constituents of the system and as the scenarios under which it might be used evolve. Thus, demonstration and validation of integrated munitions enterprise concepts will play a critical role in the TIME program.
In assessing efforts by the TIME program to demonstrate and validate key technologies, the committee turned first to definitions. The committee defined “validation” as the means of confirming that an approach is well grounded, justifiable, and correctly derived from basic premises. “Demonstrations,” on the other hand, exercise some segment of the system under a selected set of conditions, thereby serving to illustrate or provide conclusive evidence that an approach can be made to work. Demonstrations do not generally attempt to build from first principles a series of robust, logical arguments that mathematically prove that an approach will work under a wide variety of conditions or, ideally, under all foreseeable conditions.
To date, TIME has focused on demonstrations, which are reviewed and assessed in the next section. Little attention has yet been paid to validation, although the TIME literature uses this term. Specific suggestions on validation are offered in the last section of this chapter.
In response to its funding history, which has consisted primarily of a series of “one-time” congressional mandates without assurance of future funds, the TIME program has correctly been highly sensitive to the need to demonstrate short-term successes. TIME program managers were forthcoming in pointing out that if a steady funding stream were assured, they would organize the program in a more conventional manner, with less emphasis on short-term successes that might not lie on the critical path. For instance, the TIME program presented no methodology, other than the availability of technologies, for its selection of demonstration projects. This approach is well below commercial industry standards that typically rank candidate projects based on criteria such as (1) return on investment, and (2) criticality of technologies to the overall program. Without the use of such selection criteria, there is substantial risk that work on these projects may be of low value to the overall TIME program, resulting in program delays and lower overall returns on investment.
Nonetheless, with the exception of the Open Modular Architecture Controller (OMAC) project, selection and execution of demonstration projects appears to be one of the strongest aspects of the TIME program. For the most part, TIME has chosen projects that could be quickly brought to fruition or that incorporated pieces of technology that had a significant head start before the TIME program began. These projects have provided an excellent means for soliciting external reactions to the program while building stakeholder enthusiasm and support.
The TIME program is using real-world proof-of-principle projects, integral and concurrent with the development of major facets of the program, to demonstrate key capabilities. These projects enable testing, feedback, and improvement by the developer through testbed applications and they aid both the developer and the (U.S. Army) Tank-automotive and Armaments Command/ Armament Research, Development, and Engineering Center (TACOM-ARDEC) in determining the degree of success and correctness of direction of selected TIME elements.
Demonstration activities typically include efforts to accomplish the following (Burleson 1999b):
Procure testbed integration components.
Configure and maintain testbeds.
Support integration, validation, and benchmarking.
Incrementally include product realization tools as they become available.
Incrementally include OMAC capabilities on the shop floor as they become available.
Provide feedback to design and implementation activities.
These demonstration projects also serve as a means to begin to implement some of the key technologies into everyday practice with the intent that they will stay in place and remain operable as part of the gradual process of upgrading the munitions manufacturing base. The TIME program, depending on funding, has planned six major demonstrations of its integrated architectures and technologies outside of the laboratory (Burleson 1999b):
Miniaturize global positioning systems,
Scranton/General Motors Powertrain (GMPT) product data exchange,
Explosively formed penetrator, and
Each of these demonstration projects has scheduled milestones and specific technology objectives. Many of these projects consist of a 1-year primary thrust plus scheduled follow-on tasks that extend beyond the initial demonstrations. In some cases, the follow-on tasks are designed to upgrade system capabilities as they become commercially available.
The primary purpose of these demonstration projects is to accomplish the following:
Identify problems and provide valuable feedback to design and implementation teams within the TIME program.
Help to ensure deployment of only robust instantiations of the TIME architecture.
Focus the diverse efforts of TIME program participants on specific, measurable goals and schedules.
Demonstrate progress and the potential value of the TIME program to diverse constituencies, including the sources for ongoing funding.
Begin the deployment of TIME technologies and capabilities to production programs.
The concept validation project was organized into four initial segments:
OMAC version 1.0, mill and piece parts. Scheduled for program years 1 and 2;
Electronics. Scheduled for program years 2 and 3;
OMAC version 2.0, mill and piece parts. Scheduled for program years 2 and 3; and
OMAC version 2, lathe and assemblies. Scheduled for program years 3 and 4.
The purposes of the concept validation project are to test the TIME architecture incrementally in a controlled environment and also to test the TIME toolset, including the product realization technologies, a network testbed, and the OMAC, as the tools became available. Thus, it is intended that the project demonstrate a collaborative design and manufacturing environment, a fully networked infrastructure with application servers, and structured document archiving.
This project will extend the TIME product realization toolset into electronics manufacturing. The 5-year plan will begin with the application of TIME tools for requirements definition and culminate in a production facility that can produce low-cost modules for a wide variety of munitions applications (ManTech 1999). The objectives of the miniaturized GPS project are as follows (Burleson 1999b): (1) Use a previous-generation GPS subsystem as a vehicle for electronics miniaturization (redesign the module for use in munitions). (2) Drive modularization of design for multiuse circuits in families of applications. (3) Demonstrate flexible manufacturing for small lot electronics manufacturing. (4) Demonstrate the extension of the TIME toolset into electronics manufacturing.
This demonstration project was initially planned for early in the TIME program but is now in the planning phase.
Scranton/GMPT Product Data Exchange
This is one of an ongoing series of demonstration projects being conducted at the Scranton Army Ammunition Plant that are designed to gradually integrate the facility into the munitions enterprise. The objectives of this project were to demonstrate the capability to (1) integrate an initial collaborative toolset into the Scranton TIME network; (2) electronically transfer product and process design data among ARDEC, Scranton, the Louisiana Center for Manufacturing Sciences, and a potential commercial replenishment manufacturer, GMPT; and (3) manufacture discrete parts (a mortar adapter) using those data at both Scranton and GMPT (Burleson 1999b).
The project has been essentially completed, successfully demonstrating the following, in sequence (Stephens 2000):
Remote downloading of design files from the TIME file server via the internet,
Translation and processing of the computer-aided design (CAD) files,
Performance and verification of tool simulation,
Modification of machine instruction files,
Subsequent uploading of computer-aided manufacturing (CAM) files to the TIME file server,
Downloading of CAM files to the shop floor,
Programming of the commercial-off-the-shelf (COTS) machine controller, and
Manufacture of parts using all of the above data.
The M42 grenade project focuses on sharing of a shop traveler via a virtual enterprise. Its primary objectives are the following (Cary 2000) (1) Demonstration of the TIME surge manufacturing concept of using dual-use suppliers to broaden the munitions manufacturing base, (2) Establishment of a secure Web-based virtual environment at Primex (a routine munitions supplier) and GMPT (a potential dual-use supplier), (3) Demonstration of a Web Integration Manager and associated cockpits; (4) Electronic capture of Primex’s manufacturing process for the M42 grenade, (5) Demonstration of the basic TIME collaborative environment and capability to transfer both CAD and CAM information, (6) Demonstration of an OMAC front end on an existing computer-numerical-control lathe at Primex, and (7) Building on these capabilities with these key participants in the TIME replenishment effort.
The committee noted that this demonstration project served to identify an issue that TIME thus far does not appear to have addressed, that of design and fabrication of custom metalworking tools and dies. Most of TIME’s metalworking efforts to date have dealt with relatively straightforward metal removal and shaping using readily available COTS tools for cutting, knurling, grinding, facing, boring, and drilling on mills and lathes. Preparations for fabrication of metal parts at remote, dual-use sites becomes more complex when, as in the example of the M42 grenade, custom dies are required for cupping and coining. Fortunately, such dies for fabrication of munitions are typically relatively straightforward, but they can nonetheless require both redesign by the dual-use supplier to match its specific equipment interfaces and fabrication by increasingly scarce custom tool and die makers. Identification of unforeseen problems, such as the issues associated with tool and die design and fabrication, points out the importance that the TIME program has correctly been placing on demonstration projects. It also identifies a need for the TIME program to devote ongoing attention to parts of its dual-use supply chains that are typically seldom used or nonrecurring but nonetheless on the critical path for production.
Recommendation: As replenishment suppliers are added to the enterprise, the TIME program should focus attention not just on integrating recurrently used portions of dual-use supply chains, but also on nonrecurring and seldom-used suppliers, such as tool and die fabricators that can play a critical path role in the rapid ramp-up of replenishment capabilities.
Explosively Formed Penetrator1
This project is focused on the issues associated with extending the virtual enterprise to the shop floor. It is scheduled for completion in December 2000 and is designed to test a TIME toolset in a production environment. Aerojet will demonstrate the flexibility of the OMAC, version 1, for advanced munitions components (ManTech 1999). Plans are to (1) install the TIME collaborative environment at Aerojet, (2) incrementally introduce product realization tools as they become available, (3) establish and train an integrated product team, and (4) incrementally include OMAC capabilities on the milling machine as they become available (Burleson 1999b).
This project will also serve as a means to evaluate use of the TIME collaborative environment in the fabrication of a sample product. The team has succeeded in electronically forwarding CAD data for a novel explosively formed penetrator design from ARDEC in New Jersey to Aerojet in California. Aerojet, in turn, forwarded the CAM data to Lawrence Livermore National Laboratory (LLNL) where the penetrator was machined on a mill controlled by an open architecture controller (OAC) developed by LLNL.
The TIME program is also carrying out the Picatinny/Thiokol/Stevens twin-screw extruder project. The nation’s current energetics production base is decades old. The facilities were designed for large production quantities and are inflexible to varying formulations of propellant, explosive, and pyrotechnics. Recent procurements of energetics have decreased to such a low level that production facilities are operating at a small fraction of total capacity. This is causing the cost of producing energetic materials to reach unaffordable levels, necessitating significant changes within the industrial base to maintain a viable energetics production capability in the United States. In response to the current business environment, efforts to develop new energetics materials, and a trend toward “designer munitions,” the industrial base must be modified to cost-effectively produce a wider variety of new and existing products in smaller quantities. To do this efficiently, the Department of Defense (DoD) will have to partner with private industry and academia and leverage a substantial commercial infrastructure to manufacture the required energetic materials.
A process methodology has been developed and demonstrated under the TIME program that quickly transitioned technology from small-scale R&D quantities to large-scale production. A virtual enterprise network was installed that provides a link between industry, government, and academia to transfer real-time data between sites. This, according to the TIME program, will reduce product development time from 5 years down to approximately 2.5 years. It will
also significantly improve processing safety and reduce hazardous waste streams.
This project addressed the issues in two ways. First, a new CL-20-based explosive formulation was developed using the TIME methodology. Second, the TIME network was utilized with the current production base to improve the propellant manufacturing process. CL-20 is a state-of-the-art, high-energy explosive compound that is extremely sensitive to dry handling. The focus of this project was to demonstrate the safe reproducibility of CL-20 when scaling up to production quantities.
The project successfully demonstrated the use of modeling and simulation tools for product and process development, including modeling the crystallization process and determining the critical relationship between physical and chemical characteristics of the material on a microscopic scale and correlating these to bulk characteristics on in-process and end-product materials. The TIME network will be used to link the model to both the bulk laboratory experiments and the production of CL-20. The TIME program has already successfully demonstrated the continuous production of small batch lots of energetics at remote sites using real-time monitoring and control from a central location.
The Army’s Modular Artillery Charge System is currently using M30A1 triple-base propellant in the XM232 charge and has decided to utilize PAP-7993 propellant in the XM231 charge. Large quantities of these propellants are currently being produced using a batch method, and there tends to be variability in production lots. A major contributor to variability is the design of the die used to form the final propellant shape. By applying the methodology developed under the TIME program, a better understanding of the process can be achieved and improvements to the existing process can be implemented. Once the process is understood, new technology can be developed to enhance the process and eventually reduce the variability and cost of the material.
By applying this technology, warfighters will be provided with advanced energetic materials for their weapons systems with significantly enhanced effectiveness and survivability, and the energetics industry will have the ability to bring new energetic materials to the warfighter faster than ever before. Further details regarding the twin-screw extruder project are presented in Chapter 5 and Appendix C.
The TIME program intends to modify the twin-screw extruder TIME network to incorporate the OAC module developed for program logic controller interface. This effort is designed to demonstrate the ability to control the extrusion and mixing of energetic formulations through sensor feedback to a mathematical model and automatic adjustment by the control system of key parameters. This demonstration is fundamental to proof of the base concept and essential prior to implementation of the model-based control approach on production lines.
Another TIME project is taking this concept of model-based control and applying it to a melt pour production line at the Iowa Army Ammunition Plant. The operation’s control needs are described in depth in Appendix C. Following completion of this design and the completion of the twin-screw extruder project, a
project to execute the melt pour design as a prototype project funded through Production Base Support funding is anticipated.
Summary of the Demonstration Projects
These projects demonstrate several virtual enterprise concepts, including the ability to leverage commercial facilities for surge capacities. With the completion of these projects, the concept of transferring metal removal operation processes from existing metal part producers to a commercial site and the concept of using commercial nondefense producers for expansion of capability for replenishment purposes will have been demonstrated.
Conclusion: The TIME program is to be commended for its focus on demonstration projects as a means to try out complex integrated enterprise systems in a real-life environment and as a means to identify potential problems prior to full implementation and use.
Recommendation: When a long-term strategic plan for TIME has been developed, after the Army has assumed ownership of TIME, and as the TIME program moves toward the implementation phase, increased program priority should be given to demonstrations and system validation, so as to verify that the concepts can be made to work and to reduce the chances for unforeseen system problems.
Recommendation: The TIME program should, on a regular basis, review its goals and objectives, as well as its technology path for achieving these objectives, so as to avail itself of the latest, appropriate, well-proven COTS technologies from commercial industry.
To date, validation of TIME technologies has centered on demonstration projects. This is a good first step. It is an effective means to assure that the tools and technologies can be made to work in the munitions industry environment and to demonstrate that the objectives of TIME are being met in a usable fashion. However, there can be a natural tendency to demonstrate and forget. Each demonstration must be viewed as a link in a chain so that, as each element in the “demonstration matrix” is completed, the overall system is formed in place. It is important to identify and structure this process through to its ultimate form. It is true that the process may change significantly over time, but the process of demonstrating subsystems before they are integrated into the operational structure will tend to help identify issues that need to be addressed, especially the interactions between subsystems. It should also be noted, however, that demonstrations are merely “existence proofs” that show that the process will work under a narrowly defined set of conditions. The committee believes that far
more validation is needed before substantial resources are committed to broad implementation.
The committee recognizes that, in the worlds of manufacturing, business, and e-commerce, new tools, techniques, and architectures are seldom validated back to first principles for all foreseeable applications and conditions. Such validation could, in many cases, consume more time and resources than were required to initially develop and demonstrate the technologies. However, the committee believes that it is important for the TIME program and the Army to identify potential limitations and vulnerabilities that may be inadvertently built into the integrated munitions enterprise upon which the United States will base a significant portion of its national security. Without extensive validation, defects in this highly complex system may not become evident until the system is called upon to meet an urgent national need.
Validation of Product Designs and Manufacturing Processes
There are several levels of validation that the TIME program must consider. It is important that the disparate, individual systems that make up the integrated enterprise be validated to assure that they properly perform their defined tasks. Is the software error free and the hardware defect free? It is also important that the interfaces and interoperability of these systems be validated to rigorously work together within the enterprise under a variety of conditions.
More fundamentally, however, the TIME program shares responsibility with other DoD programs for electronically documenting munitions designs and manufacturing processes and ensuring that product manufactured at multiple locations, using a mixture of modern, antiquated, mothballed, and dual-use equipment along with materials from a variety of suppliers, will all consistently yield products that fully meet DoD specifications. Part of the problem is pedigree. If, for instance, processes are developed for fabricating a part on an old piece of equipment and the munitions enterprise wishes to produce the identical part on new or different machines, possibly with new or different controllers, is the new part really the same as the old? How can this be validated for safety, reliability, and performance without the building and field testing of large quantities of prototypes? Machining knowledge and processes are a complex, interactive sum of multiple factors such as geometry, “features,” materials, control characteristics, physics, rigidity, and tools. Thus, to rigorously automate a process or to transmit sufficient information to a new machining site can be a daunting challenge.
Munitions designs at government-owned/government-operated facilities, as described by participants in the TIME program, have traditionally been documented on paper or Mylar. Efforts are under way by the Army to scan these documents into an electronic format. Likewise, according to TIME participants, parts of some of the manufacturing production processes are documented on paper or Mylar. Others are relatively undocumented (McWilliams 2000a). Munitions experts are retiring or are being laid off as funding declines. Capturing production methods is almost always a difficult and complex task. Thus, TIME faces a serious challenge to (1) electronically capture all of the information that is required to manufacture the product, (2) validate on the original manufacturing equipment (if still available) that all of the necessary data have been correctly captured, and then, (3) using these data, validate that product produced on different machinery, by different operators, and using materials from different suppliers will meet all specifications. Although many of the metal parts currently used in the munitions industry are relatively straightforward to produce, slight variations in energetics processing can produce catastrophic results (McWilliams 2000a). Newer generations of increasingly smart munitions tend to be more complex. Hence, there is an increasing need for extensive validation. As equipment is modified or replaced, which can happen with relative frequency in dual-use operations, the validation process must appropriately keep up with the changes.
Product validation requires extensive attention to details, documentation, and the implementation of in-process and finished product testing. The munitions industry should pay special attention to assure that the workforce is properly skilled in the quality assurance function.
Appropriate validation of a massive, highly complex system, such as the munitions enterprise, requires attention to myriad details, concepts, and challenges. For example, capture and effective dissemination of knowledge is essential, especially for rapid replenishment. “Knowledge” can be thought of as information organized in context. Although tools are being commercially developed to capture knowledge, to date they are preliminary at best. “Intent” can play important roles in the understanding of transmitted information, yet to date intent cannot be effectively captured and transmitted. These technology limitations add risk to the TIME approach, which relies heavily on the timely transmission of product and process information from one site to another. Extensive, ongoing validation will be required to reduce these risks and to increase confidence in this approach.2 Validation will also be required to assure that all necessary process knowledge, some of which may be considered proprietary by routine producers of munitions, is transferred to replenishment dual-use suppliers in a timely manner. Exercising the process on a regular basis and evaluating the output can help to validate the completeness of the data packages in advance of need.
Recommendation: Substantial resources, including funding, must be made available to the TIME program to validate the integrated munitions enterprise so that it can be trusted to perform appropriately when needed.
The model that OAC developers are building on, using application program interfaces (APIs) and separate software modules, can never be fully tested. MS Windows and Windows NT use the same model and, despite extensive user experience and testing done by Microsoft, desktop computers using these systems crash from time to time. Adding new software and hardware, as OMAC developers are doing, only adds to the risk, no matter how well they validate. Everyday experience with current desktop systems demonstrates that this is the case.
The difference is that system crashes can be a significant cost and safety problem in the controller environment. They happen with today’s OACs. Although the argument is made by proponents of the OAC that crash problems are caused by users, the systems tend to be made more complicated than necessary by adding additional software that is not needed for the controller functionality. If well-tested, commercially available software that is not even using the real-time kernel cannot be added to OAC systems without risking system failures, then the committee questions the rationale of an OAC.
In the Microsoft model, the key intellectual property of the company, the core functionality of Windows or NT is tightly controlled. This is one of the primary differences between MS Windows and Unix. There is only one version of MS Windows and many more applications are written for Windows than Unix because of this. Source code for Windows has never been released and because of that, it is said by some that MS is not sufficiently open, thereby restricting those who would write applications for it. (This is part of the current litigation between the federal government and Microsoft). This is the same argument made by proponents of the LLNL version of OAC for continuing their work. Since the LLNL OMAC version will be “open” as in the Unix model, to which version of the core controller functionality will controller add-on applications providers be writing code? The answer can perhaps be found in the operating system “wars.” They will write applications for the core functionality that holds the biggest market share. Hence, they will tend to write applications for the current market leaders, Fanuc and Siemens, unless a huge user group forces the issue.
Following the “open controllers are like MS Windows” line of argument put forth by the OMAC developers, who is the Microsoft for this application? Who will be responsible for testing the APIs? Who will assure that the system is robust? Is it the national laboratories, the as yet unidentified companies who may someday want to commercialize the OMAC, or might it, by default, end up being the TIME program? Despite the possibility of extensive testing in government laboratories, it appears unlikely to the committee that sufficient funds will be
provided for the proposed OMACs to be sufficiently validated to pose minimal risk to the user.
Assessment of Validation Plan
To date, the TIME program has done little to outline plans for—or implement—a validation plan. The plan should start with the selection, wherever possible, of commercial industry-proven tools and concepts for the integrated enterprise. Ideally, these tools should be proven, back to first principles, to operate in all foreseeable use environments. All new tools and concepts developed by TIME should likewise be thoroughly proven out before being inserted into a production environment. Then all of these parts of the enterprise system should be proven to work together under all foreseeable scenarios. This validation process should include the introduction of system faults to determine how well the system responds to faults and whether there are single points of failure. This thorough validation process, which is essential to have a high degree of assurance that the system will work, especially in times of crisis, can be extremely expensive. The cost of validation could be equal to or exceed the development costs of the tools and components.
Although a rigorous validation of such systems is seldom undertaken due to time and cost, it is a topic that should not be ignored. Validation efforts should be commensurate (proportional) in magnitude to the criticality of system performance, especially in times of crisis. Should the validation of munitions manufacturing systems be as extensive as those undertaken for nuclear power plants, missile launches, or nuclear weapons performance and safety? Probably not. However, the importance of munitions manufacturing and the magnitude of potential investments in implementation make it abundantly clear that an extensive validation effort, well beyond the several demonstration projects envisioned to date by the TIME program, is justified. Recent highly publicized problems in the implementation of enterprise resource planning (ERP) systems at major U.S. corporations come to mind as examples. Such problems can be costly for commercial industry but have the potential to create a national disaster if not identified and resolved in advance of use in a munitions replenishment effort.
Dr. George Hazelrigg of the National Science Foundation, for example, in recent papers (Hazelrigg 1999a and 1999b) has identified flaws in frequently used engineering design and manufacturing techniques, such as quality function deployment, the Taguchi Loss Function, and the Pugh Selection Matrix, that can result in conflicting or misleading answers. Similar questions have been raised regarding the models and simulations used in engineering design and analysis as well as frequently used ERP, supply chain management, and other computer systems that may well form part of the backbone of the integrated munitions enterprise.
Conclusion: The integrated munitions enterprise will be especially vulnerable to unanticipated failures because it will consist of a large number of disparate, evolving systems, segments of which are primarily utilized by dual-use suppliers and may rarely be exercised in conjunction with other parts of the integrated munitions enterprise.
Recommendation: It is extremely important that the integrated munitions enterprise be well validated, before and during implementation, both as components and as a system. It is also extremely important that the system be regularly exercised to identify and resolve problems as participants are added and removed from the enterprise and as individual computer systems change or migrate to newer versions.
Recommendation: The validation projects of the TIME program should be subject to review by DoD managers at an appropriate level to assess the projects’ contributions to the management of munitions across all military services so that appropriate program changes can be made to assure that the services needs are met.