Potential Problems Relating to Obsolescence in Chemical Demilitarization Processing Operations
This chapter examines potential vulnerabilities to successful completion of the U.S. chemical stockpile disposal program that could become issues due to obsolescence. This chapter is focused on facilities, equipment, and the personnel required to operate, maintain, and eventually close the chemical agent stockpile incineration facilities at each operational incineration site.
As a practical matter, all facilities and equipment become obsolete at some point in time. Even so, “obsolescence” does not necessarily mean “nonfunctional,” as long as safety, environmental performance, operability, and overall schedule are not adversely impacted. The facilities and much of the equipment and supporting systems covered in this report are unique to the chemical stockpile disposal program, which is an enterprise of finite duration. As such, they are not likely to be superseded by new technology over the course of the mission for which they were designed and cannot be considered to be or become obsolete on that basis.1 Operational challenges are expected to increase because of continuing degradation of stockpile components.
While the obsolescence issues regarding facilities and equipment constitute a major focus of this report, also of critical importance is the avoidance of having the quality and capability of plant personnel degrade over time. The latter could increase the potential for a serious accident or a permit violation that may stop operations for a prolonged period and create additional challenges for the continuing operability of the disposal facilities. Since delays prolong exposure to stockpile risks, all these factors are related to overall program obsolescence.
Consequently, the key question considered by the committee was not necessarily one concerned strictly with obsolescence, but one that more broadly assessed continuing functionality and reliability through the systematic anticipation of problems and the implementation of sound obsolescence management programs. These considerations are central themes in this and subsequent chapters.
INCINERATION FACILITIES FOR CHEMICAL STOCKPILE DISPOSAL
As the chemical weapons stockpile disposal program was evolving in the 1980s and 1990s, the Army initiated plans to build chemical agent disposal facilities at each of the stockpile storage sites since Congress had proscribed the transportation of stockpiled chemical warfare items. When the decision was made to develop the baseline incineration destruction system, incineration facilities were built at five of the nine locations where large quantities of varied stockpile items were located. Although these facilities had a common design, parallel processing lines tailored to the mission of destroying the specific stockpile items at each location were added.2
Even though agent destruction was initially intended to take place over a relatively short time frame, the robust structural requirements by which chemical agent stockpile incineration facilities were built should allow their potential operating life in all probability to exceed even the most pessimistic current schedule for completion of chemical weapons destruction.
While certain components and systems of the physical facilities may become damaged or deteriorate and require repair or replacement, it is unlikely that these situations would be sufficiently frequent or serious enough to adversely impact process safety, the environment, or the overall destruction schedule except for the highly remote occurrence of rare catastrophic events of either natural or process origin.
The site contractors have done a very capable job of scheduling repairs, maintenance, and replacements to minimize schedule disruptions, having been encouraged by the Army’s contractual incentives to maintain schedule as long as safety and environmental performance remain at a high level. Major maintenance activities are planned during shutdowns between the demilitarization campaigns. Obsolescence management programs are now under development to identify possible future vulnerabilities from unplanned failures due to the aging of equipment and operational systems and to take proactive preventive actions.
The four currently operating incineration plants that are the focus of this study are expected to be in operation for periods of another five or more years to complete stockpile processing and for an additional two to three years to accomplish closure.3 The processing periods depend on the site-specific stockpiles and the continuing operability performance of the facilities. Chemical agent disposal facilities are by definition unique and are probably not readily adaptable to other uses, including chemical or conventional waste processing. Moreover, industrial waste treatment facilities would be able to process wastes more efficiently and economically than modified chemical demilitarization furnaces, and there is excess capacity in the hazardous waste treatment business at this time. Further complicating the possibility of adaptive reuse of disposal facilities is the long-term risk and liability associated with possible latent agent contamination in standing structures, especially in the facilities that processed nerve agents.
The closure periods for these facilities are dependent on the amount of accumulated secondary hazardous waste that requires disposal and the end state of clean-up established for each site. The latter is a matter for consultation and negotiation between the Army and the governors of each affected site. In recognition of the environmental contamination in these plants, the CMA currently plans to decontaminate and demolish the agent processing facilities to bare ground suitable for industrial reuse. Uncontaminated associated structures might be adapted for reuse. Transfer of ownership from the CMA would also require satisfactory closure of any permits and likely a willingness for the new owner to assume any future liability from the CMA.
The planning and execution of maintenance, repair, and replacement activities should anticipate the termination of destruction operations and facility closure. During the terminal phase of destruction operations, these activities should be strictly limited to those that are essential to ensure continuing safe and efficient operations and regulatory compliance. During visits to the facilities, the committee found them to be generally well maintained and of good appearance.
In the wake of the September 11, 2001, events, the chemical industry has undertaken steps to further enhance plant security systems by conducting periodic and comprehensive operational security analyses. The committee found good physical security programs at the sites and a high level of safety awareness, although more scrutiny has historically been given to the processing facilities than to the associated storage yards. While the CMA has conducted risk assessments for each facility in the past, these do not include acts of terrorism, nor do they identify external factors that might disrupt the continuing operability of the plants (e.g., those associated with site security or the integrity of utilities, communications, and other emergency systems). The effects of future changes in surrounding communities and the regulatory environment are not systematically tracked.
Finding 1. The four operating incineration facilities, although aging, are capable of safe and effective destruction of remaining stockpile inventories and completion of closure activities at each site, if strong and comprehensive obsolescence management programs are implemented. However, each chemical agent disposal facility is susceptible to location-specific external factors that may change over time. A need exists for these factors to be determined and evaluated at each site on a periodic basis.
Recommendation 1. An operational security analysis program should be established for each chemical agent disposal facility to evaluate external factors (including the integrity of critical utilities) that could adversely impact continuing operability. (Tier 3)
INCINERATION SYSTEM PROCESS EQUIPMENT
A detailed process diagram for a typical chemical agent incineration facility is presented in Figure 2-1. Some differences in configuration exist between the sites due to differences in the stockpiles to be processed. For example, the unpack area is reconfigured at PBCDF. There also are some differences in the number of parallel processing lines provided at individual plants. In the generic design, munitions and containment systems are moved from stockpile storage areas to the munitions demilitarization building (MDB) in sturdy on-site containers, which are received at a container handling building (CHB) and lifted to the second floor unpack area (UPA), where the munitions and bulk items are unpacked and put on input conveyors to the MDB. Four separate systems are employed to transfer munitions from the UPA to the explosion containment room (ECR) and the munitions processing building (MPB). These are the rocket handling system (RHS), the bulk handling system (BHS), the
projectile handling system (PHS), and the mine handling system (MHS). There are two ECRs within the MDB where energetics are removed by projectile/mortar disassembly (PMD) machines. After removal the energetics drop through feed chutes to the deactivation furnace system (DFS).
Multipurpose demilitarization machines (MDMs) remove the burster well, drain the agent, and then reinsert the burster well. The agent flows into agent holding tanks and is burned in a liquid agent incinerator (LIC). Drained munitions, which in some cases may contain a solidified heel of agent, are processed in a three-zone metal parts furnace (MPF).
Bulk ton containers are simply punched and drained, rinsed, and passed through the MPF for final decontamination.
Exhaust gases from each furnace and afterburner system go to a high-performance pollution abatement system (PAS) consisting of wet scrubbers and filters.
Subsequent sections examine the potential vulnerabilities to obsolescence for the major types of equipment.
Spare and Replacement Parts Availability
Essential to the continuing operability of the processing equipment is the requirement to maintain adequate stocks of spare parts to allow maintenance as equipment performance deteriorates from age or other causes. This section provides a description of the spare or replacement parts program on a facility and programmatic basis. The committee examined how the systems for managing the spares inventory function currently and how these systems might need to be improved for continuing operations.
Control System Spares
At present, control system spares are supported by an external contractor; other spare parts are managed on a site-by-site basis, with provision for sites to get spares from other sites in the event of a local shortage in inventory.
At the time this report was being prepared, the control system included some equipment that could be considered obsolete, but was still functional. To ensure continuing support for control system spares, a control system spares management contract has been negotiated with Rockwell Automation, the present owner of the Allen Bradley operations that provided the original control system components.
The committee did a careful assessment of the maintenance and spares support program (i.e., the Rockwell Automation Asset Management Program) for the older control system components: programmable logic controllers-3s (PLC-3s), PLC-5s, Data Highway, and AB 1700 series input/output (I/O) modules. In the committee’s view this program is well designed and will satisfy the required support throughout the operational and mission life of all the incineration facilities.
Under an annually renewed, systemwide contract with WGI, Rockwell Automation works with all of the incineration sites through surveys and visits to forecast requirements for control system spares (based on historical and forecast spares depletion rates), establish certified spares inventory levels for each site, repair or rebuild failed parts and components, and to special-build new parts and circuit cards as needed to keep maintenance inventories at a safe operational level. The Rockwell Automation support group, located in Cleveland, Ohio, also implemented a relatively simple soft-
ware module upgrade so that the older PLC-3s could improve performance and interface with new additions to the control systems.
Rockwell Automation also maintains a stockpile of spares at its facility in Memphis, Tennessee, equivalent to about 20 percent to 25 percent of the installed base to permit 24-hour turnaround for spares requests. The inventory of control system spares consists of about 75 percent of the installed part types. This includes a recent discovery of a cache of I/O parts at UMCDF, which will allow the inventory to provide for most part types. Any shortages are filled from Rockwell Automation’s general spares supply that it uses for all its customers. Currently, about 90 percent of the CMA/WGI control system spares inventory have been retested and recertified.
The local site inventories of control system spares are specified to be at least 10 percent of the installed base, with a minimum of two for each replaceable part. However, sites are maintaining their own supplies of additional noncertified control system spare parts and this is not well coordinated with the central spares control system at Rockwell Automation. The spares inventory control systems vary from site to site, and the committee noted that there is no comprehensive accounting of the inventory levels of spares, nor of the testing and certification status of spares, at some sites. This can result in program inefficiencies and cost penalties.
Finding 2. The contract with Rockwell Automation for management of control system spares is an effective way to ensure long-term availability of control system spares. This contract is currently subject to annual renewal as part of an existing WGI-JACADS contract.4 This holding-pattern approach to a long-term commitment is due in part to an in-demnification issue pertaining to issuance of a new longer-term contract.
Recommendation 2. The Chemical Materials Agency should implement a firm long-term contract to handle control system spares management for each of the four incineration facilities and the Chemical Demilitarization Training Facility at the Edgewood Area of Aberdeen Proving Ground. This contract should be structured to provide support through all related closures. (Tier 2)
Other Equipment Spares
The basic spare parts inventory is located in warehouses at each site. Smaller local inventories are also situated in such areas as the mechanical maintenance shop, where frequently used parts can be stored conveniently inside the perimeter fence to prevent repeated security delays in bringing spares into the main facility. Some spares backup accessibility is also provided by (1) a central warehouse colocated with TOCDF (which is stocked with parts from JACADS and surplus from other related Army activities) and (2) a cooperative mutual support program with the local warehouses at other sites using overnight shipment of parts.
At each site the warehouse uses a reorder policy for maximum and minimum stock levels with a reorder threshold level established for each item stocked. The maximum, minimum, and reorder levels are based on the initial experience of spare parts management from JACADS with modifications based on ongoing continued operating experience at active facilities. One site had a goal of 98 percent parts availability, and that goal is currently being met.
The committee saw no evidence that a mathematical-model-based inventory policy as used in many manufacturing industries was being employed. The objective of such policies is to minimize inventory storage costs while ensuring a defined and very low probability of out-of-stock items. The central warehouse uses a MAXIMO database, but relies on human intervention to notice any unusual usage that would trigger a change in required inventory levels. At the sites, warehouses use the TRACS system, which is much more comprehensive than just parts tracking, as it also provides for control over scheduling, work orders, and personnel function.
An informal network of communications exists between warehouse personnel across the sites to locate specialized major demilitarization equipment items, such as a spare furnace section. Such items have long lead times, if they are currently available at all, but usage is so low that separate inventories at each site are neither required nor economical. Warehouse managers gave many examples of this network in use, stating that parts can be shipped between sites in “one or two days.”5 Warehouses keep track of loans and paybacks to other sites for correct charging.
One obstacle is that part numbers are not consistent across sites, nor with the original government-furnished equipment identifiers. Warehouse personnel often use the original drawings as the basis for positive identification of particular parts, where the shape, size, and material can be easily communicated between sites. Site warehouse personnel reported little use of the central warehouse (adjacent the TOCDF) with its stock of refurbished JACADS parts, although all knew of its existence; however, the management of the central warehouse reported use of its parts at different sites.
If parts are no longer available from their original equipment manufacturers (OEMs), a number of techniques are used. Occasionally, parts can be fabricated by the small but well-used machine shop at each site. This also allows the
rapid production on nights and weekends of otherwise unavailable but needed parts. The second strategy is to use a replacement part from another manufacturer. This option is used for relatively common chemical and oil industry parts, such as pumps or valves, where a component with identical or improved performance can be substituted. This typically requires less modification of permits than changes involving more specialized demilitarization equipment. Such parts sometimes do not have identical couplings so that some machining of adaptors is required. The third strategy is to identify the impending loss of an OEM supplier and take steps to buy up its remaining inventory. For example, this strategy was used for self-contained breathing apparatus (SCBA) systems that are certified to use with protective suits for entry into agent-contaminated areas of the plant. In this case the manufacturer’s entire stock was purchased after a calculation was made that it would be sufficient to complete the chemical stockpile disposal program.
At one facility electrical substation, parts that were no longer supplied by the vendor had to be replaced. Through the use of the engineering change proposal (ECP) system employed at the plant, equivalent replacement parts were obtained through a local vendor. This is another example of the types of proactive steps being taken by plant management to address potential obsolescence issues.
For any of the above strategies to be used there must be a trigger that alerts warehouse personnel to the situation. This typically occurs when personnel reorder a part only to find that it is no longer available. The reordering mechanism could also involve a Web site posting of a requirement, and then having no vendor provide a quote on the item required. Automatic reordering has been instituted at some sites, where reaching the established minimum stock level automatically triggers a reorder that is then processed by hand. These mechanisms constitute essentially a system that reacts to detected changes rather than a proactive system that would anticipate the changes, although some vendors do notify sites when parts are discontinued. The reorder level is based on the assumption that a replacement part will be available within the anticipated lead time when ordered from the current supplier. Should that not be the case, an unanticipated extended delay is incurred while an alternate supplier is found (if one exists) or a replacement is approved through the appropriate management of change process. It should be possible to have office personnel make regular checks on each item in the inventory so that the discovery that a part was unavailable would not depend on and be linked to the reorder level. In this way stock-out situations could be more easily avoided.
Many examples of inventory management software and systems exist with capabilities to translate across different part identification numbers and actively track inventories. The Army should not have to engage in basic design and development of software. Conversely, not all existing applications are appropriate or effective for this particular situation. On the contrary, in the absence of a judicious choice of platforms, costs can escalate and results can be problematic. Therefore, the implementation of an improved inventory management system should be done only after the requirements of the organization have been thoroughly defined and articulated, and a conscious effort then made to implement the minimum effective software solution to support that definition.
The committee acknowledges that to date no evidence was found that major items critical to continued operability had become unavailable and thus stopped operations using the current inventory management system. However, TOCDF did experience considerable and unnecessary damage to its DFS kiln when kiln speed sensors were not accurately measuring the speeds needed for safe operation during VX mine processing. These sensors were obsolescent and were no longer supported by the vendor.
Finding 3. No systematic approach exists for managing spare parts inventories across the four different chemical agent stockpile incineration facility sites, although there is a clear need for one. The committee found effective systems at some sites, but they are not consistent in parts identification codes, do not integrate information about control system spares, do not have internal trending and tracking capabilities, and only react to loss of suppliers on discovery that a supplier is no longer able to provide needed parts, typically when parts are reordered.
Recommendation 3. The Chemical Materials Agency should implement a more formal inventory management system that models usage, usage variability, lead times, and costs to ensure a very low rate of out-of-stock items. The spare parts inventory should be modeled across the four chemical agent stockpile incineration facility sites to ensure the capability to translate across different identification numbers for the same spare part, and to facilitate the optimum distribution of inventory between site warehouses and the central parts storage warehouse. Such a coordinated spare parts inventory management system should be accessible to and updated regularly by all sites and should indicate the status of key vendors supporting spare parts availability. A proactive program should be instituted to check more regularly whether each inventory item is still available, for example, by telephone or Internet query. (Tier 2)
The rocket shear machine (RSM), bulk drain station (BDS), mine machine (MM) drain station, the projectile/ mortar disassembly (PMD) machine, and the multipurpose demilitarization machines (MDMs) are collectively referred to as disassembly robotics. These machines are automated and remotely controlled equipment connected by a conveyor system. They are unique to the chemical stockpile disposal
program and were initially designed with the assistance of Surface Combustion Corporation, under contract to Edgewood Arsenal with later design work by El Dorado Engineering Inc. (EDE) under contract to the program manager for chemical demilitarization (PMCD). The contractual terms provided for PMCD to obtain the detailed drawings for all the robotic units so that the drawings are available for replacement parts to be made as required. Both the PMDs and the MDMs were built by Wright Industries of Nashville, Tennessee. Surface Combustion Corporation, EDE, and Wright Industries are still in business and are available for equipment support as required.
General Physics Corporation continues to have personnel at the CDTF the Edgewood Area of Aberdeen Proving Ground in Maryland, where the robotic and processing control systems designed for the incineration facilities are replicated and used for training. In addition to the CDTF’s long-time training function, part of the operation at CDTF includes a skunk works6 for the purpose of troubleshooting unforeseen problems that have occurred with the disassembly robotics. Since the CDTF/skunk works is not contaminated with agent, troubleshooting can be more readily accomplished than doing so at the four operating facilities.
Experienced General Physics personnel spent considerable time with the committee, demonstrating the robotics operation on agent-free rockets, ton containers, munitions, and mines. They also answered numerous committee member questions and described the skunk works operation.7
Finding 4. The Chemical Demilitarization Training Facility/skunk works remains an extremely valuable resource for continued support of the disassembly robotics operation.
Recommendation 4. The Chemical Demilitarization Training Facility/skunk works facility and its knowledgeable personnel should be maintained as a support resource until all chemical agent stocks have been safely removed from storage and processed by the disassembly robotics at all incineration sites. (Tier 3)
Furnaces and Afterburners
The incineration facilities utilize three types of furnaces for the destruction of agent and decontamination of materials, the DFS, LIC, and MPF. Representations of the three furnaces are presented in Figure 2-2.
The DFS is a thick-shell alloy steel, counter-currentflow rotary kiln for destruction of energetic materials. It has internal spiral flights to advance and keep energetic pieces separated. Gases discharged from the kiln pass through an afterburner where they are subjected to a temperature of 2200ºF. The DFS is followed by a heated discharge conveyor (1000ºF) for further decontamination of solid materials discharged from the DFS.
The LIC is a high-temperature liquid incinerator built by T-Thermal Company, utilizing a T-Thermal burner for destruction of chemical agents. Agent is atomized and destroyed at very high-temperature (~2700ºF), with exhaust gases sent to a secondary chamber operating at 2200ºF. The LIC has a carbon steel shell lined with high-temperature refractory brick with dip joints of high-temperature mortar. Maintaining the LIC at operating temperature and minimizing the number of heating and cooling cycles is important to maximize refractory life.
The MPF is a three-zone roller hearth, refractory-lined steel shell furnace built by Wellman Furnaces Inc. to operate at 1450ºF or higher in order that materials to be processed can be heated to at least 1000ºF for at least 15 minutes. It is used for decontamination of metal munition bodies and other materials. The MPF has multiple North American Manufacturing burners, special water spray cooling nozzles for temperature control, and entrance and discharge “air lock” chambers. Gases discharged from the furnace pass through an afterburner where they are subjected to a temperature of 2200ºF.
The furnace designs were initially formulated at Surface Combustion Corporation in the early 1970s under contract to Edgewood Arsenal.8 Each furnace has an afterburner with a high-temperature refractory-lined steel shell and commercial natural-gas-fired burners. The afterburners were also originally designed by Surface Combustion Corporation. Drawings and materials specifications are available so that replacement parts including the specified refractory materials can be purchased as required.
Subsequently, PMCD took over the design program and contracted with Stearns Catalytic Corporation to provide the furnaces. The latter utilized Allis Chalmers (now part of Metso Corporation) for the DFSs, Wellman Furnaces Inc. for the MPFs, and T-Thermal Corporation for the LICs. These companies, or their successors, continue to operate and can provide replacement parts as required for the furnaces. Continental Research & Engineering (CR&E) is a more recently established company that has personnel who are specialists with over a decade of experience in the operation and maintenance of chemical demilitarization furnaces and who continue to support the four incineration facilities.
The MPFs and the LICs are similar to equipment utilized in industry. The furnaces utilize commercially available components, such as burners, refractory, and thermocouples. With good maintenance, similar furnaces in industry can continue to be operated for three decades or longer.
The DFS is a more specialized design in that the shell sections at the munitions inlet end are 2 inches thick in order to withstand intermittent explosions of the energetic materials from rockets and mines and some projectiles. The helical flights are also subject to severe intermittent stresses. At TOCDF significant holes in the DFS shell were discovered during an inspection at the end of the VX destruction campaign. This was the result of the DFS having operated above the recommended 0.7 rpm during the VX mine disposal campaign and overfeeding energetics. As previously noted, the installed kiln speed sensors were not measuring the speeds accurately and the vendor was no longer supporting these sensors. An analysis of the damage to the DFS kiln resulted in a recommendation by ANCDF management, and with which the CMA concurred, to repair the damage by welding patches over the holes rather than replacing kiln shell sections.9 Replacing one section of the kiln shell is difficult because of warping and lack of alignment with adjacent section(s). Munitions furnaces at other Army sites that process nonchemical munitions were repaired this way.
After the experiences at TOCDF, the ANCDF site project manager told the committee that there were concerns about the integrity of the DFS during the upcoming VX mines campaign at the Anniston facility. UMCDF and PBCDF also have upcoming campaigns involving mines, which were a significant processing problem at TOCDF.10 Since procurement of new 2-inch-thick shell replacement sections for the DFS require a lead time from ordering to
delivery of almost a year, a failure that could not be safely patched would lead to an unacceptable delay in processing.
Recently the DFS helical flights at ANCDF were damaged during a 105-mm projectile campaign and had to be replaced. The DFS kiln is still safe to use, but the CMA has been requested to order a spare kiln to keep in stock for possible future site or programmatic needs. The CMA expects to authorize an order for a full kiln with a delivery date in the third quarter of Fiscal Year 2007. Present plans are to start processing VX mines at ANCDF in March 2008. After the start of the mine campaign at ANCDF, the CMA will take prompt action to place the order for a second spare kiln if that is determined to be necessary.11
The MPF was designed to handle some residual agent heels, but more degraded agent gels and sludges may require modifications in operating parameters to maintain furnace performance.
The LIC (which also burns decontamination fluids), the MPF, and possibly the DFS (if carbon micronization and combustion were to be utilized as at JACADS) are needed during closure, so they must be able to continue in operation beyond the completion of agent destruction.
Refractory materials used in the LIC and MPF and in the afterburners degrade gradually during processing operations and are replaced on a schedule coordinated with the end of processing campaigns. The relining of refractory must be done by experienced personnel who closely adhere to approved drawings, materials specifications, installation procedures, bake-out, and the initial heat-up schedule.
Finding 5. Upcoming chemical agent disposal campaigns at ANCDF, UMCDF, and PBCDF have some potential to severely degrade the DFS kiln shells. Procurement of new kiln sections requires a lead time of about a year.
Recommendation 5. The Chemical Materials Agency should implement its plan to order a spare DFS kiln in time for mine disposal operations at ANCDF and continue to evaluate whether additional kilns are needed to support the subsequent campaigns at UMCDF and PBCDF. (Tier 2)
PAS/PFS Gas Cleaning Systems
Each furnace has a PAS gas cleaning system. ANCDF, UMCDF, and PBCDF also have a PAS carbon bed filter system (PFS) in series following the PAS. Figure 2-3 presents a schematic of the MPF PAS, including the PFS. The PAS for the other furnaces are similar at all sites. No PFS is presently installed at TOCDF. Plans call for a PFS to be added at TOCDF to meet the environmental requirements that must be satisfied for processing the mustard agent in storage at that site.
The PAS quench, venturi scrubber, scrubber tower, mist eliminator, and exhaust fan were originally designed by Esso Research & Engineering. Drawings and materials specifications are available for repairs and replacement parts. The mist eliminator elements (“candles”) are available from Ceco Environmental and exhaust fan replacement components are available from Robinson, both of which are still in business.
The PFS is a more recent design than that of the PAS. It includes a gas reheater, particulate filter, multiple activated carbon beds, and a final HEPA filter. Ionex designed and provided the filter equipment and remains in business.
Two commercially available electric-motor-driven exhaust fans in series are utilized to provide for the high pressure drop through the PAS/PFS system. The facilities need to have both of the two exhaust fans operating to run at full capacity. If one fan fails, the facility can do a controlled shutdown using the remaining fan.
The committee does not know of any components of the PAS/PFS gas cleaning systems that are not available to the facilities to allow their continued operability.
Process Control Systems
For purposes of this report, the process control system includes the closed-loop control of plant processes having control response times ranging from minutes to fractions of seconds. Higher-level information management and control is covered under “Information Management Systems” in Chapter 4.
The committee reviewed the process control systems at the TOCDF and ANCDF in detail to assess their capability and capacity to handle current and forecast workloads. In addition, maintenance procedures, spares supportability, reliability, and availability were evaluated. The control systems at these sites are typical of those at the other two incineration sites.
The current control system for incineration processes, HVAC, power, and alarms is relatively robust, with adequate performance margins for system response times, control capacity, and communications connectivity. The system architecture, along with the installed hardware and software, is of 1980 vintage. As such, the committee gave particular attention to the programming language, the human machine interface, built in diagnostics, processing capability, communication integrity, and supportability issues.
As indicated in the process data acquisition and recording system (PDARS) block diagram that is typical of the process control systems at all sites (see Figure 2-4), plant devices are connected to PLCs via Allen Bradley series 1700 I/O modules. PLC-3s handle digital inputs; PLC-5s handle analog inputs. The PLCs are interconnected by a Data High-
way II having redundant characteristics, and with a diagnostic unit checking the redundant paths for communication integrity. If there is a failure in one data highway, that path is disabled with the redundant path carrying traffic until the plant is shut down. Continued operation is not permitted when just one data highway is operational. In addition to being networked by Data Highway II, the PLCs are also connected via a Data Highway Plus to personal computers (PCs) that are used to provide the human-machine interface (HMI), and to accommodate control programming with ladder logic program language. The data highway network interconnects with VAX cell-level computers via Rockwell 1779 series KFMR communication interface units.
Batch process computers are networked with the VAX computers via a PDARS Ethernet network. Data query PCs are networked with Sun batch processors via the site Ethernet local area network (LAN). Data storage is managed by the cell-level VAXs in a redundant manner.
Modifications for Mustard Agent Processing at TOCDF
The management of TOCDF has an outside system integration contractor developing a new, modern control subsystem (see Figure 2-5) that will be connected via a Rockwell 1779 series KP5R communications interface unit to the installed system and will support the planned processes for the mustard agent disposal campaign. The communication LAN will be a standard industrial Ethernet.
The committee noted that the new TOCDF control system Ethernet backbone does not have a parallel or redundant pathway. Although the single Ethernet backbone may have electronic reliability as good as the former redundant data highway, it could be vulnerable to failure from physical damage. TOCDF representatives stated that the installed cost of the single Ethernet was approximately $60,000. Provision of a redundant Ethernet path at TOCDF is worthy of reconsideration.
The batch process computers will be connected to the PDARS database via the PDARS Ethernet LAN, as in the initial system. A back-end database will also be connected via the Ethernet to the cell-level processing.
In the planned modifications to their PDARS configuration at TOCDF, plant devices are to be connected to PLCs via Allen Bradley 1700 series I/O modules, and PLC-3s and PLC-5s are networked by a redundant Data Highway II. This segment of the control system remains unchanged from the initial system.
In the new configuration, Data Highway II is also con-
nected via Rockwell 1779 series KP5R interfaces (10 times faster than the original series KFMRs) to RSLogic 500 systems, which will control the new processes for the mustard agent disposal campaign. RSLinx provides the interface to the PDARS VPN Ethernet LAN. New HMI PCs will have RSView software, and will have an RSLinx connection to the new Ethernet. TOCDF was the first site to recognize limitations in the Advisor HMI as planning was begun for the more complicated mustard agent processing campaigns. The RSView software was chosen without coordination with other sites at that time. As indicated above, a communication interface will permit the control center to have operational control and visibility for both the old and new subsystems.
The old Advisor system at ANCDF was also found in need of upgrading, and with the other WGI sites, a WonderWare system was selected as the replacement HMI. All WGI sites are scheduled to have the data highway diagnostic system upgraded with a new system developed by WGI that will be able to detect trouble at any LAN tap. This will enable fixes to be implemented without shutting down the whole system.
The logic behind the selection of two different HMI subsystems (RSView at TOCDF, WonderWare at the other sites) is based on the control systems architecture modifications necessary to accommodate mustard agent processing at TOCDF. The Tooele modification provides a simple, economic interface for the RSView subsystem, which in turn will provide for more efficient programming for the RSLogic PLCs dedicated to the mustard agent processes. It is the absence of such architecture modifications at the other three sites that allow for effective connection of the WonderWare subsystem.
The programming for mustard agent processing at TOCDF is unique, so software coordination problems should not arise. However, the display interfaces processed by the RSView HMI were modified so that display and related operator control processes would be consistent across all sites. Any future modifications of this type should similarly take consistency across sites into account.
Finding 6. The process control system design is old, but the systems can be satisfactorily maintained and are being upgraded as justified to enhance performance. There is some divergence in the choice of components at individual chemical agent disposal facility sites as upgrades are made, and this may limit future interchangeability of parts and operational knowledge.
Recommendation 6. The Chemical Materials Agency should continue to oversee control system improvements at individual chemical agent stockpile incineration facility sites to ensure that total performance standards are met and that sites coordinate those parts of the control system and its operation that can provide more robust continuing operations at a programmatic level. (Tier 1)
Chemical agent monitoring systems have been thoroughly reviewed by the NRC, most recently in 2005 (NRC, 2005). A key conclusion in that report was that the monitoring systems that are currently in place (ACAMS, DAAMS, and MiniCAMS) are adequate for their intended purposes and will remain so for the foreseeable future. Even though the supporting computer architecture could be considered to be obsolete, it is completely functional and should remain so for the life of the chemical stockpile disposal program.
Because of the risk reduction strategy incorporated into agent processing schedules (i.e., nerve agents first), near-real-time monitoring becomes less critical as destruction of nerve agents is completed. For example, there is no longer a need for ACAMS monitoring in much of the TOCDF facility because only mustard agent remains to be processed. The low volatility and lack of acute toxicity for mustard via the inhalation route of exposure obviate the need for extensive near-real-time monitoring. The same will be true for other facilities as nerve agent processing is completed.
The consequence of lessened ACAMS monitoring requirements is that ACAMS equipment should be available for sites that continue to process nerve agents. Some agent detection capability is needed in support of closure activities that include demolition and/or decontamination of facility structures and equipment. However, there should be enough remaining ACAMS to support these activities.
While there may be new detection technology available in the future, it may not be advisable to adopt such technology unless there are clear and compelling reasons for doing so. Prior to adopting new detection technology, there should be demonstrations that there are distinct advantages to safety, human health, or environmental protection and that the benefits clearly outweigh the cost (NRC, 2005).
Trained and knowledgeable personnel are crucial to successful safe continuing operations at the four operating chemical agent stockpile incineration facilities. Each site requires a highly competent workforce that must be maintained throughout the life of the project. In addition to the plant workers, teams of experts with specialized knowledge are also needed to address the processing upsets, incidents, and deterioration of equipment during extended operational campaigns. Operators and maintenance personnel need to be knowledgeable about the plant configuration at all times, including current operating and maintenance procedures. As the plant ages, personnel must also be alert to detecting equipment deterioration and to responding to abnormalities in processing that may require some corrective action.
Operating and maintenance procedures change in response to the aging of systems and must be updated accordingly. Workers will need training to stay current with such
changes. The changes and the rationale for them need to be reviewed as part of the training, and workers need to demonstrate their understanding of the changes. Also important are the programs for maintaining a highly competent workforce, including site safety culture and communications of unusual process excursions or conditions. Site and programmatic aspects of these issues are further discussed in Chapters 3 and 5.
Operating procedures provide technical guidance to personnel, as well as ensure consistency in the daily operation of the plants. The effective use of operating procedures in chemical agent stockpile incineration facilities is essential to their safe operation and continued operability. Operating procedures are reviewed by oversight groups focused on safety, environmental compliance, engineering, etc., and then approved prior to their use. These written procedures are available to operating personnel on the facility’s database and are reviewed and updated periodically as needed. As facility equipment ages or is modified to maintain function, operating procedures are updated using the management of change processes described in Chapter 3. Operating personnel are involved in the review of operating procedures. The procedures address normal operation, emergency operation, temporary operation, and normal shutdown.
A particular challenge in chemical agent stockpile incineration facilities is the need for personnel to wear protective clothing in parts of the facility where agent is present or might be accidentally released. The highest level of protection is a demilitarization protective ensemble (DPE) suit that resembles a space suit with an air supply system and can allow workers to perform tasks in contaminated areas for periods of an hour or more. The suits are clumsy and hot, so work needs to be planned in advance to make sure that tasks can be performed without undue stress to the worker. The plants are designed to avoid routine worker operations in DPE suits, but nonroutine tasks and maintenance require human performance in DPE suits.
During the disposal of 1,120 overpacked leaking GB rockets at ANCDF, workers in DPE suits had to work inside the ECR to open the flanged overpack containers, remove the leaking rockets, and place them on the rocket disposal conveyors. A team of workers in DPE suits trained together to perform these operations and became quite adept at handling both routine and corroded overpacks. The committee was told that this team is going to UMCDF to work with a team there that will have to perform similar operations in the future.
In Chapter 3 the systems for managing the operating and maintenance procedures and maintaining a fully capable workforce are discussed in detail.
As plants age there is a need for increased maintenance on the process equipment to ensure continued operability. To ensure safe and reliable operation, the maintenance practices used must follow suggested vendor recommendations or be defined internally to address technical and safety considerations. The continued operability of chemical agent stockpile incineration facilities is dependent in part on the effectiveness of the maintenance systems to proactively address obsolescence in their procedures and practices and to modify systems as required to process severely degraded stockpile items.
NRC (National Research Council). 1996. Review of Systemization of the Tooele Chemical Agent Disposal Facility. Washington, D.C.: National Academy Press.
NRC. 2005. Monitoring at Chemical Agent Disposal Facilities. Washington, D.C.: The National Academies Press.