Assessment of the Continuing Operability of Chemical Agent Disposal Facilities and Equipment (2007)

In this chapter the operational factors that affect and contribute to continuing operability at the chemical agent stockpile incineration facility sites are examined. These factors constitute the organizational framework by which the potential vulnerabilities described in Chapter 2 are managed. The first section focuses on the management of problems arising from obsolescence, the second on the means available to ensure that safety and environmental standards are met, and the third on personnel management. Information management is addressed separately in Chapter 4. The final section addresses some issues associated with specific sites.

The safest and most efficient operations occur when functional systems are stable. Unusual conditions (e.g., those arising from aging, unreliable, or substituted equipment; processing degraded agent and munitions; or something not covered by standard procedures) lead to less reliance on rule-based working and greater use of knowledge-based working (Rasmussen, 1983). Human participation is required in systems precisely because people can deal with unusual circumstances not covered by standard operating procedures through knowledge. This does not mean that human knowledge should be relied upon in place of stable operations. This section examines the actual and potential impacts that arise from changes of equipment, and how any negative impacts are and can be minimized. Some equipment changes may also require regulatory permit modifications that are discussed elsewhere in this report.

Equipment obsolescence and equipment changes can impact how operations and maintenance tasks are performed in the following ways:

1. Less reliable equipment can necessitate more nonroutine or off-nominal procedures, increasing reliance on knowledge-based reasoning.

2. Obsolescence can lead to a lack of trust in equipment, in turn leading to work-arounds and changed operating procedures for the purpose of reducing stress on the equipment and keeping it working longer.

3. Older equipment can require increased maintenance, potentially increasing DPE suit entries that are expensive and not without risk, and more time with the plant in maintenance (off-nominal) conditions.

4. Changing equipment from that originally installed may involve changes in the human interface and the operating characteristics. Both of these could lead to an increase in errors; these changes may also be a source of improved capabilities and hence improved performance if properly incorporated into procedures and training.

Changes to the equipment, system, or procedures must be addressed by a process for active management of change, which will have direct implications for the personnel involved at all levels. Moreover, it is not just equipment that becomes obsolescent and requires adaptation. The personnel involved in operations, laboratories, and maintenance all have skills and knowledge that change over time. In an employment field with a limited time horizon, such as chemical stockpile demilitarization, the public needs to be assured that there will be enough skill and knowledge retained in the workforce to safely complete the process across all sites.

The consequences of error in chemical weapons demilitarization are potentially large so that any involved organization must maintain a consistently low error rate over extended time periods to ensure public safety and public confidence. Other organizations may have relevant experience that would also apply to chemical agent stockpile incineration facilities. For example, considerable research on high reliability organizations (HROs) has shown that they do in fact exist and have some common characteristics. These may

be a useful model or benchmark for the chemical stockpile disposal program, in particular, in how HROs respond to change.

Roberts (1990) examined three organizations that had a very high level of safety performance while performing hazardous operations using complex technology and found several common characteristics. The organizations (an electrical supply company, the Federal Aviation Administration’s air traffic control system, and a fighter squadron on a nuclear aircraft carrier) were all adept at “reacting to unexpected sequences of events” despite being in tightly coupled environments with highly time-dependent processes. Like the stockpile plants, these operations are too complex for reliance only on command-and-control procedure-driven processes. Workers also need to draw information from informal networks and from their own intuition when facing an unexpected situation. The characteristics Roberts (1990) reported that these organizations all use to achieve and maintain high reliability were:

1. constant training for normal and unusual conditions (e.g., one week in four);

2. redundancy in equipment and functions (e.g., a buddy system);

3. utilization of indirect as well as direct sources of information;

4. flexible layers of activities to reach goals; and

5. responsibility and accountability at all levels (e.g., any person can stop the activity).

All of these are present to some extent in chemical stockpile demilitarization, but can become increasingly important as the systems age and change, perhaps causing unexpected interactions, or what Roberts (1990, p. 107) calls “baffling interactions.” In examining the chemical stockpile disposal program, the committee used the characteristics of HROs as one benchmark.

Examples of the use of HRO principles were evident at all sites, where people at all levels were seen to be willing to use their knowledge to intervene on the side of safety. For example:

• A “culture of inquisitiveness” was among the operating principles at several sites, and it was taught extensively in training and at team briefings where reasons were presented to improve the understanding of each procedure and its steps.¹

• At ANCDF two incidents were reported where an operator had intervened through inquisitiveness. First, when the deactivation furnace system (DFS) appeared to be running hot, the problem was that a thermocouple had sagged, causing erroneous readings, and the operator took manual control. In a second reported incident, DFS pressure excursions too small to trigger the automatic action threshold were traced and corrected by the control room operator after an unpack operator had reported some unusual noises occurring after weapons had passed out of the unpack area into the adjacent processing area.

• Many examples were presented of operators stopping operations (e.g., during maintenance performed in DPE suits when the actual configuration of hardware did not match that shown in the procedure). At ANCDF and UMCDF at least, such actions are reported favorably in the daily newspaper and even are used as a basis for instant rewards of gift cards for local stores.

The Chemical Materials Agency (CMA) stated at the committee’s first meeting that it is committed to managing obsolescence problems and is also relying on its experienced site contractors who are those best equipped to understand potential issues.² As previously noted in Chapter 1, the Army does have standards and guidance on the management of obsolete weapons stockpiles and other military equipment (U.S. Army, 1999). But while the guidance does include some provisions for dealing with unavailability of parts, the guidance is largely not applicable to aging chemical agent disposal facilities. A further barrier involves differences in the nomenclature used. DOD uses Category 3 as its highest obsolescence risk, with Category 1 being routine. At stockpile disposal facilities, the safety ranking system has historically used Category 1 as the highest risk (that leads to a halt in processing) and Category 3 is routine. Introducing the inverse DOD categorization structure would be confusing from a human factors viewpoint.

Finding 7. The Department of Defense (DOD) standards on obsolescence are focused on criteria and considerations concerning obsolete weapons and other military equipment stockpiles. The DOD standards do not address the broader range of issues encountered in an operational processing facility. Another barrier to partial use of these standards is that they categorize the risk of obsolescence in a manner that is the inverse of the stockpile facilities’ long-standing categorization of safety-related risks.

Recommendation 7. The Chemical Materials Agency should disengage from using the inapplicable DOD diminishing manufacturing sources and material shortages stan-

dards and develop a program for obsolescence management that is tailored to the needs of the chemical agent stockpile disposal program. (Tier 3)

At the request of the CMA, site contractors have begun to develop plans for managing obsolescence. The TOCDF site started making plans independently; the Washington Group International (WGI) sites developed a three-site initiative for the other incineration sites.

Planning at TOCDF focused on high-risk systems—initially the critical utility systems: the emergency generators, the uninterruptible power supply, and the HVAC system.³ Major upgrades on each of these systems have been made where vulnerabilities were found. A database system to track repair orders has been initiated, and plans are to incorporate this information into the TOCDF parts reorder program.

The program at WGI sites (ANCDF, UMCDF, and PBCDF) is focused on critical components and a systematic look for components with an unreliable supply.⁴ A database on parts, spares, and frequency of repair or replacement, vendor(s), point of contact, etc., called the WGI Asset Management program, is being developed. TOCDF will eventually become part of the database since EG&G is also working on this program in close relationship with WGI.

The plan is to use the database information to develop an Obsolescent Equipment Management Lifecycle program (OEM-LP), which is an approach that was originally started at TOCDF. Aspects of this OEM-LP system were shown to the committee during a site visit to TOCDF in March 2006, and this work in progress seems to be progressing in a suitable manner.

While all of the incineration sites are participating in these activities, approaches and concerns differ to some extent from site to site. Progress on review and upgrading of the control systems was discussed in Chapter 2. A systematic review of other critical systems was started in 2004, beginning with a review of the metal parts furnace (MPF) at each of the sites and followed by a review of the conveyors for the container handling building (CHB) and the munitions demilitarization building (MDB), and the liquid agent incinerators (LICs). Reviews of the DFS and the pollution abatement system (PAS) have been scheduled. These reviews are intended to enhance the potential for identifying differences in configurations among the facility sites and the extent to which parts are interchangeable.

The incineration sites have collectively developed a list of experts for different systems and pieces of equipment. At present each expert has one or more backup people. As problems arise from the aging of equipment or changes in operations, these experts are shared by all the sites and can be dispatched for up to two weeks or longer to any site having a need for that particular expertise.

There is a daily teleconference among all the site managers where operations are discussed and where problems and responses to problems are shared.

Finding 8. The site contractors for the chemical agent stockpile incineration facility sites are developing and implementing plans for managing obsolescence at each site. The Army and its contractors recognize that a process to continuously identify and evaluate critical components and parts is necessary to offset increasing vulnerability to obsolescence. Evolving systems at the various sites differ in approach and how critical components and parts are identified. These differences can have adverse implications for future program obsolescence management.

Recommendation 8. The Chemical Materials Agency should implement an effective, consistent, and documented system to manage obsolescence, including the sharing of expertise and spare equipment and parts across the chemical agent stockpile incineration facility sites. (Tier 2)

Each of the chemical agent stockpile incineration facility sites operates within a structure of policies, procedures, and training programs that are intended to ensure operational safety and regulatory compliance. As the facilities and equipment age, changes will be needed to proactively manage obsolescence. A key challenge is to continually update plant specifications, drawings, procedures, and other items related to plant configuration, so that the past and present status of the facilities are documented in an accessible manner and that all permits are up-to-date. In this section the committee presents a review of the configuration management processes in place at the sites, especially the management of change processes.

Equipment changes are not just a function of obsolescence; they also occur periodically as disposal campaigns change and as better methods of operation are devised. Examples are the upgrading of MDB chillers to meet increased heat load demand and better diagnosis of data highway bottlenecks using a new probe test system.

There are some changes necessary for continuing operability that have human performance implications. Certain worn equipment cannot be easily replaced as it ages or circumstances change. An example is the DFS at TOCDF,

which has required changes in operation, primarily reduced loading rates, to keep it running without causing it to be damaged during mine campaigns. Another example is the hydraulics system at TOCDF that had insufficient capacity and started to damage pumps by cavitation. This also led to loading restrictions. Similarly, the data highway at TOCDF has a bandwidth bottleneck so that currently unused screens (e.g., utility screens) in the control room have to be switched off to ensure reliable operations. A final example is from UMCDF, where manual valves located outside had to be “babied” as the handles and mechanisms were aging.

Equipment meant to replace obsolete equipment is not always identical to the installed equipment so machining work may be necessary to make connections fit (e.g., the lift cylinders for the bulk drain system at TOCDF). Sometimes a change in campaigns may lead to an unanticipated incompatibility of materials. An example of this is the sump lining material at TOCDF that was designed for compatibility with decontamination fluids for GB, but was eroded by the different decontamination fluids used for VX. The sumps had to be relined with a suitable new material.

Effective configuration management in an environment where changes to equipment and procedures are occurring requires both an understanding of the change process and formal systems to ensure that the changes are indeed beneficial. As plants age there is an ongoing need for change to maintain process efficiency and safety and to reduce costs. These changes may reflect improvements in technology, lessons learned, equipment changes, or advisories (safety alerts), as well as new parts (other than replacements in kind) for existing processes because the original parts are no longer supplied by vendors. Engineering change proposals (ECPs) are the primary way the configuration of the plant is kept current as equipment is modified or procedures are changed as a result of obsolescence or lessons learned. Many ECPs are processed each year at chemical agent stockpile incineration facilities. For perspective, ANCDF has processed about 600 ECPs since starting operations and has a backlog of 40 awaiting action.⁵ Each ECP must be reviewed and accepted by quality assurance, safety, maintenance, operations, engineering, and permit compliance departments before being implemented.

While ECPs are generated at all sites for major changes in plant configuration, the policies concerning review of more minor configuration changes vary from site to site, sometimes because of regulatory requirements. Since ECPs are generated independently as needed at each site, their implementation may lead to some divergence in configuration among the sites, unless intersite communications or implementation of a lesson learned lead to a common change. From a programmatic safety and operability viewpoint, a consistent policy for handling different sorts of configuration and procedural changes is desirable.

There are variations from site to site in the criteria for requiring an ECP for more minor changes. This often involves a judgment call that may overlook the need for a more comprehensive safety review. The committee reviewed about 75 lower-level changes that were made at TOCDF without requiring an ECP. These included the following items, all of which required a documentation change:

• work order No. 122836, Install a valve at ground level for 3/4 -V-56 Venturi;

• work order No. 125927, Install a new isolation valve V-9895 on Line 5244;

• work order No. 140939, Chilled water piping modification (not replacement in kind); and

• work order No. 142294, Change flame scanner cooling air from compressed air to instrument air source.

Changes such as these could have potential impact on overall system integrity. For example, at UMCDF under procedure UM-EN-003 minor changes do not require an engineering action classification and may be implemented with only a document change notice. The regulatory definition of management of change includes all modifications to equipment, procedures, raw materials, and processing conditions other than replacement in kind. The examples given above are not replacements in kind. As the equipment continues to age and have more modifications such as these, a careful review of the criteria for requiring and implementing ECPs consistently across the sites is prudent.

To ensure that changes do not adversely impact the safety and operability of chemical agent disposal facilities, a disciplined management of change process must be followed. When an ECP is adopted and leads to changes in processes or procedures, the changes are analyzed for any potential impacts on the process hazard analysis. This is the main documentation for determining that safety levels are acceptable. Changes that pass a proper process hazard analysis safety review will not have any significant impact on the overall results of the quantitative risk assessment for the facility.

If the ECP involves a physical change, an engineering work package is generated to implement the change. This triggers a configuration management process that includes updating plant drawings and relevant procedures. The Army field office at each site also has a configuration control board that approves and tracks these changes.

A means being used to make the change management process more effective at all sites the committee visited involves placing more reliance on team review, team training,

and mutual understanding, and less reliance on multiple signoffs of ECPs. However, it is still not universal practice at some sites to have the engineers who write the ECPs verify them “on the ground” with actual users—an essential step in ensuring workable procedures and discouraging the need for subsequent revision.

Whenever an ECP results in a corresponding change in any of the written procedures or other configuration-related documents, a document change procedure (DCP) is initiated and subjected to review processes that are similar to an ECP. At UMCDF and ANCDF, the committee saw evidence of field walk-through reviews of revised procedures by teams including operators. These team reviews help to identify any errors or possible improvements to the procedures. The actual users of the procedures must sign that they have read them before work can start. Although such a process does not ensure that the procedures are well written, work documentation observed at all sites was typically well designed for operator use. For example, the work orders are self-contained and do not require the user to consult other documentation.

During this study, a DFS incident occurred at ANCDF (see Box 3-1) that highlights the importance and possible impacts of the management of change processes. Here the site used an ECP for the same operation from TOCDF and relied on an experienced contractor to provide safe implementation rather than perform an independent review that might have identified minor differences in configuration that turned out to be important. This incident notwithstanding, good configuration management and management of change procedures are in place at all sites, such that any differences between the DFS at ANCDF and the other sites can be reviewed individually. At the time this report was in development, a root cause analysis report of the ANCDF DFS bricking incident was being prepared. Recommendations are expected to result that should address contributing deficiencies in the management of change procedures.

Critical safety systems are usually considered to be those that are necessary for safe plant operation during processing and also those required for the safe shutdown of the plant in the event of an unanticipated malfunction. Within an incineration plant, examples of critical safety systems include agent monitoring systems, testing or inspection of pressure relief valves, ventilation flow and control systems, fire protection systems, emergency alarm and shutdown systems, process interlocks, and furnace temperature control systems.

As plant process equipment ages a continuing need exists to ensure the reliability and functionality of supporting systems that are critical to safe operations. Aging or leaking munitions may also cause safety systems to be activated with

some increase in frequency. Factors that could adversely impact the effectiveness of the supporting critical safety systems thus require an established means for ongoing review and evaluation.

Within the incineration facilities, critical safety systems must be properly identified and maintained to ensure safe

continuing operations. Since the incineration systems designs are similar at the four operating facilities, the critical safety systems should also be similar. The committee evaluated the consistency with which each of the plants identified critical safety systems, as well as evaluating the supporting maintenance and testing or inspection programs for validating the readiness of the systems.

Based on interviews with personnel at three of the incineration plants, the committee concluded that critical safety systems in general are clearly identified but that, as described previously, the definitions at TOCDF differ from those used for the WGI sites, although work is in process to integrate the best aspects of the two approaches into more consistent plans.

“Advisories” are safety alerts to address serious safety issues that need immediate attention. Advisories are communicated among the incineration plants and may be reviewed informally in the daily site manager teleconferences. However, no formal feedback mechanism to communicate and document that the safety alerts have been implemented was observed, although they often may become part of the lessons-learned database where follow-up is documented.

Supportive safety procedures are utilized to ensure that there are necessary controls in place for safe maintenance and repairs; that importantly includes nonroutine hazardous work. These procedures encompass rules governing lockout and tagout, confined space entry, elevated work, electrical work, etc. The committee found some minor problems in these supportive procedures and differences among the sites, especially when some of the work is done by subcontractors, who may have their own practices.

The capability of a chemical agent disposal facility to have an effective emergency response is critical to its continuing operability. In evaluating the emergency planning capability at the incineration plants, the committee focused on the capabilities of the plants to address accidents, medical emergencies, significant operational safety events, process upsets, natural disasters, acts of terrorism or sabotage, and loss of utilities. Written emergency response procedures have been developed that address these identified emergency scenarios.

For those potential hazards that could affect the surrounding community, an active outreach program has been established to inform local emergency response agencies (fire department, police, civil defense, hospitals, etc.) as well as local civic organizations of emergency response actions to take in the event of an emergency. Civic organizations in the community are immediately notified in the event of an emergency. The outreach program includes training on shelter-in-place techniques and, at one facility, respirators have been provided to families in the potentially affected areas.

However, critical emergency safety systems may also be subject to potential obsolescence. For example, at one plant a potential obsolescence issue relating to maintenance of the fire water protection system had to be proactively addressed. Also, the committee was informed that the Mine Safety Appliances (MSA) Company, which has supplied parts for the self-contained breathing apparatus (SCBA) respirators, will no longer supply these respirators in the future. In response, the Army purchased the remaining supplies. At the current use rate these supplies are expected to last through the end of the chemical stockpile disposal program. Commercial suppliers of SCBA respirators are available. The committee believes a program to identify and evaluate other qualified suppliers of the respirators is prudent.

Obsolescence issues related to emergency systems are being proactively addressed at the incineration facilities. For instance, the ventilation system and emergency generators at TOCDF have been upgraded.

The committee was informed that operating and maintenance personnel from at least two plants are trained to Level 1 of the Occupational Safety and Health Administration’s (OSHA) Hazardous Waste and Emergency Response Standard (HAZWOPER). Emergency responders also receive HAZWOPER training, which is required for cleanup operations at RCRA-permitted sites.

Finding 9. The chemical agent stockpile incineration facility sites all have adequate safety programs, but there is considerable variability from site to site in definitions of critical safety systems; dissemination and response to site-specific and programmatic safety alerts (advisories); procedures for control of nonroutine hazardous work; details in policies for management of change systems; applicable attributes of high-reliability organizations; and organization and implementation of training programs, including the testing of their effectiveness. As facilities and their equipment continue to age, effective safety programs must be maintained until each facility is fully decommissioned (closed).

Recommendation 9. In the interest of continuing safe operability, facility staff at each chemical agent stockpile incineration facility site should continue to compare their operations and performance with those at other sites and with practices in the broader chemical industry for dealing with hazardous materials to ensure continual improvement in safety performance, consistency in major safety practices, and safety-related cooperation among the sites. (Tier 1)

Each of the incineration sites operates under Resource Conservation and Recovery Act (RCRA) and other state and local permits that specify plant configuration and operating parameters approved for environmental compliance. Any significant changes to permitted conditions must be reported, and major modifications may require extended re-

view and comment before a permit modification is issued. Obsolescence planning needs to anticipate any replacements or changes in operation that require permit changes so that timely application for permit modifications can minimize the impact of permitting delays on overall schedule.

Some of the incineration sites initially had permit constraints on the amounts and types of secondary waste (e.g., used DPE suits, packing materials, filter charcoal) that might be shipped off-site for commercial disposal. For example, TOCDF has a substantial backlog of secondary waste that will ultimately have to be treated during closure. TOCDF consequently will have an extended closure period. ANCDF has found ways to dispose of secondary waste to reduce backlogs during periods when the MPF is not being used for demilitarization operations. Other sites are allowed to do some off-site shipment of secondary waste to avoid accumulation. The WGI incineration facilities are incorporating programs for disposal of wastes into operational plans to avoid any significant accumulation for treatment during closure. Off-site shipment can significantly reduce the operating duty requirements imposed on the LIC and MPF by secondary waste disposal during plant operations and during closure.

Maintaining the human capabilities of personnel within aging chemical agent stockpile incineration facilities is as important as dealing with the obsolescence of equipment. First of all, it is essential that a highly competent workforce exist at each site. A second critical issue is the retention and renewal of capable personnel and their expertise throughout the life of the facility operations.

As the facilities age it is crucial that operating and maintenance procedures are clear and current and that tasks are designed properly so that workers can be utilized most productively. As changes to obsolescent systems are incorporated, proper attention to human factors is likewise an opportunity for the continuing operability of the facilities to be enhanced.

Safe operability depends not only on equipment but also on the skill and knowledge of workers and managers. Not all information essential to continued safe operation resides in specified procedures and system documentation, including past end-of-campaign reports from other sites; the committee noted that these reports were much consulted guides to current operations.

Programs to maintain the necessary personnel skill and knowledge within the demilitarization system are needed at all stages of operations, but they become particularly important as campaigns near their end, as problems with some badly degraded agent lots are identified, and during closure of a facility. Given the large investment made in personnel training for chemical demilitarization and the long time required to bring new hires up to safe operating standards, personnel retention is preferable to the alternative of recruiting new personnel.

Still, when people leave, even at the lower organizational levels, a system of knowledge retention can help to ensure that operational wisdom is not lost. Knowledge management includes processes for technically debriefing personnel who are departing in order to retain their knowledge and skills and transfer it to replacements. The committee observed little evidence of such knowledge management efforts at any site, with exit interviews being the typical limit in this regard. Knowledge management is important at all levels of relatively unique organizations such as those engaged in chemical demilitarization, from operators through maintenance personnel to engineers and managers. The previously noted list of experts that has been compiled across the sites is a useful way to begin a knowledge management program. All-site topical workshops scheduled among the sites and CMA several times each year also reinforce communities of practice at site and programmatic levels.

Even when staff members do not leave, important knowledge may not be available to address problems at one site if it is not known that a person with such useful knowledge exists somewhere else. To avoid this situation, the organization must be able to define the skills and knowledge necessary for each technical community, identify who has such skills or knowledge, and enable accessibility by each technical community to them when needed. Such an approach implies that communities of practice—plant operators or warehousing specialists, for example—be defined and supported in a formal knowledge management program. Although there has been some activity of this type already with respect to the chemical stockpile disposal program,⁶ the overall state of practice regarding knowledge management is at this time ad hoc and incomplete.

A knowledge management program is clearly an important ingredient for continued successful operation of the disposal facilities, and will only become more critical as the number of operations decreases in the future and the current personnel are discharged. One of the challenges that will be particularly difficult in this situation is the limited number of individuals with experience in very specialized areas, making a traditional community of practice much more difficult to sustain. It is likely that more face-to-face contact between experts from the different facilities will be required to maintain necessary communications, and incentives to encourage

that communication through monetary or other motivating devices will be needed for success. Careful thought should be given to implementing this system, and an inventory of critical knowledge should initially be completed to make sure that efforts are placed in the areas having the highest value for continuing operations.

Finding 10. Only limited evidence was observed regarding programs or measures to share or protect the knowledge base important to the safe and effective functioning of the various communities that together constitute the capabilities on which the operability of chemical agent stockpile incineration facility systems depends.

Recommendation 10. A formal knowledge management program should be implemented by site managers under the guidance of the Chemical Materials Agency to identify various communities of practice among the workforce at chemical agent stockpile incineration facility sites, define the skills these communities encompass, and ensure that the skills of each member in a particular community are made available to the community as a whole as necessary. This program should also monitor the skills inventory in each community with the aim of detecting potential vulnerabilities (e.g., situations where the personnel to deliver any particular mission-critical capability are limited), and implement measures to retain staff and increase capabilities among staff where such vulnerabilities (limitations) are encountered. (Tier 2)

There is little potential for the creation of new jobs at the chemical agent stockpile incineration facilities since all of the contracts are already in place and operations are in progress. Workforce recruitment at these facilities has been based on a mixture of local personnel and experienced personnel from the Johnston Island facility or other continental U.S. facilities. The current workforce is relatively stable. The committee notes this is unlike the situation at Johnston Atoll Chemical Agent Disposal System (JACADS) where incentives were needed for retention. The turnover at current operating facilities is dependent on what other local opportunities are available and is lower (about 1 percent) at TOCDF but higher at ANCDF and UMCDF (about 10 percent) where more competing job alternatives exist. PBCDF still is not fully staffed because of a limited local skill pool and difficulties in getting nonlocal personnel to relocate. Opportunistic losses, such as when several maintenance staff at UMCDF left for jobs in an Alaska oilfield, also occur. As the operations at each site wind down, personnel may begin to leave for more permanent jobs before the stockpile demilitarization is completed. No evidence of this at present was reported to the committee, but that it happened at JACADS is a concern for the continuing operability at currently operating facilities.

Site contractors will probably employ various types of cash incentives, such as bonuses for continuing service or termination bonuses for staying to a predetermined end date based on programmatic needs. Retention incentives can go beyond cash rewards for continued employment. A variety of noncash incentives to help retain personnel were used at JACADS and continue to be used at the currently operating facilities in the continental United States. Related activities include support of community planning activities that might provide replacement jobs after facility closure. At JACADS the retention of personnel, and hence the skill and knowledge they possess, was accomplished with the introduction of retention bonuses, although these were started rather late in the operational life of that facility when many key personnel had already been lost. These included:

1. provision of educational and placement opportunities to make it easier for personnel to remain with the chemical stockpile disposal program to the end and still find employment after the closure of JACADS, as well as educational opportunities for the families of personnel employed;

2. extended medical benefits after closure; and

3. provision of portable 401k or similar retirement plans.

Finding 11. Personnel turnover varies across chemical agent stockpile incineration facility sites, but is generally low. However, this situation may not continue as facilities approach the end of disposal operations. Retention incentives have not yet been applied in an effective manner to help ensure continuing safe operations.

Recommendation 11. The Chemical Materials Agency should ensure that site contractors promptly develop staffing plans capable of being tailored to site-specific needs while recognizing the challenges of maintaining a competent staff throughout operations and closure. (Tier 2)

Another aspect essential to the continuance of safe chemical stockpile demilitarization operations is the training of personnel. Since the equipment and operations at sites have undergone a gradual divergence, training is no longer centrally provided mainly by the CDTF at the Edgewood Area of Aberdeen Proving Ground (Maryland), but is now largely site specific. Training occurs in several ways: at initial hiring; as regular refresher training; and as needed (just in time) for new and changed operations. Although most recruits arrive with some chemical industry training and are knowledgeable about basic chemical process equipment, the specialized chemical demilitarization equipment and agent safety procedures will in many ways be novel.

The content of the training must be closely aligned with the current operating procedures. Evidence was presented to

the committee that procedures and engineering changes were being revised and personnel trained more consistently at TOCDF than in the past.⁷ Training is provided in a more performance-based manner, but still consists of either classroom or on-the-job (work experience) modes. Computer simulations are used in training control room operators to respond to upset or emergency conditions. In addition to computer-based training for knowledge transmission, more use of computer simulation models may also prove helpful in training teams to work through revised procedures before they are implemented in the plant.

Each plant’s certification program for its operating personnel includes evaluation of their knowledge of the pertinent operating procedures and documentation of training and retraining. Because each site has developed its own training and testing programs, the committee noted some inconsistencies among sites. A sharing of best practices across sites could result in more effective programs at all sites.

One example of a well-thought-out change to address an issue of obsolescence is the replacement of the control room operating system (Assist) with a new one (RSView or WonderWare) when the former was no longer supported. There was a choice during this upgrading to change to an operator interface more cognizant of human factors than was possible with RSView. However, the decision was to change the operating system but to keep the interface similar to the original. This effectively decoupled the new operating system from the existing operating interface to allow each to be separately verified in operations. A second good feature of this change was to maintain the two systems in parallel on the two screens at each workplace until all operators and supervisors were comfortable with the change (Eason, 1989). This is an example of equipment redundancy essential to HROs.

At ANCDF this parallel running was preceded by using each system in turn to command changes while the other system was used to monitor the effects of these changes. However, one aspect of this change at ANCDF was not well engineered from a human factors perspective. The spatial mimic screen showing the ACAMS alarms and current readings was replaced by a tabular listing screen, to ensure good use of screen space. However, the list did not allow direct spatial perception of spreading changes in ACAMS readings, as could be experienced during upset conditions, thus increasing the time required for the operator to achieve the necessary situational awareness. Operators said, “We will soon learn it,” but that approach is tantamount to using a training solution to address a hardware issue. The lesson is that any equipment interface changes must not adversely affect human performance.

At UMCDF the committee observed that there was consistency in format across procedures in that all sentences started with an action verb presented in bold type, and branches in procedures were well designed with an IF (condition), THEN (Action 1) ELSE (Action 2) format, although this was not seen in all cases. Further improvements could be the use of techniques such as the specialized language of Simplified English, and flowcharts at the start of each procedure to provide an overview to operators before they read the detailed steps.⁸ Where changes are involved, the design and even the formatting of ECPs can help operating personnel to perform new or revised tasks with minimal or no increase in errors. Conversely, poor procedure design can lead directly to operating errors (Drury, 1999).

The maintenance of facilities includes preventive maintenance programs that identify required preventive maintenance activities and recommended intervals based on safety, reliability, and economic considerations. Factors such as failure rate, replacement costs, impact on operations and safety, lessons learned, and analysis of incident causes are considered. Computerized programs provide for preventive maintenance schedules to be planned, implemented, and monitored.

Significantly, there also is a system at the WGI sites that encourages “condition reporting.” Any employee can write a report that indicates a condition (e.g., corrosion, impaired operability, or a maintenance problem) that the employee thinks is worthy of further investigation. The condition report is entered into an online database and is directed to the employee’s supervisor for assessment. If deemed valid, this triggers an investigation that ends with an action communicated to the initiator of the report.

The effectiveness of the maintenance program at two of the facilities reviewed was increased by using predictive maintenance. The predictive maintenance program monitors key equipment indicators predictive of impending problems by detecting problems in the early stages. Plans and schedules are then made for maintenance to be performed prior to predicted equipment failure.

Failure data generated during preventive maintenance and predictive maintenance present an opportunity to identify potential equipment obsolescence issues. For equipment

whose parts are no longer supplied by vendors, ECPs are used. These include review and approval by quality assurance, safety, engineering, and operations personnel to ensure that technical and operational design considerations are evaluated.

In general, plant personnel use the recommended maintenance procedures provided by equipment vendors. These personnel have been trained and possess the skills needed for the type of maintenance work to be performed.

The maintenance system and its procedures have evolved over time and now include a 12-week planning horizon. This enables personnel to schedule procedures having similar requirements for access to areas and equipment at the same time. Another benefit is that teams are more stable and there is increased emphasis on knowledge-based operation that is cultivated within a culture of inquisitiveness. The longer-term scheduling also allows parts required for each job to be ordered well in advance. There is no formal “kitting” that gathers all parts and supplies needed for each job, but all of the more specialized equipment is stored together to make job preparation simpler and less error-prone.

This section discusses issues and concerns the committee identified at specific CMA sites. Specific and unique issues were identified at TOCDF, PBCDF, and the Newport Chemical Agent Disposal Facility (NECDF), which uses hydrolysis (neutralization) to destroy bulk VX nerve agent stored in ton containers instead of using incineration. Also included is a section on issues that may have applicability to, and should be considered in, the disposal operations planning for the two future Assembled Chemical Weapons Alternatives (ACWA) program facilities in Pueblo, Colorado, and Lexington (Blue Grass), Kentucky, which also employ neutralization as the primary destruction technology.

The stockpile at DCD represented over 42 percent of the total original U.S. chemical weapons stockpile, counting all agents and munition and container configurations. TOCDF began disposal operations in August 1996, first destroying GB agent and munitions and completing GB destruction in March 2002. After facility decontamination and changeover, destruction of VX agent and munitions began and was completed in June 2005. All GB- and VX-agent-filled munitions and containers have been successfully destroyed and represent 54.5 percent of the original DCD stockpile. As TOCDF prepares for starting mustard agent disposal operations, ways to deal with the mustard agent in ton containers that have been found to have some mercury contamination have been a subject of active planning. The consequence of this contamination is that processing times will need to be extended and significant system modifications may be required.

Preliminary testing has shown that an unknown number of the more than 6,400 ton containers of approximately 1,800 pounds of mustard agent each contain mercury contamination that could interfere with disposal operations and environmental permit compliance. Some of the containers also appear to have substantial heels of solid and gelled material. Sampling and characterization of the contents of each ton container is required. Under the joint sampling project, DCD workers will deliver bulk containers to specially outfitted and equipped igloos in the Area 10 munitions storage location, where EG&G workers will transfer them into sealed glove-box units, allowing operators to safely open them and remove samples for analysis and characterization by site laboratory personnel. Following sampling EG&G workers will close and return the containers to DCD personnel for segregation and storage according to their contents. Sampling operations will allow individual bulk containers to be categorized for later disposal. The sampling project is now underway and is expected to take two and one-half to three years to complete. Bulk container disposal operations were expected to begin in 2006, first targeting for destruction those sampled containers identified as having little or no mercury contamination and low solids content. Safety and operational risks are inherent due to the unique and largely manual operation for such a large quantity of containers.

The Pine Bluff stockpile contains approximately 12 percent of the original total U.S. stockpile and consists of GB agent in M55 rockets, VX agent in M55 rockets and M23 land mines, and HD and HT mustard agent in ton containers. PBCDF began disposal operations in March 2005. Like TOCDF, Pine Bluff has a unique feature in that the stockpile includes approximately 3,600 ton containers of mustard agent HT, which is a blend of approximately 60 percent HD (sulfur mustard) and 40 percent agent T.⁹ Concern exists about the condition of the HT in the ton containers due to the unique mixture as well as the fact it was manufactured in the early 1940s. Limited sampling conducted in the 1970s indicated that there was a significant heel of sludge and solids in the containers. Because many of the ton containers at DCD came from the Pine Bluff site, it follows that the containers of HT at Pine Bluff should also be tested and the agent condition characterized regarding mercury as well as other contaminants and solids. Planning has been done and initial work begun to sample, analyze, and characterize the contents of

these HT containers. There currently is time to do this sampling and acquire any necessary equipment for processing these containers, while PBCDF continues destroying the remaining GB and VX M55 rockets and M23 landmine stocks. Mustard agent destruction operations are scheduled to begin in September 2008.

The Newport stockpile represents approximately 4 percent of the original stockpile, consisting of bulk VX agent in ton containers. NECDF began neutralization of VX in May 2005 and had neutralized 19 percent of the stored VX as of mid-2006.

The hydrolysate resulting from the VX agent neutralization is being stored on-site pending a decision on the final disposition of the hydrolysate. Disposition or further treatment of the hydrolysate is necessary under Chemical Weapons Convention rules for agent destruction. NECDF continues to face issues regarding the disposition of the hydrolysate through shipment to an approved hazardous waste treatment facility. If off-site disposal is not permitted, an on-site treatment facility would have to be designed and built. This would have a major impact on schedule and costs and would also complicate site closure plans.

For this report the committee did not evaluate the two ACWA program sites at Pueblo Chemical Depot (Colorado) and Blue Grass Army Depot (Kentucky). However, the committee suggests that ACWA government and contractor management personnel review this report in detail for information, issues, and recommendations that have potential applicability to the ACWA program procedures and processes. This is particularly important since the ACWA sites will likely be operating after the other stockpile sites complete destruction operations. There will essentially be no opportunity for hardware overlap at that point, and both of the ACWA plants are one-of-a-kind, first-generation pilot plant facilities. Additionally, a large number (over 700,000) of cartridges and projectiles are stored at the Pueblo site and processing them will require reliable operation of the munitions disassembly machines over a period of several years. Obsolescence of these machines and other equipment must be considered. Other areas include:

• obsolescence management programs at each site and at CMA, and lessons-learned programs;

• handling of the brine streams and secondary waste;

• detection and monitoring, process control systems, and spare parts;

• recruitment, training, and retention of critical skilled and experienced personnel;

• availability of water and other utilities; and

• facility closure.

ASD (AeroSpace and Defence). 2005. ASD-STE100 Simplified Technical English. Brussels, Belgium: AeroSpace and Defence Industries of Europe.

Drury, C.G. 1999. Case Study: error rates and paperwork design. Applied Ergonomics 29(3): 213-216.

Eason, K.D. 1989. Information Technology and Organisational Change. London, U.K.: Taylor and Francis Group.

Rasmussen, J. 1983. Skills, rules, knowledge: signals, signs and symbols and other distinctions in human performance models. IEEE Transactions: Systems, Man, and Cybernetics 13(3): 257-267.

Roberts, K.H. 1990. Some characteristics of one type of high reliability organization. Organization Science 1(2): 160-176.

U.S. Army. 1999. Diminishing Manufacturing Sources and Material Shortages, Army Materiel Command Pamphlet No. 5-23. Alexandria, Va.: Department of the Army Headquarters, United States Army Materiel Command.