3
Important Parameters for Successful Closure
BACKGROUND
The committee provided a series of key parameters for overall management of the current and upcoming chemical agent disposal facility closures in an earlier interim letter report (Appendix A). As an initial basis for developing these parameters, the committee considered the lessons learned by the U.S. Army Chemical Materials Agency (CMA) from closure of the Johnston Atoll Chemical Agent Disposal System (JACADS), which was the first full-scale incineration-based disposal facility. The Aberdeen Chemical Agent Disposal Facility (ABCDF), which was the first neutralization-based disposal facility, and the Newport Chemical Agent Disposal Facility (NECDF), another neutralization-based facility, were also considered.
The committee has modified the original list of parameters given in the interim report and developed a structured approach that presents the parameters and the associated metrics more effectively. Using these in conjunction with the experiences gained from previous closures, along with earlier programmatic plans for closure of TOCDF and CAMDS and the knowledge and experience of the committee members, the committee developed recommendations that it believes can improve the current and future closures of disposal facility sites. It is imperative to track the critical criteria, including the agency and regulatory requirements, necessary to successfully managing a closure project.
The committee continues to emphasize that the foremost goal of the parameters and metrics is to promote a safe and successful program for facility closure. To accomplish this, the effective use of well-developed management reports can serve as a feedback mechanism for the continuous improvement of closure activities and, where necessary, can serve as a mechanism for stopping work until appropriate corrective actions can be made. The role of a management reporting process is best described by the plan-do-check-act cycle used by many organizations and described at the American Society for Quality website.1 As originally identified in the interim letter report, metrics are of two kinds: leading metrics, which help to predict performance, and lagging metrics, which indicate the actual performance. While the committee considers the listed parameters and corresponding metrics to be important, it notes that they need not be considered all-inclusive.
KEY PARAMETERS
The parameters in Table 3-1 and Table 3-2 are shown along with associated metrics as a starting point for the management of closure activities at the program and project levels. It is imperative that different parameters be tracked at the appropriate management and activity levels. This section develops a parameter framework for both the program and the project level that is defined later. The list of parameters has been revised from the original list in the interim letter report and includes
1 |
A discussion of the plan-do-check-act cycle is found at http://www.asq.org/learn-about-quality/project-planning-tools/overview/pdca-cycle.html. |
TABLE 3-1 Program-Level Parameters and Metrics
Metric |
Definition |
Type |
Safety, Health, and Security |
|
|
Near-miss |
Number of unsafe conditions |
Leading |
Incident investigations completed within 30 days |
Percent complete |
Leading |
Cross-training for workforce/supervisors |
Percent complete |
Leading |
Random drug testing |
Percent complete |
Leading |
Occluded space survey process |
Yes/No |
Leading |
Unventilated monitoring test process |
Yes/No |
Leading |
Numbers of recordable injuries (RIs) |
Number |
Lagging |
Number of lost workday cases |
Number |
Lagging |
Days away from work due to workplace incident/injury |
Number |
Lagging |
Fatalities (all causes) |
Number |
Lagging |
Training and Development |
|
|
Continuing education |
Hours/year |
Leading |
Communication |
|
|
Safety culture survey |
1-5 scale |
Leading |
Leadership communication sessions |
Number/year |
Leading |
Cost |
|
|
Federal requirements |
Standard metrics |
Lagging |
Program cost objectives |
Program cost targets metrics |
Lagging |
Schedule |
|
|
Schedule status |
Percent complete |
Lagging |
Earned value |
Compare progress to expenditures |
Lagging |
Environmental Compliance |
|
|
Establish facility future use for all sites (RCRA and BRAC) |
Yes/No |
Leading |
Establish RCRA closure performance standards for all sites |
Yes/No |
Leading |
Develop RCRA permit closure plan for all sites and coordinate with state regulators |
Yes/No |
Leading |
Develop supplemental closure plans at each site and coordinate with state regulators |
Yes/No |
Leading |
Develop plan in coordination with state regulators at each site to minimize waste/maximize reuse and recycling |
Yes/No |
Leading |
Develop plan in coordination with state regulators at each site for disposition of waste |
Yes/No |
Leading |
Management |
|
|
Develop roles and responsibilities for key personnel |
Yes/No |
Leading |
Develop quality assurance management program including design control, training, document control, revision control, performance review, and internal audits for a comprehensive quality assurance program |
Yes/No |
Leading |
TABLE 3-2 Project-Level Parameters and Metrics for the Chemical Agent Disposal Facilities
Metric |
Definition |
Type |
Safety, Health and Security |
|
|
Near-miss |
Number of unsafe conditions |
Leading |
Incident investigations completed within 30 days |
Percent complete |
Leading |
Cross-training for workforce/supervisors |
Percent complete |
Leading |
Metric |
Definition |
Type |
Appropriate personal protective equipment for all tasks |
Yes/No |
Leading |
Closure of open safety items in a timely manner |
Percent complete |
Leading |
First aid cases |
Number |
|
Recordable injuries and exposuresa |
Number |
Lagging |
Lost-time injuriesa |
Number |
Lagging |
Fatalities (all causes) |
Number |
Lagging |
Hazardous assessment plan |
Yes/No |
Leading |
Occluded space survey #1 |
Yes/No |
Leading |
Occluded space survey #2 |
Number of failures |
Lagging |
Unventilated monitoring test |
Number of failures |
Lagging |
Maintenance |
|
|
Planning and scheduling all maintenance |
Yes/No |
Leading |
Appropriate maintenance for deconstruction equipment |
Percent complete |
Leading |
Appropriate calibration and checking of monitoring equipment |
Yes/No |
Leading |
Preventive/predictive maintenance program for key equipment |
Percent complete |
Leading |
Monitoring/audit of maintenance plan |
Percent complete |
Leading |
Training and Development |
|
|
Cross-training for critical operation deconstruction positions |
Yes/No |
Leading |
Proper certification for employees and contractors |
Yes/No |
Leading |
Hazardous waste certification for appropriate workers |
Yes/No |
Leading |
Workforce training on the facility and on non-normal situations |
Yes/No |
Leading |
Workforce training for deconstruction personnel |
Yes/No |
Leading |
Communication |
|
|
Schedule(s) communications with local citizens advisory commission |
Yes/No |
Leading |
Proactive two-way communications with neighboring communities |
Yes/No |
Leading |
Proactive and frequent communications by senior site personnel with the state regulatory personnel |
Yes/No |
Leading |
Regularly scheduled two-way communications with workforce |
Yes/No |
Leading |
Cost |
|
|
Periodic cost spending plan |
Yes/No |
Leading |
Track costs to spending plan |
Yes/No |
Lagging |
Schedule |
|
|
Develop project schedule milestones |
Yes/No |
Leading |
Track engineering changes |
Percent complete |
Lagging |
Track project schedule milestones |
Percent complete |
Lagging |
Environmental Compliance |
|
|
Establish closure performance standards for closure wasteb |
Yes/No |
Leading |
Closure plans included in RCRA permit, supplemented by more detailed plansb |
Yes/No |
Leading |
Closure addressed in other applicable permitsb |
Yes/No |
Leading |
Approval of waste analysis plan and characterization protocolsb |
Yes/No |
Leading |
Approval of monitoring plans for other appropriate media |
Yes/No |
Leading |
Establish where generator knowledge can be usedb |
Yes/No |
Leading |
Develop plans for optimizing reuse and recycling |
Yes/No |
|
Develop protocols for segregation of hazardous and nonhazardous waste materialsb |
Yes/No |
Leading |
Monitor compliance with RCRA permit closure plan |
Monitor percent compliance and notices of violation |
Lagging |
Monitor compliance with other permits |
Monitor percent compliance and notices of violation |
Lagging |
|
Yes/No |
Leading |
Management |
|
|
Develop effective records retention process |
Yes/No |
Leading |
Regularly review lessons learned database for effectiveness |
Yes/No |
Leading |
Review material management process including reuse/recycling |
Yes/No |
Leading |
aOSHA requirement. bPer state approval. |
additional parameters from the list developed in the NRC report Evaluation of Safety and Environmental Metrics for Potential Application at Chemical Agent Disposal Facilities (NRC, 2009). That report served as an excellent framework for operating disposal facilities as well as a starting point for closure planning. Appendix B of the present report includes the complete list developed in that study. This current report expands the list with additional critical management parameters required for closure. The overriding purpose of the parameters and metrics is to instill a culture of continuous improvement for all aspects of closure and deconstruction, including safety, regulatory, and program and project management. The list developed by the committee is more complete than the one previously developed in the interim report; nevertheless, it still need not be considered all-inclusive. The following findings and recommendations are provided by the committee for consideration by the Army.
Finding 3-1. A formal and structured system of parameters and metrics, including leading as well as lagging metrics, provides important guidance for planning, organizing, and implementing efficient closure of baseline incineration chemical agent disposal facilities.
Recommendation 3-1. The Army should consider the parameters and metrics presented in this report as it plans and executes the closure of the baseline chemical agent disposal facilities.
Finding 3-2. Tracking and reporting parameters and metrics will facilitate the safe and successful management of the closure of the Army’s baseline incineration chemical agent disposal facilities.
Recommendation 3-2a. At a minimum, the Army should track parameters and metrics used for disposal facility closure at two levels: the program level and the project level. Thereafter, it should determine whether additional parameters and metrics are required.
Recommendation 3-2b. The Army should ensure that appropriate and timely management reports are developed that enable tracking results for parameters and metrics to be used to make management decisions and take necessary actions.
Following is a summary of each of the categories of parameters.
Safety, Health, and Security
The CMA programmatic staff, along with the site management personnel and workers at chemical agent disposal facilities, continues to promote an operational culture focused on safety, health, and security. The performance record of the various sites during the disposal operations phase demonstrates the management focus and employee awareness of the safety programs. Table 3-3 summarizes site injury rates for the baseline chemical disposal sites through the first quarter of 2010. These data indicate the performance of operations management in maintaining a safe work environment. However, a major concern related to safety not present in the day-to-day operations is the new and non-repetitive activities during closure, which can result in unexpected situations that present safety issues. An effective safety program for closure requires that both leading and lagging metrics be tracked, documented, reported in a timely manner, and communicated as part of the process. As reported in the interim letter report, satisfactory results concerning safety, health, and security are supported by the establishment of systematic data collection, site observations, incident reporting, and an effective investigation process. This requires the involvement of the entire organization from the programmatic and site management personnel, project planners, operational personnel, contractors, and all workers involved in the execution of the closure activities. As the site transitions from operations to closure, the potential for unforeseen/unexpected safety occurrences may increase in number and complexity, requiring close surveillance and awareness on the part of not only the closure teams but also the existing operations teams.
Maintenance
The successful closure of the baseline incineration disposal facilities requires correct execution of all maintenance work activities on equipment that will impact the closure. Execution includes effective planning and timely scheduling of all maintenance work on key equipment used for disposal operations, closure, and monitoring. In the deconstruction business, many safety and regulatory problems can be prevented through well-managed maintenance work processes. Additionally, negative impacts on schedule can be minimized when critical equipment is properly maintained. Benchmark studies have shown that results
TABLE 3-3 Chemical Demilitarization Site Recordable Injury Rates as of March 31, 2010
Facility |
Employee Hours Worked Since Last LWC (hr) |
Current 12-Month RIR |
Highest 1-Month RIRa |
Lowest 1-Month RIRb |
Highest 12-Month RIRc |
Lowest 12-Month RIRd |
ANCDF |
4.5 million (988 days) |
0.69 |
5.18 |
0.0 |
1.75 |
0.27 |
NECDF |
1.9 million (914 days) |
0.33 |
4.45 |
0.0 |
1.95 |
0.18 |
PBCDF |
3.0 million (770 days) |
0.13 |
3.32 |
0.0 |
1.17 |
0.00 |
TOCDF |
8.9 million (1,624 days) |
0.76 |
14.54e/11.26 |
0.0 |
4.82 |
0.46 |
UMCDF |
6.2 million (1,369 days) |
1.06 |
3.83 |
0.0 |
2.25 |
0.34 |
NOTE: LWC, lost workday case; RIR, recordable injury rate. aHighest 1-Month RIR in entire facility operational history. bLowest 1-Month RIR in entire facility operational history. cHighest 12-Month RIR in entire facility operational history. dLowest 12-Month RIR in entire facility operational history. eThe higher number includes 11 cases of food poisoning that occurred at a safety celebration. The lower number excludes these 11 cases. SOURCE: Cheryl Maggio, Deputy Project Manager Chemical Stockpile Elimination, CMA, May 28, 2010. |
are optimized by having 80-85 percent of all required maintenance activities planned and scheduled.2
Training and Development
The basis for successful completion of closure of baseline chemical agent disposal facilities is the active involvement and proper preparation of each site’s entire workforce. Strategic elements identified by the committee that should be tracked and reported include the employee selection, training, and development process.
The leading and lagging metrics provided in Table 3-1 for this parameter represent critical items that, when successfully addressed, will result in a more qualified workforce and a higher probability of success. For example, the technical aspects, complexity, and unique safety and environmental requirements of a facility closure mandate that the workforce be properly selected and prepared through education and training. That training drives performance results for new activities, especially when new personnel are involved, is a well-established fact, and closure will constitute a new activity at baseline chemical agent disposal facilities. The committee continues to believe that concerted efforts should be made to train the deconstruction workforce on hazards and awareness pertinent to the site and facility situations. Cross-training between personnel familiar with operations and individuals responsible for closure constitutes a critical interface for promoting safe and successful outcomes for closure activities. Training requirements for closure operations are very different from operational activities and therefore must be closely monitored. An example of the importance of workforce training is the criticality of the unventilated monitoring process for the closure facilities and the need to develop solid training materials (as discussed in detail in Chapter 6).
Communications
The committee believes that a strong, positive communication strategy engages a full range of stakeholders involved in the execution of the closure programs. The committee appreciates the strong, positive safety culture that has been developed at TOCDF, CAMDS, and other CMA sites, and it believes that that culture will continue into the closure phase as long as the frequent formal and informal sharing of information and ongoing dialogue continue. Additionally, the closure management must actively lead and support commu-
2 |
A discussion of benchmark studies that establish optimal levels of maintenance is available online at http://www.reliabilityweb.com/excerpts/excerpts/Maintenance_Benchmarks.pdf. Last accessed July 1, 2010. |
nications with all key stakeholders, including federal and state authorities and local community groups. Good communications build trust and provide more opportunities to understand the changing nature of risk. The committee has selected the metrics identified as critical to ensuring a proactive communication strategy.
Cost
Program cost objectives comprise a key strategic parameter for successful completion of the site closure as indicated in the committee’s interim report for successful completion of the site closure. The committee was neither tasked nor provided with the financial data necessary to assess how the programs are progressing in terms of cost management. The committee believes strongly that the various levels of management, including CMA at the program level and the prime and subcontractors at the project level, should be able both to forecast anticipated costs and to effectively understand, manage, and explain all expenditures during the implementation of program and project work and have a robust cost control system. It is imperative that all federal financial requirements be met for the projects to be successfully undertaken in the closure and closure process.
Schedule
Based on the complexity and cost of the satisfactory completion of the chemical agent disposal facilities site closures, the committee believes that the leadership needs to develop and adhere to a comprehensive schedule. Additionally, how management addresses programs and project changes and delays within the closure process are critical to timely and cost-effective completion and are therefore identified as a parameter to be tracked. Safety becomes a critical item when tracking schedule changes and when pressure develops to complete an activity on time.
Environmental Compliance
Environmental compliance issues associated with the closure of the baseline chemical agent disposal facilities are a priority for management. The tracking of performance in this regard will be critical. The CMA closure team and the teams at each facility have evidenced a strong understanding of regulatory compliance and monitoring requirements. Key attributes for successful closure include obtaining regulatory agreement with the closure plans with the state authorities on a timely basis. This requires close coordination with the state regulatory community to obtain early agreement on the anticipated future use for the facility and the related closure performance standards that must be achieved. State environmental regulatory authorities have been engaged throughout the process in the review of plans and specifications for the closure.
Just as important as the closure performance standards are reaching agreement with the state regulatory authorities on the manner in which performance standards will be achieved. Throughout the closure process, compliance monitoring will be required. The metrics provided herein are designed to track timely selection of future use, agreement on the performance standards, and the manner in which the performance standards will be achieved, leading eventually to final facility closure.
The committee’s metrics for environmental compliance are based in part on the recommendations established within the NRC report Evaluation of Safety and Environmental Metrics for Potential Application at Chemical Agent Disposal Facilities (NRC, 2009).
Management
The responsibility of management is to set the tone and direct the site effort for all work activities. The metrics listed for this parameter offer ways to consider how management measures its activities and their effectiveness. Management is responsible for developing and structuring parameters and metrics for each segment of the workforce. Not all of the workforce can or should receive and react to all of the information resulting from the metrics discussed in this chapter. However, each parameter and each metric must be reviewed for its intended audience.
Closure program quality is a key strategic element for successful program completion. Quality elements such as adequate and appropriate analytical capabilities, inventory and material management, records retention, and lessons learned comprise critical management items that can significantly affect the efficiency of the overall closure effort. Careful materials management is a key to successful facilities closure and management control while quality protocols for segregation of generated hazardous and nonhazardous materials must be implemented. This will require proper identification
and inventory control of these materials. The committee has identified a set of key metrics to be tracked for successful closure.
GROUPINGS OF PARAMETERS
The large number of parameters and associated metrics to be tracked and reported over the course of the closure of chemical agent disposal facilities will result in the generation of significant amounts of information. In managing the closure operations, management should divide the parameters and metrics into two levels: the program level and the project level. The committee did not attempt to address task-level activities. However, the Army and its closure contractors should do so to track the key parameters identified and the corresponding metrics.
Program Level
Program management is the process of managing one or more related projects, often with the intention of improving an organization’s performance. The management of the site and the CMA is concerned with the aggregate result or end use. Typically, a program approach is broken down into projects that reflect the overall objective. The emphasis for the program management staff involves coordination and prioritization of resources across projects as well as supervision of links between the projects and the overall costs and risks of the project. Closure and deconstruction activities require the possession or acquisition of an understanding of the unique aspects of these operations, and therefore the parameters and metrics must reflect such an understanding. At the program level, the intent is to focus more on the leading metrics in the hopes of anticipating future issues.
Project Level
The key difference between a program and a project is the finite nature of the project. A project is designed to deliver an output or deliverable, and its success will be determined in terms of delivering the right output at the right time and at the right cost. Project-level metrics will primarily be the responsibility of the contractors, both prime and subcontractors, to track the results and manage accordingly. The project-level metrics will be more heavily weighted to lagging metrics and focused on continuous improvement.
REFERENCE
NRC (National Research Council). 2009. Evaluation of Safety and Environmental Metrics for Potential Application at Chemical Agent Disposal Facilities. Washington, D.C.: The National Academies Press.