Evaluation of the Army's Draft Assessment Criteria to Aid in the Selection of Alternative Technologies for Chemical Demilitarization (1995)

The Stockpile Committee evaluated the April 26, 1995, draft of Assessment Criteria to Aid in Selection of Alternative Technologies for Chemical Demilitarization (U.S. Army, 1995a) on the basis of the four critical factors—process efficacy, process safety, schedule, and cost—discussed in Chapter 3 . The committee based its findings on the following questions:

• Do the Army's draft assessment criteria reflect the factors defined in Chapter 3 of this report?

• Do the Army's draft assessment criteria adequately address the information requirements listed in Chapter 4 ?

• Does the Army's draft document raise other questions with respect to the overall decision process?

Although the Army developed the draft assessment criteria to facilitate the Department of Defense decision on whether to proceed with construction and operation of an alternative technology demonstration facility, it is clear that these criteria will also receive extensive scrutiny by other major stakeholders—community groups, environmental organizations, and regulatory agencies.

The committee's specific findings and its recommendations, both for improving the assessment criteria and for more clearly delineating how they are to be used, are detailed below. The individual findings and applicable recommendations are grouped in three categories: presentation and use, public concerns and input, and schedule factors.

Finding 1. Side-by-side, site-specific comparisons of alternative technologies versus the baseline system are needed to decide whether to implement an alternative technology or the baseline system at a particular bulk storage site.

Although the original intent of both the NRC's earlier recommendations and the Army's program was to narrow the selection to a single alternative technology for use in a pilot program, site-specific differences and related information subsequently developed in the Army research, development, test and evaluation (RDT&E) program may make it desirable to select .different technologies for the agents stored at Aberdeen and Newport. Whether such a choice could be timely and cost effective would be an important consideration.

Recommendation 1a. The basis for evaluation of an alternative technology should be a side-by-side, site-specific comparison of each applicable assessment criterion with the baseline system.

Recommendation 1b. The draft criteria should include all of the assessment criteria needed to evaluate the baseline system versus each potential alternative technology on a site-specific basis.

Recommendation 1c. The assessment criteria document and the Army's technology evaluation should not foreclose the possibility that separate variations of neutralization technologies could be used for each agent type. These variants could lead to consideration of more than one pilot facility, if that is required to demonstrate schedule acceleration or cost effectiveness.

Finding 2. The required information for several of the Army's evaluation factors and assessment criteria cannot be fully developed within the current evaluation and decision schedule. Uncertainty in evaluation of the factors and assessment criteria will result in subjectivity in the decision process.

Recommendation 2a. Essential criteria (go versus no go) should be identified as such.

Recommendation 2b. The absence of specific information at the level of detail needed should be noted as part of the evaluation. Data relative to the cost and schedule required to obtain such information, as well as the programmatic risks associated with uncertainty due to its absence, should be provided. Thus, although the lack of needed information may have a negative effect on the program cost, schedule, and overall uncertainty, it should be considered as a decision uncertainty, not as an absolute negative, if the time and funds required allow timely completion of the program.

Finding 3. The responses to individual assessment criteria, as well as the integration of the responses as part of the decision process, will have both objective and subjective components. The final assessment will reflect the experience and bias of those who evaluate that information.

Recommendation 3. Subjectivity should be recognized and, to the extent possible, minimized by defining how the assessment criteria will be used in the decision-making process and by identifying the assumptions, projections, and uncertainties involved.

Finding 4. The draft assessment criteria will generate a vast amount of information, and there is no indication of how the criteria will be used to develop an integrated evaluation. The evaluation factors and assessment criteria have varying degrees of importance in terms of the overall decision. In addition, there are numerous relationships between criteria and between evaluation factors that must be considered. An integrated evaluation requires assignment of importance and definition of the critical interrelations between and among evaluation factors and criteria.

Recommendation 4a. A methodology for using the assessment criteria and for integrating the information obtained from responses to the criteria should be described in detail. The relationships among evaluation factors and assessment criteria should be identified, and an explanation of their use in the evaluation process should be provided.

Recommendation 4b. The relative importance of each assessment criterion should be specified, at least to the extent of categorizing it as essential, important, or desirable.

Finding 5. The draft assessment criteria document is inconsistent in the level of detail required for evaluation. Furthermore, the specificity required and the information needed to respond to each criterion are not indicated.

Recommendation 5a. The assessment criteria document should specify the level of detail required and types of the information necessary to conduct the subsequent assessment and evaluation activities.

Recommendation 5b. On technical issues, enough specificity should be provided to support a detailed conceptual design for each site.

Recommendation 5c. The assessment criteria that address the impact of federal, state, and local environmental regulations on proposed alternatives must be much more detailed.

Finding 6. Concerns about potential long-term health hazards and environmental burdens from emissions of an agent or its reaction products have been expressed by host communities. Although the Army's assessment criteria elicit information about the potential for release of agents or other hazardous materials, they do not address the assessment of the health risks and environmental burdens that could result from such a release. Host communities have also expressed concern that installation of facilities that are applicable to additional waste types may lead to prolonged risks through the destruction of imported wastes.

Recommendation 6a. Site- and technology-specific preliminary accident and health impact comparisons should be included in the assessment criteria.

Recommendation 6b. The assessment criteria should require the use of system mass balances to specify the amount, physical state, and composition of all system effluents under both normal and

upset conditions. This information should be used to evaluate the potential impacts on air, water, and soil.

Recommendation 6c. The assessment criteria should elicit the variety of waste types for which a destruction process may be applicable. The Army's commitment to decommission a disposal facility at the conclusion of stockpile destruction should be reiterated.

Finding 7. The potential economic impacts of a disposal facility will be both positive and negative and are a major concern to local communities. These communities are a valuable source of knowledge about these concerns.

Recommendation 7. If public benefits are to be considered in the assessment criteria, negative factors, such as the potential for decreased property values and decreased sales of agricultural products, as well as impaired use of local recreational areas, should also be addressed.

Finding 8a. The assessment criteria proposed under the basic category “Public Acceptance” are unlikely to yield information that will be useful to the Army in making a decision regarding the implementation of competing technologies.

Finding 8b. All of the criteria in the draft assessment criteria document appear designed to include input only from the RDT&E program. Relevant stakeholders in the program include, but are not limited to, Citizens Advisory Commissions, regulating agencies, host communities, and legislative bodies with jurisdiction at the specific sites. These groups can be a valuable source of information about program issues.

Finding 8c. Community involvement throughout the Chemical Stockpile Disposal Program is essential to facilitate program implementation and reduce programmatic delays.

Recommendation 8a. The Army should identify relevant stakeholders and should solicit input on program issues from them. The Army should provide both a description of the intended use of the input and feedback on the effect of the input on decision-making.

Recommendation 8b. The Army should continue to improve and expand its efforts to involve the public in the decision process regarding technology implementation in lieu of attempting to measure the public's concerns of risk or acceptance of technology.

Finding 9. The draft assessment criteria assume programmatic schedule dates and completion of stockpile destruction by December 31, 2004, as specified in Public Law 102-484. This statutory date is based on a 10-year period beyond the assumed date on which the Chemical Weapons Convention will become effective. Given the delay in ratification that has already occurred, the date specified in the statute might be extended for months or years. Adjustment of programmatic schedules might allow time for the additional RDT&E needed for an adequate comparison between an alternative technology and the well-developed baseline system.

Recommendation 9. The Army should consider revising the intermediate programmatic schedule dates (including the decision about pilot-scale demonstration) and the targeted completion of stockpile destruction to reflect the most current projection of treaty obligations, provided that the changes do not adversely affect schedule-related risks and costs.

Finding 10. The draft criteria do not recognize the possibility that an alternative technology may accelerate the schedule for agent destruction and thereby reduce schedule-related cost and risk. Acceleration may occur either by advances in technology (such as in situ neutralization of VX) or by avoidance of regulatory delays.

Recommendation 10. The Army should evaluate the effect of an alternative technology on site-specific schedule, risk, and cost for use in comparison with the baseline system.

Finding 11. Community resistance, permitting issues, or other factors may delay implementation of either the baseline system or an alternative technology during the regulatory process. Effective community involvement can facilitate implementation and reduce

delays. The extent of possible delays during the permit-acquisition process can be assessed on the basis of experience with other similar facilities and by consultation with state and federal regulatory authorities. Potential delays can impact not only schedule but also cost and risk.

Recommendation 11a. The Army should assess the range of potential delays associated with implementing a baseline system or any alternative technology.

Recommendation 11b. The Army should constructively engage both the host community and the appropriate regulatory agencies in key decisions and the development of project milestones to minimize potential delays.

Recommendation 11c. The impact of potential delays on site-specific schedule, risk, and cost should be evaluated for use in the comparison of the baseline system with any alternative technology.