Evaluation of the Army's Draft Assessment Criteria to Aid in the Selection of Alternative Technologies for Chemical Demilitarization (1995)

The committee evaluated the Army's draft document, Assessment Criteria to Aid in Selection of Alternative Technologies for Chemical Demilitarization (U.S. Army, 1995a), on the basis of the following questions:

1. Does the draft criteria document address, and give proper emphasis to, the factors that the committee considers central to the comparison between the alternative technologies and the baseline system?

2. Will use of the draft assessment criteria generate the data that the committee believes are necessary for making a decision on whether to proceed with pilot-scale demonstration for one of the alternative technologies?

3. Does the draft assessment criteria document raise other issues with respect to the decision process regarding an alternative technology?

This chapter presents a description of the Army's. draft assessment criteria document, then discusses the committee 's evaluation of the assessment criteria with regard to each of these questions.

The draft assessment criteria document consists of three substantive elements: a chapter describing how the assessment criteria were developed, a chapter summarizing the 10 basic criteria categories developed by the Army, and an appendix setting forth the assessment criteria formulated as 169 questions that the Army intends to use to decide whether to implement an alternative technology. The Army's proposed assessment criteria are reproduced as Appendix C of this report.

Section 2 of the draft assessment criteria document describes the development and implementation of the criteria and identifies the following objectives for such development:

• Establish evaluation factors for characterizing and comparing bulk agent demilitarization alternatives.

• Communicate the basis of the assessment process in support of public discussion of this process.

• Provide information in an organized format to support the selection process for neutralization, neutralization/biodegradation, or program cancellation.

• Provide guidance to the research, development, test and evaluation (RDT&E) program with respect to technical information that will meet the programmatic requirements.

• Form a starting point for design criteria for a facility that implements the selected technology for chemical agent disposal.

The document then states that the goal for criteria development is to ensure that the assessment of candidate technologies, including the baseline system, is systematic, objective, and comprehensive. The document also states that it is important that the assessment criteria be “discrete and unambiguous, so as to minimize the potential for bias in the assessment process.” To accomplish this goal, the assessment criteria are formulated as questions that have the following characteristics:

• address only one specific subject or parameter;

• call for the identification of a specific item or component;

• call for a “yes” or “no” answer when appropriate;

• require a numerical answer when a parameter is involved; and

• include a specification of the units for numerical answers.

The document next explains how the assessment criteria were developed, beginning with the identification of primary programmatic requirements. These programmatic requirements are defined as “public law, international agreements/treaties, regulations, and the related NRC report and Army response” (U.S. Army, 1995a, p. 1-4) and, alternatively, as “public policy, chemical treaty, statutes, and public perception” (U.S. Army, 1995a, p. 2-4, Figure 2-1).

The document states that these programmatic requirements were divided into four areas of consideration: regulations, public awareness, programmatic, and engineering (U.S. Army, 1995a, p. 3-2, Figure 3-1). “Basic categories” were then established within each of these areas based on the type of expertise needed to identify the evaluation factors in each area and to perform the assessment.

The document identifies 55 evaluation factors deriving from the basic categories. These evaluation factors are elaborated by division into 169 assessment criteria, which are formulated as questions. Appendix B of the draft assessment criteria document presents the evaluation factors and the 169 questions that comprise the assessment criteria for the alternative technologies under consideration.

Section 3 of the draft assessment criteria document presents a discussion of each of the 10 basic categories and briefly describes the evaluation factors for each category. The basic categories and examples of the issues that the Army intends to address in each category are discussed below.

The Army will consider whether the alternative technology is significantly safer than the baseline system, is equally or more cost effective than the baseline system, and is suitable for completing stockpile destruction by December 31, 2004. The Army draft assessment criteria document cites the National Research Council's 1994 report as indicating that no alternative can meet the requirements imposed by PL 102-484 (NRC, 1994b). This citation requires clarification. In Finding 5 of the report, the NRC stated that it “did not foresee that any alternative agent-destruction technology will substantially reduce the total agent-processing risk.” The committee believes that, although the risk from agent processing is a significant component of the overall risk to the public and to stockpile workers, storage risk is generally greater. An alternative technology that permits early detoxification of agent at a stockpile site may be significantly safer than the baseline system and may meet other Public Law requirements by reducing storage risk and costs.

The Army will consider issues such as whether the alternative technology can satisfy treaty requirements and whether it can be permitted under and can comply with the federal, state, and local pollution control laws and whether its implementation raises issues under the National Environmental Policy Act.

The Army will consider issues such as waste minimization, monitoring requirements, and the risk of the release of hazardous materials other than agent.

The Army will consider a number of worker-safety issues. The Army plans to address issues related to the possibility of agent release outside of engineering controls, the largest possible release of agent, the potential for severe facility damage, the ability to mitigate process upsets, and the potential for worker exposure to agent as the result of such incidents.

The Army intends to consider community involvement issues, public concerns of risk and environmental impact, and potential community benefits.

The Army will consider issues such as the potential for off-site agent release caused by equipment failure, human error, or external events. The Army also states that toxicological studies will be performed to assess long-term health effects of plant operation.

The Army intends to consider issues such as the time required for (a) testing, construction, and environmental permitting; (b) acquiring, installing, and preparing the facility for operation; (c) processing the agent using the technology; and (d) decommissioning the facility.

The Army intends to consider such issues as the cost of research and development; construction, procurement, operation, and closure; and the impact of scheduling delays associated with implementing a technology other than the baseline system.

The Army intends to consider the results of laboratory- and bench-scale testing of the process, agent quality, materials to be used in the equipment, and decontamination procedures for ton containers.

The Army will consider issues such as the availability of necessary materials and equipment, the complexity and flexibility of the process, and the extent to which agent materials will have to be prepared prior to introduction into the destruction system. The effectiveness of conducting agent destruction, process monitoring, and control of residue and effluent will be considered, as will the requirements for utility and power and methods for decontaminating metal parts.

The remainder of this chapter contains the committee's evaluation of the Army's draft assessment criteria in the context of the three genetic questions listed in the chapter's introductory paragraph.

The committee believes that there are four primary factors, identified in Chapter 3 , critical for comparing the baseline system (i.e., incineration) to any alternative technology. Those factors are process efficacy, process safety, schedule, and cost. Although the committee does not believe that the Army must organize its analysis in this same fashion, it does believe that any assessment process that does not thoroughly address these four factors would be inadequate to conduct the decision-making process. Thus, this section considers whether the assessment criteria proposed by the Army adequately address these factors.

It is important that the assessment of an alternative process be carried out as a comparison with the baseline system because the Public Law criteria underlying consideration of an alternative technology are based on such a comparison. Public Law 102-484 directs the Army to utilize an alternative disposal process at a site if it “is significantly safer and equally or more cost effective than the baseline disassembly and incineration process.” The committee has interpreted “significantly safer” to mean reduction of total integrated risk to the public and to workers at a specific site. Thus, a disposal technology should be selected if it accelerates agent destruction and thereby reduces storage risk and cost—and if it does not increase risks during destruction.

As discussed in Chapter 3 , evaluating process efficacy involves finding out whether the process can destroy agent and whether it can be controlled, function reliably, and meet applicable treaty and regulatory requirements. The following is a discussion of each of the subfactors for process efficacy that were identified by the committee and the ways in which the Army' s draft assessment criteria address them.

The considerations identified by the committee with respect to detoxification of agent are addressed by the Army's criteria. Army assessment criteria 3.2.2.1, 3.2.4.1, 3.2.5.1, 3.2.5.2, and 3.2.5.3; 3.2.6.1, 3.2.6.2, and 3.2.6.3; 3.5.1.1 and 3.5.1.2; and 3.10.7.1, 3.10.7.2, and 3.10.7.3 address issues related to detoxification of agent. (See Appendix C of this report). The criteria address not only the question “Does this technology achieve irreversible destruction?” but also whether process emissions comply with environmental regulations as discussed subsequently.

The committee defined the key treaty requirement as the reaction or destruction of primary agents so that the end product is not a “scheduled” compound, which can readily be reconverted to agent. Although the degree of destruction required is subject to negotiation, strict standards of accountability and verification are set forth in the Chemical Weapons Convention. These issues are explicitly addressed by the Army in the assessment criteria under evaluation factor 3.2.1.

The committee stated that the agent-destruction process must comply with applicable state and federal regulatory requirements. These issues are addressed by the same assessment criteria as those noted in the discussion of detoxification of agent. The committee notes that other regulatory compliance issues should be addressed. For example, assessment criteria related to management of nonprocess wastes, such as decontamination fluids, wastes from analytical procedures, and entry suits, should be included. Some of these issues do not appear to be addressed explicitly in the Army's assessment criteria. The committee also believes that a listing of all applicable regulatory requirements should be compiled. The assessment criteria assembled under evaluation factors 3.2.2, 3.2.3, 3.2.4, 3.2.5, and 3.2.6 are a good start in this direction. Issues related to Occupational Safety and Health Administration compliance are not addressed specifically, although many are covered in the worker-safety category.

Some of the assessment criteria have many subquestions embedded in them. For example, assessment criterion 3.2.2.1 (“Can the operating standards of 40 CFR 264 and permit requirements of 40 CFR 270 be met?”) generates many other specific questions that need to be answered. Some of these questions may be very important for determining whether a technology is viable and for identifying research needs.

The committee believes that identifying the nature of the process residuals, the regulatory limitations applicable to such process residuals, and the environmental risks posed by such process residuals is very important to determine the environmental acceptability and the overall risk associated with any destruction process. Although several of the assessment criteria (e.g., 3.2.2.4 and 3.2.6.2) and evaluation factor 3.3.1 suggest that the Army will examine such issues, some should be treated more extensively. For example, the physical state and chemical composition of the process residuals must be characterized to determine whether the requirements of the Resource Conservation and Recovery Act, the Clean Air Act, the Clean Water Act, and state and local legal requirements can be met. The treatment of this issue under evaluation factor 3.3.1 should be expanded to include air and water discharges, as suggested in the text of the assessment criteria document. Appropriate criteria should be developed to identify the risks associated with these process residuals and their ultimate disposal methods.

The committee's concerns regarding the impact of agent purity and composition are addressed by Army assessment criteria 3.9.3.1 and 3.9.3.2, as well as criteria 3.10.5.1 through 3.10.5.4. Other assessment criteria (3.10.1.1 through 3.10.3.1) are aimed at ensuring process reliability and “robustness” using standard equipment that has demonstrated its reliability in commercial service. Avoidance of “exotic materials” to resist harsh thermal or corrosive conditions, such as those in the liquid incinerator train of the baseline system, can also enhance reliability. The committee also believes that, in the context of improving public understanding of the text supporting the assessment criteria, it would be

helpful to avoid terms like “exotic materials.” These terms have specific meanings for a technical audience, but the general public may not understand them.

The Army's assessment criteria address whether current monitoring technology can meet process control requirements (3.10.8.1) and characterize the effluents of the process (3.3.4.1 through 3.3.4.3). These questions are phrased in terms of whether proven monitoring technologies are available. The criteria should address the possibility that fairy straightforward modifications of existing reliable technology could be used. In addition, the committee believes that sampling procedures, response times, and required detection limits need to be addressed by the assessment criteria. These issues, although they may be implicit in the Army's stated criteria, are not explicitly discussed.

The Army specifically addresses the availability of the required natural resources in assessment criteria 3.10.11.1, 3.10.11.2, 3.10.11.3, and 3.10.11.4. However, there is no indication that the Army's assessment of these issues will be site-specific, that is, whether the assessment will be tailored to the limitations of the proposed implementation site.

The committee believes that the decision process for implementing an alternative technology requires a determination of whether an appropriate demonstration site can be chosen. This issue is partially addressed by Army assessment criteria 3.5.1.3 and 3.5.1.4.

The Army does not specifically address the applicability of the proposed technology to other wastes. The potential for treatment of imported wastes (e.g., industrial wastes, other stockpile materials, or nonstockpile materials) is a specific concern of host communities. The applicability of the proposed technologies to foreign chemical weapons stockpile may be viewed as beneficial.

The concept of process safety, as described by the committee, encompasses a broad range of issues. These include on-site safety and health risks; community safety, health, and environmental risks; and storage risks. The Army's draft criteria examine many aspects of the safety issue—from the very broad question of whether the alternative technology is significantly safer than the baseline system (3.1.1.1) to very specific questions about the potential for worker exposure and how it is measured. Reviewing the Army's discussion of safety issues on the basis of the committee's subfactor groupings, it appears that agent-related hazards are addressed adequately but nonagent hazard criteria could be expressed more clearly.

The committee believes that an evaluation of on-site safety and health risks should center around four main issues: catastrophic failure and agent release, worker exposure to agent, worker exposure to nonagents, and risk from hazards associated with process reactants and conditions.

The Army extensively addresses the first of these issues, risk of catastrophic failure and agent release. Evaluation factor 3.4.1 includes 10 assessment criteria on this issue. The Army plans to evaluate (a) the extent of possible agent release ( 3.4.2.1); (b) the mechanisms that could lead to catastrophic failure or an uncontrolled chemical reaction (3.4.3.1 and 3.4.3.2); and (c) the ability to mitigate process upsets (3.4.4.1 and 3.4.4.2).

The second aspect of plant safety and health risks identified by the committee is the risk of exposing plant workers to agent. Again, the Army has addressed this factor quite extensively. There are 11 assessment criteria related to worker exposure to agent under evaluation factor 3.4.5.

The committee's third subfactor in this area, the risk of worker nonagent exposure, is addressed by evaluation factors 3.3.2 and 3.3.3 according to the draft criteria document text, although it is not clearly expressed in the derived assessment criteria. In addition, assessment criterion 3.4.7.1, which inquires into

potential long-term effects for plant workers, touches on the subject. However, this criterion does not clarify whether such long-term effects are being assessed for agent, nonagent, or both.

The final area identified by the committee is risks from hazards associated with process reactants and conditions. Although the Army 's assessment criteria do not discuss the subject in these terms, this area seems to be addressed by the assessment criteria under evaluation factors 3.3.3 and 3.4.5. In addition to addressing issues identified by the committee, the Army developed four assessment criteria that address (a) the expected number of worker injuries during construction, processing, and facility decommissioning; and (b) the expected level of exposure to industrial hygiene hazards.

The committee believes that a number of issues should be addressed within this rubric. The first is the risk to the community from acute exposure to agent or process residuals. This concept is not directly addressed by the Army's assessment criteria. Some of the information necessary to address the issue may be developed in response to assessment criterion 3.6.1.1, which considers whether the facility is designed to contain escaping materials. The assessment criteria grouped under evaluation factors 3.4.1, 3.4.2, and 3.4.3 also raise numerous questions regarding the possibility of agent release outside of engineering controls. Although these assessment criteria are designed to assess the possibility that agent may escape the plant boundaries, they do not focus on the level of risk posed by such events. Similarly, they do not look at such issues for process residuals.

The committee believes that consideration of community safety includes the assessment of risks from low-level exposure to process emissions and discharges. These risks include potential for both carcinogenic and noncarcinogenic health effects. The Army's assessment criteria do not address this issue specifically; however, assessment criterion 3.6.2.1, which calls for an assessment of potential long-term health effects associated with the alternative technology, may embrace this issue. The intent of assessment criterion 3.6.2.1 should be clarified.

The committee believes that the assessment criteria should address potential impacts on the surrounding environment from the implementation of the proposed technologies. Clear presentation of the quantities and disposition of all process emissions and residuals per ton of agent treated and the cumulative quantities for each site should be specified. The estimated quantities should allow for process idle periods. This issue does not appear to be addressed by the Army' s assessment criteria.

Finally, the committee believes that special considerations with respect to emergency preparedness or emergency response should be addressed. This topic does not appear to be addressed in the draft assessment criteria.

As stated in its earlier report (NRC, 1994b), the committee believes that the risks associated with storage prior to completion of agent destruction are a critical element in deciding whether implementing an alternative technology reduces overall risk to the public. This issue is not specifically addressed by the Army's assessment criteria. The Army should consider this issue in answering the question of whether the alternative technology is significantly safer than the baseline system (3.1.1.1). Within the basic category, “Schedule,” there are 18 detailed assessment criteria that question whether the implementation of an alternative technology will increase storage time. However, the criteria do not focus on whether any delay occasioned by implementing an alternative technology would result in a significant change in storage risk or whether any change would be offset by other factors associated with the alternative technology. In addition, the assessment criteria do not address the possibility of significant reduction of storage risk by early in situ detoxification of VX stored in bulk containers, as discussed in Chapter 2 of this report.

The committee believes that the impact of technology selection on program schedule, although impossible to predict accurately, should be estimated based on the best possible information. Allowances should be made to address the effect of scheduling delays due to permitting and community opposition to the baseline system. As noted above, the Army included 18 assessment criteria that attempt to predict the scheduling impacts of implementing an alternative technology. These criteria do not explicitly recognize either the possibility that an

alternative technology might accelerate chemical agent destruction or the possibility of permit delays or delays occasioned by community opposition to one or another technology. The Army included 10 assessment criteria within the basic category “Public Acceptance”; however, none of these directly addresses the effect of public acceptance or opposition on permitting or on implementation schedules for a specific technology.

The committee believes there are four basic areas of process cost that should be examined as part of the evaluation process: extended stockpile storage interval, process development, process implementation and operation, and plant disassembly. The Army seems to address these issues thoroughly, as discussed below.

The committee believes the cost associated with maintaining the stockpile until destruction is completed should be included in the Army's evaluation. Assessment criteria 3.8.7.1 through 3.8.7.3 directly address this issue. However, the assessment criteria do not address either the possibility that implementation of the baseline system may be delayed because of permitting issues or opposition by a hostile community or the possibility that an alternative technology may reduce storage duration and cost.

The costs associated with development of the alternative technology are addressed in detail by assessment criteria 3.8.1.1 through 3.8.1.5.

The costs associated with full-scale facility construction and operation will be developed in response to the criteria set forth in evaluation factors 3.8.2 through 3.8.5. It is important that the Army develop site-specific cost figures for the baseline system comparable to those for the alternative technology being considered.

The costs associated with decommissioning the destruction facility will be assessed under assessment criterion 3.8.6.1. This information should be developed for both the baseline system facility and the alternative technology facility.

The Army's draft assessment criteria concerning cost do not address the issues that arise when discounting is applied to storage costs as affected by schedule delays or accelerations specific to a particular destruction technology. Similarly, discounting should be applied to technology-specific construction, operations, and decommissioning costs.

In comparing the Army draft criteria document with the critical factors identified in Chapter 3 of this report, the committee found that the draft criteria document addresses most of the issues the committee considers to be important with respect to process efficacy, process safety, schedule, and cost. Many of the issues in these areas were covered quite thoroughly. For example, the document addressed certain aspects of process safety, particularly the potential for worker exposure and the monitoring thereof, in great detail. Nonetheless, there are a number of areas that the committee believes are important to the evaluation of an alternative technology that the Army has not addressed thoroughly or has addressed only by implication.

A key area in which the draft assessment criteria document could be improved is in the treatment of the effect of federal, state, and local regulations on the destruction process. Despite the fact that the text of the document acknowledges that many of these issues could be go/no-go issues, the assessment criteria in this area are very broadly phrased. Much finer detail is required to ensure that appropriate decisions are made in the area of responses to regulations. It is important that the effect of these regulations on destruction efficiency, operation of the plant, and disposal of process residuals is understood.

Another area that appears to be inadequately addressed by the draft criteria is the assessment of risks to

the public and burdens to the surrounding environment from the competing destruction technologies. Although the assessment criteria require the development of significant amounts of information about the potential for the release of agents and other hazardous materials, they do not appear to require the use of this information to assess the relative health risks. Given the level of concern expressed in the host communities, the criteria should specifically address the development of such comparative-risk information and should specify the potential burdens to air, soil, and water.

On a related note, although the draft assessment criteria include many questions regarding the impact of technology choice on schedule, there is no indication in the criteria that this information will be used to relate schedule changes to changes in total programmatic risk. In particular, the assessment criteria do not appear to recognize the possibility that alternative technologies may reduce storage risk, nor do they seek information on the potential schedule impacts of public opposition to a particular technology.

With regard to monitoring issues, the assessment criteria are phrased in a way that does not allow for the possibility that monitoring needs for an alternative technology may be met by achievable advances in existing monitoring technology. In addition, the committee believes that sampling procedures, response times, and required detection limits need to be addressed more specifically.

Although the development of assessment criteria is important to the decision-making process with regard to an alternative technology, these criteria will serve the process well only if they elicit responses that provide a body of information adequate to support a decision about whether to go forward with the alternative technology. For this reason, the committee summarized (in Chapter 4) the kinds of information that will be needed to (a) make a decision regarding implementation of an alternative destruction technology and (b) design a demonstration facility for the process. A later NRC report will evaluate the adequacy of the information generated by the Army's RDT&E program. As noted in Chapter 4, some of the information necessary for decision-making is specific to the site and process being considered; other required information is generic to process development activities.

Many characteristics of a disposal technology are generic and must be met by any process. An example is destruction efficacy, that is, the attainment of an appropriate level of destruction of chemical agent, including the characteristic of irreversibility. The efficacy question is a basic evaluation factor and is addressed well by the Army's assessment criteria, as noted previously. Similarly, the proposed assessment criteria adequately address the need to characterize the specific agent to be destroyed at a stockpile site.

Broadly speaking, allowable discharge limits for a disposal process are generic. For example, the Clean Water Act prohibits the discharge of chemical warfare agents into navigable waters of the United States without any qualification as to concentration of agent in the discharge. An agent-destruction process must, therefore, eliminate agent from any aqueous discharge that might enter a navigable river or bay.

Although the performance standard for disposal technologies is generic, destruction processes are affected in significantly differing ways, depending on the nature of the process residuals, that is, the materials remaining when agent destruction is complete (e.g., combustion gases, salt solutions, and biomass). The baseline system implemented at Johnston Island in the Pacific and at Tooele Army Depot, Utah, has no liquid discharge because the solutions of inorganic salts are evaporated to dryness and the moisture is discharged into the atmosphere. As a consequence, the question of discharge of agent into navigable waters does not arise. Conversely, the most efficient design of an alternative process might involve discharge of an aqueous salt solution into an existing waste-water treatment plant. As a result, potential discharge of traces of agent into nearby navigable waters may become a significant question.

The Army's proposed assessment criteria in the basic category “Regulatory/Legal” may address the need to meet allowable discharge standards for a specific process; but, as currently developed, the criteria lack sufficient specificity to ensure that the necessary information

will be developed. As noted later in this chapter, their utility in comparing an alternative technology with the baseline system will depend on how the criteria are applied in the decision-making process. Also, as discussed later, it is important that the Army work with both federal and local regulatory authorities to gain a sound understanding of how environmental regulations will be applied to a specific process at a particular site.

Some generic safety aspects for the design and operation of a disposal process should probably be addressed more explicitly in the assessment criteria. For example, fire and explosion hazards and reaction instability need to be treated explicitly. They may be implicit in the worker-safety category, for example, in 3.4.3.2 (“What is the probability of an uncontrolled chemical reaction?”), but specific consideration is vital. Although the safety considerations are generic, they need to be evaluated specifically for each alternative technology as well as for the baseline system.

A site-specific conceptual design for a full-scale agent-destruction facility should be started before a decision is made to proceed with a pilot-scale demonstration facility. As noted previously, building and operating a pilot plant is so costly that there must be a high level of confidence that the alternative technology to be demonstrated will actually be implemented. The process of designing a full-scale facility should provide the critical information needed to answer the assessment criteria questions concerning cost and overall schedule and the derived impact on storage risk and overall risks. The design itself is not specified by the assessment criteria, but responses to the criteria must include information critical to the design, such as process description; physical and chemical properties of agent, reagents, and reaction products; and energy balances. In general, the assessment criteria for the “Process Design” and “Research, Development, Test and Evaluation” basic categories address these topics to the extent required.

Preliminary evaluations of risk that are specific to the site and to the technology selected are also needed. A preliminary comparison of risks posed by differing technologies should cover many of the topics incorporated in the worker-safety and environmental basic categories, particularly those dealing with hazardous materials, which are addressed by assessment criteria in evaluation factors 3.3.2 and 3.3.3. Similarly, health-impact comparisons should deal with health risks, both to workers and to the public. Some health-risk factors are addressed under the headings of “Worker Safety” and “Public Health.” The assessment criteria in both the accident and health areas are helpful; but simply answering the questions posed by the assessment criteria will not provide the level of understanding of the critical factors elicited by a professional risk assessment study. While the time available before the 1996 Defense Acquisition Board decision does not allow full quantitative risk assessments, these preliminary comparative hazards analyses should be performed before the decision is made, with more comprehensive risk assessments completed before the start of full-scale agent destruction (as was done with the baseline system).

Beyond the evaluation factors of “Worker Safety” and “Public Health,” the assessment criteria do not appear to address the total integrated risk, whereby the findings from all the preliminary risk assessments on disposal risk and storage risk for the remaining life of the stockpile at a particular location are summarized. The total integrated risk would appear to be the proper measure of whether an alternative process is “significantly safer,” as stipulated in the congressional directive, than the baseline system.

In addition to the issues identified by comparing the Army's draft assessment criteria to the factors that the committee considers central to evaluating competing destruction technologies and those identified by assessing whether the use of the draft criteria will generate the data the committee believes are necessary for making the decision, the committee identified other concerns that merit discussion. These concerns include the way in which the responses to the assessment criteria will be used in decision-making, the objectivity of the decision

process, the character of the “Public Acceptance” criteria, and the use of host-community input in the decision-making process. These concerns—use of information, objectivity of the decision process, and public acceptance—are discussed below.

Nothing in the draft assessment criteria document indicates how the Army will use the information generated by the 169 assessment criteria to decide whether to demonstrate an alternative destruction technology. Describing a plan to use the answers to the questions posed by the assessment criteria is important to guide the decision-making process and to gain stakeholder understanding of the ultimate decision.

On the most basic level, a decision cannot be made based on 169 criteria without some method for distilling that amount of information into a much smaller group of constructs. This problem is magnified by the many subquestions embedded in the Army's assessment criteria. Thus, the actual number of responses to the assessment criteria may be well over 200. This amount of information is unmanageable without an accompanying methodology for using it. In the absence of a defined methodology, each person who evaluates the data will create an individual methodology to simplify it.

This observation leads to the second reason why the decision-making methodology must be described. Among the five identified objectives for the development and implementation of assessment criteria are the following: “to communicate the basis of the assessment process in support of public discussion of this process” and “to provide information in an organized format to support the selection process.” As noted above, in the absence of a stated methodology for utilizing the information from the responses to the criteria, each person evaluating the information will develop an individual process to distill the data. Inevitably different evaluators will arrive at different conclusions based on the same data. Moreover, since most evaluators will not clearly articulate the process used for such distillation, it will be difficult for them to understand why others come to different conclusions. Such a result does not “provide information in an organized format to support the selection process.” More importantly, the lack of a defined methodology will be a major impediment to the public discussion of the assessment process. Thus, without a clearly articulated methodology for managing this large amount of data, an important function of the assessment process—facilitating public understanding and agreement regarding the choice of destruction technology—will be undermined.

There are many ways that the Army could approach constructing a methodology for using this information to come to a decision. At a minimum, any system to integrate this data into a meaningful number of constructs should address two key issues. First, there are clearly some criteria which, if not met, would preclude the implementation of an alternative technology. For example, if the answer to the question “What is the probability of an uncontrollable chemical reaction?” is that the probability is “high,” it is a “no go” decision. The Army already recognizes that “most of the evaluation factors in the “Regulatory Legal” category will serve as go/no go decision points” (U.S. Army, 1995a). Similar criteria should be clearly identified, and the reasons why they constitute “go/no go” decisions should be clearly and fully explained.

The second issue in integrating data for a decision is recognition of the interrelationships among the proposed assessment criteria. One of the most obvious and important examples is the relationship between schedule and risk. As noted in previous NRC reports on this subject, an increase in the length of time that stockpile materials remain in storage increases the total risk to the public. Thus, the relationship of all schedule factors, including schedule delays caused by permitting issues or by public opposition, must be considered in assessing whether one particular technology is safer than another. A less obvious example of such interrelationships is provided by the assessment criteria related to monitoring capability (3.3.4.1 through 3.3.4.3 and 3.10.8). The ability to monitor environmental media, both for agent and for other dangerous materials (either breakdown products or materials needed to support the process), may affect whether regulatory compliance can be achieved. Similarly, monitoring capability is needed to establish whether the requirements of international agreements can be met, to determine the risk of worker exposure to agent, and to underpin the level of public

acceptance of the technology. Unless these interrelationships are recognized and explained, those who review the answers to the questions posed in the assessment criteria may come to conclusions that are inconsistent with the data or may fail to understand the results of the selection process.

One of the Army's stated goals for criteria development is to ensure that the technology assessment “is comprehensive, systematic, and objective.” To further this goal, the Army states that it sought to formulate questions that required a “yes” or “no” answer when appropriate or a numerical answer when a parameter was involved.

While the committee agrees that these goals are desirable and worthy of considerable effort, review of the draft assessment criteria indicates that many of the criteria cannot be answered with a simple yes/no or numerical answer. Even when numerical answers can be provided, those answers will not necessarily be completely objective. As an illustration, consider the first two environmental monitoring criteria:

The first of these assessment criteria meets the objective of requiring a numerical answer; the second does not and may require a complex response. In the context of the overall comparison of an alternative technology with the baseline incineration system, neither of these criteria may be as objective as it appears on the surface.

The baseline system nominally has only two effluents: scrubbed and filtered gases and dried salts from the brine reduction area. A third effluent stream, the slag from the liquid incinerator, is not usually noted. A “level playing field” comparison of the baseline system versus an alternative technology must deal with aqueous effluent streams consisting of salts dissolved in water. Both processes produce such streams. In the baseline system, the aqueous stream arises from scrubbing of the gaseous emissions. It is sent to an evaporator in the brine reduction area and the water is discharged into the atmosphere.

Characterization of the effluent streams may also be complex because neutralization processes often produce traces of organic materials and solid precipitates of metal salts derived from corrosion of the agent storage containers. Effluents from incineration seldom contain high levels of organic compounds; however, metals have been detected on several occasions in the salt formed through the drying of the brine from the pollution abatement systems. Detection limits for monitoring both agent and process reactants may differ in the aqueous effluents from different destruction processes because of different interfering chemicals.

The point in citing these two assessment criteria as examples is to illustrate the complexity of answers to the seemingly simple assessment criteria questions. Almost inevitably, subjectivity creeps into responses that involve such complexity. The same is frequently true in responses to criteria requiring numerical answers.

As an example of the subjectivity of numerical criteria, consider criterion 3.3.4.5. The question “How often must monitoring take place, both general area and personal?” can be answered in terms of minutes, hours, or days. The process for arriving at the answer to this “ simple” question may be complex and may involve personal judgments. The response might involve making measurements or estimates of monitor response times; ventilation air-handling characteristics; timeweighted allowable exposures of personnel; and other variables, such as personnel activities. Because some of these inputs involve estimates and weighing of various factors, the numerical result is best expressed as a range rather than as a single number.

Thus, even seemingly well-designed technical criteria may elicit responses that are not altogether objective. The effect of summing up 170–200 such responses when assessing an alternative technology is that the overall integrated assessment is subjective to some undetermined extent. This outcome seems inevitable. The final assessment will depend on the biases and experiences of the evaluators (both at the primary technical level and at the ultimate decision-making levels). To the extent that subjectivity can be minimized by defining the use of the criteria in the decision-making process and by identifying assumptions, projections, and uncertainties that affect the responses, it should be done.

Clear guidelines for use of the criteria may facilitate communication and achievement of consensus among the members of an evaluating committee. Guidelines may also assist in communicating the basis for the decision to the public.

The committee found the assessment criteria assembled under the basic category of “Public Acceptance” to be problematic for several reasons. The first four criteria (3.5.1.1 through 3.5.1.4), which apparently were suggested by the Citizens Advisory Commissions in Indiana and Maryland, are actually directed to whether the alternative technology can destroy agent and can meet legal permit requirements in the Citizens Advisory Commissions home states. While it is noteworthy that the Citizens Advisory Commissions regard these questions as the most important, these criteria more appropriately belong in the “Regulatory/Legal ” basic category.

Assessment criterion 3.5.4.1 solicits information only on potential benefits to the community from the process. Input received during the committee's meetings with citizens living near the Aberdeen and Tooele sites, as well as the written public responses that the committee has received ( Appendix B ), suggest that negative economic impacts are a major concern to the community, second only to that of adverse health affects. Significant concerns were expressed about potential adverse affects on property values due to the presence of a nearby disposal facility, especially an incinerator. Similarly, concerns were expressed about how emissions from a facility might affect the salability of agricultural products, such as meat, milk, and fresh produce. In the vicinity of Aberdeen, concerns were expressed about the negative impact on recreational activities on the Chesapeake Bay, with ensuing financial losses for local businesses. Thus, to the extent the Army intends to examine potential benefits to the community from the stockpile disposal program, it should also examine potential detriments.

The remaining assessment criteria assembled under the basic category of public acceptance ask questions such as:

• What is the anticipated success of community involvement programs aimed at improving understanding and acceptance of the technology? (3.5.1.5)

• What is the acceptability of the technology to environmental regulators and environmental interest groups? (3.5.3.1)

Questions such as these are troublesome for several reasons. First, they cannot be given either a numerical or a yes/no answer. The methodology for answering such questions is not immediately apparent, and the criteria document does not suggest any. However, even if a methodology for answering such questions were developed, the answers, to be accurate, would have to reflect the diversity of views within the interested communities. It is not clear how such complicated answers could achieve any of the five objectives for the development of the assessment criteria that are identified in the Army's draft assessment criteria document.

The committee recognizes that public acceptance of the Army's choice of technology for the Chemical Stockpile Disposal Program is critical for many reasons. On the most basic level, it is undesirable for a government to implement programs to which the immediately affected community is openly hostile. In terms of program goals, community opposition—even by a small, but determined, segment of the community —can result in schedule delays. These delays, in turn, drive up costs, increase total program risks, and jeopardize the country's ability to abide by international treaty obligations.

Given the difficulty of assessing community acceptance or opposition to a particular technology, and the fact that determined minorities can potentially cause significant program delays, the committee believes that, instead of trying to measure public acceptance of any particular technology, the Army should focus on improving and expanding its efforts to involve the affected communities in the decision process. Case studies have repeatedly shown that the degree of acceptance will be related to the public's knowledge of the technology and to people's sense of involvement in the technology selection process. Although the technical aspects of a specific disposal technology may not be fully understood by the general public, there are several aspects of the technology selection process to which the public can make useful input. Some of the points on which community input may be helpful are suggested in the section below on “Opportunities for Involving Host Communities in Program-Milestone Decisions.”

Successful and timely implementation of any agent-destruction process at Aberdeen and Newport will require extensive dialog with representatives from the

host communities. This should be accomplished through information dissemination programs, such as public meetings, and by providing opportunities for meaningful input from affected stakeholders within the host communities throughout the entire life cycle of the Chemical Stockpile Disposal Program (CSDP).

Two specific groups, the Citizens Advisory Commissions and the state environmental protection agencies, have critical roles in this process. The Citizens Advisory Commissions are the state-designated community representatives for providing input to the CSDP. The state environmental protection agencies are the regulatory agencies required to review and approve required facility permits. While the two groups have different missions, their responses may reflect each other's concerns and those of the community at large.

The Army has made significant progress in dialog with host communities. Information-dissemination programs have received increased emphasis and should continue to be enhanced. A significant step in gaining input was achieved through the Army's request for the Citizens Advisory Commissions at Aberdeen and Newport to review and comment on the first draft version of the Army's criteria for evaluating alternative technologies.

Several future milestones in the destruction program also will provide important opportunities for the Army to obtain input from the Citizens Advisory Commissions and state environmental protection agencies. These are discussed in the following sections.

Process Design Choices That Impact Process Effluents. An important concern to host communities has been the issue of process effluents. This issue has been particularly contentious for incineration processes, which result in atmospheric discharge of combustion gases. Alternative technologies, which rely on neutralization or neutralization followed by biodegradation, require several design choices that impact process effluents. A central design choice will involve water management. Most of the neutralization processes being considered require the use of substantial quantities of water to carry out the required chemical and biological reactions. Water may be used either in a single-pass mode or it may be recycled. The extent to which water is recycled will impact both process complexity and cost.

Disposal of Aqueous-process Waste Streams. This subject is complex and presents several options. Aqueous-process waste streams containing reaction products, such as inorganic salts and residual organic species, may be discharged to a waste-water treatment facility or may be evaporated, which requires either an atmospheric discharge or condensation and recycling. In either case, excess water will have to be removed from the process. In addition, evaporation will most likely require subsequent treatment (e.g., solidification/stabilization) and disposal of salts as solid waste. Thus, a trade-off exists when choosing the production of waste water, solid waste, or atmospheric emissions. Selection of biodegradation as a process component also will impact process effluents. Aerobic biodegradation processes will require aeration, which results in an atmospheric discharge. Biodegradation will also generate biomass (sludge), which will require disposal.

Pilot-plant Locations. The options for locating an alternative technology pilot plant are either at the Chemical Agent Munitions Disposal System (CAMDS) at Tooele Army Depot, Utah, or at Aberdeen for mustard and Newport for VX. Army considerations must include the availability of a trained work force, appropriate facilities, permit requirements, cost, and schedule. Locating a pilot plant at either Aberdeen or Newport may result in accelerated permitting because of the community's vested interest in program success or may be delayed because of the need to demonstrate technology at a previously untested scale.

Review of Alternative Research and Development Results. The decision about conducting a pilot demonstration of an alternative technology and, if successful, proceeding with full-scale implementation will depend in large part on the results of the research and development program currently in progress. It is important that the Citizens Advisory Commissions and state environmental protection agencies have adequate opportunities to review research and development results and raise concerns.

Permitting Schedules. Permitting schedules and response intervals are controlled by the state environmental protection agencies and may be delayed as a consequence of adverse community input. Both the state environmental protection agencies and the Citizens Advisory Commissions should be consulted about expected permit schedules under baseline system and

alternative technology scenarios and about approaches for expediting the permitting process.

Review of Preliminary Site-specific Risk Assessments. Minimization of cumulative risk has been a guiding NRC Stockpile Committee principle during review of the Chemical Stockpile Disposal Program. Comparative risk assessments will provide critical information during tech- nology selection. Potential health and environmental impacts of a disposal facility also are central concerns to host communities. Input should be solicited during the development of the risk assessments. In addition, community review of the study results and risk mitigation measures will assist in building public confidence.

Facility Decommissioning. The potential for extended use of a chemical demilitarization facility has been a concern to host communities. Facility decommissioning requirements and schedule may be significantly different for baseline system and alternative technology facilities.