5

Evaluation Criteria

This chapter describes the criteria that the committee uses in evaluating alternative technologies that may be used in place of open burning (OB) and open detonation (OD). These criteria establish a means for comparing OB/OD to alternative technologies and allow for a systematic evaluation (see Chapter 8). The ordering of the presentation of the criteria in this chapter does not reflect any prioritization. The committee made no judgment on which are more or less important. The four criteria mentioned in the statement of task—throughput, personnel safety, environmental impacts, and cost—are discussed before additional criteria that the committee adopted.

Four evaluation criteria (throughput capacity, personnel safety, environmental impacts, and cost) were explicitly specified in the legislative language that became the committee’s statement of task (National Defense Authorization Act for Fiscal Year 2017; SEC 1421, 2016). The committee applied five additional criteria: public health impacts, technology maturity, permitability or other approvals, monitorability, and public confidence. The committee has identified these as important criteria that also need to be considered when selecting alternative approaches to OB/OD. In selecting the additional five criteria, the committee considered criteria used in prior reviews of alternatives to OB/OD performed by the Army, the National Research Council (NRC), the Cease Fire! Campaign (see Appendix B), as well as the experience and judgment of the members of this committee.

The actual selection of alternative technologies for deployment would require a detailed engineering analysis based on the design and construction of each munition, which is outside the committee’s scope and ability to conduct within the limits of this study. Further, the committee has chosen not to weight these criteria in terms of importance, because the committee firmly believes that the application of the criteria is munition, technology, site, and community specific. Clearly some criteria, such as personnel safety, need to be addressed rigorously, while others, such as maturity, might be desirable but would probably carry less weight than safety in selecting a specific technology. However, in all cases there needs to be a high level of confidence that the technology employed will demilitarize the munitions, that it can be implemented, and that it will be protective of workers, the nearby community, and the environment.

Munition demilitarization can be achieved via a combination of different technologies for each demilitarization process. Criteria used in selecting a treatment train not only need to focus on how effective a process is for a given munition but also need to take into account the ease with which a single process in such a treatment train can be upgraded or substituted so that the full treatment process can be easily adapted to a different type of munition. Consideration also needs to be given to the inclusion of a new promising technology, either as an integrated whole-munition process or as a process or component within an existing treatment train that might not have matured enough for immediate implementation when the treatment train was originally selected.

The committee’s nine evaluation criteria are discussed in detail below. The committee provides a general narrative on how these criteria are applied to the alternative technologies in Chapter 8.

THROUGHPUT CAPACITY

Throughput is the rate of material that can be processed in a given time and can affect treatment costs. When a technology in and of itself provides treatment for the whole munition, determining throughput may be straightforward. Where treatment of the whole munition is based on a treatment train, including pretreatment, treatment, and post-treatment activities, determining throughput is less precise. It is important to identify any potential impacts on throughput that arise in a specific treatment configuration. Some factors that affect throughput can be mitigated relatively easily by adding more low-cost units or components (e.g., band saws for increased size reduction capability), but in other cases the cost of additional treatment capability (e.g., more or larger detona-

tion chambers) may significantly increase costs. Ideally, the design of an overall treatment system would allow for the flexibility to process different munitions or to be reconfigured relatively easily for that purpose. Finally, it is important to recognize that whole-munition treatment systems may also have rate-limiting steps (i.e., loading, cooling, etc.) that affect their throughput.

PERSONNEL SAFETY

The avoidance and prevention of possible injury or death to workers conducting demilitarization operations is paramount. There are risks to personnel when transporting or otherwise moving an item to the facility or location where it will be processed, and there are risks in any preprocessing step (i.e., size reduction for alternative technologies or positioning munitions for OB/OD), in the treatment processing step, and during post-processing operations. The risks in all of these steps need to be identified, evaluated, and compared when selecting technologies. The safety risks inherent to alternative technologies and many individual steps within a treatment train can be mitigated with fully automated systems that minimize potential injury to workers.

In addition to injuries from catastrophic events, personnel health and safety risks from cumulative and aggregate direct and indirect exposures to potential toxicants associated with munitions and their destruction, including noise, also need to be considered.

ENVIRONMENTAL IMPACTS

A technology’s ability to monitor, prevent, minimize, and control emissions of contaminants to all environmental media (water, air, soil), both during the demilitarization process and during process upsets, will determine its environmental impact. Furthermore, because various treatment technologies and pollution abatement systems result in the accumulation of secondary waste streams, it is important to consider the ability to meet the management and disposal requirements of these streams. Ideally, the treatment method is a complete solution with no long-term storage or disposal requirements for hazardous process waste streams.

Given the nature of the source material being treated, damage to equipment during treatment is possible. Hence, technologies for the demilitarization of conventional munitions need to have safety controls to prevent accidental releases of emissions, due to equipment damage, that have environmental or health concerns.

Other environmental impacts that need to be considered include those on ecosystems, including wildlife, marine, and aquatic receptors, as well as cultural, recreational, and commercial activities depending on such ecosystems.

Finally, the effects of vibration, noise and shock, visual plumes, and odor need to be considered, as they affect nearby communities.

COST

The costs of demilitarization technologies include capital costs, startup costs, operational and maintenance costs, and closure costs. Alternative technologies will have higher capital costs, which include development, design, construction, site preparation, and installation costs. Site preparation may also include the installation of required utilities. In addition to the capital costs, the implementation of any technology needs to account for permitting and regulatory compliance costs, including state, federal, and internal approvals, including those required by the Department of Defense Explosives Safety Board (DDESB).

Operation and maintenance costs comprise the costs of staff and operators including their procedural and safety training, scheduled and emergency maintenance, environmental and health monitoring, and the ultimate disposal of secondary wastes generated during the demilitarization process. Given that some munitions may have to be transported to sites where the appropriate demilitarization technology is available, transportation costs might need to be accounted for. Recycling may generate income streams that can reduce the overall operational cost of a demilitarization activity.

The cost evaluation of any demilitarization technology also needs to include closure costs. Equipment and sites will have to be decontaminated after closure. Given the much larger land area affected by OB/OD operations, and lack of containment, their closure costs are expected to be highest. Closure costs are usually not considered in the cost of demilitarization activities but need to be considered in an overall cost evaluation A life cycle cost (LCC) analysis is required if a true cost comparison of alternative technologies to OB/OD is to be made. According to Department of Defense (DoD) guidance, LCC is defined “as the cost to the government of a program over its full life, including costs for research and development; testing; production; facilities; operations; maintenance; personnel; environmental compliance; and disposal.”¹ However, due to the lack of complete information on costs, the committee was not able to conduct an LCC.

PUBLIC HEALTH IMPACTS

The potential impacts of different technologies on public health is an important consideration. Permitted OB/OD and alternative technologies are required by the Resource Conservation and Recovery Act (RCRA) to be protective of human health and the environment.

The potential risks from exposure to all of the substances likely to be emitted during demilitarization of particular munitions (cumulative exposure) need to be considered, as do all significant potential sources of direct exposure and indirect exposure (aggregate exposure). Sources of direct exposure include air, water, and soil that a person might come

___________________

¹ For LCC definition, see https://www.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid=e8a6d81f-3798-4cd3-ae18-d1abafaacf9f.

into direct contact with, while sources of indirect exposure include, for example, consumption of contaminated species such as fish or hunted game. Because susceptibility to the potential effects of exposure can vary with age and other factors, differences in susceptibility need to be evaluated during any risk assessment of alternative technologies.

Potential impacts on public health arising from all activities of a demilitarization process other than emissions would also need to be considered, such as the shifting of risk burdens among communities caused by transportation of munitions or munition components from one depot to another or to commercial facilities for treatment.

TECHNOLOGY MATURITY

When selecting technologies for conventional munition demilitarization, it is important to assess how far the technology has progressed toward being able to be used in industrial operation and hence the level of confidence that the technology will operate successfully once implemented. The closer a technology or system is to being ready for industrial use, the more mature it is. There are alternative technologies at several stages of maturity, ranging from novel ideas that have not yet been tested at the bench scale to treatment systems that have a proven track record and for which data are available for the effectiveness, throughput, cost, safety, maintenance requirements and downtime, environmental impacts, and reliability of the full-scale process.

A good measure of technology maturity is the number of munitions that have been treated successfully so far with a specific technology. The Army method when estimating technology maturity during the acquisition process is to assign a Technology Readiness Level (TRL) to the technology. TRLs range from 1 to 9, with 9 being the most mature technology. A TRL of 6 indicates a “system/subsystem model or prototype demonstration in a relevant environment,” the lowest TRL considered by the committee (DoD, 2011). In assessing whether a technology has reached a certain maturity level, and when comparing the maturity of different technologies, one also needs to consider whether that specific technology is in operation under existing permits or other approvals. A history of successful permitting or approval may provide some confidence that the emplacement and operation of the technology at a different site will be successful.

Data from a demilitarization technology with a proven track record may also enable a more accurate cost estimate as well as an estimate of the throughput rate when processing a given type of munition.

PERMITABILITY AND OTHER APPROVALS

Permitability is the ability, or expected ease, to permit an alternative technology under applicable federal, state, tribal, and local laws and regulations. This criterion also includes other approvals, as not all technologies will require permits. There are also instances where alternative technologies may be exempt from RCRA permitting based on the RCRA regulations applicable to a given treatment process (e.g., recycling or certain RCRA exemptions). However, a declaration of exemption from RCRA can also require an application and approval, and the facility may need other permits (e.g., Clean Air Act [CAA]).

The most significant law for the demilitarization of conventional munitions is RCRA (see Chapter 6). The RCRA regulations applicable to an alternative technology depend on the type of technology used to treat the conventional munition or munition component. RCRA regulations focus on protecting human health and the environment based on risks from exposures to contaminants in the air, surface and groundwater, or soil. In addition, there may be other permits required when permitting an alternative technology unit, including the CAA, the Clean Water Act, water use laws, or other cultural and natural resources laws.

Several factors can impact permitability or other approvals. First, as a practical matter, an alternative treatment technology is more likely to be permitted or otherwise approved in a more efficient manner if that technology has been permitted or approved at similar locations and for the treatment of the same or similar wastes. Similarly, the more mature the munition access, treatment, and pollution control technology, the more data are available and, again, the more likely the process will be permitted or otherwise approved in a more efficient manner.

Also, public opposition can impact permitability or approvals by lengthening the permitting process or by asking regulators to impose operational conditions, health risk studies, and other conditions on OB/OD or the alternative technology permits.

MONITORABILITY

It is important that any treatment method can be monitored accurately for both emissions and completeness of treatment. In terms of monitoring emissions, treatment methods need to allow for the effective monitoring of pollutants of concern in all waste streams to determine, for example, if air pollution abatement systems are performing as designed, or if liquid or solid secondary wastes need specific treatment before their disposal. Furthermore, monitoring of surrounding soil, water, and air for pollutants that might be emitted during demilitarization activities may be necessary based on permit or other requirements. Monitoring of soil and groundwater can support rapid corrective action and significantly reduce closure costs. Monitoring for ambient air pollutants (associated with the demilitarization technology) is key to ascertain whether the demilitarization activity could affect public health.

Although costly and a negative impact on throughput, some technologies allow for holding a waste stream for monitoring or testing before it is released. This can be done

for solid, liquid, and gas streams. However, hold, test, and release is usually applied to treatment of chemical weapons due to the toxicity of the chemical warfare agents.

Last, any demilitarization technology would ideally allow for a post-treatment inspection and testing of the munition or treated residue to determine that explosives and or propellants have been sufficiently treated, to allow for disposal or recycling.

PUBLIC CONFIDENCE

Experience has shown that building public confidence and trust via public consultation and participation in the technology selection process is instrumental (it will promote legitimacy); substantive (it will lead to better decisions); and normative (it is the right thing to do in our society; see Fiorino, 1989, and NRC, 2008). A lack of public confidence can impact schedules, cost, and ultimately, the implementation of technologies. Also, as noted earlier, public opposition can cause delays in the permitting process and influence the permit conditions imposed on alternative technology implementations.

Public confidence in a technology and related program management can be linked to core questions, such as “how safe is safe enough,” and cannot solely be answered by science nor technology; they are questions of human values (NRC, 1996). These values, along with social trust, are key factors shaping both management and public perceptions of technologies, including (of relevance to this study) the selection and prioritization of criteria that are used to evaluate them. For example, if the public conceives that some variables are weighted more heavily in the decision-making process than those that are a priority for them (e.g., public health or environmental impacts), public confidence could be undermined.

To evaluate public confidence in making decisions on alternative technologies, the committee considered the following factors:

• The characteristics of a technology and associated risks (e.g., the potential for catastrophic releases, familiarity of technology and risks, types of secondary wastes generated, distribution of risks and benefits within and among communities);
• The management of the technology (e.g., information is publicly available about how the technology and its pollution abatement system work, monitoring data are immediately available and accessible); and
• The processes for making decisions (e.g., are they viewed as being fair, transparent, based on accepted and appropriate criteria).

REFERENCES