Metrics and System Evaluation
The ability to protect occupants, activities, contents, and buildings themselves is the overarching goal of the different architectures for building protection. Several measures of effectiveness have been developed and used to assess protective ability. The appropriate choice of metrics and criteria for evaluation depends on the goals of protection and the objectives of the system design. The goals could be protecting resources (personnel, contents, or buildings) and meeting mission requirements (ensuring continuity of operation). The objectives of a system design could be low maintenance and service of system components or maintaining a preset budget. Preset metrics and criteria for evaluation are key elements in assessing whether a building protection system achieves its protection and operation goals and meets its design objectives.
Metrics are measures used to assess and compare the performance outcomes of different systems. The metrics of building protection systems are measured indicators of the impact of a protection system on the occupants and operations of a building in the event of an attack, and most common ones include protection and operational performance metrics.
In building protection, the primary focus is on protecting building occupants and contents. Protection metrics provide a quantifiable measure of a system’s efficacy in protecting building occupants and contents. The degree of protec-
tion offered by a system can be assessed by comparing protection metrics in a building with or without (or before and after deployment of) a protection system. According to some prior and existing building protection studies, the protection of people could outweigh the protection of contents. Some commonly used protection metrics seen in prior and existing building protection programs include fraction of building exposed (FBE), fraction of occupants exposed (FOE), and lives saved.
The Immune Building Program of the Defense Advanced Research Projects Agency (DARPA) used FBE as a primary metric. FBE is defined as the fraction of the building (by volume) in which occupant exposures would exceed a prescribed level or guideline, typically evaluated as a function of the mass of agent released.
For chemical warfare agents and most toxic industrial chemicals, the exposure criterion is the acute exposure guideline level (AEGL) (NRC, 2000, 2002, 2003b, 2004, 2006) or other similar estimate of acute toxic potency. For biological agents, the exposure criterion is the infectious dose, that is, the number of organisms believed to be necessary to overwhelm host defense mechanisms and establish an infection.1 FOE measures the fraction of occupants that are exposed to a threat agent. For this metric to be useful, the amount of exposure at a given time or duration and the type of exposure (for example, skin, inhalation, or ingestion) need to be specified. FOE then can be derived in a similar manner, provided information is given about the location of the occupants and their exposure levels. FOE is not commonly used as a primary quantitative metric, but it is inherent in some of the chemical and biological protection architectures that have been developed and deployed. (See Chapter 5 for specific demonstrations that use FOE and FBE metrics.)
FBE relies on an experimental measurement or analysis of the transport and dispersal of agents or toxic materials within the interior spaces as a result of releases either outdoors or within the building. For example, DARPA’s Immune Building Program uses multizone contaminant transport modeling as a principal means of estimating the concentration and exposure profiles within the interior spaces. Several such models are available; the Immune Building Program adopted the use of the CONTAM2 multizone code (NIST, 2006a; Walton and Dols, 2006). Tracer experiments conducted as part of the Immune Building test bed program have shown that the modeling approach can provide results comparable to actual measurements. As is the case with most models, accuracy is highly dependent on an adequate understanding of the input parameters and inherent model limitations.
FBE and FOE have several limitations. They do not take into account the
time between initial release and when the exposure guideline is exceeded (if this occurs). A building can be evacuated before the threat agent released has spread in the building. Another limitation of either FBE or FOE is that not all spaces within a building require equal protection. This is especially true for buildings that are operations centers where the highest value is protection of continuity of operations. FBE and FOE also assume that the occupants or the state of connectivity in a building is static even though personnel move around the building and the opening and closing of doors could change the connectivity of a building. They also do not take into account whether and when the diseases caused by exposure can be treated.
Another protection metric, which is related to FOE, is the number of lives saved or the reduction in the number of exposed occupants as a result of protective architecture. If the disease progression of a threat agent and the countermeasures are known, then the number of lives saved or reduction of exposed occupants as a result of building protection can be estimated. Number of lives saved or the reduction of exposed occupants can be measured as decreased mortality, morbidity, or costs associated with human health effects.
Operational Performance Metrics
The operational aspects of building protection are important considerations, even though they are less quantifiable than the protection metrics. The need for continuous operation has an important influence on protection system design. If part of the protective response is building evacuation or movement of personnel to an interior shelter, essential operations could be disrupted. Key activities that cannot be disrupted must be accounted for by the protective architecture. Similarly, the tolerance for false alarms could vary from building to building depending on the need for continuous operation.
If part of the protective architecture requires an active response from occupants, such as evacuation or seeking shelter, then the response time can be an important criterion. The paucity of data on response times does not allow realistic estimates to be made. The overall performance metric analyses need to ensure that response times are estimated cautiously and with full awareness of the impact of their variability on the overall performance.
The time for recovery, restoration, and return to service is linked to the issue of operational continuity or future use of a material asset. If there are alternative means to meet operational criteria (for example, personnel at different sites or buildings can maintain critical operations off-site), then return to service might not be a major consideration. Nonetheless, a protective architecture that results in the temporary loss of an operational asset for a few days could be desirable when restoration to service of a building without protection takes months or years.
Operationally, user acceptance can affect the performance of protective systems, especially those in which some response action by personnel is neces-
sary (for example, evacuation and sheltering). Acceptance by users also affects maintenance of the building protection system because users who see the value of a system are more likely to maintain or replace its components or comply with rules and regulations set to ensure its efficacy. If a protection system is to be implemented, the benefits of the system and its proper operation need to be communicated to users and even to the building occupants.
There could be ancillary benefits for some building protection systems, such as improved air filtration leading to higher-quality indoor air. Recent research suggests that air quality is correlated with sick days, health care costs, and productivity, but it could take some time before such measures are sufficiently reliable to be useful in a cost-benefit analysis. In the near term, however, the likelihood that such trends exist and are statistically significant might serve as an ancillary motivation for security improvements (Fisk, 2000a,b; Seppänen and Fisk, 2006).
MAINTENANCE AND COST EVALUATION
In addition to quantifiable protection metrics, service, maintenance, and cost issues must be considered in protection system evaluation. Compliance with cost and maintenance criteria can be used for decision making and to assess whether the maintenance and budget objectives are met. During the design phase of a system, preconceived standards for the service and maintenance of its components and a budget for the total cost of the system are set. The evaluation phase assesses whether these standards and the budget are met; if not, the system or the design plan could be adjusted. Even if adjustments cannot be made, the evaluation provides valuable lessons learned to guide future protection efforts.
Service and Maintenance
As with operational metrics, maintenance issues have important consequences for decision making. In many cases, maintenance has an impact on costs, particularly over the lifetime of the building. Almost all mechanical and electronic systems require periodic maintenance to ensure that they are performing as originally expected or required. These maintenance activities, such as periodic calibration and performance testing, incur costs including labor, supplies, and consumables. Because maintenance of protection systems is often essential to ensure that they work properly, additional training and staffing beyond normal building or heating, ventilating, and air-conditioning (HVAC) system maintenance are needed. For example, installation of improved filtration requires greater care to ensure that there is no air bypass around the filters. In some situations, follow-up aerosol testing is needed to verify system efficacy.
In addition to routine maintenance, lifetimes or failure rates of the protection system components influence decisions about the types of systems to deploy.
Some of the criteria that have been used for evaluation are mean time to failure or system or component durability. A related issue is the ease and cost of component replacement. System or component obsolescence should also be taken into consideration. Obsolescence is related to the adaptability of the overall system to changes in some of the components or changes in protection requirements.
It is probably safe to say that the more people or interior area a building protection system saves from exposure to a release, the better the system is. It is not plausible to say unequivocally that the cheaper the system, the more desirable it is, or vice versa. For cost to be used as a criterion for evaluation, it has to be considered in terms of benefits received. This is known as cost-benefit analysis and is used frequently in the decision-making process by federal agencies (White House, 1992). A detailed discussion of cost and its role in risk assessment, risk management, and decision making can be found in Chapter 6.
The methodology for performing cost-benefit analyses is complex and beyond the scope of this report to discuss in detail. However, general principles can be adapted from Cost-Benefit Analysis Guide for NIH IT Projects (NIH, 1999), even though it is directed at selecting information technology systems, and from the Standard Practice for Measuring Life-Cycle Costs of Building and Building Systems (ASTM, 1999) and Measuring Benefit-to-Cost and Savings-to-Investment Ratios for Buildings and Building Systems (ASTM, 1998),both published by the American Society for Testing and Materials. Cost-benefit analyses, as generally performed by federal agencies in compliance with Office of Management and Budget Circular A-94 (White House, 1992), typically include comprehensive estimates of the projected benefits (such as lives saved or reduction in FBE as discussed earlier in this chapter) and costs for all alternatives. Cost-benefit analyses are performed before and during design of a system to optimize design performance and expenditure (Chapter 6). However, they also could be performed after implementation to determine whether the expected benefits are achieved within the preset budget. To assess whether budget objectives are achieved for building protection designs, costs have to be ascertained over the system’s life cycle.
Components of cost for protection include the initial cost, cost of operation, and cost of maintenance. Initial cost includes design fees and cost of personnel (such as building operators) and subject matter experts (such as architects, engineers, and risk assessment and management experts) who dedicate their time and labor to plan the protection system, cost of required building modifications, and cost of equipment, installation, and commissioning. Operating costs include all impacts on building operation staff and their necessary training and on utility use such as incremental HVAC fan energy use associated with higher-pressure-drop filters or the cost of operating lamps in an ultraviolet germicidal irradiation system. Maintenance costs include replacement of used parts such as fully loaded
filters, periodic calibration of instrumentation, and testing to confirm proper system function. Collectively, these cost factors determine the lifetime cost of ownership.
A life-cycle cost analysis frequently is used to justify a larger initial cost to obtain the benefits of lower operating or maintenance cost. In the case of security enhancements, consideration of the system life cycle takes on increased importance because failure to commit to the ongoing costs of maintaining such systems will compromise their ability to perform as intended.
Representative cost information for particulate and gas-phase filtration systems has been published by the National Institute for Occupational Safety and Health (NIOSH, 2003). This source cites a range of $6 to $40 per square foot from continuous high-efficiency particulate air (HEPA) filtration and activated carbon filtration to sensor-activated military filtration systems. Associated operating costs (primarily the cost of moving air against the higher resistance of such filters) are estimated at up to $2 per square foot each year, which is comparable to the energy cost incurred by a typical commercial building (NIOSH, 2003). HEPA filters are roughly 10 times as expensive as standard efficiency particulate filters of the same size, but the cost of gas-phase media can be an order of magnitude more expensive than a HEPA filter. Site factors such as space limitations and fan characteristics can add cost or limit the range of options, particularly in the case of retrofits. Qualitatively, these data indicate that building protection can be costly and that cost-benefit analyses are, therefore, important in justifying the costs of protective measures. Some passive security measures, particularly if implemented in a new building design, might carry no first-cost penalty and could reduce operation and maintenance costs. An example is architectural compartmentalization combined with the use of a dedicated outside air (once-through) system for ventilation.
The appropriate measures of effectiveness and criteria for evaluating a building protection system depend on the goals and objectives of building protection. Although the focus often is on the number of lives saved, considerations related to operational performance, maintenance, and cost should not be overlooked when planning for building protection and should be used to evaluate system-wide risks and benefits.