Hazardous Materials Transportation Risk Assessment: State of the Practice (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

76 S e c t i o n 5 5.1 General Analysis 5.1.1 Types of Decisions The types of decisions presented in Section 4.1 show the range of issues that can benefit from either safety or security risk analyses (or both). As stated by the NRC (see Section A.1.7 and page 10 of the reference discussed in that section), “dif- ferent categories of decisions require different approaches to risk analysis; strict reliance on quantitative models is not always the best approach.” In many cases, the types of decisions themselves are more closely tied to the type of decision maker. The strongest delineation between types of decision makers is whether they are in the public or private sector. One of the key objectives for this research effort is to highlight the differ- ences between risk assessments done by government agencies and industry. The fundamental difference in the decisions they must make has led to the differences in the risk assessment approaches that each of these types of stakeholders employ. Scope and Timeframe for Analyses One fundamental difference among types of hazmat risk assessment decisions is in the applicability of their scope and the timeframe for implementation. Industry analyses are gen- erally focused on very specific alternatives with short-term timeframes, whereas those performed by public sector enti- ties are generally focused at the system level and may involve implementation timeframes extending several years or more into the future. At another level, industry models tend to be more holistic, covering a wider range of variables than many of the public sector models (DHS CTRA is one notable excep- tion). Using routing as an example, a chemical shipper or car- rier may examine all the feasible mode and route alternatives for a shipment between a manufacturing plant and the ulti- mate customer to determine the choice that best meets its risk/cost tolerance for that product. A routing authority, on the other hand, is limited to only its jurisdiction and may use similar techniques to analyze route risks, but would need to consider a wider range of potential origins and destinations to determine whether any route restrictions or designations through its jurisdiction are warranted. Decisions Calling for Screening, Semi-Quantitative, and Quantitative Risk Assessments As evident in the range of component processes in the over- all CCPS methodology, the use of quantitative risk assess- ments in industrial transportation of hazardous materials is reserved for special cases. Depending on the risk levels among the alternatives being assessed, a resource-intensive quantita- tive risk assessment may not be worth the cost if a less costly (and less detailed) approach is sufficient to discern relative risks. For some decisions, the exact magnitude of the differ- ence in risks between two alternatives is less important than the fact there is a large relative difference. This distinction may be more pronounced when considering the potential effects of low-probability, high-consequence events in which the deci- sion maker may focus less on assessing the risk value and more on managing the transportation operation to reduce the like- lihood of the event as much as reasonably possible and pre- paring to address the consequences should they materialize. In some cases, where the available data are sufficient across the geographic and operational spectrum for the alternatives under consideration, more quantitative risk assessments can be effectively automated. Models such as RCRMS and Fedtrak, which can be used for route choice, are good examples. While simplifying assumptions are used to determine the extent of the impact area from a potential release (using a bandwidth approach that examines the exposure within an appropriate ‘band’ or distance on either side of the transportation infra- structure), that impact area can be applied to the entire length of transportation routes, or at any specific location, and used in concert with the most detailed level of population data to estimate potential human exposure. Analysis and Recommendations

77 One example of how different analyses can use varied approaches to address the same element is in assessing envi- ronmental consequences. For some models, potential envi- ronmental exposure is estimated through counts of river and stream crossings or sizes of sensitive land areas within an impact distance. For others, the counts and land areas are combined with estimated cleanup costs to obtain more quantitative loss estimates. Specific Decisions While each of the models used to address the decisions listed in Section 4.1 are presented with specific comments about the approach used and relevant issues for that deci- sion, each of the decision types and how the available models address those decisions is discussed here separately. Mode Choice. Industry decision makers examining mode choice often have limited options due to the locations of their manufacturing and customer facilities. Many shippers begin with a qualitative risk assessment of the different options they have and only use a quantitative analysis when they determine that their risks are very high. For some companies, the focus is less on mode- and route-choice from a risk minimization per- spective and more on ensuring that they are meeting all regu- latory requirements in their transportation operations. The step beyond this compliance-level focus is to follow industry best practices that may exceed the regulatory requirements. Based on the project team’s experience, the more quantita- tive the analysis, the more that mode and route choice deci- sions are made in concert, with the candidate routes from each mode all being analyzed using the same modeling approach. Care should be taken, however, since the differences in how components, such as accident rates, are measured or estimated across modes can introduce hard-to-measure biases. Generally, mode choice is a private sector decision process, though there are exceptions. The most notable exception is the movement of high-level radioactive materials, in which the Department of Energy is essentially acting like a shipper rather than a regulator. The same applies to Department of Defense munitions shipments. In other cases, the government may try to understand the relative risks of shipping specific materials to effect policy that encourages or prohibits transportation on various modes in order to reduce public risks. This tactic is evident in the Hazardous Materials Regulations, in which some materials are allowed in highway or rail transportation, but are prohibited in air or marine transportation. Route Choice. As with mode choice, route choice is gen- erally a decision limited to industry, except where government entities are acting as shippers. In situations in which quantita- tive risk assessments are determined to be warranted, such as for high-hazard or security-sensitive products, mode-specific routing models and software systems are often used. The TRAGIS/RADTRAN combination is specific to radio- active materials and includes the measurement of risk from incident-free exposure, which is derived from the radiation that emanates from an intact packaging. Risks from accidents and attacks are included as well. For explosives, the IMESAFR model can be applied to trans- portation decisions, but is geared toward fixed-facility risk analysis. The use of IMESAFR in route choice would be mostly focused on modeling the potential consequences from an acci- dent or incident at a specific location and would only consider likelihood components associated with facility activities, such as loading and unloading, rather than those associated with shipments in transit, such as highway accidents. RCRMS was developed by the rail industry to meet the reg- ulatory mandate to perform a combined safety and security route risk assessment for each of their cargos that met certain high-hazard conditions, primarily radioactive, explosive, and TIH materials. Many factors are considered within the model and some qualitative metrics are provided as additional infor- mation for the decision makers, but not included in the risk calculations. The UIUC Tank Car Risk Analysis process can also be applied to routing decisions, as the route informa- tion to evaluate national-level risk values for different tank car designs can be applied to compare alternative routes. This model is focused exclusively on safety, however. The Boston Hazmat Route Evaluation makes use of a wide range of data from different sources and is the most recent known use of the 1996 Federal Routing Guidelines for highway that define the analysis framework and process that states must follow to implement any changes to the Hazardous Ma terials Route Registry. Where possible, local data were obtained at dif- ferent levels of precision, based on availability and suitability. Fedtrak is also a highway-specific model that is focused on near-real-time security risk situational awareness, but has a component to compute a planned route’s safety risk as well. The industry processes (CCPS processes the Large Chemical/ Plastics Manufacturer’s approach) can be used to develop route comparisons at any level from screening to quantitative. Dif- ferent companies may consider different items in their analy- ses, such as the presence of bridges and tunnels. As with mode choice, most companies seem to use a qualitative approach for most of their hazmat shipments, elevating the analysis to a more quantitative approach where the material hazards or estimated risks indicate that more details are needed to make an informed decision. Facility Siting. IMESAFR, the Large Chemical/Plastics Manufacturer’s approach, and the CCPS Guidelines were the only models that were suitable to facility siting decisions. This decision was not one that the interviewees focused on

78 can be quantified, they can be used with a more detailed analy- sis. Otherwise, a screening or qualitative level analysis can be performed. Security Measure Identification, Prioritization, and Evaluation. There are many different approaches for assess- ing the appropriateness of security measures. Generally, any of the models that address security risk can be applied to this decision process as long as they accommodate the ability to incorporate the benefits of each measure in terms of reduced vulnerability. In many of the models, including TSSRA, THTRA, CTRA, and Fedtrak, expert elicitation is used to determine the relative vulnerability of different scenarios. These same elicita- tions can address the changes that would be expected if each of the security measures were applied separately or in combina- tion. Of course, the unmitigated risk values from these models can be used to determine where risk mitigation is best focused and can be the first step in this decision process. The Fedtrak system is specifically designed to measure the reduction in risk from different mitigation strategies. The CREATE Model is designed to assist in determining both the types and best place- ment of radiological/nuclear detection devices to best prevent or deter terrorist attacks. One note related to security countermeasures is that the NRC stated that probabilistic risk assessment may not be the right way to deal with adaptive adversaries—those that make adjustments in their strategies as security countermeasures are deployed—resulting in reduced effectiveness of those countermeasures over time. Security Risk Situational Awareness. There are two high- level approaches for security risk situational awareness: at the systems level and in near-real time. Both THTRA and TSSRA look at the current state of their respective domains that include hazmat transportation. On the other hand, Fedtrak and TRACC attempt to capture and report on the current con- ditions throughout the country and raise awareness of poten- tial security concerns with specific shipments. 5.1.2 Model Components The following sections present the various approaches for dealing with each of the major model components. Data sources, assumptions, limitations, biases, and availability are discussed. Frequency Safety-related models discussed in Section 4 most often incorporate a frequency element in terms of historical event rates, such as accident rates in the case of highway risk analy- sis. Quantitative route risk models, regardless of mode, use or discussed in any detail. In general, where different loca- tions are being considered, the analysis will likely include traditional route choice-based assessment that considers the potential locations as components of different route options. Packaging Selection. The UIUC Tank Car Risk Analysis was specifically developed to address this decision area. It is used beyond selection for specific shipments, but in analyz- ing the effects on risk from different tank car designs. The other industry-specific models and approaches are suitable for addressing packaging selection, but would focus only on elements that change with each alternative. Stronger packages may be more costly and have lower capacities due to additional weight from added features, but may reduce risk sufficiently to offset any needed increase in the number of shipments. Such a difference may be discernible at the screening level or may require a semi-quantitative assessment to estimate, for exam- ple, assuming sufficient data were available. Alternate Product Selection. The general models, such as the CCPS Guidelines, can be used to examine the impacts of alternate product selection. Product alternatives will be a function of the manufacturing needs of the customers and the ability to discern the differences in the shipping and material characteristics. In addition to differences in the hazards and the potential consequences in the event of a release, there might be differences in origin, shipment size and number, and other factors to consider. Models that focus on human health consequences (arguably the primary consideration for most decision makers) may not identify significant environ- mental consequences that may warrant specific attention. Emergency Management Resource Planning. When local agencies develop their emergency management resource plans, they need to be aware of the types of materials moving through their jurisdictions, the quantities and frequencies in which they are shipped, and the hazards that they present. A key element of this planning is to identify the areas of particu- lar concern and to ensure proper response coverage. Many of the models reviewed can assist public planning agencies and the industry entities that desire to provide assistance. Specific models in this area are material-centric (RADTRAN), mode- specific (Pipeline Risk Management Manual and TRACC), or able to address multiple modes (GeoCTA and RRAS). Operational Changes. There are many potential opera- tional changes that may be considered. From an industry perspective and not considering the other decisions already mentioned above, these changes can include varying the time of day for shipments and loading/unloading operations, select- ing alternative carriers, improving training to reduce human factors-related issues, providing escorts, and many others. To the extent that information about these operational changes

79 tainer release probabilities to include qualifying factors, such as container speed at impact in the RCRMS model. The more holistic shipping industry models extend multi-dimensional probability factors even further to include, for example, the type of potential release (e.g., pool, BLEVE,12 jet fire, etc.), as in the CCPS’ Quantitative Risk Analysis Process and the size of a potential release in the CCPS’ Quantitative and Semi- Quantitative Risk Analysis Processes. In the CCPS’ Qualita- tive Risk Analysis Process, additional consideration is given to elements that may affect release probability but are dif- ficult to quantify, such as methods of container securement, inspection procedures, and personnel qualifications. Exclusions to the general rule of the use of release-focused probability terms include TRACC, RADTRAN, and IME- SAFR. TRACC is intended to gauge the potential for elevated barge-based safety or security risks, rather than calculating the risk of release from an incident. In this way, the model acts as a situational awareness tool and calculates the prob- ability of elevated risk based on deviations in a barge’s behav- ior from historical route paths and proximity to other barges and infrastructure. IMESAFR, while employing a probabil- ity term to describe the potential for release (in this case, an explosion), is primarily focused on the potential for injury and loss of life. Thus, the IMESAFR probability term ultimately represents the conditional probability of fatalities or injuries given a release event. RADTRAN was originally designed to calculate the risk of transporting radiological materials in the absence of transportation accidents or releases. The model is flexible enough to be applied to accident risk calculations, however, and several sources exist that define conditional probabilities for various user-defined accident scenarios.13, 14, 15 The probability data that supports these safety models tend to fall into two categories: those that are publicly available or built into the analysis tool and those that must be provided by the user, either through measurement or institutional knowl- edge. Publicly available release probabilities include rail car release rates used in the UIUC Tank Car Analysis method and built into the RCRMS tool (sourced from Treichel et al., these rates explicitly, while semi-quantitative and qualitative approaches direct users to approximate the impact of these rates through the use of relative ratings or to otherwise con- sider their effects on potential safety risks. Highway accident frequency data, such as that employed in Fedtrak, the Boston Hazmat Route Evaluation, and poten- tially in RADTRAN, is publicly available through a variety of state and national sources, including state and federal DOT databases. These sources often categorize accident rates according to roadway functional classifications, providing an added level of precision over generic accident rates for a given location. Available data sources generally do not sup- port further segmenting accident rates based on accident type or cause, however, and truck traffic density estimates are a source of uncertainty. In addition, hazmat-specific accident rates are usually not available and truck accident rates are often used as a proxy. Rail accident frequency data, unlike highway data, is not widely available to the public. While limited accident data are publicly available through the FRA, recent studies support- ing the development of RCRMS have produced accident rates specific to individual carriers, methods of operations, track class, and traffic densities. This detailed rate information is proprietary to the AAR and its constituent organizations, however, and is unavailable for public dissemination. Pub- licly available published estimates of more generalized acci- dent rates exist, however, such as those employed in the UIUC Tank Car Analysis.11 The chemical industry risk assessment methods call for consideration of the detailed route and carrier-specific acci- dent frequencies seen in RCRMS, but a lack of detailed pub- lic data for all modes of shipment require chemical shippers to rely largely upon carrier-supplied information for such parameters. Probability The probability elements of safety models presented in Sec- tion 4 focus primarily on the potential for the release of the hazmat being transported. In most cases, these probabilities are conditional release probabilities contingent on the occur- rence of an accident, such as in Fedtrak, the CCPS Guidelines, and the UIUC Tank Car analysis, among others. These con- ditional release probabilities are dependent upon the hazmat container used for the shipment. Typically, however, the risk methodologies include multi-criteria probability elements that go beyond a single prescribed value for conditional con- 11 Anderson, R. and C. P. L. Barkan 2004. Railroad Accident rates for use in rail transportation risk analysis. Transportation Research Record: Journal of the Transportation Research Board, No. 1863: 88-98. 12 Boiling Liquid Expanding Vapor Explosion 13 Sprung, J. L., D. J. Ammerman, N. L. Breivik, R. J. Dukart, and F. L. Kanipe, 2000, “Reexamination of Spent Fuel Shipment Risk Estimates,” NUREG/CR-6672, Washington, DC: US NRC. pp. 7-73 to 7-76. 14 US DOE, 2002, “Final Environmental Impact Statement for a Geo- logic Repository for the Disposal of Spent Nuclear Fuel and High-Level Radioactive Waste at Yucca Mountain, Nye County, Nevada,” DOE/ EIS-0250F, Washington, DC: US DOE. Appendix J and Transportation Health and Safety Calculation/Analysis Documentation, CAL-HSS- ND-000003, Section 5.3.2. 15 Fischer, L. E., C. K. Chou, and M. A. Gerhard, 1987, Shipping Con- tainer Response to Severe Highway and Railway Accident Conditions. NUREG/CR-4829. Two volumes. Washington, D.C.: U.S. Nuclear Regu- latory Commission.

80 deals with probability uncertainties through operational practices, such as using the most protective containers avail- able for a given product, even if the container exceeds regula- tory requirements. Threat Whereas safety risk methodologies most often determine the likelihood of an accident based on historical data to cal- culate frequency and/or probability, security risk models and methodologies tend to use qualitative data to define an event’s likelihood as a function of threat and vulnerability. The most common sources of threat data tend to be from elicitations of internal or external subject-matter experts (SMEs) or internal intelligence agencies/divisions, due to a scarcity of or sensitive nature of historical event data. SME Elicitation. SME elicitations can come in many variations, but the core characteristic of the elicitation is to convene a discussion among several experts within the field of interest. For hazardous materials transportation, an elicitation may include first-responders, hazmat response team members, transport operators (drivers, barge captains, railroad engi- neers, etc.), federal and state and local law enforcement offi- cers, transport company security representatives, academic researchers, and governmental department of transportation representatives. The elicitation results’ structures vary depend- ing on the thought-experiment’s goal and the methodology employed. In some of the models described in this study, an SME elicitation is conducted at the beginning of the risk assess- ment to determine which attack scenarios, security areas, or target assets will be analyzed. For the most part, SME elicita- tions are used as a substitute for quantitative datasets. Unlike safety risk assessments, which often draw on historical data, security risk assessments often do not have robust histori- cal incident data; none of this study’s interview participants discussed any publically available threat dataset. Addition- ally, security risk assessments typically consider an “adaptive adversary,” who will shift targets and attack modes to optimize the adversary’s goal of a successful attack, often measured in consequences. Consequently, as additional iterations of SME elicitations are conducted, the results may change. Internal Intelligence Data Using “Black Boxes.” Another method of determining threat metrics used by hazmat secu- rity risk models is the use of “black box” threat data. The term “black box” is used because it is unclear what data are being used, how they are used, and/or the final result. Some stakeholders are hesitant, for business-proprietary reasons, to share information about perceived threats or how they calcu- late threat data. Additionally, government agencies use data that is classified security information that cannot be disclosed 200616) and truck release rates incorporated into the Fedtrak system (sourced from Harwood et al., 199317). The latter of these two sources, while continuing to be widely used in high- way transportation research, presents an opportunity for the development of improved data for use in contemporary risk analyses, owing to its age and lack of information on many high-priority hazmat transportation packaging options. Similarly, a portion of the probability data required by the IMESAFR tool is supplied by the software’s database. This information is derived primarily from military use and test- ing data, however, and could be improved for commercial analyses through expansion of the data to include probability information for commodities and quantities typically found in industrial uses. User-supplied probability data is a component of a major- ity of the available safety approaches. In some cases the supporting data is easily estimated simply by being on-site or having operational experience with the analyzing institu- tion, like in the case of estimating building characteristics in IMESAFR and various operational considerations in the qual- itative, semi-qualitative, and ranking methods of the CCPS Guidelines. In other cases, like in RADTRAN and the Pipeline Risk Management Manual approach, probability informa- tion may require difficult or costly measurements or calcula- tions, but may be easily estimated by using values or sources suggested in the model methodology documents. The chemi- cal industry models all require a high degree of user input to calculate or estimate probabilities, with quantitative methods tending to rely more heavily on in-house databases and more qualitative approaches drawing more from institutional or expert knowledge. While large chemical manufacturers often have the needed information on hand, or available through their carriers, to support these kinds of probability determi- nations, obstacles, such the proprietary nature of data and reluctance to share data, can make external data collection for analyses arduous. In general, hazmat transportation safety models handle uncertainty in probability data sources through the selection of conservative model parameter estimates or the use of con- servative operational procedures or equipment. For example, IMESAFR allows users to employ highly conservative model- ing of blast particle movements and intra-facility casualties to determine fatality probabilities. The large chemical/plastics manufacturer, whose methodology is profiled, meanwhile, 16 T. T. Treichel, J. P. Hughes, C. P. L. Barkan, R. D. Sims, E. A. Philips, and M. R. Saat, 2006, Safety performance of tank cars in accidents: prob- ability of lading loss, in: RSI-AAR Railroad Tank Car Safety Research and Test Project, Association of American Railroads, Washington, DC. 17 Harwood D.W, G. Viner, and E. R. Russel, 1993, Procedure for devel- opment truck accident and release rates for hazmat routing. Journal of Transportation Engineering, 119, 189-199.

81 ities and actions (RCRMS). Similar to threat data gathered from SME elicitations, vulnerability data is sometimes scaled (THTRA and Fedtrak use the Kent Scale) or scored relative to other vulnerability scenarios discussed. Two models (Readiness and Resiliency Assessment Frame- work and RCRMS) use datasets that either involve “black boxes” or their developers were unwilling to disclose their information for the study. Consequence The consequence component of the safety and security risk models presented in Section 4 focuses on one or more of four categories for each model: human, economic, environ- mental, or critical infrastructure. All but two of the models, the Readiness and Resiliency Assessment Framework and the CREATE Model, have a con- sequence component that explicitly focuses on impacts to the population exposed to an incident. These models incorpo- rate population-based consequences using various degrees of detail. On the most basic level are models that employ simple population counts for exposed persons. Such models include TRACC, TRAGIS, and the UIUC Tank Car Analysis, among others, who source spatial population data from the U.S. Census Bureau. Several models, such as Fedtrak, RCRMS, and GeoCTA add a level of precision to this population informa- tion by employing distinct daytime and nighttime popula- tion count data, which are most commonly sourced from FEMA’s HAZUS-MH software database. A number of the models investigated consider population consequences beyond simple counts of potentially exposed persons. For example, TSSRA and IMESAFR calculate casualty estimates in terms of the number of fatalities versus injuries. Casualty estimates are approached with the greatest level of detail, however, in models such as RADTRANS, CTRA, and most of the chemical industry assessment methodologies, by accounting for personal exposures through specific poten- tial pathways (i.e., respiration, ingestion, percutaneous). The high degree of detail in these models’ population consequence components tend to be particularly data-intensive and require the user to employ data that must be assumed through expert knowledge or estimated using additional models (e.g., plume dispersion models). Models that consider the economic consequences of an incident include a range of direct cost impacts, including the costs associated with replacing lost infrastructure, equipment, or products, as well as indirect economic impacts, such as traf- fic or product delays. Seven of the risk models consider the consequences of losing CIKR, which include assets or materi- als that are vital to the organization’s mission or goal. The loss of a CIKR asset could have substantial consequences across due to public safety concerns. In either case, it is unclear how this group of stakeholders calculates threat. Outside of the actual calculation or data gathering, the models identified in this project followed a similar process for defining scenarios and potential targets. The first step to obtain threat data is to identify and define the scenario to be analyzed. Within security risk assessments, threat is usually defined along the lines of a target’s attractiveness to an attacker and can be a function of the consequences of a successful attack (casualties, economic effects, symbolic or psychological impact, etc.). Of the security methodologies and models reviewed, those that identify potential targeted assets and attack scenarios used SME elicitations, potential-consequence analysis, internal information, or a combination of the three. After identifying the scenario, its threat score is typically obtained either from an internal intelligence group that does not disclose its methods and/or sources or through relative- scoring against the other scenarios and assets through SME elicitations. Additionally, some models account for proxim- ity to high-population areas (identified through Census or FEMA’s HAZUS-MH data), potential or unclassified critical infrastructure, or TSA HTUAs through either emphasizing the targets’ geographic locations to the SMEs or scaling the initial threat score. Fedtrak and RCRMS are two examples that embed this information into the methodology. Outside of these security-focused models, a few safety-based risk models have been adapted to measure security risk, usu- ally through scaling the safety risk score or components (fre- quency and probability). For instance, IMESAFR multiplies probability and frequency components by a scaling factor that is predetermined based on the DHS communicated public threat level. Vulnerability Similar to the threat component, vulnerability datasets are largely structured around SME elicitations. Instead of identify- ing potential attack scenarios, targets, and the threats associated with those items, however, stakeholders define vulnerability as the likelihood that a defined attack will be successful for a given target. The majority of the models assess vulnerabil- ity based on the likelihood the attack scenario will overcome target-specific countermeasures, while two models (Fedtrak and the CCPS’ Security Vulnerability Assessment Process) iden- tified in this project also consider the likelihood the attacker will fail on their own volition. In the case of vulnerability elicitations, some of the coun- termeasures that SMEs consider are detection equipment positions along hazmat routes (CREATE Model), emergency response rate information (Readiness and Resiliency Assess- ment Framework), and target or route-specific security activ-

82 mined for each alternative and then the relative differences between these values is used in the decision making pro- cess. Risk scores are often thought of as being unitless. If the risk for one alternative is 10 times that of another, there is no attempt to determine whether the higher-level risk is within an acceptable range and requires no mitigation efforts. Generally, the lower risk option will be chosen in this case, with all other parameters being equal. In safety analyses, risk is generally computed at a seg- ment level and aggregated along the length of a route. For security analyses, risk is generally computed at specific loca- tions along a route and the maximum value is taken as the overall risk for the route. This approach is used because an attacker will chose the location of an attack to maximize their intended impacts. Some adjustments may be made for routes that have more high-security risk locations than other routes, but these adjustments would not be linear. • Individual risk. This presentation of risk is used to indicate the aggregate risk from all sources (or scenarios) at a given location. • Societal risk. This method of communicating risk takes the risk index for each possible outcome and location and most often uses F-N curves. These curves will usually have the consequences along the x-axis and the frequency with which those consequences are expected to occur along the y-axis. As stated previously, it appears that the large majority of hazmat transportation quantitative risk assessments estimate relative risk values rather than absolute risk values. The latter, if computed, could be applied to established ranges for accept- ability. When this is done, the three most often used ranges are: (a) acceptable with no additional mitigation needed, (b) accept- able but mitigation is appropriate if it can be accomplished in a cost-effective manner (this might be called ‘tolerable risk’), and (c) unacceptable so that additional mitigation is neces- sary to bring the risk at least into the second range. Absolute values might also facilitate comparison of alternatives across dissimilar choices, such as comparison of routing across dif- ferent modes. While uncertainty is a concern expressed by many of those interviewed for this project, it seemed that little effort is made on the industry side to formally quantify it. Margins of error are assumed to exist and the risks of alternates need to be significantly different to support incurring additional costs to achieve the reported safety or security benefits. Depend- ing on the analysts’ perspective of the variability in the fac- tors that go into their assessment, they would adjust the error bounds that they would feel comfortable to use in making a distinction between alternatives. In the RCRMS, for exam- ple, the risk scores are grouped into ‘attractiveness’ categories such that any score in an attractiveness band is considered equivalent to any other in that same band. several areas, including, but not limited to, the economy, the stakeholder’s ability to function, and public safety. For exam- ple, a water treatment plant would consider the local railway as critical to its supply of chlorine, which is typically shipped via rail. Major bridges, tunnels, roadways, and transportation nodes may also be considered CIKR. Government agencies tend to reference the publically available DHS CIKR defini- tions, but the DHS CIKR list is classified and mainly focused on aggregate impact to the U.S. public. For private stakehold- ers, CIKR may include DHS CIKR listed and micro-assets, such as the railway referenced in the previous example. Additionally, economic and environmental consequence data can overlap in certain models as environmental conse- quences may be calculated in terms of hazmat clean-up costs or the opportunity costs associated with contaminated land; however, not all environmental costs are calculated in eco- nomic terms. Instead, some models in this study measure geo- graphic areas or features that are environmentally sensitive. For instance, the Boston Hazmat Route Evaluation measures environmental impact in affected acreage while RCRMS takes into account affected bodies of water and national park lands. Finally, two of the security risk assessments developed for TSA (THTRA and TSSRA) include a consequence factor for psychological or symbolic losses. By including this factor, TSA was hoping to account for unquantifiable losses that may occur as a result of an attack. The costs associated with rebuilding, replacing or repairing damaged infrastructure may not include the full impact that the accident or incident has on the general public. For instance, consider the loss of an iconic or historic building; while the building may be restored, the event may result in a psychological change in the public that cannot be easily quantified. While the two assess- ments included discussion on the psychological impact, the most current methodology does allow assessors to omit psy- chological consequences from the final score. 5.1.3 Interpreting and Applying Results Results from screening-level and qualitative analyses often are presented in the form of a general category (such as high, medium, or low), based on one or more elements (such as haz- ard, consequence, frequency, or risk), or in a risk matrix, with consequence and frequency as the two axes. Operations or alter- natives that approach the corner with the greatest consequences and frequency are those that warrant the most attention. For most quantitative models, the results are presented in terms of one of these types: • Risk indices. This type of result appears to be the most commonly used form and is almost always considered to be a relative risk value. In this method, a risk score is deter-

83 riers because hazmat shipments by highway are perceived to be restricted to routes predetermined by governing agencies. Furthermore, there are no significant regulatory incentives to compare potential alternative routes for highway shipments as there are with rail. As such, carriers cannot justify the cost of performing route analyses or developing analytical tools of their own. However, the existence of a widely available model with minimal burden on the user organization, in terms of input data, necessary expertise, and cost, would provide carri- ers a better understanding of the risks associated with current or planned shipments. Such knowledge would promote better informed shipment and operational decision making, benefit- ting company and public welfare. It is the goal of the TSA to address this gap through the continued development of the Fedtrak tool, at least in terms of Tier 1 HSSM. A gap may remain, however, for the analysis of other classes of hazmat. Tools for barge route analysis are similarly unavailable. Barge routes are strictly constrained by river geography. As with the perceived route restrictions in the trucking industry, the inflexibility of waterway paths may have hampered the development of barge risk analysis tools to this point. How- ever, as with trucking, shipment and operational decisions by barge carriers could be bolstered by a better understanding of the risks associated with shipment options. While the TRACC tool addresses situational awareness with regard to current shipments, no tool exists to calculate risk values for the plan- ning or comparison of individual barge shipments. 5.2.2 Gaps in Data for Models Inadequate Highway Exposure Data and Accident Rates There is a significant lack of detailed (i.e., disaggregated) exposure data on hazmat transportation shipments of vary- ing materials, packaging types, and operational parameters. Data sources could be improved by adding information that facilitates correlation with hazmat accident data to determine hazmat-specific accident rates. In many cases, the accident rates are desired for some segment of all hazmat transportation and that segmentation is not available. For local or regional intra- state assessments, truck accident rates may be available from the state department of transportation and are sometimes available at the roadway segment level (for roads over which that state has jurisdiction). In other cases, they are aggregated to roadway functional classification. Data on bulk shipments would be the most valuable on an industry-wide basis. Conditional Probability Data Available data on the conditional probabilities of release for containers used in highway, rail, and intermodal hazmat 5.2 Gaps 5.2.1 Gaps in Models for Decisions Multi-Modal/Intermodal Risk Analyses Other than those carried out by hazmat shippers, most risk assessments seem to focus on a single mode. There remains a lack of a suitable integrated model to consider multi-modal and intermodal transportation options. For example, there is a cur- rent focus of attention directed toward human factors issues as a leading cause of hazmat incidents. Because interchange/ transfer operations increase the handling that a shipment may experience, it is important to be able to model these operations when considering different shipment alternatives that may have varying levels and opportunities for these interactions. Validation of Prior Assessments From the models reviewed, it appears that when hazmat risk analyses utilize the results of previous analyses, that information is not typically subject to any type of validation. If the previous study’s results fill a need in modeling or data, they tend to be used without further scrutiny. Comparability of Model Results Most of the models covered in this document use varying methodologies and metrics, rendering their results incompa- rable to each other. More importantly, the differences make it more difficult to easily understand/interpret results from individual assessments that may differ from those that are more familiar to the user or decision maker. Uncertainty Most of those interviewed indicated that uncertainty is acknowledged but usually not quantified or even qualified (TSA and IME are notable exceptions). For some risk assess- ments, the practitioner will vary the values for one or more parameters that they feel have sufficient levels of uncertainty in order to understand the sensitivity of the assessment’s out- come on those parameters. This sensitivity analysis does not help in understanding the true uncertainty of the parame- ters, but does provide some confidence in how important the parameters are for the decision they are making. Route Analysis Tools There is currently a lack of analysis tools for truck-based carriers to use in determining the risks of transporting hazmat by a given route. Conversations with trucking industry repre- sentatives indicate that route analyses are rarely done by car-

84 Security Assessment Credibility and Transparency Several risk models rely on SMEs or involve classified or proprietary information or processes (black boxes) to identify, calculate, score, or compare risk components. Both processes are most often used to assess threat and vulnerability, which are security-risk components, because necessary datasets either are not publically available, do not contain relevant data to the risk-scenario, or contain information that could harm the business’ competitiveness or have an impact on public opin- ion or behavior. However, private entities may also employ a black box process for business-sensitive safety risk assessments. Both processes lead to gaps in credibility through the introduc- tion of biases and the reduction of transparency, which in turn reduces the ability to reproduce a risk assessment’s results. Black box calculations or datasets, which are not transpar- ent to the public, reduce the credibility of a risk assessment by hiding key processes or inputs into safety and security risk assessments. Black boxes may be employed across security and safety assessments and are used to protect information from external entities. Private risk assessors are reluctant to publically disclose the information or processes they use to access risk for several reasons. • Incident/accident data may be used against them by affected parties or by third parties that are involved in their business model or strategy. • A private entity’s risk-assessment process may also be con- sidered proprietary business information for competitive reasons or in the case that the private stakeholder considers risk assessments to be a marketable service. • Public risk assessors may label security-risk data as classi- fied for public safety issues. • Security-risk data may be withheld to prevent attackers from adapting their attack to other areas. Risk scores may inform resource allocations, so attackers may change their attack strategies and targets according to available information. • Black boxes may be employed to hide incomplete or skewed data that is being used because the desired data is not available. Whether the reasons for using a black box or not disclos- ing data are for market competitiveness, public safety, or that the dataset does not actually exist, the end result is a lack of transparency that hinders the ability of external entities to verify the model through replication. Typical safety-related datasets for frequency and probability are built on historical data pertaining to incidents. Security risk assessments, on the other hand, may consider previous events, but typically must employ hypothetical scenarios that do not have any historical data. In order to find suitable supporting data, many security risk assessments turn to SME elicitation to transport suffer from having a limited number of sources, from being outdated, or from having information gaps for particular commonly used containers. For example, highway hazmat risk analyses commonly use release rates estimated by Harwood et al. in 1993.18 This source, for which there is currently no comparable alternative, remains a staple of hazmat risk studies despite not accounting for any of the transportation technol- ogy, operations, or infrastructure innovations of the past two decades. An update of the Harwood report could, for example, account for release statistics on all high-priority hazmat con- tainers, including radioactive and explosive material packages. Development of conditional release probabilities for intermodal portable tanks and containers would add value to models such as the UIUC Tank Car Risk Analysis and RCRMS by enabling evaluation of a wider range of common container types. Available probability information for explosives risk analy- sis also presents several opportunities for further development. For example, explosives testing data concerning quantities likely to be present in commercial transport, as opposed to military applications, would enhance the applicability of analyses of industrial models, such as IMESAFR, as would conditional probabilities of release due to sympathetic detonation of explo- sives in close proximity to one another. Development of these data would enable more accurate risk analysis of transporta- tion facilities where explosives are stored, loaded, or unloaded and would contribute to a better understanding of the risks associated with packages of explosives in transit. Disparate Data Quality across Modes The degree of data accuracy and precision found in avail- able risk metric data tend to be dissimilar across modes. For example, rail carriers have access to accident rates that are spe- cific to their organizations, track classes, method of operation, and traffic density. Highway accident rates, on the other hand, are available only by roadway functional classification. Another example is the precision with which a shipment can be located during a movement from an origin to a destination along the relatively fixed-path modes of pipeline, rail, and barge, in com- parison to highways, where the street network provides a large number of potential paths. This variation in data quality further complicates the comparison of transportation options from mode to mode. Moreover, there is currently no methodology for calibrating or accounting for this disparity when comparing risk analyses for a shipment by different modes. The develop- ment of such a method would allow shippers and regulators to make better informed decisions about mode choice. 18 Harwood D.W, G. Viner, and E. R. Russell, 1993, Procedure for devel- opment of truck accident and release rates for hazmat routing. Journal of Transportation Engineering, 119, 189-199.

85 conducting risk assessments, which may result in inefficient risk-mitigation resource allocation and, with regard to hazmat risk consequences, negative effects on public safety. Public disclosure of intelligence reports or target-specific threat and vulnerability assessments will reduce their value by signaling that information to potential attackers. Validation of Supporting Data From the research conducted, it appears that where data are available, models that utilize that information perform limited validation on the prior work. If the data fill a need in modeling or data, they are used. This is particularly true for GIS datasets. 5.2.3 Gaps in Model, Data, or Results Availability The gaps listed in this section relate to those that are related to obtaining access to models or data that exist, but are pro- prietary or classified. Where gaps refer to the absence of such models or data, they are addressed in Sections 5.2.1 or 5.2.2. Formal Risk Management Process For transportation companies and hazmat shippers that consider risk assessments, the key to success is implementa- tion within a structured risk management system, program, or process within which to conduct risk assessments. Com- panies that implement the CCPS Guidelines or a similar approach are following such a system, but anecdotal evidence suggests there are many companies that do not. From the research conducted, it seems that some companies follow a less structured process with less formal risk assessments, even at the qualitative level. Data Building Blocks for Assessments For those entities that build their assessments internally using component pieces and not an integrated software prod- uct, there is a desire to have a repository of needed standard data, particularly geographically connected data that can relate potential exposure to affected populations and envi- ronmentally sensitive areas. Such a repository can include GIS data for day and night residential and employment pop- ulation, waterways, parks, and similar items. Transportation networks are already available from the BTS through their National Transportation Atlas Database (NTAD). Models can standardize the methodologies that integrate these ele- ments together, which may make it easier for companies to perform more quantitative analyses. However, having a data repository without accompanying standard implementation determine variables such as attack scenarios, potential targets, and relative threat and vulnerability scores. Additionally, SMEs may be used to verify model results or assumptions, but the inclusion of their opinions in the data-creation process intro- duces the potential for biases. While certain models attempt to mitigate SME biases through multiple elicitations with differ- ent SME groups, scaling results, and using three-tier scoring, an elicitation process has not been found to completely elimi- nate SMEs’ preconceptions. Thus, the data collected from an SME elicitation may change based on several factors, including the participants, the number of participants, the number of iterations of the elicitation, how the participants interact with one another, the elicitation process or how it is moderated, etc. As a result, the use of SME elicitations introduces a variable that is difficult to control that makes replication of results dif- ficult and decreases overall credibility. Lack of Public Vulnerability and Threat Data Another identified data gap is the lack of publically avail- able datasets for security and safety risk assessments. Several interviewed stakeholders stated that larger or better data is one desired improvement to their model or methodology. One potential solution would be to increase available data on fre- quency, probability, potential threats, and vulnerability to the same degree that consequence data can be obtained through U.S. Census. Since these data are not currently publically available, private entities that possess this information have a competitive advantage in conducting risk assessments and, therefore, an incentive not to share the data. Moreover, publi- cally shared datasets may reduce the use of SME elicitations and black boxes as attempts to mitigate, patch over, or cover up data deficiencies. The lack of publically available data has led to an increased importance being placed on models’ consequence components, which usually is the most well-defined and available dataset, especially in security risk assessments. For instance, Fedtrak and the CCPS Guidelines take potential consequences into account when identifying targets in their threat assessments. Another explanation for the emphasis on consequence could be the common assumption in the security risk field that an adversary is consequence-maximizing; however, that assump- tion may be made in an attempt to focus on data that is most readily available and understandable. Some potential attackers, such as those that have issues with particular organization or location, may direct their focus there rather than on other tar- gets with potentially greater consequences. Others may attempt attacks that offer the least resistance to maximize their chances of success and minimizing their likelihood of capture. A lack of publicly available data leads to imperfect infor- mation and can result in ill-informed and skewed risk assess- ments. Furthermore, data discovery increases the costs of

86 nation of the three that are used by government agencies are usually considered sensitive information and classified, mostly due to the intelligence data used to calculate the threat compo- nent. As described earlier, disclosure of threat scores may cause an adversary to adapt their attack strategy, which could then invalidate the model’s assumptions and results. Some govern- ment stakeholders, such as TSA, publish their risk methodolo- gies but withhold the data used to calculate risk. Furthermore, the final scores or results for government security-risk meth- odologies are normally communicated internally and only to external stakeholders on a need to know basis. Additionally, data is not always shared between all the stake- holders in a risk assessment. Again, stakeholders may have market-based incentives to withhold or skew/adapt data that they share externally. For instance a private carrier may attempt to understate accident information that is reported to the government (to prevent fines or avoid increased oversight), potential clients (to prevent losing customers and revenue), or general public (to prevent increased scrutiny or oversight). Withholding of critical information also occurs between the government and third parties. For instance, certain risk mod- els (e.g., TRACC, THTRA, TSSRA, CTRA, and Fedtrak) use classified CIKR information for identifying targets, high- consequence areas, and potential attack scenarios to be evaluated. While DHS releases definitions for what may be considered a critical infrastructure or key resource, the actual tiered list of CIKR sites is considered security sensitive infor- mation and is, therefore, not publically available. 5.3 Recommended Paths Forward 5.3.1 Recommendations for Model Development • Develop a single risk assessment approach across all modes using a standard architecture that would include a stan- dard (ideal or baseline) model for addressing hazmat trans- portation risk. Each relevant agency could augment the common approach with their specific area of expertise. Such a model could have varying, yet prescribed, levels of detail for each component, with data availability and cost determining which level that is actually implemented in a particular assessment. This could support both qualita- tive and quantitative assessments. The common approach could also include methods for measuring the validity and uncertainty in data and results and methods for conveying that information to decision makers along with the results. • Develop a highway hazmat route risk assessment tool that implements the standard approach described earlier and also considers the FMCSA Hazardous Materials Route Registry and other state and local truck restrictions. This tool, unlike RCRMS, would need the ability to suggest candidate routes for consideration given the significantly guidelines can still result in incomparable results from one implementation to another. Lack of Awareness of Available Data, Tools, and Methods Based on the interviews conducted, a knowledge gap about the availability of risk data, models, and methodologies was identified. Stakeholders were not always aware of the full range of available data sources or other hazmat stakeholders’ risk frameworks or methods. This knowledge gap can lead to risk decision-making results that are based on incomplete information or do not take advantage of fully developed risk analysis. Additionally, stakeholders may not be using resources in the most efficient manner. An increased awareness of avail- able data and models will prevent new risk assessors from start- ing from scratch and current stakeholders from using resources to research and implement improvements to their risk models that have already been employed by others. Additionally, this increased awareness would promote cooperation among all stakeholders in developing future improvements. Lack of Public Disclosure of Datasets One of the reasons for the knowledge gap between hazmat risk stakeholders is that developers and users are unwilling to disclose information regarding risk calculations or the data that is used in the model. Private stakeholders protect data and information through claims of business proprietary informa- tion and the need to maintain competitiveness in the market. For instance, risk models developed through funding from professional associations may only share the model, data, and results with members, which prevents the free-rider problem and incentivizes membership. Furthermore, private users may be averse to disclosing accident rates because the rates may be used against them by competitors or third parties (e.g., insur- ers, watchdog groups, government). AAR’s RCRMS model avoids this data gap by not allowing members access to other members’ data and only sharing limited information within the model with the FRA. By limiting the information acces- sible to users, RCRMS is able to provide a direct benefit to those stakeholders that supply information and fulfill a federal reporting requirement while guaranteeing information will not be divulged to competitors. Safety risk models developed and used by government stake- holders, such as the Boston Hazmat Route Evaluation and Pipe- line Risk Management Manual Risk Assessment Method, often are publically available to promote the model’s use and thereby decrease the risk posed to the public by hazmat transportation. In a few cases like the DOE-owned RADTRAN tool, the risk model or tool is available but requires the government to grant the user access. Security risk data, scores, models, or a combi-

87 in underreporting accidents across modes could be used to obtain a more normalized comparison of accident rates. This would involve a research project to develop a method- ology to estimate the levels of underreporting. • The hazmat transportation risk assessment community would benefit from the development of a guidebook that describes different types of elicitation methods and their applicability to the unique issues that are typically addressed in this field. This could include best practices, scenario devel- opment, how to present scenarios to SMEs to avoid intro- ducing biases, and appropriate qualitative scaling methods. • Further research is needed on the benefits that may be real- ized by sharing security-sensitive threat and vulnerability data with private sector risk assessors with a need to know, as well as the potential pitfalls and security risks that may result. If the outcome suggests that increased sharing is worth pursuing, the subsequent step is to study and iden- tify the best methods and safeguards to use. 5.3.3 Recommendations for Communication and Data/Model Sharing • Building on the success of the CCPS Guidelines and evolv- ing DHS risk assessment methodologies, an integrated framework document could be developed with more spe- cific checklists to encourage greater adoption of risk man- agement principles and decision making. Such a document would integrate the best practices from both the private and public sectors and facilitate their adoption. • Develop a single data repository for transportation network data that have the requisite data elements to support hazmat transportation risk assessment. This could also include a catalog of other relevant data sources that is kept evergreen. A less expensive alternative is to expand the catalog to have explicit pointers to all relevant datasets and forgo the repos- itory aspect for the data itself. For example, listings of data sources suitable for use in transportation risk assessment like FEMA’s HAZUS-MH daytime and nighttime popula- tion data would reduce the effort required by risk assessors looking to model their operations. greater extent of the highway network and the lack of oper- ational constraints with which the interlining railroads have to contend. • Develop a waterway hazmat route risk assessment tool that implements the standard approach described above and also considers dam and lock infrastructure restrictions. This tool could be developed, like the RCRMS, to only analyze prescribed route options. • Develop an approach for addressing low-probability, high- consequence events into hazmat transportation risk models. 5.3.2 Recommendations for Data Development • Continued enhancement of the BTS’ CFS and the FHWA’s Freight Analysis Framework (FAF) may ultimately provide sufficient national-level data on hazmat flows to support improved hazmat-specific highway accident rates. While material-specific rates are desired, rates that include all hazmat would be a marked improvement. Data on multi- modal hazmat flows, as an important subset of all com- modity flows, should be collected to support risk analysis as part of the CFS and FAF. • Expand and extend the results of HMCRP project HM-07, Accident Performance Data of Bulk Packages Used for Haz- ardous Materials Transportation, to develop conditional release probabilities for different packaging types, with the emphasis on bulk packages. At a minimum, using the dated Harwood et al. report from 1993 cited earlier to update their analysis would be beneficial. • Research on commercial-scale explosives would enhance the modeling for fixed facilities and eliminate the use of less-appropriate military explosives characteristics. Trans- portation risk assessments would also be able to benefit from this more accurate information about these materials when shipped in commerce. • Similar to the discussion for a standard risk assessment model, there would be benefits to developing a system for calibrating the differences in similar data across modes or other categories. For example, quantifying the differences