Being Prepared for IROPS: A Business-Planning and Decision-Making Approach (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

21 As a described in Chapter 2, in a benefit-cost analysis the lifecycle costs of a mitigation initiative would normally be compared against a monetized evaluation of benefits. The basic objective is to provide decision makers with the net financial value of a project. This process can be labor intensive and time consuming, however, especially if many alternatives need to be considered or if the level of uncertainty is unusually high. Also, in many cases, the finances of a project are not the only considerations. The definition of benefits can vary, not all aspects may be possible to monetize, or levels of uncertainty may be higher than normal, and stakeholder preferences may need to be included. In these cases, decision analysis (or decision theory) can be used as an alternative to other techniques. Decision analysis identifies the factors that enter into a decision, and quantifies their impacts and uncertainties to arrive at an optimal decision. The method can be implemented using dif- ferent mathematical techniques to identify the key factors of a decision and then recommend a course of action. First, a framework is defined that models the decision-making process, includ- ing the key factors and alternatives being considered. The selected mathematical method is then used to quantify the effectiveness of each potential decision. The results can be used to rank the initiatives and present recommendations to the decision maker. 3.1 The Analytic Hierarchy Process AHP helps decision makers structure a decision problem into layers of sub-problems. The resulting layers form a hierarchy, which gives the method its name. AHP is a specific appli- cation of multi-criteria decision analysis (MCDA), a discipline within operations research (Saaty 1980). The goal of MCDA is to support decision-making under multiple and opposing criteria—in this case, cost of a mitigation initiative, likelihood and severity of an IROPS event, and the effectiveness of the proposed mitigation. AHP augments traditional benefit-cost analysis to provide a comprehensive evaluation of each potential mitigation initiative. Under AHP, decisions are evaluated using a mathemati- cal formulation of effectiveness that takes into account both quantitative analysis and human judgment. Incorporating both perspectives has the advantage of allowing the analyst to quickly examine decisions that would otherwise be difficult to measure. The resulting evaluation of effectiveness replaces the monetary evaluation of benefits that would be used in a traditional benefit-cost analysis. Figure 2 diagrams the high-level approach for determining the impact of IROPS events, estimating the costs, and evaluating the impact of mitigation initiatives. The impact of an IROPS event refers to the potential severity of the consequences. For exam- ple, at an airport that is used to winter weather conditions, an icing event may be disruptive, C H A P T E R 3 IROPS Business Planning Using Decision Analysis

22 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach but not severely so. At an airport that normally does not experience snow or ice, however, an icing event has the potential to essentially shut down the airport for several hours, resulting in extended tarmac delays. If all other factors are equal, a mitigation initiative that mitigates IROPS events with more severe impacts should be ranked higher than alternatives that address less severe ones. Impact must also take into account the likelihood of occurrence: An IROPS event with a severe impact that is unlikely to occur should carry less weight in the decision-making process than one with a lower impact but that is much more likely to happen. In this decision analysis framework, benefits cannot be compared directly against costs because the benefits are not monetized. Instead, a notional form of benefit is used, which com- bines impact and effectiveness: = ⊗Benefit Impact Effectiveness, where the operator ⊗ represents a positive interaction between impact and effectiveness. In gen- eral, the greater the effectiveness of the mitigation initiative, the greater is the resulting benefit. Similarly, the greater the disruptive impact of an IROPS event—and the more likely the event is to occur—the greater is the benefit of investing in a mitigation initiative. Using the decision analysis approach allows both impact and effectiveness to be quantified. This, in turn, allows for the benefit of an IROPS mitigation initiative to be quantified. The benefit values can then be compared against relative cost to determine which alternatives provide the best investment. (Relative cost is used since the comparisons are not made in monetary terms.) The remaining sections of this chapter describe how these concepts are combined to design a business-planning approach for IROPS. The description uses a notional example that illustrates application of the decision analysis approach. Chapter 4 describes the particulars of how the approach is implemented in the Microsoft Excel-based decision support tool IRIS. 3.2 IROPS Investment Portfolio The basic element being evaluated under this approach is the IROPS investment portfolio. Here, the term portfolio refers to a list of alternative mitigation initiatives under consideration. The airport management team determines which IROPS events and impacts they would like to Figure 2. Decision analysis approach for IROPS business planning.

IROPS Business Planning Using Decision Analysis 23 address and then develops potential mitigation initiatives to address their concerns. The com- pletely user-defined portfolio defines the investment options under consideration by matching potential mitigation initiatives to the IROPS events and impacts they are designed to address. Figure 3 provides an example of an IROPS investment portfolio. 3.3 IROPS Impact Contingency planning includes the unique challenge of addressing a variety of events that often occur very infrequently and have a wide range of impacts on operations. The objective of this step in the process is to determine both the frequency and impact of each IROPS event under consideration. This step defines the opportunity space for the mitigation initiatives. For example, using scarce resources to develop an optimal mitigation approach for an IROPS event that is extremely likely to occur but has a minimal disruptive impact may not be in the best interest of the airport. For this reason, the business-planning approach requires both an esti- mate of the likelihood (or probability) of an IROPS event occurring and an evaluation of its impact on airport operations. The overall severity of an IROPS event can be thought of as a combination of two things: (1) the likelihood of the event happening and (2) the impact on the airport and its customers if the event does occur. This combination can be expressed through a conceptual impact rating, which is written as follows: = ⊗Impact Likelihood of IROPS Event Severity of IROPS Event. Such evaluations of likelihood versus severity are commonly used in the risk management field. The approach presented in this guidebook for IROPS business planning uses the risk assessment matrix from the FAA draft Advisory Circular Safety Management System for Air- ports (FAA 2012b, 30) and is shown in Figure 4. This matrix is substantially similar to the risk assessment matrix used in ACRP Report 74: Application of Enterprise Risk Management at Air- ports (Marsh Risk Consulting 2012, 31). For each event in the investment portfolio, the user provides subjective evaluations for both the likelihood and the severity of the event. Each rating is quantified using a five-step scale as shown in Figure 4. The intersection of the likelihood rating and severity rating determines the overall level of risk using a three-step scale (i.e., low, medium, and high). This risk level is then assigned a numerical value for the quantitative assessment of impact of each IROPS event associated with the mitigation initiatives in the IROPS investment portfolio. Figure 3. Sample investment portfolio for IROPS mitigation initiatives.

24 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach 3.4 Costs As described in Chapter 2, the cost input required for the IROPS mitigation business-planning approach is the same as that used for traditional benefit-cost analyses. The total lifecycle cost is needed (in IRIS it is entered as an initial acquisition cost and a recurring O&M cost). Lifecycle cost estimates include the total costs to acquire, implement, operate, maintain, tech refresh, and dispose of the proposed initiative. The lifecycle must be selected commensurate with the measures of effectiveness. Also, to allow for an economic analysis to be performed, the same lifecycle must be used for all alternatives in the IROPS investment portfolio. It is expected that airports will obtain costs from vendor quotes, existing engineering esti- mates, or through the separate development of a cost estimate. Figure 5 presents a typical cost- estimating process. General best practices for developing cost estimates are described in the business case analysis primer in Appendix A. This guidebook is not, however, intended as an exhaustive reference on cost estimating. Users of IRIS may need to refer to other sources for some cost data, depending on the scope and complexity of the proposed mitigation initiative. Low Low Low Low Low Low Low Low Low Low Low Medium Medium Medium Medium High High High High High High High High High High Medium Figure 4. Risk matrix from FAA Safety Management System for Airports. CARD = cost analysis requirements description; CES = cost-estimating structure; WBS = work breakdown structure. Figure 5. Generalized cost-estimating process.

IROPS Business Planning Using Decision Analysis 25 For complex mitigation initiatives (e.g., those involving construction), the airport would nor- mally be expected to have a cost estimate developed as part of its long-term capital planning process. Cost estimates from the ACIP can be used as inputs to the decision support tool, although they may need to be supplemented with O&M and upgrade costs. For these reasons, the IROPS business-planning approach uses cost estimates at a highly dis- tilled level. The user provides initial costs and recurring costs, with single fields for total acquisi- tion costs and O&M costs. For simpler mitigation strategies, these totals may be readily derived using simple arithmetic in a spreadsheet. To derive the cost inputs for more complex mitigation strategies, the standard cost-estimating practice is to break down total system costs into a stan- dardized work breakdown structure (WBS). The WBS facilitates the comparison of alternatives, as differences in the individual cost elements are readily visible. To simplify the exercise for airport operators, use of an abridged WBS such as the one presented in Table 7 is suggested. 3.5 Evaluating the Effectiveness of IROPS Mitigation Initiatives The rarity and potential severity of IROPS events prevents the use of traditional data-driven benefits or similar measurements of effectiveness, especially when designing a business- planning approach for use by a broad range of airports. The approach presented in this guide- book leverages AHP, an established decision analysis technique. It merges the stakeholder’s Acquisition Costs Year 1 Year 2 Year 3 … Last Year Total Cost Program office management Research and development Site surveys Engineer technical solution Purchase COTS materials, equipment, software Develop/customize software Site preparation Construction of facilities Utilities and telecommunications Installation Testing and activation Documentation Training Operations and Maintenance (O&M) Costs Year 1 Year 2 Year 3 … Last Year Total Cost Program office management System operations Maintenance labor Maintenance parts Utilities and telecommunications Fuel Periodic tech refresh Modifications Training Total Costs Table 7. Simplified WBS.

26 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach objectives with their evaluation of the problem to analyze complex business-planning problems in a structured way. As applied here, the effectiveness of IROPS mitigation initiatives is assessed in three distinct steps: 1. Review and understand the evaluation criteria. 2. Weight the criteria to incorporate stakeholder preferences. 3. Evaluate each mitigation initiative against the criteria. A common understanding of the predetermined evaluation criteria is essential among evalu- ators and serves as the foundation of the effectiveness rating process. At the core of an AHP- based approach is the hierarchy that breaks down the decision analysis process into distinct manageable parts. The hierarchy adopted in this guidebook evaluates the effectiveness of miti- gation initiatives across a set of criteria categorized into three major focus areas: strategic chal- lenges, user benefits, and tactical complexity. Figure 6 illustrates the hierarchy. Definitions for each criterion in the hierarchy are listed in Table 8. One of the most valuable aspects of AHP is allowing each evaluation to be driven by the preferences of the airport management team that is executing the approach. This flexibility enables AHP to be used for evaluation by a wide range of airports that have varying concerns, while still providing a structured and predetermined approach. After reviewing the criteria, the airport management team must determine which evaluation criteria are most important to its concerns. This ranking of the criteria establishes the most influential factors in analysis. The IROPS business-planning approach uses pairwise comparison to measure and quantify stakeholder preferences. This method assists in deriving clear preferences, as other approaches tend to include too little differentiation. Pairwise comparisons also ease understanding of the evaluation process by providing simple one-on-one comparisons. A sample pairwise compari- son is shown in Figure 7. A numerical scale is used to value the relative importance of each pair of choices. The rating scale is based on the perception and experience of the respondent (Table 9). It is important to remember that the pairwise comparisons are based on human judgment and that stakeholder preferences can change over time. Consequently, different evaluations of criteria can yield dif- ferent results. Figure 6. Hierarchy of effectiveness criteria.

IROPS Business Planning Using Decision Analysis 27 Evaluation Criteria Description Strategic Challenges Airport Master Plan alignment How well does the mitigation initiative align with the current Airport Master Plan? Was it already considered in the capital plan? Is it a completely new concept? Funding availability How accessible will funding be for this initiative? Does it qualify for a federal grant, PFC funding, PFC-backed bonds or other public funding? Airport-generated funds? Does this create a significant ongoing operational expense to the airport? Stakeholder coordination How many stakeholders must be involved for this initiative? What level of coordination is required across different interested parties? What are the potential related complications? What is the impact on rates and charges? Airline use agreements? Implementation timeline How long will the initiative take to procure/implement? User Benefits Reduction in airline impact How will this mitigation initiative reduce disruption to airlines in terms of time? Consider flights delayed, missed connections, crew scheduling, and extended tarmac delays. Reduction in traveler delay How will this mitigation initiative reduce delay experienced by travelers? Consider the value of the travelers’ time, missed connections, and baggage lost. Traveler comfort What level of comfort can be provided to travelers during the IROPS event? Consider access to food, water, bathrooms, cots, telephone, the Internet, airport/airline information, onsite overnight accommodations. Consider impacts on travelers with special needs and persons with mobility impairments. Improvement in airport operations How does the mitigation strategy impact the work conditions for airport staff during IROPS events? Tactical Complexity Disruption level during implementation What level of disruption will this mitigation initiative cause to normal airport operations when it is in effect? Note: This criterion should not include disruption associated with the acquisition/construction of the initiative (e.g., temporary disruptions because of construction activity). Execution response time How quickly can this mitigation initiative be executed to address the IROPS event? Policy and regulatory compliance difficulty How difficult will it be to maintain policy and regulatory compliance during the execution of the mitigation initiative? Consider security, Federal Aviation Regulations Part 139, safety, and so forth. Table 8. Definitions of IROPS mitigation effectiveness criteria. Figure 7. Sample pairwise comparison.

28 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach The AHP method requires that the pairwise valuations maintain internal logical consis- tency. For example, it is not possible that initiative A be more important than B, initiative B be more important than C, and initiative C be more important than A. For this reason, the IRIS decision support tool includes a validation process of the pairwise comparisons. The validation test computes a consistency ratio that measures the extent to which the pair- wise comparisons are logically consistent. The resulting ratio cannot exceed a prespecified error threshold. If the error threshold is exceeded, the user will be prompted to repeat the pairwise evaluation or to use suggested values provided by the decision support tool. This feature of the tool prevents the user from executing the decision analysis with inconsistent choices in place. After each pairwise comparison is evaluated for relative importance, the responses are auto- matically encoded as shown in Table 10 and then converted into matrix form. The resulting scores are used to estimate the airport’s stakeholder priorities by using them as weighting fac- tors in the evaluation of the effectiveness of each IROPS mitigation initiative. An example of the processed results is shown in Figure 8. The final step in assessing the effectiveness of mitigation initiatives is to evaluate each initia- tive within the IROPS investment portfolio against the criteria applied to the associated IROPS event(s) and impact(s). Initiatives are scored through a subjective evaluation of how effective each proposed initiative is at mitigating the impact of the IROPS event. A Likert scale modified for this purpose is used to obtain a quantitative assessment of effectiveness. For example, for the criterion “Airport Master Plan Alignment,” the IRIS decision support tool poses the following question to the user: How well does the mitigation initiative align with the current Airport Master Plan? Intensity of Importance Definition Explanation 1 Equal importance Two elements contribute equally to the objective. 3 Moderate importance Experience and judgment moderately favor one element over another. 5 Strong importance Experience and judgment strongly favor one element over another. 7 Very strong importance One element is favored very strongly over another; its dominance is demonstrated in practice. 9 Extreme importance The evidence favoring one element over another is of the highest possible order of affirmation. Note: Intensities of 2, 4, 6, and 8 can be used to express intermediate values. Intensities of 1.1, 1.2, 1.3, etc., can be used for elements that are very close in importance. Table 9. Rating scale for pairwise comparisons. Category: Strategic Challenges Criterion Score Criterion Score Airport Master Plan alignment 1 Funding availability 1 1 Stakeholder coordination 5 1 Implementation timeline 7 Funding availability 1 Stakeholder coordination 7 1 Implementation timeline 9 Stakeholder coordination 1 Implementation timeline 1 Table 10. Sample analysis of pairwise comparisons.

IROPS Business Planning Using Decision Analysis 29 For each question, a five-step scale is provided to evaluate the effectiveness of the criterion in question. In this example, the user would be given the following response options: 5 = Very Well 4 = Well 3 = Adequately 2 = Poorly 1 = Very Poorly Evaluations for each possible pairing of a proposed mitigation initiative with its associated IROPS event and impact are then computed by incorporating the effectiveness evaluations as weighted by stakeholder priorities. This process is repeated to iterate through each element in the investment portfolio. An overall effectiveness score for each initiative is calculated by apply- ing the weights computed from the pairwise comparisons to the evaluation of each effectiveness criterion. The results for a sample IROPS investment portfolio are shown in Figure 9. 3.6 Rank Ordering the Investment Portfolio The final step in the IROPS business-planning approach is the rank ordering of each initia- tive in the investment portfolio. This ranking compares the estimated benefit of each initiative against its lifecycle cost. As described previously, the evaluation of the overall benefit of each Figure 8. Sample computation of weightings of effectiveness criteria. 57 43 46 54 25 75 Figure 9. Computed effectiveness scores (relative weights) for sample IROPS investment portfolio.

30 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach initiative is computed from its rated effectiveness and the impact score of the IROPS event. The impact score, in turn, is based on the estimated likelihood and severity of the event. Given that the resulting benefit score does not represent a monetary value (as it would in a traditional benefit-cost analysis), it cannot be directly compared against a monetary cost. Con- sequently, a relative cost score is computed for each initiative in the investment portfolio. This cost score is the lifecycle cost of the initiative divided by the total lifecycle cost for all initiatives. The ranking of the IROPS investment portfolio is based on the ratio of the benefit score to this relative cost value. The higher this ratio, the more benefit is received for each dollar invested. Notice that in a traditional benefit-cost analysis, the value of the benefit-cost ratio can be used to determine whether or not the benefits are likely to exceed the cost of the investment. This occurs if the benefit-cost ratio is greater than 1. This sort of test is not possible for the ratio used to rank the IROPS investment portfolio. These ratios can be used to rank the relative merit of each IROPS mitigation initiative, but because benefits are not monetized in this method, the absolute value of the ratio conveys no meaning. The ratio should not be interpreted as a benefit- cost ratio.