Being Prepared for IROPS: A Business-Planning and Decision-Making Approach (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

31 To implement the IROPS business-planning approach in a pragmatic way, this guidebook is accompanied by the IRIS decision support tool, an application developed in Microsoft Excel. IRIS uses AHP to prioritize and rank IROPS mitigation initiatives under consideration by an airport. IRIS provides a guided process that leads the user through each step in the business- planning approach and performs all necessary calculations, such as converting initial and recurring costs to a lifecycle cost and deriving the weightings used in AHP. This chapter describes how the decision-making approach identified in Chapter 3 is imple- mented in IRIS. IRIS has been designed so that no formal training in business planning or decision analysis methodologies such as AHP is required to use the software tool. Nonetheless, most users will find it helpful to first read Chapter 1 through Chapter 3 in this report as well as the business case analysis primer in Appendix A. Chapter 4 is organized to follow the sequence of tasks in IRIS. This workflow is defined in the sections that follow with an explanation of the purpose of each step. This chapter does not include detail on the mechanics of using IRIS, however. Information on how to install IRIS, save information, export results, and so on, is provided in the IRIS User Guide (Appendix B). A quick reference guide also is included on CRP-CD 144 for installation with the IRIS software, and it can be accessed from the main interface. 4.1 Before Getting Started with IRIS To ensure a smooth experience with IRIS, some preparations are necessary before running the application. These preparations include the collection of information that constitutes inputs to the IROPS business-planning approach. IROPS business planning is interdisciplinary and involves management, policy, planning, finance, emergency response, security, and safety functions at the airport; therefore, the inputs should be vetted with relevant personnel and/or departments. Alternatively, IRIS can be run in a group setting to allow consensus discussion on the subjective inputs to the tool while it is being used. Some of the inputs required by IRIS should be collected prior to starting. These are the inputs that define the portfolio under consideration. The portfolio is made up of the IROPS event(s), mitigation initiatives, initial costs, and recurring costs. The lifecycle duration to be used in the business plan and the cost escalation rate (if different from the default rate of 1.9%) should also be specified. A sample IROPS portfolio is shown in Appendix C. Subjective evaluations of effectiveness are an important contributor to the final results, so it is important that IRIS users be familiar with the effectiveness criteria and the associated rating scale. Users should review the hierarchy of the criteria shown in Figure 6 in Chapter 3 and the C H A P T E R 4 IROPS Investment Support (IRIS) Decision Support Tool

32 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach definitions of the criteria listed in Table 8. The scale used to rate the relative importance of the criteria is shown in Table 9. The definitions of the criteria and the rating scale can be displayed and printed from within IRIS. It may be useful to have a printed reference copy of this material while using IRIS, especially when using it for the first time. 4.2 IRIS Workflow The main IRIS user interface serves both as a visual map of the workflow and as the interface for executing each step in the process (see Figure 10). These steps are as follows: 1. Portfolio: Defining the IROPS investment portfolio. 2. Cost: Specifying the initial and recurring costs, which are used to determine the lifecycle costs of the investment initiatives. 3. Comparisons: Conducting the pairwise comparisons between each criterion in the AHP hierarchy. The pairwise comparisons are used to compute the values for weighting the effectiveness scores. 4. Effectiveness: Evaluating the effectiveness of each investment initiative using the criteria in each level in the AHP hierarchy. 5. Results: Rank ordering the investments initiatives in the portfolio, with an accompanying table showing the impact, effectiveness, benefit, cost, and combined scores. The implementation of the business-planning approach in IRIS uses the hierarchy and scoring definitions described in Chapter 3. For example, the AHP hierarchy is as shown in Figure 6, the criteria in each level of the hierarchy are as shown in Table 8, and the rating scale used for the pairwise comparisons used to calculate weightings is as shown in Table 9. Each step is implemented using a wizard (a common user interface technique that guides the user through a complex task). The wizard consists of a short sequence of dialog boxes (i.e., windows), accompanied by brief instructions, that solicit the input required to execute each task. The required calculations occur in the background as the user completes the wizards. Figure 10. Main IRIS user interface.

IROPS Investment Support (IRIS) Decision Support Tool 33 In short, IRIS can be said to consist of five wizards: The portfolio wizard, cost wizard, comparisons wizard, effectiveness wizard, and results wizard. Each wizard contains a “Save” button that can be used to save any partial progress made within the wizard. Pressing this button saves any information that has already been entered to the Excel file, and it will remain even if the user exits the wizard. IRIS also supports the file-saving functions embedded in Microsoft Excel (under the “File” menu). 4.2.1 Defining the IROPS Portfolio The portfolio wizard helps the user execute the first step in the workflow when using IRIS. This wizard is used to define the IROPS portfolio. The portfolio has two dimensions: (1) the IROPS events under consideration, and (2) the initiatives under consideration to potentially mitigate each event. The information that must be entered for each IROPS event under consideration includes: • Description: The name or identifying characteristics of the event (e.g., “International widebody aircraft diverted to airport”). • Likelihood: The user’s best estimate of the probability that the event might occur at the airport. The options are: – Extremely improbable – Extremely remote – Remote – Probable – Frequent • Severity: The user’s best estimate of the magnitude of the impact that the occurrence of the event would have on airport operations. The options are: – Minimal – Minor – Major – Severe – Catastrophic In theory, the business approach described in this guidebook can be used to consider any number of IROPS events and mitigations; however, IRIS has built-in restrictions on the maximum allowable number. Specifically, IRIS supports up to five IROPS events in one portfolio. For each event, a maximum number of five mitigations can be specified. This means IRIS can consider investment portfolios of up to 25 mitigations. This limitation is the result of constraints on the user interface and computational limitations. The business-planning problem under consideration should determine whether or not more than one IROPS event should be included when defining the portfolio. For example, if the airport is applying for a public safety grant that can be used for a broad range of IROPS-related mitigations, a portfolio with several events is likely to be the best choice. The events can be selected from among those that have the highest overall impact (i.e., combination of likelihood and severity). Conversely, if the airport wishes to address a known weakness in preparedness for a specific event (e.g., flooding), a single event should be used in the portfolio. Regardless of the number of events, up to five mitigations can be listed per event. Because the portfolio wizard is the first step in the IRIS business-planning process, it is a convenient point in the workflow to collect information that supports the entire process. This includes the file name that will be used whenever work in progress is saved. It also includes the duration of the lifecycle, specified in years. The lifecycle is usually tied to the expected physical lifecycle of the initiative (e.g., 10 years for a vehicle), but this is not always the case. Because the rankings are based on comparing mitigations against each other, each with different recurring

34 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach costs, the same lifecycle duration must be used for all mitigations across all events. Doing this ensures that the rankings in the portfolio are not biased by having different numbers of years of recurring costs when computing the lifecycle cost used in the decision-making analysis. The need for uniform lifecycle duration across the portfolio raises the challenge of how to handle initiatives with different service lives. For example, consider the comparison of an electronic communications system with a 10-year lifecycle against a pavement project with a 30-year lifecycle. Such disparities can be handled by including recurring costs that represent replacement or refurbishment costs for mitigation initiatives with shorter service lives. Such costs need to be converted into an average cost for each year in the lifecycle, but they can then be entered in the same way as any other recurring cost. This conversion is important to ensure that the results for all mitigation initiatives in the portfolio are comparable to each other. 4.2.2 Defining Lifecycle Costs Cost estimating is a discipline in its own right with its own set of software tools employed by cost analysts and engineers. To keep the process of defining lifecycle costs manageable while preserving the integrity of the cost-estimating process, IRIS uses a highly distilled structure for the input of costs. The lifecycle cost for each mitigation initiative is determined entirely through the following four inputs: • Initial Investment Cost: This input is the estimated initial investment cost for each mitiga- tion option, typically either a construction cost or an acquisition cost (e.g., in the case of equipment and vehicles). For planning projects, however, the initial investment cost may be the value of the labor hours spent by the airport’s staff. • Recurring Cost: This input is an estimate of the average cost expected to recur each year in the lifecycle. The estimate typically includes O&M costs, but may also include replacement or major refurbishment costs for investments with service lives shorter than the lifecycle duration. Replacement or refurbishment costs should be entered as the annual recurring cost, which can be estimated by spreading out the associated costs over each year in the lifecycle. • Lifecycle Duration: This input is the number of years in the economic lifecycle of the invest- ment analysis. Because the lifecycle duration is the same for all mitigation initiatives in the analysis, it is entered in the portfolio wizard instead of in the cost wizard. • Escalation Rate: This input is a cost escalation rate expressed as an estimated annual per- centage growth in cost. In the IRIS decision support tool, an escalation rate of 1.9% is used by default unless the rate is changed by the user. The default rate is based on the predicted year-to-year change in the chained price index provided by the Office of Management and Budget in its Analytical Perspectives: Budget of the U.S. Government (OMB 2013, 11). The most recent value can be found in the annual update of this document, if desired. These four inputs are used to calculate lifecycle costs by combining the initial cost with the recurring cost for each year in the lifecycle duration, with the cost escalation rate applied. The calculation is represented by the formula given in Equation 1: ∑ )(= + + − = C C c r i i n 1 (1)0 1 1 where C = lifecycle cost, C0 = initial cost, c = recurring cost, n = lifecycle duration, and r = escalation rate.

IROPS Investment Support (IRIS) Decision Support Tool 35 The resulting lifecycle cost for each initiative in the investment portfolio appears in the summary report in the results wizard. In addition to the lifecycle cost, which is expressed in dollars, a cost score is shown in the summary report. The cost score is simply the lifecycle cost normalized by the sum of all lifecycle costs in the investment portfolio. This score is compared against the benefit score, to rank order the portfolio. 4.2.3 Determining Stakeholder Preferences Through Pairwise Comparisons The fundamental concept of AHP is to decompose a decision-making problem into a hierarchy of sub-problems. In this case, the sub-problems are expressed as the evaluation of effectiveness criteria. The effectiveness criteria are structured into a hierarchy or decision-tree consisting of multiple levels, as shown in Figure 6. Within each level, the criteria are evaluated for importance and assigned a numerical score. Each level in the hierarchy represents a common area (e.g., “user benefits”) and is also evaluated. The resulting values are used as weightings for the evaluation of the criteria, and serve to indicate which decision areas are of highest priority to the user. A strength of the business-planning approach presented in this guidebook is that it combines objective measures with subjective ones. For example, lifecycle cost is an objective measure computed from the initial cost, recurring cost, lifecycle duration, and cost escalation rate. Conversely, the benefit side, including the estimation of impact and effectiveness, often involves subjective estimates. Using subjective measures allows the method to capture stakeholder preferences and the expertise that is held by the airport management team. It also solves the problem of quantifying benefits that otherwise might be impossible to monetize. At the same time, combining objective measures with subjective ones makes the process subject to biases, as is the case in any process that uses subjective input. To minimize biases and obtain a true evaluation of stakeholder preferences, the AHP method relies on pairwise evaluations. When conducting pairwise evaluations, two criteria are pre- sented at a time, to be evaluated against each other. In IRIS, for example, the user may be asked to compare the criteria “Airport Master Plan Alignment” and “Stakeholder Coordination” and indicate which is of higher priority when investing in airport improvements to mitigate IROPS. (An example of a prompt in IRIS to perform a pairwise comparison appears in Figure 7). By presenting the criteria in a pairwise fashion, the decision-making process is decomposed into elements that are tangible and more manageable. The goal is to obtain a more accurate mapping of the user’s preferences than would be obtained if the user were asked simply to assign a score to each criterion under consideration. In addition to removing bias, the process provides better differentiation between factors because the method forces the user to express clear choices. When performing pairwise comparisons, the user decides which criterion is more important and also grades the relative importance of the two using the 1 through 9 rating scale shown in Table 9. The mathematical algorithms in AHP use these ratings to derive numerical scores that are then used to weight the criteria when their level of effectiveness is evaluated. One challenge of the pairwise comparisons method is that the choices have to be logically consistent with each other. Even though the process is designed to elicit true stakeholder pref- erences, in practice it is possible to end up with entries that are logically inconsistent. This possibility is especially likely when the number of comparisons that must be made is relatively large. IRIS includes a test that checks the outcome of the pairwise comparisons for logical consistency. Although the test does not require absolute logical consistency, the entries must meet a reasonable threshold of consistency. If that threshold is not met, an error message is generated and the user is asked to review and edit the comparisons. If the user is unable to

36 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach identify and correct the error, a “Suggest Values” option is available to help the user. Selecting “Suggest Values” starts an iterative process that adjusts the user-entered values so that the final values are logically consistent. This option should only be used as a last resort, however, because the evaluation of stakeholder priorities is an important input to the decision analysis modeling implemented in IRIS. 4.2.4 Evaluating the Effectiveness of Each IROPS Investment Initiative Once the pairwise evaluations have been completed, the user is prompted to rate each mitigation initiative in the IROPS portfolio. An evaluation is required for each criterion in the AHP hierarchy. This evaluation uses drop-down menus to display a modified Likert scale from which the user can select an entry. In a typical Likert scale, a five-step rating scheme is used, as shown in Figure 11. In IRIS, these choices are replaced with choices that are relevant to the IROPS effectiveness criterion in question. For example, the criterion, “How long will this mitigation initiative take to procure/implement?” is rated using the five options “Very Long,” “Long,” “Average,” “Short,” and “Very Short.” When evaluating the criteria, more differentiation generally improves the performance of the model. If the mitigations in a portfolio receive evaluations with little variation, cost considerations may dominate the ranking of the portfolio. For this reason, users should not be reluctant in selecting evaluations from the extreme ends of the scale (e.g., “Very Long” or “Very Short” in the example above). The choices should be considered in relation to the scope of the portfolio. If a particular mitigation is better than the rest under any given criterion, it should be a candidate for receiving the highest score in the ranking system. After all effectiveness criteria have been evaluated, the selections are converted to numerical values from 1 through 5. These evaluations are combined with the weightings computed from the pairwise evaluations to calculate an overall effectiveness score for each mitigation initiative. The effectiveness score is the final input requirement prior to generating the results of the decision analysis. 4.2.5 Ranking the IROPS Investment Portfolio After all necessary inputs have been provided the results wizard can be used to generate the main output of IRIS—a rank ordering of the investment portfolio. The ranking prioritizes the initiatives using the combined score, which is an assessment of overall economic value that compares the benefit score with the cost score. For each entry in the portfolio, the impact, effectiveness, benefit, and cost scores are shown, along with the lifecycle cost in dollars. These values provide additional information that helps document the ranking of the IROPS mitigation initiatives. For example, the cost scores can help explain whether cost was the dominant factor in ranking the portfolio or whether the lifecycle cost of the highest ranked initiative was out- weighed by other considerations, such as impact or effectiveness. A sample summary report with a ranked IROPS portfolio is shown in Figure 12. In this particular example, lifecycle cost is, in fact, the dominant factor. In this case, the costs of the most expensive alternatives are driven by the recurring cost of the deicing agent over the Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Figure 11. Typical Likert scale.

IROPS Investment Support (IRIS) Decision Support Tool 37 lifecycle of 10 years. The impact (i.e., severity and likelihood) of the two IROPS events is found to be the same. Although some variation occurs in the benefit score, based on the outcome of the pairwise comparisons and evaluation of effectiveness, this variation is not sufficient to offset the large differences in lifecycle cost among the different initiatives. Notice that because only cost is measured in dollars, the units for the other scores are arbitrary. For this reason, IRIS normalizes most of the scores so that the values always fall between zero and 1. This normalization process applies to the impact, effectiveness, benefit, and cost scores. The combined score can exceed 1, however, because it is computed as the ratio between the benefit and cost score. As discussed in Chapter 3, the value of the combined score should not be interpreted as a traditional benefit-cost score. In particular, exceeding a value of 1 has no significance. What matters is the magnitude of the combined scores relative to each other. For this reason, the combined score is also expressed as a percentage of the sum of all combined scores across the portfolio. The percentage values can be used to judge how close two competing initiatives are to each other. The summary report is the main output of the IRIS decision support tool. In addition to the summary report, a number of additional reports are also created. Their purpose is primarily to document the inputs that went into the decision-making process. The reports available in the results wizard include: • Portfolio Report: This report lists the IROPS events and each mitigation initiative in the portfolio. For each mitigation initiative, the specified likelihood and severity are shown. • Cost Report: This report provides the initial and recurring costs for each mitigation initiative, as well as the lifecycle duration in years and the annual cost escalation (expressed as a percentage rate). The lifecycle duration is specified for the entire portfolio, so it is the same value across all mitigation initiatives. The final column shows the total lifecycle cost, which is computed from the other cost inputs. • Criteria Report: This report documents the outcome of the pairwise comparison for each criterion in the three categories in the AHP hierarchy (i.e., strategic challenges, user benefits, and tactical complexity). The report shows the comparison scale for reference. For each pair, the criterion judged to be more important is indicated by a score greater than 1 (as specified by the user). Criteria judged to be equally important are both shown with a score of 1. • Effectiveness Report: This report displays the evaluated level of effectiveness of each mitigation initiative for each criterion in the AHP hierarchy. The contents of these reports are explained in more detail in the IRIS User Guide (see Appendix B). The reports are generated as standard Microsoft Excel tables, which means they can be exported, printed, copied, or saved using the normal functionality of Microsoft Excel. Preprogrammed buttons for printing and saving also are provided in IRIS for users who are less proficient with Microsoft Excel. Finally, a button is provided to modify the inputs and recalculate the results. This capability supports the generation of alternative cases in which one or more inputs are adjusted, thus providing a basic “What if?” modeling capability. This capability also can be used to conduct sensitivity analyses of the results to any of the inputs used in the decision-making process. Figure 12. Sample summary report.