Prioritization of Public Transportation Investments: A Guide for Decision-Makers (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

41   Demonstration of Cross-Modal Prioritization is section of the guide includes a pilot demonstration illus- trating dierent approaches to prioritizing capital projects across multiple modes. e pilot demonstrates how measures, representing multiple objectives with various units of analysis, can be combined in a quantitative prioritization framework to identify funding priori- ties. e pilot includes a single set of projects with two approaches for prioritization: a data-intensive approach with multiple quantitative measures and a streamlined approach with fewer, more qualitative measures. e pilot was implemented in a spreadsheet and used to populate the Multiple Objective Decision Analysis Tool (MODAT), a web-based, publicly available tool hosted by the American Association of State Highway and Transporta- tion Ocials (AASHTO) for using in investment prioritization to provide further illustration of results (AASHTO 2021). Additional information on this tool, including previously developed training materials and guidance, is also available on the AASHTO Transportation Performance Management Portal (AASHTO n.d.) e following sections detail the objectives and measures, test projects, analysis approach, analysis results, and conclusions from the pilot demonstration. Section 6.1 Objectives and Measures identies a set of prioritization objectives (e.g., mobility, equity) selected for the pilot based on examples from practice. It also outlines the specic quan- titative and qualitative measures selected under each objective to demonstrate a data-intensive and less “data-hungry,” streamlined approach. Section 6.2 Test Projects describes the set of 20 multimodal projects developed to demonstrate prioritization in the pilot. ese projects were adapted from actual projects in two states but include anonymized data. e pilot analyzes how dierent prioritization approaches lead to dierent project rankings. Section 6.3 Analysis Approach walks the reader through the steps of prioritization. is includes analytical steps of (1) scaling qualitative measures with ordinal (e.g., 1–5) scoring; (2) normalization so all values lie on a scale from 0–100%; and (3) computing a score for each project based on a weighted combination of objectives and measures. is section also denes a series of analysis scenarios with dierent objectives and measure weights to demonstrate the type of sensitivity testing that agencies can conduct before settling on their nal distribution of weights. Section 6.4 Analysis Results presents and discusses the results for the various prioritization scenarios. is includes a discussion of dierences between the streamlined and data-intensive cases and of the impact of cost, criteria, and measure weights on project ranking. C H A P T E R 6 The pilot demonstration shows how measures, representing multiple objec- tives with various units of analysis, can be combined together in a quantitative prioritization framework to identify funding priorities.

42 Prioritization of Public Transportation Investments: A Guide for Decision-Makers Section 6.5 Pilot Demonstration Conclusions summarizes the results from the pilot, which include the nding that project ranking can be very sensitive to the absence of an important objective and that one can approximate the results of a data-intensive prioritization approach using a streamlined, less sophisticated set of measures. 6.1 Objectives and Measures e objectives included in the pilot prioritization process were identied based on the research team’s review of practice, and the measures were selected to be similar to actual measures used in practice. e objectives are as follows: • Mobility • Safety and Security • Stewardship (omitted with the streamlined approach) • Environmental Performance • Economic Development • Equity • Consistency with Agency Policies and Plans Agency examples reviewed to guide the development of the pilot include: • Atlanta Transit Link Authority (ATL) • Baltimore Metropolitan Council (BMC) • Broward County Metropolitan Planning Organization (MPO) • Chicago Metropolitan Agency for Planning (CMAP) • Delaware Valley Regional Planning Commission (DVRPC) • Maryland Department of Transportation (MDOT) • Metropolitan Transportation Commission (MTC) • Oregon DOT (ODOT) • Virginia Oce of Intermodal Planning and Investment (OIPI) Table 9 presents the selected measures and their denitions for the data-intensive case. In the table, measures are organized by objective. e table also lists examples of similar measures used by the agencies listed above. For this case, seven objectives and 14 measures are identied. For Mobility, three measures are dened, and for Consistency with Agency Plans and Priorities, a single measure is dened. For each of the remaining ve objectives, two measures are dened. e measures include a mix of quantitative measures and scores that are assigned qualitatively based on a review of the project scope and eects. e qualitative measures are all dened to be scores that range from 0 to 100. ese are identied with an asterisk in the table. Table  10 presents the selected measures and their denitions for the streamlined case. In this case, there are six objectives and seven measures identied. For Mobility, two measures are dened, while for each of the other objectives, a single measure is dened. All of the measures for this case are qualitative scores, demonstrating how project values may be estimated with limited complexity based on the presence or absence of dierent project elements. While few agencies use only qualitative scores in their prioritization, many select a combination of dierent score-based variables to stand in for charac- teristics too resource-intensive to measure. Quantitative Measure A measure based on a numeric value that represents the measurable size or quantity. Examples include travel-time savings and reduced fuel consumption. Qualitative Measure A measure based on expert judgment and/or non-numeric data. Frequently, such measures are expressed using a numeric scale, such as a rating on a scale from 1 to 5 established by expert judgment.

Demonstration of Cross-Modal Prioritization 43   Mobility Travel-Time Savings (hours/day) Total travel-time savings in hours per day • Virginia OIPI: Person-Hours of Delay Increased Transit Ridership (passengers/day) New transit passengers per day • DVRPC: Multimodal Use • Maryland DOT: Daily New Passengers Multimodal Mobility Improvement Score* Score reflecting mobility for non- highway modes • Oregon DOT: Integration of Transit Services • Virginia OIPI: Access to Multimodal Choices • Broward County MPO: Impact on Multimodal Connectivity Safety and Security Crash Reduction (crashes/year) Anticipated reduction in equivalent property damage only (EPDO) crashes in the project area • Virginia OIPI: EPDO of Fatal and Injury Crashes Safety and Security Score* Score reflecting the count and extent of improvements to safety and/or security • BMC: Complete Streets Features & Evacuation Route or Parallels • Maryland DOT: Reduction in Fatalities and Injuries & Complete Streets Stewardship Increase in Asset Useful Life ($) Increase in asset value for existing assets resulting from the project • DVRPC: Facility/Asset Condition and Maintenance • ATL: Increased Useful Life • Maryland DOT: Facility Lifespan Asset Risk Reduction Score* Score reflecting reduced risk resulting from the project, such as a reduction in assets vulnerable to flooding • Broward County MPO: Improvements Related to Sea Level Rise Mitigation/Extreme Weather Resiliency • Maryland DOT: Facility Resiliency Reduced Fuel Consumption (gallons/year) Reduction in fuel consumption resulting from the project, either from congestion reduction or shift to non-highway modes • Maryland DOT: Emissions Reduction Environmental Performance Score* Score reflecting environmental improvements incorporated in the project • CMAP: Environmental Quality Score • Virginia OIPI: Air Quality and Energy Environmental Effect Measure Description Examples Environmental Performance Table 9. Objectives and measures – Data-Intensive Case. (continued on next page)

44 Prioritization of Public Transportation Investments: A Guide for Decision-Makers Note that all of the measures except one—the Highway Mobility Improvement Score—also appear in Table 9. Further, note that relative to the data-intensive case, in the streamlined case, the Stewardship objective is omitted. is omission was made to reect what the team observed in the review. Namely, many of the examples reviewed in the research omitted this objective, focusing on expansion investments rather than SGR or preservation investments where this objective is most applicable. 6.2 Test Projects A set of 20 projects was developed for the pilot. e projects were adapted from actual proj- ects identied by agencies in two states. Original project names and descriptions have been anonymized. Where available, the analysis uses measure values from actual projects. Because no one agency uses the specic combination of measures listed in Table 9 and Table 10, these data Economic Development Increased Job Access (jobs) Increase in number of jobs accessible to households within a specified commute time • Maryland DOT: Job Accessibility • Virginia OIPI: Access to Jobs Economic Development Score* Score reflecting the extent to which the project supports the local economic development and serves as a catalyst for future development • BMC: Connection to Priority Funding Area • ATL: Re/Development Potential • Virginia OIPI: Project Support for Economic Development Equity Increased Job Access for Disadvantaged Areas (jobs) Increase in the number of jobs accessible to disadvantaged households within a specified commute time • Maryland DOT: Job Accessibility for Disadvantaged Populations • Virginia OIPI: Access to Jobs for Disadvantaged Populations Increased Modal Accessibility Score* Score reflecting an increase in travel modes accessible to disadvantaged households (may apply to new service or enhancement of existing service) • Oregon DOT: Social Equity • CMAP: Equity • ATL: Existing Population – Communities of Interest • DVRPC: Equity Consistency with Agency Plans and Policies Plans and Priorities Score* Score reflecting the level of consistency and continuity with agency plans and priorities, including land-use plans, long- term plans, and other documents identifying priority investments • ATL: Regional Integration • MTC: Guiding Principles Assessment • Maryland DOT: Local Priorities * = Qualitative Measure. Measure Description Examples Table 9. (Continued).

Demonstration of Cross-Modal Prioritization 45   Measure Description Examples Mobility Highway Mobility Improvement Score Score reflecting improvement to mobility for the highway mode • BMC: Highway Level of Service (LOS) Multimodal Mobility Improvement Score Score reflecting mobility for non-highway modes • Oregon DOT: Integration of Transit Services • Virginia OIPI: Access to Multimodal Choices • MTC: Accessibility • Broward County MPO: Impact on Multimodal Connectivity Safety and Security Safety and Security Score Score reflecting the count and extent of improvements to safety and/or security • BMC: Complete Streets Features & Evacuation Route or Parallels • Maryland DOT: Reduction in Fatalities and Injuries & Complete Streets Environmental Performance Environmental Performance Score Score reflecting environmental improvements incorporated in the project • CMAP: Environmental Quality Score • Virginia OIPI: Air Quality and Energy Environmental Effect Economic Development Economic Development Score Score reflecting the extent to which the project supports the local economic • BMC: Connection to Priority Funding Area • ATL: Re/Development Potential development and serves as a catalyst for future development • Virginia OIPI: Project Support for Economic Development Equity Increased Modal Accessibility Score Score reflecting the increase in travel modes accessible to disadvantaged households (may apply to new service or enhancement of existing service) • Oregon DOT: Social Equity • CMAP: Equity • ATL: Existing Population – Communities of Interest • DVRPC: Equity Consistency with Agency Plans and Policies Plans and Priorities Score Score reflecting the level of consistency and continuity with agency plans and priorities, including land-use plans, long-term plans, and other documents identifying priority investments • ATL: Regional Integration • MTC: Guiding Principles Assessment • Maryland DOT: Local Priorities Table 10. Objectives and measures – Streamlined Case.

46 Prioritization of Public Transportation Investments: A Guide for Decision-Makers were also supplemented with synthetic information appropriate to the nature of the projects being analyzed. Additionally, since the approach for scaling and normalizing measures varies by agency, much of the data from the original projects have been transformed to provide a unied demonstration data set. Table 11 lists the 20 projects included in the analysis. For each project, the table lists an ID number, short description, predominant mode, and cost in millions. Of the 20 projects, 10 are transit projects, eight are highway projects, and two combine transit and highway elements. e transit projects include a mix of bus rapid transit (BRT) projects and eet purchases plus a signal prioritization project, the creation of a transit plaza, a commuter rail extension, and a light-rail extension. e highway projects include improvement of existing roads, safety improvements, ITS, and improvements to facilitate pedestrian and bicycle access. e multimodal projects include the construction of multimodal stations/pedestrian access and a corridor improvement project. 6.3 Analysis Approach is section describes how the analysis was performed following the denition of measures, objectives, and test projects. e following subsections describe the approach for scaling and normalizing the data, and prioritizing and testing dierent scenarios. Testing Projects The pilot tested anonymized data from a set of 20 projects, including eight highway projects, 10 transit projects, and two combined transit/highway projects. ID Description Mode Cost ($M) 1 Airport BRT Line Transit 60 2 ITS and Signal Upgrades Highway 10 3 Magenta Ave Roadway, Safety and Pedestrian Improvements Highway 3 4 Route 4 Roundabout Highway 6 5 Maple Road Safety and Bike/Pedestrian Improvements Highway 3 6 Main Street Safety and Streetscaping Highway 13 7 Elevated BRT Line Transit 100 8 Commuter Rail Extension Transit 14 9 Commuter Bus Fleet Transit 1 10 Army Road Roundabout Highway 7 11 Zero-Emission Bus Fleet Transit 23 12 Citywide Transit Signal Priority Transit 2 13 Multimodal Transit Plaza Transit 9 14 Intersection Restriping Highway 2 15 New Traffic Signals and Sidewalks Highway 4 16 BRT Southern Line Extension Transit 8 17 Median-Separated BRT and Station Upgrades Transit 50 18 Multimodal Stations and Pedestrian Access Mixed 2 19 Multimodal Corridor Improvements Mixed 6 20 Crosstown Light-Rail Line Extension Transit 300 Table 11. Example projects.

Demonstration of Cross-Modal Prioritization 47   For the pilot, projects are prioritized based on an overall score divided by the project’s cost. e project score is a weighted sum of scores calculated for each of the seven objectives. Each objective score is a weighted sum of one or more scaled and normalized measure values. e measure values are scaled so that they are proportional to the benet of investment and normalized so that they lie on a scale from 0% (lowest value) to 100% (highest value). Scaling and Normalizing Scaling and normalizing the measure data are an important part of the prioritization process. e dierent measures have dierent ranges and measurement units. Also, in some cases, the measure value is proportional to the size of a project, while in others the measure is a score that is not strictly proportional to project size. e following approach was used to scale and normalize the data. Scaling. e rst step was to scale the measure values so that the resulting scaled value for each is proportional to the utility generated by the project with respect to the measure. e quantitative measures are self-scaling, but most of the qualitative ordinal scores are not. For this latter category, the approach used for scaling was to multiply the measure value by the project cost, which serves as a proxy for the magnitude of the project. is approach was used to scale the value for all of the score mea- sures except Economic Development Score, which is assumed to scale appropriately without additional adjustment given that in some of the examples, the measure value is proportional to the amount of new devel- opment expected to be initiated as a result of the project. is approach was used with the assumption that the scores essentially represent unit benets that must be adjusted to reect the magnitude of the project. Normalization. Next, each measure was normalized such that the values for the measure lie on a scale from 0% (lowest value) to 100% (highest value). is step simplies the specication of measure weights given the measures are in dierent units. Once the normalization approach is established, it is important to keep the approach consistent throughout the analysis. Measure val- ues were normalized by dividing them by the maximum value for the measure across all projects. Prioritizing Projects Given the scaled, normalized measure values, it is possible to then compute a score for each objective based on a weighted combination of individual measures. e score for a given objective i is calculated as follows: , 1 s w mi i j j j k ∑= = where si = score for objective i for a given project, i, j = indices, k = number of measures, wi,j = weight for objective i, measure j, and mj = scaled, normalized value for measure j for a given project. Measure Scaling Performance measures used for prioritizing investments should be scaled such that the measure value is proportional to the benefit yielded by the investment. This may require multiplying the measure value by a measure of project size, such as the project length, area, or cost. Measure Normalization When measures are combined to calculate a score, they should be normalized so that they lie on a common scale. Two approaches to normalizing measure values are to monetize all measures (convert them to dollars) or to transform the measure so that all values are on a scale from 0 to 1. Score Weighting For prioritizing investments considering multiple objectives, it is common to compute an overall score that combines scores for different objectives. These objective scores are typically multiplied by an objective weight that ranges from 0% to 100%, with the sum of weights equal to 100%.

48 Prioritization of Public Transportation Investments: A Guide for Decision-Makers ere are dierent approaches for prioritizing projects once objective scores are calculated. A typical approach is to calculate an overall score for the project that combines scores for dierent objectives using weights on each objective that overall sum to 100%. From there, projects are prioritized based on decreasing order of score/cost. is approach has been used for the pilot. e overall score for a project is calculated as follows: 1 S b si i i n ∑= = where S = score for a given project, i = index, n = number of objectives, and bi = weight for objective i. One consideration in prioritization is that of which cost to use when calculating the ratio of score to cost. Strictly speaking, if one is performing a one-time allocation using a single budget, then the budget consumed by the project should be used (e.g., the initial capital budget). However, in practice, agencies consider their capital plans over multiple periods utilizing multiple dierent budgets. e overall life cycle cost of the project provides a more comprehensive view of the project cost, and if this is available, it may be preferable for use in prioritizing over a range of programs and periods. Dening Analysis Scenarios Based on the above formulation, the variables that impact the prioritization are the weights on objectives and weights on measures with respect to each objective. Thus, the team developed a set of 20 different analysis scenarios for each test case to help compare the two approaches and establish the degree of sensitivity in the results to changes in weights. In a real-world application of this type of prior- itization scheme, weights are selected through an iterative process that considers the relative importance of an agency’s goals and the reliability of each variable to reflect the true intent of the objective. The following analysis demonstrates the type of sensitivity testing that agencies can conduct before settling on their final distribution of weights. Figure 10 shows the weights on objectives and measures for the data-intensive case. Each scenario is presented as a separate column. The width of the bars in Figure 10 is used to indicate the relative size of the weights, with higher weights represented by longer bars and lower weights represented by shorter bars. Weights are first applied to each of the objectives, and then they are applied to the collec- tion of variables comprising the objectives. The objective weights are shown in the top portion of the table, and the measure weights are shown below. For example, Figure 10 shows that in Scenario 1 (Base), Stewardship is weighted at 10% of the overall score across objectives. Stewardship is then comprised of two measures, Increase in Asset Useful Life and Reduction in Asset Risk, weighted at 75% and 25%, respectively. The analysis of scenarios with different objective and measure weights demonstrates the type of sensitivity testing that agencies can conduct before settling on their final distribution of weights.

1 2 3 4 5 6 7 8 9 10 Description Base Transit- Focused Highway- Focused Safety- Focused E onomy- Focused Equity- Focused Envi onment- Focused Consistency with Plans Perspective Focus on Quantitative Measures Focus on Measures of Time & Money Objective Weights Mobility 20% 30% 25% 10% 10% 5% 10% 10% 30% 30% Safety & Security 25% 10% 25% 40% 10% 15% 5% 10% 10% 5% Stewardship 10% 5% 25% 15% 10% 5% 15% 10% 10% 15% Environmental Performance 10% 20% 5% 15% 5% 15% 40% 10% 5% 5% Economic Development 15% 10% 5% 5% 40% 10% 5% 10% 10% 30% Equity 10% 20% 5% 5% 20% 40% 15% 10% 30% 10% Consistency with Agency Plans & Priorities 10% 5% 10% 10% 5% 10% 10% 40% 5% 5% Measure Weights Mobility Trave Time Savings (hours/day) 40% 5% 80% 20% 70% 10% 5% 40% 50% 80% Increased Transit Ridership (pass/day) Multimodal Mobility Imp. Score (scaled) 40% 70% 5% 60% 20% 40% 60% 30% 40% 10% 20% 25% 15% 20% 10% 50% 35% 30% 10% 10% Safety & Security Crash Reduction (annual EPDO crashes) Safety and Security Score (scaled) Crash Reduction (annual EPDO crashes) Safety and Security Score (scaled) Increase in Asset Useful Life ($) 50% 50% 50% 50% 90% 20% 50% 60% 90% 80% Env. Perf. Red Fuel Con. (000 gallons/yr) Reduction in Asset Risk (scaled) Reduction in Asset Risk (scaled) 75% 90% 50% 40% 80% 50% 70% 60% 90% 80% Eco. Dev. Increased Job Access (jobs) Eco. Dev. Score 50% 40% 50% 30% 60% 10% 20% 70% 90% 60% Equity Inc. Job Access - Disad. Areas (jobs) Increased Modal Accessibility Score (scaled) 50% 30% 80% 30% 80% 50% 20% 60% 90% 80% 50% 50% 50% 50% 10% 80% 50% 40% 10% 20% 25% 10% 50% 60% 20% 50% 30% 40% 10% 20% 50% 60% 50% 70% 40% 90% 80% 30% 10% 40% 50% 70% 20% 70% 20% 50% 80% 40% 10% 20% Consist. Plans and Priorities Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 11 12 13 14 15 16 17 18 19 20 Description Focus on Equity and Safety Balanced Split Evenly Split Random Mix 1 Random Mix 2 Random Mix 3 Maximized S o Minimized S o Max (Hwy S o - Transit S o Max (Transit S o - Hwy S o Objective Weights Mobility 5% 20% 15% 15% 10% 15% 5% 5% 5% 5% Safety & Security 30% 20% 15% 30% 20% 10% 5% 5% 5% 5% Stewardship Stewardship 10% 15% 15% 10% 20% 20% 5% 5% 70% 5% Environmental Performance 15% 10% 15% 10% 25% 15% 5% 5% 5% 70% Economic Development 5% 20% 15% 20% 10% 10% 70% 5% 5% 5% Equity 30% 10% 15% 5% 5% 10% 5% 70% 5% 5% Consistency with Agency Plans & Priorities 5% 5% 10% 10% 10% 20% 5% 5% 5% 5% Measure Weights Mobility Highway Mobility Imp. Score (scaled) 10% 40% 40% 40% 20% 30% 90% 5% 90% 5% Multimodal Mobility Imp. Score (scaled) Increased Transit Ridership (pass/day) 60% 30% 40% 50% 30% 20% 5% 5% 5% 90% Safety & Security 50% 30% 50% 80% 60% 30% 5% 95% 5% 95% 75% 75% 75% 20% 80% 20% 5% 80% 80% 90% 25% 25% 25% 80% 20% 80% 95% 20% 20% 10% Increase in Asset Useful Life (S) Env. Perf. Red Fuel Con. (000 gallons/yr) Enviro. Perf. Score (scaled) Enviro. Perf. Score (scaled) 40% 70% 50% 50% 80% 60% 95% 5% 5% 95% Eco. Dev. Increased Job Access (jobs) Eco. Dev. Score 40% 70% 50% 80% 50% 20% 5% 95% 5% 95% Equity Inc. Job Access - Disad. Areas (jobs) Increased Modal Accessibility Score (scaled) 50% 70% 50% 10% 50% 10% 95% 5% 5% 95% 60% 30% 50% 50% 20% 40% 5% 95% 95% 5% 60% 30% 50% 20% 50% 80% 95% 5% 95% 5% 50% 30% 50% 90% 50% 90% 5% 95% 95% 5% Consist. Plans and Priorities Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Stewardship 30% 70% 50% 50% 40% 40% 95% 5% 95% 5% 70% 30% 50% 50% 60% 60% 5% 95% 5% 95% 30% 30% 20% 10% 50% 50% 5% 90% 5% 5% 50% 70% 50% 20% 40% 70% 95% 5% 95% 5% Figure 10. Analysis scenarios with different weights – Data-Intensive Case. The width of the bars represents the relative weighting of the objects and measures.

50 Prioritization of Public Transportation Investments: A Guide for Decision-Makers e logic behind the scenario denition is as follows: • Scenario 1 is the Base case, with objective weights set at typical values based on the review of practice. • Scenarios 2 to 8 are similar to Scenario 1, except that the objective weights are revised to emphasize a single mode or objective. Specically, in Scenario 2 transit projects are given more emphasis. is is accomplished by increasing the weight on Mobility, Environmental Perfor- mance, and Equity by 10% each, and reducing the weights on other objectives. In Scenario 3, highway projects are given more emphasis by increasing the objective weights on Mobility and Stewardship and adjusting various measure weights to reduce the impact of transit and multimodality. In Scenarios 4 to 8, greater weight is placed on Safety and Security, Economic Development, Equity, Environmental Performance, and Consistence with Agency Plans and Priorities, respectively. Each of these represents the impact of shiing weights somewhat to emphasize a target mode or objective. • Scenarios 9 to 11 reect an emphasis on particular combinations of measures: quantitative measures (9); measures of time and money (10); equity and safety (11). • Scenarios 12 and 13 demonstrate equivalent or nearly equivalent weight given to each of the objectives. Scenario 12 takes a balanced approach, not allowing any objective weight to exceed 20% nor the measure weights to fall below 30%. Scenario 13 presents an even split across all objective weights. • Scenarios 14 to 16 have randomly generated weights. • Scenarios 17 to 20 were developed using the Excel Solver to maximize or minimize either the total score or the dierence between the cumulative transit and highway scores with a mini- mum weight of 5% on each objective and measure. Figure 11 shows the weights on objectives and measures for the streamlined case. As before, the width of the bars is used to indicate the relative size of the weights, with higher weights repre- sented by longer bars and lower weights represented by shorter bars. Generally, the weights were set to be comparable to the corresponding scenarios in the data-intensive case, while accounting for the dierent number of objectives and measures. Once the scenarios were defined, they were loaded into MODAT to help visualize the analysis results. Also, the analysis spreadsheet described previously was configured to allow for rapid testing of different scenarios and store the results of the analysis. For each scenario, the team calculated scores for each project, and the project rank was assigned based on the score/cost ratio. The resulting ranks and the correlation between the ranks for different cases and scenarios were calculated and reviewed as described in the next section. 6.4 Analysis Results e analysis yielded a set of scores, score/cost ratios, and ranks for each project, test case, and scenario. e results were reviewed to address the following topics: • What results were obtained? • How much do the results vary between the two test cases (data-intensive and streamlined)? • What is the degree of correlation between the cost, scores, and score/cost ratios for the two test cases? • How much do the ranks vary between the dierent scenarios? • What factors have the greatest impact on the results? The Multi-Objective Data Analysis Tool (MODAT) is a web-based tool intended to aid agencies in prioritizing a set of investments based upon multiple objectives (AASHTO 2021).

1 2 3 4 5 6 7 8 9 10 Description Base Transit- Focused Highway- Focused Safety- Focused E onomy- Focused Equity- Focused Envi onment- Focused Consistency with Plans Perspective Focus on Quantitative Measures Focus on Measures of Time & Money Objective Weights Mobility 22% 32% 33% 12% 11% 5% 12% 11% 33% 35% Safety & Security 28% 11% 33% 47% 11% 16% 6% 11% 11% 6% Stewardship 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Environmental Performance 11% 21% 7% 18% 6% 16% 47% 11% 6% 6% Economic Development 17% 11% 7% 6% 44% 11% 6% 11% 11% 35% Equity 11% 21% 7% 6% 22% 42% 18% 11% 33% 12% Consistency with Agency Plans & Priorities 11% 5% 13% 12% 6% 11% 12% 44% 6% 6% Measure Weights Mobility Highway Mobility Imp. Score (scaled) 40% 5% 80% 20% 70% 10% 5% 40% 50% 80% Multimodal Mobility Imp. Score (scaled) 60% 95% 20% 80% 30% 90% 95% 60% 50% 20% Safety & Security Safety and Security Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Env. Perf. Enviro. Perf. Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Eco. Dev. Eco. Dev. Score 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Equity Increased Modal Accessibility Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Consist. Plans and Priorities Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 11 12 13 14 15 16 17 18 19 20 Description Focus on Equity and Safety Balanced Split Evenly Split Random Mix 1 Random Mix 2 Random Mix 3 Maximized S o Minimized S o Max (Hwy S o - Transit S o Max (Transit S o - Hwy S o Objective Weights Mobility 6% 24% 18% 17% 13% 19% 5% 5% 17% 5% Safety & Security 33% 24% 18% 33% 25% 13% 5% 5% 17% 5% Stewardship 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Environmental Performance 17% 12% 18% 11% 31% 19% 5% 5% 17% 74% Economic Development 6% 24% 18% 22% 13% 13% 74% 5% 17% 5% Equity 33% 12% 18% 6% 6% 13% 5% 74% 17% 5% Consistency with Agency Plans & Priorities 6% 6% 12% 11% 13% 25% 5% 5% 17% 5% Measure Weights Mobility Highway Mobility Imp. Score (scaled) 10% 40% 40% 40% 20% 30% 90% 5% 90% 5% Multimodal Mobility Imp. Score (scaled) 90% 60% 60% 60% 80% 70% 10% 95% 10% 95% Safety & Security Safety and Security Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Env. Perf. Enviro. Perf. Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Eco. Dev. Eco. Dev. Score 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Equity Increased Modal Accessibility Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Consist. Plans and Priorities Score (scaled) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Figure 11. Analysis scenarios with different weights – Streamlined Case. The width of the bars represents the relative weighting of the objects and measures.

52 Prioritization of Public Transportation Investments: A Guide for Decision-Makers Project Rankings in the Streamlined and Data-Intensive Scenarios Table 12 summarizes the rank order results for the data-intensive and streamlined scenarios. Projects are listed in order of increasing rank using the ranks for the data-intensive scenario. Based on the results shown in the table, it is clear that projects are ranked similarly in the two test cases, with the notable exception of SGR projects (e.g., eet replace- ments). Qualitative scores may closely approximate the measures repre- sented by qualitative data. However, removing an objective (in this case Stewardship) can have a signicant impact on project ranks. It is also notable that there is no clear pattern by mode in the rankings—both highway and transit projects appear near the top and near the bottom of the rankings for both cases. Table 16 in Appendix D shows more detail on results in the base-weighting scenario (Scenario 1) including, by project, the project cost, scores for each test case, resulting score/cost ratios, and project rank based on the score/cost ratio. Correlation in Data Table 13 quanties the degree of correlation in the data shown in Table 6. It shows the correlation coecient between the project-level data for cost, scores, and the score/cost ratio for the two test cases. A correlation coecient of 1 indicates that there is a perfect, positive Removing an objective can have a significant impact on project ranks. ID Description Mode DI S 18 Multimodal Stations and Pedestrian Access Mixed 1 1 3 Magenta Ave Roadway, Safety and Pedestrian Improvements Highway 2 2 12 Citywide Transit Signal Priority Transit 3 6 10 Army Road Roundabout Highway 4 3 8 Commuter Rail Extension Transit 5 9 16 BRT Southern Line Extension Transit 6 4 9 Commuter Bus Fleet Transit 7 20 2 ITS and Signal Upgrades Highway 8 8 17 Median-Separated BRT and Station Upgrades Transit 9 14 4 Route 4 Roundabout Highway 10 5 19 Multimodal Corridor Improvements Mixed 11 7 11 Zero-Emission Bus Fleet Transit 12 19 15 New Traffic Signals and Sidewalks Highway 13 11 13 Multimodal Transit Plaza Transit 14 12 14 Intersection Restriping Highway 15 15 5 Maple Road Safety and Bike/Pedestrian Improvements Highway 16 13 1 Airport BRT Line Transit 17 17 7 Elevated BRT Line Transit 18 18 6 Main Street Safety and Streetscaping Highway 19 10 20 Crosstown Light-Rail Line Extension Transit 20 16 DI = data-intensive rank; S = streamlined rank. Table 12. Summary rank results for the Base Scenario 1.

Demonstration of Cross-Modal Prioritization 53   correlation between a pair of variables. A value of −1 indicates they are negatively correlated, and a value of 0 indicates the variables are uncorrelated. e table shows there is a high degree of correlation between the two test cases based on the correlation coecients for the scores of 0.96 and the score/cost ratios of 0.88. Further, for both test cases, the scores are highly correlated with cost. is results in part from the scaling process, in which cost was explicitly used to scale certain measures. In addition, the score/cost ratio has a weak negative correlation with cost and by extension, with score. This indicates that more costly projects are somewhat likely to rank lower. e fact that there is a negative correlation is to be expected given the score/cost ratio is inversely proportional to cost. It is reassuring that the correlation coecient is relatively low; otherwise, it would suggest that the prioritization approach is biased against larger projects. When designing a prioritization process, it is important to be aware of the impact of cost on the project ranking. Examining the correlation coefficients between score and cost, as illustrated in Table 13, can indicate whether there is a particular bias in the rankings toward higher-cost or lower-cost projects. All things being equal, one would expect the score for a project to be correlated with its cost but uncorrelated with the score/cost ratio, which captures cost-eectiveness. is is because higher-cost projects tend to have greater benets overall due to their larger scale and the greater number of people they therefore aect. Where there is a strong correlation between cost and score/cost ratio, this suggests that further consideration is needed regarding how measures are scaled. Variation in Results by Scenario e next set of gures shows how the results vary by scenario. Figure 12 is a box-and-whiskers plot showing projects ranked in order for Scenarios 1 to 16 for the data-intensive case. In this gure, each column shows the range of project ranks across the scenarios for a single project. When designing a prioritization process, it is important to be aware of the impact of cost on the project ranking. Examining the correlation coefficients between score and cost can indicate whether there is a particular bias in the rankings toward higher cost or lower cost projects. All things being equal, one would expect the score for a project to be correlated with its cost (higher cost projects tend to have greater benefits overall) but uncorrelated with score/cost ratio. Variables Compared Cost Data- Intensive Score Streamlined Score Data- Intensive Score/Cost Ratio Streamlined Score/Cost Ratio Cost 1.00 0.98 0.97 −0.31 −0.29 Data- Intensive Score NA 1.00 0.96 −0.27 −0.27 Streamlined Score NA NA 1.00 −0.21 −0.15 Data- Intensive Score/Cost Ratio NA NA NA 1.00 0.88 Streamlined Score/Cost Ratio NA NA NA NA 1.00 NA = not applicable. Table 13. Correlation coefcients calculated using project data for selected variables.

54 Prioritization of Public Transportation Investments: A Guide for Decision-Makers e projects are ordered by average rank (which is slightly dierent from the order in Table 12). e following is shown for each project: • Mean rank (indicated with an “x”). • Median rank (indicated with a horizontal line). • Range of ranks excluding outliers (vertical line). • Interquartile range (IQR), or range of the middle 50% of the data (thick bar). • Outliers, dened as lying 1.5 outside the IQR by a distance of at least 1.5 multiplied by the size of the IQR (circles). e gure illustrates that the ranks do vary for the dierent scenarios, but in most cases, the rank does not shi signicantly despite large dierences in the weights between the scenarios, particularly for the highest-ranked projects. For instance, for the top-ranked project (ID 18, Multimodal Stations and Pedestrian Access), the rank is 1 or 2 for all scenarios, with a single scenario at 4. e second-ranked project (ID 3, Magenta Ave Roadway, Safety and Pedestrian Improvements) has ranks from 1 to 3 for all scenarios, while the third- and fourth-ranked proj- ects (ID 12, Citywide Transit Signal Priority; ID 10, Army Road Roundabout) have ranks from 1 to 5 and 4 to 6, respectively. e project with the greatest variation in ranks is ID 15, New Trac Signals and Sidewalks. e rank for this project ranges from 6 to 20. Similarly, the rank for ITS and Signal Updates The rank of projects that emphasize a single objective are strongly dependent on objective weights, whereas the rank of projects that contribute to multiple objectives are less dependent on changes to objective weights. Figure 12. Project ranks for the data-intensive case, Scenarios 1–16.

Demonstration of Cross-Modal Prioritization 55   (ID 2) ranges from 3 to 16. Both of these projects have high values for measures under Mobility (ID 2 also scores well under Stewardship), but they have low values for most of the other measures. Since both projects emphasize a single objective, their ranks are strongly dependent on the objective weights. For projects that contribute to multiple objectives, the results are less dependent on changes to objective weights. Figure 13 shows a box-and-whiskers plot for the data-intensive case for Scenarios 17 to 20. Here the ranks vary signicantly for nearly every project, even ranging between 1 and 20 for one project (ID 20, Cross- town Light-Rail Line Extension). e results for the nal four scenarios illustrate that it is possible to achieve large shis in ranks by emphasizing selected measures to the near exclusion of others. e outcomes of these scenarios demonstrate what would happen if projects were prioritized based on a single measure. While emphasizing key objectives is impor- tant for reecting the values of the agency, it is important to balance the weight of one key objective with the benets provided by the others. Figure 14 shows the ranks for the streamlined case. In this gure, results are shown for all of the scenarios, and projects are shown in the same order as in Figures 12 and 13. In this case, there are several projects where the rank is unchanged across all scenarios and several with large ranges in ranks. e dierences between these results and those for the While placing an emphasis on key objectives is important for reflecting the values of the agency, it is important to balance the weight of one key objective with the benefits provided by the others. Additional measures in the data- intensive case do add information to the ranking process allowing for greater variation in ranks. Figure 13. Project ranks for the data-intensive case, Scenarios 17–20.

56 Prioritization of Public Transportation Investments: A Guide for Decision-Makers data-intensive case show that the additional measures in the data-intensive case do add informa- tion to the ranking process, allowing for greater variation in ranks. Practitioners should under- stand that altering the method of calculation for a measure or removing the measure entirely would impact the project scores and ranks. However, in many cases, the nal ranks will still closely resemble the previous iteration. e inclusion of additional data through extra measures lends to results that are more robust because with each measure added, less weight is given to each existing measure, and the impact of outliers is reduced. Visualization of Analysis Results in MODAT e analysis data were loaded into MODAT to help facilitate the analysis and illustrate the results. Appendix D provides details on this process. Figure 15 is a ow diagram produced by MODAT that shows the relationship between projects and objectives in the data-intensive case. Projects are listed on the le side of the diagram, and objectives are listed on the right. e relative contribution of each project to each objective is represented by ows from the le to the right. is gure is useful for establishing and understanding the impact of dierent objective weights on the relative strength of each project. e links that stand out the most are those for which one project is dominated by a single objective or one objective is dominated by a Figure 14. Project ranks for the streamlined case, Scenarios 1–20. Visualization can help with establishing and understanding the impact of differ- ent objective weights on the relative strength of each project.

Demonstration of Cross-Modal Prioritization 57   Figure 15. Flow diagram in MODAT.

58 Prioritization of Public Transportation Investments: A Guide for Decision-Makers single project. Raising and lowering an objective’s weight will increase or decrease the size of specic links, adjusting the bias between projects and objectives. 6.5 Pilot Demonstration Conclusions e following are major conclusions derived from and/or illustrated by the pilot. • It is feasible to design a prioritization process that prioritizes capital investments across multiple modes. e pilot analysis illustrates an example process, and albeit using a mix of real and hypothetical data, shows that the process can be structured to prioritize transit, highway, and multimodal projects. • Measure selection is important. Careful consideration is needed to design the objectives and measures to incorporate all of the factors that lead to selecting a project and to structure the measures to address dierent types of projects. Performing a sensitivity analysis, such as that described here, to test the impact of changes to measure and objective weights can help document the degree to which the scores for dierent objectives contribute to the overall scores for each project based on dierent assumptions. Also, such an analysis can help estab- lish whether removing selected measures changes the resulting prioritization. • Careful consideration must be given to the approach for scaling and normalizing measures. For the pilot, the measures used included a mix of quantitative measures (e.g., travel time and fuel savings) that are proportional to the benets of the project and other more qualitative measures that do not scale with project size. It is important to consider what each measure represents and what adjustments are needed to support prioritization. Ideally, measures should be scaled such that their values are proportional to the benets generated by a project. Where this is achieved, projects can be prioritized in decreasing order of the ratio of the project score to project cost. • Results are not highly sensitive to weights on objectives and measures, except where project performance is highly skewed toward one performance area. Any changes to objective or measure weights will impact the project score. However, Figures 12 to 14 show that in most cases, the rank of a project varies only somewhat as a result of changes in weights. Greater variation in rank is found in projects that perform very well in only one area as compared to projects that perform moderately well across several areas. • Project ranking is sensitive to the removal of an objective. Dramatic changes in rankings between scenarios are observed with very large changes in weight (as illustrated in Figure 13) or with the removal of certain objectives (e.g., the removal of Stewardship lowered the rank of SGR projects in the streamlined case). • One can approximate the results of a data-intensive approach using a streamlined set of measures. e results of the data-intensive and streamlined cases are highly correlated, as demonstrated by the correlation coecient of 0.88 between the score/cost ratio values calculated between the two cases, and many projects received similar ranks in both cases. In a streamlined case, it is important to retain as many objectives as possible, though the measures are usually less sophisticated than those of a data-intensive approach. In this study, the streamlined case was limited by the removal of the Stewardship objective.