Mainstreaming System Resilience Concepts into Transportation Agencies: A Guide (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

175   Monitor and Manage System Performance (Step 10) The 2012 legislation, “Moving Ahead for Progress in the 21st Cen- tury Act” (MAP-21), called for statewide and metro politan planning processes to incorporate a more comprehensive, performance- based approach to decision-making. According to the FHWA: Performance-based planning and programming apply performance management principles to trans- portation system policy and investment decisions, providing a link between management and long-range decisions about policies and investments that an agency makes in its transportation system. Perfor- mance-based planning and programming is a system-level, data-driven process to identify strategies and investments. Long-range planning helps to define key goals and objectives and to analyze and evaluate strategies and scenarios for meeting goals. Connecting performance measures to goals and objectives through target setting provides a basis for understanding and sharing information with stakeholders and the public (FHWA n.d.). Such an objectives-driven, performance-based approach to decision-making has also been applied to other functional areas of a transportation agency such as traffic safety, asset manage- ment, freight, and others. With respect to system resilience, performance metrics relating to an agency’s efforts to enhance system resilience would be a logical addition to such metrics. Indeed, the FHWA Climate Change Adaptation Guide for Transportation Systems Manage- ment, Operations, and Maintenance recommends that identifying operations performance metrics for system resilience is one of the steps that should be taken by O&M managers (FHWA 2015). Step 10 emphasizes two aspects of performance monitoring and management: (1) the performance (e.g., effectiveness) of agency actions in achieving a more resilient transportation system and (2) the monitoring of transportation system resilience to assess the extent to which agency actions are making a difference. This step also examines how well performance monitor- ing information feeds back into Step 8 actions and future systems-level vulnerability assessment work in Step 5 (hence the arrow shown to these steps from Step 10 in Figure 1). In a simple sense, Step 10 can be viewed as a feedback loop in the overall decision-making and program manage- ment structure for an agency. It provides information to decision-makers on the current state of transportation system performance that can be used to identify areas of opportunity for further actions to enhance system resilience. Capability Factors and Levels of Maturity Factor 10.1: Have you identified and are you using resilience metrics to monitor transportation system performance and your agency’s contribution to system resilience? This factor reflects a fundamental component of a performance-based approach to system resilience: the existence of resilience metrics that both monitor system performance and keep C H A P T E R   1 6

176 Mainstreaming System Resilience Concepts into Transportation Agencies: A Guide track of an agency’s programmatic contribution to this outcome. Examples of the first perfor- mance category include such outcome topics as average system delay or user cost per type of disruption, number of road closures due to system disruption, number of successful cyber- attacks against an agency’s IT capability, and number of debris-ridden culverts. Examples of the second category of output performance metrics include such topics as the number of culverts cleaned per year, average agency response time to major incidents, amount of fund- ing targeting resilience-enhancing projects, and the number of agency staff trained in system resilience courses. The major distinctions among the maturity levels in this factor consist of the level of comprehensiveness associated with the number of identified performance metrics and how the resulting information is used in agency decision-making. • Level 1: We have examined and expect to use (but are not currently doing so) resilience metrics that are most relevant to the types of decisions our agency makes relating to system resilience. • Level 2: We have identified and are using system and agency resilience metrics in several of our agency’s units. These metrics are both outcome-oriented (that is, focused on transporta- tion system resilience) and output-oriented (that is, focused on internal agency activities and actions leading to a more resilient transportation system). The performance measures are used primarily to guide management decisions in these units. • Level 3: We have a comprehensive set of resilience metrics relating to our agency’s entire resilience program and strategy. Factor 10.2: Are the results of resilience performance monitoring reported on a periodic basis to agency leadership? Reporting performance information can occur for many reasons, one of the most important being to influence decision-making within an agency. The key concept in this factor is to focus on the extent to which resilience performance information is reported to the agency leaders. This idea is similar in concept to many state DOTs having the number of traffic crashes and fatalities reported to top management weekly or monthly (given its importance to the agency and the public). The rationale for this factor is that what is reported to top management reflects what is important to the agency. The major distinctions among the maturity levels in this factor reflect the extent to which resilience performance monitoring is agency-wide versus unit- specific and the degree to which this information is used for decision-making. • Level 1: Individual unit leads in our agency receive reports relating to their resilience-oriented functional responsibilities. This information is used to enhance the unit’s efforts at improving that component of our system resilience program. • Level 2: Agency leaders and relevant unit managers are presented the results of agency-wide resilience monitoring as part of our normal system performance monitoring. This informa- tion is used by agency leaders to allocate resources to improve the resilience component of our transportation capital program and operating budgets. • Level 3: We have achieved Maturity Level 2 maturity. In addition, agency leaders receive after-event reports in order to identify how the agency can improve its resilience efforts. These reports are linked to agency performance measures relating to emergency response and incident/event recovery efforts. Factor 10.3: Has the monitoring of system resilience led to changes in agency processes or outputs in order to achieve better system resilience? This factor represents the bottom line in the use of performance measures: has anything changed based on the information provided? Thus, the different levels of maturity reflect the

Monitor and Manage System Performance (Step 10) 177   degree to which changes have actually occurred to enhance agency resilience actions based on the information provided. • Level 1: The performance measure information is used to monitor trends and acts as an early warning system when system performance is starting to deteriorate. This information often leads to resilience-oriented actions implemented by unit managers, although there is no monitoring of how this information is used. • Level 2: The information is used by our resilience-related units to improve their efforts. There are many examples of where such efforts have occurred because of the information produced from performance measures. • Level 3: We systematically examine the trends in system resilience performance and use this information to identify where organizational and programmatic changes should occur. There is also systematic and comprehensive monitoring of steps taken by agency units that feed into an agency-wide perspective on resilience-related actions. There are many examples where organizational change has occurred due to information provided from the performance monitoring system. Factor 10.4: Is your resilience monitoring internally consistent amongst agency units? One of the early experiences with the use of performance measures was that different units in an agency developed their own metrics, often inconsistent with the measures being used by other units or with overall agency objectives. The major distinctions among the maturity levels in this factor examine the degree to which resilience measures used throughout the agency are consistent with one another and the extent to which they reflect and contribute to overall agency resilience goals and objectives. • Level 1: We have developed unit-specific, resilience-oriented performance metrics but have not looked at resilience measurement organization-wide to assure consistency and compat- ibility among all the units. • Level 2: We have developed unit-specific performance metrics with specific consideration of similar and/or linked responsibilities in other resilience-oriented units (e.g., traffic operations and emergency response). • Level 3: Our agency’s resilience monitoring program was developed collaboratively amongst all units with internal consistency and interconnection in mind. Each unit’s performance measures are linked directly to an agency-wide resilience performance metric. Factor 10.5: Do you monitor the performance metrics of other agencies and partners who have a role in enhancing system resilience (e.g., enforcement or emergency response providers) to determine how your agency’s actions affect their overall performance? Transportation agency resilience programs and activities are often conducted in collabora- tion with other agencies (e.g., emergency response and disaster recovery efforts). This factor reflects the extent to which collaboration occurs with your agency’s partners in defining and monitoring the performance and effectiveness of jointly conducted resilience efforts. This factor represents a very proactive, multi-organization effort at enhancing system resilience. The major distinction among the maturity levels reflects the level to which partner agencies are brought into your agency’s resilience efforts to improve system resilience performance. • Level 1: We informally monitor what partner agencies are reporting as their key perfor- mance metrics as they relate to system resilience. We consider this information to make our collaborative efforts more effective.

178 Mainstreaming System Resilience Concepts into Transportation Agencies: A Guide • Level 2: Certain units in our agency monitor the performance metrics of their partner agency units and collaborate closely with key partners to determine if our agency can implement actions to improve overall system resilience and benefit all those involved. Our collaborative efforts are linked closely to the unit’s resilience performance metrics. • Level 3: We monitor the performance metrics of our key partners and use this information to determine jointly and periodically what actions can be taken to improve system resilience performance. These efforts to improve system resilience are linked closely to our agency’s system resilience performance measures. Factor 10.6: Have you used resilience metrics in communicating to key policy leaders and other decision-makers outside of your agency? Besides monitoring an agency’s effectiveness in affecting the performance and condition of the transportation system, performance measures can be used to communicate to key individuals about what your agency is doing and why it is important. Many transportation agencies, for example, have developed annual “state-of-transportation” reports for widespread dissemina- tion on the trends in transportation system performance. This factor focuses on how resilience performance measures are used as part of an agency’s communications strategy. The major distinctions among maturity levels are the degree to which resilience performance measures are used as part of this communications strategy and how this information is used as part of budget justification. • Level 1: We provide resilience information from our agency’s performance measures when requested by key decision-makers and stakeholders. • Level 2: The resilience metrics are part of an overall agency communications strategy for informing key government leaders on what our agency has been doing to improve the resilience of the transportation system. The information is sent to these leaders and constitu- ency groups as part of a periodic outreach effort. • Level 3: We have achieved Maturity Level 2. In addition, the resilience performance measures are used to showcase our agency’s contribution to a resilient transportation system to the general public. Factor 10.7: Have you developed and reported on performance metrics relating to incident/event response? Traffic crashes and other incidents are some of the most visible and most frequent disruptions to a transportation system. They not only disrupt travel in key mobility corridors, but they also directly influence the public’s perception of a transportation agency’s competence in managing the system. The creation and ongoing evolution of TMCs is an example of how transportation agencies have been more active in improving their response and recovery efforts associated with such incidents. The major distinction among the maturity levels is the degree to which such information is provided to different groups. • Level 1: We have developed incident/event performance metrics primarily for internal purposes. The information is provided to key decision-makers and stakeholders when requested. • Level 2: The incident/event performance metrics are part of an overall agency communi- cations strategy for handling incidents and disruptive events. The information is provided to the media as part of an incident/disruptive event communications effort. • Level 3: We have achieved Maturity Level 2. In addition, the incident/disruptive event per- formance information is regularly provided to key governmental leaders and constituency groups as part of our agency’s resilience outreach efforts.

Monitor and Manage System Performance (Step 10) 179   Factor 10.8: Have you assigned monetary estimates to the benefits of showing positive resilience outcomes? Estimating the monetary value of the benefits of a transportation program is one of the major ways of attracting decision-maker attention on transportation program effectiveness. This factor focuses on the extent to which such benefits have been assigned monetary values. The major distinctions among the maturity levels include the degree to which monetary benefits have been defined and the extent to which they are used in supporting agency resilience programs. • Level 1: We have considered (or are considering) the use of monetary benefits and expect to adopt them as part of the agency’s performance measurement strategy. • Level 2: Some of our agency’s units use widely accepted BCA analysis to illustrate the benefits of resilience actions (e.g., monetary benefits of responding to incidents quickly). • Level 3: We have developed monetary measures on a wide range of parameters relating to system resilience (e.g., economic impacts, user costs, freight impacts, and the like). Our agency regularly reports on what the expected monetary benefits are for agency-wide actions to improve system resilience. Table 30 summarizes the factors that are included in the self-assessment tool for Step 10. The maturity levels for each factor are presented in the descriptions of each factor. The total score for this step is found by summing the number of points given for each factor. Recommended Actions to Maintain the Highest Level of Agency Resilience Capability The highest level of capability for Step 10: Monitor and Manage System Performance focuses on continual improvement in agency capability and actions leading to a more resilient trans- portation system. If your agency has reached a Level 3 maturity, the steps that can be taken to maintain this level include: • Continue to monitor the resilience of the transportation system and consider changes in the performance measures currently used for the agency’s resilience program. • Monitor changing data collection technologies from the perspective of whether they provide new opportunities for defining and collecting data on new resilience performance measures. • Review how after-event reports are used by decision-makers and identify changes in the format or timeliness to make them more useful. • Periodically review how performance measures are used to communicate with key stake- holders, support program changes, and affect budgets. Make changes as appropriate to ensure such uses are occurring. • Review agency resilience efforts concerning collaboration with other organizations. Identify where changes in your agency and other organizations’ procedures could result in improved, jointly conducted resilience efforts. • Hold key agency/unit leadership meetings to brainstorm how resilience program monitoring and management can be made more effective. • Hold meetings with key stakeholders to brainstorm how resilience program monitoring and management can be made more effective in terms of informing others of agency efforts. • Reassess the use of traffic incident performance metrics as to their usefulness in improving agency response capabilities and in conveying to the public your agency’s effectiveness in handling such disruptions. • Periodically update monetary estimates of the benefits of your agency’s resilience efforts. These updates should reflect the latest cost estimates from your agency’s asset management, maintenance, and other relevant unit programs.

Maturity Characteristic Level 1 (1 point) Level 2 (2 points) Level 3 (3 points) 10.1 Have you identified and are you using resilience metrics to monitor transportation system performance and your agency’s contribution to system resilience? We have examined and expect to use (but are not currently doing so) resilience metrics that are most relevant to the types of decisions our agency makes relating to system resilience. We have identified and are using system and agency resilience metrics in several of our agency’s units. These metrics are both outcome-oriented (that is, focused on transportation system resilience) and output-oriented (that is, focused on internal agency activities and actions leading to a more resilient transportation system). The performance measures are used primarily to guide management decisions in these units. We have a comprehensive set of resilience metrics relating to our agency’s entire resilience program and strategy. 10.2 Are the results of resilience performance monitoring reported on a periodic basis to agency leadership? Individual unit leads in our agency receive reports relating to their resilience-oriented functional responsibilities. This information is used to enhance the unit’s efforts at improving that component of our system resilience program. Agency leaders and relevant unit managers are presented the results of agency-wide resilience monitoring as part of our normal system performance monitoring. This information is used by agency leaders to allocate resources to improve the resilience component of our transportation capital program and operating budgets. We have achieved Maturity Level 2. In addition, agency leaders receive after- event reports in order to identify how the agency can improve its resilience efforts. These reports are linked to agency performance measures relating to emergency response and incident/event recovery efforts. 10.3 Has the monitoring of system resilience led to changes in agency processes or outputs in order to achieve better system resilience? The performance measure information is used to monitor trends and acts as an early warning system when system performance is starting to deteriorate. This information often leads to resilience-oriented actions implemented by unit managers, although there is no monitoring of how this information is used. The information is used by our resilience- related units to improve their efforts. There are many examples of where such efforts have occurred because of the information produced from performance measures. We systematically examine the trends in system resilience performance and use this information to identify where organizational and programmatic changes should occur. There is also systematic and comprehensive monitoring of steps taken by agency units that feed into an agency- wide perspective on resilience-related actions. There are many examples of where organizational change has occurred due to information provided from the performance monitoring system. 10.4 Is your resilience monitoring internally consistent amongst agency units? We have developed unit-specific, resilience-oriented performance metrics but have not looked at resilience measurement organization- wide to assure consistency and compatibility among all the units. We have developed unit-specific performance metrics with specific consideration of similar and/or linked responsibilities in other resilience- oriented units (e.g., traffic operations and emergency response). Our agency’s resilience monitoring program was developed collaboratively amongst all units with internal consistency and interconnection in mind. Each unit’s performance measures are linked directly to an agency-wide resilience performance metric. Table 30. Assessment table for Step 10: Monitor and Manage System Performance.

10.5 Do you monitor the performance metrics of other agencies and partners who have a role in enhancing system resilience (e.g., enforcement or emergency response providers) to determine how your agency’s actions affect their overall performance? We informally monitor what partner agencies are reporting as their key performance metrics as they relate to system resilience. We consider this information to make our collaborative efforts more effective. Certain units in our agency monitor the performance metrics of their partner agency units and collaborate closely with key partners to determine if our agency can implement actions to improve overall system resilience and benefit all those involved. Our collaborative efforts are linked closely to the unit’s resilience performance metrics. We monitor the performance metrics of our key partners and uses this information to determine jointly and periodically what actions can be taken to improve system resilience performance. These efforts to improve system resilience are linked closely to our agency’s system resilience performance measures. 10.6 Have you used resilience metrics in communicating to key policy leaders and other decision-makers outside of your agency? We provide resilience information from our agency’s performance measures when requested by key decision-makers and stakeholders. The resilience metrics are part of an overall agency communications strategy for informing key government leaders on what our agency has been doing to improve the resilience of the transportation system. The information is sent to these leaders and constituency groups as part of a periodic outreach effort. We have achieved Maturity Level 2. In addition, the resilience performance measures are used to showcase our agency’s contribution to a resilient transportation system to the general public. 10.7 Have you developed and reported on performance metrics relating to incident/event response? We have developed incident/event performance metrics primarily for internal purposes. The information is provided to key decision-makers and stakeholders when requested. The incident/event performance metrics are part of an overall agency communications strategy for handling incidents and disruptive events. The information is provided to the media as part of an incident/disruptive event communications effort. We have achieved Maturity Level 2. In addition, the incident/disruptive event performance information is regularly provided to key governmental leaders and constituency groups regularly as part of our agency’s resilience outreach efforts. 10.8 Have you assigned monetary estimates to the benefits of showing positive resilience outcomes? We have considered (or are considering) the use of monetary benefits and expect to adopt them as part of the agency’s performance measurement strategy. Some of our agency’s units use widely accepted BCA analysis to illustrate the benefits of resilience actions (e.g., monetary benefits of responding to incidents quickly). We have developed monetary measures on a wide range of parameters relating to system resilience (e.g., economic impacts, user costs, freight impacts, and the like). Our agency regularly reports on what the expected monetary benefits are for agency-wide actions to improve system resilience. Score Range Description of Agency Maturity in Monitoring and Managing System Performance 0 to 11 Your agency is emerging into this area and has taken initial steps to develop a resilience performance measurement program. However, much more can be done. 12 to 20 Your agency has implemented several strategies for measuring performance, not so much as part of an agency-wide strategy but primarily at the unit level. There are still important actions that can be taken to reach the next level of maturity. 21 to 24 Your agency has reached significant maturity in developing and implementing a resilience performance management program. The major focus should be on maintaining and enhancing existing efforts when appropriate and taking advantage of new opportunities as they become available.

182 Mainstreaming System Resilience Concepts into Transportation Agencies: A Guide If you did not score a 24 in the assessment (a perfect score in Level 3 efforts), identify those factors that were rated lower and identify a strategy or action steps to improve these particular components of Step 10. Recommended Actions to Achieve Higher Levels of Resilience Capability If you scored at Level 1 or 2, you can take steps to continue your evolution toward a more resilience-oriented agency as it relates to enhancements to monitor and manage system perfor- mance activities. In such cases, agency managers should identify which of the factors in Table 30 are most deficient and determine priorities for improving your agency’s organizational resilience capabilities in monitoring and managing system performance. Table 31 is offered as a template to determine which steps your agency can take to improve its resilience capabilities in monitoring and managing system performance, who should be responsible, the timeframe for the implementation, and expected outcomes. Table 31. Actions to achieve higher maturity for Step 10: Monitor and Manage System Performance. Let’s do this. (check) Action Re sp on si bi lit y? Ti m ef ra m e? Ex pe ct ed ou tc om es ? Assess your agency’s use of resilience measures and how they can be improved and enhanced, making sure all your agency’s relevant units are involved in the process. Examine how agency leadership is using resilience performance measurement information (such as after-event reports) to identify how the use of such information can be improved. This would likely require internal meetings with the agency’s top leaders. Track the use of resilience performance measure information and how it has been used to adopt changes in the agency. Use this information to illustrate both the usefulness of performance measure information as well as the effectiveness of your agency’s resilience program. Periodically monitor the trends in system disruptions from both natural and human-made causes. Relate this tracking to the appropriateness of currently used resilience measures. Update your agency’s resilience-related procedures and processes based on jointly conducted assessments of incident/disruptive event performance with participation from those organizations your agency collaborates with. Identify jointly adopted performance measures for incidents and disruptions where collaborative efforts are common. Periodically examine the use of resilience performance measurement information in your agency’s communications efforts to key stakeholders and decision-makers. Make changes where necessary. Hold meetings with agency leaders and staff, especially those who often speak in public forums, to make sure they understand the meaning, implications, and benefits associated with your agency’s resilience program (as reflected in reported performance measures).

Monitor and Manage System Performance (Step 10) 183   Chapter 16 References FHWA. n.d. Performance-Based Planning and Programming. Website. Retrieved June 30, 2020, from https:// ops.fhwa.dot.gov/plan4ops/performance_based.htm FHWA. 2015. Climate Change Adaptation Guide for Transportation Systems Management, Operations, and Maintenance. Report FHWA-HOP-15-026. Retrieved June  30, 2020, from https://ops.fhwa.dot.gov/ publications/fhwahop15026/index.htm Useful Resources Nelson, P. and G. Dongjun. 2016. Final Report: Metrics, Models, and Data for Assessment of Resilience of Urban Infrastructure Systems. University Transportation Research Center, Region 2. Retrieved June 30, 2020, from https://rosap.ntl.bts.gov/view/dot/32331/dot_32331_DS1.pdf Sun, W., P. Bocchini, and B. Davison. 2018. Resilience Metrics and Measurement Methods for Transporta- tion Infrastructure: The State of the Art. ResearchGate. DOI: 10.1080/23789689.2018.1448663. Retrieved June  30, 2020, from https://www.researchgate.net/publication/326920176_Resilience_metrics_and_ measurement_methods_for_transportation_infrastructure_The_state_of_the_art Transportation Research Circular E-C226: Transportation Systems Resilience—Preparation, Recovery, and Adaptation. Transportation Research Board. Washington, DC. Nov. 2017. Retrieved June 30, 2020, from http://onlinepubs.trb.org/onlinepubs/circulars/ec226.pdf Weilant, S., A. Strong, and B. Miller. 2019. Incorporating Resilience into Transportation Planning and Assessment. RR-3038-TRB. RAND Corporation. Retrieved June 30, 2020, from https://doi.org/10.7249/RR3038 Let’s do this. (check) Action Re sp on si bi lit y? Ti m ef ra m e? Ex pe ct ed ou tc om es ? Given the importance of traffic incidents to system performance and perceived agency credibility, conduct after-event assessments of the use of incident response and recovery performance information for both internal and external audiences. Continue to investigate the most appropriate strategy for monetizing resilience benefits. Monitor the literature and examine best practices from other transportation agencies in how this can be done most convincingly. Possible steps for Step 4: Implement Early Wins Table 31. (Continued).