Framework and Tools for Estimating Benefits of Specific Freight Network Investments (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

113 7.1 Critical Lessons Learned from Phase I Research This section highlights some of the more critical lessons learned throughout the development and testing of the Freight Evaluation Framework as a prelude to a proposed approach for further research to refine, test, and implement the Framework to support freight investment decisions. There are numerous available tools that can be used to assess benefits, costs, and risk of freight investments. What is needed are clear procedures that help analysts and deci- sionmakers integrate these tools and that guide the analysis to ensure consistency from project to project. As described earlier, there is a wide variety of investment decision-making techniques and tools that are used to assess user benefits, conduct return on investment assessments, and perform benefit/cost analysis, economic impact analysis, and risk analysis. Yet, there was general agreement at the Freight Investment Workshop that the Framework was very useful to help frame an investment decision analysis, even when there are multiple stakeholders and decision processes involved. Many analysts find it difficult to wade through the variety of tools and data and determine which are the most appropriate for their particular situations. They also felt that having a structure that guides the analyst through steps of an analysis would be very useful. Some specific features of the Framework that were felt to be particularly useful included the following: • The identification of stakeholders and relationships between benefit categories and stakeholders, which helps in alloca- tion of costs among potential beneficiaries. • The categorization of benefits and relationships among ben- efits, project types, and modes. This essentially provides a checklist for the analysts to make sure they have considered all appropriate benefit types for a particular project type. • The ability to conduct multimodal comparisons as well as to consider cross-modal impacts of projects. • Incorporation of risk analysis, which is a critical element of private-sector decision-making, but is not often explicitly accounted for in public-sector analyses. Incorporating a risk analysis module also can help compensate for uncertainty introduced as a result of data or methodology weaknesses. • The incorporation of case studies as part of the Framework testing process, which provided how-to examples as well as examples of the different data and tools that are available for conducting analysis. Users would benefit from more detailed procedures and guidance to facilitate using the Framework. Many participants in the Freight Investment Workshop described in Chapter 5 were concerned that the Framework as currently configured was complex and difficult to follow. Some were looking for “cookbook” procedures that provided detailed examples of how the Framework could be applied in practice. One participant suggested a guided approach that would walk the user through an analysis based on a series of questions about the nature of the project and the types of information available. Other participants felt that it would be difficult to develop a procedure that was this deterministic and wanted to maintain the flexibility inherent in the idea of a framework. Despite this range of opinions on how to do it, there was near universal agreement that more detailed procedures and examples would be beneficial to potential users of the Freight Evaluation Framework. This might include worksheets, sam- ple problems, and references on analytical tools and data sources organized in a guidebook format with users in mind. Analysis of small-scale local projects would benefit from more detailed information about current and prospective users of the project. The analyses suggested by the Framework are most effective when the analyst has some information about the types of industries/commodities that will use the project. This is par- ticularly important in the analysis of supply chain, reliability, C H A P T E R 7 Lessons Learned and Suggested Topics for Future Research

and inventory cost impacts. Some participants in the workshop lamented the state of the practice in subnational commodity flow data and freight forecasting that could produce this kind of information. Given this problem, more focus on local data collection may be justified for significant investments. Some of the case studies did involve this type of local interviewing or data collection to enhance the type of information that was available from models. This kind of supplemental data and information collection is a critical component of the success- ful implementation of the Framework. There are a variety of multimodal issues that need to be incorporated in any freight investment decision. The Freight Evaluation Framework was initially structured around modal modules, primarily because the analytical techniques tend to be slightly different for different modes. However, in actual practice (as was made clear from the case studies in Chapter 5), most projects will require the use of multiple modal modules. This is a feature of freight invest- ments that tends to distinguish them from passenger trans- portation investments. Reasons multimodal analysis is so critical include the following: • Projects often have explicit intermodal dependencies or multiple modes are present in the project. This is espe- cially true in cases involving cargo handling facility and port (air and marine) projects. Even though the investment may focus on one mode, the interaction of all the modes present at a facility requires a comprehensive intermodal analysis. • Projects often have cross-modal impacts. The most obvi- ous case is one in which modal diversion is an explicit objec- tive or expected outcome of the project (e.g., the Heartland Corridor Clearance Initiative). But there are cross-modal interactions in most projects. Developing a new air cargo facility will affect trucking access routes and transportation costs associated with connecting to the facility. Grade cross- ing projects clearly affect highway modes at the crossing as well as rail operations through the crossing. • Supply chains typically are integrated across multiple modes and reliability and inventory-cost impacts must be considered from one end of the supply chain to the other. This is one of the most challenging aspects of freight investment analysis because if done properly, it almost always requires consideration of secondary impacts of a project well beyond the jurisdictional boundaries of the investing entities. However, if the users of the project can be identified by industry type, it may be possible to focus on how the project affects nearby intermodal links and relate this to overall supply chain performance. Allocation of benefits and costs among stakeholders is a critical feature of the Framework but could be enhanced in future research. The research team’s testing process uncovered a number of issues related to how freight stakeholders are engaged through- out the application of the Framework, including the following: • Disaggregating benefits by stakeholder type. The Frame- work identifies and classifies stakeholders into different groups (asset providers, service providers, end users, and other impacted parties), and then adds a table to assign or allocate the various elements of benefit and cost to specific stakeholder groups. However, in carrying out the analysis, it can become a challenge to effectively assign various classes of benefits to specific stakeholders when there are dynamic interactions among them. This is illustrated by the DIA WorldPort case, where freight transport firms were projected to gain net revenue from expanded facility capacity but their actual gain would be reduced to the extent that they have to pay ground lease payments to the air freight facility operator, which, in turn, has to pay a share of its revenue to the property owner (airport author- ity). Tracking the string of payments can be challenging and estimating their final allocations may require the type of risk analysis that is included in the Framework. • Consistency among stakeholders and benefits. Maintain- ing consistency with how stakeholders are identified and how they might benefit from particular projects will add value to the Framework. For example, the results and find- ings from a study can look very different depending on the level of detail in which stakeholders are defined and the degree of depth to which their interactions are traced. Both detail and consistency are required to generate useful results. • Accounting for sensitivity differences. There are poten- tially large differences in the sensitivity to cost, benefits, and risk among different stakeholder types that are not all cap- tured within the Framework. This becomes important if the Framework is used to help rank projects from the perspec- tives of various stakeholder groups. In some cases, there may be issues of such importance to a particular stake- holder group that they outweigh any and all other possible costs and benefits to that particular agent. In such cases, group preferences may include factors not all captured in the Framework as currently configured. It may be possible for the Framework to be expanded to account for, and incorporate, these types of preferences. Alternatively, it may be necessary to just note cases where the Framework does not (or cannot) encompass other major considerations. Freight investment analyses need to be structured to ensure that they capture the wider economic effects of a project that may extend well beyond traditional study area boundary definitions. All of the case studies involved projects leading to savings in the cost of doing business (for at least some industries at some 114

locations). These transport efficiency and business productiv- ity enhancements typically lead to broader impacts on local economic growth. One of the cases—Huntsville—was selected to illustrate how a regional economic impact model can be suc- cessfully applied to assess the broader job and income growth impacts of a freight facility improvement project. The Framework could benefit from a more consistent approach to identifying the sources of risk and uncertainty that should be incorporated in the analysis. As described earlier, risk analysis often is focused on the market and cost risks that create the greatest uncertainties and that could lead to different project outcomes. The mar- ket risks may be a result of normal fluctuations (such as busi- ness cycles), which may be reasonably predictable, whereas other random events may be important to consider. Guidance could be developed to help identify the most typ- ical sources of each type of risk and uncertainty for different types of projects. In addition, guidance could be provided for how to account for methodological uncertainty in the analy- sis. Given that there are a number of key performance attrib- utes of freight investments that are difficult to predict with currently available tools and data, having a way to assess the level of uncertainty this introduces into investment decisions would be helpful. 7.2 Proposed Phase II Research Approach [Unfunded] In discussions with the NCFRP-05 Research Panel immedi- ately after the Freight Investment Workshop, it was clear that a number of the initial ideas about the focus of future research needed to be revisited. Although there clearly are areas where analytical tools and data that are necessary to assess specific freight investment benefits could use further development, these would be more appropriate for other ongoing research projects (e.g., improvements to freight forecasting and subna- tional commodity flow data development are, or have been, the subject of several NCFRP or Strategic Highway Research Program [SHRP] research projects). In general, the panel, the research team, and the workshop participants agree that there are many tools that already exist that can be used in a freight investment decision analysis. Where the Framework developed as part of this research effort adds most value is in providing a structured approach to con- ducting analysis, integrating existing tools and data, relating the benefit analysis to stakeholder perspectives, and introduc- ing risk analysis into the freight investment decision. There- fore, the researchers recommend that the focus of future research be shifted to accomplish the following: • Work with prospective users to identify any critical gaps in the Framework beyond those identified in this report. The Framework should be updated and modified to address these gaps and the critical lessons learned (as described in Chapter 5 and Section 7.1 of this report). • Develop a more detailed set of procedures for using the Framework and prepare a guidebook that includes these procedures and reference links to data and tools. The guide- book format should be determined with additional user input but could include both a hard copy and Web-based tool. • Provide the draft procedures to a number of users to fur- ther test the procedures in actual practice. During this test, the research team should be available to provide techni- cal support and assistance. Based on the test, recommenda- tions would be made to the consultant team for modifying and finalizing the guidebook. • Conduct outreach workshop and presentations at key user group meetings to ensure that the guidebook gets out to the widest audience. Each of these elements is described in additional detail below as a series of tasks for Phase II. Task 1—Identify and Correct Gaps in the Framework This task would begin with the identification of gaps/lessons learned from Phase I and identified in this report. These would include • Draft an outline of more detailed procedures, including a quick reference to specific data sources and tools that already have been identified in Phase I. The draft outline would serve as a quick roadmap of the Framework to review with prospective test users and could provide a basis for finaliz- ing the procedures that would be tested. • Review the benefit categories identified by stakeholder type and project type to ensure a final consistent set of benefit categories to be used in the final Framework. Ensure that these benefit categories are consistent across modes and that mode-specific performance metrics can be reported in a consistent multimodal format. • Investigate ways of incorporating various sensitivities of stakeholders to the importance of specific benefit categories. This could include the possibility of introducing a multi- attribute weighting procedure in addition to standard benefit/cost analysis. • Develop a more detailed reference list of tools and data sources for each type of analysis recommended in the project drawing on those already identified as part of this research. This would be reviewed by the panel and the test users. • Review the risk analysis procedures and develop a list of risk categories by project type to ensure consistency in risk analysis across project types. 115

While the research team is conducting this review of prior- ity refinements to the Framework, three beta testers will be identified. These testers will be agencies or entities that are willing to work closely with the research team to further refine and test the Framework in actual use. The beta testers should be selected, to the maximum extent possible, to reflect the fol- lowing three primary application types: 1. Need to make a go/no-go decision on a project, 2. Need to develop a project prioritization methodology, and 3. Need to determine cost allocation for a public-private partnership. To the extent that testers can be selected who have real proj- ect decisions and who have at least some data already compiled for their projects, this would facilitate testing. It would also be beneficial if the testers represent cases in which both state DOTs and MPOs are involved and in which projects involve some cross-section of different modes. The research team believes that it would be relatively easy to identify prospective test users from the Freight Investment Workshop participants. Once the testers have been identified and have agreed to par- ticipate, the research team will provide the results of the initial review of refinements to the test users and will meet in one-on- one interviews to identify any additional gaps or issues that the beta testers feel need to be addressed prior to the testing of the Framework. Task 2—Develop Procedures and Draft Guidebook Working with the beta testers, the research team and the NCFRP-05 Panel will agree upon a format for a draft set of procedures to facilitate testing. This format may not be the final format of the guidebook in order to facilitate quicker completion of the draft, but will need to have sufficient detail for the test users to be able to conduct analysis with limited assistance from the consultant team. The research team will then develop detailed procedures and links to existing data, tools, and other resources for each step in the procedures for as many project types as can be rea- sonably assembled (including all of the project types that will be the subject of the tests), and will prepare a guidebook to be tested by the test users. This guidebook will incorporate the revised Framework, a summary of analysis tools, and other associated information within a how-to approach to allow freight stakeholders to more consistently and effectively assess freight benefits and evaluate freight investment decisions. The research team has led the development of a number of guidebooks for NCHRP, AASHTO, FHWA, and others. This experience has allowed us to understand five key elements of successful guidebooks, as follows: 1. Start with an annotated outline—The team will first develop an annotated guidebook outline to share with the panel and the beta testers (as appropriate). This outline will document the proposed structure and organization of the guidebook and allow the research team to ensure that the information will be presented in an easy-to-use format. 2. Understand the target audience—When developing a guidebook, it is critical to understand who will be using it and, by extension, how it will be used. Understanding the target audience for this guidebook will allow us to develop its structure and content appropriately and in a way that meets the needs of its intended users. 3. Ensure that the guidebook complements existing resources—As described earlier, there are a number of existing guidebooks and other resources describing freight planning and programming and the use of freight benefit assessment tools and data. The guidebook developed as part of this task will be designed to be used in conjunction with these other resources and will provide links to addi- tional data and information. 4. Use case studies to provide how-to examples—Freight planning practitioners can benefit tremendously from understanding lessons learned and critical success factors from other agencies that already have undertaken such activities. Our experience working with a wide range of state DOTs, MPOs, and regional coalitions has shown that case study examples are effective ways to demonstrate how freight concepts—including the use of analytical tools to evaluate potential investments—can be used to address real-life problems. The techniques, processes, and practices described within this guidebook will be supported with case study vignettes derived from the case studies described in Chapter 5 of this report and others, as appropriate. 5. Organize the guidebook so that it is useful to both novice and advanced freight planning practitioners—Freight investment activities differ among different stakeholder types, as well as geographic scale. To become a meaningful resource to a wide range of freight stakeholders, the guide- book will be developed so that it provides useful informa- tion and techniques for both novice users and those users who are interested in more sophisticated freight tech- niques. The guidebook will be organized so that those that may be new to freight planning or to assessing freight ben- efits planning can obtain general guidance and informa- tion, while advanced users (or those wanting guidance on specific investment scenarios) will be pointed to specific tools or methods. The research team will develop the guidebook so that it is consistent with the key elements described above and meets the needs of its intended audience. The beta testers will receive an early copy of the draft to ensure that it is being developed in a manner that will best facilitate testing. 116

Task 3—Test the Draft Guidebook The beta testers will be provided with a copy of the draft guidebook for use in an actual implementation test case. At the initiation of the test, the research team and the testers will specify a test plan that will describe exactly how the test will be accomplished (i.e., the specific application of the guide- book), particular elements of the guidebook that will need to be evaluated as related to the particular application covered by the test case, and the schedule for the test. During the test, the users should be encouraged to attempt to use the guidebook without assistance first. However, during the duration of the test, the research team will be available to provide technical assistance to the test users on an as-needed basis. At regularly defined intervals during the test, the research team will conduct telephone check-in interviews with the test users to ensure that the test is proceeding and to answer any questions that may have emerged during the test. The research team will record the results of these check-ins and provide them as part of the monthly progress reports. At the conclusion of the test, the research team will meet with each beta tester and conduct an exit debriefing that will obtain detailed feedback on what worked and what did not, as well as specific recommendations for improvements in the final guidebook. Task 4—Revise the Draft Guidebook Based on the input received from the test users, the research team will prepare a work plan for making revisions to the guidebook and preparing it in the final format agreed upon with the panel. Prior to undertaking these revisions, the research team will meet with the panel and present the pro- posed changes. In addition, at this meeting, the NCFRP-05 Panel and the research team will discuss plans for outreach. After receiving approval from the panel, the research team will prepare a draft final guidebook that will be submitted to the panel. The research team will respond to any comments from the panel in the final version of the guidebook. The team also will prepare presentation materials that summarize the approach, key findings, important concepts, and notable con- clusions of the research. These materials, which will include detailed speaker notes, will be designed to be used by states, MPOs, regional coalitions, industry groups, and others as an off-the-shelf summary of the research and its importance. Task 5—Finalize, Publicize, and Conduct Outreach Upon completion of the guidebook, the research team will implement and disseminate it, working with the members of the NCFRP Project 05 Panel and other interested parties. There are a variety of stakeholders that could take an active role in implementing/disseminating the final product. At the federal level, FHWA has an established freight program, and supports many of the tools and datasets that are likely to be included within the investment framework developed as part of this research. Because of the multijurisdictional nature of freight movements, freight investments, and freight benefits, multi- state coalitions, such as the I-95 Corridor Coalition, the Mis- sissippi Valley Freight Coalition, the West Coast Corridor Coalition, and others will be important leaders in implement- ing the product. There also are opportunities for focused outreach by inter- ested parties at key conferences/gatherings, such as the Asso- ciation of Metropolitan Planning Organizations (AMPO), National Association of Regional Councils (NARC), TRB, and the Intermodal Association of North America (IANA). This would require individual champions, such as the research team and panel members. Key implementation activities would consist of coordinated outreach through established federal freight programs such as the “Talking Freight” seminar series, conference presentations and workshops, and ongoing promotion by panel members and identified leadership. One key opportunity may be to leverage FHWA’s investment in Financing Freight Improve- ments (29) and associated workshops. The results of this research could be a logical “add-on” to this effort that uses the workshop process to gain wider dissemination. In addition, the presentation materials and speaker notes developed as part of Task 4 will facilitate presentation in multiple forums. 117