Review of the Federal Railroad Administration’s Research and Development Program (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

77 6 Office of Research, Development, and Technology Support Functions In addition to reviewing each of the four research divisions in the Federal Railroad Administration’s (FRA’s) Office of Research, Development, and Technology (RD&T), the Statement of Task calls for a review of and advice on the central office’s support functions. As discussed in Chapter 1, the questions in the Statement of Task that concern these support functions ask whether RD&T • Engages in planning that is guided by a well-defined mission with associated goals and priorities that reflect safety needs in the rail- road industry (Question 3); • Takes steps to evaluate and ensure the usability and likelihood of adoption of research results by the railroad industry (Question 10); • Assesses the overall impact of the research and communicates the results to key stakeholders using means such as summative evalua- tion reports, technical reports, and conference presentations (Ques- tion 11); and • Sets budgets and staffing levels that are suited to addressing estab- lished goals and priorities (Question 6). To respond to these questions, the committee reviewed the means by which RD&T engages in, directs, and oversees • Strategic planning to guide the program, including the articulation of goals and strategies for the identification of research needs; establishment of program priorities; pursuit of different kinds

78 REVIEW OF FRA’S R&D PROGRAM of research (e.g., applied and knowledge-based research); dis- semination, implementation, and evaluation of research results; and alignment of staffing and budget levels with these goals and strategies; • Communications and other interactions with industry, other FRA offices, the research community, and railroad labor to identify priorities, recruit partners, and ensure that results are adopted and effective in the field; and • Evaluations of the performance and impact of work of the indi- vidual divisions and the program overall. In the committee’s view, all four research divisions should engage in strategic planning, communications, and evaluation along with the RD&T management team, but the management team has a supportive role in providing the needed direction, guidance, and resources. Therefore, when reviewing these three general support functions, the committee also refers back to its observations and recommendations from the four division re- views. Strengths and weaknesses observed in their planning, communica- tions, and evaluation can be indicative of how well RD&T is providing the needed support to the overarching research program. Upon commencement of the study, the committee met with RD&T’s senior management and division chiefs, who provided high-level presenta- tions describing RD&T’s mission and goals, as well as overviews of their focus areas, budgets, and staffing of the four divisions and program over- all. The committee reviewed RD&T’s planning documents, including the most recent Annual Modal Research Plan for FY 2019 (Federal Railroad Administration Office of Research, Development, and Technology 2018b), and asked the senior management team to explain how data on safety trends are used to inform priorities, staffing levels, and budget allocations. In subsequent meetings, the committee queried senior managers about how they engage in and guide the approaches used by the four divisions for communications and impact evaluations, including means used to commu- nicate with and consult FRA’s Office of Railroad Safety (RRS), other modal administrations within the U.S. Department of Transportation (U.S. DOT), freight and commuter railroads, railroad labor, and the broader research and technical communities. Informed by these discussions and document reviews, as well as the observations and recommendations from the subcommittee reviews of the four research divisions (summarized in Box 6-1), the committee offers the following observations with respect to RD&T’s strategic planning, com- munications, and evaluation support functions.

RD&T SUPPORT FUNCTIONS 79 BOX 6-1 Observations and Advice from Reviews of the Four Research Divisions Track and Structures Value of Safety Data and Industry Collaboration in Project Selection The Track and Structures Division (Track Division) budgets for the two research programs on track and structures and track–train interaction align with data on track-related derailment causes. Decision-support tools can be an effective means for developing and articulating the rationale for project selection, which, once recalibrated, should lead to even more effective means for project selec- tion and even more effective communication of research priorities with industry and other audiences, especially if combined with other means such as periodic conferences with industry to discuss the basis for past project selections. Reliance on Phased Procurement The use of separate procurements for distinct phases of larger projects can be an effective means of reducing risk through the establishment of clear checks on project quality. However, this approach can slow progress on low-risk proj- ects that have a higher likelihood of commercial viability and lead to a portfolio consisting of many active projects with a burdensome level of administrative challenges for program managers. Advance Planning to Assure Widespread Deployment Capability The willingness of railroads, suppliers, and others from industry to participate in a project is a clear sign that the research topic is important and the innovations being pursued by the research promise to have application in the field. Ensur- ing longer-term and more wide-scale deployment of successful innovations from research, however, may require even more front-end technology transfer planning to avoid deployment obstacles, such as the management of intellectual property, including user and data rights. Recommendation Ensure that technology transfer planning is sufficiently thorough and anticipatory at the outset of projects to take into account and manage the challenges that can hinder desired levels of deployment of research products and services, including the management of intellectual property such as user and data rights. Human Factors Trespassing and Suicide Countermeasures Program Holds Promise All indications are that the Human Factors Division’s ongoing trespassing and suicide prevention programs are coordinated effectively with the railroad industry. Because trespassing and suicide prevention are major concerns for operators continued

80 REVIEW OF FRA’S R&D PROGRAM of transit rail and highway systems, opportunities may exist for wider application of this knowledge through increased coordination with the Federal Transit Ad- ministration, Federal Highway Administration (FHWA), the public transportation industry, and state transportation departments. The Cab Technology Integration Laboratory’s Strategic Planning Promises More Industry Application and Collaboration The Human Factors Division has successfully developed its strategic plan for the Cab Technology Integration Laboratory and placed greater emphasis on industry consultation, collaboration, and communication. If continued as planned, this plan will help ensure that this world-class human factors laboratory is used to the maximum extent to address many safety issues. Phasing Research into Small Projects Has Pros and Cons The Human Factors Division seeks to maximize the use of funds for research by staging projects in phases, which allows funding to be spread across multiple budget cycles and can provide a checkpoint mechanism to monitor project progress and modify research plans as necessary. A disadvantage is that shorter duration projects may not sustain the development of a talent pool of budding hu- man factors researchers whose involvement in projects is essential for a research management agency that is dependent on the expertise of contractors. The Short Line Safety Institute Is a Success Story The Short Line Safety Institute has provided a needed focus on improving the orga- nizational safety culture of smaller freight railroads and illustrates how the Federal Railroad Administration’s (FRA’s) collaborations with industry can lead to cooperative solutions to problems and generate support for implementation across the industry. Opportunities Remain for Increased Communication and Dissemination of Research Results The importance of the work conducted by the Human Factors Division does not appear to be matched by its capability to communicate and disseminate the results. Greater use of communication means such as e-newsletters, webinars, presentations, and workshops in conjunction with major industry conferences and technical events would seem to be warranted to ensure that the products of the division’s human factors research are known and available to potential us- ers, including those in the U.S. rail transit and highway sectors. Ultimately, these efforts and those in the other divisions will only demonstrate value if there are observable improvements in safety trends that can be attributed to the Office of Research, Development, and Technology’s (RD&T’s) programs. Recommendations • The Human Factors Division should become more involved in the design, development, and programming of research projects across FRA’s RD&T portfolio. BOX 6-1 Continued

RD&T SUPPORT FUNCTIONS 81 • The Human Factors Division should engage more frequently with other U.S. Department of Transportation (U.S. DOT) modal administrations and other relevant federal agencies to identify opportunities for its work to inform, and be informed by, their human factors–related challenges and research, and to collaborate on the design and conduct of relevant research and the dissemination of those research results and products that have broad, multi-modal application. Rolling Stock Rolling Stock Safety Spans a Wide Domain That Demands Consideration and Balancing of Multiple Interests When Programming Research The wide purview of the Rolling Stock Division has led to a large and varied re- search portfolio—as diverse as wheel failure research, fire prevention, passenger railcar structural integrity, and tank car impact testing—that responds to prob- lems identified through examinations of causal factors reported in FRA incident data, legislative mandates, the rulemaking and enforcement requirements of FRA safety regulators, safety concerns identified in FRA and National Transportation Safety Board incident investigations, and other inputs. Industry Collaboration Is Critical to Ensuring That the Rolling Stock Division’s Research Capacity Is Used to Its Greatest Advantage The Rolling Stock Division’s data-driven and longstanding practice of industry- partnered research on railcar equipment and components was lauded by the industry representatives consulted for this review. However, their comments on the research programmed in some other areas, such as on wayside detec- tion technologies, surfaced concerns about whether the pace of the division’s research programming and execution is sufficient to keep up with the rate of technology development and deployment by industry in the field. Recommendation To better ensure that RD&T projects align not only with the most important safety problems but also focus on research needs that are best suited to the unique strengths of a government contract research program, the Rolling Stock Divi- sion should make industry consultations and collaborations a core feature of all research that will ultimately require industry acceptance and application of the results. In having so many diverse research responsibilities, such an alignment and focus is essential to ensuring the effective and judicious use of the division’s limited research budget. Moreover, the division should prioritize projects based on safety data; these priorities should be used to determine the number of active projects commensurate with the division’s funding level and allocated personnel. Train Control and Communications The Train Control and Communications Division’s Work to Improve the Op- erational Capabilities and Performance of Positive Train Control Aligns with Its Safety Mission continued

82 REVIEW OF FRA’S R&D PROGRAM In recognizing the importance of efficient, precise, and reliable operations to achieving the safety promise of positive train control (PTC), the division has forged strong connections with the railroad industry to identify and articulate critical PTC- related issues requiring research and industry involvement in projects. While the legislative mandate for PTC deployment underpins this collaboration, it appears to have promoted collaboration across the division’s train control and communica- tions portfolio. The Train Control and Communications Division’s Focus on Automation, Intel- ligent Transportation Systems, Communications, and Sensor Technology Recog- nizes the Importance of Human Factors to Train Operations and Grade-Crossing Safety Having such a wide breadth of responsibility—from furthering PTC to grade- crossing safety—the Train Control and Communications (TCC) Division has dem- onstrated creativity in harnessing advanced communications, sensors, intelligent transportation systems, and automation technologies to address human perfor- mance and behavior issues that can affect safety. Collaborations with other U.S. DOT agencies, such as FHWA and the Federal Motor Carrier Safety Administra- tion, on using connected vehicle technologies to alert drivers to grade crossings exemplify this creativity. Recommendation While all of the RD&T divisions face obstacles to getting the results of their re- search published and disseminated, overcoming these obstacles is especially important for ensuring the timely application of the TCC Division’s technology- oriented work. To this end, the TCC Division should make a concerted effort to ensure that research results are made available to industry and other users as quickly as possible to contribute to the advancement of PTC and other systems being deployed in the field. BOX 6-1 Continued STRATEGIC PLANNING When asked about the status of RD&T’s strategic planning, the committee was informed that a late-stage draft of the FY 2018–2022 strategic plan was pending internal (FRA/U.S. DOT) review. In not having access to this plan, the committee reviewed FRA’s Annual Modal Research Plan for FY 2019 and concluded that it was not a strategic plan but rather a compendium of the various RD&T programs and projects. While the plan maps these programs and projects to the goals of U.S. DOT’s strategic plan, it does not articulate agency priorities, strategies for pursuing them, or justifications for RD&T programs and budgets as one would expect from a guiding strategic plan. Hence, at the time of this review RD&T did not have an up-to-date stra- tegic plan and was presumably being guided in part by the FY 2013–2017

RD&T SUPPORT FUNCTIONS 83 strategic plan issued in May 2013 (Federal Railroad Administration Office of Research, Development, and Technology 2013). That plan—to the extent that it remains operative and influential on current RD&T programs—de- scribes strategies for stakeholder engagement and partnering, procurement, the identification of research needs and selection of projects, and the evalu- ation of programs and projects. As a model for future strategic planning, however, the committee determined that this plan needs improvement, not only by providing more complete descriptions of the strategies being pur- sued but also by giving the rationale for pursuing each. The articulation of stakeholder communications and partnering strate- gies is a strength of the FY 2013–2017 plan, which states that stakeholder engagement is the cornerstone of a successful R&D program and that en- gagement with stakeholders throughout the life cycle of a research project is crucial to identifying, prioritizing, and implementing research that is both relevant and feasible to intended users.1 The plan identifies key stakehold- ers inside FRA (e.g., RRS) and external to FRA (e.g., passenger and freight railroads, labor, and suppliers). The plan emphasizes the importance of conducting research in partnership with organizations such as the Associa- tion of American Railroads (AAR) and the American Public Transportation Association (APTA) and on co-funding and jointly conducting projects with industry to ensure research relevance and quality and to avoid duplication. As an example, the plan calls for ongoing collaborations with AAR to scan universities for technological developments that may have railroad appli- cation. However, it merits noting that the process of integrating input on research needs from external stakeholders during the development of the RD&T strategic plan is not part of a formalized process. Likewise, the strategies for identifying research needs and prioritizing projects are well-explained and reasoned. In addition to industry engage- ment, emphasis is placed on using incident data and consultations with RRS for identifying safety research needs and project concepts. The plan also points to the importance of funding some novel research for the purpose of developing fundamental knowledge, although it does not explain how this goal should factor into prioritizations. An example of novel research includes the early development of machine learning for track inspection in 2002, as noted during an RD&T staff presentation (Carr 2019). In meetings with the RD&T management team about priority setting, the committee was informed about the office’s use of Decision Lens, a decision-support software program, to assist with project selection that had been in use for several years. The algorithm in the software scores candidate 1 This point on stakeholder engagement was made in the Committee for Review of the Federal Railroad Administration (FRA) Research, Development, and Demonstration Programs Letter Report: May 31, 2012 as RD&T was preparing its strategic plan for FY 2013–2017 (National Academies of Sciences, Engineering, and Medicine 2012).

84 REVIEW OF FRA’S R&D PROGRAM projects based on weights assigned to various criteria and considerations such as whether a candidate research topic addresses a need identified through analysis of safety data and consultations with stakeholders, whether it aligns with U.S. DOT and FRA strategic priorities, and how much risk it presents in terms of cost and schedule. A benefit from the use of a decision- support tool such as this is the ability to produce a score that helps to communicate the rationale for selecting specific projects for funding. At the time of this review, however, this decision-support tool was not being used for such scoring because its weighting factors were undergoing review to account for changes in U.S. DOT strategic goals. The committee was thus unable to consider project prioritization scoring for RD&T’s portfolio of active projects. The FY 2013–2017 plan is not as effective in explaining RD&T pro- curement and evaluation strategies and their functional role and value. While it describes the mechanisms used for procurement, including contract research, grants, Broad Agency Announcements (BAAs),2 and the stepwise phasing of projects into a series of smaller projects, it does not do enough to explain the reasons for each mechanism under different circumstances and for different purposes. As discussed in Chapters 2–4, phased procure- ment can lead to a burdensome administrative load or an overly broad set of project topics, which leads to a slowdown in deployment of safety benefits and questionable priority setting. The section on evaluation strate- gies points to the importance of conducting evaluations that are credible, relevant, and timely but lacks a common framework for how such evalua- tions should be conducted. Also noteworthy is that the plan does not pro- vide—ideally at the outset—a compelling articulation of the role and value of FRA’s RD&T, including the range of outcomes intended, which presum- ably consists of not only technology development to enhance safety but also information for decisions, operational solutions to problems, knowledge to support future research, technology transfer, and the development of a skilled pool of railroad safety researchers. Another gap in the plan involves personnel strategies. Means for ensur- ing the effective allocation of staff and filling vacant positions in a timely manner should be part of a strategic plan, which would assist in fulfilling long-term needs in domain knowledge and talent among RD&T’s personnel and key contractors. Although aware of the slow pace of the federal hiring process, the committee questions whether longstanding senior staff vacan- cies will ultimately hinder the work of the divisions. It is exceedingly critical to the safety mission of RD&T to ensure continuity in key leadership and project management positions. 2 A BAA allows FRA to publicize a set of general research needs and to receive proposals from potential suppliers.

RD&T SUPPORT FUNCTIONS 85 Although an updated strategic plan is apparently being drafted, its pro- longed absence means that RD&T senior leadership may not be providing sufficient strategic guidance, direction, and support for the four research di- visions. Indeed, the preceding reviews of the four divisions yielded examples of needs that a strategic assessment could surface and help resolve. The committee has recommended, for instance, that the Human Factors Divi- sion be given a more prominent role in the design, development, and pro- gramming of research projects across RD&T’s portfolio—which requires cross-organizational strategies that can only be developed and implemented by the RD&T management team. The committee further recommended that the Track Division make more judicious use of phased project procurement to ensure that valuable research results are not delayed as a result of mul- tiple procurements and to ensure that the number of projects under contract does not create oversight and administrative burdens that risk delays across the portfolio. An articulation of the purpose of such a phased procurement approach, and when it should be used, is warranted as part of a broader procurement strategy. Another committee recommendation pertaining to the Track Divi- sion was for more thorough technology transfer planning at the outset of projects to take into account and manage the challenges that can hinder deployment of research products and services. A particular issue identified as being problematic—the management of intellectual property rights—is beyond the capabilities of the Track Division to resolve alone and requires strategic actions pursued by the RD&T management team in concert with other FRA offices. While the committee expects its biennial review of the four divisions to help inform strategic planning, it is not a substitute for an ongoing and more comprehensive strategic planning capability. While there is evidence that RD&T has such a capability, the delays in releasing the plan can be problematic for executing the strategies and ensuring that they are sound. It is important to recognize that even if informal steps are taken internally to align with the draft plan, many of the strategies are likely to involve in- terests outside RD&T, including other FRA offices, the railroads and their suppliers, labor, and the research community. The opportunity for them to have input into the plan at a formative stage is essential. COMMUNICATIONS The committee was interested in understanding how, in practice not just in its strategic planning, RD&T demonstrates the importance of commu- nication with industry, other FRA offices, the research community, and labor to identify priorities, recruit partners, and implement results. The FY 2013–2017 strategic plan placed a great deal of emphasis on these

86 REVIEW OF FRA’S R&D PROGRAM connections and interactions for identifying research needs and priorities, partnering with industry, and ensuring that the research results are usable and applied. The results of that planning should now be evident in practice. The committee therefore asked senior management to identify the specific means by which RD&T consults and collaborates with other FRA offices, including RRS, the Office of Railroad Policy and Development (RPD), and the Office Safety Analysis; railroads, shippers, and suppliers; and technical experts from academia and research and consulting organizations. Both in planning documents and from discussions with senior manage- ment, communications was described as integral to the program. Providing avenues for the broad array of stakeholders to communicate their research needs was characterized as critical to ensuring that the right problems are being targeted for research and that appropriate research mechanisms and procurement methods are being used (e.g., industry partnerships, BAAs, phased procurements). Communications was also described as being inte- grated into the entire project life cycle to ensure that the work is being done well and that the results will be used in the field to yield positive safety im- pacts. The committee was told that the divisions are directed to ensure that staff evaluate the progress of the research and usefulness of the results on an ongoing basis by partnering with stakeholders in the conduct of the work and by making a coordinated effort to disseminate the results to the rail- road community. The committee’s review of the work of the four divisions suggested that this direction is being followed, but gaps surfaced in certain aspects of communications, particularly in the area of dissemination (e.g., hastening publication review and enhancing the office’s electronic library) that may warrant central office attention and support. As noted in Chapter 3, the eLibrary suffers from issues with retrieval of project reports and other technical difficulties. Moreover, it is concerning that a senior employee of a commuter rail line was unaware of the RD&T divisions and how their research products and services might have delivered safety benefits to this passenger service. The committee would be better placed to evaluate RD&T if the office could better demonstrate the influence of its research through measurable improvements in safety and operational data. Asked to explain the approach to communications with the railroad industry, RD&T senior management pointed to several examples of the individual divisions engaging with railroads and suppliers; indeed, the sub- committees found many examples of such engagement when reviewing the work of the four divisions. The examples given by senior management, and those cited in the chapters of this report, show a wide variety of approaches being employed, including RD&T staff participating in AAR and APTA research and technical committee meetings, staff presentations at industry conferences, and the co-funding and joint conduct of projects, including

RD&T SUPPORT FUNCTIONS 87 pilot projects to test and demonstrate the technologies and concepts in the field. The RD&T management team, however, did not explain how it sup- ports such communications activity with needed guidance and resources— for instance, by providing funds for staff to attend conferences and to convene workshops with stakeholders. In the absence of an updated strate- gic plan, it proved difficult for the committee to assess the role that RD&T plays in supporting communications. While the FY 2013–2017 plan empha- sizes stakeholder engagement and partnering, there is sufficient variability in the communication mechanisms used by the divisions to suggest that division-level initiative, not support and guidance from the management team, is central to building and sustaining stakeholder communications. An example of a meaningful communications strategy to further strengthen stakeholder communication would be support for an open and free ex- change of ideas on a regular basis between RD&T and the railroad industry about safety improvements outside of the enforcement context. With regard to interactions with other FRA offices, the Annual Modal Research Program states that RD&T “works closely with FRA RRS to provide the basis for science-based requirements, standards, and recom- mendations that have been tested in real-world environments with the help of companies and organizations who will adopt and use the technology” (Federal Railroad Administration Office of Research, Development, and Technology 2018b, 106). In asking about the means by which these con- sultations and those with other FRA offices are carried out, the committee was told that interactions have occurred on an informal basis for years but that since 2017 the RD&T division chiefs and officials from RPD and RRS have been meeting regularly to discuss safety and research needs. To further expand communication with RRS, the RD&T division chiefs attend week-long RRS offsite training sessions on functions such as track and equipment inspection, which are intended to improve their understanding of the practical challenges and needs of FRA enforcement. Hence, the evidence from the reviews of the four research divisions suggests that they maintain regular and largely effective communication with other FRA offices and with industry to aid in the identification of priorities, recruit partners, and implement research results. While the sup- portive role of RD&T senior management in ensuring such engagement is not clear, there appear to be opportunities for more support to fill gaps, particularly in the timely and broader communication of research results. For instance, following the review of the Rolling Stock Division’s work the committee recommended that the division make industry consultations and collaborations a core feature of all research to make more judicious use of its limited research budget in a field in which there are so many research needs. Likewise, the committee recommended that the Train Control and

88 REVIEW OF FRA’S R&D PROGRAM Communications Division make a concerted effort to ensure that its re- search results are made available to industry and other users as quickly as possible given the rapid pace of information and communications technol- ogy development and deployment in the field. In its review of the Human Factors Division, the committee recommended that staff consult regularly with other U.S. DOT modal administrations—as well as other agencies that conduct research into human factors such as the Occupational Safety and Health Administration in the U.S. Department of Labor, the Human Fac- tors Program in the Office of Safety and Mission Assurance at the National Aeronautics and Space Administration, and the U.S. Department of Defense Human Factors Engineering Technical Advisory Group—to identify op- portunities for the division’s work to inform, and be informed by, work in the other modes and federal agencies. RD&T could be proactive in trying to identify where such additional communications efforts are desirable and provide the needed guidance and resources to support and sustain them. EVALUATION Stakeholder engagement in the selection of projects, conduct of work, and dissemination of results is viewed as providing built-in evaluations. Industry, for instance, is not likely to prioritize research topics that do not address needs in the field or that have minimal prospects for adoption. Likewise, industry is not likely to agree to co-fund or continue to collabo- rate on projects that are poorly planned and poorly executed. The quality and relevance of many products, concepts, or innovations developed by research can be ascertained in large part by the extent to which a railroad or supplier partner adopts them and promotes their use across the broader industry. Thus, RD&T senior leadership and its divisions depend on close connections with stakeholders to ensure that the work is relevant and that it remains on course and has impact when completed. The committee agrees that stakeholder connections are desirable in large part because they ground the research program in reality. There is ample evidence—including projects sampled in this report—that the four research divisions cultivate and nurture such connections and that by doing so they have produced many deployable products for improving railroad safety. However, there are many reasons for engaging in formal and deliber- ate evaluations and for not relying solely on this built-in process for ensur- ing program effectiveness. One practical reason is that FRA research seeks to do more than develop deployable safety-enhancing products, services, and operational concepts. Research is programmed to inform agency deci- sions (e.g., regulatory decisions), find operational solutions to problems, create knowledge to support future research, and help build and sustain

RD&T SUPPORT FUNCTIONS 89 the U.S. railroad research workforce and experimental capacity. Another reason is that the evaluations themselves can have many purposes—for instance, they can be undertaken to assess the conduct of the work itself, identify areas where improvements are needed in programs and the orga- nization, establish effective oversight, and ensure compliance. Evaluations help FRA set priorities and make adjustments to research procedures, and they can be used to communicate the value of the research to a variety of audiences internal and external to the agency, including Congress. While responsibility for undertaking program-wide evaluations clearly lies with RD&T’s divisions, the management team can also support the divisions in conducting their own program and project evaluations. For instance, the management team could develop a common evaluation frame- work that distinguishes among research outputs, outcomes, and ultimate impacts. Similarly, they could report more extensively on industry adoption of RD&T results, as was noted for deployment of autonomous track geom- etry measurement systems in Chapter 2, though otherwise briefings to the committee lacked such instructive detail. Measures of research outcomes that relate to the value to the public (safety and efficiency), government (scientific and wholistic bases for actions), labor (safety and ergonomics), or industry (safety, efficiency, and profitability) are imperative. Because FRA’s mission is to improve railroad safety, complete evaluations would naturally also determine whether outputs and outcomes produce safety benefits—a vital consideration in program evaluation. As noted above, the FY 2013–2017 strategic plan is short on details de- scribing how RD&T senior management engages in and supports program and project evaluations. When the committee asked senior management about methods of evaluation, a common response was to cite the impor- tance of having continuous input and reviews by stakeholder panels. When asked whether RD&T requires that each project have built-in features to aid in evaluation, senior management reported that each is required to provide status reporting, usually on a monthly basis. RD&T has been de- veloping a handbook for staff that contains guidance on conducting project evaluations, including reporting on key indicators of project performance. The standardized report requires expenditure data, work in progress per- centages on deliverables, and summaries of recently completed and planned activities. The committee learned that the management team has been build- ing internal capabilities for program evaluation to ensure that projects have the highest probability of delivering benefits, but few additional details were provided on the status and accomplishments of this initiative. A general sense of the committee is that the management team is in- terested in conducting more informative evaluations as well as providing the divisions with more guidance and capacity for conducting their own

90 REVIEW OF FRA’S R&D PROGRAM evaluations. The RD&T senior management team is responsible for lead- ing the group to establish effective long-range metrics, including periodic benchmark exercises with other agency research units. It merits noting that FRA’s sister agency, FHWA, has implemented a program and project evalu- ation framework for its R&D program that may serve as model for RD&T (Federal Highway Administration 2018). RECOMMENDATIONS The committee’s review of the support functions that RD&T should fulfill under the rubrics of strategic planning, communications, and evaluation surfaced both strengths and weaknesses. A major weakness is the lack of an updated strategic plan that articulates the management team’s role in supporting the work of the four divisions and provides strategies for implementing that role. In the absence of such a plan, the committee found it difficult to understand how RD&T senior management guides, oversees, funds, and otherwise supports the work of the four divisions. More enlight- ening, in the committee’s view, were the reviews of the individual divisions. They provided insight into where additional support by RD&T senior management is needed. Hence, the recommendations offered next can be viewed as pertinent to strengthening the supportive role of the RD&T man- agement team—especially in strategic planning—and addressing several of the division-level needs identified in the earlier chapters. • RD&T should engage in ongoing strategic planning that not only articulates agency priorities, strategies for pursuing them, and jus- tifications for its programs and budgets, but also clearly defines the support functions that are its responsibility and explains how those functions will be implemented. The plan should provide strategies for nurturing new technologies and techniques, such as the applica- tion of machine learning to railroad applications. The plan should also provide direction for assessing the impact of the research pro- gram over time, such as for the period covered by the most recent strategic plan, and it should articulate strategies for future impact assessments. Because many of the strategies in the plan are likely to involve interests outside RD&T, including other FRA offices, the railroads and their suppliers, labor, and the research community, such stakeholders should be formally invited to offer input early in the planning process. • RD&T should make more judicious use of phased project procure- ment to ensure that valuable research results from well-scoped, low-risk projects are not delayed as a result of multiple procure- ments and that the number of projects under contract does not

RD&T SUPPORT FUNCTIONS 91 create oversight and administrative burdens that risk delays across the portfolio. • RD&T should consult the four research divisions about their com- munications needs and challenges and look for opportunities where support by the management team would be advantageous, such as in providing resources for staff travel and participation in indus- try conferences and for convening annual, multi-day stakeholder workshops for sharing research results, identifying and prioritizing research needs, and exploring opportunities for collaboration in the conduct of research and in the demonstration and deployment of research results. • RD&T should work to develop a more comprehensive approach to program and project evaluation, including the development of a common evaluation framework that can be used by the four divi- sions to assess outputs, outcomes, and ultimate safety impacts of their work. FRA should adopt best practices that become apparent through periodic benchmarking exercises with other agencies in U.S. DOT and other federal research agencies about their research program evaluation methods and support functions, including FHWA. Note: While the study committee’s report was in peer review, RD&T released its Research, Development, and Technology Strategic Plan for 2020–2024 (Fed- eral Railroad Administration Office of Research, Development, and Technology 2020b). Unfortunately, the release occurred too late in the study process for the committee to thoroughly deliberate over and critique the document. One general observation that can be made, however, is that the strategies identified in the plan are similar to those of the earlier plan, with the exception of a new strategy for addressing safety risks in rural areas. A notable improvement is the inclusion of benchmark safety outcomes or goals as well as other metrics to evaluate a project’s success, such as reduce the cost per mile of recording track conditions. Apart from these observations, however, the updated plan does not appear to address the points raised above in the recommendation on strategic planning. It also bears noting that for 3 years RD&T has not had the benefit of this refreshed strategic plan.