Managing State Transportation Research Programs (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

43 This Synthesis ties together research topics that have been treated disparately in the past in order to understand how state transportation research programs organize themselves for pro- gram quality and value, given their program capabilities and program management attributes. This survey was sent to all Research Program Managers at state DOTs across the country and resulted in 44 responses, an 88% participation rate. For a comprehensive list of survey questions and responses, please see Appendix C. 2.1 Program Capability Program Capability is the ability of a state DOT to undertake research, in terms of its goals, knowledge, expertise, resources, organizational structure, and external relationships. Organizational Structure and Program Responsibilities There is not an immediate correlation between the role or size of the research program and where the program is located; there are relatively few programs located in specialized divisions. The Synthesis survey found that most state transportation research programs are located in Planning or Chief Engineer divisions of state DOTs, where they report to the Chief Engineer, Planning Director, or equivalent or affiliated leadership. A few are located in more specialized divisions such as in Highway, Materials, Construction, and Field Services. Almost all state research programs administer research funding and manage research proj- ects (Figure 12). These functions are possibly the most common and most basic functional attribute of research programs. Beyond this, however, agency functions start to become more differentiated. Most telling is the observation that only about half of agency programs conduct research “in-house” within the DOT. Outsourcing all research could influence research culture and capability because DOT personnel may have lower exposure or experience with research and may not benefit from the perspectives that are available to those who directly engage in research. Outsourcing all research may also reduce the options and discretion available to agen- cies on projects that are more applied in nature and better suited to a DOT practitioner’s way of thinking. Furthermore, not all programs monitor implementation, which is surprising because implementation is an integral part of the research lifecycle. Many programs also view themselves as entrusted with the responsibility to broadly pro- mote innovation within the agency, which is consistent with the overall nature and mandate of research programs. Supporting this idea is that agency research units frequently provide library services, and a few provide training and other services (such as the Local Technical Assistance Program—LTAP) to disseminate research findings, support practitioner education, and so on. C H A P T E R 2 Survey Results

44 Managing State Transportation Research Programs Few agencies currently have programmatic responsibilities for staff training or continuing edu- cation to disseminate new knowledge, however (Figure 13). These ancillary knowledge manage- ment functions can play an important role in developing a culture of research and promoting innovation. This ability to innovate is closely tied to how other parts of the DOT champion and engage with the research program. Research Direction Setting a vision, goals, and objectives for research allows a program to prioritize efforts and focus resources. In this study’s survey, agencies reported that they find research direction from a variety of methods. Two methods stand out—an organization-wide strategic plan for the DOT and a research strategic plan specifically for the research program as shown in Figure 14. Half of states use either a DOT-wide strategic plan or a research strategic plan, and over half of research programs do not set their research direction through a research strategic plan. For more than one-third of research programs, research direction is set in a less formalized manner and occurs 0 10 20 30 40 50 Administer Research Funding Manage Research Projects Perform In-House Research Promote Agency Innovation Monitor Implementation Other Agency Count Figure 12. Research program responsibilities. Almost all state research programs administer research funding and manage research projects. Only half conduct “in-house” research within the DOT. 0 5 10 15 20 25 30 Asset Management Library Services Training Services Information Technology Other Agency Count Figure 13. Research program’s non-research responsibilities. A majority of programs manage library services. Very few perform training services or asset management. None are responsible for IT. Figure 14. Research visioning through strategic plans.

Survey Results 45 through discussions within the research program itself, or with executive leadership, mid-level managers, or various advisory boards or committees. Many programs feel that more intentional research direction setting is needed within their DOTs. These programs feel that breaking down barriers and pulling together various stakeholders from across the DOT would help improve the program and meet concrete objectives of the DOT. Many respondents stated that they hope to develop or redevelop a research strategic plan, which can be a fairly comprehensive but resource-consuming effort. Others prefer a simpler method for defining their overall program strategy through greater engagement by senior leadership. The Florida DOT commented on a desire for cross-functional coordination within the larger agency to identify, set, and implement vision, goals, and objectives. For many agencies, research direction is set through the needs assessment stage of the research lifecycle. This means that the strategic direction of the research program is based on that year’s research topic submissions and is there- fore likely to change on an annual basis. Additionally, this type of research direction is more likely to be focused on research projects and not on program-wide improvements. Research Expertise How research is conducted, and who ultimately conducts actual research, affect the overall research culture of the agency. Programs face a challenging and evolving task of obtaining varying types of expertise for research projects, research management, and research support to maximize both the quality and the value of their programs. The actual research component of projects themselves is most often contracted out to universities and non-university contractors, meaning that the research program does not have direct control over the conduct of research outside of periodic evaluations and check-ins with contractors. However, the tendency for contractors to conduct actual research also means that DOTs may be able to source more appropriate, specialized expertise for research. Research programs thus rely heavily on external groups to perform the core research function. Research programs handle relationships with a diverse group of stakeholders from their research project managers, executive leadership, research committees, state transportation inno- vation council, university transportation centers, university faculty, non-university contractors, AASHTO, TRB, FHWA, local associations, technology vendors, and so on. These stakeholders are a vast wealth of resources for transportation research programs, but how each program configures itself to tap into this research expertise must be tailored to each specific program and its needs. Note on presentation of survey results: A box and whisker plot (Figure 15) is better at showing variance than simple averages or ranges, because such a plot shows the statistical dis- persion of data. The ends of the two “whiskers” of the graph mark the range from minimum to maximum. The box itself depicts the interquartile range, meaning the 25th percentile to the 75th percentile of results. The median is shown by the line inside of the box. In other words, Figure 15. Reading a box and whisker graph.

46 Managing State Transportation Research Programs the box depicts the middle 50% results for state agencies. Finally, outliers are depicted by points beyond the whiskers of each graph. Although half of research programs may conduct in-house research, this Synthesis survey found that the most research sponsored by the research program is conducted by university con- tractors, as seen in the box and whisker plot in Figure 16. State agencies use university research- ers in differing amounts, ranging from none to 100%. However, state agencies are concentrated within 50%–87% of research budgets allocated to research projects conducted by universities. Many transportation research projects may require specialized engineering expertise that is more readily available at universities compared to most DOTs. Universities may also have a university transportation center, a consortium of colleges and universities that partner on spe- cific research topics to advance transportation research and technology while developing the next generation of transportation professionals. This makes universities a natural partner to DOT research. Half of the state transportation innovation councils also benefit from university participation; this provides additional perspectives on innovation and potential for accelerated implementation. However, this participation does not substitute for a current lack of integration between half of research programs and their STICs. Although university contractors conduct most of the research, non-university contractors are also used. Whereas university researchers span both basic and applied research, non-university contractors tend to focus on applied research. This gives non-university contractors a unique perspective and an ability to frame, execute, and present research to a variety of audiences. Research programs perceive non-university contractors as valuable because they offer special- ized skill sets that the state DOT may not have internally. Non-university contractors are also valued for their ability to provide capacity that the DOT might otherwise lack (Figure 17). Outside of universities and contractors, agencies also utilize AASHTO and TRB to sponsor cooperative research; to facilitate knowledge sharing; to provide training and state-of-practice updates; to identify ongoing research, national connections, technology transfer, and program development through pooled funds; and to facilitate peer exchanges. Survey respondents also report that FHWA division office staff serve important roles, including on technical advisory committees for all research projects in some states. Partnerships with local associations typ- ically occur on a more ad hoc basis and are done at a project level rather than through an Figure 16. Who conducts actual research for the research program? Most research programs use university contractors to conduct a majority of research.

Survey Results 47 institutionalized approach for most agencies. Technology vendors also assist state transporta- tion research programs by providing technologies to improve processes and construction prac- tices, software support, data hosting, research and deployment ideas, and at times, technology transfer and development on some specific projects. Research Funding The capabilities of a research program are affected by the source, spending, and require- ments of their research funding. Approximately half of states surveyed had an agency budget between $1 million and $6 million for the last fiscal year (Figure 18). Florida, California, and Texas are outliers with $15 million, $25 million, and $31 million in annual research budgets, respectively. A majority of states receive most funds from State Planning and Research (SPR) funds with an associated 20% match from state funds. Some state DOTs receive substantial Figure 17. How do agencies value the role of non-university contractors? Non-university contractors are valued by research programs most often for their specialized skill sets and ability to provide additional capacity to the program. Figure 18. What were agency budgets for the last fiscal year? Half of research programs had a total of $1 million to $6 million in research budget for the last fiscal year.

48 Managing State Transportation Research Programs funds from other federal grants, such as Colorado and New York, while others also gain sub- stantial funds from multistate pooled funds. SPR and state funds tend to have wider purposes than other federal sources and multistate pooled funds, which affect what type of research can be funded. Research programs allocate their budgets to a variety of activities, such as national programs, multistate pooled funds, university transportation centers, DOT research, DOT research imple- mentation, other DOT activities such as knowledge management and training, research pro- gram administration, local technical assistance programs, or various other activities. Figure 19 demonstrates how research programs allocate budgets across seven common research activity expenditures. Many state research programs do not pre-allocate funding into separate buckets at the begin- ning of a fiscal year. Instead, programs use the research topic solicitation process in order to determine how funds are spent. Most research programs also maintain reserve or discretionary funds, which may provide more flexibility for meeting immediate research needs outside of the normal research solicitation process. According to survey responses, pavements and bridges or structures receive the most funding from research programs. Fulfilling funding compliance requirements is of “moderate difficulty” across the country. Research programs tend to have set accounting systems, procurement procedures, and dedi- cated DOT staff in place to manage specific fund compliance and reporting requirements. How- ever, disaggregation of these data reveals disparities between state research programs. Some smaller programs expressed capacity challenges in keeping up with everything that needs to be done, such as scheduling peer exchanges. The next section details survey findings on how research programs manage their programs specifically throughout the research lifecycle. 2.2 Program Management Program Management comprises the processes and protocols in place to execute, manage, and deliver the research function over the research lifecycle. other research program administration DOT’s other activities DOT’s research implementation DOT’s research university transportation centers pooled funds national programs (AASHTO, TRB, NCHRP, etc.) 0 25 50 75 pu rp os e (% of total) Figure 19. How do agencies spend their research budgets? Most of research programs’ budgets are spent on DOT research, followed by national programs, pooled funds, and research program administration. Most research programs spend only a small percentage of research budgets on research implementation.

Survey Results 49 Given the diversity of purposes, responsibilities, funding levels, and ways of sourcing exper- tise, research programs naturally follow different practices to deliver research throughout the research lifecycle. However, six distinct stages are common across all research: assess needs, select topics, award projects, manage projects, disseminate findings, and implement research (Figure 20). Assessing needs and selecting topics differ by agency depending on how broadly the state DOT defines its research needs. These processes continue to balance between top-down and bottom-up approaches. The survey found that most programs utilize top-down approaches through a research advisory committee that prioritizes research topics, and half of programs also use a bottom-up open submission and voting process. Approximately one-fourth of states use focus groups as a more active bottom-up needs solicitation strategy. A number of states also use a “vetting” process where open solicitation is accepted and research staff ensure that the needs can be met before the topics are submitted to a research advisory committee. New Mexico and Arizona are two example states that use this vetting process. Some unique examples of needs assessment and topic selection revealed by the Synthesis survey come from the larger research programs of Caltrans and the Florida DOT. Caltrans uses a research prioritization method to score projects based on how well they address the depart- ment’s strategic goals. The Florida DOT uses functional area working groups to prioritize func- tional area submittals with subsequent agency voting and executive review. How programs manage individual research projects vary by state but generally require coordination across many DOT divisions (Figure 21). Non-research program individuals serve as research project managers in a majority of states: approximately three times as many state DOTs use research project managers from other programs and offices within the state DOT than research project managers from within the research program. As a result, project Figure 20. Research lifecycle. 0 5 10 15 20 25 30 35 40 Research Project Managers (employed by your DOT) Research Project Managers (contracted by your DOT) Other Research staff employed by your research program Other DOT staff members Technical Panels Other Agency Count Figure 21. Who manages research projects at your DOT? Most agencies use research project managers employed within the DOT. Around half of research programs use technical panels. A quarter of all programs use contracted research project managers or other research program staff.

50 Managing State Transportation Research Programs managers may need to balance multidivisional responsibilities while having a clear under- standing of the procedural and quality control requirements of research. It is common for non-research DOT staff members to manage the research projects that they request in the research solicitation process, and half of states also use technical panels to help manage the technical aspects of research. Research project managers serve the important role of seeing research through its entire process. State agencies prioritize different types of qualifications for these individuals as shown in Figure 22, but they generally find experiences with managing research projects and experi- ences working in specialized research areas to be of highest importance. Given the diverse set of research topics that a research program may handle, smaller programs appear to prioritize research management experience over specialized research experience for their research project managers. Less important to agencies was professional expertise with the geographical context of the research area (i.e., same state or locality). About half of states manage the technical aspects of project management through their technical panels. This allows states to look for more general expertise in their research proj- ect managers, such as “critical thinking, problem solving, decision making, ability to work independently, etc.” (Montana DOT). Other state agencies have tiered strategies in which a research advisor may have a technical background, while project coordinators are considered management analysts (Minnesota DOT). Sometimes, the project manager is the same individual that requests the research. Although this may help ensure that the research need is being met, the project manager may not always have the appropriate knowledge of how to oversee the research process. Support for research management is accomplished in different ways as well. Research pro- grams frequently have staff members with expertise in finance, accounting, and contracting or procurement or both, which is supplemented by other DOT divisions in most cases. Research administration and library services are more often done within the research program itself, while information technology expertise is by-and-large obtained from other DOT divisions when necessary. Research project managers serve the important role of overseeing project execution from beginning to end. The next section reviews how research programs manage their projects to Figure 22. What types of experiences are most important in a project manager?

Survey Results 51 deliver program quality. The last section synthesizes how research programs manage research dissemination and implementation processes to maximize the value of their programs. 2.3 Program Quality Program Quality is the rigor and diligence with which the DOT adheres to scientific prin- ciples and best practices and is efficient and effective at pursuing research. The ultimate purpose of transportation research is to enhance the transportation system by generating new knowledge, increasing the stock of knowledge, and using this knowledge for new applications. Maintaining a research program of high quality improves program outcomes, ultimately providing a better transportation system. This means that standards and procedures must be put in place to ensure that projects sponsored by the program are accountable to a baseline level of research quality. Research program oversight responsibilities come in the form of strategic direction, project selection, project execution, budgeting, and implementation. Yet, for most programs, the sheer number of research projects sponsored by the research program may mean that oversight on execution falls to the project manager and not to the research program manager. Consequently, most research program managers do not have oversight on research project execution because that responsibility falls to project managers. Instead, research program managers have over- sight over the strategic direction, project selection, budgeting, and implementation processes. A majority of states have some type of quality control method in place for research activities. The most common method is quality control review by a project manager, sometimes supple- mented with technical review by a technical panel. Some states also use additional quality control tools such as copyediting, manual of research procedures, deliverable-based payment, payment withholding for quality, or quality assurance and quality control plan requirements within proposals. Most agencies do not currently use peer review processes for their projects, meaning that most research is not reviewed by another researcher unaffiliated with the project who works in the same field of research. However, responses may be limited by knowledge of those surveyed, because peer reviews may be done by papers generated from research projects, separate from actual projects themselves. Research evaluation helps research programs to better understand program processes and effectiveness, to evaluate individual research performance, to inform new research project selection, and to share lessons learned from the research program. However, the majority of states do not use research evaluation to learn how to direct funding. This may be because agencies place a greater priority on the DOT’s current strategic research needs than on developing a research agenda based on research evaluation. Notably, around one-fourth of states surveyed do not have a consistent process for evaluating research programs. Many state DOTs have basic project-level measures that ensure research quality (Figure 23). This includes whether projects are completed in line with the budget and schedule, tracking project deliverables, formatting and presentation of findings in documents, and so on. Programs may rely on research project managers and technical advisory panels to review for quality; some are beginning to create quality control standards to control for quality across the research program, such as through a research manual. While research may be executed efficiently and effectively, follow best-in-class practices, and adhere to scientific principles, research must be useful in order to reach meaningful conclusions and to generate value. The next section explains how research programs maximize their value.

52 Managing State Transportation Research Programs 2.4 Program Value Program Value is the usefulness of new and enhanced knowledge gained through research and the degree to which it is applied to enhance the transportation system. Research must be intentionally shared with target audiences in tailored methods and then strategically implemented to realize value. Both research dissemination and research implemen- tation tracking tend to be done in full or in part by research programs. A lack of external knowledge sharing for practitioner learning remains at many state DOTs. The survey found that most states do not consider researchers in other state DOTs or in universities to be main recipients of research. Rather, research dissemination is intended to be for the DOT in general, and to the DOT’s leadership. Research dissemination is most frequently pursued through a research report and the DOT’s website, which previous literature labeled “passive” forms of dissemination. These methods are simple ways of shar- ing research but may not always be appropriate ways to reach the target research recipients. Passive dissemination may reach a limited number of DOT employees who are naturally already interested in the research topic. At the same time, by not targeting other researchers through other state DOTs and universities, the risk is run of duplicating research done by others in the past. Research programs often also struggle with methods to participate in research implementa- tion, and this is currently also done in “passive” forms at most agencies. A disconnect forms frequently due to different personnel and a time lapse between the research and the implemen- tation. Research programs traditionally view the individual who requests research on a topic as the individual ultimately responsible for translating research findings into practice. However, research programs often help manage or administer implementation activities, and half of them are involved in selecting candidate implementation projects. For example, most programs state that they track implementation by maintaining a list of completed research that is implementable. At the same time, only about half of states also follow the progress of implemented projects for a predetermined amount of time, and a few agencies assess the benefits or return on investment of research implementation. 0 5 10 15 20 25 30 35 Peer assessment Alignment to state DOT's strategic initiatives No consistent method for evaluating completed research projects Other Post-project tracking for implementation Assessing potential benefits prior to beginning project Fulfillment of research objectives Completion of project on schedule Completion of project in line with the budget Agency Count Figure 23. Performance measures. The majority of research programs track the completion of their projects in line with the budget and schedule. Approximately half of programs conduct post-project tracking for implementation.

Survey Results 53 The main barriers to research implementation come in two forms: limited resources for implementation and DOT-wide barriers (Figure 24). Limited funding for implementation and lack of dedicated staff empowered to lead and to be held accountable for implementation consti- tute the main resource constraints. DOT-wide barriers are those such as misalignment between research projects and DOT priorities, shifting DOT priorities over time, or the inertia of the status quo. There is a need to solicit internal DOT acceptance, funding, and designated person- nel ahead of time so that the value of research can be fully captured. There is great variance among state agencies as to how they maximize implementation suc- cess. A few states have dedicated implementation engineers or other individuals specifically dedicated to support implementation, such as the Performance Coordinator at Florida DOT or the Innovation and Implementation Group at Utah DOT. This dedicated position is not always feasible for state agencies, however. Additionally, research funding does not depend on successful research implementation in a majority of state DOTs, with a few exceptions. There are some plausible reasons why success- ful research implementation is not a prerequisite for funding. Many research programs believe that not all projects should be implemented; as a center of innovation, research programs may want to encourage a degree of risk taking and discovery. This means that some research projects that are of “higher risk,” meaning not immediately implementable, might still be encouraged by the agency. An atmosphere conducive to innovation might, therefore, wish to be accepting of non-implementation in some cases. A hybrid model to efficiently allocate funds to projects with greater chances of implementation success might include a phased research structure with built-in checkpoints to determine whether to continue pursuing a research project. In some cases, it may also be helpful to alter the research project scope after a phase is complete, in order to derive findings that are tangibly meaningful to the agency when the project is complete. Agencies are moving toward program-wide assessment of research outcomes and program value. Program value can be actively communicated through regular assessment of the program’s value, outcomes, and the overall information generated that is used to make decisions. Most research programs do not assess their value and outcomes from a program-wide basis. However, more than half of them currently do so on an individual project level which means that pro- gram value and outcomes can be derived by “rolling up” individual project outcomes. Of states that measure on a program-wide basis, many more agencies use quantitative measures than use qualitative ones. While creating quantitative performance dashboards across an entire research Figure 24. Factors inhibiting research implementation.

54 Managing State Transportation Research Programs program is by no means an easy endeavor, tangible measurements such as cost savings, lives saved, reduction in hours spent, and so on are not the only methods for defining value. Qualita- tive measures add new dimensions by adding claim to how the research results in lessons learned, improved processes, and new standards and policies that may be more difficult to quantify. 2.5 Conclusion This Synthesis adds to the wide body of literature on managing state transportation research programs by evaluating programs as a whole with their various structural and pro- cedural attributes and constraints. By evaluating research program capability, management, quality, and value simultaneously, this study is able to pinpoint how different state agencies organize themselves to generate program quality and value. The next chapter provides an in-depth analysis into five state agencies: District (of Columbia) Department of Transportation, Louisiana Department of Transportation and Development, Minnesota Department of Transportation, Ohio Department of Transportation, and Utah Department of Transportation.