Managing State Transportation Research Programs (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 State transportation agencies conduct applied research with a goal of ultimately creating new knowledge to enhance the transportation system. Agency research as an activity requires special skills and capabilities—it convenes practitioners, scholars, and policy makers to identify and pursue the knowledge that is most needed. These and other attributes of research make it unlike other department of transportation (DOT) functions such as planning, programming, construction, maintenance, and operations, even though it eventually enables agencies to perform those functions. The payoffs and innovative outcomes of research can be significant and valuable, although they are rarely immediate. Applied transportation research is a challenging pursuit. Research at state agencies is time intensive, often integrates diverse disciplines to address unanswered questions and to solve specific agency problems, requires deep technical or policy knowledge, exhibits uncertain outcomes, and must be assessed using a mix of quantitative and qualitative assessments. Applied transportation research strives to integrate knowledge across diverse disciplines to solve practical problems facing the transportation system. Agencies must intentionally organize for research so that they can navigate its challenges and make the pursuit meaningful. There is little explicit guidance on how to organize for research. In the absence of program management tools and resources guidance, some agency research programs have matured and become more effective over time. Others continue to struggle with developing programs that are capable of sustained quality and value over time. The diversity of agency circumstances and approaches obscures how some have intention- ally assembled various “building blocks” or ingredients to make their research programs more effective. What is missing is a framework to understand what these basic ingredi- ents are and how they often come together in a research program to deliver quality and value. Such a framework could help agencies assess their own programs and identify the missing ingredients in developing their own recipes for success. Such a framework could also inform future research and policy efforts to develop guidance, recommendations, and prescriptions. This synthesis study accordingly addresses the question “How do transportation agen- cies organize and manage their research programs to strive for quality and positive impacts on the transportation system over time (value)?” In response, the study team developed a four-dimensional framework to analyze and shed light on how state DOT research programs with differences in agency needs, resources, and constraints are able to produce programs of high quality and value. S U M M A R Y Managing State Transportation Research Programs

2 Managing State Transportation Research Programs The study team used multiple methods to address this scope, including a literature review, survey-based data collection, and case example interviews. • The team first evaluated an extensive body of literature published over the last 15 years to orient the study. These secondary sources shed light on a wide range of research program structures, practices, challenges, and lessons learned. However, there is no comprehensive scientific body of literature on “best practices” for managing state transportation research programs. There also continue to be major gaps in the body of work. For example, there are few published statistics on the range of agency research budgets across state DOTs, the distribution of amounts from various funding sources, and how agencies allocate these to different streams of research. There have been few attempts to connect these resource aspects with program management choices or with program outcomes such as the degree of implementation. Other outstanding questions include how agencies assess the quality of their research, and which means of dissemination they deem most valuable. While there are many variations in program structures and processes, it becomes clear that these can be better characterized through targeted primary data collection. • To address these gaps, the team collected primary data in two ways. The first was a detailed online survey to which 44 agency research program representatives responded (88%). The survey was composed of 46 closed-ended and open-ended questions. The second approach followed with interview-based case examples of five agency programs: District (of Columbia) Department of Transportation, Louisiana Research Transportation Center, Minnesota Department of Transportation, Ohio Department of Transportation, and Utah Department of Transportation. The team chose these five agencies because of their diversity and unique organizational arrangements and practices. The primary data from surveys and case examples were analyzed to complement the knowledge gleaned from the literature review. Information from surveys, case examples, and the literature review were synthesized to document how state transportation research programs organize and manage themselves for program quality and value. This study documents state transportation research program structures and processes as seen from a bespoke framework of four dimensions: program capability, program manage- ment, program quality, and program value. Figure 1 illustrates the framework developed in the course of this study. Figure 1. Managing state transportation research programs framework.

Summary 3 Main study findings correspond to the four framework dimensions and are as follows: Program Capability is the ability of the DOT to undertake research. This dimension touches on goals, knowledge, expertise, resources, organizational structure, and external stakeholder relationships. 1. DOT research budgets range widely from slightly less than $1 million to about $30 mil- lion per fiscal cycle. About half of DOT research program budgets are between $1 million and $6 million. Most research funding comes from State Planning and Research (SP&R) Part B. Most research spending comprises the areas of pavements and bridges or structures. 2. A research program’s role or size does not appear to depend on where it is situated within the DOT organizational structure. Most state transportation research programs tend to be located within the Planning or Chief Engineer Division of the state DOT or some- times in more specialized divisions. Day-to-day working relationships between research program staff and other DOT employees dominate organizational reporting relationships. 3. Intentional planning for research continues to be an agency need. Research programs set strategic priorities and establish the overall direction of their research efforts in a variety of ways, but most wish to better engage with executive leadership and broader groups of research stakeholders to refine DOT needs and priorities. Some DOTs have research strategic plans to guide their program, but most research programs look to the broader DOT strategic plan and participate in conversations with DOT leadership and advisory committees. 4. Research programs rely on external groups to administer research administrative functions. Agencies rely on national programs such as TRB—including NCHRP—and AASHTO to conduct research and to provide training, publications, and surveys. In addi- tion to these forums for knowledge sharing, ongoing research identification, and technical support, agencies periodically participate in peer exchanges to enhance their capabilities. The U.S. DOT’s FHWA provides additional resources and local office support to state trans- portation research programs, such as engagement in periodic research peer exchanges. 5. Research programs also frequently use external groups to perform the core research function. Universities conduct most of the research sponsored by DOT research pro- grams across the country. University faculty may also participate on a research advisory committee and/or serve as a technical expert for projects. Non-university contractors conduct a small percentage of research projects at most agencies. Only about half of research programs conduct any in-house research. 6. Desired research program staff skill sets include a fundamental research background or technical expertise or both, research administrative support competencies, and management and communication skills. While technical expertise may vary depending on whether the research program conducts in-house research, a fundamental research background to ensure the technical quality of research with strong scientific rigor and inquiry is a core competency for research project managers. Research programs respon- sible for research support functions also require competencies in contracting, legal, accounting, and other administrative services. In addition, management and communi- cation expertise is needed to foster collaboration between different research stakeholders and to successfully administer research contracts. 7. A DOT’s ability to innovate is closely tied to how other parts of the DOT champion and engage with the research program. Half of research programs consider “promot- ing agency innovation” as one of their main responsibilities. Research programs also manage knowledge of research methodologies, funding programs and resources, and library services to assist DOT staff with their needs. Few agencies currently have pro- grammatic responsibilities for staff training or continuing education to disseminate new knowledge, however.

4 Managing State Transportation Research Programs Program Management comprises the processes and protocols that the DOT uses over the research lifecycle. This dimension covers different aspects of management, execution, and administration. 1. Research needs assessment processes continue to be a balancing act between top- down and bottom-up approaches. Most research programs currently use top-down processes through research advisory committees, and half of programs also engage in bottom-up open submission and voting processes. Many programs are supplementing these with stakeholder workshops and other exercises to refine research needs and to determine how best to channel resources. 2. Managing projects and administering the research program require program staff to coordinate across many DOT divisions. Subject matter experts from various functional divisions often participate in research projects, as either idea champions, project managers, or on advisory panels. Ancillary research management and administrative functions such as finance, accounting, information technology (IT), and contracting and procurement tend to be conducted outside the research program itself by DOT-wide groups. 3. Research programs emphasize the importance of making the outcomes of research more accessible to stakeholders. Programs most often passively convey research out- comes through project final reports and program websites. Some DOTs develop research briefs, presentations, and newsletters to not only engage the DOT’s own leaders but also to touch a wider audience, albeit in a limited manner. Program Quality is the rigor and diligence with which the DOT approaches research to make it efficient and effective. Following scientific principles, consistent research and experimental techniques, and other best practices fall under this dimension. 1. Most agencies focus on project-level measures of quality. This includes whether proj- ects are completed in line with the budget and on schedule, tracking project deliverables, formatting and presentation of findings in documents, and so on. 2. Quality control reviews provide additional rigor to research methodology and find- ings. Many programs rely on DOT project managers to critique the research deliverables, and some consistently establish technical advisory panels to provide oversight over tech- nical aspects of research. 3. Agency research typically is not subject to peer review. This means that DOT research tends not to be reviewed by another researcher working in the same field of research. However, research studies published in journal articles could be peer reviewed and are currently not tracked. Some agencies are developing peer assessment programs to enhance research quality. 4. Some agencies are creating quality control standards. To standardize quality across the program, some agencies have created research manuals, payment schemes based on qual- ity, and other methods. Others require robust quality control plans from researchers at the start of the project to introduce more specific measures of quality for the project. Program Value is the usefulness of new and enhanced knowledge and the degree to which it leads to an improved transportation system. This dimension focuses on research out- comes and impacts. 1. The barriers to external knowledge sharing and practitioner learning remain high. Research programs currently prioritize research dissemination within the DOT over the external dissemination of research findings. Programs tend to consider their state DOT— and in particular, executive leadership—to be the main audience of their research. Exter- nal recipients such as local practitioners, other state DOTs, and universities or researchers are usually not dissemination targets.

Summary 5 2. Research programs often struggle to participate in research implementation. Pro- grams have traditionally viewed their research “customers”—those who champion or request research on a topic—as responsible for translating research findings into practice. At the same time, programs often help manage or administer implementation activities by maintaining a list of completed research deemed implementable, coordinating the selection of implementation projects, and administering related budgets. Not all agency research programs participate in implementation and of those who do, only about half track project progress over time. Few agencies assess the long-term impacts of research implementation. 3. Research implementation by the DOT itself is most often unsuccessful due to the limited resources for implementation and DOT-wide barriers. The lack of (or very limited) funding for implementation and lack of staff who are empowered to lead and are held accountable for implementation are the main resource constraints. Wider orga- nizational barriers include misalignment between research projects and DOT priorities, shifting priorities over time, or the inertia of the status quo. 4. Research and implementation performance are not linked to current or future research funding at most programs. Research evaluation is typically not used to direct or allocate research funding for future projects. Additionally, funding is not premised on successful implementation at most research programs. However, research evalua- tion is linked with individual researcher performance evaluations, and some research programs require quality control plans and consider implementability as a part of the project selection process. 5. Agencies are moving toward program-wide assessment of research outcomes and pro- gram value. More than half of agencies only assess project-level indicators and do not use program-wide measures. Of those that do report program-wide outcomes, more programs use quantitative measures rather than qualitative measures. Although quanti- tative measures may be easier to “roll up” into aggregate program outcomes, agencies find that qualitative narratives can help document the effects of improved processes, lessons learned, the rationale for new standards and policies, and other lessons that may be dif- ficult to quantify. Beyond these synthesized findings, the team also identified several areas of future research in the course of this study, including gaining knowledge of research visioning, program quality, and program value from those to whom the research program is accountable, as well as the nuances of program expertise, funding, and project management from research program staff and project managers. This Synthesis also recommends that further case examples be conducted based on this research program framework to understand how the mandates, structures, and processes both internal and external to a research program influ- ence its ability to deliver program quality and value to the department, other states, other researchers, and ultimately to the wider transportation system.