Interim Guide for Developing a State Transportation Research Manual (1995)

As the research effort focuses on customer benefit, it is conducted with an eye toward implementation. The implementation process is aided by the exchange of information, which starts with clear, concise and complete project reports. These reports detail the progress and accomplishments of a research project and are written with the customer in mind.

The proper reporting of the individual parts, represented by the projects, will enhance the evaluation of the entire research program.

Commentary:

For this phase of the guide, the section on project reporting is placed under Program Evaluation. In the final guide, the project report section will be placed with a section on Reporting.

Each of the major tasks outlined in the work plan will be briefly described, whether they have been completed or are still in progress.

The planned and actual time schedule for each of the tasks will be shown. The overall percent complete will be shown, using the expended versus planned budget.

Financial, staff, equipment and technical problems will be discussed, as they affect the individual tasks. Their resolution, or attempts at resolution, will also be stated.

Each task will result in a milestone. The milestone could take the form of a summary report, a design of an installation, an installation completion, data collection completion, a specification, etc. A brief description of the steps taken to reach the milestones will be given for each task.

As each milestone will describe the completion of a task, each task may result in an accomplishment. The significance of the accomplishment will be discussed with respect to its advancement of an implementable product. This section of the report is the most important to the client. The potential success of the research and implementation of the results is stated here. The implementation results of the entire program are discussed in section 4.2.2.A.

The line items of budgeted funds for salaries, overhead, travel, equipment and a miscellaneous category will be shown. Contracts will list the same items.

The line item expenditure of funds will be shown for salaries, overhead, travel, equipment and a miscellaneous category. The same line items will be listed for contracts. The budget and expenditures will be shown in the same table.

Commentary:

The reasons for giving the technical and financial information, presented in A., B. and C. above, are to keep the client informed and let the manager track progress against expenses. This level of data may be unrealistic for very small or short term projects. There may also be a lag between the time the reports are issued and the availability of the expense information. To account for the lag the reports can either be delayed to enter the financial data or the financial data can lag the technical data by a cycle. Small or short term projects may not require detailed information for reports. These projects can be defined by level of staff effort, total cost or some other criteria.

COMMENTARY:

Every agency produces reports. The type and frequency of the reports will vary with the research project, the expectations of the customer, funding source requirements and other factors. The reports listed below are used most commonly.

The cyclical report can be produced on a quarterly, semiannual or annual basis.

The Project Committee ( Section 2.2.2.C ) members are the principal reviewers of the cyclical and interim reports. Their meetings will include a review of the findings in these reports.

Project reports will be written quarterly, incorporating the information in 4.1.2.A, 4.1.2.B through 4.1.2.C.

Projects that are expected to take more than two years to complete or are expected to have a significant accomplishment during the course of the research will be detailed in an interim report. In addition to the information in 4.1.2.A, 4.1.2.B. through 4.1.2.C., the interim report will discuss the implementation process and expectations. This report covers a significant part of the research, including impediments to implementation, and suggestions for overcoming the impediments.

The committee members associated with the project are aware of the findings prior to the final report. However, the research community and operational units affected by the work must be informed. The final report is the most lasting and complete document of the research and will be

carefully assembled to include at least the following information:

• Summary, including a brief description of the work and conclusions;

• Recommendations, based on the findings and conclusions; suggestions for additional research;

• Implementation Plan, defining the procedure to introduce the results into practice, including suggestions for organizational responsibility;

• Introduction, including the problem, its background and a concise history of research;

• Work Plan, including the experimental research plan, data collection, description of sites and activities and an analysis of the data;

• Findings and Conclusions.

The project reports are the official documentation of the research the quarterly reports are used to monitor progress. The interim and final reports form the basis for discussion of the research and presentations to the transportation community. The output of this section is the technical and financial status of a project in cyclical and final report form that is the basis for the implementation effort.

The public expenditure of funds is subject to careful scrutiny. The profit motive doesn't exist in the public arena, hence, the programs in the public arena that receive these funds must prove their value in other ways. After carefully selecting problem statements and developing the work program, the research effort must follow scrupulously defined procedures that ensure unbiased and meaningful results. On an individual project basis, these results are very meaningful. On a program basis, the projects should be aggregated to appreciate the cumulative effect of the program.

COMMENTARY:

The phase 1 guide coverage of Program Evaluation does not include some important aspects of the program, namely, the success of technology transfer efforts and an evaluation of economic benefits from the program. Subsequent phases of the guide will cover these areas. This section is focused more on the implementation, administrative and scheduling features.

COMMENTARY:

This aspect of program evaluation involves a large record keeping effort. As with previous sections of the guide, the research managers are reminded that they should be selective in their choice of subsections and content of each subsection.

The section on implementation results is too important to the research program to omit it. Besides, the FHWA regulations require a tracking effort for this activity.

The easiest method of tracking may be the use of tabulations; updates can be made quarterly. The actual items to be tracked can be selectively chosen by the manager, but the actual implementation successes should be documented.

The section on Milestones is actually an activities list. Perhaps not all of the suggestions are necessary, but they do show the range of effort in which the research unit is involved.

The data collected for the Funding and Schedule Adherence sections are necessary to satisfy the requirements of the FHWA. The cyclical report (quarterly, semiannual or annual) for each project can be the vehicle to transmit this information to the FHWA. Tracking the total program funds may require interaction with the accounting unit of the agency.

The last section, Benchmarking, is an attempt to put generalized data for the research unit into a performance setting. This is not a

mandated element of the manual, but the research manager should consider some aspect of benchmarking. There may be several qualifications that mitigate each of the suggestions made. But there may be other more acceptable parameters that can be used.

The implementation efforts of the individual projects were discussed in section 4.1.2.B. Summary tabulations of the project efforts will document the progress for the entire program. The tabulations will include implementation discussions and actions:

• during the project work plan preparation,

• at all project level-level meetings,

• during project field visits, and

• at any specific implementation meeting.

In addition to the tabulations listed, all partial or full implementations will be documented. Although a project may have been formally closed out, records of the subsequent implementation successes will be maintained.

The work program is the sum of all activities planned for the year. These activities are primarily projects, technology transfer efforts and technical assistance, seminars and implementation efforts. Milestones are set for each of these activities:

• such as installation designs or interim reports for projects (refer to section 4.1, Project Level Reporting);

• LTAP meetings or field inspection visits for technical assistance;

• public technical meetings for seminars (refer to section 2.1, Customer Support Development); and

• discussions and field visits for project problems, information dissemination and advancement.

A record of each of these activities will be kept.

COMMENTARY:

The achievement of the milestones that are set at the beginning of the year may also result in technical accomplishments. The technical accomplishments are defined under 4.1 , Project Level Reporting. They don't have to be repeated here.

Each funding source used for research has been programmed for the various activities in the work program. In addition, each activity has a specific budget (Section 3.2, Project Prioritization). A record will be kept for both the project level and funding source expenditures. Allowances are made for overexpending on the individual State Planning and Research (SPR) projects for the year, but the total program funds for SPR or other funding sources cannot be exceeded. The record will reflect the reasons for the individual project over or under runs.

The projects are the most important activities as far as schedules are concerned. Most other activities can be planned throughout the year. The ability to adhere to the schedule for a project is contingent on many factors. The principal investigator and unit manager will be in frequent communication with each other to avert major slippage.

The quarterly (semiannual or annual) report (section 4.1, Project Level Reporting) will reflect the percent complete for each project; the planned and actual time schedules will also be shown.

The achievements of the research program cannot easily be reflected on a total performance basis. The diversity of the activities is too large to permit their summation. However, the quality of the program can be judged by observing the progress of some of the measurable parameters. Benchmarking not only demonstrates research's progress but it also improves the progress by a quality improvement thrust. Some of the factors that will be benchmarked to show the performance of the program are:

• programmed funds

• staff research projects

• contract research projects

• milestones

COMMENTARY:

The benchmarking process requires an attention to data collection that will result in the comparison of similar information. The factors suggested above can be best compared on an annual basis. Other parameters may be compared on a semiannual or biennual basis. But the point of benchmarking is to run a time series of the factor to judge its trend.

The selection of parameters is very important. Qualification of the data is cumbersome in a graph, so it is necessary to select parameters that can stand on their own. It should also be kept in mind that the factors listed above, when put in parameter form, may not necessarily reflect that more is better.

The documentation of a successful performance of the research effort is important to continue to receive the management and financial support that it requires. Objective and quantifiable parameters can give the basis for this support. Overall program performance can be measured by a combination of the achievement of implementation and milestones and a qualified adherence to financial and scheduling limits.

A quality research program depends upon its ability to implement effective and timely solutions to the problems of the Agency. It is the execution of the staff-and-management-developed procedures and processes that ensures the attainment of this objective. One technique designed to improve the quality of the program is a peer review examination of the deliverables of the research unit through the management system. A panel, with knowledge of state research programs, will bring that expertise to a study of the research process and advance recommendations structured to enhance its performance.

COMMENTARY:

The peer review is meant to assist the research staff with program performance. (Refer to Sections 4.1, Project Level Reporting and Section 4.2, Overall Program Performance ). The FHWA mandated its use in 23 CFR Part 420.207 (refer to the Appendix ). The peer review is an activity independent of the research process, conducted by non-agency staff. The agency can accept its reporting as it would any other report that is designed to improve management processes.

The review team of at least two members will consist of representatives of the FHWA, universities, the Transportation Research Board, the private sector, other agencies and the research units of other states. At least two of the members of the team will be drawn from a preapproved list compiled by the FHWA.

COMMENTARY:

The cost of travel of the peer review team will be charged against the SPR program. The state can assist the individual members with accommodations for the trip. The suggestion for the representation of a university on the panel is based on the fact that much of the contract work performed for the state research program is accomplished by universities.

The peer review team will spend at least two days with staff of the research unit. Although the items of the agenda may vary due to requests of the review team, the basic agenda will cover:

• Discussion of the research unit's management system, as described in the research manual.

• Development of the strategic plan.

• Scope of the research program, including all the activities in the work program.

• Examples of a project as it advances through the system, including the solicitation, selection, choice of researcher, project progress and technology transfer activities.

• Discussion with select clients.

• Review of resources.

• Review of staff training program.

• Review of contract process.

• Review of technology transfer efforts and implementation activities.

• A discussion of recommendations in the form of the processes of other states.

COMMENTARY: The FHWA has not yet developed procedures to conduct peer reviews. However, the state should include a peer review process in its manual and address those aspects of the state research program that are important to it. When definitive guidelines are issued for the process the state can revise its manual. If the state ignores the peer review process it will not be in compliance with the regulations.

The agenda of the peer review is intended to advance the more successful research processes within the agency. Unless all aspects of the program are discussed, such a discussion would be less than complete. The meeting can be viewed as a regularly scheduled session that looks at the progress and success of each of the instituted procedures. The harmonious spirit of the meeting can be enhanced by reducing the feeling that it is an audit review. Prior to the meeting, the members of the review team should be given the opportunity to scan the strategic plan or its substitute, the research manual and the current year's work program. This could be an excellent opportunity to fine tune the research procedures.

COMMENTARY:

The program development portion of the peer review will give the team the background information needed to understand the mechanics of the research process in the agency. The guide covers several of these areas under sections 3.1 Problem Solicitation , 3.2, Project Prioritization , 4.1, Project Level Reporting , and 4.2, Overall Program Performance .

The suggested list of material to be made available to the review team is subject to the specifics of the management plan adopted by the state.

As in previous sections, reference to the strategic plan can be omitted and a substitution can be made of the state's critical issues or the solicitation process.

As the commentary in section A.2 , Meeting Agenda, suggested, the FHWA requirements may not compel the states to include all review issues but a review of at least the suggested issues could be beneficial to the state's research program. The program can gain from the peer review teams experience.

1. Problem Solicitation and Selection

The research process will be reviewed from the beginning. Therefore, copies of the strategic plan, the solicitation letter for problem statements and the problem statements received in the current fiscal year, will be made available to the review team. The research unit will keep an up-to-date list of the members of the Research Advisory Committee and the Research Management Committee. These will be made available to the review team with the minutes of each of the meetings.

2. Work Program Process

The process for putting together a work program is a description of the early stages of the management system. The process to input the various elements is subject to policy, financial and management considerations. These issues will be discussed with the review team.

3. Contract Research Process

The magnitude of the research program conducted by contract warrants a review. The review team will be given copies of the contract research process, including the proposal review forms, a listing of all projects that were put into the contract process in the last three years, a listing of all proposals received in the last three years, the results of the proposal review process for each project and the names of all contractors selected.

COMMENTARY:

There is an expected overlap between A.3.1 Program Development and A.3.2 Project Progress . A discussion of the program cannot be complete without including the projects. The inclusion of select project discussions can highlight the elements of the Overall Program Performance ( Section 4.2 ) and serve as examples of reporting on funding and schedule adherence (section 4.2.2.C and section 4.2.2.D ).

1. Project Monitoring

The satisfactory progress and transfer of information on the projects is essential to a well managed and harmonious relationship with the customer. As a means of assessing this element, copies of each project 's most recent cyclical report and recently completed final reports will be made available to the review team. The review team will also be given a schedule and the minutes of the committee meetings for state selected projects. Research staff will detail the different types of meetings, namely, contractor vs. in-house project meetings,

and the frequency of the meetings.

Examples of the efforts made in the implementation of the project results will also be given for state selected projects.

2. Reporting

Select projects will be discussed with the review team and project staff. The basis for the discussions will be the type and distribution of reports generated by a project.

All aspects of the technology transfer effort, as described in Section 5.0, including implementation activities, will be explained to the review team. Examples of the activities will be defined, using the project briefings in section 4.3.2. A.3.2., Project Progress, above.

COMMENTARY:

This could be the most important aspect of the peer review of the research program. Although only select projects are suggested for discussion, the research unit would be in an excellent position, particularly with the agency's administration, if a complete history of technology transfer progress is maintained for all projects. Obviously, project activities are only a portion of the technology transfer effort. All customer oriented activities should be discussed with the team. The team's input to this activity could be the most rewarding part of the meeting for the research unit.

COMMENTARY:

The administrative aspects of the program are intended to show the extent and limits of resources available to the research unit. The peer review team can only use them as guidelines in a discussion of potential improvements to the program. The research unit has the option of redeploying its resources and reordering its process if the benefits of suggested recommendations are warranted. The administrative factors are not expected to be review issues. The program development, project progress and technology transfer items are expected to be the only issues on which the review team will offer positive recommendations.

For those states that do not make use of a committee structure, item a.,below, should be omitted. If a limited scale of task forces or project groups are actively involved with the program or projects, these should be listed below.

1. Committee Structure

The interactive potential of the research unit is evident in the type of committees and the representation on them.. The review team will be given all details of the committees that

interact with research, as defined in section 2.2, Research Committees Structure. The current membership, minutes from the latest committee meeting and the research process as it relates to the committees will be explained.

2. Resources

The limitations and expectations of the research unit are best defined in terms of the size of staff and budget. The current financial and staff resources, as defined in the work program, will be shown to the review team. The current budget appropriation, source of funds, allocation of funds between activities in the program, organization chart and explanation of the use of staff will be detailed.

3. Staff Training

The technical capabilities of the research team can be defined by its educational and practical experience background. The review team will be given a listing of all training programs available to staff, including state sponsored courses, research developed courses, FHWA courses and university programs in transportation. In addition, the team will be told the process that supervisors use to advise staff of the training courses of which they should avail themselves. A list of all personnel and their degrees, training courses and years of experience will be made available to the team.

The peer review team will write a report on the visit that will cover all aspects of the agenda items. The report will summarize the discussions, itemize the findings and reiterate the recommendations discussed with the research unit. Copies of the report will be filed with the research unit and the Division office of the FHWA.

COMMENTARY:

The review report can have added significance if all the points of recommendation have been thoroughly discussed and the agency supports at least a further review of them.

The research unit will request a peer review at least every three years to be held at the research office.

The peer review is a vigorous effort conducted for the benefit of the research unit. It will be accomplished by qualified peers to improve the research process. The recommendations of the team will be discussed with research staff and agency management. Every effort will be made to incorporate those recommendations that can improve the quality of the research process.

The peer review team will send a report to the agency. In addition, the research unit will also report on the outcome of discussions of the peer review recommendations within the agency. Both reports will be forwarded to the FHWA for further discussion, at their discretion.

COMMENTARY:

If changes to the research process are made, the research manual may also have to be revised. But the recommendations may only affect the manner in which the procedures are carried out and not the process.

The review process is an important aspect of the FHWA regulations and it is an important procedure in its own right. The agency has the potential for considerable gains from an extensive and intensive review of the functioning of its management system.

COMMENTARY:

The subsections below suggest the allocation of research staff expenses to participate in peer reviews of other states. When a peer review team visits the state, this section does not suggest direct billing to the state. Obviously, the research manager should revise these suggestions to conform to the policy of the state. It is assumed that a state would have no difficulty with contributing to an SPR pooled fund project to cover the travel costs.

Staff of the research unit will program time to serve as peer review team members. The staff will perform, in another state, the same review that was described in Section 4.3.2.A, Team Review of Research Unit.

Research will allocate staff expenses for an annual peer review trip. Travel funds will not be programmed to cover the cost of other team members. If a pooled fund project is set up by the FHWA, to cover the costs of the peer review team visiting the state, the state may participate in the pooled fund project.

The research unit does not expect to serve as a team member more frequently than once every three years. However, in the event that other states cannot serve, or more than one state representative is used on the teams, time will be programmed annually.

The peer review process is designed to let the states interact with other states on a formal review basis. Staff can both learn from and give guidance to other agencies on the research process. This is an excellent opportunity to participate in and gain the benefits of a nonintrusive review of the agency's research process.

The process should result in recommendations covering the problem solicitation process, work program, contract research effort, project monitoring, project reporting, technology transfer and implementation efforts.