Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

10 Developing a Measure of Effective Public Involvement The research team developed a measurement framework, crafted measurement items, and designed methods for implementing the measurement tools, all informed by lessons learned from the literature review. The research team used the following definitions of terms when developing the survey instrument, [which can be found on the TRB website (www.trb.org) by searching for NCHRP Research Report 905]: • Indicators: qualities that are indicative of effective public involvement, such as “influence and impact,” “accessibility,” and “timing.” • Items: questions that make up each indicator. For example, for the indicator of timing, items include such questions as whether the public involvement started early enough in the process, whether it lasted long enough to be effective, and so forth. • Sub-index: the score calculated from all the items under each indicator. • Index: the overall public involvement score calculated from all indicators. • Survey: the entire measurement tool, including demographic questions, questions that are used in scoring the index part of the survey, and qualitative open-ended questions. An Evolving Process The research team’s goal was to create a tool that validly measures the effectiveness of public involvement, while also being user-friendly and doable given the typical constraints faced by transportation agencies. Creating a tool that is too complicated or difficult to administer (even if valid and reliable) would not be useful. The team was guided by the axiom that the purpose of program evaluation is not to prove but to improve. The objectives were to create a survey and scoring system as follows: • The items were worded so that they could be completed by both the public and those conducting the public involvement, thereby allowing for comparable perspectives. • A consistent scale was used so that the scores on each item could easily be rolled up into an overall index (using all indicators), as well as into sub-indices for each indicator. • The scoring scale allowed responses to be differentiated at a useful level that would work for all types of transportation projects at any phase of a project. A five-point scale, ranging from “strongly disagree” to “strongly agree,” along with “don’t know” and “not applicable” response categories was used. The development of a way to measure the effectiveness of public involvement was an evolving process, with the lessons learned at each phase of the project informing the next phase, as C H A P T E R 3 Developing the Survey and Securing Transportation Projects for Testing

Developing the Survey and Securing Transportation Projects for Testing 11 well as potentially revising decisions made at earlier phases. The following are examples of this process: • As items were created, the idea of whether to combine certain indicators was considered. The team decided to keep indicators separate at the beginning, because these could always be combined later, whereas combining indicators too soon would make it more difficult to separate them later, if needed. • The team considered creating a weighted index (in which some indicators count more toward the overall measure of public involvement effectiveness). However, the team decided to use an unweighted index due to the subjective nature of assigning the weights, the fact that the weights might need to be adjusted depending on the particular transportation project involved, and the fact that a weighted index would be more complicated. • Finally, the team considered customizing the survey for each project or project stage/phase, but this idea was also decided against. The survey needs to work for different types of trans- portation projects at different project stages/phases. A customizable survey puts a burden on the transportation agencies, because they may not have the internal resources or the budget to hire outside experts to create one. Cognitive Interviews In-depth cognitive interviews were conducted with three members of the public who had participated in a transportation project in-person open house. Cognitive interviews are in-depth interviews where the respondents talk through their understanding of the survey questions and their experience completing the survey. The purpose of the cognitive interviews was to identify ways to further reduce the length of the survey, as well as identify any areas of confusion or concern about specific items. Each cognitive interview lasted about 1 hour and was audio-recorded. Interviewees went through an online version of a draft of the survey, item by item, as a researcher probed for such issues as clarity of each item, whether an item seemed duplicative of other items, and on what basis participants were deciding their rating for each item. The results of the cognitive interviews were positive. Interviewees found the survey easy to understand and pertinent to their public involvement experience. The key changes made to the survey based on the cognitive interview results were as follows: • Several items were rewritten to clarify that the survey was asking for the respondents’ personal experience or perception instead of their perception of the public’s opinion. On several items, participants responded that they could answer for themselves but not for the public. • A few items were deleted that were not able to be answered from the public’s experience. However, those items were retained for the agency version of the survey, because agency representatives would be able to answer them. • Open-ended questions were deleted at the end of each indicator section. It was the inter- viewer’s experience that these questions had several unintended consequences: they slowed down the respondent and resulted in getting answers that, for the most part, mirrored the actual items in the respective indicator section. However, a general open-ended question was added at the very end of the survey that asked the respondent what could be done to improve the overall public involvement process. Creating Paper and Online Versions of the Survey for Testing After the cognitive interviews, paper and online test versions of the survey were created. The paper version of the test survey for use with the public is included in Appendix E, which can

12 Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development be found on the TRB website (www.trb.org) by searching for “NCHRP Research Report 905”. (Note that the paper version was designed to be printed on 11 × 17 in. paper and then folded into thirds.) This was the version distributed at open houses and it offered the opportunity for participants to complete it at the open house or to mail it back to the research team. The online version of the test survey was optimized for use on tablets and smart phones. An online version of the test survey to be used by transportation agencies to rate themselves was also created. Securing Transportation Projects for Survey Testing Because testing and validating the test version of the survey required a high degree of cooperation and access, the research team linked the testing to the following projects with which the team had an existing and positive relationship: • Washington State Department of Transportation Puget Sound Gateway Program (SR 509 and SR 167 extension) • Washington State Ferries’ Long-Range Plan Project • Washington State Department of Transportation I-405 Bellevue Widening and Express Toll Lane Project The research team’s firm was the public involvement firm for these three projects. The only potential disadvantage of using projects in which the team’s firm was conducting the public involvement is the perception that the testing would not be objective. However, members of the public participating in the testing did not know that the team’s firm was conducting the public involvement; all public involvement staff were presented as WSDOT consultants. As for concerns about the public involvement staff completing the agency version of the survey and inflating their ratings, this would likely be an issue with any transportation projects chosen. Public involvement agencies need to take the evaluation of their public involvement seriously and should be open to hearing “the good, the bad, and the ugly.” For this testing, the public involvement staff had a vested interest in truly learning if the public involvement they did was effective and where it could be improved. Meanwhile, the research team’s focus was on testing the measurement tool and not on achieving high scores. For the participating agency, evaluating the effectiveness of the public involvement processes would show the funder the agency’s sincere interest in quality assessment/quality improvement. Puget Sound Gateway Program The Puget Sound Gateway Program involves the extension of SR 509 and SR 167. This project offers substantial economic benefits to the region. However, because the SR 167 and SR 509 projects create new transportation facilities, they will impact surrounding communities with displacements, effects to community cohesion, and environmental impacts. Both projects will also include tolling, which is controversial in the region.

Developing the Survey and Securing Transportation Projects for Testing 13 Washington State Ferries’ Long-Range Plan Washington State Ferries is developing a long-range plan to better understand and prepare for the ferry system’s changing needs through 2040. The process includes robust community engagement centered on two milestones: identification of issues and priorities in spring 2018 and review of a draft long-range plan in fall 2018. The final long-range plan is due to the Legislature January 1, 2019, and will guide future service and investments in vessels, terminals, and technology. I-405 Renton to Bellevue Widening and Express Toll Lanes Project The Washington State Department of Transportation is building a dual express toll lane system on Interstate 405 between downtown Bellevue and Renton. These new lanes would connect with existing toll lanes north and south of the project area to create a continuous 40-mi-toll lane system along this major north-south transportation corridor in the Seattle metropolitan area. The scope of this project also includes other highway and interchange improvements, new infrastructure to support a bus rapid transit system, and portions of a new regional bicycle and pedestrian trail. The project is currently in the preliminary engineering and environmental review phase, with construction scheduled to start in 2019, and the improvements scheduled to open to traffic in 2024.