Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

C-1 A P P E N D I X C Review of Thirty-Six Articles That Include Frameworks or Tools Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results Atlee, T., Buckley, S., Godec, J., Transportation Framework Methods Principles of why effectiveness should be evaluated. Good list of indicators. Careful planning, diversity, collaboration, openness/learning, transparency/trust, impact, sustained engagement. A small working group, listed on page one, formed to consider how to move forward quickly in as collaborative and transparent a way as possible, and the group decided to encourage broad involvement among networks in the formation of a set of principles for public engagement that all can get behind. A simple forum (bulletin board) was created on the website of the National Coalition for Dialogue & Deliberation (NCDD) to enable people to post and comment on existing sets of public engagement principles, values, and guidelines issued by various organizations and governing institutions, and to start developing a unified set of principles based on the work that had already been done. These seven principles reflect the common beliefs and understandings of those working in the fields of public engagement, conflict resolution, and collaboration: careful planning and preparation, inclusion and demographic diversity, collaboration and shared purpose, openness and learning, transparency and trust, impact and action, and sustained engagement and participatory culture. Bailey, K., Grossdart, T., & Ripy, J. (2015). Transportation Framework Methods Good framework. Importance of measuring all aspects/ audiences. Process, inclusion, clarity/utility of decision supports, efficiency. In the light of social and political currents calling for increased accountability for public processes and discussions between federal officials about the performance of the public involvement process, this paper examines public involvement in transportation and develops frameworks for identifying suitable approaches and evaluating outcomes. A significant Arnstein gap is identified in the quality of public involvement in transportation. Beierle, T. C. (2002). Environment Framework Methods Tool Good example of trying to assess effectiveness with a specific online dialogue. Self-assessment tool. Satisfaction, inclusion, communication, mutual impact. This report evaluates the National Dialogue on Public Involvement in EPA Decisions (the Dialogue) as a case study of electronic public participation. It examines the dynamics of the participation process and how participants felt about it. It describes the quality of communication when public participation moves from the meeting room to the computer screen. Finally, it looks at what participants and EPA got out of the process. By most of the criteria one could use to evaluate the Dialogue, it was a great success. Most people reported being satisfied by the process and thought similar on-line Dialogues should be conducted in the future. Communication in the Dialogue was rich and respectful. Both participants and EPA staff reported learning a great deal, and EPA now has much broader input to useas it goes about finalizing and implementing its Public Involvement Policy. R., Heierbacher, S., Nurse, L., Pyser, S., & McCallum, S. R. (2009). Harris, (continued on next page)

C-2 Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results Boyd, D. S., Bell, J., Doyle, D., Hammond, P., Weeks, J., Winnick R., & Schwartz, M. (1999). Transportation Framework Methods Tool High-level self- assessment tool for DOTs focused on project-based PI. 10 indicators: information exchange, shared decision making, project efficiency, project/decision acceptability, media participation, media coverage, decision implementation, mutual learning, mutual respect, cost avoidance. This document is an adaptable and practical guide which produces output in a “scorecard” format. It is intended to provide the practitioner with a means of conducting a self-assessment of the effectiveness of a specific public involvement campaign for a specific planning or project development activity (e.g., the development of a long-range plan or a specific capital improvement). N/A Brevard Metropolitan Planning Organization. (2013). Transportation Framework Methods Tool Evaluation matrix/ framework. Has criteria and goals with high-level measures. No clear link to indicators. Evaluation matrix/framework to evaluate public involvement tools. N/A Charbonneau, D., Avey, H., Gilhluly, K., Staton, B., & Harris, L. (2016). Health Framework Methods Tool Good self- assessment tool with multi- method approach. Inform, consult, involve. This evaluation offers new data related to community participation in the United States-based practice of Health Impact Assessment (HIA). The findings are intended to inform the work of HIA practitioners but are relevant to all researchers and organizations intending to authentically engage community members in addressing policy, program, or planning solutions, as well as funders of HIA or similar types of community- based participatory research. Incorporating potentially impacted community members in HIA activities at higher levels on the Spectrum of Public Participation shows promise to increase transparency, accountability, and credibility of HIA findings. Côté, M., & Bouthillier, L. (2002). Environment Framework Methods Tool Self-assessment tool. Although it is for forest management, it provides the actual assessment questions. Informing the public, incorporating public values into decision making, improving the quality of decision making, increasing trust, reducing conflict. The aim of the research project was to develop a method for assessing the impact of public involvement processes in Quebec, and to test this method in the context of forest certification initiatives. The research project determined that the public involvement processes tested in the Haute-Mauricie region, Quebec (Canada): (1) fostered better information sharing among parties interested in forest management; (2) brought about changes in forestry planning; (3) decreased mistrust between local stakeholders; and (4) reduced potential negative conflicts in this region. However, the experiment also showed that further institutional support should complement a public involvement initiative in order to increase its impact on forest management planning and on the relationships among parties interested in forest management. Edwards, P. B., Hindmarsh, R., Mercer, H., Bond, M., & Rowland, A. (2008). Framework Tool Evaluation of climate change PI using Rowe and Frewer technique. Provides good framework and tools, although Rowe & Fewer 2000. Report on the formulation of a three- stage approach to deliberative evaluation, which was applied in an independent evaluation of a Courageous Conversation on Climate Change and Transforming Energy (March 2007), designed by the Ethos Foundation, Queensland, Australia. Overall, we found our schema successful in identifying both the positives and negatives of the design, of what could be applauded and what needed revision. More broadly, we advance that it offers a useful approach for practitioners to develop further, especially in getting the balance right on evaluating process and Environment mostly a self- assessment. outputs, to which we have added inputs in reflection of deliberative design advances.

Review of Thirty-Six Articles That Include Frameworks or Tools C-3 Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results Florida Department of Transportation. (2015). Transportation Framework Methods Tool Several tool examples. Provides framework of methodology. Provides examples of indicators to use. Looks at many types. This Public Involvement Handbook provides techniques and methods to encourage meaningful public participation throughout the transportation decision-making process. N/A Genesee County Metropolitan Planning Organization. (2010). Transportation Framework Tool Some participant self-assessments provided. Brief participant evaluation surveys. No clear link to indicators. The following framework presents a basic format to facilitate “grass roots participation” in the planning process. N/A Hendricks, A. (2009). Government Framework Methods Tool Assessments and survey tools for stakeholders, process implementers, and participants. Integrity, inclusion, deliberation, influence, capacity, sustainable decisions. Late last year I conceived the idea of developing a framework for evaluating any and all community engagement processes based on the Brisbane Declaration. This document is a revised version of the draft framework, based on the feedback from people who have trialed it, as well as discussions with a number of people including Dr. Lyn Carson and Dr. Jeanette Hartz-Karp. N/A Huck, D. (2012). Environment Tool Sample outside of transportation but limited in scope. Only interviewed participants involved in the public involvement process. No general public participants were included. Early involvement, integrity, influence, adequate notice, dialogue, multiple engagement methods, adequate information, educating participants. This study set out to determine if collaborative approaches to watershed management planning have incorporated effective public engagement. Based on the framework for effective public engagement and a thorough analysis of public participation in the development of watershed management plans in Manitoba, it was apparent to me that the public is being genuinely engaged by Water Planning Authorities as a true partner in identifying and addressing watershed concerns. Manitoba Water Stewardship and conservation districts as Water Planning Authorities wholly recognize and support public engagement as fundamental to the overall success of integrated water resources management. In fact, the data show that watershed management planning as implemented under the Water Protection Act has incorporated many of the components of effective public engagement. Institute for Local Government. (2011). Methods Tool Methods and actual survey tools to measure and compare Loosely related to indicators of preparation, participants, These Rapid Review Worksheets can help guide the planning and design of the engagement process as well as ensure that the design and purposes of N/A participation experience to experience of local officials/ stakeholders. Worksheets to analyze differences and make improvements. process, and results but not strongly described. the engagement process match up with what the agency plans to evaluate. (continued on next page)

C-4 Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results Jackson, K., Johnson, P., & Jolley, M. (2011). Environment Framework Methods Tool High level self- assessment with case study analysis. Interview guide to actually measure the indicators/ concepts. Cooperation, transparency, openness, inclusiveness, involvement. This research aimed to find strategic methods in community engagement related to regional sustainable development, specifically within the context of regions in Europe and North America that are applying for the UNESCO Biosphere Reserve designation. We initially found 60 methods that contributed to higher scoring on Process Characteristics. Filtering that list of 60 to ones with the most significant relationship yielded a list of the nine strongest and most synergistic methods in community engagement that included: Representative Organization, Communications Strategy, Facilitated/Hosted Dialogue, Invitation to Co-create, Neutral Spaces, Bridge Building and Networking, Co-learning Reciprocity Approach, Trust Building, and Working with the Positive. Klaers, K. D. (2013). Government Framework Methods Tool Tested Rowe and Frewer 2000 framework for online applications. Rowe and Frewer 2000, modified for online applications. Regulation Room represents a systematically designed and evaluated Web-based public engagement exercise and serves as the case this paper uses to address whether and if so, how Web 2.0 has been an effective means of public engagement. The case is analyzed with an adaptation of Rowe and Frewer’s (2000) classic public engagement evaluation framework influenced by Macintosh and Whyte’s (2008) novel e-participation evaluation framework. Regulation Room demonstrates e- participation exercises have the potential to be effective public participation mechanisms though there is still room for future improvements. Martin Metropolitan Planning Organization. (2014). Transportation Framework Methods Tool Self-assessment with objectives, strategies, tools, and measures. Hold regular meetings, equitable access, engage early and often, use variety of PI methods, and get feedback. This public involvement plan is intended to define how the Martin MPO will engage the public and gather public input. N/A McGrow, G., & Greenaway, L. (2013). Health Framework Review of several different evaluation frameworks. Looks at many types. This Guide has been developed by the Scottish Health Council as a tool for supporting the evaluation of public involvement and participation in health services. N/A Metropolitan Planning Organization for the Miami Urbanized Area. (2001). Framework Methods Tool Good framework with full effectiveness but no actual measures for each. Very high- level measures for each Looks at many types. The purpose of this study is twofold: (1) to review and update the current PIP goals where appropriate and (2) to develop and implement a systematic process to evaluate the effectiveness of the PIP. N/A Transportation indicator.

Review of Thirty-Six Articles That Include Frameworks or Tools C-5 Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results MIG. (2013). Transportation Framework Methods Tool Highly structured framework with goals/indicators and measures. Diversity, reach, accessibility, impact, education, participant satisfaction. This evaluation report is intended as a companion piece to the reports describing and documenting the three- year public outreach and involvement process conducted as part of the development of Plan Bay Area. Plan Bay Area is a long-range integrated transportation and land-use/housing strategy through 2040 for the San Francisco Bay Area that gets updated every four years. Plan Bay Area drew an unprecedented number of participants. The number of workshop participants, activity on the Plan’s website, and written correspondence received on the Draft Plan all dramatically exceeded levels from previous plans. The amount of media coverage also exceeded previous updates and the sheer volume of engagement opportunities — including 270 public meetings — allowed for a more inclusive and robust public engagement process. Minnesota Department of Transportation. (1999). Framework Detailed self- assessment tool on evaluating all sorts of PI methods. Collaborate and empower. Multi- Method approach. The following public involvement guidelines were developed to assist Mn/DOT personnel in implementing public involvement plans and activities. They reflect the mandates of ISTEA, reinforced by TEA-21, as well as public agency best practices. N/A Neshkova, M. & Guo, H. (2012). Transportation Framework Methods Self-Assessment tool with staff/managers/ leaders of DOTs to investigate success with different types of PI activities. International Association for Public Participation (IAP2) classifies into five stages of increasing public impact: inform, consult, involve, collaborate, and empower. We draw on data from U.S. state transportation agencies to test the relevance of two theoretical perspectives about the effect of public participation on organizational performance. Our results show that there is not necessarily a trade-off between the values of democracy and bureaucracy, with clear implications for the theory and practice of democratic governance. North Jersey Transportation Planning Authority. (2007). Transportation Framework Methods Tool Basic framework of how to evaluate. Mostly self-assessment. Has measurement goals, but not linked to specific indicators. The NJTPA seeks to evaluate and upgrade, on a periodic basis, the public participation efforts of the NJTPA to ensure that public involvement has a demonstrable impact on the transportation planning and decision- making process in the region. N/A Null, M. & Huckstep, R. (2011, July). Environment Framework Methods Tool Self-Assessment tool with guidelines of how to rate and measures provided for each Indicator. Accessibility, understanding, diversity of views, integration of concerns, information exchange, project efficiency, decision acceptability, mutual learning, cost avoidance. Conference presentation at EPA Community Involvement Conference about measuring community engagement effectiveness. N/A O'Leary, A. A., Arnold, Jr., E. D., Kyte, C. A., & Perfater, M. A. (1999). Transportation Methods Tool Includes complete survey given to participants in public involvement. No clear link to indicators. The Virginia Transportation Research Council (VTRC) developed a plan for a two-phase study of VDOT's public involvement practices. Phase I, summarized in this report, focused primarily on hearing formats and related issues. Open format public meetings appear to have advantages over traditional public meetings. (continued on next page)

C-6 Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results O'Leary, A. A., Kyte, C. A., Arnold, Jr., E D., & Perfater, M. A. (2003). Transportation Methods Tool Sample citizen hearing survey. Self-assessment survey for VDOT engineers, public affairs managers in other states, and consultants (transportation leaders). No clear link to indicators. The objectives of this study included a broad assessment of the Virginia Department of Transportation’s (VDOT’s) public involvement practices and the development of a public involvement “toolkit” for use by VDOT staff. Responses to the citizen and VDOT staff surveys indicated that the public is often unclear about the steps in VDOT’s planning, project development, and public involvement processes. Citizens and VDOT staff also agreed that frequent updates on project status are desirable and that the public should get more feedback about how their input is really used in decision making. VDOT staff sees a need for more strategic communications planning and evaluation for major projects and more coordinated project communications within the agency. VDOT staff also believe that broader [staff] understanding of the responsibilities of different VDOT divisions in the project development process would improve communications within VDOT. Olszak, L. M., Goldback, R. L., & Long, J. R. (2007). Transportation Framework Tool Good tool and framework, although limited in actual measures. Self- assessment based on surveys with DAT members, facilitator surveys, and meeting minutes. Has survey samples. Inform, consult, involve, collaborate, empower. The aim of this research was to understand the impact and outcome of a context sensitive design process utilizing five community-based Design Advisory Teams (DATs) working directly with design. 1. Empowered Decision Making Can Save Costs 2. Developing Public Trust and Confidence Positively Influences Outcomes 3. Direct and Frequent Access to Designers and the Owner is Essential 4. Ongoing Communication and Exchange Opportunities Facilitate Decision Making Quick, K. (2014). Transportation Framework Methods Tool Good use of a framework. Multi-method (interviews with stakeholders, media audit, interviews with participants by a third party). Provides actual survey questions. Impacts of participation, quality of decision outcomes, long- term results, participant satisfaction, impacts of participation. This research is a comparative analysis of public engagement methods for involving stakeholders in decision making about complex issues. This research identifies convergences and divergences in information and perspectives among stakeholders. Some public engagement methods allowed study participants to change their perspectives on what road management options were achievable and acceptable. This occurred through active recruitment of diverse stakeholders, focus groups with individuals of similar backgrounds, and a facilitated policy roundtable among all the different stakeholders. An additional finding relates to evaluation measures for public participation, which scholars and practitioners acknowledge are poorly developed. Rowe, G. & Frewer, L. J. (2000). Government Framework Methods Source of many studies on effectiveness and evaluation. Provides an effectiveness evaluation framework. Acceptance Criteria (representativeness, independence, early involvement, influence, transparency) and Process Criteria (resource accessibility, task definition, structured decision Given that the quality of the output of any participation exercise is difficult to determine, the authors suggest the need to consider which aspects of the process are desirable and then to measure the presence or quality of these process aspects. To this end, a number of theoretical evaluation criteria that are essential for effective public participation are specified. From our analysis, it is difficult at this juncture to categorically declare any one method to be the best. Indeed, Smith, Nell, and Prystupa (1997) conclude—and in this we agree—that the most appropriate techniques for public participation are likely to be hybrids of more traditional methods. making, cost effectiveness).

Review of Thirty-Six Articles That Include Frameworks or Tools C-7 Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results Rowe, G., Horlick-Jones, T., Walls, J., Poortinga, W., & Pidgeon, N. F. (2008). Science Framework Methods Tool Self-assessment using input from participants rather than stakeholders about success of a PI event. Good survey questions to review. Rowe & Frewer 2000 In this paper one normative framework for evaluating engagement processes is considered. This framework was operationalized and used as part of the evaluation of a recent major UK public engagement initiative: the 2003 GM Nation? debate Results suggest the chosen evaluation criteria have some validity, though they do not exhaustively cover all appropriate criteria by which engagement exercises could be evaluated. SEDA-COG Metropolitan Planning Organization (2014). Transportation Framework Methods No tool or instruments, but has a good solid framework with goals, objectives and indicators. Reach, diversity, impact, satisfaction, education, performance. The Public Participation Plan (PPP) was developed based on consultation with SEDA-COG and includes regional overview information and a framework of goals, objectives, and strategies for accomplishing an effective and compliant public participation process in its transportation planning efforts. The PPP includes procedures for implementing public involvement as well as indicators for evaluating the performance of the plan and suggesting future improvements. N/A Skogan, W. G., Hartnett, S. M., DuBois, J., Comey, J. T., Twedt-Ball, K., & Gudell, J. E. (2000). Law Enforcement Framework Methods Tool Some survey questions might be good to duplicate. Limited tool for transportation. No clear link to indicators. This report summarizes recent research on citizen involvement in Chicago’s community policing initiative, known as the Chicago Alternative Policing Strategy (CAPS). Community factors and personal contacts play the largest role in stimulating actual participation; but meeting attendance is strongly linked to involvement with groups that spread the word about CAPS, as are word-of-mouth discussions about the program and the fliers and newsletters that circulate easily in well-organized neighborhoods. United States Environmental Protection Agency. (2004). Environment Framework Methods Tool Self-assessment of stakeholders/ officials. Provides a framework with actual surveys used to measure. Planning for involvement, informing, conducting, community response, benefits of involving. The purpose of this evaluation is to gain a better understanding of the dynamics of Community Involvement in Risk Assessment and its impact on the risk assessment process. The risk assessment is part of a larger process that begins with the Remedial Investigation and continues through the Record of Decision. The best results stem from community involvement that commences early in the risk assessment process, such as during the RI phase. CICs, RPMs, and RAs emphasized the importance of coordinating with local health departments and ATSDR to reach out, to collect information from, and to report results to the public. The involvement of technical advisors can also raise the community's level of respect for EPA and their satisfaction with the cleanup, particularly if the community group that hired the technical advisor is representative of the larger community. While community input did not affect the outcome of the risk assessment in all cases, interviewees noted in all cases, the comments often encouraged EPA to look at new sources of exposure and to reconsider assumptions about exposure frequency, duration, and intensity. The evaluation team asked nine RAs to discuss the benefits of involving the community in risk assessment, and they unanimously stated that community involvement increased public confidence in EPA. (continued on next page)

C-8 Measuring the Effectiveness of Public Involvement in Transportation Planning and Project Development Citation/ Sector Tool Type Reason to Consider Framework Indicators/Goals Purpose Results Wagner, J. (2012). Transportation Framework Performance scorecard rubric to operationalize principles. Scorecard provides an intuitive and concise method to measure and display performance information. 11 indicators around 3 principles: accessible, engaging, and outcome-oriented. However, the measures are high- level assessment and not concrete. No tools associated. Measuring the performance of public engagement in transportation planning has proved difficult due to a variety of issues, including unclear goals, limited funding, and the situational nature of engagement activities. To address some of these issues, the author analyzes the literature of best practices for public engagement in transportation planning. The author derives three principles that encapsulate the contained concepts. The three principles include accessible events, engaging interactions, and an outcome- oriented process. The author developed a performance scorecard rubric to operationalize the three principles. The scorecard provides an intuitive and concise method to measure and display performance information. Wang, X. (2001). Government Methods Tool Good survey measures for DOT self- assessment of PI practices with public. 1. Participation leads to satisfaction. 2. Participation builds consensus on goals, services, performance, and fiscal commitment. Three research questions: How much public participation occurs in U.S. municipal governments? What are the possible causes of participation? What are the possible impacts of participation on governmental agencies? Cities use a variety of participation mechanisms. The size of government influences participation. The use of participation mechanisms is perceived as effective in meeting public needs, building consensus, and improving public trust. The use of participation mechanisms is significantly associated with all participation impacts. Warburton, D., Wilson, R., & Rainbow, E. (2011). Government Framework Methods Goals, indicators, and recommended methods, though no actual measures. Cost savings, create awareness, improved services, accessibility. This document provides some specific ideas for using evaluation to set objectives for engagement, monitor progress, measure achievements, identify lessons, and help improve practice. N/A Wemple, E. & Wu, D. (2014). Transportation Tool Potentially useful survey instruments, though more focused on the success of the planning process rather than the PI itself. No clear link to indicators. The Planning Assistance for Rural Areas (PARA) program, sponsored by the Arizona Department of Transportation’s (ADOT) Multimodal Planning Division (MPD), dedicates funding and staff to conduct multimodal transportation planning studies for local jurisdictions. This research study’s goal was to evaluate and refine ADOT’s PARA program to better meet the goals of both ADOT and its local jurisdictional partners. The study found that overall, PARA stakeholders were satisfied with the PARA program. Stakeholders stated that PARA programs provided resources for identifying needs and addressing issues that were not otherwise readily available to small communities with budget constraints.