Measuring Quality of Life in Communities Surrounding Airports (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

86 Process for Developing Quality of Life Assessment Methodology A P P E N D I X D The following appendix provides a more detailed record of how the Quality of Life Assessment Methodology was developed. Chapter 2 of the guidebook provides an overview and explanation of the Quality of Life Assessment Methodology, including a description of the quantitative and qualitative indicators, the QOL scoring mechanism, and the process of importance weighting. The information in this appendix provides additional detail and context describing the development of the methodology by the research team. As noted in Chapter 2, the Quality of Life Assessment Methodology developed for this guidebook was based primarily on a similar tool developed by ERG for evaluating resilience of communities to the impacts of climate change created for the EPA (Blue et al. 2017). The EPA tool is known as the Multi- Sector Evaluation Tool for identifying Resilience Opportunities, or METRO. Most of the indicators developed for the Quality of Life Assessment Methodology are different than those developed for the resilience assessment tool. However, the overall methodology is similar, and some of the relevant indicators were retained. Indicators related to QOL, sustainability, and resilience are often related to each other. Robust methods of measuring QOL can help governments and other interested parties, such as airports, understand livability issues and both positive and negative outcomes of changes or planned changes in a community. Some attempts to measure QOL rely mainly on economic or demographic metrics (i.e., hard data), while others rely on subjective, perception-based measurements (Haslauer et al. 2014). One difficulty in creating a useful assessment of QOL lies in the wide variety of factors that affect overall QOL and the challenges related to broadly measuring, interpreting, and integrating metrics. Some factors affecting QOL in a community—such as economic health, air quality, and water quality— can be measured quantitatively through publicly available data sets. Other factors are not reflected in this type of available data and are best addressed by asking community members for input on issues that are important to them, as well as how important they are and how the issue is currently affecting their QOL. For this project, the research team selected a mixed-methods approach (i.e., one that incorporates both hard data, when it is available, and qualitative information collected from community members to address QOL more comprehensively) to assessing QOL in communities surrounding airports. This was the same approach used in the EPA resilience assessment tool previously discussed. A mixed-methods approach that integrates both quantitative and qualitative information increases understanding of the broad influence that airports have on local communities and can support airport

Process for Developing Quality of Life Assessment Methodology 87 quantitative data sets, where available, and qualitative data related to subjective aspects of QOL collected via survey from community stakeholders and airport personnel. This mixed-methods approach will introduce more possibilities for analysis and interpretation than a strictly quantitative analysis. Most importantly, little data is available on some of the most important aspects of QOL, particularly those in the social relationship category, and an assessment based exclusively on quantitative data would not be robust. Quantitative and Qualitative Indicators The Quality of Life Assessment Methodology is based in part on well-vetted decision-support methods known as multicriteria analysis or multicriteria assessment (MCA) and mixed-methods evaluations. MCA methods evaluate decision alternatives based on multiple criteria or objectives (Hajkowicz and Higgins 2008). MCA studies typically involve participant engagement to collect input on preferences that is often converted to quantitative data. Mixed-methods evaluations incorporate quantitative information (i.e., hard data) in addition to qualitative information collected via MCA methods or other means. The list of 100 indicators selected for evaluating QOL under this methodology is a mix of quantitative and qualitative indicators. For the quantitative indicators, existing data sets are used to score QOL. Input from tool users is still needed to score the importance of that indicator to the overall assessment. For the qualitative indicators, tool users must answer questions to determine a QOL score for the indicator (essentially, ranking QOL on a scale from 1 to 4 with respect to the issue represented by the indicator), in addition to providing an importance score for the issue represented by the indicator (reflecting the indicator’s relative importance in the overall QOL assessment). Selection of Indicators The research team’s goal during the indicator selection process was to identify a set of indicators that would—when applied to an assessment of QOL in any community surrounding an airport in the United States—provide a reasonably accurate and comprehensive measure of all aspects affecting the QOL of all members of the community. The team balanced the degree to which a large number of indicators would provide more specific information on the largest number of community members with the degree to which a small number of indicators would provide an efficient means of conducting such an assessment. The indicators had to be general enough that a reasonably small number of them could capture major factors affecting QOL but specific enough that they would be meaningful and there would be consistency in how they are interpreted and scored by various communities. Similar high-level indicator categories—including economic, health, environmental, and social categories—were identified in many of the resources reviewed during the literature review process. To leaders in making decisions that are beneficial to local QOL. Perhaps even more importantly, involvement in assessing the QOL in the local community demonstrates to communities that the airport is aware of and considerate of its effects on its neighbors. This can help foster improved relationships between airports and communities. With guidance from case studies and literature on mixed-methods, multicriteria approaches, the research team revised and adapted METRO’s indicator-based approach using a combination of detailed D.1 D.2

88 Measuring Quality of Life in Communities Surrounding Airports transportation categories to categorize additional indicators related to ensure a holistic assessment methodology in relation to measuring QOL in communities surrounding airports. The transportation category was of great interest to the research team, given that this tool is intended to be used in part to inform airport leaders regarding issues that are important to the surrounding communities, which will ultimately inform decisions the airport may make that will affect community QOL. The tool is robust enough that—although it is intended to be used by airports—it can also be implemented by other local organizations interested in obtaining information concerning community QOL. The research team identified more than 400 QOL indicators from the literature. These were narrowed down to 100 indicators of greatest relevance to this study. The research team removed redundant indicators and those deemed to be insignificant to the general assessment methodology (compared with remaining indicators). Each of the 100 indicators in the Quality of Life Assessment Methodology was then organized into one of the following six categories. Indicator titles were revised as necessary to reflect the subcategory of the QOL assessment it was intended to cover. The following list indicates the number of indicators (including both qualitative and quantitative indicators) that reside within each of the tool’s high-level QOL categories: Environmental (14 indicators) Health (22 indicators) Economic (23 indicators) Transportation (12 indicators) Social Relationships (11 indicators) Local Governance–Community Services (17 indicators) The number of indicators within each category is not a reflection of the importance of the category in assessing QOL. Rather, for some categories it was possible to cover the main topics of interest using a smaller number of indicators, while for other categories the diversity of QOL topics covered within the category required a larger number of indicators to capture QOL. In addition to the indicator categories listed above, a general indicator on overall QOL was included in the assessment methodology that does not fall into a category. Data related to many QOL indicators are not available in existing data sets describing communities surrounding airports (or any community, for that matter). Therefore, most of the indicators selected are qualitative. The qualitative indicators are framed as questions administered to the assessment participants through a survey, and there is some subjectivity expected in the answers to be provided reflect the research team’s findings during the literature review process, the final Quality of Life Assessment Methodology includes six categories of indicators: Environmental, Health, Economic, Transportation, Social Relationships, and Local Governance–Community Services. The research team used the term “social relationships” to reflect the social category (as it best reflected the set of individual indicators that were selected) and added local governance–community services and

Process for Developing Quality of Life Assessment Methodology 89 Supplemental Indicators The Quality of Life Assessment Methodology is designed to provide flexibility to meet the needs of unique airports. Collecting data for all 100 indicators is suggested under the Quality of Life Assessment Methodology (described in Chapter 3) in order to conduct a comprehensive study of community QOL, but the methodology allows for some flexibility when choosing indicators by allowing the list to be altered by individual airports to best meet their needs when completing the assessment. As discussed in this guidebook, the primary set of 100 indicators was developed based on input from subject matter experts, airport personnel, the ACRP project panel (an advisory panel of technical industry experts), and community stakeholders at three partner airports. However, local conditions will vary from airport to airport, and if the airport or stakeholders identify significant information gaps in the provided list of indicators, the airport can use some customized, supplemental indicators in order to make the QOL assessment more locally relevant. The supplemental indicator list in Appendix B contains example indicators that can be added to the survey tool or used to replace indicators already in the survey tool; for example, if the data does not exist for all the quantitative indicators recommended or if an existing indicator is not applicable to a certain airport. Airports are free to develop additional qualitative or quantitative indicators to address specific issues of importance to them or their stakeholders beyond those examples that exist in the current list of supplemental indicators. The research team advises that supplemental indicators be added to a QOL assessment judiciously. The addition of too many custom supplemental indicators may jeopardize the ability of a QOL assessment to capture information under the six categories of QOL, as identified by the research team and validated through the research process. Thus, the research team recommends that supplemental indicators be used sparingly unless an airport’s situation diverges significantly from those of most U.S. airports for a particular reason. It is also acceptable to use the original list of 100 indicators without making any modifications. Either approach will lead to a valuable assessment of community QOL to inform future decision making and improve understanding between the airport and the surrounding community. Indicator Quality of Life Scoring Mechanism As described in Chapter 2, each qualitative and quantitative indicator was assigned scores of 1 to 4 corresponding to responses representing low to high QOL. Additional details follow. Quantitative Indicators For the quantitative indicators presented in Table 1, QOL scores ranging from 1 to 4 were assigned by dividing data into four categories of values separated by three threshold indicator values. Indicator thresholds were identified to represent the points at which an indicator value likely changes from one during the assessment. Regardless, the research team believes that the wealth of information that community members have about what influences their own QOL is better captured by using a full range of qualitative indicators than by limiting the assessment to information that could be gleaned from the narrower set of relevant quantitative indicators. Table 1 in Chapter 2 includes information on which indicators are quantitative and which are qualitative. Information on how data can be gathered for these indicators is described in Chapter 4. D.2.1 D.3

90 Measuring Quality of Life in Communities Surrounding Airports A QOL score of 2 is assigned to communities where 16 percent to less than 20 percent of the population live below the poverty line. A QOL score of 3 is assigned to communities where 12 percent to less than 16 percent of the population live below the poverty line. A QOL score of 4 is assigned to communities where less than 12 percent of the population live below the poverty line. Due to the complex processes represented by each quantitative indicator and the variable ways in which each indicator can influence overall QOL, indicator thresholds can be somewhat challenging to assign; especially when the goal is to have them represent all U.S. cities. As such, and to the extent possible, the thresholds used in this assessment were based primarily on validated indicators developed to support similar assessment tools or information gleaned from the peer-reviewed literature. For example, the thresholds for Indicator G15 (emergency medical service response time) and Indicator E17 (economic growth) are based on those previously developed by Julie Blue, Nupur Hiremath, Carolyn Gillette, and Susan Julius and published in EPA’s Evaluating Urban Resilience to Climate Change: A Multi-Sector Approach (Blue et al. 2017). Even though the primary purpose of that tool is to evaluate resilience of communities to the impacts of climate change, the research team reviewed the technical approach and concluded that the thresholds used to evaluate quantitative data for that purpose are equally relevant and appropriate when measuring QOL. When indicator thresholds were not readily available in the peer-reviewed literature or other sources, publicly available data for U.S. cities were examined to establish a range of values for the indicator across the U.S. For these indicators, the median value from within the data distribution was often selected as a starting point for identifying the value for a middle threshold (i.e., separating a QOL score of 2 from a QOL score of 3), with high and low values based on natural upper or lower limits in the data. Recognizing that the distribution alone is not enough for defining absolute thresholds for important QOL issues, the research team consulted the published literature (e.g., academic literature and government reports) to calibrate the initial thresholds, based exclusively on the data. As an example, Indicator H21 (obesity prevalence) is based largely on data maintained by the Centers for Disease Control and Prevention. However, since obesity is considered a national epidemic, indicator thresholds based solely on the distribution of prevalence estimates may not appropriately reflect QOL. For indicators such as this, the thresholds identified from the data were adjusted upwards or downwards to reflect information gathered from other sources. In the example of obesity prevalence, the research team adjusted the thresholds downwards based on findings published in various public health resources. Qualitative Indicators For qualitative indicators, QOL scores ranging from 1 to 4 were assigned to each response to the indicator question. As described earlier, qualitative indicators are represented by a question with a representing poor QOL (an indicator score equal to 1) to a fair QOL (an indicator score equal to 2), for example. To illustrate this concept, imagine a scenario in which data were collected from the U.S. Census Bureau for Indicator E19 (the percentage of people living below the poverty line). For this indicator, a QOL score is assigned using the indicator’s unique thresholds as follows: A QOL score of 1 is assigned to communities where 20 percent or more of the population live below the poverty line.

Process for Developing Quality of Life Assessment Methodology 91 A QOL score of 2 is assigned for a response of “somewhat dissatisfied.” A QOL score of 3 is assigned for a response of “somewhat satisfied.” A QOL score of 4 is assigned for a response of “satisfied.” As with the scoring mechanism described above for the quantitative indicators, there can be challenges when assigning QOL scores to the qualitative indicator responses. As such, several members of the research team independently assigned QOL scores to each response for each qualitative indicator. These scores were then compared, and any discrepancies were discussed and reconciled with the larger research team. Indicator Weighting Mechanism An important part of the Quality of Life Assessment Methodology includes prioritizing which indicators contribute the most to QOL or detract the most from QOL. As noted previously, for each of the 100 indicators, the QOL score shows how an individual community member is faring (or, collectively, how the community is faring) with relation to the component represented by that QOL indicator (i.e., high QOL is represented by a QOL score equal to 4, and a low QOL is represented by a QOL score equal to 1 for the specific indicator). However, the importance of one indicator is unlikely to be equivalent to the importance of all the other indicators in assessment of QOL, so it is not advisable to weight each indicator equally in the assessment. Some indicators may be very important to overall QOL, and some indicators may be minor in terms of how they affect overall QOL for an individual (regardless of whether the QOL score is high or low for that indicator). For this reason, the methodology uses the calculation of importance scores to reflect the degree to which an indicator contributes to overall QOL so that each indicator is not weighted equally. Importance scores vary from 1 to 4 (an importance score equal to 1 represents low importance for an indicator with respect to a respondent’s overall QOL, and a 4 represents high importance). Importance scores for each indicator are captured using the Quality of Life Assessment Survey Tool (Appendix A). For example, the research team anticipates that many participants may rank Indicator H1 (personal satisfaction with health) with an importance score of 4 (high importance), since it is universally applicable and poor health has the ability to affect most aspects of an individual’s life. However, an indicator such as T4 (transportation system redundancy) may receive a lower importance score. Some participants may believe low redundancy in the public transportation system is unlikely to affect their QOL significantly if they have access to a reliable personal vehicle. Other individuals may rank Indicator T4 with a higher importance score if they are dependent on public transportation for commuting to work or school or live in areas where the transportation system is unreliable due to scheduling, weather, or recurring maintenance problems. defined set of four responses. For example, Indicator EN1 (satisfaction with local air and water quality) asks an individual to respond to “How satisfied are you with air and water quality in your community?” by selecting one of the following responses: “dissatisfied,” “somewhat dissatisfied,” “somewhat satisfied,” or “satisfied.” In this case, QOL scores are assigned as follows: A QOL score of 1 is assigned for a response of “dissatisfied.” D.4

92 Measuring Quality of Life in Communities Surrounding Airports quadrant plots are used in this way, QOL scores and importance scores must be averaged across participants so that a single score for each can be plotted for each of the indicators. Chapter 5 presents example quadrant plots using simple averages to obtain a single QOL score for each of the qualitative indicators and a single importance score for each of the qualitative and quantitative indicators. These values were then rounded to the nearest whole number to determine which quadrant each indicator falls within. While this approach offers an easy and quick way to visualize the data, it has the potential to dilute or mask QOL issues (i.e., indicators with a low QOL score or indicators with a high importance score). This could result in implicitly optimistic assessments of QOL, potentially overlooking important QOL issues simply because low QOL scores or high importance scores were averaged out. This can also occur if rounding of average QOL scores or importance scores moves an indicator from one quadrant to another (e.g., from the “monitor for changes” quadrant to the “low priority” quadrant). To address this concern, various advanced statistical approaches are available that would allow the upper and lower bounds of the range of possible QOL scores or importance scores to be more heavily weighted in the average estimate (Runfola et al. 2017). Simply put, this means that instead of all data points contributing equally to the final average QOL score, certain data points (i.e., QOL scores of 1 or importance scores of 4) would contribute more than others. This conservative approach would help to ensure that important QOL issues are not overlooked due to traditional average and rounding conventions. However, these alternative approaches require additional resources and, ideally, guidance by an expert statistician. In lieu of these advanced statistical methods, airports can remain aware of the potential limitations associated with using a simple average to plot QOL scores and importance scores for a community. Additional insight can be gained by reviewing the distribution of individual responses. Refer to Appendix G for more information. Refinement of Methodology To ensure that the Quality of Life Assessment Methodology is usable by airports and results in meaningful data to determine a community’s overall QOL, the approach was reviewed with three partner airports. This collaboration consisted of convening a small group of airport personnel and external community stakeholders from each airport to discuss and refine the indicator list and the survey process. Three airports volunteered to participate in the methodology refinement process: Dallas–Fort Worth International Airport, Tampa International Airport, and Portland International Jetport (Portland, Maine). Appendix E contains summaries of the information obtained from the three airport workshops. The methodology refinement process involved outreach through teleconferences and webinars to airport Quadrant Plots As described and presented in Chapter 5, results for qualitative and quantitative indicators can be presented in a quadrant plot, with importance scores on one axis and QOL scores on the other. These plots allow users to quickly identify QOL issues that need to be addressed or monitored for changes over time. As also mentioned in that same section, quadrant charts can be used to visualize results for a single participant but are far more useful when results are plotted for a group of individuals. When D.5 D.6

Process for Developing Quality of Life Assessment Methodology 93 information for a holistic QOL assessment, including input on the weighting process for indicators and suggestions for refinements to the assessment tool (i.e., revised indicators and addition or removal of indicators). Participants had the opportunity to evaluate the appropriateness and scope of the proposed qualitative QOL indicators in both small and large group settings. Engagement with Partner Airports Internal airport stakeholders included a variety of representatives from across the airports, such as executive leadership, operations staff, environmental staff (including noise and sustainability-focused staff), communications and marketing staff, and government affairs representatives. Introductory calls , including an introduction to ACRP, provided background information; the purpose and objective of the research project; research approach; draft Quality of Life Assessment Methodology; project schedule; and anticipated final analyses. The introductory calls served to engage airport staff. The three airports requested follow-up teleconferences with additional airport staff to clarify the research process and airport responsibilities and address any concerns about the workshops prior to inviting external stakeholders. The partner airports were asked to identify external stakeholders to include in the workshops and to initiate outreach. To ensure consistency and clarity, the research team drafted outreach emails for the airport points of contact, as well as prepared a package of background information for participants. The outreach email introduced the research team to the external stakeholders, and further communications were sent directly from the research team. To allow for as much time as possible during the workshops for stakeholder discussions, the research team scheduled introductory web-enabled teleconferences for the external stakeholders approximately 2 weeks in advance of the workshops. The purpose of the calls was to introduce the project and research team members to stakeholders, discuss the research and the workshop objectives, and review workshop logistics. As noted previously, participants were also provided with a read-ahead document describing the project and objectives of the research, workshop logistics, an overview of the methodology, and instructions for participation. The read-ahead packet also included the draft survey instrument for collecting data on the qualitative indicators. However, the participants were instructed not to fill out the survey. Participants were asked to review the packet prior to the introductory calls and the workshop. Although much of this process is specific to the step of refining the research methodology, similar preliminary web-enabled teleconferences and introductory information can be used when conducting a full QOL assessment. staff and, later, the external community stakeholders, culminating in in-person workshops at each of the airports. The research team worked with the main contact for each partner airport to identify and convene the group of volunteer stakeholders for each workshop. One objective of the workshops was to review and discuss all the qualitative indicators in a collaborative group setting. During the three workshops, participants engaged in discussions concerning the ability of the methodology to capture critical D.6.1

94 Measuring Quality of Life in Communities Surrounding Airports Collecting Input on Quantitative Indicators Quantitative indicators were not the primary subject of discussion at the workshops because the methodology requires that airport personnel (or a contracted third party) gather data on the quantitative indicators from publicly available data sets during the QOL assessment process. To obtain airport stakeholder input and review during the research process, quantitative indicators and suggested data sources for each were shared with the three airports. The airport personnel were asked to share the draft quantitative indicators with colleagues to obtain feedback from various airport departments, although they were not tasked with collecting the data. The research team scheduled discussions with airport stakeholders to obtain feedback on the quantitative indicators after the in-person workshops. Airports were also asked to provide input related to the following items: Feedback concerning whether the airport already collects any of the data contained in the list of quantitative indicators and, if so, for what purposes; Feasibility of gathering the data that is not already collected for other airport purposes; Any foreseen quantitative data gaps; Feedback concerning whether the data sources that the research team identified are accessible or reasonable for data collection by airport staff; Clarity of indicator descriptions; Relative importance of each indicator to overall QOL in their airport and community perspective as a whole; and Feedback concerning whether the level of effort estimated to gather information regarding each indicator is accurate. The research team incorporated feedback concerning specific indicators and importance scores in the final list of indicators presented in Appendix B of this guidebook, as appropriate after further evaluation of feedback by the research team. Multiple airport participants suggested that it may be more efficient to obtain data sources related to the quantitative indicators by contacting city departments as opposed to the originally identified national data sets. They noted that many cities or local governments regularly collect and analyze data on a local level for their own purposes, specifically many of the economic quantitative indicators. Some participants felt that it may be more realistic and efficient to contact and work with local entities to ensure the assessment captures the most up-to-date local data in their communities. Airport participants also mentioned that some of these indicators have been considered during airport master planning efforts and would not require much additional analysis to include. Discussions also focused on how community perspectives and expectations have shifted over time, and, Workshop Overview The three in-person workshops were each a half day and included both large group and small group discussions facilitated by members of the research team. The stakeholders were given adequate time and opportunities to voice their opinions and provide feedback on the project in the workshop, as well as in a follow-up survey to the workshop. Further details concerning input obtained during each workshop are provided in Appendix E. D.6.3 D.6.2

Process for Developing Quality of Life Assessment Methodology 95 input on the applicability and appropriateness of the qualitative indicators. During the workshop, participants were separated into small groups to discuss specific categories of indicators and provide feedback to the research team. Small group discussions were facilitated by research team members but were designed to provide a forum where every stakeholder felt comfortable providing input. Feedback from the workshops included suggestions to add or remove indicators, revisions to the language of indicators, switching the order of questions on the survey, and more. The outcome from these discussions—along with the feedback from airports on the quantitative indicators—formed the basis for revisions to the methodology. Potential Concerns Regarding Collection of Sensitive Information Development and administration of a QOL assessment can be a cause for concern because the survey instrument facilitates the collection of potentially sensitive information. The qualitative indicator survey asks respondents for their personal opinions on a range of topics and includes optional demographic questions that may help the airport analyze and better understand evaluation results. The research team is aware of the sensitivities of the research and sought to maintain a transparent and collaborative relationship with both internal and external airport partners throughout the research process and the workshops. A few examples of concerns identified by the internal airport stakeholders include legal concerns about data privacy, political concerns associated with obtaining data that may further illustrate disparities among community populations, and airport concerns that some stakeholder groups may intentionally have their members provide biased responses in an attempt to influence airport operations. Partner airports also expressed concerns that holding a workshop could provide a forum for stakeholders with preexisting complaints about or disputes with the airport to bias the evaluation process. The research team received feedback from airport personnel that many airports interested in undertaking a QOL assessment would likely prefer to initially undertake a streamlined version of the assessment to demonstrate value to internal stakeholders before embarking on a full QOL assessment. The rationale is that once the resulting QOL data is recognized as reliable and important, the allocation of financial and human resources for a full QOL assessment could be more easily justified. Airports also recognized that partnerships with local governments, associations, or local universities may provide another avenue for funding a full QOL assessment as the outcomes would be of interest to these organizations. In addition to providing potential funding, another benefit of partnering with outside organizations is that the study might be viewed as more legitimate by communities if it is not undertaken solely by the airport. therefore, it is increasingly valuable for airports to track and collect data related to many of the QOL indicators that the airport previously may not have considered. Collecting Input on Qualitative Indicators Qualitative indicators were the main subject of discussion at the workshops because the methodology suggests using a participant survey to gather this information. The research team sought stakeholder D.6.4 D.6.5