Assessing Aircraft Noise Conditions Affecting Student Learning, Volume 1: Final Report (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

CHAPTER 4. SELECTION OF RESEARCH DESIGN PLAN 4.1. Introduction The objectives of this study are twofold, namely to: (1) to identify and evaluate conditions under which aircraft noise affects student learning, and (2) to identify and evaluate one or more alternative noise metrics that best define those conditions. To these, we have added a third requirement (3) to identify the most effective of the current criteria for school sound insulation projects To achieve these objectives, and in response to the identified gaps in knowledge noted in Chapter 3, five research plan candidates were formulated for evaluation by the ACRP panel members using the principles of the Pugh Matrix methodology (see Section 4.4). The candidates were selected in an attempt to provide answers to the following questions: 1. To what extent is student learning affected by aircraft noise? 2. What is the most appropriate noise metric for describing aircraft noise as it affects learning? 3. What is the threshold above which the effect is observable? 4. Has insulation meeting existing classroom acoustic criteria improved student achievement? 5. How does aircraft noise affect learning for students with different characteristics? The first of the five candidates represents the initial thinking, and is termed the Datum candidate. It consists of a nationwide macro-analysis at a large sample of US airports to examine the relationship between student performance, as measured by test scores, and noise level as measured by different metrics. The analysis considers schools exposed to aircraft noise, schools that have been sound insulated, and control schools not exposed to noise. The other four candidates are modifications of the Datum, formed by introducing on-site case studies at the expense of a reduced number of schools to be analyzed in the macro-analysis. This section of the report summarizes the five candidate plans and describes the process used for selection of the plan that best meets the objectives of the study. Additional details on the candidate plans, together with pros and cons of each approach, can be found in Appendix D to this report. 4.2. Research Plan Candidates The research plan candidates are summarized as follows: Datum: Macro-analysis (60 airports) Conduct a nationwide macro-analysis of the relationship between aircraft noise exposure and student performance taking into account the effect of school sound insulation and other confounding factors. The analysis will use the top 60 US airports sorted by the number of schools exposed to DNL 55 dB and higher. The student performance measure is the standardized test scores (reading and mathematics). Alternative 1: Macro-analysis (50 airports) with Follow-up Analysis This is the same type of macro-analysis as the Datum except that the top 50 US airports will be used instead of the top 60. Resources are shifted in order to follow-up the macro-analysis 4-1

with a more detailed examination at a small sample of schools that the analysis identifies as atypical. Alternative 2: Macro-analysis (40 airports) with Observation Case Study This is the same type of macro-analysis as the Datum and Alternative 1 except that the top 40 airports will be used instead of the top 60 or 50, respectively. Resources are shifted in order to conduct a single school case study to observe changes in classrooms when exposed to aircraft noise and to measure the individual noise events. Alternative 3: Macro-analysis (30 airports) with Follow-up Analysis and Case Study This is the same type of macro-analysis as the Datum and Alternatives 1 and 2 except that the top 30 airports will be used. Resources are shifted in order to conduct both a follow-up study and a single school case study. The follow-up analysis is like Alternative 1. The case study is the same as Alternative 2. Alternative 4: Macro-analysis (15 airports) with Follow-up Analysis and Expanded Case Study This is the same type of macro-analysis as the Datum and Alternatives 1, 2, and 3 except that the top 15 airports will be used. Resources are shifted in order to conduct both a follow-up analysis and expanded case study. The follow-up analysis is like Alternatives 1 and 3. The case study involves classroom observations as proposed in Alternatives 2 and 3, but includes two schools with the addition of student and teacher questionnaires given through focus groups. Table 4-1 presents a preliminary estimate of the data sample sizes for each candidate. The differences in sample sizes among the candidates reflect the different distribution of resources in order to accommodate the various analyses and studies within the study budget. TABLE 4-1 Data Gathering for the Research Candidates Data/Candidate Datum Alt 1 Alt 2 Alt 3 Alt 4 Macro-analysis # Airports 60 50 40 30 15 # Target Schools DNL 55-60 694 662 624 576 437 DNL 60-65 240 234 219 199 154 > DNL 65 76 76 74 70 59 Follow-up Schools NA < 20 NA < 20 < 20 Case Study # Airports NA NA 1 1 1 # Schools 1 1 2 A power analysis was conducted to determine if the target school sample sizes are sufficient to provide statistically significant results. With assumptions on the expected effect size taken from the recent RANCH study (Stansfeld 2005), the corresponding minimum sample sizes would be approximately 210 at DNL 60-65, 120 at DNL 65-70, and 80 at DNL 70-75 to answer these same research questions. A detailed description of the power analysis conducted can be found in Appendix D.6. 4-2

It was concluded that the research plan candidates fall short of meeting sample size minimums to varying degrees However, the power analysis was based on previous research involving only Leq-based aircraft noise metric. The current study is planned to explore metrics that are distinctly different from Leq in hopes of finding one that has a better relationship with learning. Thus, the preliminary estimates of sample size requirements could be viewed as a worst case scenario. 4.3. Evaluation of Research Plan Candidates A two-step procedure was used to evaluate the five candidate research plans, select the best overall alternative, and modify the individual components to achieve the study objectives. First, a formal quantitative evaluation of the five candidates was performed using the Pugh Matrix method. This was followed by a qualitative analysis performed by the ACRP 02-26 Project Panel and the research team to take into account comments received from the formal evaluation. The final decision on selection of a research design was then made by the Project Panel. The evaluation process is described in the following sections. 4.3.1. Pugh Matrix The Pugh Matrix is a method for concept selection using a scoring matrix in which alternatives are scored relative to weighted criteria. It is widely used in the Six Sigma Method as it provides a straightforward means to choose the best alternative with limited information. It uses simple scoring of the relative merits of the alternatives based upon criteria that attempt to take into consideration the needs of the user. The Pugh Matrix compares a concept to a reference concept, usually referred to as the Datum, using a matrix of the form in Table 4-2. TABLE 4-2 Example of the Pugh Matrix Evaluation Criteria Concepts Weight Datum A1 A2 A3 A4 R1 W1 S11 S12 S13 S14 R2 W2 S21 S22 S23 S24 R3 W3 S31 S32 S33 S34 R4 W4 S41 S42 S43 S44 R5 W5 S51 S52 S53 S54 Score T1 T2 T3 T4 First, a set of criterion (R) is established. The weight applied to each evaluation criterion (W) is derived from the relative importance of that factor. The sum of the weights (Wi) must equal 1. The rating scheme assigns a criteria score (S) for each alternative (A). Each score is based on a comparative judgment against the datum or reference concept. The total score (T) for each alternative is simply the summation of the individual rating multiplied by the criterion weight as shown in the following equation:. ( )∑ = ⋅= 6 1i ijikj SWT Where k = individual evaluator. The most preferred concept is the one that achieves the highest overall score (T). An example of rating scheme (S) to evaluate concepts against the datum is given in Table 4-3. 4-3

TABLE 4-3 Pugh Matrix Rating Scheme Relative Performance Rating Much worse than reference concept (Datum) 1 Worse than reference concept (Datum) 2 Same as reference concept (Datum) 3 Better than reference concept (Datum) 4 Much better than reference concept (Datum) 5 The evaluation criteria for comparing alternatives is based on rigorous, defensible experimental design, successful plan execution within the funding and time constraints (i.e. risk), and implementable findings that meet the project objectives. To these three prime elements, is added the desire that whatever is learned from this study can be of value to future research. The evaluation criteria (R) and weighting (W) for the assessment of alternative research plans are presented in Table 4-4. TABLE 4-4 Research Plan Evaluation Criteria Criteria Description Weighting Experiment Design: • Internal validity Degree to which observed changes (student performance) can be attributed to the intervention (aircraft noise) and not to other possible causes. 10% • Content validity Degree to which a test measures an intended content area. 10% • External validity Degree to which the conclusions would hold for other persons in other places and at other times. 10% Risk Potential for relative success or failure, such as running out of time or money before completion. 30% Reward Degree to which the plan answers the project research questions and the potential for immediate application of findings to airport practices 30% Knowledge Value of the knowledge gained for future research or application. 10% Total 100% The scores are used to fill the Pugh Matrix to find the alternative with the highest weighted total score. Table 4-5 presents an example of the Pugh Matrix that was used. TABLE 4-5 Pugh Matrix for ACRP 02-26 Research Plan Evaluation Performance Evaluation Criteria Concepts Weight Datum A1 A2 A3 A4 Design: Internal Validity 0.10 3 Design: Content Validity 0.10 3 Design: External Validity 0.10 3 Risk 0.30 3 Reward 0.30 3 Knowledge 0.10 3 Total Score (T) 3 Note that the Datum scores are all equal to 3. The Datum is the yardstick by which all alternative designs are evaluated and a score of 3 means “Same as the reference concept (Datum).” 4-4

4.3.2. Results of the Formal Evaluation Process Seventeen members of ACRP 02-26 Panel and the project team evaluated the five research study candidates based on the Pugh Matrix. Nonparametric statistics were used instead of typical (parametric) statistics, like mean and standard deviation, because there is not enough information to determine whether the evaluators’ ratings conform to a normal distribution - a necessary requirement for use of parametric statistics. Table 4-6 presents the median ratings and scores for the Datum and each alternative. The median is the middle value of an ordered set of values; in this case the ratings and scores given by the 17 evaluators. The solid green cells identify the maximum value for a given criteria or total score. The diagonal red line cells identify the minimum values. TABLE 4-6 Median Values of the Ratings and Scores Alternative #3, Macro-Analysis with Follow-up Analysis and Case Study, received the highest total score and the highest ratings for 4 of the 6 evaluation criteria. It also tied for the lowest rating for the Risk criteria. 4.3.3. Qualitative Evaluation of Candidate Plans Following the formal quantitative evaluation of the candidate research plans the ACRP 02-26 Panel convened with the research team to discuss the comments received from the Pugh Matrix process and make a final selection of the plan to be performed. The major comments and responses are listed below (in some cases the actual comments are paraphrased without change in meaning). Comment: There is another aspect of this problem of variation in outdoor-to-indoor noise reduction which does not appear to be explicitly identified - the variation in outdoor-to-indoor noise reduction for schools located in different cities with very different climates Response: We agree that variation of indoor to outdoor levels at classroom and school level is a factor that we cannot control within the study design, but believe that the potential confounding effect is mitigated by the large sample sizes involved and the approach that we plan to take. The analysis to find whether aircraft noise has an effect is done by assessing achievement differences between target schools and comparable control schools at the state level and then to aggregate Datum #1 #2 #3 #4 Experimental Design: Internal Validity 0.10 3.00 3.00 3.00 3.00 2.00 Experimental Design: Content Validity 0.10 3.00 3.00 3.00 2.90 2.00 Experimental Design: External Validity 0.10 3.00 3.00 2.90 3.00 2.50 Risk 0.30 3.00 3.00 2.00 2.00 2.00 Reward 0.30 3.00 3.00 4.00 4.00 2.00 Knowledge 0.10 3.00 4.00 4.00 4.50 4.00 3.00 3.08 3.10 3.23 2.58Total Score Research Plan Candidates Performance Evaluation Criteria Weight Alternative 4-5

across states; the assumption being that target and control schools at the state level are similar enough that any difference in achievement is due to the aircraft noise exposure. Comment: If highway is feeder to airport, then it should be viewed as airport-related. Response: The overall objective of the study is to determine the effect of aircraft noise on student learning. Some of the schools (target and control) will be exposed to highway noise. It is not possible within the scope of the study to calculate the actual highway noise levels at each school, but, as stated in our research plan, we will be conducting a spatial analysis to determine the distance from each school to a major highway, and including this distance as a confounding variable in the analysis. Comment: I am concerned about the overall data analysis plan. I agree that normalization is appropriate to compare schools across states, but it’s unclear to me why the state effect was not explicitly modeled using a random or fixed effect approach. It also seems to me that there will be other group effects that need to be controlled for that are not accounted for in the current plan, including differences by school district, and by school when subgroups or grades are analyzed. Also, the analytical plan describes the dependent variable as ‘school-level’ test scores, but then also describes a longitudinal analysis that will follow students by grade. So is the analysis by grade level or school level? If it is by grade level, there need to be additional controls to capture unobserved differences across schools. Also, some of the independent variables appear to be constructed at the ‘grade level’, such as the ‘percentage of different racial/ethnic backgrounds by grade’, but there is no approach outlined to account for the mismatch among the dependent (school level) and independent (grade level) variables. Response: The dependent variable data available for this study are aggregate mathematics and reading/language arts achievement indicators for students in each tested grade in each public school, in some cases also disaggregated by gender, race/ethnicity, poverty, or disability status. The comment concerns three potential group effects in the study: states, school districts, and grades. States. Because individual states administer different achievement tests, with different properties, while within each state all schools administer the same test battery, it is necessary to eliminate state effects from the analysis. For this reason, all analyses to be carried out in this study compare achievement measures in schools exposed to airport noise with other schools in the same state. State effects are eliminated from the analysis by setting the means of variables included in the analysis to zero within each state. Because the number of schools exposed to significant airport noise in a single state is small, it is necessary to aggregate the results of the within-state comparisons across states to achieve acceptable power for statistical tests. Nevertheless, results for individual states will be compared to explore potential explanations of outliers if they should occur. School districts. School district effects on achievement are primarily due to demographic and resource factors (e.g., race/ethnic distributions, percentages of students eligible for free lunch, urbanicity, enrollment size, pupil-teacher ratios), which will already be statistically controlled as predictors of school-level achievement variation in the analysis. A test will be made for 4-6

significant district-level variation not accounted for by school-level predictors, and where it is found, it will be included in the analysis. Grades. Although airport noise levels are at the school level, the available achievement level data are for individual grades at each public school. For example, for analysis of the relationship between airport noise and third-grade achievement, we can use third-grade achievement scores. However, for various states in various years, achievement data are only available for a subset of grades, because those states only required testing in some grades. For the analysis, the achievement data for each grade in each school in a state exposed to airport noise are compared to all other students in the same grade in the same state. In order to maximize the power of the study, data must be aggregated across states and years in which testing was in different grades. In addition, because average achievement for one grade is only moderately reliable as a measure of average achievement in an entire school, we plan to aggregate achievement across grades in each school for the main analysis. If separate grade level effects are significant, the average grade level of tested students in each school will be included as a control. General Comments: • Although Alternative 3 may offer more rewards, it may sacrifice statistical validity and ability to analyze alternative metrics. It also carries more risk. I would be hesitant to recommend this alternative. • The expanded case study approach would add value in concept but remains too limited to justify the significant decrease in the number of airports included in the basic analysis. • Even though the Panel member was a huge proponent of case studies, he felt that by only selecting one school, the results would not be scientific. • The purpose of the project was first and foremost the identification of the dose-response relationship, and advocated either all case studies or maximizing the number of schools to analyze. • The gain to the industry of one case study could be minimal, and would depend on, among other factors, luck. • A Panel member stated that he initially had negative feelings about the case studies, but has since had additional issues with decreasing the number of schools. • A general discussion of Alternative 3 noted that it has the risk of not including enough airports, thus affecting the ability to distinguish impacts because of the decreased number of airports included in the survey. • Alternative 3 does not include getting verbal response from students or teachers. A case study may improve the ability of identifying effect of intermittent significant noise events. 4.4 Conclusion In summary, the Panel recognized the potential advantage of including a case study, but that since only one or two schools would be studied, the results might not be representative. The Panel was of the opinion that including a case study would reduce the number of schools in the 4-7

macro-analysis, but that the most important objective of the study was to develop a relationship between noise and test scores. The number of schools should be maximized to reduce risk. As a result, the Panel agreed that Alternative 1 was the first choice, with suitable documentation being provided for the tradeoffs, variables, and potential for future research. 4-8