Appendix A
Assessing the Evidence
Members of the committee were drawn from different disciplines with different methodological and analytic approaches to developing and interpreting evidence. The committee relied on the range of experimental and observational approaches described in Table A-1.
One of the committee’s central challenges was to summarize what is known about the causes of disparities in adolescent behavior, health, and well-being and the effects of those disparities on adult outcomes. Of course, these explanations depend on the synthesis of many studies, each of which has methodological strengths and limitations relating to the nature of the study design and the data collected. For example, researchers who have observed a positive association between student test scores and later adult economic outcomes are careful not to assume a causal relationship between educational performance and employment, due to the possibility of confounders—variables that are correlated with both test scores and economic outcomes—causing a spurious association between the two. By contrast, data generated by randomized controlled trials (RCTs) do not suffer from confounding, because the act of randomization severs any relationship between a confounder and the independent variable (e.g., education, in the example above). As such, data generated from an RCT can produce an estimate of the causal relationship between education and adult economic status without concern that confounders are biasing the estimate. RCTs are increasingly being used in social science research (Jackson and Cox, 2013). Nevertheless, RCTs are not appropriate in all settings, because they can be infeasible or expensive, and for some interventions
TABLE A-1 Description of General Uses of Experimental and Observational Approaches
| Experimental Approaches | |
| Large-Scale Randomized Controlled Trials (RCTS) | RCTs are frequently used to assess the effects of a particular educational intervention, such as school infrastructure, teacher characteristics, or school organization, on student outcomes (see, e.g., Connolly et al., 2018). RCTs tend to be implemented only after observational research has provided evidence that a proposed intervention is likely to be effective. Reasonable assumptions can therefore be made about the likely risks and benefits of the research to participants. |
| Small-Scale Experiments | Small-scale experiments are often used as a precursor to an RCT and are used frequently in educational research in psychology and economics. They rely on small sample sizes and are frequently implemented without a strong observational evidence base. Because the effects of such experiments are more difficult to predict in advance, risks and benefits to participants are also more difficult to predict. |
| Observational Approaches | |
| Analysis of Administrative Data | Statistical analyses of administrative data are common in the educational context (see e.g., Fahle and Reardon, 2018). Data collected from administrative sources are “de-identified” and held in specialist data centers, so that risks to subjects are minimal. |
| Analysis of Survey Data | Statistical analyses of survey data are also common. They rely on data collected by individual researchers or survey agencies. Participation in surveys is voluntary, and risks to subjects are usually minimal. |
| Collection of Qualitative Data | In some disciplines, and especially in sociology, ethnographic and interview-based studies make up a substantial part of the evidence base on educational inequality. Due to the importance of thickly detailed description in such studies, anonymity is a special concern, so researchers go to considerable lengths to de-identify individuals and contexts when reporting research results. |
randomization can be considered ethically objectionable if it denies an adolescent a service or treatment known to be beneficial.
Because of these concerns about RCTs, researchers have increasingly used “natural experiments” to estimate causal effects. Such studies harness changes in state and local policies that generate plausibly random or quasi-random variation in exposure to a given service or treatment to estimate its causal effect on outcomes of interest (see Meyer, 1995 and Angrist and Pishke, 2009 for an overview of these methods). One such example is the use of the draft lottery to estimate the impact on future earnings of service in the military among youth during the Vietnam War (Angrist, 1990).
Because one’s draft number is randomly generated and draft numbers are very predictive (though not perfectly predictive) of military service, the draft generates variation in military service that is not confounded. A researcher using various statistical and econometric techniques can use this variation to generate a causal estimate.
Despite the limitations of estimation based on observational data, careful use of observational data has many advantages: First, it is very useful for identifying associations that can be more rigorously studied using other approaches; second, in some cases careful use of natural experiments or other research designs can minimize the bias from confounding; and finally, some questions are by their nature not amenable to randomized trials (such as the question of what the effects are of a new state law that changes the age of majority in the criminal justice system) and so can only be studied using observational data.1
___________________
1 In addition to concerns over research design, there are significant ethical considerations related to research on adolescents. All research on vulnerable populations raises special ethical considerations, because members of these populations may be less capable of providing informed consent, of assessing the costs and benefits of participating in research, and of entering into research voluntarily. Adolescents are particularly likely to be vulnerable: their capacity for self-determination is understood to be related to maturity, they are particularly likely to be members of social and economic groups seen to require greater protection, and they may be members of institutionalized groups that are vulnerable by virtue of being institutionalized.
This page intentionally left blank.
