Multilevel Modeling of Intervention Effects

Multilevel modeling of contextual effects, such as the school, has also been well integrated into the evaluation of preventive trials. At the time of the 1994 IOM report, it was rare for published analyses of group-based randomized trials to correct for nonindependence among the participants in a group. As a result, they could erroneously report impact when it was not statistically significant. In a trial with 20 schools, half of which are randomized to a prevention program, the correct statistical test of impact is based on the number of schools, not the number of children, which may be several orders of magnitude larger (Murray, 1998). Now it is expected that published papers of group-based randomized experiments will use multilevel analysis (Raudenbush, 1997) or generalized estimating equations and sandwich-type estimators (Zeger, Liang, and Albert, 1988; Brown, 1993b; Flay, Biglan, et al., 2005) to account for group randomization.

Modeling That Incorporates Growth and Context in the Same Analysis

At the time of the 1994 IOM report, it was customary to report only the overall impact of an intervention in a population. Since then, statistical modeling has advanced so that longitudinal and multilevel modeling can now be handled in the same analysis. It is common to see analyses that include both individual growth and multiple levels of nesting, such as children nested within classrooms and schools (Gibbons, Hedeker, et al., 1988; Brown, Costigan, and Kendziora, 2008). Analyses can examine how change occurs across multiple levels (Raudenbush and Bryk, 2002) and examine impact across both individuals and contextual levels with different types of growth trajectories (Muthén, Brown, et al., 2002; Muthén and Asparouhov, 2006; Asparouhov and Muthén, 2007).

Handling of Missing or Incomplete Data

A major advance has been the treatment of missing data in statistical analysis of longitudinal data. When the previous IOM report was written, most published analyses of intervention impact simply deleted any missing cases. Now most impact analyses make use of full maximum likelihood methods (Dempster, Laird, and Rubin, 1977) or multiple imputations (Rubin, 1987; Schafer, 1997; Schafer and Graham, 2002; Demirtas and Schafer, 2003; Graham, 2003; Graham, Cumsille, and Elek-Fisk, 2003). These techniques are especially important for evaluating impact across long periods of time, because data will be incomplete for many of the participants and differentially across contexts.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement