3
Methodological Issues

Mobility is a complex phenomenon to study. Directly or indirectly, each of the presenters highlighted some of the methodological challenges in collecting and analyzing data that would enable them to make solid inferences about the effects of mobility on children’s development and the effectiveness of possible interventions. Mobility is not a single event that happens at a particular point in time, but a series of processes and changes that may have complex and cumulative effects— and these effects are likely to vary with the characteristics of the children who experience the mobility. Eric Hanushek and Jens Ludwig provided two perspectives on ways of identifying and assessing the impact of mobility.

ISOLATING THE EFFECTS OF MOBILITY ON INDIVIDUALS AND SCHOOLS

Hanushek began with the point that had emerged so clearly from the data already discussed—that although national residential mobility rates may have declined, they are consistently highest among low-income families. The challenge is to distinguish between positive moves, made in search of better schools and neighborhoods, and moves that are made because of some sort of disruption and that cause harm. Disentangling the effects of these two kinds of moves is difficult because researchers lack reliable measures of school quality and of family choices and behaviors, and because ways of observing the causes of mobility are limited. Never-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 33
3 Methodological Issues M obility is a complex phenomenon to study. Directly or indirectly, each of the presenters highlighted some of the methodological challenges in collecting and analyzing data that would enable them to make solid inferences about the effects of mobility on children’s development and the effectiveness of possible interventions. Mobility is not a single event that happens at a particular point in time, but a series of processes and changes that may have complex and cumulative effects— and these effects are likely to vary with the characteristics of the children who experience the mobility. Eric Hanushek and Jens Ludwig provided two perspectives on ways of identifying and assessing the impact of mobility. ISOLATING THE EFFECTS OF MOBILITY ON INDIVIDUALS AND SCHOOLS Hanushek began with the point that had emerged so clearly from the data already discussed—that although national residential mobility rates may have declined, they are consistently highest among low-income families. The challenge is to distinguish between positive moves, made in search of better schools and neighborhoods, and moves that are made because of some sort of disruption and that cause harm. Disentangling the effects of these two kinds of moves is difficult because researchers lack reliable measures of school quality and of family choices and behaviors, and because ways of observing the causes of mobility are limited. Never- 

OCR for page 33
 STUDENT MOBILITY theless it is important to identify possible negative effects of mobility, both for individual students and the schools into and out of which they transition. To address this need, Hanushek and his colleagues attempted to hold all other factors (e.g., family and neighborhood characteristics) constant and examine the independent effect of changing schools. They developed a model that uses measures of achievement growth to examine the differ- ent ways in which mobility might affect individual students and schools.1 Achievement is the product of many factors, so they made a few assump- tions to simplify the analysis. First, they assumed that achievement during the school year prior to a move was not unusually bad or good. Second, they assumed that students are generally on a growth path, and that this path is likely to follow a particular trajectory, assuming a constant school quality from grade to grade. That is, if a student is learning at grade level, he or she is likely to continue at that level after moving, assuming the new school is of similar quality. Finally, they assumed that the disruption that caused the move (e.g., divorce, economic upheaval) lasted only for the year in which the move took place. These assumptions allowed them to isolate any changes in school quality that students experienced when they moved. Hanushek and his colleagues applied the model to data from the Texas Schools Project, which provided attendance and mathematics achieve- ment data for three cohorts of students in grades 4 through 7. For the general student population, including students who move for different reasons, they found that the move itself had little effect on the students’ academic trajectory (past the disruption in the year of the move). In other words, Hanushek said, “if they’re in a bad situation, the move has very little marginal effect on their bad situation.” More challenging was to discover whether high mobility has a measurable effect on peers, teachers, and the quality of the schools into and out of which students move. To do this, they compared student achievement among, for example, fourth graders in consecutive years, and used the differences in mobility rates “to see whether mobility shows up in differences in achievement, other things being equal.” They found that “higher student mobility in a school during the school year really hurts everybody, and it hurts people in a fairly dramatic way.” Hanushek emphasized that for people in high-mobility schools these effects persist throughout their school careers. Moreover, African Ameri - can students in the Texas sample had much higher mobility rates than other children, and they also tended to go to schools with much higher mobility. Hanushek and his colleagues estimated that the difference in mobility rates between white and African American students in Texas 1 The model is described in detail in Hanushek, Kain, and Rivkin (2004).

OCR for page 33
 METHODOLOGICAL ISSUES “can explain about 15 percent of the achievement gap by grade seven.” Participants questioned the representativeness of the Texas study, and particularly the assumption in that study that the effects of mobility are temporary. Hanushek concluded that understanding the impact of mobility is difficult because measuring the characteristics of schools and families that make a difference is so difficult. But, by isolating individual fixed effects, they were able to show that while the effects of mobility on individuals are fairly small, the effect on schools is quite large. SEEKING CAUSALITY Jens Ludwig described the methodological challenges associated with estimates of the effects of residential mobility on outcomes for children. As others had already discussed, there are numerous ways in which mobil- ity might affect development, positively or negatively. Moving may be disruptive, and it also changes the social networks, routines, and perhaps the available coping strategies in a child’s life. After a move, a child may have more, fewer, or simply different community, social, institutional, and physical resources. Because the reasons for a move and its consequences are so varied, it is difficult to think about a single effect of moving. The implications for child development and schooling are likely to depend in significant ways on the motivation, circumstances, and options facing a family that moves. Thus, to understand the effect of a move, it is necessary to understand the circumstances the family might have experienced if they had not moved—or the “counterfactual” condition. For example, if the head of a household loses his or her job, the family is evicted from their home, and they move in with relatives, should the counterfactual be identified as the scenario in which the job was not lost, the job was lost but the family had the wherewithal to avoid eviction, or the family was evicted but moved into a different sort of neighborhood? Alternatively, if a head of house - hold receives a promotion and a raise in pay and moves the family to a wealthier neighborhood, is the counterfactual the adult is not getting the promotion, the family not moving, or the family not moving but sending their children to private school? In seeking the correct counterfactual, one might consider the vari - ous constraints on the choices families are making. A family that experi- ences a loss of income may or may not move, but it will have to adjust in some way, and most of the possible adjustments could affect children’s development. So, Ludwig suggested, from a social science point of view, one important implication is that in conducting statistical analyses it is possible to “overcontrol” as well as to “undercontrol” for relevant con- founding factors. That is, some of the explanatory variables researchers

OCR for page 33
 STUDENT MOBILITY include in their models may be factors that families would be forced (or able) to adjust whether or not they chose to make a move. The most promising way to understand the messy world of people’s choices can best be studied in terms of questions about how they respond to par- ticular interventions, whether they are policy interventions or natural experiments that are induced by changes in the economy or the housing market. If one begins with this sort of policy evaluation question, it can be researched using a variety of observational approaches, in which two groups that naturally experience different treatments (without interven - tion) are compared and the researcher controls for factors other than the treatment that might have affected the outcome. Alternatively, one might use a randomized approach, in which subjects are randomly assigned to different treatment groups, or so-called natural or quasi-experiments, in which the treatment is “assigned” by a change of policy or other factor beyond the control of subjects. These distinctions have stimulated passionate disagreements about establishing causation in many contexts, and, Ludwig noted, an empiri- cal literature has emerged that describes ways of using nonexperimental estimation when a randomized study is not feasible. For example, Robert LaLonde (1986) developed an influential approach to evaluating estima - tion methods. Researchers who have data from a randomized experi - ment, and therefore know the “right” answer regarding the effect of a policy intervention on particular outcomes, can then try to use different observational methods to see if it is possible to reproduce the results that were obtained experimentally. “The answer that LaLonde got,” Ludwig explained, was that there were big differences between the two sets of results, a finding that “was shockingly grim and has had a profound influ- ence on the field of empirical economics and applied statistics.” 2 This approach was initially developed for research on employment, but it has been used to examine education questions as well. Although the results of nonexperimental estimates vary in practice, depending on the context and the quality of the available data, Ludwig has found that, on average, they do not look particularly impressive. He described a comparison between experimental and nonexperimental results related to mobility to illustrate the problem, using 1990s data from the U.S. Department of Housing and Urban Development’s (HUD’s) Moving to Opportunity (MTO) study. In that study, a total of 4,600 families in 5 cit- ies were recruited to participate and randomly assigned to 1 of 3 groups. Two of the groups received different sorts of housing vouchers designed to help them move to neighborhoods with lower poverty rates, and the 2 Other researchers have done similar work with somewhat different results (see Dehejia, 1999; Heckman, 2008; Smith and Todd, 2005).

OCR for page 33
 METHODOLOGICAL ISSUES control group received no voucher. Thus, Ludwig explained, “the random assignment resulted in nontrivial differences in the average neighborhood environment for otherwise similar types of families.” These results could be used to compare the effects of different specific neighborhood charac- teristics and mobility on outcomes for children. To develop a nonexperimental estimate for comparison with the empirical results of the randomized experiments, researchers can use dif - ferent methods, such as standard regression analysis and its close cousin, propensity score matching, in which certain variables are held constant and others are varied with the goal of isolating a particular effect. The ability of these sorts of nonexperimental or observational approaches to reproduce the experimental answer probably depends on the qual - ity of the data that are available, so it is important that a fairly rich set of background characteristics are available for the families and children in the MTO study, including demographics (e.g., age, household size), socioeconomic data (e.g., income, parental employment, history of pub - lic assistance), housing and mobility history, perceptions of the baseline neighborhood (e.g., having local family or friends, perception of safety), and, perhaps most important, the family’s motivation for wanting to move and past experiences in different types of neighborhoods. Figure 3-1 shows the results of the comparison between the empiri- cal results of the randomized experiment and the estimated results of applying nonexperimental methods to the MTO data on the effects of housing vouchers. The light gray bars, labeled OLS, are the estimated, nonexperimental results for various outcomes, and the dark gray bars, labeled IV, are the experimental results. The estimated results indicate, for example, that boys who live in high-poverty areas are slightly less likely to be involved in risky behavior than boys who live in low-poverty areas. However, the darker bar indicates that the experimental results were quite different. This result occurs, Ludwig suggested, because factors not observed in the study lower the likelihood that these boys would engage in risky behaviors. In his view, the stark disparity on the outcomes for the two different approaches to examining mobility in this context makes clear that the nonexperimental method rested on assumptions that the empirical evidence indicates were wrong. However, he pointed out, ran- domized or natural experiments can reduce the selection bias that limits the usefulness of the estimates, but they also limit the range of questions that can be explored. Ludwig argued that the research portfolio regarding the factors that determine mobility is currently out of balance. He suggested that random- ized and natural experiments should make up a larger proportion of the research portfolio than they do now. At the same time, rich descriptive

OCR for page 33
 STUDENT MOBILITY 2.0 1.5 Standard Deviation Units 1.0 0.5 0.0 OLS IV – 0.5 – 1.0 – 1.5 – 2.0 – 2.5 K6 K6 y y y isk lth k ris om er ea _r n_ ht M er h ug So n_ ht Da ug So Da FIGURE 3-1 Comparison between empirical and estimated results. NOTE: OLS = the estimated, nonexperimental results for various outcomes; IV = the experimental results. SOURCE: Adapted from Kling, Liebman, and1 Figure 3- Katz (2007). studies would support the design of better experiments down the road, by highlighting possible mechanisms and other researchable questions. Participants generally liked his approach, although(2009). reiterated Source: Ludwig several the point that using strict randomly controlled trials to study families’ housing decisions and other questions related to mobility is often not feasible and perhaps unethical. Moreover, recent improvements in non - experimental methods mean that more feasible alternatives exist. 3 Sev- eral stressed the value of natural experiments, looking at, for example, the impacts of foreclosures on students’ academic achievement. Another suggested that, given the practical and ethical difficulties of randomly controlled studies, the solution is to seek “overwhelming data of the 3 Nonexperimental research designs that might be used when randomized controlled stud- ies are not feasible include more qualitative approaches, such as case studies or ethnographic surveys; longitudinal studies; correlational studies; and statistical analyses, such as regres - sion discontinuity, use of instrumental variables, or propensity score matching.

OCR for page 33
 METHODOLOGICAL ISSUES weaker kind that [describe how] a process works in the natural world.” These discussions suggest that a combination of rich qualitative reports, studies using the weaker regression design, and a theoretical model about the mechanisms through which an intervention works might be very compelling. Ludwig acknowledged the point, adding that because so many fami- lies who would be eligible for housing assistance are not receiving it, there are multiple opportunities for comparing outcomes among differ- ent groups. Ludwig stressed that answering questions about causation is most important at the point when policy makers are contemplating a specific intervention. The key concern with nonexperimental methods, he suggested, is that they can give a misleading reading on the size or even the existence of key causal relationships. More important, he said, “We currently do not have very good methods for determining how well our nonexperimental approaches are working. If we cannot be sure that our nonexperimental estimates are close to the truth we must be appropriately cautious in basing policy decisions on such results, and continue to push harder to develop a body of evidence that gives us greater confidence that we understand what the consequences of different policy decisions will be.” Reflecting on the state of mobility research in light of the method- ological issues that had been raised, Stephen Raudenbush offered several observations. First, he noted that the field has benefited recently from a body of carefully done empirical studies that represent an important step forward. These studies have established a strong basis for conclu - sions about which children are exposed to the most mobility and which children seem to be at greatest risk for harm from mobility. But the causal questions are complex, and the interpretations of what these data mean are less straightforward. Raudenbush highlighted the value of Hanushek, Kain, and Rivkin’s approach and Ludwig’s challenge. Their model bypassed the lack of complete data to show the short-term disruptive effects of a move in a way that takes into account the effects of the new school and neighbor- hood of a move. It also has a useful way to reframe the causal question by looking for the impact of attending a school that is characterized by high levels of mobility. This is valuable, Raudenbush observed, because it demonstrates the need for a policy intervention based on the results of this causal question. For his part, Ludwig challenged researchers and policy makers not to take shortcuts on causal questions. Strong design considerations—either randomized experiments, regression discontinuity studies in which the selection process is fully known, or natural experi - ments in which there is a clear instrumental variable—are needed to sup- port causal interpretations.

OCR for page 33