Early Warning Indicators
Dropping out is not something that occurs at a single point in time. A growing body of research suggests that dropping out is but the final stage in a dynamic and cumulative process of disengagement from school (Appleton et al., 2006; Finn and Cox, 1992; Glanville and Wildhagen, 2007; Klem and Connell, 2004; Rossi and Montgomery, 1994; Rumberger, 1987; forthcoming; Rumberger and Arelleno, 2007). Disengagement may begin as early as elementary school, when students fail to become involved in either the academic or the social aspects of school. Poor performance on assignments, misbehavior, failure to do homework, and lack of participation in extracurricular activities are all signs of disengagement, which often leads to frequent absences, retention in grade, and repeated transfers to other schools.
A number of research studies substantiate that these signs of disengagement are precursors to dropping out, and students may advertise their intentions fairly early on. As Robert Balfanz, a researcher with Johns Hopkins University, stated at the workshop, some students are “waving their hands wildly saying ‘if you don’t do something drastic, I’m not going to graduate.’” The key to reducing the dropout rate is to notice these behaviors and intervene at a stage when there is a chance for correction.
It is important to note that much of the research on this topic is descriptive. The research documents associations between certain behaviors and dropping out, but does not necessarily support conclusions that these characteristics cause students to drop out. It could well be that other factors are the underlying problem, and these factors cause students both to become disengaged from school and to drop out.
During the workshop, several presentations focused on research identifying the precursors of dropping out. In this chapter, we first discuss the findings from this research and then describe ways that data systems can be developed to incorporate these indicators and used to develop intervention strategies.
RESEARCH ON PRECURSORS TO DROPPING OUT
A considerable body of research exists on precursors to dropping out, but research findings are not entirely definitive, and the advice they offer has evolved over time (see, e.g., literature reviews in Gleason and Dynarski, 2002; Jerald, 2006; National Research Council, 2001). Early research suggested that certain social and family background factors were associated with an increased risk of dropping out, such as being poor, minority, from a single-parent family, or from a family with low educational attainment or low support for education (Barro and Kolstad, 1987; Eckstrom et al., 1987; Haveman, Wolfe, and Spaulding, 1991; Mare, 1980, National Center for Education Statistics, 1990, 1992; Natriello, McDill, and Pallas, 1990; Rumberger, 1995). In the 1980s, researchers began questioning the role of individual factors—in part because these variables are beyond the control of school systems—and research was designed to identify school-related factors associated with dropping out (Whelage and Rutter, 1986, cited in Jerald, 2006). This research documented that although individual demographic factors are related to dropping out, students’ educational experiences are equally important. These studies showed that students who dropped out reported that they disliked school and found it boring and not relevant to their needs; had low achievement, poor grades, or academic failure; or had financial needs that required them to work full-time (ERIC Digest, 1987; Jerald, 2006; Jordan, Lara, and McPartland, 1999). Other research has identified school-related factors associated with lower dropout rates, including high schools with smaller enrollments, more supportive teachers, positive relationships among students and school staff, and a more rigorous curriculum (Croninger and Lee, 2001; Lee and Burkham, 2000; McPartland and Jordan, 2001).
The late 1980s and early 1990s brought concerted efforts to develop intervention programs designed to prevent at-risk students from dropping out. These programs were supported, in part, by federal grants from the School Dropout Demonstration Assistance Program. Federally funded evaluations of these efforts examined the effectiveness of the approaches the programs used for identifying at-risk students. These reviews found that the approaches tended to misclassify students, resulting in programs serving students who would not have dropped out and failing to serve students in most need of preventive services (Dynarski, 2000; Gleason and Dynarski, 1998, 2002). In these studies, Gleason and Dynarski reported that many of the variables used to identify at-
risk students were poor predictors of dropping out, correctly identifying less than a third of dropouts.1
At the time, identification of at-risk students was primarily accomplished through single-point-in-time indicators, such as checklists or questionnaires that reflected performance and attitudes in the given year only. Longitudinal information that tracked students and cohorts over time was rarely available. Insights learned from this research suggested that risk factors may cumulate from year to year and that there may be benefit to measuring trends in students’ status on the risk factors over time (Gleason and Dynarski, 2002). For instance, for some students, a year of poor performance may be regarded as a temporary setback that causes them to buckle down and work harder the next year. Other students may not be as resilient—one year of poor performance may lead to another, causing the student to become discouraged, to increasingly detach from school, and ultimately to drop out.
Several comprehensive studies followed that made use of longitudinal data (i.e., Allensworth and Easton, 2005; Neild, Stoner-Eby, and Furstenberg, 2001; Roderick and Camburn, 1999). These studies considered some of the same variables evaluated by Gleason and Dynarski, but the existence of data collected over time allowed for examination of the interactions among potential precursors to dropping out and students’ individual levels of resiliency and persistence. Below we discuss two series of studies that grew from work with middle school students in Philadelphia and high school students in Chicago.
Identifying At-Risk Students in Middle School
Robert Balfanz and his colleagues have conducted a series of studies on early precursors of dropping out, warning signs that become apparent before students begin high school. This work began with the Philadelphia school system. At the time the study started, most of the grade 9 students in the 21 Philadelphia neighborhood high schools were over age for the grade (older than the typical ninth grade student). At some schools as many as 80 percent of the freshmen were repeating the grade for the second or third time. Students had poor attendance records in grade 8, and their achievement in mathematics and reading was below grade level.
Balfanz and colleagues followed a cohort of students for seven years, from the 1995-96 school year (when they were enrolled in grade 6) to the 2003-04 school year (one year past their expected time of graduation). The researchers
sought to identify variables that were clear and useful signals of being at risk for not graduating. Aware of the findings reported by Gleason and Dynarski (2002), they set a decision rule for the variables that minimized the number of false positives—that is, variables that minimized the chance of incorrectly classifying a student as at risk of dropping out. For this study, they selected variables that identified grade 6 students with a 75 percent chance or higher of not graduating on time. The variables that met this rule were
attending grade 6 less than 80 percent of the time, and
receiving a poor final behavior grade in a course.
Each of the individual indicators was designed to identify students with a 25 percent or less chance of graduating. For this cohort, less than 30 percent with any one of these indicators graduated by 2003-04. Collectively, the four indicators identified 59 percent of the students in the cohort who did not earn a diploma. To corroborate these findings, Balfanz and his colleagues conducted follow-up studies in Boston, Indianapolis, Mobile, Pueblo (Colorado), and Baltimore.
In all of the districts, grade 6 students who failed English or mathematics were at high risk of not graduating.2 More recent studies in California have corroborated the relationship between course failure in middle school and the risk of dropping out. Studying the 2006 graduating class in Fresno Unified School District, Kurlaender, Reardon, and Jackson (2008) found that students who failed two or more courses in grade 7 were much less likely to graduate than students who did not fail any classes. In this study only 24 percent of the students who failed two or more classes in grade 7 graduated, whereas 71 percent of the students who did not fail any class graduated. Another study, with the Los Angeles Unified School District, found that each failed course in the middle school grades reduced the odds of graduating, with failed classes in middle school reducing the odds more than failed classes in high school (Silver, Saunders, and Zarate, 2008).
Although absenteeism was a consistent indicator across the districts studied by Balfanz, there was no common absolute attendance threshold that met the researchers’ decision rule. In Boston, for instance, the attendance threshold was raised to 90 percent (missing 90 percent of grade 6) in order to meet the decision rule. The researchers hypothesized that it may be the distribution of absences that matters, not the absolute number of days missed; that is, being
in the tail end of the district’s attendance distribution may be the important predictor.
Balfanz and colleagues could not corroborate their findings with regard to behavior grades, because this indicator was limited to Philadelphia. They investigated substituting school suspensions for behavior grades, but the findings were inconclusive.
Balfanz also examined the extent to which low achievement test scores, being over age for grade, status as an English language learner (ELL), and enrollment in special education were associated with dropping out. For the most part, these variables did not add to the prediction of dropping out, once the other variables were considered (attendance, failure in English or mathematics). Further research on several of these variables appears to be warranted, however. In a study of students in Boston, the Parthenon Group (2007) found that being over age was predictive of dropping out. The study also revealed that students who were late-entrance English language learners and special education students taught in substantially separate classrooms had a 75 percent risk of not graduating.
Balfanz and colleagues also considered the grade in which the indicator (failing math, failing English, high absenteeism, poor behavior grade) first became apparent and the relationship between the number of indicators and the likelihood of graduating. The findings suggested that the earlier the indicator first appeared, the lower the students’ chances of graduating. In Boston, for instance, students who first had an off-track indicator in grade 9 graduated at nearly twice the rate as students with an off-track indicator in grade 6. Furthermore, they found that the likelihood of graduating decreased as the number of indicators increased. Students who had multiple indicators had extremely poor graduation outcomes—in some districts, only a few percent of students with all four indicators graduated.
Identifying At-Risk Ninth Graders
Another series of studies focused on identifying risk factors for ninth graders. Allensworth and Easton (2005, 2007) created indicators to classify freshman as on-track or off-track in terms of following a path likely to lead to graduation. In those studies, freshmen were classified as on-track if
the student had earned the course credits needed to be promoted to grade 10, and
the student had no more than one failing semester grade in the core subjects of English, mathematics, science, or social studies.
In their first study, the researchers followed a cohort of students in Chicago who entered high school for the first time in 1999, who should have graduated
by summer 2003 (n = 21,203). Their studies showed that ninth graders who were on-track at the end of the freshman year were 3.5 times more likely to graduate than the students who ended their freshman year off-track. Of the ontrack students, 82 percent graduated from high school within four years, while only 22 percent of the off-track students graduated on time. After five years, the graduation rate was 85 percent for the on-track students and 28 percent for the off-track students. Subsequent studies found the same patterns among later cohorts of students.
Similar to Balfanz, Allensworth and Easton found that performance in coursework was more highly associated with graduation than performance on standardized achievement tests. Their findings showed that the on-track indicator was nearly eight times more predictive of graduation than grade 8 achievement test scores. In their sample, 46 percent of the entering freshmen with achievement test scores in the bottom quartile were on-track by the end of the year, and 71 percent of those on-track students with low test scores graduated on time. Furthermore, nearly 25 percent of the entering freshmen with achievement test scores in the top quartile were off-track by the end of the year, and only 38 percent of this group graduated from high school on time.
The researchers also examined the relationships between on-time graduation and background factors, including race/ethnicity, gender, economic status, parental education, and achievement in elementary school. They found that although there is a relationship between on-track rates and background characteristics, these factors do not predetermine graduation. The on-track indicator predicted on-time graduation equally well for students regardless of their background characteristics. Furthermore, background factors did not substantially improve the prediction of graduation once students’ grade 9 course performance was considered.
Although the on-track variable is easy to understand and to calculate, one drawback is that it cannot be calculated until students complete the freshman year. To compensate for this shortcoming, the researchers studied indicators available earlier in the freshman year, including grade-point average (GPA), the number of semester course failures, and absences. All three variables were as predictive as on-track status, correctly predicting graduation status about 80 percent of the time. Again, once these variables were considered, background characteristics did not add to the prediction.
Of the variables studied, GPA was a slightly better predictor than a simple indicator of passing or failing a course because of its rank-ordered nature (i.e., some students pass their courses with very low grades). Students who ended their freshman year with a GPA of 2.5 or higher had graduation rates of at least 86 percent. Students with freshman year GPAs of 1.5 or lower had a much lower graduation rate: 53 percent for those with a GPA of 1.5, and 28 percent or less for those with a GPA of 1.0 or lower.
Absenteeism was slightly less predictive than GPA or the on-track indicator
(since attending class is not the same as performing well), but even moderate absentee rates were predictive of being at risk for not graduating. Of the students who missed between 10 and 14 days of school in a semester, only 41 percent graduated on time. A follow-up study found that absenteeism and freshman year grades were as strongly predictive of graduation for students with disabilities as they were for students without identified disabilities (Gwynne et al., 2009). Furthermore, absences were the primary reason that dropout rates were so high among students with disabilities.
DEVELOPING DATA SYSTEMS TO IDENTIFY AT-RISK STUDENTS
The studies discussed above suggest that a number of routinely collected variables can be used to identify students at risk of dropping out. Early intervention is likely to be key to reducing dropout rates, and these findings provide guidance on ways to identify at-risk students as early as middle school. Building data systems that incorporate the necessary information can facilitate early identification and intervention.
The research findings imply that early warning systems need to be able to capture, at a minimum, students’ course grades and attendance records, beginning as early as grade 6, as well as credit hours earned for ninth graders. In addition, behavior measures and basic demographic, test score, and status variables (special education, ELL, etc.) are likely to be useful. Together, the findings suggest that data should be captured in its “rawest” form so that districts and states can conduct their own studies to determine the best predictors in their school systems. For instance, Balfanz advised recording actual course grades, not simply an indicator of course failures, because research is not entirely conclusive on the grade threshold that is predictive of dropping out. In some settings, D’s may be as predictive as course failures (Rumberger and Arellano, 2007). Capturing both semester and final grades and noting whether the course is a core academic course are important, since a low semester grade in a core course can be the earliest sign of becoming off-track. Likewise, Balfanz advised that attendance data be recorded in terms of the number of days attended, not an overall percentage of days attended (without providing information on days enrolled), because the research was not conclusive about the absolute number of absences that was predictive of dropping out. At the high school level, absences should be recorded at the course level, because students may cut a particular course but not miss the entire day of school
A number of states and districts (such as Albuquerque, Dallas, Omaha, and Prince George’s County, Maryland) have begun working on early indicator systems, conducting their own research to identify the variables predictive of
dropping out in their jurisdictions.3 At the workshop, Bill Smith, with the Sioux Falls School District in South Dakota, made a presentation on research that has been conducted in his district. We include this as an example of ways in which school systems can design studies to identify early risk factors, determine the variables that need to be incorporated in early indicator data systems, evaluate current policy, and inform decision making about interventions.
An Example of an Early Indicator System
In Sioux Falls, officials were aware of the findings about precursors of dropping out from research on students in Philadelphia and Chicago. Drawing from these results, district researchers designed several studies of students who dropped out of the Sioux Falls School District. The researchers identified three categories of risk factors for their students:
Academic: a semester grade of F in two or more classes or dismissal from an Individualized Education Plan at the middle school level.
Transition: moving into the district in grade 5 or later and multiple moves to schools into and/or out of the district.
Attendance: having more than 10 absences in a year.
The school system then worked to develop interventions for students with these risk factors and now has three kinds of programs in place. “Universal interventions” are implemented for all students. “Targeted interventions” focus on smaller group of students (10-20 percent) who may need some additional support to remain connected and successful in school. “Individualized interventions” are for those students (roughly 5 to 10 percent) who need intensive support in order to stay in school and be successful. The interventions used by the school system are displayed in Box 5-1.
Smith also recounted a story of how this research uncovered unintended negative consequences of a long-standing district policy. A portion of their research focused on the deleterious effects of absenteeism, and these studies revealed a statistically significant negative correlation between academic performance and days absent from school. Smith said that the data indicated that students establish patterns of missing school as early as grades 2 and 3. As he put it, “students begin dropping out one day at a time.” Missing 10 days a year appeared to be the threshold after which academic performance steadily declined as the number of absences increased; the more absences, the more likely a student was to have a GPA below 3.0.
As part of this study, the researchers examined reasons for absences and
discovered that a school policy was actually contributing to absenteeism. At the time, school policy called for high school students with excessive absences to be reprimanded with out-of-school suspensions. Two-thirds of the out-of-school suspensions were for this reason. When they realized the unintended impact of this policy, district officials revised it. Current policy now requires students with excessive absences to spend time before and after school making up the missed assignments. (For additional details about the Sioux Falls studies, see National Forum on Education Statistics, 2009.)
Dropping out is a process that begins well before a student actually leaves school. Research has identified early warning signs of dropping out, such as poor grades, frequent absences, being over age for the grade, low achievement, and frequent transfers from school to school. Building data systems to accommodate these indicators is fundamental to developing systematic prevention efforts.
Although the precursor variables discussed in this chapter are useful in identifying at-risk students, they are not perfect predictors. Balfanz and colleagues used a high-yield rule to select precursor variables. Although this rule identified students who would almost certainly not graduate without intervention, it did not identify all nongraduates: approximately 41 percent of the eventual dropouts were not identified by any of these indicators. Thus, it is important that precursor variables be only one part of a comprehensive effort to identify at-risk students and target interventions. Multiple levels of interventions may be needed, as the example of Sioux Falls shows. Developing a variety of interventions that can be targeted to different audiences—some that are universal interventions that can be cost-effectively provided to all students and some that are triggered by certain factors (e.g., a sudden pattern of absences, receiving a low semester grade in a core course)—may be the best way for schools to focus their efforts to reduce dropout rates and increase the number of graduates.
Other factors, such as parental practices, have been less widely studied but may be amendable to interventions through, for instance, parent training programs. Similarly, some school factors, such as the demographic composition of the students in a school, have been shown to exert a powerful influence on student outcomes, but can be altered only through policies that directly address school segregation (Orfield and Lee, 2006; Rumberger and Palardy, 2005). Other school factors, such as those related to school policies and practices affecting potential dropouts, could be more directly altered through policy.
Although the studies discussed in this chapter provide a general range of the kinds of variables to be considered in these systems, there is clearly benefit to locally designed research. As Balfanz’s work demonstrates, the variables (and
Student Engagement Action Plan: Sioux Falls School District
Academic (Semester grade of F in two or more classes/Dismissal from an Individualized Education Plan at the middle school level)
Transition (Move into the District in grade 5 or higher/Multiple moves to schools in and/or out of the district)
Attendance (More than 10 absences)
The “Student Engagement Case Manager” program is modeled after the “Check and Connect” program developed at the University of Minnesota, which has had
levels of those variables) predictive of dropping out in one school district may not be equally predictive in another. We encourage states and local school districts to conduct studies to determine the extent to which the research findings apply to their students. We therefore recommend:
RECOMMENDATION 5-1: States and districts should build data systems that incorporate variables that are documented early indicators of students
nationally recognized success in all three areas: academics, transitions, and attendance. The strategies used and record-keeping system developed for the program will be implemented by the student engagement case manager, as well as by the Title I, Part D Success Coordinators at Axtell Park Middle School and Whittier Middle School. It is recommended that the Behavior Facilitators in each of the elementary Title I buildings incorporates the “Check and Connect” strategies and record-keeping into the current program. All data will be kept weekly and monthly to monitor effectiveness. These data will be analyzed for program effectiveness and future staffing needs.
Committee Participation: Committee involvement included four teachers, two school counselors, a school social worker, three elementary principals, two middle school principals, four curriculum services and special services administrators, and two instructional support services administrators.
Administrative Recommendation to the School Board: Acknowledge review of the Elementary and Middle School At-Risk Intervention Committee Report.
SOURCE: Box 5-1 was reprinted with permission from Sioux Falls School District 49-5, Copyright 2008.
at risk for dropping out, such as days absent, semester and course grades, credit hours accrued, and indicators of behavior problems. They should use these variables to develop user-friendly systems for monitoring students’ risk of dropping out and for supporting them based on their level of risk.
An important implication of this recommendation is that the interface for the data systems should be exceptionally user-friendly, enabling teachers and
administrators to access information that will be useful to them in the course of usual educational practice.
We also note that it will be important for states and districts to evaluate the impact of any policy interventions that are implemented to determine their effectiveness and to consider any unintended consequences associated with the policies. The studies we discussed in this chapter identified precursors that were related to dropping out, but none of these variables were prefect predictors. Thus, it is important that any policy measures that are implemented achieve the appropriate balance between over-identifying students at risk and under-identifying students who might be at risk for dropping out. The problems associated with under-identifying are clear—students who need an intervention are missed. However, problems can also result from targeting students for intervention who are not in need of it, such as by over responding when a student misses a few days of school. Such a policy can be counter-productive. Careful evaluations of policies and programs can help to ensure that they are effective and appropriately target the students most in need of intervention.