5

Student Achievement Under PERAA: First Impressions

To ask how well the schools are doing under the Public Education Reform Amendment Act (PERAA) is to ask countless specific—and often complicated—questions, which is why a thorough, 5-year evaluation is called for in the law. That evaluation may be even more important than originally envisioned because—even in the short time that this committee has been developing the evaluation plan—there has been complete turnover in the primary leadership positions for education in the District. There is a new mayor, deputy mayor, chancellor, and interim state superintendent of schools. Yet, 3 years after PERAA was enacted and after many significant changes have been implemented, it is not unreasonable to consider what has happened, what we call first impressions.

In this and the next chapter, we present our first impressions on several goals of the legislation: this chapter considers student achievement data. Chapter 6 looks at the other aspects of the system that also must be measured: the quality of district staff, the quality of classroom teaching and learning, service to vulnerable children and youth, family and community engagement, and operations.

For the purposes of this first phase of the evaluation effort, the committee was able to collect only preliminary information about student achievement and the five primary areas of district responsibility we discuss in Chapter 6. We stress that these first impressions are useful only as a basis for further inquiry and not as reliable evidence about the effectiveness of the changes under PERAA or how best to fine-tune programs and strategies in the future. Those tasks require an ongoing program of evaluation and research, which we offer in Chapter 7.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 63
5 Student Achievement Under PERAA: First Impressions To ask how well the schools are doing under the Public Education Reform Amendment Act (PERAA) is to ask countless specific—and often complicated—questions, which is why a thorough, 5-year evaluation is called for in the law. That evaluation may be even more important than originally envisioned because—even in the short time that this committee has been developing the evaluation plan—there has been complete turnover in the primary leadership positions for education in the District. There is a new mayor, deputy mayor, chancellor, and interim state superintendent of schools. Yet, 3 years after PERAA was enacted and after many significant changes have been implemented, it is not unreasonable to consider what has happened, what we call first impressions. In this and the next chapter, we present our first impressions on several goals of the legislation: this chapter considers student achievement data. Chapter 6 looks at the other aspects of the system that also must be mea- sured: the quality of district staff, the quality of classroom teaching and learning, service to vulnerable children and youth, family and community engagement, and operations. For the purposes of this first phase of the evaluation effort, the com- mittee was able to collect only preliminary information about student achievement and the five primary areas of district responsibility we discuss in Chapter 6. We stress that these first impressions are useful only as a basis for further inquiry and not as reliable evidence about the effectiveness of the changes under PERAA or how best to fine-tune programs and strategies in the future. Those tasks require an ongoing program of evaluation and research, which we offer in Chapter 7. 63

OCR for page 63
64 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS STUDENT ACHIEVEMENT AND TEST DATA The most readily available first impressions of student achievement are provided by test scores. There is a long history of relying on student test data as a measure of the effectiveness of public education, and it is tempting to simply rely on those readily available data for judgments about student achievement and about causes and effects. However, student test scores alone provide useful but limited information about the causes of improve- ments or variability in student performance. The results of achievement tests provide only estimates about students’ skills and knowledge in selected areas—usually, what they know and can do in mathematics and reading and sometimes other subjects. Aggregate year-to-year comparisons of test scores in the District’s schools are con- founded by changes in student populations that result from student moves in and out of the city and between DC Public Schools (DCPS) and charter schools, dropout and reentry, and also from variations in testing practices that may exclude or include particular groups of students.1 For these and other reasons, therefore, it is important to remember that the consensus of measurement and testing experts has long been to use test scores cautiously. For this discussion, it is perhaps most important to underscore that most tests are not designed to support inferences about related questions, such as how well students were taught, what effects their teachers had on their learning, why students in some schools or classrooms succeed while those in similar schools and classrooms do not, whether conditions in the schools have improved as a result of a policy change, or what policy makers should do to solidify gains or reverse declines. Answering those sorts of questions requires other kinds of evidence besides test scores. Looking at test scores should be only a first step—not an end point—in considering questions about student achievement, or even more broadly, about student learning. Nevertheless, changes in student test scores since 2007 provide one set of impressions regarding progress in DC schools. We offer here an overview of publicly available data from both the District of Columbia Comprehen- sive Assessment System (DC CAS) and the U.S. Department of Education’s National Assessment of Educational Progress (NAEP). We first discuss these data sources, then look at the trend data, and end the chapter with a discus- sion of how to interpret the data. But we note again that a systematic and comprehensive analysis of achievement data for DC was beyond the scope of this report; the readily available information provides only a useful first 1 Test scores also come with measurement issues that have to be considered if they are to provide an accurate picture of even those areas they do measure (Koretz, 2008; National Research Council, 1999; Office of Technology Assessment, 1992).

OCR for page 63
65 STUDENT ACHIEVEMENT UNDER PERAA look and hints about issues related to student achievement that will need to be addressed in the long-term evaluation. THE DATA SOURCES The District’s assessment system, the DC CAS, assesses students in grades 3 through 8 in reading and mathematics and in selected grades in science and composition.2 The assessment system, which has been in place since 2006, is designed to measure individual students’ progress toward meeting the District of Columbia’s standards3 and is used to meet federal requirements under the Elementary and Secondary Education Act, as amended by the No Child Left Behind (NCLB) Act (District of Columbia Public Schools, 2010a).4 DC CAS scores are used to determine if a given school is making sufficient progress under NCLB, and the media and the public look to them for an indication of how well district schools are doing. DC CAS results are reported using four performance levels: advanced, proficient, basic, and below basic. Box 5-1 provides an example of the performance descriptions used in DC CAS. The NAEP, known popularly as the Nation’s Report Card, is an assess- ment administered by the U.S. Department of Education and overseen by the autonomous National Assessment Governing Board that provides independent data about what students know and can do in mathematics, reading, and other subjects. NAEP is valuable in part because it is not a high-stakes test—scores for individual students or schools are not reported, and there are no consequences to students, teachers, or schools associated with NAEP scores. Results are reported for states, selected urban districts, and the nation, and all students are measured against common performance expectations; consequently, the results can be used to make comparisons among jurisdictions.5 Changes to the assessment are infrequent and come with careful studies of comparability, so NAEP is also used to track student 2 Information about DC-CAS can be found at: http://www.dc.gov/DCPS/In+the+Classroom/ How+Students+Are+Assessed/Assessments/DC+Comprehensive+Assessment+System-Alternate +Assessment+Portfolio+(DC+CAS-Alt) [accessed October 2010]. 3 Information about the academic standards can be found at: http://osse.dc.gov/seo/cwp/ view,A,1274,Q,561249,seoNav,%7C31193%7C.asp. According to the 2011 DC CAS guide (Office of the State Superintendent of Education, 2010b), the assessments in reading and mathematics are aligned to both the District of Columbia standards and to the Common Core Standards, a set of standards that the majority of states have recently adopted to ensure greater consistency in public education from state to state (http://www.corestandards.org/ [accessed December 2010]). 4 States must meet the requirements of the No Child Left Behind Act in order to receive federal financial assistance to support the education of poor children. 5 The comparisons are subject to some caveats related to such issues as inclusion rates for students with disabilities and English language learners.

OCR for page 63
66 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS BOX 5-1 DC CAS Performance Descriptions for 3rd Grade Reading The DC CAS is a standards-based assessment. Based on performance, each student is classified as performing at one of four performance levels: below basic, basic, proficient, and advanced. The descriptions below are examples of perfor- mance descriptions for each level. Below Basic Students are able to use vocabulary skills, such as identifying literal or com- mon meanings of words and phrases, sometimes using context clues. Students are able to read some 3rd grade informational and literary texts and can identify a main idea, make some meaning of text features and graphics, form questions, locate text details, and identify simple relationships (e.g., cause/effect) in texts. Basic Students are able to use vocabulary skills, such as identifying words with prefixes and suffixes, and distinguishing between literal and nonliteral meanings of some common words and phrases. Students are able to read some 3rd grade informational and literary texts and can identify main points and some supporting facts, locate stated facts and specific information in graphics, form questions, iden- tify lessons in a text, make simple connections within and between texts, describe and compare characters, and make simple interpretations. Proficient Students are able to use vocabulary skills, such as identifying affixes and root words and using context clues to interpret nonliteral words and meanings of nknown words. Students are able to read 3rd grade informational and liter- u ary texts and can distinguish between stated and implied facts and cause/effect relationships, determine and synthesize steps in a process, connect procedures to real-life situations, explain key ideas in stories, explain relationships among char- acters, identify subtle personality traits of characters, and connect story details to prior knowledge. Advanced Students are able to use vocabulary skills, such as identifying the figurative meanings or nonliteral meanings of some words and phrases in a moderately complex text. Students are able to read 3rd grade informational and literary texts and summarize the information or story with supporting details, apply text infor- mation to graphics, identify and explain relationships of facts and cause/ ffect e relationships, use text features to make predictions, distinguish between fact and fiction, identify a speaker in a poem or narrator in a story, explain key ideas with supporting details, use context to interpret simple figurative language, and deter- mine simple patterns in poetry. SOURCE: Office of the State Superintendent of Education (2010a, p. 1).

OCR for page 63
67 STUDENT ACHIEVEMENT UNDER PERAA performance over time. For example, the NAEP mathematics assessment scores go back to 1992, and those for reading to 1990. DC has participated in NAEP as a “state” since the early 1990s, and the scores for grades 4 and 8 reflect the performance of all District public schools, including all public charter schools. When NAEP began the Trial Urban District Assessment (TUDA) in 2002, the District was included in this assessment as well, one of only five such districts. In 2009, 18 districts participated. Until 2009, the scores for the District were the same for both the state and district assessments. Beginning in 2009 most charter schools were excluded from the District’s TUDA results, but they remained in the state score calculation. The charter schools that are excluded from DCPS’s adequate yearly progress (AYP) report under NCLB were also excluded from the NAEP TUDA. This look at the data presents both state and TUDA scores together. The state scores include all DC schools and therefore serve as the basis for comparison. All NAEP data in this section refer to the state DC scores unless otherwise noted. The TUDA scores are presented in graphics for completeness, but are not discussed and should be evaluated cautiously, particularly in the context of comparisons between 2007 and 2009 because of the change in charter school exclusion in 2009. These two assessments provide different ways of measuring student progress. DC CAS provides evidence of the progress of individual students and groups (such as 3rd graders in a school or in the district) toward mastering specific objectives in the DC standards. NAEP provides a picture of what students at each grade in the District as a whole know and can do in terms of nationwide definitions of achievement in each subject. TEST SCORE TRENDS The percentage of tested students who performed at or above the pro- ficient level (proficiency rate) in all grades in the District on the DC CAS increased from 2006 to 2010. Figure 5-1 shows the upward trend prior to PERAA’s passage in 2007. After 2007, the trend in both reading and mathematics increased more steeply for 2 years, then flattened out, and then declined slightly in the 2009-2010 school year. Figure 5-2 shows the percentages of students (by grade and subject) performing at each of the four proficiency levels on DC CAS and state NAEP for 2007 and 2009. These data show that, in general, the percent- ages of students in both the below basic and basic categories decreased for both assessments, while the percentages of students performing at both the proficient and advanced levels increased. That is, the distribution of students shifted to higher performance levels from 2007 to 2009. Thus, the NAEP scores appear to confirm the improvement shown on the DC CAS scores. However, the percentages of students performing at the proficient

OCR for page 63
68 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS 50 Percentage Above Proficient 40 Reading 30 Mathematics 20 10 0 2006 2007 2008 2009 2010 Year FIGURE 5-1 Percentage of District students at or above the proficient level on the DC CAS in reading and mathematics, 2006-2010. SOURCE: Adapted from http://www.nclb.osse.dc.gov/index.asp [accessed December Fig 5-1.eps 2010]. level or above on NAEP is significantly smaller than the percentage who perform at those levels on DC CAS—a finding that suggests that DC CAS is a less challenging assessment than NAEP (we discuss this issue further below). The average (scaled) score on NAEP shows a similar positive trend. The 2009 NAEP scores for DC as a state (see Figure 5-3) in grade 4 for both reading and mathematics were statistically significantly higher than they had been in all previous years (2003, 2005, and 2007). This was also true for grade 8 mathematics. In grade 8 reading, the DC state scores in 2009 were also significantly higher than those for 2003 and 2005, but not than those for 2007. That is, grade 8 reading was the only assessment in which the District did not show a significant gain from 2007 to 2009. In comparison with states, the District’s scores were notable. In grade 4 reading, only two other states improved since 2007; in math- ematics, only four other states showed significant improvement at both grades 4 and 8. Only three states, Kentucky, Rhode Island, and Vermont, showed improvement in three of the four assessments, and no state im- proved in all four. However, in comparison with other urban districts, the District’s scores were similar: many others also showed consistently significant gains.

OCR for page 63
69 STUDENT ACHIEVEMENT UNDER PERAA FIGURE 5-2 Proficiency levels of District students from DC CAS and NAEP for Fig 5-2.eps 2007 and 2009 in reading and mathematics. SOURCES: National Center for Education Statistics, NAEP Data Explorer, see 2 bitmaps http://nces.ed.gov/nationsreportcard/naepdata/ [accessed September 2010]; DC CAS, see http://www.nclb.osse.dc.gov/index.asp [accessed March 2011].

OCR for page 63
70 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS Grade 4 Mathematics 250 Charlotte Austin 240 New York Houston 230 Boston Average Scores Average Scores Atlanta San Diego Los Angeles 220 DC - TUDA DC - State Chicago Cleveland 210 200 190 180 2003 2005 2007 2009 Year Grade 4 Reading 250 240 230 Average Scores Average Scores Charlotte Austin 220 New York San Diego 210 Boston DC - TUDA Houston Atlanta DC - State 200 Chicago Cleveland Los Angeles 190 180 2003 2005 2007 2009 2002 Year FIGURE 5-3 NAEP TUDA average scores for 10 urban districts and DC, as well as DC state NAEP, for mathematics and reading, grades 4 and 8, 2002-2009. NOTES: State and TUDA scores are presented together for completeness. State scores include all DC schools and thus are the focus of this analysis. TUDA scores should be evaluated cautiously particularly when comparing 2007 to 2009 because most charter schools were excluded in 2009 (but included in 2007—see text). Fig 5-3.ep SOURCE: National Center for Education Statistics, NAEP Data Explorer, see http://nces.ed.gov/nationsreportcard/naepdata/ [accessed September 2010]. landscap (lower right--New York doesn’t seem

OCR for page 63
71 STUDENT ACHIEVEMENT UNDER PERAA Grade 8 Mathematics 300 290 Austin Charlotte 280 San Diego Houston Boston Average Scores 270 New York 260 Atlanta Chicago DC - State Cleveland 250 DC - TUDA Los Angeles 240 230 220 2003 2005 2007 2009 Year Grade 8 Reading 300 290 280 Average Scores 270 Charlotte New York 260 Austin San Diego Boston Chicago 250 Atlanta e Houston DC - State 240 DC - TUDA Cleveland Los Angeles 230 220 2002 2003 2005 2007 2009 Year FIGURE 5-3 Continued. Fig 5-3.eps landscape sn’t seem to have any plotted points)

OCR for page 63
72 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS Figure 5-3 also shows trends for other school districts assessed by NAEP in mathematics and reading.6 Two points are worth noting: the District’s average scores are low compared with those of most of the other 10 school districts in both the 2007 and 2009 TUDA (including Boston, Chicago, and New York), but DC and its peer districts are improving at similar rates. Most districts showed gains from 2007 to 2009: see Figure 5-4.7,8 It is important to note, however, that scores that are averaged across large numbers of students can obscure which students are improving and by how much. It may be that only a small group of students is making gains while others are not improving or may even be doing worse than previ- ously. For example, the highest achievers may be showing gains while the lowest achievers are not, or vice versa. The committee was limited by time and resources in the number of disaggregations we could carry out for this report, but a few examples demonstrate the importance of looking beyond average scores. It appears that students at every level in the District are gaining ground. As Figure 5-5 shows, for example, in DC state NAEP grade 4 reading, students in the lowest, middle, and highest groups all made gains, with the lowest scoring students gaining at a faster rate than the others. We note, too, that black, Hispanic and white 4th graders on average scored higher on the DC CAS mathematics in 2010 than in 2007, while English language learners and students with disabilities also showed some improvements relative to their peers: see Figure 5-6. The NAEP data show different results. For grade 4 reading, there was no significant change in the performance of white 4th grade students in the District from 2005 to 2009, while scores for both black and Hispanic 4th graders showed a significant gain for 2009: see Figure 5-7. For grade 8 reading, the NAEP data show large achievement gaps when scores are bro- ken out by the educational attainment of students’ mothers: see Figure 5-8. These data show improvements for those students whose mothers did not finish high school. 6 Although 18 urban districts participated in the 2009 mathematics and reading assessments, only 10 districts other than DC also participated in assessments in previous years, so it is only possible to examine changes over time for those 10. 7 These data findings come from the test of differences in gains performed through NAEP Data Explorer, see http://nces.ed.gov/nationsreportcard/naepdata/ [accessed September 2010]. 8 Although the DC state scores are the focus of this analysis because they reflect all public schools, the TUDA scores are also presented in Figure 5-3. It should be noted that if the non- DCPS charter schools had been excluded in 2007 as they were in 2009 (i.e., if NAEP had used comparable samples in both years), the District would have also shown a statistically signifi- cant increase from 244 in 2007 to 251 in 2009 in grade 8 mathematics, rather than the non- significant change from 248 to 251: see “comparability of samples” at http://nationsreportcard. gov/math_2009/about_math.asp [accessed December 2010].

OCR for page 63
Change in NAEP Scores 2007-2009 Grade 4 Reading Grade 8 Reading Grade 4 Mathema cs Grade 8 Mathema cs 8 6 6 6 5 5 5 5 5 4 4 4 4 4 4 4 3 333 3 3 3 3 3 2 2 2 2 2 1 1 1 1 1 11 1 0 0 -1 -1 -1 -1 -1 ATLANTA AUSTIN BOSTON CHARLOTTE CHICAGO CLEVELAND DC TUDA DC STATE HOUSTON LOS NEW YORK SAN DIEGO -2 ANGELES -4 -4 FIGURE 5-4 Changes in NAEP scores for selected urban districts, 2007-2009. Numbers indicate the amount of the increase or decrease in the average scaled score. NOTES: State and TUDA scores are presented together for completeness. State scores include all DC schools and thus are the focus of this analysis. TUDA scores should be evaluated cautiously particularly when comparing 2007 to 2009 because most charter schools were excluded in 2009 (but included in 2007—see text). SOURCE: National Center for Education Statistics, NAEP Data Explorer, see http://nces.ed.gov/nationsreportcard/naepdata/ redrawn new Figure 5-4 73 [accessed September 2010].

OCR for page 63
78 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS (see, e.g., Linn, 2000). One hypothesis as to the reason for this pattern is that as teachers and students gradually become accustomed to the new test format and new expectations, student performance improves, but that once the test is familiar, performance stays flat (Koretz et al., 1991). Additional evidence would be needed to show whether this phenomenon might explain the observed changes in DC. In short, the DC CAS scores did rise, but there is insufficient evidence to establish the reason for the improvement. In the case of the District, the fact that NAEP shows increases similar to those seen on the DC CAS suggests that the new-test phenomenon may not be the primary explanation; however, other changes that occurred in the same period could be responsible. Demographic shifts—changes in the composition of the student population that occur when students leave or enter the system (which will also change the groups of students being compared from year to year) are another potential source of change in test scores. Since the tests compared cohorts of students, scores will be affected if the populations are not similar. For example, if more higher-scoring 4th graders move into (or opt not to leave) the district’s public schools from one year to the next, average scores would likely rise—but that rise would not reflect improved learning. Such changes could occur because of in- or out- migration from the city or transfers between public and charter schools. If there are only small differences in the composition of students being tested across years, the effect would be slight. However, if, substantially more or fewer students in one year came from families of low socioeconomic status than in the next year, test results might show substantial changes that have nothing to do with the quality of instruction in schools or improved student learning. This is a serious issue in DC, which has a highly mobile stu- dent population, where many students move into and out of the charter school system, and which has a history in which the most disadvantaged residents have sometimes been forced by changing political and economic forces to move within the city or into neighboring jurisdictions. This issue is not just theoretical. The composition of students in tested grades in the District of Columbia’s public schools, has changed markedly since 2007 (see Table 5-1).11 The number of students in all tested grades in DCPS has dropped by almost 21 percent, while the number of tested stu- dents in the charters has increased.12 However this decrease within the DCPS has not been consistent across demographic groups; in contrast the subgroup composition of students attending public charter schools in the district has remained relatively stable over this same time period—see Table 5-1. 11 Table5-1 was revised after the prepublication report was released; data are now presented separately for DCPS and charter schools (previously the combined data were presented). 12 Discussion in this paragraph relies on data about students enrolled in the tested grades of 3-8 and 10 only and not to all students. See Table 5-1.

OCR for page 63
79 STUDENT ACHIEVEMENT UNDER PERAA For example, the enrollment of students who were not economically dis- advantaged fell considerably in DCPS between 2007 and 2010 while the enrollment of economically disadvantaged students also declined but not at the same rate. This means that economically disadvantaged students now make up a larger proportion of the total population of DCPS students— an increase of 8.2 percentage points; economically disadvantaged students were 62.2 percent of the DCPS tested population in 2007 and 70.4 percent in 2010. A similar pattern can be found for black students whose overall numbers fell in DCPS while those of whites and Hispanics increased slightly resulting in a shift in the overall demographic composition of the DCPS student body. The effects of families leaving the district or returning to the district are not generally factored in to summary proficiency statistics, yet these patterns could significantly bias the summary statistics (including co- hort averages) either up or down. As we discussed in Chapter 3, the District has witnessed changes in movement between DCPS and charter schools and in the composition of particular neighborhoods (as well as tensions regard- ing school closures and school improvements) that are likely to affect local school student populations; consequently, this issue should be carefully considered when interpreting changes in student achievement data. Dropout rates raise similar concerns. As students drop out of schools, their test scores are no longer included in their schools’ data. Thus, those schools’ average test scores may improve if significant numbers of low- achieving students leave, even if the remaining students’ scores have not gone up and the school has not actually improved. This is an important consideration in assessing DC’s test scores because a recent report from the National Center for Education Statistics found that the rate of students who enter 9th grade and later graduate from a DC school has steadily declined, from 68 percent in the 2001-2002 school year to only 56 percent in the 2007-2008 school year. The validity of data on dropout rates is, in itself, an issue of serious concern in interpreting achievement data (see, e.g., National Research Council and National Academy of Education, 2011). For all of these reasons, reports of test score gains are complete and valid only when they include analysis of the demography of the student population—including examinations of the distribution of students by geo- graphic area (e.g., ward) and movement into and out of charter schools, pri- vate schools, and suburban school districts. One means of factoring out the effects of population changes is to track individual students in the system over time to determine whether their performance is on an upward trajectory, that is, to follow actual cohorts of students across time. Doing so makes it pos- sible to see the performance of the students who remain in the system without any distortion that could come from changes in demographic composition. Thus, it is important to complement the average scaled scores and demo- graphic analyses with assessments of individual student growth over time.

OCR for page 63
80 TABLE 5-1 Changes in Demographic Subgroups Enrolled in Tested Grades for DCPS and Charter Schools, 2007-2010 DCPS Non Non- Non- Year Econ Dis Econ Dis Black Hispanic Whitea LEP LEP SPED SPED Total 2007 subgroup% 62.2 37.8 83.5 9.5 5.3 8.2 91.8 21.8 78.2 # 16,283 9,916 21,881 2,487 1,376 2,185 24,422b 5,707 20,492 26,199 2008 % 64.2 35.8 83.7 10.1 5.7 8.4 91.6 19.7 80.3 # 15,125 8,440b 19,463 2,353 1,331 1,993 21,872b 4,638 18,927b 23,259 2009 % 68.5 31.5 80.0 11.2 6.9 11.6 88.4 20.8 79.2 # 14,631 6,738 17,091 2,389 1,473 2,470 18,899 4,446 16,923 21,369 2010 % 70.4 29.6 78.1 12.1 7.8 8.0 92.0 21.0 79.0 # 14,587 6,140 16,181 2,518 1,610 1,663 19,064 4,352 16,375 20,727 Percentage Point Change in Subgroup 8.2 –8.2 –5.5 2.6 2.5 –0.2 0.2 –0.8 0.8 Composition (2010%-2007%) Percentage Change –10.4 –38.1 –26.0 1.2 17.0 –23.9 –21.9 –23.7 –20.1 –20.9 in Number Enrolled (2010#-2007#/2007#)

OCR for page 63
Public Charter Schools in DC Non Non- Non- Year Econ Dis Econ Dis Black Hispanic Whitea LEP LEP SPED SPED Total 2007 subgroup% 66.3 33.7 90.4 6.6 2.2 4.3 95.7 12.8 87.2 # 5,971 3,037 8,140 599 194 390 8,690b 1,155 7,853 9,008 2008 % 70.0 30.0 86.9 6.9 2.4 5.2 94.8 13.7 86.3 # 6,718 2,876 8,608 687 241 501 9,184 1,310 8,284 9,900 2009 % 72.0 28.0 89.6 7.6 2.2 6.0 94.0 13.4 86.6 # 8,174 3,183 10,173 859 251 677 10,680 1,519 9,838 11,357 2010 % 68.3 31.7 88.7 8.2 2.5 4.7 95.3 12.5 87.5 # 7,962 3,698 10,339 952 297 550 11,110 1,452 10,208 11,660 Percentage Point Change in Subgroup 2.0 –2.0 –1.7 1.5 0.4 0.4 –0.4 –0.4 0.4 Composition (2010%-2007%) Percentage Change 33.3 21.8 27.0 58.9 53.1 41.0 27.8 25.7 30.0 29.4 in Number Enrolled (2010#-2007#/2007#) NOTES: Tested grades are 3-8 and 10. Econ Dis = economically disadvantaged, LEP = limited English proficient, SPED = special education. aPercentages (across black, Hispanic, white) do not sum to 100 because two subgroups (with very low numbers) are not shown. bThe total across these two subgroups is greater than the total number of students reported for that year (see “Total” column at far right). Data are presented as they appear on the OSSE website; we were unable to determine the reason for the discrepancy. For these cases, percentages were calculated based on the sum across subgroups (not the total number from the far right column). SOURCE: Compiled from http://www.nclb.osse.dc.gov/index.asp [accessed April 2011]. 81

OCR for page 63
82 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS Looking Beyond Proficiency Rates The primary data point reported for DC CAS (as for many assessment programs) is the proficiency rate, the percentage of students who perform at or above the proficient level. However, using proficiency rates has more significant limitations than using measures that more accurately reflect the spread of scores, such as averages. One limitation is that states have widely varying definitions of proficiency in core subjects. For example, a study for the U.S. Department of Education (Bandeira de Mello et al., 2009) found that the difference between the most and least challenging state standards for proficient performance in reading and mathematics was as large as the difference between the basic and proficient performance levels on NAEP. This study did not include the District because data were not available, but it is possible to compare the percentage of students at or above proficient on NAEP to that of DC CAS during the same year: see Figure 5-2, above. The reasons for the differences in the tests may be that the DC CAS is more closely aligned to the District’s—not NAEP’s—standards and therefore mea- sures different things. It is also possible that the District, like many other states, has a lower bar for proficiency than does NAEP. Another limitation to consider about data on the percentage of stu- dents performing at or above the proficient level is that this figure provides no information about students who are performing significantly above or below that level. Thus, this measure cannot reveal change that occurs at all other points on the scale—such as students who move from below basic to basic or from proficient to advanced. If a school or the district as a whole has focused on helping the students who are performing just below the proficiency cutoff point to cross that cutoff (sometimes called bubble kids), other students might receive less attention (Booher-Jennings, 2005; Neal and Schanzenbach, 2007). Another and perhaps most important limitation is that the percent proficient statistic does not account for the weight (relative numbers of students) around the proficiency cut scores, and the fact that a slightly different choice in cut score may even reverse trends (Ho, 2008). Using proficiency scores to assess gains and gaps leads to “unrepresentative depic- tions of large-scale test score trends, gaps, and gap trends” and “incorrect or incomplete inferences about distributional change” (Ho, 2008, p. 1). Because of this limitation, analysts recommend statistics or summaries that accurately reflect the performance of all students, such as the average scaled scores and the distribution of these scores (Ho, 2008).

OCR for page 63
83 STUDENT ACHIEVEMENT UNDER PERAA Disaggregating Test Results Although average scores provide a measure of whole group perfor- mance, the average may mask important subgroup differences. For example, it is possible for the overall average to be increasing while some subgroup scores are decreasing. Alternatively, the average may not show a change, even though some subgroups’ scores are significantly increasing. Thus, dis- aggregating results is essential to understanding of score trends. A thorough evaluation of test scores in the District would examine how achievement has been changing across a number of student groups, considering: • grade level, • subject (and, in some cases, strands), • types of schools (e.g., charter or traditional), • student achievement levels (e.g., 10th, 25th, 50th, 75th, 90th percentiles), • geography (e.g., in the District, ward), • ethnicity, • income level, and • special populations, such as students with disabilities and English language learners. We also note that policies that change the standards for classifying English language learners have potentially significant effects on the charac- teristics of the whole population, and, therefore, on average performance. Students who move into the proficient category, for example, are often automatically reclassified as non-English language learners (even though they may not have attained complete fluency) and, thus, are no longer counted in the subgroup. In this situation, overall scores would appear to decrease simply because the composition of the tested group changed. Disaggregating data is complicated for DC because the city’s black population is large in comparison with that of many other school districts. Significant demographic differences within the city, including differences in levels of income and education, may therefore be obscured in analyses of achievement by racial group. DC’s unique population demographics make the black-white achievement gap less informative than comparisons within the demographic groups in the District and surrounding areas. Although there is little argument about the importance of striving to eliminate long-standing achievement gaps, it would be misleading to focus on such aggregate gaps within the District population as was done, for example, in the 2008-2009 progress report of the DC Public Schools (Dis- trict of Columbia Public Schools, 2009). The District’s black population

OCR for page 63
84 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS is very diverse, and includes both a concentration of very highly educated and successful black residents and many who are poorly educated and economically insecure. Socioeconomic differences are especially large be- tween the northwest and southeast areas of the city, whose populations are dominated, respectively, by well-off whites and poor blacks. For example, recently released data from the American Community Survey—aggregated from 2005 to 2009—show that in northwest Washington more than 80 percent of adults have at least a bachelor’s degree and more than 50 per- cent have at least a master’s degree, while in southeast Washington fewer than 10 percent have a bachelor’s degree. And in most areas of northwest Washington, the median household income is well over $100,000 per year, while in southeast Washington, the median household income is well under $50,000 (U.S. Census Bureau, 2010). It is highly misleading to compare academic achievement between populations of such different social and economic standing. Even in the absence of improved measures of individual students’ socioeconomic status (discussed below), when the new common core standards and common assessments become available, it should at least be possible to compare academic performance levels of white, black, and Hispanic students in the District with those in other, comparable student populations. In the mean- time, naïve aggregate comparison of test scores among race-ethnic groups in the District should be interpreted critically and cautiously. Thus, analysts need to carefully consider student backgrounds when comparing average scores, for example, by disaggregating by socioeconomic background. One way that is sometimes proposed to capture socioeconomic differ- ences is to use eligibility for the National School Lunch Program (which provides free or reduced-price lunch for income-eligible students), but re- search suggests that this is not in fact a valid proxy (Harwell and Lebeau, 2010). Students are eligible for the lunch program if their family incomes fall below 125 percent of the official federal poverty guideline (for free lunch) or between 125 percent and 175 percent of the poverty line (for reduced- price lunch). However, the program serves only those students who apply, and not all who are eligible apply. The percentages of students identified as low-income using the NAEP lunch program are lower than the percentages identified by Census Bureau data (Booher-Jennings, 2005). Another diffi- culty with using the lunch program data as a measure comes from changes in policies regarding eligibility. During the past decade, the program has been offered to the entire populations of schools that meet certain criteria, as well as to individual students in any school. Thus, in some cases individual students who do not meet the criteria actually participate in the program. Moreover, the federal definition of the poverty threshold has risen signifi- cantly less than the standard of living since the 1960s, so the official poverty designation has come to refer to a relatively more deprived segment of the

OCR for page 63
85 STUDENT ACHIEVEMENT UNDER PERAA population over time (see National Research Council, 1995). Because of these variations, eligibility for free or reduced-price lunch has limited value as a measure of socioeconomic status. Further research is needed to establish an improved measure of socioeconomic status that will capture differences in the District. We reiterate that DC NAEP results should be disaggregated by socio- economic status, as well as by race and ethnicity, to support meaningful inferences about student learning. Multiple methods should be used to track income level, such as parental education and home ownership status, as reported by parents or other responsible adults. The percentage of students tested (of all students enrolled) for DC CAS and the inclusion rates of English language learners and students with dis- abilities for NAEP are also factors that can affect population scores while masking subgroup scores. For example, if there were a significant decrease in the percentage of students tested, it could significantly affect test scores because the students most likely to be excluded are low-performing ones. For NAEP, state or district policies may differ on the inclusion or exclusion of students with disabilities or English language learners. If larger numbers of these students are excluded in one district or state in comparison with another, the test’s results for that state or district may be inflated. For the District, the percentage of students with disabilities or who are English language learners and were excluded from the NAEP assessments dropped from 2007 to 2009: in mathematics, the exclusion rate declined from 6 to 4 percent in grade 4 and from 10 to 6 percent in grade 8; in reading, the exclusion rate dropped from 14 to 11 percent in grade 4 and from 13 to 12 percent in grade 8. This decrease in the percentage of excluded students provides additional evidence that the assessment gains for District students are real in every NAEP assessment. Comparing Test Results Even individual student-level data will have significant limitations. Tracking students who leave the city is a challenge for the District, which has high rates of mobility to and from neighboring jurisdictions. It is also not generally possible to compare student performance across districts unless they use the same assessments (or ones that have significant overlap; see National Research Council, 2010b, for a discussion of cross-state compari- sons). Since the DC CAS is only administered to students in public schools in the District, it is not possible to assess whether students in DC are “catching up” over time with students outside of the system: one can only track the relative movement of DC students in comparison with one another. The DC State Board of Education voted in 2010 to adopt the common core standards, a set of standards in English language arts and mathematics

OCR for page 63
86 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS that have been developed cooperatively by the states and have been adopted by 40 other states.13 Since these standards are different from the current standards used for the DC CAS, a new set of assessments will be needed to replace the DC CAS. The District currently plans to adopt a new common assessment system that will align with the common core standards; such an assessment system is being developed by a multistate consortium.14 Once the new assessment system is operational, it will be possible to compare the progress of DC students with those in other jurisdictions, and thus to acquire additional evidence regarding changes in student performance since the pas- sage of PERAA. However, switching assessments also has disadvantages. If the DC CAS is not retained in some form for trend purposes, the District will no longer be able to compare current performance with that of the years prior to the implementation of a new assessment. It is possible to do a braided study (in which questions from the old test are nested within the new test) or to use the old test in a sample of schools for a few years to provide some in- formation on trends. Since, as we noted above, performance typically falls in the first year after a new test is introduced and then rapidly improves as teachers and students become familiar with the new format and new stan- dards, it will be important to take that into account in drawing conclusions about the results from a new test (see Koretz et al., 1991). A second issue we note is that assessment scores are part of DCPS’s teacher performance management system. There is considerable debate over pay-for-performance and the reliability of value-added measures; we note here only that attaching direct consequences to student test scores may provide an added incentive for teachers to focus on tested content, at the expense of other important educational goals, or even to cheat by offering students help or information they are not intended to have (see Jacob and Levitt, 2003; Lazear, 2006; National Research Council, 2010a). Comparing overall and disaggregated student performance on DC CAS and NAEP can help to provide a check on the integrity of results. REFERENCES Allensworth, E.M., and Easton, J.Q. (2007). What Matters for Staying On­Track and Gradu­ ating in Chicago Public High Schools: A Close Look at Course Grades, Failures, and Attendance in the Freshman Year, Research Report. Chicago: Consortium on Chicago School Research at the University of Chicago. 13 Fordetails, see http://www.corestandards.org/in-the-states [accessed January 2010]. 14 Forthe DC government press release announcing the State Board’s adoption, see http:// newsroom.dc.gov/show.aspx/agency/seo/section/2/release/20261 [accessed March 2011].

OCR for page 63
87 STUDENT ACHIEVEMENT UNDER PERAA Bandeira de Mello, V., Blankenship, C., and McLaughlin, D. (2009). Mapping State Profi­ ciency Standards onto NAEP Scales: 2005­2007 (Research and Development Report, NCES 2010-456). Washington, DC: National Center for Education Statistics. Booher-Jennings, J. (2005). Below the bubble: “Educational triage” and the Texas account- ability system. American Educational Research Journal, 42(2), 231-268. District of Columbia Public Schools. (2009). Progress: Second Year of Reform. Washington, DC: Author. Available: http://www.dc.gov/DCPS/Files/downloads/ABOUT%20DCPS/ Strategic%20Documents/Progress%20Report%20-%202008-2009/DCPS-Annual- Report-7-21-2010-Full.pdf [accessed March 2011]. District of Columbia Public Schools. (2010). Learning Standards for Grades Pre­K­8. Available: http://dcps.dc.gov/DCPS/In+the+Classroom/What+Students+Are+Learning/Learning+ Standards+for+Grades+Pre-K-8 [accessed October 2010]. Harwell, M., and LeBeau, B. (2010). Student eligibility for a free lunch as an SES measure in education research. Educational Researcher, 39(2), 120-131. Ho, A.D. (2008). The problem with “proficiency”: Limitations of statistics and policy under No Child Left Behind. Educational Researcher, 37(6), 351-360. Jacob, B.A., and Levitt, S.D. (2003). Catching cheating teachers: The results of an unusual experiment in implementing theory. In W.G. Gale and J. Rothenberg Pack (Eds.), Brookings­Wharton Papers on Urban Affairs 2003 (pp. 185-209). Washington, DC: Brookings Institution Press. Koretz, D.M. (2008). Measuring Up: What Educational Testing Really Tells Us. Cambridge, MA: Harvard University Press. Koretz, D.M., Linn, R.L., Dunbar, S.B., and Shepard, L.A. (1991). The Effects of High­ Stakes Testing on Achievement: Preliminary Findings About Generalization Across Tests. Paper presented at the Annual Meetings of the American Educational Research Associa- tion (April 3-7) and the National Council on Measurement in Education (April 4-6), Chicago, IL. Lazear, E.P. (2006). Speeding, terrorism, and teaching to the test. The Quarterly Journal of Economics, 121(3), 1029-1061. Available: http://www.mitpressjournals.org/doi/ abs/10.1162/qjec.121.3.1029?journalCode=qjec [accessed March 2011]. Linn, R.L. (2000). Assessments and accountability. Educational Researcher, 29(2), 4-16. Miller, R.T., Murnane, R.J., and Willett, J.B. (2007). Do Teacher Absences Impact Student Achievement? Longitudinal Evidence from One Urban School District. (NBER Working Paper No. 13356). Cambridge, MA: National Bureau of Economic Research. Available: http://www.nber.org/papers/w13356 [accessed March 2011]. National Research Council. (1995). Measuring Poverty: A New Approach. C.F. Citro and R.T. Michael (Eds.). Panel on Poverty and Family Assistance: Concepts, Information Needs, and Measurement Methods. Committee on National Statistics. Commission on Behav- ioral and Social Sciences and Education. Washington, DC: National Academy Press. National Research Council. (1999). High Stakes: Testing for Tracking, Promotion, and Gradu­ ation. J.P. Heubert and R.M. Hauser (Eds.). Committee on Appropriate Test Use. Board on Testing and Assessment. Commission on Behavioral and Social Sciences and Educa- tion. Washington, DC: National Academy Press. National Research Council. (2010a). Getting Value Out of Value­Added: Report of a Work­ shop. H. Braun, N. Chudowsky, and J. Koenig (Eds.). Committee on Value-Added Methodology for Instructional Improvement, Program Evaluation, and Accountability. Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

OCR for page 63
88 EVALUATING THE DISTRICT OF COLUMBIA’S PUBLIC SCHOOLS National Research Council. (2010b). State Assessment Systems: Exploring Best Practices and Innovations, Summary of Two Workshops. A. Beatty, Rapporteur. Committee on Best Practices for State Assessment Systems: Improving Assessment While Revisiting Stan- dards. Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. National Research Council and National Academy of Education. (2011). High School Drop­ out, Graduation, and Completion Rates: Better Data, Better Measures, Better Decisions. R.M. Hauser and J.A. Koenig (Eds.). Committee for Improved Measurement of High School Dropout and Completion Rates: Expert Guidance on Next Steps for Research and Policy Workshop. Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. Neal, D., and Schanzenbach, D.W. (2007). Left Behind by Design: Proficiency Counts and Test­Based Accountability. (NBER Working Paper No. 13293). Cambridge, MA: National Bureau of Economic Research. Available: http://www.nber.org/papers/w13293.pdf [accessed March 2011]. Office of Technology Assessment. (1992). Testing in American Schools: Asking the Right Questions. Summary. Washington, DC: U.S. Government Printing Office. Available: http://govinfo.library.unt.edu/ota/Ota_1/DATA/1992/9236.PDF [accessed March 2011]. Office of the State Superintendent of Education. (2010a). DC­CAS Grade 3 Performance Level Descriptors. Washington, DC: Author. Available: http://osse.dc.gov/seo/frames.asp?doc=/ seo/lib/seo/Grade_3_Performance_Level_Description.pdf [accessed November 2010]. Office of the State Superintendent of Education. (2010b). District of Columbia Comprehensive Assessment System: Resource Guide 2011. Washington, DC: Author. Available: http:// osse.dc.gov/seo/frames.asp?doc=/seo/lib/seo/assessment_and_accountability/2011_dc_ cas_resource_guide.pdf [accessed November 2010]. U.S. Census Bureau. (2010). 2005­2009 American Community Survey Five­Year Estimates. Available: http://factfinder.census.gov/servlet/DatasetMainPageServlet?_program=ACS&_ submenuId=&_lang=en&_ds_name=ACS_2009_5YR_G00_&ts= [accessed March 2011].