A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence (2011)

1

Introduction

THE CITY AND ITS SCHOOLS

The nation’s capital is a small city with big challenges. Home to the government of the richest and most powerful country on earth, the District of Columbia has a population of about 600,000 (excluding its ever widening suburban ring), which makes it roughly comparable in size to Boston and about 1/13th the size of New York City.¹ The fiscal 2010 operating budget for the city government was about $10 billion, and it employs more than 32,000 full-time staff.

Washington is a diverse city. Over half the residents are black, and almost one in five speaks a language other than English in the home. It is home to the nation’s largest concentration of college-educated blacks, and black residents hold prominent leadership positions in corporations, universities, and federal, state, and municipal government agencies. However, blacks also make up the largest group of economically disadvantaged residents in the city.

Although median household income and the share of residents who are college educated are higher than national averages, poverty rates are also higher (17 percent compared to 13 percent nationally), and there is large variation in economic well-being by neighborhood. The city includes neighborhoods that have been impoverished for decades, extremely affluent sections similar to the most well-to-do suburbs of nearby Maryland

_______________

¹The city of Washington, District of Columbia, is commonly referred to as Washington, the District, or simply DC, and we use all three names in this report. Where the word district is not capitalized, we are using it to refer to school districts in general.

and Virginia, and many in between. The most affluent section (the city’s Ward 3) has a median household income that is almost 200 percent of the citywide average; in contrast, the poorest neighborhoods have incomes that are 37 percent below the citywide average. The phrases “east of the river” (the Anacostia) and “west of the park” (Rock Creek Park) are understood by DC residents as euphemisms for the city’s enduring race and class divide, a divide mirrored in the city’s public schools.

The city’s most significant political peculiarity is that it was designated in the U.S. Constitution (Article One, Section 8) as a district under the jurisdiction of the federal government and is not part of any state. Until 1973, the city had no independent governing authority, with virtually all municipal functions under the control of the U.S. Congress. In that year the Home Rule Act granted the District limited governance authority, but Congress still retains considerable authority over its affairs and budget, and the city’s elected Representative to Congress does not have a vote in that body. This situation has long been a flash point for DC residents, and many car owners have license plates with the slogan “taxation without representation,” to echo Patrick Henry’s famous phrase about tyranny.

The public policy arena under the purview of the DC government that is most fraught, most politically contested, and most socially complex is education. The city has a relatively small public school system, with about 45,000 students enrolled in traditional elementary and secondary schools and another 28,000 in public charter schools.² Formally segregated until 1954, the schools serve a city in which residential patterns continue to play a prominent role in the politics of education. In the 2006-2007 school year, for example, less than 33 percent of all white school-age children attended DC public schools (including charter schools), while more than 90 percent of all black and 88 percent of all Hispanic school-age children did so. Looking at it another way, white children made up over 13 percent of the city’s school-age population, but accounted for only 5 percent of all students in public or charter schools.³

Reforms to the education system, then, inevitably evoke concerns about neighborhood cohesion, gentrification, and the power of commercial and economic development interests, as well as the potentially negative effects that change may have on the city’s poor and minority populations.

Given the city’s uniquely complicated historical, political, and economic history, the governance of the DC Public Schools (DCPS) has necessarily

_______________

²For contrast, New York City has more than 1 million children enrolled in public schools and about 40,000 in charter schools.

³For 2006-2007 (the latest year for which complete data are available), 3,521 of 11,298 white school-age children were in public or charter schools, 57,706 of 63,861 black children, and 7,130 of 8,017 Hispanic children (21st Century School Fund, Brookings Institution, and Urban Institute, 2008).

been significantly different from that of any other school district in the country. Rather than being one of a number of school districts governed by a state department of education, DCPS has been overseen by a changing combination of entities and individuals, including Congress and local officials. As summarized by two experts who have studied the system closely (Hannaway and Usdan, 2008, p. 116):

The school system is well known not only for its struggles with governance, but also for its students’ persistently low average achievement, and particularly the achievement of poor and minority students. Although recent data from the National Assessment of Educational Progress (NAEP) show modest gains in student achievement for some DC students between 2007 and 2009, the district’s average test performance historically has been poor, contributing to the system’s dismal reputation for at least three decades.

Since state-by-state comparisons of student achievement on NAEP first became available in the 1990s, DC schools have performed at the low end of the scale; numerous reports prior to that time had also documented DC students’ low average performance and other shortcomings. On the District’s own assessments, average performance has fluctuated, although there have been pockets of excellence. DCPS has frequently been publicly criticized not only for its students’ low achievement, but also for its poor financial management, dilapidated facilities, inadequate resources, and other failings. Mounting frustration about the quality of the public schools has led DC (like other cities confronting similar challenges) to approve 60 public charter schools, now serving roughly 28,000 students. Some local activists have urged an even more dramatic change by supporting school vouchers that can be used toward tuition costs at private schools (District of Columbia Charter School Board, 2010). As this report goes to press, there is a movement in the U.S. Congress to restore the city’s voucher program, which was suspended in 2006.

It was in this context that the DC City Council passed the Public Education Reform Amendment Act (PERAA) of 2007, which established

mayoral control of the city’s public schools and a state department of education and instituted other significant management changes (Office of the Chief Financial Officer, 2007). PERAA also mandated an independent, comprehensive, 5-year evaluation to determine “whether sufficient progress in public education has been achieved to warrant continuation of the provisions and requirements of this act or whether a new law, and a new system of education, should be enacted by the District government…” (p. 9).

THE COMMITTEE’S CHARGE AND ITS WORK

In response to PERAA’s requirement for an independent evaluation, the DC City Council, with the concurrence and cooperation of the mayor, the chancellor of DCPS, and the new State Superintendent of Education (a position created by PERAA), turned to the National Research Council (NRC) of the National Academies. This report, the first in what is expected to be a series issued during the next 5 years, is the product of an expert panel convened by the NRC in response to that request.

Although the design and oversight of an evaluation is an unusual assignment for the NRC, which does not routinely conduct program evaluations or address the circumstances of a single jurisdiction, the institution recognized both the special circumstances motivating the request and the extraordinary opportunity the initiative represents. The financial and moral support of local business and civic leaders reinforced the NRC’s vision that the initiative could provide a valuable contribution to the ongoing public debate about public education in the District.

The committee’s charge for the first phase of the project was not to conduct an evaluation but to design a potential multiyear, multiphase evaluation of the District of Columbia Public Schools. The committee was asked to identify available data and assess its quality and utility; develop a preliminary set of indicators; engage with various stakeholder groups, including researchers, national and local reform experts, and civic leaders; and explore the desirability, feasibility, and scope of the optional next phases of the initiative. The committee has aspired to provide as comprehensive a response as possible, and our interpretation of the charge is based on a number of basic assumptions, shown in Box 1-1.

The committee addresses all aspects of its charge in this report, but we do so with varying degrees of analytical depth as allowed by existing and accessible information. Constraints of budget, time, and data availability limited what we could accomplish. Indeed, our experience developing this foundational evaluation plan demonstrated that answering complex questions about a rapidly changing urban school reform requires a sustainable program that takes into account ongoing community input.

We learned a great deal about the circumstances in the DC school system from our review of preliminary data, but the committee had neither the time nor the resources to conduct a thorough analysis of available data or to collect new data. However, even a more systematic analysis of the information that is available would likely not provide a sufficient basis for conclusions about the effects of PERAA or about how well the system is faring more generally. Moreover, even as this first report goes to press, the situation in DC has changed significantly from when the committee first met: although much attention was focused as this project began on the decisions of the first mayor and chancellor who served under PERAA, both offices have since changed hands.

Thus, our focus was the development of a plan for a sustained, independent evaluation program. We hope that this plan will be of use to other districts but it was developed specifically for Washington, DC. Our report begins with a review of the historical background and context in which PERAA was enacted and a discussion of what we have learned about the public education system in the city—all of which influenced our plan for the evaluation.

Planning for the committee’s work began with a public meeting in July 2009, which approximately 80 people attended. Many spoke about what they saw as the most important educational outcomes for DCPS. Perspectives were offered by DC government officials (the chair of the city council, the deputy mayor, the chancellor and the state superintendent all spoke), as well as civic, business, and labor leaders, and DCPS parents. Expert input was also obtained from education researchers and evaluators.

The committee was formally appointed in early 2010, and it held three meetings that year, as well as a public forum through which we again sought the views of stakeholders from across the city. At that forum, principals and school administrators; teachers; charter school representatives; special education providers; education providers for children and youth; representatives of colleges, universities, and job training programs; students; and parents were asked to discuss the education issues they viewed as most important for the city.

We also commissioned two background papers and have sought input from researchers, DCPS officials, national and local experts in education reform, civic leaders, and members of the school community. On behalf of the committee, staff attended DCPS hearings and community meetings. We have also reviewed much of the published literature on recent reforms in the District, as well as other relevant research on reforms elsewhere and have examined available accounts of developments in the history of PERAA.

The primary result of the committee’s work is the design for a comprehensive and continuing program for evaluating the District’s schools. Our recommended design is presented and discussed in Chapter 7. We note the logic that leads us to this conclusion. In general terms we differentiate among the intent of the reform (as articulated in the law), its implementation (actions taken by DCPS and the city government), and its effects (on student learning and other valued outcomes). Though questions about the reform’s effects on learning are perhaps the most important and the ones with greatest long-term impact, they also require the most data, the most rigorous analysis, and the most patience.

The structure of this report reflects that logic and those three elements. We begin in the next two chapters with the background needed to understand the District of Columbia’s schools and the intents underlying the passage of PERAA. Chapter 2 provides a brief overview of education reform

nationally; Chapter 3 provides the historical context for PERAA. Chapter 4 describes the city’s response thus far (up to the end of 2010, when this report went into final production) to the requirements of PERAA, focusing on its implementation. The next two chapters provide a preliminary look at the very limited evidence that is available about effects on learning and other valued outcomes: Chapter 5 looks at student achievement, and Chapter 6 considers a wide array of other issues that need to be considered in any evaluation. In both these chapters we offer the committee’s cautions and caveats about how to interpret this kind of early evidence. Chapter 7 presents the committee’s consensus regarding the fragility of existing information as a basis for reaching summative judgments—positive or negative—about the effectiveness of the reform, and our recommendation for a robust, sustainable, and independent program of evaluation and research.

REFERENCES

District of Columbia Public Charter School Board. (2010). SY2009 to 2010 Charter School Profile. Available: http://www.dcpubliccharter.com/Enrollment-and-Demographics/SY2009-to-2010-Charter-School-Profile.aspx [accessed July 2010].

Hannaway, J., and Usdan, M.D. (2008). Mayoral takeover in the District of Columbia. In W.L. Boyd, C.T. Kerchner, and M. Blyth (Eds.), The Transformation of Great American School Districts: How Big Cities Are Reshaping Public Education (pp. 113-188). Cambridge, MA: Harvard Education Press.

Office of the Chief Financial Officer. (2007). Public Education Reform Amendment Act of 2007. Washington, DC: West Group.

21st Century School Fund, Brookings Institution, and Urban Institute. (2008). Quality Schools, Healthy Neighborhoods, and the Future of DC, Policy Report. Washington, DC: Author. Available: http://www.21csf.org/csf-home/publications/QualitySchoolsResearchReport/QualitySchoolsPolicyReport9-18-08.pdf [accessed March 2011].

This page intentionally left blank.