In 2007, the District of Columbia made a bold change in the way it governs public education with the goal of shaking up the system and bringing new energy to efforts to improve outcomes for students. The Public Education Reform Amendment Act (PERAA) shifted control of the city’s public schools from an elected school board to the mayor, created a new state department of education, created the position of chancellor, and made other significant management changes. PERAA also mandated an independent, comprehensive, 5-year evaluation to determine “whether sufficient progress in public education has been achieved to warrant continuation of the provisions and requirements of this act or whether a new law, and a new system of education, should be enacted by the District government.…”
To plan that evaluation, the Committee on the Independent Evaluation of DC Schools was convened by the National Research Council in response to a request from the City Council of the District of Columbia. The committee was asked not to conduct the evaluation, but to provide initial guidance on the focus and structure of the required evaluation. The work included identifying available data and assessing its quality and utility; developing a preliminary set of indicators; engaging with various stakeholder groups, including civic leaders, parents, researchers, and national and local reform experts; and exploring the desirability, feasibility, and scope of the optional next phases of the evaluation.
This report documents the committee’s plan for the evaluation. It lays out a plan for a comprehensive, long-term program of evaluation that is designed not only to examine short-term effects of the changes made under PERAA, but also to provide the District with a structure for continuous,
independent monitoring of important features of its school system. The plan is based on the committee’s review of preliminary data and on its conclusion that first impressions of the implementation of PERAA and its effects, though informative, are not sufficient as a basis for decisions about PERAA or continued improvement of the city’s education system.
The committee agreed on several basic assumptions and goals that have guided our work. First, although many U.S. cities have undertaken significant reforms to change their schools and researchers have examined what they have done, there is no established model for evaluating a district involved in reform—or, for that matter, any district. Second, school districts are judged primarily on the academic achievement of their students, but achievement depends on how effectively a school district accomplishes its many responsibilities and pursues many valued educational outcomes. Third, we interpreted PERAA’s requirement for an evaluation broadly: to establish for the residents and leaders of DC a sustainable ongoing program of evaluation that provides reliable information they can use to improve the school system continuously, regardless of future political or personnel changes. Last, the committee approached the most challenging part of its charge—to explore the effects of the reform legislation itself—by distinguishing among the intent of the reform, as articulated in the law; its implementation, that is, the actions taken by the DC Public Schools (DCPS) and other responsible city agencies; and its effects on student learning and other valued outcomes.
CONTEXT
PERAA is the latest in a long line of changes in the way the city’s public schools are governed. Since 1804, there have been 17 different governance and administrative structures, and PERAA was the second new approach since 2000. Many of these changes were responses to concerns about students’ academic performance, the quality of the schools and the teachers, and an ineffective central bureaucracy, as well as the perception that many DC residents were indifferent to the persistent problems.
The city’s education problems have been intensified by a history of segregation, and the city continues to struggle with many challenges related to race, poverty, and geography. Those challenges include inequitable distribution of resources and supports to schools in the lowest-income sections of the city, which are largely black, tensions over demographic shifts that change the character of neighborhoods, and a strong charter school movement. They have made reform efforts more urgent while complicating the city’s response to them.
Another factor in DC has been the city’s distinctive political status as a small geographic area under the jurisdiction of the federal government.
Because DC is not part of any state and elected its first mayor and city council only in 1973, it does not have a long tradition of self-governance. The U.S. Congress retains considerable authority over its affairs and budget.
PERAA was a response in part to these historical circumstances, but it was also spurred by impressions of the effectiveness of reforms in other urban districts facing at least somewhat similar economic, social, and historical challenges. Districts in Boston, Chicago, Cincinnati, Minneapolis, and New York City (among others) have focused on the alignment of content and performance standards with curricula, instruction, and other aspects of the school system. They have used data to guide their decisions, emphasizing such goals as improved professional development for teachers and principals; more frequent formative assessments; and the development of a culture of learning and collaboration among teachers. These approaches are widely used and are supported by some promising evidence, but the research literature is not yet settled enough to provide firm guidance on best practices for district reform or evaluation.
Some districts have also focused on the governance of schools, and a few (e.g., Boston, Chicago, Cleveland, and New York City) have given their mayors control over the public schools. Such reforms are designed to “jolt” the system by changing dysfunctional institutional relationships and giving leaders new lines of authority and accountability. Evaluation of these governance reforms is critical to knowing what really works and what does not, but few cities have made this a priority, so there are neither clear exemplars nor substantial evidence to guide the District as it implements PERAA.
IMPLEMENTATION OF PERAA
The District of Columbia has made many changes called for in PERAA. Thoroughly documenting the city’s efforts will be a critical component of the comprehensive evaluation the law requires, and until this is done, no firm conclusions should be drawn about how well the city has implemented PERAA and fulfilled its intentions. As a first step, however, we offer an outline of the city’s response to PERAA.
The new structures mandated in PERAA have largely been put into place. The mayor now has responsibility for most key aspects of the school system, including appointment of a chancellor who establishes educational priorities, adopts curricula and assessments, and ensures that the schools are appropriately staffed and managed. Also in place are the Department of Education, the Deputy Mayor for Education, the Office of the State Superintendent of Education (OSSE), the State Board of Education, and the Public Charter School Board. Not currently in place are the Office of Ombudsman for Public Education (a position that had been filled but was later eliminated) and the comprehensive data system.
DCPS has also adopted strategies to meet the goals of PERAA. Among them are efforts intended to improve the quality of teachers, principals, and administrators, including a new system for evaluating teacher performance; a new teaching and learning framework, which describes the specific instructional practices the district has identified as most likely to promote student learning; and improvements to school facilities.
FIRST IMPRESSIONS OF THE DC SCHOOL SYSTEM UNDER PERAA
Student Achievement
Public attention frequently is focused on fluctuations in student achievement scores, in DC as in the rest of the nation. First impressions offer a mixed picture: in general, scores on the District of Columbia Comprehensive Assessment System (DC CAS) have continued on an upward trajectory (which began before PERAA was enacted), and they have flattened slightly during the most recent 2 school years. However, definitive conclusions about PERAA’s effects cannot be drawn from these preliminary results for three major reasons:
- The DC CAS is designed to measure students’ mastery of specific academic skills, but determining whether and how the changes in district policies or strategies have contributed to those skills requires additional empirical evidence: the scores themselves do not provide evidence about what accounts for them.
- The available scores are averaged across the entire student population, and do not provide information on the status or progress of specific groups: some may be making sharp gains while others are not.
- Because DC is a highly mobile district and the student population changes every year, score fluctuations may be the result of changes in the characteristics of the students taking the test, rather than improvements or declines in students’ knowledge and skills.
Thus, in order to draw any conclusions about the effect of PERAA on student achievement as measured by DC CAS, further study of patterns for types of schools, individual schools, grade levels, neighborhoods, wards, and population subgroups is needed, and this should include longitudinal studies of cohorts of students within the District.
Scores from the National Assessment of Educational Progress (NAEP) provide independent information about achievement trends in all 50 states and DC, and these results also suggest that, in general, DC students’ performance has been improving. However, as with the DC CAS scores, more
study is needed to understand the reasons for trends. Like DC CAS, NAEP does not account for changes in the demographics of the population, so it is not possible to tell from these scores alone whether the improvements are the result of demographic shifts rather than changes in educational policies, programs, or practices.
School Quality and Operations
Like any district, DC is responsible for setting high expectations for all students and providing them with the instruction and resources necessary to meet them. Test scores only provide evidence, partial at that, about one aspect of the system. A school system’s responsibilities are more complicated, and can be categorized in five broad areas:
- quality of personnel (teachers, principals, and others),
- quality of classroom teaching and learning,
- capacity to serve vulnerable children and youth,
- promotion of family and community engagement, and
- quality and equity of operations, management, and facilities.
The District seems to have made changes in these areas, but a comprehensive evaluation would be needed to determine whether, how, and where conditions are improving. The District is already collecting data on many of these functions, and a first step in the evaluation will be to systematically assess these measures, determine which will be useful for the evaluation program, and identify priorities for new data collection—a task that was beyond the resources of this committee.
FROM IMPRESSIONS TO EVIDENCE: AN EVALUATION PLAN
RECOMMENDATION 1 We recommend that the District of Columbia establish an evaluation program that includes long-term monitoring and public reporting of key indicators as well as a portfolio of in-depth studies of high-priority issues. The indicator system should provide long-term trend data to track how well the programs and structure of the city’s public schools are working, the quality and implementation of key strategies undertaken to improve education, the conditions for student learning, and the capacity of the system to attain valued outcomes. The in-depth studies should build on indicator data. Both types of analysis should answer specific questions about each of the primary aspects of public education for which the District is responsible: personnel (teachers, principals, and others); classroom teaching and learning; vulnerable children and youth; fam-
ily and community engagement; and operations, management, and facilities.
Figure S-1 depicts our proposed evaluation framework. It begins with the goals the district has set for itself, as shown in the horizontal box that appears at the top of the figure. The logic of this framework reflects the point that passing a law does not automatically result in increased student learning, reduced achievement gaps, increased graduation rates, or other valued outcomes. To achieve these outcomes, the new structures and relationships that PERAA mandated have to be established and working as intended; school system leaders have to have implemented strategies that are likely to be effective; those strategies have to be well implemented; and the conditions for student learning—such as the quality of school staff and instruction—have to have improved.
In addition to the elements of reform, the evaluation has to cover the broad areas of the school district’s responsibility: see the shaded horizontal stripes that cut across the elements of reform in Figure S-2. In this elaboration of Figure S-1, the elements of reform and the broad evaluation questions pertaining to them are depicted in the vertical boxes, and the substantive areas of responsibility are depicted with shaded horizontal bands. This framework is designed to guide the evaluation so that it is comprehensive: even if resources limit the specific analyses that can be undertaken at a given time, use of the framework will ensure that the most important aspects of the system are examined.
The framework is a depiction of the primary components of reform and of the district’s responsibilities. The basic questions to be asked under each of the four elements and across the five areas of responsibility will need to be answered using many different study designs, data collection methods, and types of analysis. Thus, the evaluation framework provides a guide to the kinds of information that are needed to fully inform policy makers and the public. The indicators—which should be developed in conjunction with OSSE and DCPS and members of the community—should provide long-term disaggregated trend data to track how well district roles and structures are working, the quality and implementation of key strategies undertaken to improve education, the conditions for student learning, and valued outcomes. The in-depth studies should draw from the indicators, as well as other data, to provide detailed answers to specific questions about key aspects of public education in the District.
It will be critical to establish stable indicators as soon as possible, supplementing and refining those the District is already collecting as needed; however, the program of focused evaluation studies will evolve over time as changes in the city’s policies, challenges, and circumstances require. Many empirical questions are subsumed in the elements of reform and
the broad categories of district responsibility. The evaluators will look to District leaders and other members of the community to establish the priorities and available resources that will guide the choice of specific indicators and studies, and the long-term indicator system will build on data collection efforts already in place in the District. The evaluation needs to engage the perspectives, concerns, and needs of all who are part of and care about the system: students (and youth who are disconnected from school), families, educators, administrators, and the community. The key evaluation questions, the data used to answer them, and how these answers are shared and used need to be designed with the concerns and goals of the community in mind.
RECOMMENDATION 2 The Office of the Mayor of the District of Columbia should produce an annual report to the city on the status of the public schools, drawing on information produced by the District of Columbia Public Schools and other education agencies and by the independent evaluation program that includes
- summary and analysis of trends in regularly collected indicators,
- summary of key points from in-depth studies of target issues, and
- an appendix with complete data and analysis.
Building and maintaining a high-quality indicator system, designing studies that address pressing issues, and presenting and disseminating findings so that all stakeholders can act on them will require deliberate and skillful management. An independent evaluation program that is an ongoing source of objective information and analysis will be an invaluable resource for the city under changing political circumstances. To make such a program work, the District of Columbia will need to engage potential research partners and funders in planning and developing an infrastructure for ongoing independent evaluation of the city’s public schools.
Urban districts face some of the most difficult challenges in U.S. public education, and many have pursued ambitious reforms. Valuable lessons have begun to emerge from their experiences; systematically evaluating these efforts and their effects is a critical part of education reform. Objective evidence derived from multiple sources of data is a tool for monitoring progress and guiding continuous improvement in a city’s schools—and also for ensuring that their benefits can be sustained and replicated in other districts. It is our hope that this model will be of use to districts around the country.