Indicators are intended to focus attention on a few, key, readily interpretable facts that tell users about the current status of a topic or field and highlight areas in need of improvement. For the purposes of educational equity, indicators should convey—in an easily understood way—the range, nature, and magnitude of disparities, as well as measures of any narrowing or widening over time. Indicators might be useful for many purposes, including research or accountability. The committee, however, was most attentive to information that could inform efforts to improve policy and practice.
Currently, equity is a prominent focus for education policy makers and, in turn, for those who implement policy; this is in part because equity was a major theme in the Every Student Succeeds Act (ESSA, 2015) and its predecessor No Child Left Behind (NCLB, 2002). States are working to develop and incorporate equity indicators in their required plans, while the federal government is working to review and approve state plans. As stated by one of the individuals we interviewed, “[Nearly] every national level organization related to education is having a discussion about equity. They are including equity in their priorities, core principles, strategic plans, and member activities.” A set of research-based indicators could help to ground the related but disparate conversations that are taking place and provide a common starting point for many stakeholders in pursuing their own missions. Another interviewee said that the committee’s indicators would serve as a “North Star” to guide their process for exploring, investigating, and studying disparities.
This chapter begins by summarizing some of the information about indicator systems we gleaned from: (1) expert guidance on indicator sys-
tems; (2) existing programs and initiatives focused on educational equity; and (3) stakeholder insights on the purposes and uses of indicators. It then describes our own framework, or architecture, for a comprehensive set of key equity indicators.
There are numerous publications that offer guidance on how to set up an indicator system. We primarily relied on the six, listed below in chronological order, written by authorities in the field of education indicators:
- Jeanie Oakes (1986), Education Indicators: A Guide for Policy Makers
- Shavelson et al. (1987), Indicator Systems for Monitoring Science and Mathematics Education and the accompanying sourcebook
- Marshall Smith (1988), Education Indicators
- Field, Kuczera, and Pont (1988), Ten Steps to Equity in Education
- Bryk and Hermanson (1993), Educational Indicator Systems: Observations on Their Structure, Interpretation, and Use
- Planty and Carlson (2010), Understanding Education Indicators
These writings discuss different types of indicators (based on a single statistic or a compound statistic) and different uses for indicators (to report on status, monitor change, project future patterns). They offer advice on ways to design a system, and they provide criteria for evaluating the technical qualities essential for the indicators included in the system. They also provide guidance on the process of developing both the conceptual and operational definitions of indicators, determining how to measure the construct, and collecting the needed data.
Indicators have long been used in the education arena—one of the earliest was the National Assessment of Educational Progress (NAEP).1 Designed to track educational achievement over time, NAEP has reported reading and mathematics achievement results for students aged 9, 13, and 17 since 1971. Although not usually referred to as a system of equity indicators, NAEP has highlighted the achievement gaps among the nation’s students. Routinely published reports track performance differences for students grouped by race, ethnicity, and sex, and provide disaggregated results for students with disabilities, English learners, and students who receive free
and reduced-price school lunch. NAEP maintains a sophisticated website with a user-friendly dashboard that allows users to select the information they want to see. Over the years, NAEP has evolved and adapted its methods to meet various policy needs. NAEP has expanded the grade levels included, the subject matter tested, and the reporting levels, first to the state level and then to the urban district level. It is difficult to overestimate the impact NAEP has had on education in the country, particularly the role it has played in raising awareness about disparities in achievement. These impacts are well documented (Bourque, 2009; Casserly et al., 2011; Glaser, Linn, and Bohrnstedt, 1997; Jones and Olkin, 2004; NCES, 2012).
There are numerous other initiatives that have contributed to understanding of disparities in education. Government agencies, such as the National Center for Education Statistics (NCES) and the Census Bureau, collect an enormous amount of data that have been used to evaluate educational equity. For example, NCES annually prepares the Condition of Education report that contains indicators on the state of education in the United States, from pre-kindergarten through postsecondary education, as well as labor force outcomes and international comparisons. The data for these indicators are obtained from many different providers—including students and teachers, state education agencies, local elementary and secondary schools, and colleges and universities—using surveys and compilations of administrative records. These data also become part of the Education Digest published by NCES and are easily available and widely used.
In addition to the Condition of Education, NCES uses these data to prepare other reports, of which the most relevant for our purposes is a series of annual publications called Status and Trends in the Education of Racial and Ethnic Groups. The data are also used by others. The Annie E. Casey Foundation draws from these data to prepare its Kids Count reports; more recently, the foundation uses the data to monitor and evaluate educational equity in a series of reports, Race for Results: Building a Path to Opportunity for All Children. Another user is the Federal Interagency Forum on Child and Family Statistics, which annually publishes America’s Children: Key National Indicators of Well-Being. Child Trends draws from these data to create short easily digested policy briefs.
Another important data resource is the Civil Rights Data Collection (CRDC) program of the Office for Civil Rights (OCR) of the U.S. Department of Education. Since 1968, a wide variety of data have been collected on key education and civil rights issues in public school, including information about student enrollment and educational programs and services, most of which are disaggregated by race and ethnicity, sex, English proficiency, and disability. “The CRDC is a longstanding and important aspect of the [Office for Civil Rights’] overall strategy for administering and enforcing the civil
rights statutes for which it is responsible.”2 OCR prepares “first look” briefs and focused reports that are easily downloaded from its website. The topics of these briefs differ from year to year. The two briefs issued in April 2018 based on data from the 2015-2016 data collection are the STEM [science, technology, engineering, and math] Course Taking Issue Brief and the School Climate and Safety Issue Brief. The CRDC also provides ready access through a search feature to three special reports for school districts and schools: English Learner Report, Discipline Report, and Educational Equity Report: they are provided in Excel spreadsheets. The search feature allows users to generate state, district, and school reports of user-selected data elements disaggregated by user-specified demographics.
Given that a variety of equity indicators are already reported, we considered the ways that our efforts might improve the situation. In what ways do the existing initiatives fall short? What can we suggest that would fill existing gaps or make the whole (collection of indicators) greater than the sum of its parts?
To answer these questions, we first conducted a broad review of the equity indicators reported by other organizations. We considered efforts by both government agencies and nongovernmental organizations (NGOs), and we looked at initiatives that targeted equity from the outset as their primary purpose (e.g., CRDC, Annie E. Casey Foundation’s Race for Results project), as well as those that just touch the periphery of equity, such as by reporting disaggregated results. We considered 19 of the organizations that produce various reports and briefs intended for a wide spectrum of audiences. Some are involved in all the steps of producing indicator reports (e.g., National Center for Education Statistics), from collecting data to reporting the results. Others make use of data collected by government agencies to develop their own indicators and associated reports (e.g., Child Trends). Still others make use of indicators developed by others to include in their own reports. Some organizations publish reports on a regular basis, most often annually (e.g., Kids Count, from the Annie E. Casey Foundation); others publish briefs when the findings warrant (e.g., Child Trends, CRDC). Appendix B provides details about these 19 existing initiatives. We selected reports from seven organizations to explore in depth—to learn more about the indicators that are reported, the data they are derived from, how and when they are reported, and how they are intended to be used and by whom. The reports include:
- Annie E. Casey Foundation: Kids Count Data Book and Race for Results
- Council of the Great City Schools: Academic Key Performance Indicators: 2018 Report
- Education Law Center and Rutgers University: Is School Funding Fair? A National Report Card on Funding Fairness
- Federal Interagency Forum on Child and Family Statistics: America’s Children: Key National Indicators of Well-Being
- National Center for Education Statistics: Condition of Education and Status and Trends in the Education of Racial and Ethnic Groups
- National Institute for Early Education Research, Rutgers University: State of Preschool yearbooks
- U.S. Department of Education, Office of Civil Rights: Civil Rights Data Collection First Look Issue Briefs and Special Reports
In a related vein, we considered whether an equity indicator system has already been created, in effect, by the data collection and “report card” obligations states face under the federal Every Student Succeeds Act of 2015 (ESSA).3 Because federal impositions on states are politically problematic, the statutory reporting requirements are loosely defined and cover a very limited domain of education inputs and outcomes, at least when compared with the range of variables discussed in the research literature (see National Urban League, 2019). Moreover, state indicator systems under ESSA continue to evolve, have varied designs, and produce data in ways that may not be comparable with each other or even wholly consistent within a state. Finally, the federal requirements not only are underspecified in a technical sense, but also are only the minimum requirements; states and districts are free to augment these base indicator systems.
To gather information about ways equity indicators might be used, the committee heard presentations during its first two meetings (see agendas in Appendix D) and solicited informal input from representatives of a small set of government agencies and organizations that provide information, support, and analytic assistance to the education policy community. This group included the following:
- representatives from various member organizations that work on behalf of cities, states, school boards, and school administrators, such as the Council of Chief State School Officers, the Council of
- independent organizations, such as the Education Trust and the Alliance for Excellent Education;
- organizations that have developed related indicator and reporting systems, such as the Annie E. Casey Foundation and Child Trends; and
- government agencies that collect relevant data, such as the CRDC, the Census Bureau, and NCES.
the Great City Schools, the National Governors Association, and the national Parent Teacher Association (PTA);
As noted above, equity is a prominent focus for education policy makers and, in turn, for those who implement policy. Several individuals talked about the need to understand disparities and determine actions to take. States, districts, and other entities may be aware of inequities in their school systems, but they need help in judging their magnitude and sources and in identifying strategies for addressing them. These groups would benefit from clear, understandable metrics that highlight problem areas and could be used in conversations with various constituencies. They would be seeking ways to measure, define, and track their progress in closing gaps and would especially value information that helps them to make research-based decisions about investments and strategies that are likely to increase educational equity over the long term. Figure 2-1 summarizes the information we learned from our information gathering.
State education agencies are among the most likely organizations to adopt or use information from the committee’s proposed indicators. The existing data collection and reporting burdens on states are a major consideration for any data-related effort, and states (and school districts) would be more likely to adopt indicators of educational equity that dovetail with how they are thinking about equity and the types of data they already collect as part of their own accountability and monitoring efforts, and if they believe they can learn something about themselves from the indicators. Given the current focus on equity in the country, key intermediaries can use the indicators to start and advance conversations about educational equity with states.
Another challenge identified by some stakeholders relates to data that are applicable across state lines. For example, it can be difficult to compile academic indicators in grades 3-8 across state lines because there are not many common data points that illustrate whether states or districts are doing well academically. As one respondent said, “You would think it would be easy with all the testing that we do. But the data that you might collect doesn’t mean the same thing across state lines.”
Finally, some individuals with whom the committee consulted noted that the design of an indicator system should avoid making the perfect the enemy of the useful; an exhaustive, research-based system would likely be overwhelming and unwieldy for decision makers and those who would implement a system. Thus, the committee reasoned, attention to parsimony—while challenging in a research-driven exercise—would be a valuable contribution to the cacophonous discussion of how best to define and gauge educational equity.
To be effective, a system of equity indicators should provide information that users view as important, credible, and valuable. The system should include indicators that represent constructs that are malleable (capable of being changed) and actionable (easily translated into a plan of action). They should be amenable to change as a consequence of educational policy or practice interventions, and this relationship should be backed by empirical research. Some indicators can play a descriptive, signaling role by calling attention to significant disparities in resources and learning opportunities, such as the distribution of school suspensions and enrollment in advanced placement courses by race and ethnicity across schools and over time. Indicators are much more powerful if the conditions they measure can be shown to be consequential for valued outcomes, such as high school completion and successful transitions to postsecondary education.
Our review of existing initiatives revealed that much of the needed data are available. There is, in fact, a wealth of data on pre-K to grade 12
education and beyond and a large number of education indicator reports prepared by government and nongovernment agencies. However, these existing data and reports are not sufficient for the system of educational equity indicators as we have conceptualized it. One set of problems pertains to the ways the data are collected and stored, in comparison with what our proposed system will require. The data are scattered across different databases, collected through different sampling procedures of different populations, based on data collections from different years and administered at different intervals, and varied across jurisdictions in their technical specifications. Another important set of issues concerns the extent to which existing data can be disaggregated for relevant groups of children and reported for different jurisdiction levels (nation, states, school districts, and individual schools).
The information we reviewed led us to conclude that a set of indicators to monitor educational equity should focus both on valued student outcomes and on students’ access to opportunities and resources needed to achieve those outcomes. The measures should reflect multiple dimensions of educational progress and well-being, including both academic achievement and engagement in schooling, since both contribute to students’ likelihood of achieving valued outcomes, such as earning a high school diploma. The measures should be sufficiently precise and meaningful to identify important between-group disparities at single points in time and consistent enough to track them across years. To be useful to a variety of stakeholders, the indicator system should have the capacity to report results at multiple geographic and organizational scales, such as at the classroom, school, district, state, or national level. Critical for the system is the identification and definition of the population groups to be tracked. The system should allow for evaluation of disparities between salient, well-defined population groups.
We have already discussed the fact that students of color and students from financially disadvantaged families experience the K–12 public education process differently from their white and more affluent counterparts. In the indicator system we propose, we do not attempt to address the societal factors that underlie these differences, but we are aware that family and neighborhood factors play important roles in the educational resources available to students. The indicator system should include measures to evaluate and monitor availability of resources that bear on school learning, such as experienced teachers, safe schools, and strong curricula.
Reports of these indicators should be produced on a frequent and regular schedule so that stakeholders can anticipate their release, and they should be easily accessed and understood. The statistics that are reported should be
based on data collected in a scientifically sound manner that can support the intended inferences. The indicator system will likely evolve as new kinds of data become available or new equity issues arise. Despite the system’s overall focus on maintaining consistency to support the integrity of the trend information, its development should anticipate that changes will be needed.
Of course, an indicator system is merely information and cannot itself directly improve equity by altering policies and practices. Any project like ours rests on the familiar assumption that decision makers—officials, practitioners, voters—who have good, useful information will make better decisions. Admittedly, this is akin to an economist’s assumption of rational behavior and may be just as problematic. However, a thorough exploration of how the behavioral sciences might bear on design and implementation of an indicator system is beyond the scope of this report.
CONCLUSION 2-1: To be useful to policy makers, educators, and other stakeholders, two types of equity indicators are needed: (1) indicators that measure disparities in students’ academic achievement and attainment outcomes and engagement in schooling; and (2) indicators that measure equitable access to resources and opportunities, including the structural aspects of school systems that may affect opportunity and exacerbate existing disparities in family and community contexts and contribute to unequal outcomes for students.
CONCLUSION 2-2: To ensure that the pursuit of equity encompasses both the goals to which the nation aspires for its children and the mechanisms to attain those goals, a system of educational equity indicators should balance breadth of coverage with specificity to the appropriate stages of child development and to relevant groups facing disparity. It should also balance consistency across time and place with sensitivity to temporal and geographic context. The committee identified the following characteristics as crucial to striking this balance:
- measure multiple dimensions of educational outcomes and opportunities, including changes over time;
- focus on disparities between the population subgroups most salient for policy attention;
- use measures that are comparable across time and place, and useful at several organizational scales (classrooms, schools, districts, states, nation);
- use indicators and measures appropriate to grade level;
- measure contextual and structural characteristics of or affecting the educational system, such as racial segregation and concentrated poverty;
- produce frequent, readily understood, high-level summary statistics, in addition to more nuanced statistics;
- be based on scientifically sound measures; and
- incorporate mechanisms for continuous improvement based on research and other developments.
CONCLUSION 2-3: Existing data collection programs and related publications present a mixed picture with regard to their ability to support the committee’s proposed set of K–12 educational equity indicators.
The committee’s charge calls for us to identify equity indicators for pre-K to grade 12 and then on to the transition to postsecondary education and work. We use a step ladder as a visual depiction of the stages of education from preschool to graduation and to emphasize important characteristics of the indicators we are proposing: see Figure 2-2.
- First, at each step, there are a multitude of outcomes that we could have selected. Since the point of an equity indicator system is to highlight a small set of statistics that are most useful for addressing disparities, we selected outcomes we judged to be milestones, critical for success at the next successive step. We based these judgments on our reviews of the literature, including both qualitative and quantitative empirical research, edited volumes that summarize this research, and thought pieces about research findings by experts in the field.
- Second, the outcomes reflect a progression: achievement at one step builds on achievement at the preceding step and in turn serves as the building block for the next step. What one learns in one step is cumulative and carries into next step.
- Third, the outcomes are predictive. The indicators at one stage are predictive (or strongly related to) the indicators at the next stage, and ultimately, they should be predictive of educational attainment.
At each step, we considered multiple dimensions of learning and achievement. Those dimensions included achievement in multiple subject areas (reading, math, science), multiple measures of achievement (e.g., performance in coursework, performance on standardized tests), and multiple aspects of learning, such as showing achievement progress and being engaged in what one is learning. We group factors related to student outcomes into three domains:
- Domain A: Kindergarten Readiness
- Domain B: K–12 Learning and Engagement
- Domain C: Educational Attainment
Societal conditions such as income inequality and residential segregation intersect with the educational process in ways that have profound implications for efforts to reduce group disparities in educational progress, achievement, and attainment. These conditions lead to disparities in the resources that are available to support children’s learning and development in their families, schools, and neighborhoods. Combined with the differential treatment of children and parents who are racial, ethnic, and linguistic minorities, these disparities contribute to between-group differences in educational outcomes. Between-group differences in education outcomes are important to understand and monitor because they reflect differences in the structure of educational opportunity—thus signaling disparities in the education system. The goals for education that we have described are
easier to attain when students have access to well-resourced, high-quality preschool experiences and K–12 schools.
Many students do not have access to these important resources, and their access is strongly related to life circumstances beyond their control—where they live, their parents’ education and income, their race or ethnicity, and the language spoken in their home. Schools cannot be expected to address the root causes of income inequality, residential segregation, or structural racism, but as long as these conditions exist, schools and school systems must grapple with their effects. Without meaningful actions on the part of schools, communities, and states, the education system will simply replicate societal disparities. Chapters 3 through 5 explore these factors in more detail. Chapter 3 discusses ways in which family, neighborhood, and societal factors shape a child or student’s context in ways that affect school readiness and student outcomes generally. Several dimensions of these domains lack an adequate consensus or research base to fashion indicators and measures at this time. That is work for the future, contributing to continuous improvement of the indicator system.
We group factors related to opportunity and resources in the education system into four domains:
- Domain D: Extent of Racial, Ethnic, and Economic Segregation
- Domain E: Equitable Access to High-Quality Early Learning Programs
- Domain F: Equitable Access to High-Quality Curricula and Instruction
- Domain G: Equitable Access to Supportive School and Classroom Environments