Data and Methodological Opportunities and Challenges for Developing K–12 Educational Equity Indicators
To make the committee’s conclusions and recommendations more concrete, this appendix illustrates for each of the committee’s seven recommended domains and 16 indicators the data sources and methods that could be used to measure relevant constructs in appropriate ways. In some instances, the currently available data support proxy measures rather than measures that more directly capture the recommended indicator, or good measures are available but not for all student groups of interest, or the data are very sparse at the scale needed for an educational equity system—that is, comparable, high-quality information nationwide for the nation, states, school districts, and schools. When possible, we indicate ways in which available data could be enhanced or scaled up to fill the gaps. We also note when indicators are included in the publications reviewed in Appendix B.
DOMAIN A: KINDERGARTEN READINESS
The research literature amply supports the importance of kindergarten readiness, both academically and behaviorally, for children’s continued educational success (see Chapter 4). Yet this is a domain for which the data are sparse for the committee’s two proposed indicators: academic readiness (Indicator 1) and behavioral readiness (Indicator 2).
Indicator 1: Disparities in Academic Readiness
At present, there is no satisfactory data source for developing measures of the two constructs under this indicator—reading/literacy skills
and numeracy/math skills—for all levels of needed geography and student groups. Although some states and districts in the country currently assess the literacy and numeracy skills of their entering kindergarten students, there are no broadly used assessments that could provide comparable results nationwide.
The Early Childhood Longitudinal Study: Kindergarten Class of 2010–2011 (ECLS-K:2011; see Appendix A) of the National Center for Education Statistics (NCES) included assessments of sampled students entering kindergarten that school year for reading, math, and science. The assessments contained items developed for the ECLS, items adapted from commercial assessments, and items adapted from other NCES studies within assessment frameworks based on the National Assessment of Educational Progress (NAEP; see Appendix A), the ECLS-K (kindergarten cohort for 1998-1999; see Appendix A), and selected states’ curriculum standards.
Because of small sample sizes, the NCES longitudinal surveys program itself is able to provide only national estimates; also, entering kindergarten students are tested only periodically as samples for new cohorts are drawn. The national portrait of equity on early literacy and numeracy skills provided by ECLS-K:2011 tests could form a model for continued, standardized testing nationwide. It would be important to balance the need for age-appropriate individual assessment tools against the need for tools that could be feasibly used at a nationwide scale.
Indicator 2: Disparities in Self-Regulation and Attention Skills
As with early literacy and numeracy skills, there is at present no data source for developing measures of the two constructs under this indicator—self-regulation and attention skills for students entering kindergarten. The ECLS-K:2011 included direct assessments of kindergartner’s social skills (e.g., social interaction, attentional focus, and self-control) and problem behaviors (e.g., impulsivity and externalizing problem behaviors). It also included parents’ and teachers’ assessments of such learning behaviors as the ability to keep belongings organized and work independently. Some states and districts have independently developed assessments of kindergartners’ behavioral readiness skills. Research and development could perhaps result in streamlined assessments for use in schools nationwide, but the road to that outcome would be challenging.
DOMAIN B: K–12 LEARNING AND ENGAGEMENT
Monitoring students’ achievement as they progress through K–12 is essential for signaling whether they are on track or whether interventions are needed for one or more student groups of interest. From the available
research (see Chapter 4), the committee identified three indicators for which measures of constructs are needed for several grades, such as grades 4, 8, and 10, or for levels, such as elementary school, middle school, and after the first year of high school. The three indicators are engagement (Indicator 3), performance in course work (Indicator 4), and performance on tests (Indicator 5).
Indicator 3: Disparities in Engagement in Schooling
For one of the two constructs under this indicator—academic engagement—data are not readily available on the scale that is needed to develop measures as students proceed through K–12. However, there are developments that may change the picture. The National Center on Safe, Supportive, and Learning Environments, operated by the American Institutes for Research (AIR) for the U.S. Department of Education’s Office of Safe and Healthy Schools, has a data collection initiative that is relevant.
In 2015 NCES piloted what is now called the ED School Climate Surveys (EDSCS) in 50 schools. Based on this work, the AIR center offers tested survey instruments and a data reporting platform to states, school districts, and schools to survey school climate as seen by 5th- to 12th-grade students, staff, and parents. The center maintains a list with links to school climate surveys conducted by states, and NCES itself plans to conduct a national-level survey. The EDSCS includes engagement as a topic, including measures of relationships among students, teachers, families, and schools, participation in school, and respect for diversity. At present, the EDSCS is provided as a resource, with an explicit promise that a jurisdiction’s results will not be seen by the U.S. Department of Education unless the jurisdiction chooses to make its results public.1
For the other construct under Indicator 3—attendance/absenteeism, which captures the inverse of student engagement—relevant data are available. The Civil Rights Data Collection (CRDC) program in the U.S. Department of Education’s Office of Civil Rights (see Appendix A) collected information on chronic absenteeism (defined as missing 15 or more days of school a year) for the 2013-2014 and 2015-2016 school years. Going forward, data on chronic absenteeism are being collected annually through EDFacts as part of each state’s reporting requirements under the Elementary and Secondary Education Act, as amended by the Every Student Succeeds
The data in EDFacts could be used to develop appropriate measures of engagement (disengagement) for students categorized by gender, race and ethnicity, disability status, and English-language learner status. The location of schools (e.g., urban, suburban, town, rural) could also be used as a reporting classification, as could school level (elementary, middle, secondary, other—the data are not collected by grade). Although EDFacts does not collect data on absenteeism for students classified by a measure of socioeconomic status, it could be possible to use poverty estimates from the American Community Survey (ACS; see Appendix A) for school districts and school attendance areas as a proxy (see discussion under Indicator 8, below).
Indicator 4: Disparities in Performance in Coursework
Chapter 4 reviewed the literature on the role that continued academic success in school courses plays in enabling students to graduate on time from high school and be ready for college or other postsecondary pursuits. From this review, the committee concluded that measures of the following three constructs, obtained at several grades or levels, would most appropriately indicate performance in course work: success in classes, accumulating credits (being on track to graduate), and grade point average (GPA).
While the available data cannot now be used for developing measures that aggregate from student-level information, the CRDC has two directly relevant variables for groups of students. These variables are collected biannually for schools and districts and disaggregated by gender, race, disability status, and English-language learner status: number of students enrolled in and passing algebra I in middle school (separately for grades 7 and 8); and number of students enrolled in and passing algebra I in grades 9–10.4
Other CRDC variables that might serve as proxy measures (although they depend on school offerings—see Domain F, below) include students
2 See Chang, Bauer, and Byrnes (2018) for an analysis of the CRDC data on chronic absenteeism, which puts schools into five categories, from low chronic absence (0-4.9% of students meeting the 15 or more days absent definition) to extreme chronic absence (30%+ of students meeting the definition). They find that poverty relates strongly to high rates of chronic absenteeism.
3 The Council of the Great City Schools includes absenteeism among its Academic Key Performance Indicators based on a survey of its members (see Appendix B, Table B-3). Absenteeism—from the CRDC to date— is also an indicator in the NCES publication, Status and Trends in the Education of Racial and Ethnic Groups (see Appendix B, Table B-7).
enrolled in gifted and talented programs, in the International Baccalaureate (IB) Diploma Program, and in at least one Advanced Placement (AP) course and who took an AP exam.5 The CRDC also collects data on the number of students retained in each grade, which could be used as a measure of lack of progress in course work.
At present, readily available information on course passing, accumulation of credits, and GPA is only available in national-level data sources, such as NAEP transcript studies, transcript studies conducted as part of various NCES longitudinal surveys, and parental reports in the Parent and Family Involvement in Education Survey, conducted periodically as part of the National Household Education Surveys (see Appendix A). As more states fully develop their Statewide Longitudinal Data Systems (SLDS, see Appendix A) and arrange for access to the data for statistical purposes, it should be very possible to develop the three measures of passing courses, accumulating credits, and GPA.
Indicator 5: Disparities in Performance on Tests
Given the focus over the past few decades on testing and assessment of student performance, there is a plethora of data on student test scores on reading, math, and other subjects for schools and school districts. The problem for a nationwide indicator system is that states do not all use the same tests, so that some way to make the results comparable across states is needed. The researchers behind the Stanford Education Data Archive (see Appendix A) have used NAEP test results to develop calibration factors for interpreting state test results. These factors, applied to state test scores, make the adjusted state scores more nearly comparable across states. For example, one state’s test scores might need to be multiplied by a factor of 0.9 because the NAEP results indicate that that state’s test gives higher scores than are justified by how students in the state perform on NAEP. Conversely, another state’s test scores might need to be multiplied by a factor of 1.1 because the NAEP results indicate that that state’s test gives lower scores than are merited by how the students in the state perform on NAEP. For purposes of an ongoing educational equity indicator system, there would be a need to periodically review the calibrations to take account of changes in state tests. Another issue for presentation of measures is how best to display the results: for example, in terms of the percentage of
5 The Council of the Great City Schools includes participation in and passing of AP courses in its Academic Key Performance Indicators based on a survey of its members (see Appendix B, Table B-3). America’s Children, produced by the Federal Interagency Forum on Child and Family Statistics, includes percentages of high school students enrolled in selected mathematics and science courses as an indicator from the CRDC (see Appendix B, Table B-5).
students scoring higher than the proficient level or by using another metric. Relatedly, there is the issue of how to capture improvement (or not) over time (see discussion of Indicator 5 in Chapter 4).
DOMAIN C: EDUCATIONAL ATTAINMENT
In Chapter 4 the committee identifies two outcomes of the K–12 education system for which it is important to measure equity among student groups of interest: on-time high school graduation (Indicator 6) and postsecondary readiness (Indicator 7).
Indicator 6: Disparities in On-Time Graduation
The current standard for measuring high school graduation rates, developed by NCES after research and consultation with stakeholders and introduced for the 2010-2011 school year, is the adjusted cohort graduation rate (ACGR). The ACGR represents the percentage of students in a state (adjusted for migration) who enter the 9th grade and earn a regular diploma in that state within 4 years. The measure is also calculated for school districts and schools. As an example, a school’s ACGR for the school year 2017-2018 would be calculated as:
ACGR rates are readily available for school districts and states from the Common Core of Data (CCD; see Appendix A) for most student groups of interest, and the rates are included in many publications (see Appendix B). Rates are also available for schools from EDFacts but not broken down for student groups of interest. Presumably, school rates for student groups of interest could be made available from the SLDS.
Indicator 7: Disparities in Postsecondary Readiness
The committee concluded (see Chapter 4) that perhaps the most useful construct to measure regarding disparities in postsecondary readiness is whether young people are actually enrolled in college, or employed, or enlisted in the military immediately following high school graduation. The ACS provides information for the nation, states, and school districts on education and employment status; however, this information cannot be tied back to the individual’s high school. The most promising source for a useful measure would likely be the SLDS in states that are tracking students beyond high school.
DOMAIN D: EXTENT OF RACIAL, ETHNIC, AND ECONOMIC SEGREGATION
Indicator 8: Disparities in Students’ Exposure to Racial, Ethnic, and Economic Segregation
As discussed in Chapter 5, to capture fully the aspects of school segregation that can adversely affect student outcomes and increase resource needs for schools, the committee concluded that it would be useful to develop measures for two constructs—namely, concentration of poverty in schools and racial segregation within and across schools.
For concentration of poverty in schools, a widely used measure to date has been the percentage of students eligible for free and reduced-price school lunches under the National School Lunch Program (NSLP). This measure, however, is less and less useful for this purpose for several reasons: the eligibility thresholds for reduced-price and free lunches are 185 percent and 130 percent, respectively, of the official poverty threshold; the percentage of enrolled students may vary as a function of the outreach and encouragement of each school and district to eligible families; enrollment tends to drop off with age due to stigma for older students; and more and more schools and districts are taking advantage of a provision in the NSLP program to provide free lunches to all students in schools with high percentages of eligible students in order to reduce the burden and stigma of application and verification.6
What would be preferable for schools is a direct measure of the percentage of poor students to use to assign schools to a few categories—say, low, medium, and high percentages of poor students, tagged as, say, “little poverty,” “less concentrated poverty,” and “highly concentrated poverty.” Then, for multi-school districts, states, and the nation, the measure would
be the percentages of students attending schools in each category. With knowledge of school attendance areas and how they correspond to census tracts and block groups, it would be possible to use estimates from the Small-Area Income and Poverty Estimates program (SAIPE; see Appendix A)—or estimates constructed using SAIPE methods—to categorize schools where the attendance boundaries closely overlap the geographic areas recognized in the ACS.7 A related method, which would categorize all schools, would be to provide address information for students attending a school to the Census Bureau to keep secure and use to model the school’s poverty percentage using ACS and administrative records data.
For racial segregation within and across schools, there are extensive data available in virtually every data set the committee reviewed (see Appendix A). The challenge is to develop a measure that most nearly relates to the deleterious effects of racial segregation, including determination of which racial and ethnic groups to use in the measure and which percentage values for, say, high, medium, and low racial segregation are most useful.
DOMAIN E: EQUITABLE ACCESS TO HIGH-QUALITY EARLY LEARNING PROGRAMS
Indicator 9: Disparities in Access to and Participation in High-Quality Pre-K Programs
Licensed pre-K programs include those offered by school districts, Head Start programs, and other programs licensed by their state. The CRDC provides biannual measures of whether school districts offer preschool together with the enrollment, and ages covered for most student groups of interest. Given that a sizable number of states and districts do not offer and no state requires enrollment in pre-K, a simple measure of how many students aged 3-5 are enrolled in a pre-K program offered by the district could be a barebones proxy for this indicator. The National Institute for Early Education Research (NIEER) at Rutgers University has a program to survey states about their pre-K programs, and the measures developed as part of that effort (see Appendix B, Table B-8) could suggest paths forward for this domain.
DOMAIN F: EQUITABLE ACCESS TO HIGH-QUALITY CURRICULA AND INSTRUCTION
As discussed in Chapter 5, for all students to have an equitable opportunity to succeed, school systems need to offer high-quality curricula and
instruction. Specifically, the committee concluded that school systems need to provide the following four things: effective teaching (Indicator 10); access to and enrollment in rigorous coursework (Indicator 11); curricular breadth (Indicator 12); and access to high-quality academic interventions and support (Indicator 13). Without these features, students will be at a disadvantage relative to other students if they wish to pursue postsecondary education and training.
Indicator 10: Disparities in Access to Effective Teaching
It is well known that teacher effectiveness matters a great deal for students’ engagement with and achievement in the K–12 education system. The challenge lies in identifying constructs and measures for them that capture actual effectiveness and are feasible to obtain on a comparable basis nationwide. The committee concluded that measures of the following three constructs, obtained at several grades or levels, would most appropriately indicate effective teaching: teacher experience, teacher certification in the subjects they teach, and the racial and ethnic diversity of the teaching staff and how well teachers match their students in term of race and ethnicity.
The CRDC, biannually, and the CCD, annually, both obtain information on teacher experience and training at the school level (see Appendix A). The NCES National Teacher and Principal Survey (NTPS; see Appendix A) also obtains data on teachers’ experience and training, biannually, together with their demographic characteristics such as race and ethnicity, although the sample size only permits national estimates. The SLDS in some states includes teacher as well as student characteristics, which would make it possible to construct measures of teacher diversity and teachers’ racial and ethnic match with students.
Finally, states all have systems in place to measure teacher effectiveness directly, such as value-added models, student ratings in surveys, and classroom observations (see Chapter 5). It could be possible to develop a measure of teacher effectiveness using one or another of these methods, according to their use in a state. Although the resulting measures would not be uniform, they could still provide useful information.
All of these measures would need to be constructed on a school basis. Corresponding measures for multi-school districts, states, and the nation would be the percentage of students in each group of interest attending schools with effective teachers. For example, a measure might be the percentage of students attending schools that have low, moderate, or high percentages of teachers with at least a specified number of years teaching.
Indicator 11: Disparities in Access to and Enrollment in Rigorous Coursework
Opportunities to successfully enroll in and complete postsecondary education or training are very dependent on having access to required preparatory courses starting in middle school—for example, access to algebra courses in middle school or the first year of high school at the latest and access to AP or IB courses in high school. If students, particularly English-language learners, are tracked into less rigorous courses, it can be a barrier to achievement.
The CRDC obtains relevant measures biannually, including availability of and enrollment in AP courses, IB courses, and dual enrollment programs and enrollment in algebra 1 in grades 7–8 and in grades 9–10 (see Appendix A). Information on tracking is not readily available. As with Indicator 10 (above), measures of access to rigorous coursework would need to be constructed at the school level, with measures for larger areas cast in terms of the percentages of students (in each group of interest) enrolled in schools with the applicable characteristic, such as availability of AP classes. Enrollment measures for larger areas could be constructed in terms of percentages of students attending schools with no, low, medium, and high percentages of students enrolled in, say, AP courses. Alternatively, they could be constructed more directly, as the percentage of students enrolled in, say, AP courses among students attending schools that offer such courses.
Indicator 12: Disparities in Curricular Breadth
Chapter 5 discusses the value of a broad curriculum, covering much more than reading, math, and standardized test preparation, for students’ educational achievement and ability to function well as adults. As the committee concluded, although it is not known which specific combination of courses is best for students’ long-term outcomes, no educational system should differentially deprive students of exposure to a broad range of subjects. A measure of curricular breadth could be developed by examining state standards for subject offerings and determining the extent to which schools serving less advantaged students either do not offer some kinds of courses at all (e.g., social studies, art, a broad range of languages) or spend less time on courses other than reading and math in comparison with schools serving more advantaged students.
The CRDC biannually collects data for high schools (grades 9–12) on the number of classes in biology, chemistry, and physics combined and enrollment in those classes; it also collects data on the number of computer science classes and their enrollment. Other than those data, information on
classes in social studies, art, languages, geography, and other subjects is not readily available, for elementary, middle, or high schools. It is possible that measures could be constructed from the SLDS.
Indicator 13: Disparities in Access to High-Quality Academic Supports
In addition to effective teaching, access to rigorous coursework, and curricular breadth, it is important for schools and districts to provide high-quality academic interventions and support, such as supplemental tutoring, enrichment programs or activities, additional instructional time, and personalized academic counseling, including college and career counseling. In addition, it is important for English-language learners and students with disabilities to receive the most appropriate mix of core and specialized instruction and not be isolated in instructional ghettos.
Only limited data are currently available with which to construct appropriate measures of these constructs. For example, the CRDC has information on numbers of FTE instructional aides and their aggregate salaries, which could be used to assess the additional instructional resources that are available to students (see discussion under Indicator 16, below, of how such a measure might be constructed). The CRDC also has information about access to and enrollment in various courses for student groups, including English-language learners and students with disabilities, which could help identify the extent to which these groups are receiving appropriate academic support.
DOMAIN G: EQUITABLE ACCESS TO SUPPORTIVE SCHOOLS AND CLASSROOMS
Chapter 5 discusses how supportive schools and classrooms are important for good educational outcomes, particularly for disadvantaged children. The committee identified three indicators in this domain: supportive school climates (Indicator 14); nonexclusionary discipline practices (Indicator 15); and supports for student success (Indicator 16) (other than academic supports, which are covered under Indicator 13).
Indicator 14: Disparities in School Climate
Definitions of school climate vary widely (see Chapter 5), but, in general, “climate” refers to the way that a school feels to students, the adults who work in the school, and students’ families. Aspects of climate can include safety, supportiveness of staff, an academically focused culture, absence of harassment and discrimination, connectedness among students and staff, sense of fairness, and trust of adults and peers.
Although climate measures are not routinely collected by schools across the United States, several states have adopted climate measures for use in their accountability systems under the Every Student Succeeds Act, and many school districts also administer climate surveys. The National Center on Safe, Supportive, and Learning Environments in the U.S. Department of Education, has an initiative (see description under Indicator 3, above) to provide tested questions and other aids to states to administer climate surveys of middle and high school students, instructional staff, and parents or guardians. The three topic areas for which questions are available include engagement (see Indicator 3, above), safety, and the school environment. The relevant topics under safety include emotional safety, physical safety, bullying/cyberbullying, and substance abuse; the relevant topics under environment include physical environment and instructional environment. Another topic under environment is disciplinary practices, which is relevant to Indicator 15 (see below).
The CRDC currently provides comparable data for all schools and districts on some aspects of school climate. Specifically, there are data on harassment and bullying and school safety, which could be aggregated into one or more scales, based on research into which factors most strongly affect student outcomes. These data are provided by school administrators, so they represent documented instances of, for example, harassment and bullying or various kinds of violence. As such, they represent limited measures of school climate. More robust measures that capture the full spectrum of school climate to use to categorize schools as having, say, a strongly supportive climate, moderately supportive climate, or hostile climate, would require work with the states and the National Center on Safe, Supportive, and Learning Environments to develop survey measures that are as comparable as possible across jurisdictions and feasible to administer at a nationwide scale.
Indicator 15: Disparities in Nonexclusionary Discipline Practices
A school’s approach to student discipline can influence students’ opportunities to learn (see discussion in Chapter 5). Such exclusionary discipline policies as in- or out-of-school suspension remove students from the classroom, thereby reducing their opportunities to learn and to become engaged in their school work. As a result, these practices could negatively affect student learning and other outcomes for students who are subjected to them. It is currently not possible to measure schools’ use of nonexclusionary disciplinary policies, the extent to which teachers are trained to use nonpunitive approaches, or the extent to which they effectively implement these approaches.
It is possible, however, to use suspension and expulsion rates to measure the lack of nonexclusionary methods. States are required to report
those rates biannually for school districts and schools to the Office of Civil Rights as part of the CRDC, including, specifically: counts of K–12 students with and without disabilities who received one or more than one out-of-school suspension or who were expelled with educational services, without educational services, or because of zero-tolerance policies. Both sets of counts are reported by race, gender, and English-language learner status. These data could be used to classify schools into categories, such as low, moderate, and high percentages of suspended and expelled students; they could also be used to report student groups (e.g., by race and ethnicity) who are suspended or expelled at rates above, about the same, or below the average for their school, district, state, and the nation.8
Indicator 16: Disparities in Nonacademic Supports for Student Success
As discussed in Chapter 5, schools that serve students from poor families, students lacking in English proficiency, and students with special needs due to one or more disabilities require resources to ensure that those students have an opportunity to learn and achieve. The range of supports that schools could offer to ensure student success is almost boundless—especially in schools in which the student population has multiple needs. One category of support focuses on academics, which is the focus of Indicator 13, above. Another category relates to supporting students’ socioemotional development through specific curricular programs and other means. A third type of support relates to meeting the emotional and behavioral needs of students who are exposed to violence and other stressors in their homes and neighborhoods—for example, screening (using, e.g., the Adverse Childhood Experiences tool)9 and providing onsite counseling or appropriate referral services to students. A fourth type of support addresses students’ physical health—for example, through dental or medical screenings for students who otherwise may not have access to such screenings. Of course, these supports all require resources, principally for staff.
Currently, the CRDC obtains relevant counts of non-instructional staff support for schools and districts, including number of FTE school counselors, psychologists, nurses, and social workers. The CRDC also obtains total salaries funded with state or local funds for support services staff (e.g.,
8 The Council of the Great City Schools includes percent of students with out-of-school suspensions by number of days suspended in its Academic Key Performance Indicators based on responses to a survey of its members (see Appendix B, Table B-3). The NCES report on Status and Trends in the Education of Race and Ethnic Groups also includes percent of students by grade who received out-of-school suspensions from the CRDC (see Appendix B, Table B-7).
counselors).10 When coupled with a measure of the percentage of students in a school who are poor or classified as English-language learners or with a disability, the above information could be used to categorize schools (and the students attending them) by the ratio of their resources to their students’ needs.
Working out the technical details of an appropriate measure of nonacademic supports for student success would be challenging, but one approach could proceed something like the following.11 For a within-state indicator based on state and local (but not federal) funding, start with the statewide average per pupil costs of support staff, determine an average to allocate per nonpoor, non-English-language-learning, and nondisabled students, which would be lower than the overall average, and an average to allocate per poor, English-language-learning, and disabled students, which would be higher than the overall average. Then, determine each school’s proportion of extra-needs students and non-extra-needs students, apply the appropriate per pupil dollar amount to each group, and calculate the overall average for the school. Finally, examine the school’s actual per pupil costs of support staff and compare it to the needs-based ratio. Schools could then be classified as having more than adequate resources for the non-academic needs of their student body, adequate resources, or less than adequate resources.
10 The financial portion of the CCD, conducted by the Census Bureau, provides data on staff expenditures that are federally funded, but only at the district level and not for individual schools.
11 Examining the approaches used by the Education Law Center and Rutgers University for the Is Funding Fair? series of annual reports could also be helpful to suggest useful measures of resources relative to needs (see Appendix B).