National Academies Press: OpenBook

Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress (1999)

Chapter: 3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP

« Previous: 2 Streamlining the Design of NAEP
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

3

Enhancing the Participation and Meaningful Assessment of All Students in NAEP

Summary Conclusion 3. NAEP has the goal of reporting results that reflect the achievement of all students in the nation. However, many students with disabilities and English-language learners have been excluded from the assessments. Some steps have been taken recently to expand the participation of these students in NAEP, but their performance remains largely invisible.

Summary Recommendation 3. NAEP should enhance the participation, appropriate assessment, and meaningful interpretation of data for students with disabilities and English-language learners. NAEP and the proposed system of education indicators should include measures that improve understanding of the performance and educational needs of these populations.

INTRODUCTION

Over the past decade, national concern about and attention to the assessment of students with disabilities and English-language learners has intensified, paralleling the growth in numbers of students with limited English proficiency1 and

1  

The most commonly used term to refer to students who come from language backgrounds other than English and whose English proficiency is not yet developed to the point at which they can profit fully from English-only instruction is limited English proficient (LEP). LEP is the term used in many national and state data collections, federal and state legislation, and court cases involving these students (National Research Council and Institute of Medicine, 1997). In this report, we refer to these students as English-language learners, which is a more positive term. We consider the terms ''English-language learner'' and "student with limited English proficiency" to be synonymous.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

the numbers of students who are identified with physical, learning, or emotional disabilities. This concern and attention is also in large measure a response to several pieces of legislation passed by the United States Congress, which specifically required the participation of all students in assessments used to measure student performance.

Enacted by Congress in 1994, the Goals 2000: Educate America Act (P.L. 103-227) provided resources to implement systemic education reforms to help all students meet challenging academic standards; it specifically provided funds to support the participation and accommodation of students with disabilities and English-language learners in assessments. Similarly, the Perkins Act (P.L. 98-524), although primarily focusing on vocational and technical education, mandated the use of appropriate methodologies in testing both students with disabilities and English-language learners. Both Title I and Title VII of the Improving America's Schools Act of 1994 (P.L. 103-328) state the need to assess all children and to provide reasonable adaptations and accommodations for students with disabilities, as well as for children who are in the process of learning English. The Department of Education Organization Act of 1994 specifically states that the secretary must ensure that English-language learners are included in assessments in ways that are valid, reliable, and fair (National Research Council and Institute of Medicine, 1997:132-133). Other legislation, such as the Americans with Disabilities Act (1990) and the Individuals with Disabilities Education Act (as amended, 1997) specifically urges the participation and reasonable accommodation of students with disabilities.

The 1997 amendments to the Individuals with Disabilities Education Act (IDEA) include provisions intended to increase the participation of students with disabilities in state-and district-wide assessment programs, with appropriate accommodations when necessary. The individualized education plans (IEPs) of students with disabilities, which are required by law for each child with a disability, must include statements of any accommodations or other modifications needed by the student to participate in the state-and district-wide assessments. If the student's IEP team determines that the student cannot participate, the IEP plan must include a statement of why the student will not participate and describe how the student will be assessed. Alternate assessments for these students must be in place not later than July 1, 2000. Finally, states must ensure proper reporting of information regarding the performance of students with disabilities on large-scale assessments.

NAEP and other assessment programs thus have a clear federal mandate to enhance the participation and meaningful assessment of English-language learners and students with disabilities. This participation is especially important for NAEP. NAEP's mission, to serve as a key indicator of the academic achievement of the nation's students, can be satisfactorily accomplished only if the assessment results include and portray data gathered from all groups of students, including students with disabilities and English-language learners.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

In this chapter we present an analysis of NAEP's progress, through the 1996 assessments, to enhance the participation and accommodation of students with disabilities and English-language learners. We begin by presenting population data that make clear why the participation of these students is needed to provide an accurate view of national-level achievement. We then review progress in participation and accommodation that has been made in large-scale assessment programs across the nation, recounting NAEP's efforts and accomplishments during the 1990s. We discuss the underlying challenge of accurate and consistent identification and classification of English-language learners and students with disabilities, and close the chapter by presenting a series of recommendations to guide NAEP's continuing efforts on this front.

ENGLISH-LANGUAGE LEARNERS AND STUDENTS WITH DISABILITIES

An examination of recent trends in the nation's student population provides pointed insight as to why better understanding of the educational achievements and experiences of students with disabilities and English-language learners are needed. Students with disabilities are approximately 12 percent of the kindergarten through grade 12 (K-12) student population (Olson and Goldstein, 1997:154). In recent years, the numbers of students participating in federal programs for students with disabilities have increased at a faster rate than total K-12 public school enrollment, at least in part because students with disabilities are increasingly better identified. Between 1977 and 1995, the number of students with disabilities increased by 47 percent and the total public school population decreased by 2 percent. During that same period, the percentage of children with specific learning disabilities increased from 1.8 to 5.7 percent of the total public K-12 enrollment, and those with speech and language impairments and mental retardation decreased slightly (Olson and Goldstein, 1997:154). Better understanding of the performance of these students and their place in the educational system is needed, because 25 percent of children ages 5 to 17 who have a disabling condition repeat at least one grade in school (Olson and Goldstein, 1997:54), and students with disabilities have higher dropout rates and lower graduation rates (Valdés et al., 1990; U.S. Department of Education, 1996).

In 1991, 2.3 million students—5.5 percent of the total K-12 student population—were classified as English-language learners (Fleischman and Hopstock, 1993:10). Nearly 1.7 million (73 percent) of these English-language learners are native speakers of Spanish. No other language is spoken by more than 4 percent of English-language learners (Fleischman and Hopstock, 1993:11). The percentage of English-language learners in the K-12 population decreases from 8.4 percent in kindergarten to 6.0 percent in grade 4, 4.2 percent in grade 8, and 3.2 percent in grade 12 (see Table 3-1; Fleischman and Hopstock, 1993:10).

English-language learners are concentrated in the West, in urban areas, and

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

TABLE 3-1 English-Language Learners in Each Grade Level

Grade Level

Number of English-Language Learners

Percentage of English-Language Learners in Grade Level

Total Students in U.S.

Percentage of English-Language Learners of Total Students

Kindergarten

277,914

12.1

3,305,619

8.4%

1st grade

279,257

12.1

3,554,274

7.9

2nd grade

246,979

10.7

3,359,193

7.4

3rd grade

221,936

9.6

3,333,285

6.7

4th grade

197,211

8.6

3,312,443

6.0

5th grade

177,412

7.7

3,268,381

5.4

6th grade

150,421

6.5

3,238,095

4.6

7th grade

134,907

5.9

3,180,120

4.2

8th grade

125,849

5.5

3,019,826

4.2

9th grade

159,208

6.9

3,310,290

4.8

10th grade

137,101

5.9

2,913,951

4.7

11th grade

103,337

4.5

2,642,554

3.9

12th grade

75,423

3.3

2,390,329

3.2

Ungraded

16,469

0.7

Total

2,303,424

100.0%

40,828,360

5.5%

NOTES: Data based on mail survey to school districts. Data for the grades at which main NAEP assessments are administered are highlighted in bold-faced type.

SOURCE: Fleischman and Hopstock (1993:10).

in large schools with 750 or more students. Schools with 20 percent or more minority students and 20 percent or more students receiving free or reduced-price lunches are also more likely to enroll larger proportions of English-language learners. A total of 42 percent of all public school teachers have at least one English-language learner in their classes; 7 percent of these teachers have classes in which over 50 percent of their students are identified as English-language learners (Olson and Goldstein, 1997).

This group of students warrants national attention because a large number of non-English-speaking children have both low levels of academic performance in English and high dropout rates. On average, English-language learners are classified as underachievers by their teachers and receive lower grades. They also score below their classmates on standardized mathematics and reading tests (Bradby, 1992).

In summary, students with disabilities and English-language learners comprise a significant proportion of the students in U.S. classrooms (12 percent and 5.5 percent, respectively). Their achievement must be reflected in NAEP results. A NAEP that does not include these students is a report card for only 85 to 90 percent of the nation's students.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

EFFORTS TO ENHANCE PARTICIPATION IN NAEP AND OTHER LARGE-SCALE ASSESSMENTS

In the last several years, a number of activities have focused on increasing the participation of students with disabilities and English-language learners in large-scale assessments from which they had previously been excluded. These activities have focused on approaches that can be used in developing and administering assessments in ways that are meaningful, challenging, and appropriate for all students. A number of interested offices within the U.S. Department of Education have strongly supported emerging efforts to increase the representation of these groups of students in NAEP and other large-scale assessments, including the Office of Educational Research and Improvement, the Office of Civil Rights, the Office of Bilingual Education and Minority Languages Affairs, the Office of Special Education and Rehabilitative Services, the Office of the General Counsel, and the National Center for Education Statistics. Outside the Department of Education, a number of organizations have also contributed to these efforts. These activities underscore the importance of the problem for various groups around the country, as well as the need to explore thoroughly the feasibility of developing valid and reliable procedures for including previously excluded students.

Two reports issued by the National Research Council in 1997 provide an important foundation from which strategies for enhancing participation and accommodation of students with disabilities and English-language learners can be built. Educating One and All: Students with Disabilities and Standards-Based Reform (National Research Council, 1997) provides a review and analysis of the current status of students with disabilities in assessment systems. The report presents two broad recommendations that provide guidance for program-specific efforts, such as those we describe later in this chapter for NAEP:

Even if the individual needs of some students require alterations of the common standards and assessments, the committee strongly recommends that these students should be counted in a universal, public accountability system (pp. 9-10). Assessment accommodations should be provided, but they should be used only to offset the impact of disabilities unrelated to the knowledge and skills being measured. They should also be justified on a case-by-case basis, but individual decisions should be guided by a uniform set of criteria (p. 10).

The second report, Improving Schooling for Language-Minority Children: A Research Agenda (National Research Council and Institute of Medicine, 1997) summarizes the state of knowledge and key issues in the assessment of English-language proficiency and subject-matter knowledge for language-minority students. The report also provides a detailed agenda for research on both classroom-level and large-scale assessments. One recommendation in particular captures the key challenge for NAEP and other large-scale assessments:

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Research is needed to develop assessments and assessment procedures that incorporate more English-language learners. Further, research is needed toward developing guidelines for determining when English-language learners are ready to take the same assessments as their English-proficient peers and when versions of the assessment other than the "standard" English version should be administered (p. 130).

To date, a number of different activities have taken place that parallel the directions recommended in these reports. Efforts to increase the participation of students with disabilities and English-language learners in both NAEP and other large-scale assessments have included a number of conferences, reports, commissioned working papers, and funded studies. Table 3-2 summarizes a number of efforts made in the past 15 years to increase the participation of students with disabilities in large-scale assessments. Table 3-3 summarizes similar efforts to increase the participation of English-language learners in large-scale assessments. For additional information in this area, a clear and thorough summary and analysis of recent progress is presented in the 1997 National Center for Education Statistics (NCES) report, The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress (Olson and Goldstein, 1997).

REVIEW OF PROGRESS THROUGH 1996

Prior to 1995, NAEP was administered in classroom-sized sessions as a timed assessment, exclusively in English, and without testing accommodations or adaptations. Schools had therefore been allowed to exclude students from NAEP if, in the judgment of knowledgeable school personnel, such students could not meaningfully participate in the assessment; NAEP personnel did provide specific criteria that schools were expected to use to inform their judgments about which students to exclude from the assessment. As a result of the implementation of these rules, 44 percent of students with disabilities and 41 percent of English-language learners were not included in the 1994 NAEP assessment (Mazzeo, 1997). It was also clear that the application of the criteria for excluding students varied widely across states, districts, and schools, affecting the state NAEP comparisons in unquantifiable ways.

For NAEP, the need to undertake efforts to enhance the participation of students with disabilities and English-language learners was stimulated in part by research from the National Academy of Education's evaluations of NAEP. These studies showed that large numbers of excluded students were capable of taking the NAEP assessment, some with various accommodations and others with no accommodations (National Academy of Education, 1993, 1996, 1997). Also, studies from the National Center on Educational Outcomes showed that up to 85 percent of traditionally excluded students were, in fact, capable of participating in large-scale assessments with appropriate accommodations (National Center on

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

TABLE 3-2 Efforts and Activities Directed at Increasing the Participation of Students with Disabilities in Large-Scale Assessments

Date

Title

Focus

Reports and Papers

1980s

Testing Persons with Disabilities: A Report for ETS Programs and Their Constituents (ETS, n.d.)

Summarizes findings of a four-year study on accommodations and test scores.

1988

Testing Handicapped People (Willingham et al., 1988)

Reports on ETS studies conducted during the 1980s in the SAT and GRE testing programs.

1993

Testing Accommodations for Students with Disabilities: A Review of the Literature (NCEO SR4 1993)

Literature review. Recommends that guidelines be developed on both exclusion and inclusion, on accommodations and adaptations, and on score reporting.

1994

Making Decisions about the Inclusion of Students with Disabilities in Large-Scale Assessments: A Report on a Working Conference to Develop Guidelines on Inclusion and Accommodations (NCEO SR13 1994b)

Reports on the 1994 NCES conference listed below. Contains six main recommendations.

1994

Recommendations for Making Decisions about the Participation of Students with Disabilities in Statewide Assessment Programs (NCEO SR 15 1994a)

Includes recommendations for including and accommodating students with disabilities and for reporting results.

1995

A Compilation of States' Guidelines for Including Students with Disabilities in Assessments (NCEO SR17 1995b)

Survey of states' guidelines and policies.

1995

A Compilation of States' Guidelines for Accommodations in Assessments for Students with Disabilities (NCEO SR18 1995a)

Survey of states' guidelines and policies.

1995

Working Paper on Assessing Students with Disabilities and Limited English Proficiency (Houser, 1995)

NCES-commissioned paper. Discusses current NCES policies of exclusion of students with disabilities and LEP students.

1996

Statewide Assessment of Students with Disabilities (Bond, 1996)

Reports on data from the Association of State Assessment Programs. Thirty-seven states reported using special testing accommodations.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Date

Title

Focus

1997

The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress (Olson and Goldstein, 1997)

NCES-commissioned report. Summarizes recent progress.

Conference

1994

Working Conference on Guidelines for Inclusion of Students with Disabilities and Accommodations in Large-Scale Assessment Programs

NCES-sponsored conference.

Studies and Projects

1997

Study of Exclusion and Assessability of Students with Disabilities in the 1994 Trial State Assessment of the National Assessment of Educational Progress (Stancavage et al., 1997b)

National Academy of Education-sponsored study. Examines exclusion and assessability of students in 1994 trial state assessment in reading. Findings suggested that 83 percent of fourth grade students with individualized education programs would have been assessable on the NAEP reading instrument based on their reading scores. Includes information about teachers' views on appropriate accommodations.

1997

Educating One and All: Students with Disabilities and Standards-based Reform (National Research Council, 1997)

Topics covered include accountability and assessment, assessment in standards-based reform, and implications of increased participation of students with disabilities in local and large-scale assessments.

Ongoing

Office of Educational Research and Improvement-funded projects designed to examine the exclusion/inclusion of students with disabilities in state assessments.

Projects funded in the following states: Delaware, Maryland, Minnesota, North Dakota, Oregon, Pennsylvania.

Ongoing

State Collaborative on Assessment and Student Standards (SCASS)

Involves consortium of 20 states with CCSSO coordinating work on inclusion of students with disabilities in state assessments. SCASS Consortium on Technical Guidelines for Performance Assessment sponsors research projects focusing on common issues.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Date

Title

Focus

On-going

Investigating the Validity of the Accommodation of Oral Presentation (Weston, 1997)

Research proposal. Focuses on validating oral presentation as an accommodation used with students with disabilities in the category of learning-disabled.

CCSSO: Council of Chief State School Officers

NCEO: National Center for Educational Outcomes

SR: Summary Report

Educational Outcomes, 1994a). Over the past few years, NAEP has taken significant steps to implement enhanced inclusion and accommodations. These are summarized below; it is clear that these efforts have contributed much important information, but they have raised important questions as well (for a more detailed description of these efforts, see Olson and Goldstein, 1997).

The Puerto Rico Special Assessment

In 1994 a special project was carried out involving the development of a Spanish-language mathematics assessment instrument for use in Puerto Rico (the Puerto Rico Assessment of Educational Progress— PRAEP). NAEP assessment items and background questions were translated by staff in Puerto Rico into the Puerto Rican dialect of Spanish. Blocks of mathematics items used in previous assessments were adapted for use in PRAEP. All administration and data collection procedures used were similar to those used in NAEP. Extensive analyses of the data were conducted including examination of responses to background questions, as well as standard item analyses and differential item functioning analyses for the mathematics items. Item response theory (IRT) analyses were performed and results placed on a NAEP-like scale.

A number of problem areas were identified, including the finding that some items were found to have inappropriate translations or content that was not meaningful for students in Puerto Rico (Anderson and Olson, 1996). Researchers also concluded that "it was not possible to express the IRT results for PRAEP on the same scales as the NAEP results. In other words, the scales established for reporting Puerto Rico results were unique for that jurisdiction" (Olson and Goldstein, 1997:69).

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

TABLE 3-3 Efforts and Activities Directed at Increasing the Participation of English-Language Learners in Large-Scale Assessments

Date

Title

Focus

Reports and Papers

1991

Summary of State Practices Concerning the Assessment of and the Data Collection about LEP Students (CCSSO, 1991)

CCSSO-sponsored report.

1994

The Feasibility of Collecting Comparable National Statistics about Students with Limited English Proficiency (Cheung et al., 1994)

CCSSO-sponsored report.

1994

Issues in the Development of Spanish-Language Versions of the National Assessment of Educational Progress (Secada, 1994)

National Academy of Education-sponsored. Recommends pilot study.

1994

A Study of Eligibility Exclusions and Sampling: 1992 Trial State Assessment (Spencer, 1994)

National Academy of Education-sponsored. Recommends examination of the cost-benefit analysis implicit in the exclusion of English-language learners.

1994

For All Students: Limited English Proficient Students and Goals 2000 (August et al., 1994)

Makes recommendations about assessments for English-language learners.

1995

Assessment Practices: Developing and Modifying Statewide Assessment for LEP Students (Hafner, 1995)

Paper presented at CCSSO conference.

1995

Working Paper on Assessing Students with Disabilities and Limited English Proficiency (Houser, 1995)

NCES-commissioned paper. Discusses current NCES policies of exclusion of students with disabilities and English-language learners.

1996

Quality and Utility: The 1994 Trial State Assessment in Reading (National Academy of Education, 1996)

Reports that a high proportion of English-language learners would have been assessable. States that most disturbing finding is the exclusion of students with four or more years in English-speaking schools.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Date

Title

Focus

1996

Proceedings from the Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students in NAEP (August and McArthur, 1996)

Reports on the 1994 NCES conference listed below. Group cautioned on assessing students in their native language and voiced concern about translations.

1996

The Status Report of Assessment Programs in the United States: State Student Assessment Programs Database, School Year 1994-1995 (Bond et al., 1996)

Reports on data from the Association of State Assessment Programs. Thirty-seven states allowed for the exclusion of English-language learners. A smaller number of accommodations were provided than for students with disabilities. Only 4 states allowed the use of other languages.

1997

The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress (Olsen and Goldstein, 1977)

NCES-commissioned report. Summarizes recent progress.

1997

A Study Design to Evaluate Strategies for the Inclusion of LEP Students in the NAEP State Trial Assessment (Hakuta and Valdés, 1997)

Suggests general approaches for studying inclusion.

Conference

1994

NCES Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students

NCES-sponsored conference.

Studies and Projects

1997

The Impact of the Linguistic Features of the NAEP Test Items on Student's Performance in NAEP Assessments (Abedi et al., 1997)

Study sponsored by the National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Concludes that the language of mathematics may disproportionately impact the scores of less language-proficient students.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Date

Title

Focus

1997

Study of Exclusion and Assessability of Students with Limited English Proficiency in the 1994 Trial State Assessment of the National Assessment of Education Progress (Stancavage et al., 1997a)

National Academy of Education-sponsored study. Examines exclusion and assessability of students in 1994 trial state assessment of reading. Findings suggest that students enrolled in bilingual programs are more likely to be excluded from NAEP.

1997

Statewide Assessment Programs: Policies and Practices for the Inclusion of LEP Students

Center for Equity and Excellence (CEEE) sponsored study. Presents findings on state assessment policies.

1997

Accommodation Strategies for English Language Learners on Large-Scale Assessments: Student Characteristics and Other Considerations (Butler and Stevens, 1997)

CRESST-sponsored study.

1997

Improving Schooling for Language Minority Students: A Research Agenda (National Research Council and Institute of Medicine, 1997)

Topics covered include the measurement of English language proficiency, student assessment, and inclusion of students in large-scale assessments.

On-going

State Collaborative on Assessment and Student Standards (SCASS)

Involves consortium of 20 states with CCSSO coordinating work on inclusion of English-language learners in state assessments. SCASS Consortium on Technical Guidelines for Performance Assessment sponsors research projects focusing on common issues.

CCSSO: Council of Chief State School Officers

The 1995 Mathematics Field Test

As part of the 1995 mathematics field test for the 1996 national and state NAEP assessments, NAEP undertook efforts (1) to include more students in the assessment by providing a range of accommodations and (2) to increase the consistency with which decisions about inclusion were made. These efforts attempted to clearly define which students to include and to align these criteria more closely with those of state testing programs.

In order to achieve these goals, the inclusion criteria were modified, beginning with the 1995 NAEP field test. As Olson and Goldstein (1997:61) point out,

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

prior to 1995, the procedures used by NAEP to determine who could participate in the assessment ''were based on criteria for excluding students.'' Beginning with the 1995 field test, "the criteria were revised with the intention of making appropriate and consistent decisions about the inclusion of students with disabilities and LEP students" [emphasis added]. Schools were instructed to include all English-language learners and students with disabilities who, in the judgment of school staff, were capable of taking the assessment. They were also instructed to err on the side of inclusion. The old and new exclusion or inclusion rules for NAEP are summarized in Table 3-4.

In the 1995 field test, the following accommodations were provided if they were part of the students' normal testing procedure:

For students with disabilities:

  • extra testing time,

  • multiple sessions,

  • individual or small group administrations,

  • allowing a facilitator to read directions, items, and/or interpret diagrams or graphs,

  • allowing students to give responses orally, using sign language, or point to the response option,

  • allowing students to give answers using a special mechanical apparatus, (e.g., a tape recorder, braille typewriter, computer, etc.),

  • large-print booklets and large-face calculators, and

  • braille booklets and talking calculators.

For English-language learners:

  • extra testing time,

  • multiple sessions,

  • individual or small group administrations,

  • allowing a facilitator to read directions, items, and/or interpret diagrams or graphs,

  • Spanish-English bilingual assessment booklets, with items in different languages printed on facing pages, and

  • Spanish-only assessment booklets.

A number of these accommodations were offered simultaneously, for example, students assessed in bilingual sessions using Spanish-English bilingual assessment booklets could also be given extra time.

The results of the field test—involving both a study of English-language learners and a study of students with disabilities who had individual education plans (IEPs) —indicated that accommodation strategies and procedures could be

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

TABLE 3-4 Old and New Exclusion or Inclusion Rules for NAEP

Students

Old (1990-1994)

New (1995-1996)

 

Exclude if:

Include if:

Students with Disabilities

—Student is mainstreamed less than 50 percent of the time in academic subjects and is judged incapable of participating meaningfully in the assessment, OR

—IEP team or equivalent group determines that the student is incapable of participating meaningfully in the assessment.

—Student has an IEP, unless the IEP team or equivalent group determine that the student cannot participate, or if the student's cognitive functioning is so severely impaired that he or she cannot participate, even with accommodations.

English-Language Learners

—Student is native speaker of a language other than English, AND—Enrolled in an English-speaking school (not including bilingual education program) for less than two years, AND

—Judged to be incapable of taking part in the assessment.

—Student has received academic instruction primarily in English for at least three years, OR

—Student has received academic instruction in English for less than three years, if school staff determine that the student is capable of participating in the assessment in English, OR

—Student, whose native language is Spanish, has received academic instruction in English for less than three years, if school staff determine that the student is capable of participating in the assessment in Spanish (if available).

IEP: Individualized education program.

SOURCE: Olson and Goldstein (1997:62).

implemented as part of the NAEP assessment and would allow for the participation of more students in the national assessment.

The study of English-language learners (Anderson et al., 1996) sought to examine the feasibility of using translated versions of NAEP instruments. The study included a close examination of the Spanish language characteristics of the items themselves in order to identify patterns of difficulty in producing equivalent

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

items. The study also included a study of the degree to which it would be possible to scale data from a bilingual version or a Spanish-only version of the mathematics items to fit the NAEP reporting scale.

The issue of scalability was examined by scrutinizing item statistics (average item score and item-total correlation) and by comparing empirical item responses for English-language learners to the IRT functions of the respective English items. The results of the first analysis revealed that, for grade 4 study blocks (both bilingual and Spanish), 71 percent of the items were not parallel to the English version and, for grade 8, 76 percent of the items were not parallel to their English-language analogues. Andersen et al. (1996:31) concluded "[that] the translated versions of the assessment are not parallel in measurement properties to the English version and that scores are not comparable." More recent research (Abedi et al., 1997) has also revealed that simple grammatical translations of NAEP mathematics items to produce Spanish-language versions of the assessment are problematic, especially for linguistically complex items; lexical and semantic translations deserve further exploration as strategies for future accommodations for English-language learners are investigated.

The study of the field test results for students with disabilities (Anderson et al., 1996) provided limited information about psychometric issues because, for the group of students with disabilities, sample sizes were extremely small. The authors conclude that (p. 39):

descriptive information provides evidence that students with disabilities are responding to many of the items differently than are students in the full sample. This does not indicate noncomparable measurement if the difference exists only due to the IEP group being of lower ability (and thus having lower percent corrects), but the differences in item-total correlations and omit rates indicated that many items may be measuring differently for students with disabilities than for full-sample groups.

The results of the 1995 NAEP field test in mathematics also indicated that use of the new inclusion criteria and the use of accommodations may affect the measurement of trends in student achievement in two ways, and the magnitude of these effects is difficult to evaluate. First, trends could be altered by the inclusion of students who previously would not have participated in NAEP but now appear in the national sample. Second, the availability of accommodations to students who previously would have participated in NAEP without them may have unknown impacts on their achievement.

The 1996 NAEP Assessment

Despite outstanding questions regarding the impact on constructs measured under alternative assessment conditions for English-language learners and students with disabilities, NAEP appropriately decided to move ahead with the new

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

inclusion criteria and the testing accommodations for the 1996 mathematics and science assessments in a way that would both permit measurement of trends and further exploration of the effects of the new criteria and the accommodation procedures.

In 1996, national NAEP was administered to three subsamples. In Sample 1, as in previous NAEP administrations, old inclusion criteria were applied and no accommodations were provided (for trend measurement in mathematics); in Sample 2, the new inclusion criteria were applied, but no accommodations were provided (to measure the effect of the new criteria independent of the effect of providing accommodations); in Sample 3, the new inclusion criteria were applied and accommodations were provided (to measure the impact of accommodations above and beyond the new inclusion criteria). The numbers of students included and accommodated in the 1996 mathematics assessment are presented in Table 3-5.

For both students with disabilities and English-language learners and at all three grades, there were no significant differences between the percentages of students included in Sample 1 and Sample 2. Thus, the changes to the inclusion criteria did not have any impact on the rate of inclusion of these groups of students in the assessment. However, at grades 4 and 8, significantly greater percentages of both students with disabilities and English-language learners were included in Sample 3 as compared with Sample 2. Thus, the provision of an expanded array of accommodations positively affected participation in grades 4 and 8, although no similar increase occurred at grade 12 (Reese et al., 1997). Similar results were obtained for the 1996 NAEP science assessment (O'Sullivan et al., 1997). Providing accommodations moves NAEP solidly toward the goal of including all students; in Sample 3 at grade 4, only 4 percent of the national population was excluded from the assessment sample, and at grades 8 and 12, 3 percent were excluded.

The importance of enhanced participation through the provision of accommodations notwithstanding, it is important to note that nothing is known about how the accommodations affected students' performance on NAEP, compared with how they would have performed without accommodations. An analysis and discussion of the impact of accommodations on scaling and the actual and potential effects on the measurement of trends will be forthcoming from NCES (Mazzeo et al., 1998). However, the encouraging results presented here do appear to be laying a foundation for the operational use of accommodations and the accompanying enhanced participation of students with disabilities and English-language learners in future NAEP assessments.

PROBLEM OF CONSISTENT AND ACCURATE IDENTIFICATION

NAEP's efforts to date make clear that implementing enhanced inclusion and accommodation strategies for English-language learners and students with

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

TABLE 3-5 Percentage of Students with Disabilities and English-Language Learners in the National Population Included in the 1996 Main NAEP Mathematics Assessment, Public Schools Only

 

SD

     

ELL

 

Students

S1: Using Original Inclusion Criteria

S2: Using Revised Inclusion Criteriaa

S3: Using Revised Criteria And Providing Accommodations/Adaptions

S1: Using Original Inclusion Criteria

S2: Using Revised Inclusion Criteriaa

S3: Using Revised Criteria and Providing Accommodations/Adaptations

Grade 4

Assessed under standard conditions

58

47

35b,c

61

41

47

Assessed with accommodation

   

37

   

30

Total assessed

58

47

72b,c

61

41

76b

Grade 8

Assessed under standard conditions

55

58

46b

60

63

61

Assessed with accommodation

   

26

   

18

Total assessed

55

58

71b,c

60

63

78b

Grade 12

Assessed under standard conditions

48

51

35b

84

73

81

Assessed with accommodation

   

19

   

6

Total assessed

48

51

54

84

73

87

a Differences between S1 and S2 results are not statistically significant.

b Indicates a significant difference between S2 and S3 results.

c Indicates a significant difference between S1 and S3 results.

SOURCE: Reese et al. (1997:72).

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

disabilities is not an easy task. They also show that the provision of accommodations can increase the rate of participation of both students with disabilities and English-language learners in the data collections on which national-level summaries are based. However, even with the availability of a broader array of accommodations, the appropriate participation of students in assessments and the appropriate analysis and reporting of results for these groups depend directly on accurate and consistent methods for identifying, classifying, and assigning accommodations to these students. Unfortunately, according to a number of scholars (e.g., National Center on Educational Outcomes, 1995b; Reschly, 1996; National Research Council, 1997; National Research Council and Institute of Medicine, 1997), existing methods for identifying and classifying students with disabilities and for identifying students with different levels of proficiency in English are highly variable and often unsatisfactory. This is an issue faced not only by NAEP, but also by all large-scale assessment programs.

According to recent work carried out on students with disabilities (for example, Lewit and Baker, 1996; Reschly, 1996), no single classification system for special education is used uniformly around the country. Students labeled "disabled" in one state may not be so labeled in another. Moreover, according to Reschly (1996:44), 78 percent of children ages 6 to 11 who are classified as disabled in schools are so classified because of mild learning disabilities and speech and language disorders; such disabilities are the most difficult to diagnose accurately and consistently.

For NAEP, the impact of this lack of uniformity is of particular concern when considering state comparisons, since state-to-state variations in identifying, classifying, and determining appropriate accommodations for students with disabilities are little understood and have unknown impacts on inclusion rates, state comparisons, and summary results.

The identification and classification of English-language learners is similarly problematic. According to a number of recently conducted surveys (Fleischman and Hopstock, 1993; Cheung et al., 1994; Rivera, 1995; August and Lara, 1996), both states and districts use a variety of methods to determine whether students can be classified as an English-language learner and placed in special language programs. These methods include surveys of language(s) used at home, observations, interviews, referrals, grades, and testing. According to Fleischman and Hopstock (1993), 83 percent of school districts use English-language-proficiency examinations to identify English-language learners.

A total of 74 percent of school districts also use these tests for monitoring student progress and for reclassifying English-language learners as fluent in speaking English. Unfortunately, as the recent study of English-language learners also pointed out (National Research Council and Institute of Medicine, 1997), English-language-proficiency instruments are highly flawed and focus on discrete language skills that are not well attuned to the language demands faced by students in schools and classrooms. The report concluded that "most measures

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

used not only have been characterized by the measurement of decontextualized skills but also have set fairly low standards for language proficiency" (National Research Council and Institute of Medicine, 1997:118).

In addition, Valdés and Figueroa (1994) argue that a number of instruments currently used to assess the language proficiency of English-language learners tend to resemble paper-and-pencil tests administered orally. In spite of certain similarities, however, these instruments are quite different from each other and are based on often contradictory views about the nature of language competence. So different, indeed, are they that even the three instruments most widely used in California—i.e., the Bilingual Syntax Measure, the Basic Inventory of Natural Language, and the Language Assessment Scales classify very different proportions of students as non-English-speaking, limited-English-speaking, and fluent-English-speaking. All three of the measures, moreover, placed the very same students in different categories. So great were the discrepancies between the numbers of children included in the non-English-speaking and limited-English-speaking categories by different tests that cynical consultants often recommended (in jest) one "state-approved" instrument or another to school districts depending on whether administrators wanted to identify large or small numbers of English-language learners.

To date, the dilemmas described above have not been resolved. Children potentially in need of native language support are still being assessed at entry level using one of several instruments that many scholars have questioned, and some years later they are tested again using another of such instruments that is in no way comparable to the first. The field is no closer to developing means for assessing whether a child can or cannot function satisfactorily in an all-English program —or participate in all-English large-scale assessments—than it was in 1964.2

The problems associated with identifying and classifying students with disabilities and English-language learners reviewed here are not problems that the NAEP program can or should solve on its own. However, as a leader in the design and conduct of large-scale assessments, the NAEP program (and the National Center for Education Statistics and U.S. Department of Education) can act as a lever to push for improved accuracy and consistency of identification and classification methods. In the next section, we describe important next steps to enhance participation and accommodation in NAEP; however, it is important to keep in mind that improved identification and classification are a critical prerequisite if the goals of such enhancements are to be fully realized.

2  

Segments from the above discussion draw extensively from Valdés and Figueroa (1994).

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

GOALS FOR ENHANCING PARTICIPATION AND ACCOMMODATION

In response to recent federal legislation that reflects a national commitment to the education of students with disabilities and English-language learners—and given the fact that these groups of students collectively comprise more than 15 percent of the nation's students—the goals for the participation of these students in NAEP and their representation in a coordinated system of indicators for assessing educational progress should be:

  • to ensure that the national samples used in NAEP's standard large-scale surveys are as representative of the nation as possible, and thus that the overall proficiency scores that are reported include results from students with disabilities and English-language learners.

  • to collect data on the achievement of these special populations and to be able to make interpretive statements about how these students are performing relative to the nation and relative to their educational opportunities and instructional experiences.

To accomplish these goals, the U.S. Department of Education and NCES should further strengthen the good efforts already begun in NAEP to increase the participation rates and the provision of accommodations to students with disabilities and English-language learners. As part of this commitment, NCES should play a central role in the ongoing effort to improve data collection on these populations of students. In particular, NCES should work with states and other jurisdictions to improve the consistency of the identification and classification of these students, given the current variability in definitions of English-language proficiency and of having a disability. NCES should also press for further steps to ensure the consistent application of inclusion criteria for students with disabilities and students with language needs across all jurisdictions that participate in NAEP.

NAEP's goals in relation to the assessment of these populations via large-scale surveys should be to produce summary results that include data from as many students with disabilities and English-language learners as possible, and to report reliable subgroup information for these two populations, so that their performance relative to the nation as a whole can be known.

When students cannot be included in NAEP's large-scale survey assessments, or when the impact of the accommodation raises serious questions about the validity of combining data for these students with overall NAEP data, then the use of alternative assessment methods should be explored, targeting the assessment method to the particular subgroup of students in question. These assessments would serve as vehicles for gathering information about the achievements of these students and provide a basis for qualitative reports of the results. Although

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

such assessments and associated reports do not meet the same statistical standards as data from large-scale surveys and undoubtedly will be costly to develop and administer, these trade-offs are necessary in order to prevent the achievements of students not included in large-scale surveys from remaining invisible.

In addition to the current paucity of achievement data from students with disabilities and English-language learners, there is also a dearth of accompanying contextual data about these students' educational experiences that would allow educators and policy makers to better understand the performance and educational needs of these populations. An important focus of the integrated system of data collections that we propose in Chapter 1 is the reporting of information on the educational opportunities and instructional experiences of these students—designed to be linked to student achievement data from NAEP's large-scale survey assessments and the alternative assessments we have proposed.

Finally, NAEP's goals and plans for the participation, meaningful assessment, and reporting of results of students with disabilities and English-language learners should be clearly defined and broadly disseminated. NAEP's users should be made cognizant of the degree to which its national samples do or do not include these groups of students; in fact, a key measure in the assessment of educational progress should be the reporting of progress in the numbers of students with disabilities and English-language learners who participate in NAEP's large-scale surveys.

We recommend that (1) NAEP continue to strive to ensure that as many English-language learners and students with disabilities participate in NAEP's large-scale surveys as possible, (2) reliable subgroup results for these two groups be reported in conjunction with national Report Card results, (3) alternative assessments be developed for English-language learners and students with disabilities who cannot be included in the large-scale surveys, (4) contextual data regarding the educational experiences of these two groups of students be collected within the coordinated system of data collections that we have proposed in Chapter 1, (5) quantitative and qualitative reports of these students' achievements be prepared and disseminated, including contextual information that helps enhance the understanding of the educational needs of these students, and (6) NAEP's goals and plans for the participation, meaningful assessment, and reporting of results for these students be broadly disseminated.

The financial and operational ramifications of these goals are tremendous and the technical issues associated with implementing these goals are numerous. Reporting reliable summary achievement results for a single subgroup that includes all types of students with disabilities and for a single subgroup that includes all types of English-language learners requires extensive (and expensive) oversampling in order to produce samples of sufficient size to report reliable subgroup results at the national level. (The expense of producing reliable subgroup results at the state level may be prohibitive.) The development of alternative

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

methods for students who cannot be assessed in the large-scale survey with accommodations will undoubtedly require a large financial investment, and, for many students, it is still not known what alternatives are best suited for assessing their achievements. The analysis and reporting of these students' achievements in relation to contextual data is an added expense not currently in NAEP's annual budget. However, the social obligation and the legislative mandate to include these groups of students in assessments and to understand their achievements and educational needs makes striving to overcome these obstacles a necessary goal for the federal government and for NAEP. Neither NAEP nor NCES can finance the steps necessary to accomplish these goals within current budgets. Additional appropriations must be forthcoming if NAEP is to make progress toward the same kinds of goals for participation and assessment of students with disabilities and English-language learners that Congress has mandated for state and district assessment programs.

Even with the availability of additional funding, significant technical issues are associated with the goals we have put forth. For example, although the provision of accommodations appears to increase the participation of students in NAEP, research to date has not resolved whether data obtained using modified versions of assessments or altered administrative conditions have the same meaning (i.e., reflect measurement of the same constructs) as data collected using standard assessment materials and conditions. NAEP's research agenda must address this lack of knowledge.

In addition, both groups for which we have recommended reporting subgroup results—students with disabilities and English-language learners—are very heterogeneous populations. The achievements and educational needs of students with learning disabilities could differ greatly from that of students with physical disabilities, and, even within subcategories, the skills of students with different types of learning or physical disabilities can be very different. There also can be differences in the achievements and educational needs of English-language learners based on native language, as well as within native language groups, depending on how much English is spoken within their home or their specific ethnic identification within their language group (e.g., Puerto Rican, Chicano, or Central American within the Spanish-speaking language group).

Reporting information on all English-language learners as a subgroup and all students with disabilities as a subgroup must be accompanied by clear caveats about the lack of certainty regarding the generalizability of this information to all types of language groups or types of students with disabilities within these large, heterogeneous, subgroups. Still, overall subgroup results can help alert the nation as to how these groups of students are performing—in the aggregate—compared with the nation as a whole.

Also, as mentioned above, for those English-language learners and students with disabilities who cannot be included in the large-scale survey with accommodations, it is not yet well established which types of alternative assessment

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

methods are most appropriate for assessing different types of students within these major subgroups. States are now investigating and implementing such strategies in their own assessment programs, and NAEP should work in partnership with them to capitalize on progress to date in planning alternative assessment methods for NAEP.

These and additional technical issues are discussed further in the next section, which outlines a research agenda for enhancing the participation, meaningful assessment, and reporting of results for students with disabilities and English-language learners in NAEP and other large-scale assessments.

In previous chapters we described a new paradigm NAEP, one in which assessment methods were tightly matched to assessment purpose. We described a core NAEP in which large-scale surveys are an important method for reporting overall results in NAEP's core subjects. Based on the discussion in this chapter, we envision a core NAEP that includes as many students with disabilities and English-language learners as possible and reports reliable subgroup information for these two populations of students.

We also described a multiple-methods component of NAEP, and the discussion in this chapter outlines another purpose for such a component. NAEP should implement a variety of appropriate alternative assessment methods that capture the achievements of students with disabilities and English-language learners who cannot be included in the large-scale surveys (or whatever other methods are used to assess the general student population). Figure 3-1 shows the general structure of the new paradigm NAEP with multiple methods for assessing students with disabilities and English-language learners not included in the standard assessments.

A RESEARCH AGENDA

The implementation of the ambitious goals that we have outlined above cannot be accomplished without an increased federal commitment to and funding for research that extends beyond NAEP and NCES to the U.S. Department of Education. Much of this research has applications well beyond NAEP, extending to assessments at the state, district, and even classroom levels. NAEP clearly cannot bear the full responsibility for this research effort, but the program should serve as a leader in pushing the research agenda forward. Specifically, NAEP, NCES, and the U.S. Department of Education should define a research agenda that includes:

  • the determination of the most appropriate methods for assessing and providing accommodations to students with disabilities and English-language learners and

  • the effects of changes in inclusion criteria and accommodations over time on trends in achievement results.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

FIGURE 3-1 Measures of student achievement, including new paradigm NAEP. NOTE: TIMSS = Third International Mathematics and Science Study; NELS = National Education Longitudinal Study; ECLS = Early Childhood Longitudinal Study.

Methods for Assessing and Providing Accommodations

The research agenda to be defined must include attention to issues and questions such as the following:

  • The need for particular types of accommodations and the adequacy and appropriateness of the accommodations provided to various categories of students with disabilities and English-language learners. Research in this area must include work on the demands (linguistic and nonlinguistic) made by different assessments on students with disabilities and English-language learners. For example, the NAEP reading assessment may place very different demands on the language abilities of students than does the mathematics assessment. Research on accommodations must also include attention to how different groups of students respond to different kinds of accommodations (e.g., students with language and speech disorders versus students with specific learning disabilities, or recently arrived English-language learners versus those who have received instruction in English over several years). Work also should focus on ways in which the appropriateness of particular accommodations for particular groups of students can be determined. Attention should also be given to the examination of whether different accommodations can provide students with disabilities and English-language learners a fair opportunity to answer questions across the range of item difficulties being tested.

  • The validity of different types of accommodations. A reasonable accommodation should provide both students with disabilities and English-language

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

learners access to tests through an adaptation of the assessment itself or a modification of the administration procedure that does not change the nature of the construct being measured. The accommodated scores should not be an irrelevant measure of the disability or the language limitation of examinees, but a reflection of what examinees know and are able to do.

  • The feasibility and cost-effectiveness of particularly expensive accommodations such as translation. As discussed earlier in this chapter, the analysis of the recent use of translated versions of NAEP suggests that a number of nontrivial issues about the meaning of score data from translated assessments need to be resolved before the translation of instruments is adopted as a key accommodation method.

  • Scaling and reporting. Research in this area should include attention to such questions as: What is the impact of accommodations on scores for students with disabilities and English-language learners? Does the NAEP scale accurately reflect results for respondents assessed under nonstandard conditions? Can the scores of students with disabilities and English-language learners be combined with scores of the general population for reporting NAEP results? If results for accommodated students cannot be reported on the NAEP scale, how might these results be best reported?

  • Questions about access to the curriculum and opportunity to learn . Research in this area should include attention to such questions as: Do students with disabilities and English-language learners study the same curricula as other students? Is the content of the NAEP assessments appropriate for students with disabilities and English-language learners, given their educational experiences?

  • Alternative assessment methods for describing the achievements of students with disabilities and English-language learners who cannot participate in NAEP's assessments of the general student population. To date, the focus of research efforts has been on strategies for including English-language learners and students with disabilities in large-scale survey assessments. However, the development of methods for assessing and reporting results for the diverse body of students not able to participate in the general assessments must also become a focus of research. Such research has the added benefit that it could provide the foundations for a generation of methodologies that are designed to assess student performance in ways that are appropriate for everybody.

Effects of Changes in Inclusion Criteria and Accommodations

As discussed previously, changes in inclusion criteria and in the availability of accommodations can potentially affect the measurement of trend results in two ways: (1) trends could be altered by the inclusion of students who previously would not have participated in NAEP but now are included in the national sample and (2) the availability of accommodations to students who previously would

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

have participated in NAEP without them may have unknown impacts on their performance on the assessment.

Research currently being conducted as part of NAEP's explorations of enhanced participation and accommodations in the 1996 NAEP mathematics assessment will begin to address these issues. Comparisons of NAEP results—for the nation and for key subgroups—obtained from Sample 1 (old criteria, no accommodations) will be compared with those obtained from Sample 3 (new criteria, with accommodations). Equivalency of these two sets of results would indicate that the enhanced inclusion has no significant impact on overall NAEP scale score results and that trend lines could be continued with results from assessments administered with the new inclusion criteria and accommodations. If the two sets of results are not equivalent, then it may be necessary to continue to administer the assessment to both Sample 1 and Sample 3 in subsequent administrations or conduct other studies to gauge the effects of the use of new criteria and the provision of accommodations on trend lines. Results from this initial work will be forthcoming in fall 1998, but continued work of this nature will be required across NAEP's core subject-area assessments and over time to ensure that strategies for continuing trend lines are in place.

MAJOR CONCLUSIONS AND RECOMMENDATIONS

Conclusions

Conclusion 3A. The participation and accommodation of students with disabilities and English-language learners are necessary if NAEP results are to be representative of the nation's students. There is currently a paucity of interpretable achievement data and accompanying contextual data on the performance and educational needs of these populations.

Conclusion 3B. Enhanced participation of students with disabilities and English-language learners in NAEP depends on (1) the consistent application of well-defined criteria to identify these students and (2) accurate collection and reporting of information about them.

Recommendations

Recommendation 3A. NAEP should include sufficient numbers of students with disabilities and English-language learners in the large-scale assessment so that the results are representative of the nation and reliable subgroup information can be reported.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Recommendation 3B. Criteria for identifying students with disabilities and English-language learners for inclusion in the large-scale survey need to be more clearly defined and consistently applied.

Recommendation 3C. For those students who cannot participate in NAEP's standard large-scale surveys, appropriate, alternative methods should be devised for the ongoing collection of data on their achievement, educational opportunities, and instructional experiences.

Recommendation 3D. In order to accomplish the committee's recommendations, the NAEP program should investigate the following:

  • Methods for appropriately assessing, providing accommodations, and reporting on the achievements of students with disabilities and English-language learners, and

  • Effects of changes in inclusion criteria and accommodations on trends in achievement results.

Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 87
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 88
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 89
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 90
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 91
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 92
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 93
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 94
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 95
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 96
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 97
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 98
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 99
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 100
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 101
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 102
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 103
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 104
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 105
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 106
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 107
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 108
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 109
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 110
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 111
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 112
Suggested Citation:"3 Enhancing the Participation and Meaningful Assessment of all Students in NAEP." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 113
Next: 4 Frameworks and the Assessment Development Process: Providing More Informative Portrayals of Student Performance »
Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress Get This Book
×
Buy Hardback | $55.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since the late 1960s, the National Assessment of Educational Progress (NAEP)—the nation's report card—has been the only continuing measure of student achievement in key subject areas. Increasingly, educators and policymakers have expected NAEP to serve as a lever for education reform and many other purposes beyond its original role.

Grading the Nation's Report Card examines ways NAEP can be strengthened to provide more informative portrayals of student achievement and the school and system factors that influence it. The committee offers specific recommendations and strategies for improving NAEP's effectiveness and utility, including:

  • Linking achievement data to other education indicators.
  • Streamlining data collection and other aspects of its design.
  • Including students with disabilities and English-language learners.
  • Revamping the process by which achievement levels are set.

The book explores how to improve NAEP framework documents—which identify knowledge and skills to be assessed—with a clearer eye toward the inferences that will be drawn from the results.

What should the nation expect from NAEP? What should NAEP do to meet these expectations? This book provides a blueprint for a new paradigm, important to education policymakers, professors, and students, as well as school administrators and teachers, and education advocates.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!