3

Enhancing the Participation and Meaningful Assessment of All Students in NAEP

Summary Conclusion 3. NAEP has the goal of reporting results that reflect the achievement of all students in the nation. However, many students with disabilities and English-language learners have been excluded from the assessments. Some steps have been taken recently to expand the participation of these students in NAEP, but their performance remains largely invisible.

Summary Recommendation 3. NAEP should enhance the participation, appropriate assessment, and meaningful interpretation of data for students with disabilities and English-language learners. NAEP and the proposed system of education indicators should include measures that improve understanding of the performance and educational needs of these populations.

INTRODUCTION

Over the past decade, national concern about and attention to the assessment of students with disabilities and English-language learners has intensified, paralleling the growth in numbers of students with limited English proficiency1 and

1  

The most commonly used term to refer to students who come from language backgrounds other than English and whose English proficiency is not yet developed to the point at which they can profit fully from English-only instruction is limited English proficient (LEP). LEP is the term used in many national and state data collections, federal and state legislation, and court cases involving these students (National Research Council and Institute of Medicine, 1997). In this report, we refer to these students as English-language learners, which is a more positive term. We consider the terms ''English-language learner'' and "student with limited English proficiency" to be synonymous.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress 3 Enhancing the Participation and Meaningful Assessment of All Students in NAEP Summary Conclusion 3. NAEP has the goal of reporting results that reflect the achievement of all students in the nation. However, many students with disabilities and English-language learners have been excluded from the assessments. Some steps have been taken recently to expand the participation of these students in NAEP, but their performance remains largely invisible. Summary Recommendation 3. NAEP should enhance the participation, appropriate assessment, and meaningful interpretation of data for students with disabilities and English-language learners. NAEP and the proposed system of education indicators should include measures that improve understanding of the performance and educational needs of these populations. INTRODUCTION Over the past decade, national concern about and attention to the assessment of students with disabilities and English-language learners has intensified, paralleling the growth in numbers of students with limited English proficiency1 and 1   The most commonly used term to refer to students who come from language backgrounds other than English and whose English proficiency is not yet developed to the point at which they can profit fully from English-only instruction is limited English proficient (LEP). LEP is the term used in many national and state data collections, federal and state legislation, and court cases involving these students (National Research Council and Institute of Medicine, 1997). In this report, we refer to these students as English-language learners, which is a more positive term. We consider the terms ''English-language learner'' and "student with limited English proficiency" to be synonymous.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress the numbers of students who are identified with physical, learning, or emotional disabilities. This concern and attention is also in large measure a response to several pieces of legislation passed by the United States Congress, which specifically required the participation of all students in assessments used to measure student performance. Enacted by Congress in 1994, the Goals 2000: Educate America Act (P.L. 103-227) provided resources to implement systemic education reforms to help all students meet challenging academic standards; it specifically provided funds to support the participation and accommodation of students with disabilities and English-language learners in assessments. Similarly, the Perkins Act (P.L. 98-524), although primarily focusing on vocational and technical education, mandated the use of appropriate methodologies in testing both students with disabilities and English-language learners. Both Title I and Title VII of the Improving America's Schools Act of 1994 (P.L. 103-328) state the need to assess all children and to provide reasonable adaptations and accommodations for students with disabilities, as well as for children who are in the process of learning English. The Department of Education Organization Act of 1994 specifically states that the secretary must ensure that English-language learners are included in assessments in ways that are valid, reliable, and fair (National Research Council and Institute of Medicine, 1997:132-133). Other legislation, such as the Americans with Disabilities Act (1990) and the Individuals with Disabilities Education Act (as amended, 1997) specifically urges the participation and reasonable accommodation of students with disabilities. The 1997 amendments to the Individuals with Disabilities Education Act (IDEA) include provisions intended to increase the participation of students with disabilities in state-and district-wide assessment programs, with appropriate accommodations when necessary. The individualized education plans (IEPs) of students with disabilities, which are required by law for each child with a disability, must include statements of any accommodations or other modifications needed by the student to participate in the state-and district-wide assessments. If the student's IEP team determines that the student cannot participate, the IEP plan must include a statement of why the student will not participate and describe how the student will be assessed. Alternate assessments for these students must be in place not later than July 1, 2000. Finally, states must ensure proper reporting of information regarding the performance of students with disabilities on large-scale assessments. NAEP and other assessment programs thus have a clear federal mandate to enhance the participation and meaningful assessment of English-language learners and students with disabilities. This participation is especially important for NAEP. NAEP's mission, to serve as a key indicator of the academic achievement of the nation's students, can be satisfactorily accomplished only if the assessment results include and portray data gathered from all groups of students, including students with disabilities and English-language learners.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress In this chapter we present an analysis of NAEP's progress, through the 1996 assessments, to enhance the participation and accommodation of students with disabilities and English-language learners. We begin by presenting population data that make clear why the participation of these students is needed to provide an accurate view of national-level achievement. We then review progress in participation and accommodation that has been made in large-scale assessment programs across the nation, recounting NAEP's efforts and accomplishments during the 1990s. We discuss the underlying challenge of accurate and consistent identification and classification of English-language learners and students with disabilities, and close the chapter by presenting a series of recommendations to guide NAEP's continuing efforts on this front. ENGLISH-LANGUAGE LEARNERS AND STUDENTS WITH DISABILITIES An examination of recent trends in the nation's student population provides pointed insight as to why better understanding of the educational achievements and experiences of students with disabilities and English-language learners are needed. Students with disabilities are approximately 12 percent of the kindergarten through grade 12 (K-12) student population (Olson and Goldstein, 1997:154). In recent years, the numbers of students participating in federal programs for students with disabilities have increased at a faster rate than total K-12 public school enrollment, at least in part because students with disabilities are increasingly better identified. Between 1977 and 1995, the number of students with disabilities increased by 47 percent and the total public school population decreased by 2 percent. During that same period, the percentage of children with specific learning disabilities increased from 1.8 to 5.7 percent of the total public K-12 enrollment, and those with speech and language impairments and mental retardation decreased slightly (Olson and Goldstein, 1997:154). Better understanding of the performance of these students and their place in the educational system is needed, because 25 percent of children ages 5 to 17 who have a disabling condition repeat at least one grade in school (Olson and Goldstein, 1997:54), and students with disabilities have higher dropout rates and lower graduation rates (Valdés et al., 1990; U.S. Department of Education, 1996). In 1991, 2.3 million students—5.5 percent of the total K-12 student population—were classified as English-language learners (Fleischman and Hopstock, 1993:10). Nearly 1.7 million (73 percent) of these English-language learners are native speakers of Spanish. No other language is spoken by more than 4 percent of English-language learners (Fleischman and Hopstock, 1993:11). The percentage of English-language learners in the K-12 population decreases from 8.4 percent in kindergarten to 6.0 percent in grade 4, 4.2 percent in grade 8, and 3.2 percent in grade 12 (see Table 3-1; Fleischman and Hopstock, 1993:10). English-language learners are concentrated in the West, in urban areas, and

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress TABLE 3-1 English-Language Learners in Each Grade Level Grade Level Number of English-Language Learners Percentage of English-Language Learners in Grade Level Total Students in U.S. Percentage of English-Language Learners of Total Students Kindergarten 277,914 12.1 3,305,619 8.4% 1st grade 279,257 12.1 3,554,274 7.9 2nd grade 246,979 10.7 3,359,193 7.4 3rd grade 221,936 9.6 3,333,285 6.7 4th grade 197,211 8.6 3,312,443 6.0 5th grade 177,412 7.7 3,268,381 5.4 6th grade 150,421 6.5 3,238,095 4.6 7th grade 134,907 5.9 3,180,120 4.2 8th grade 125,849 5.5 3,019,826 4.2 9th grade 159,208 6.9 3,310,290 4.8 10th grade 137,101 5.9 2,913,951 4.7 11th grade 103,337 4.5 2,642,554 3.9 12th grade 75,423 3.3 2,390,329 3.2 Ungraded 16,469 0.7 — — Total 2,303,424 100.0% 40,828,360 5.5% NOTES: Data based on mail survey to school districts. Data for the grades at which main NAEP assessments are administered are highlighted in bold-faced type. SOURCE: Fleischman and Hopstock (1993:10). in large schools with 750 or more students. Schools with 20 percent or more minority students and 20 percent or more students receiving free or reduced-price lunches are also more likely to enroll larger proportions of English-language learners. A total of 42 percent of all public school teachers have at least one English-language learner in their classes; 7 percent of these teachers have classes in which over 50 percent of their students are identified as English-language learners (Olson and Goldstein, 1997). This group of students warrants national attention because a large number of non-English-speaking children have both low levels of academic performance in English and high dropout rates. On average, English-language learners are classified as underachievers by their teachers and receive lower grades. They also score below their classmates on standardized mathematics and reading tests (Bradby, 1992). In summary, students with disabilities and English-language learners comprise a significant proportion of the students in U.S. classrooms (12 percent and 5.5 percent, respectively). Their achievement must be reflected in NAEP results. A NAEP that does not include these students is a report card for only 85 to 90 percent of the nation's students.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress EFFORTS TO ENHANCE PARTICIPATION IN NAEP AND OTHER LARGE-SCALE ASSESSMENTS In the last several years, a number of activities have focused on increasing the participation of students with disabilities and English-language learners in large-scale assessments from which they had previously been excluded. These activities have focused on approaches that can be used in developing and administering assessments in ways that are meaningful, challenging, and appropriate for all students. A number of interested offices within the U.S. Department of Education have strongly supported emerging efforts to increase the representation of these groups of students in NAEP and other large-scale assessments, including the Office of Educational Research and Improvement, the Office of Civil Rights, the Office of Bilingual Education and Minority Languages Affairs, the Office of Special Education and Rehabilitative Services, the Office of the General Counsel, and the National Center for Education Statistics. Outside the Department of Education, a number of organizations have also contributed to these efforts. These activities underscore the importance of the problem for various groups around the country, as well as the need to explore thoroughly the feasibility of developing valid and reliable procedures for including previously excluded students. Two reports issued by the National Research Council in 1997 provide an important foundation from which strategies for enhancing participation and accommodation of students with disabilities and English-language learners can be built. Educating One and All: Students with Disabilities and Standards-Based Reform (National Research Council, 1997) provides a review and analysis of the current status of students with disabilities in assessment systems. The report presents two broad recommendations that provide guidance for program-specific efforts, such as those we describe later in this chapter for NAEP: Even if the individual needs of some students require alterations of the common standards and assessments, the committee strongly recommends that these students should be counted in a universal, public accountability system (pp. 9-10). Assessment accommodations should be provided, but they should be used only to offset the impact of disabilities unrelated to the knowledge and skills being measured. They should also be justified on a case-by-case basis, but individual decisions should be guided by a uniform set of criteria (p. 10). The second report, Improving Schooling for Language-Minority Children: A Research Agenda (National Research Council and Institute of Medicine, 1997) summarizes the state of knowledge and key issues in the assessment of English-language proficiency and subject-matter knowledge for language-minority students. The report also provides a detailed agenda for research on both classroom-level and large-scale assessments. One recommendation in particular captures the key challenge for NAEP and other large-scale assessments:

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Research is needed to develop assessments and assessment procedures that incorporate more English-language learners. Further, research is needed toward developing guidelines for determining when English-language learners are ready to take the same assessments as their English-proficient peers and when versions of the assessment other than the "standard" English version should be administered (p. 130). To date, a number of different activities have taken place that parallel the directions recommended in these reports. Efforts to increase the participation of students with disabilities and English-language learners in both NAEP and other large-scale assessments have included a number of conferences, reports, commissioned working papers, and funded studies. Table 3-2 summarizes a number of efforts made in the past 15 years to increase the participation of students with disabilities in large-scale assessments. Table 3-3 summarizes similar efforts to increase the participation of English-language learners in large-scale assessments. For additional information in this area, a clear and thorough summary and analysis of recent progress is presented in the 1997 National Center for Education Statistics (NCES) report, The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress (Olson and Goldstein, 1997). REVIEW OF PROGRESS THROUGH 1996 Prior to 1995, NAEP was administered in classroom-sized sessions as a timed assessment, exclusively in English, and without testing accommodations or adaptations. Schools had therefore been allowed to exclude students from NAEP if, in the judgment of knowledgeable school personnel, such students could not meaningfully participate in the assessment; NAEP personnel did provide specific criteria that schools were expected to use to inform their judgments about which students to exclude from the assessment. As a result of the implementation of these rules, 44 percent of students with disabilities and 41 percent of English-language learners were not included in the 1994 NAEP assessment (Mazzeo, 1997). It was also clear that the application of the criteria for excluding students varied widely across states, districts, and schools, affecting the state NAEP comparisons in unquantifiable ways. For NAEP, the need to undertake efforts to enhance the participation of students with disabilities and English-language learners was stimulated in part by research from the National Academy of Education's evaluations of NAEP. These studies showed that large numbers of excluded students were capable of taking the NAEP assessment, some with various accommodations and others with no accommodations (National Academy of Education, 1993, 1996, 1997). Also, studies from the National Center on Educational Outcomes showed that up to 85 percent of traditionally excluded students were, in fact, capable of participating in large-scale assessments with appropriate accommodations (National Center on

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress TABLE 3-2 Efforts and Activities Directed at Increasing the Participation of Students with Disabilities in Large-Scale Assessments Date Title Focus Reports and Papers 1980s Testing Persons with Disabilities: A Report for ETS Programs and Their Constituents (ETS, n.d.) Summarizes findings of a four-year study on accommodations and test scores. 1988 Testing Handicapped People (Willingham et al., 1988) Reports on ETS studies conducted during the 1980s in the SAT and GRE testing programs. 1993 Testing Accommodations for Students with Disabilities: A Review of the Literature (NCEO SR4 1993) Literature review. Recommends that guidelines be developed on both exclusion and inclusion, on accommodations and adaptations, and on score reporting. 1994 Making Decisions about the Inclusion of Students with Disabilities in Large-Scale Assessments: A Report on a Working Conference to Develop Guidelines on Inclusion and Accommodations (NCEO SR13 1994b) Reports on the 1994 NCES conference listed below. Contains six main recommendations. 1994 Recommendations for Making Decisions about the Participation of Students with Disabilities in Statewide Assessment Programs (NCEO SR 15 1994a) Includes recommendations for including and accommodating students with disabilities and for reporting results. 1995 A Compilation of States' Guidelines for Including Students with Disabilities in Assessments (NCEO SR17 1995b) Survey of states' guidelines and policies. 1995 A Compilation of States' Guidelines for Accommodations in Assessments for Students with Disabilities (NCEO SR18 1995a) Survey of states' guidelines and policies. 1995 Working Paper on Assessing Students with Disabilities and Limited English Proficiency (Houser, 1995) NCES-commissioned paper. Discusses current NCES policies of exclusion of students with disabilities and LEP students. 1996 Statewide Assessment of Students with Disabilities (Bond, 1996) Reports on data from the Association of State Assessment Programs. Thirty-seven states reported using special testing accommodations.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Date Title Focus 1997 The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress (Olson and Goldstein, 1997) NCES-commissioned report. Summarizes recent progress. Conference 1994 Working Conference on Guidelines for Inclusion of Students with Disabilities and Accommodations in Large-Scale Assessment Programs NCES-sponsored conference. Studies and Projects 1997 Study of Exclusion and Assessability of Students with Disabilities in the 1994 Trial State Assessment of the National Assessment of Educational Progress (Stancavage et al., 1997b) National Academy of Education-sponsored study. Examines exclusion and assessability of students in 1994 trial state assessment in reading. Findings suggested that 83 percent of fourth grade students with individualized education programs would have been assessable on the NAEP reading instrument based on their reading scores. Includes information about teachers' views on appropriate accommodations. 1997 Educating One and All: Students with Disabilities and Standards-based Reform (National Research Council, 1997) Topics covered include accountability and assessment, assessment in standards-based reform, and implications of increased participation of students with disabilities in local and large-scale assessments. Ongoing Office of Educational Research and Improvement-funded projects designed to examine the exclusion/inclusion of students with disabilities in state assessments. Projects funded in the following states: Delaware, Maryland, Minnesota, North Dakota, Oregon, Pennsylvania. Ongoing State Collaborative on Assessment and Student Standards (SCASS) Involves consortium of 20 states with CCSSO coordinating work on inclusion of students with disabilities in state assessments. SCASS Consortium on Technical Guidelines for Performance Assessment sponsors research projects focusing on common issues.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Date Title Focus On-going Investigating the Validity of the Accommodation of Oral Presentation (Weston, 1997) Research proposal. Focuses on validating oral presentation as an accommodation used with students with disabilities in the category of learning-disabled. CCSSO: Council of Chief State School Officers NCEO: National Center for Educational Outcomes SR: Summary Report Educational Outcomes, 1994a). Over the past few years, NAEP has taken significant steps to implement enhanced inclusion and accommodations. These are summarized below; it is clear that these efforts have contributed much important information, but they have raised important questions as well (for a more detailed description of these efforts, see Olson and Goldstein, 1997). The Puerto Rico Special Assessment In 1994 a special project was carried out involving the development of a Spanish-language mathematics assessment instrument for use in Puerto Rico (the Puerto Rico Assessment of Educational Progress— PRAEP). NAEP assessment items and background questions were translated by staff in Puerto Rico into the Puerto Rican dialect of Spanish. Blocks of mathematics items used in previous assessments were adapted for use in PRAEP. All administration and data collection procedures used were similar to those used in NAEP. Extensive analyses of the data were conducted including examination of responses to background questions, as well as standard item analyses and differential item functioning analyses for the mathematics items. Item response theory (IRT) analyses were performed and results placed on a NAEP-like scale. A number of problem areas were identified, including the finding that some items were found to have inappropriate translations or content that was not meaningful for students in Puerto Rico (Anderson and Olson, 1996). Researchers also concluded that "it was not possible to express the IRT results for PRAEP on the same scales as the NAEP results. In other words, the scales established for reporting Puerto Rico results were unique for that jurisdiction" (Olson and Goldstein, 1997:69).

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress TABLE 3-3 Efforts and Activities Directed at Increasing the Participation of English-Language Learners in Large-Scale Assessments Date Title Focus Reports and Papers 1991 Summary of State Practices Concerning the Assessment of and the Data Collection about LEP Students (CCSSO, 1991) CCSSO-sponsored report. 1994 The Feasibility of Collecting Comparable National Statistics about Students with Limited English Proficiency (Cheung et al., 1994) CCSSO-sponsored report. 1994 Issues in the Development of Spanish-Language Versions of the National Assessment of Educational Progress (Secada, 1994) National Academy of Education-sponsored. Recommends pilot study. 1994 A Study of Eligibility Exclusions and Sampling: 1992 Trial State Assessment (Spencer, 1994) National Academy of Education-sponsored. Recommends examination of the cost-benefit analysis implicit in the exclusion of English-language learners. 1994 For All Students: Limited English Proficient Students and Goals 2000 (August et al., 1994) Makes recommendations about assessments for English-language learners. 1995 Assessment Practices: Developing and Modifying Statewide Assessment for LEP Students (Hafner, 1995) Paper presented at CCSSO conference. 1995 Working Paper on Assessing Students with Disabilities and Limited English Proficiency (Houser, 1995) NCES-commissioned paper. Discusses current NCES policies of exclusion of students with disabilities and English-language learners. 1996 Quality and Utility: The 1994 Trial State Assessment in Reading (National Academy of Education, 1996) Reports that a high proportion of English-language learners would have been assessable. States that most disturbing finding is the exclusion of students with four or more years in English-speaking schools.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Date Title Focus 1996 Proceedings from the Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students in NAEP (August and McArthur, 1996) Reports on the 1994 NCES conference listed below. Group cautioned on assessing students in their native language and voiced concern about translations. 1996 The Status Report of Assessment Programs in the United States: State Student Assessment Programs Database, School Year 1994-1995 (Bond et al., 1996) Reports on data from the Association of State Assessment Programs. Thirty-seven states allowed for the exclusion of English-language learners. A smaller number of accommodations were provided than for students with disabilities. Only 4 states allowed the use of other languages. 1997 The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress (Olsen and Goldstein, 1977) NCES-commissioned report. Summarizes recent progress. 1997 A Study Design to Evaluate Strategies for the Inclusion of LEP Students in the NAEP State Trial Assessment (Hakuta and Valdés, 1997) Suggests general approaches for studying inclusion. Conference 1994 NCES Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students NCES-sponsored conference. Studies and Projects 1997 The Impact of the Linguistic Features of the NAEP Test Items on Student's Performance in NAEP Assessments (Abedi et al., 1997) Study sponsored by the National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Concludes that the language of mathematics may disproportionately impact the scores of less language-proficient students.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress TABLE 3-5 Percentage of Students with Disabilities and English-Language Learners in the National Population Included in the 1996 Main NAEP Mathematics Assessment, Public Schools Only   SD       ELL   Students S1: Using Original Inclusion Criteria S2: Using Revised Inclusion Criteriaa S3: Using Revised Criteria And Providing Accommodations/Adaptions S1: Using Original Inclusion Criteria S2: Using Revised Inclusion Criteriaa S3: Using Revised Criteria and Providing Accommodations/Adaptations Grade 4 Assessed under standard conditions 58 47 35b,c 61 41 47 Assessed with accommodation     37     30 Total assessed 58 47 72b,c 61 41 76b Grade 8 Assessed under standard conditions 55 58 46b 60 63 61 Assessed with accommodation     26     18 Total assessed 55 58 71b,c 60 63 78b Grade 12 Assessed under standard conditions 48 51 35b 84 73 81 Assessed with accommodation     19     6 Total assessed 48 51 54 84 73 87 a Differences between S1 and S2 results are not statistically significant. b Indicates a significant difference between S2 and S3 results. c Indicates a significant difference between S1 and S3 results. SOURCE: Reese et al. (1997:72).

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress disabilities is not an easy task. They also show that the provision of accommodations can increase the rate of participation of both students with disabilities and English-language learners in the data collections on which national-level summaries are based. However, even with the availability of a broader array of accommodations, the appropriate participation of students in assessments and the appropriate analysis and reporting of results for these groups depend directly on accurate and consistent methods for identifying, classifying, and assigning accommodations to these students. Unfortunately, according to a number of scholars (e.g., National Center on Educational Outcomes, 1995b; Reschly, 1996; National Research Council, 1997; National Research Council and Institute of Medicine, 1997), existing methods for identifying and classifying students with disabilities and for identifying students with different levels of proficiency in English are highly variable and often unsatisfactory. This is an issue faced not only by NAEP, but also by all large-scale assessment programs. According to recent work carried out on students with disabilities (for example, Lewit and Baker, 1996; Reschly, 1996), no single classification system for special education is used uniformly around the country. Students labeled "disabled" in one state may not be so labeled in another. Moreover, according to Reschly (1996:44), 78 percent of children ages 6 to 11 who are classified as disabled in schools are so classified because of mild learning disabilities and speech and language disorders; such disabilities are the most difficult to diagnose accurately and consistently. For NAEP, the impact of this lack of uniformity is of particular concern when considering state comparisons, since state-to-state variations in identifying, classifying, and determining appropriate accommodations for students with disabilities are little understood and have unknown impacts on inclusion rates, state comparisons, and summary results. The identification and classification of English-language learners is similarly problematic. According to a number of recently conducted surveys (Fleischman and Hopstock, 1993; Cheung et al., 1994; Rivera, 1995; August and Lara, 1996), both states and districts use a variety of methods to determine whether students can be classified as an English-language learner and placed in special language programs. These methods include surveys of language(s) used at home, observations, interviews, referrals, grades, and testing. According to Fleischman and Hopstock (1993), 83 percent of school districts use English-language-proficiency examinations to identify English-language learners. A total of 74 percent of school districts also use these tests for monitoring student progress and for reclassifying English-language learners as fluent in speaking English. Unfortunately, as the recent study of English-language learners also pointed out (National Research Council and Institute of Medicine, 1997), English-language-proficiency instruments are highly flawed and focus on discrete language skills that are not well attuned to the language demands faced by students in schools and classrooms. The report concluded that "most measures

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress used not only have been characterized by the measurement of decontextualized skills but also have set fairly low standards for language proficiency" (National Research Council and Institute of Medicine, 1997:118). In addition, Valdés and Figueroa (1994) argue that a number of instruments currently used to assess the language proficiency of English-language learners tend to resemble paper-and-pencil tests administered orally. In spite of certain similarities, however, these instruments are quite different from each other and are based on often contradictory views about the nature of language competence. So different, indeed, are they that even the three instruments most widely used in California—i.e., the Bilingual Syntax Measure, the Basic Inventory of Natural Language, and the Language Assessment Scales classify very different proportions of students as non-English-speaking, limited-English-speaking, and fluent-English-speaking. All three of the measures, moreover, placed the very same students in different categories. So great were the discrepancies between the numbers of children included in the non-English-speaking and limited-English-speaking categories by different tests that cynical consultants often recommended (in jest) one "state-approved" instrument or another to school districts depending on whether administrators wanted to identify large or small numbers of English-language learners. To date, the dilemmas described above have not been resolved. Children potentially in need of native language support are still being assessed at entry level using one of several instruments that many scholars have questioned, and some years later they are tested again using another of such instruments that is in no way comparable to the first. The field is no closer to developing means for assessing whether a child can or cannot function satisfactorily in an all-English program —or participate in all-English large-scale assessments—than it was in 1964.2 The problems associated with identifying and classifying students with disabilities and English-language learners reviewed here are not problems that the NAEP program can or should solve on its own. However, as a leader in the design and conduct of large-scale assessments, the NAEP program (and the National Center for Education Statistics and U.S. Department of Education) can act as a lever to push for improved accuracy and consistency of identification and classification methods. In the next section, we describe important next steps to enhance participation and accommodation in NAEP; however, it is important to keep in mind that improved identification and classification are a critical prerequisite if the goals of such enhancements are to be fully realized. 2   Segments from the above discussion draw extensively from Valdés and Figueroa (1994).

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress GOALS FOR ENHANCING PARTICIPATION AND ACCOMMODATION In response to recent federal legislation that reflects a national commitment to the education of students with disabilities and English-language learners—and given the fact that these groups of students collectively comprise more than 15 percent of the nation's students—the goals for the participation of these students in NAEP and their representation in a coordinated system of indicators for assessing educational progress should be: to ensure that the national samples used in NAEP's standard large-scale surveys are as representative of the nation as possible, and thus that the overall proficiency scores that are reported include results from students with disabilities and English-language learners. to collect data on the achievement of these special populations and to be able to make interpretive statements about how these students are performing relative to the nation and relative to their educational opportunities and instructional experiences. To accomplish these goals, the U.S. Department of Education and NCES should further strengthen the good efforts already begun in NAEP to increase the participation rates and the provision of accommodations to students with disabilities and English-language learners. As part of this commitment, NCES should play a central role in the ongoing effort to improve data collection on these populations of students. In particular, NCES should work with states and other jurisdictions to improve the consistency of the identification and classification of these students, given the current variability in definitions of English-language proficiency and of having a disability. NCES should also press for further steps to ensure the consistent application of inclusion criteria for students with disabilities and students with language needs across all jurisdictions that participate in NAEP. NAEP's goals in relation to the assessment of these populations via large-scale surveys should be to produce summary results that include data from as many students with disabilities and English-language learners as possible, and to report reliable subgroup information for these two populations, so that their performance relative to the nation as a whole can be known. When students cannot be included in NAEP's large-scale survey assessments, or when the impact of the accommodation raises serious questions about the validity of combining data for these students with overall NAEP data, then the use of alternative assessment methods should be explored, targeting the assessment method to the particular subgroup of students in question. These assessments would serve as vehicles for gathering information about the achievements of these students and provide a basis for qualitative reports of the results. Although

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress such assessments and associated reports do not meet the same statistical standards as data from large-scale surveys and undoubtedly will be costly to develop and administer, these trade-offs are necessary in order to prevent the achievements of students not included in large-scale surveys from remaining invisible. In addition to the current paucity of achievement data from students with disabilities and English-language learners, there is also a dearth of accompanying contextual data about these students' educational experiences that would allow educators and policy makers to better understand the performance and educational needs of these populations. An important focus of the integrated system of data collections that we propose in Chapter 1 is the reporting of information on the educational opportunities and instructional experiences of these students—designed to be linked to student achievement data from NAEP's large-scale survey assessments and the alternative assessments we have proposed. Finally, NAEP's goals and plans for the participation, meaningful assessment, and reporting of results of students with disabilities and English-language learners should be clearly defined and broadly disseminated. NAEP's users should be made cognizant of the degree to which its national samples do or do not include these groups of students; in fact, a key measure in the assessment of educational progress should be the reporting of progress in the numbers of students with disabilities and English-language learners who participate in NAEP's large-scale surveys. We recommend that (1) NAEP continue to strive to ensure that as many English-language learners and students with disabilities participate in NAEP's large-scale surveys as possible, (2) reliable subgroup results for these two groups be reported in conjunction with national Report Card results, (3) alternative assessments be developed for English-language learners and students with disabilities who cannot be included in the large-scale surveys, (4) contextual data regarding the educational experiences of these two groups of students be collected within the coordinated system of data collections that we have proposed in Chapter 1, (5) quantitative and qualitative reports of these students' achievements be prepared and disseminated, including contextual information that helps enhance the understanding of the educational needs of these students, and (6) NAEP's goals and plans for the participation, meaningful assessment, and reporting of results for these students be broadly disseminated. The financial and operational ramifications of these goals are tremendous and the technical issues associated with implementing these goals are numerous. Reporting reliable summary achievement results for a single subgroup that includes all types of students with disabilities and for a single subgroup that includes all types of English-language learners requires extensive (and expensive) oversampling in order to produce samples of sufficient size to report reliable subgroup results at the national level. (The expense of producing reliable subgroup results at the state level may be prohibitive.) The development of alternative

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress methods for students who cannot be assessed in the large-scale survey with accommodations will undoubtedly require a large financial investment, and, for many students, it is still not known what alternatives are best suited for assessing their achievements. The analysis and reporting of these students' achievements in relation to contextual data is an added expense not currently in NAEP's annual budget. However, the social obligation and the legislative mandate to include these groups of students in assessments and to understand their achievements and educational needs makes striving to overcome these obstacles a necessary goal for the federal government and for NAEP. Neither NAEP nor NCES can finance the steps necessary to accomplish these goals within current budgets. Additional appropriations must be forthcoming if NAEP is to make progress toward the same kinds of goals for participation and assessment of students with disabilities and English-language learners that Congress has mandated for state and district assessment programs. Even with the availability of additional funding, significant technical issues are associated with the goals we have put forth. For example, although the provision of accommodations appears to increase the participation of students in NAEP, research to date has not resolved whether data obtained using modified versions of assessments or altered administrative conditions have the same meaning (i.e., reflect measurement of the same constructs) as data collected using standard assessment materials and conditions. NAEP's research agenda must address this lack of knowledge. In addition, both groups for which we have recommended reporting subgroup results—students with disabilities and English-language learners—are very heterogeneous populations. The achievements and educational needs of students with learning disabilities could differ greatly from that of students with physical disabilities, and, even within subcategories, the skills of students with different types of learning or physical disabilities can be very different. There also can be differences in the achievements and educational needs of English-language learners based on native language, as well as within native language groups, depending on how much English is spoken within their home or their specific ethnic identification within their language group (e.g., Puerto Rican, Chicano, or Central American within the Spanish-speaking language group). Reporting information on all English-language learners as a subgroup and all students with disabilities as a subgroup must be accompanied by clear caveats about the lack of certainty regarding the generalizability of this information to all types of language groups or types of students with disabilities within these large, heterogeneous, subgroups. Still, overall subgroup results can help alert the nation as to how these groups of students are performing—in the aggregate—compared with the nation as a whole. Also, as mentioned above, for those English-language learners and students with disabilities who cannot be included in the large-scale survey with accommodations, it is not yet well established which types of alternative assessment

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress methods are most appropriate for assessing different types of students within these major subgroups. States are now investigating and implementing such strategies in their own assessment programs, and NAEP should work in partnership with them to capitalize on progress to date in planning alternative assessment methods for NAEP. These and additional technical issues are discussed further in the next section, which outlines a research agenda for enhancing the participation, meaningful assessment, and reporting of results for students with disabilities and English-language learners in NAEP and other large-scale assessments. In previous chapters we described a new paradigm NAEP, one in which assessment methods were tightly matched to assessment purpose. We described a core NAEP in which large-scale surveys are an important method for reporting overall results in NAEP's core subjects. Based on the discussion in this chapter, we envision a core NAEP that includes as many students with disabilities and English-language learners as possible and reports reliable subgroup information for these two populations of students. We also described a multiple-methods component of NAEP, and the discussion in this chapter outlines another purpose for such a component. NAEP should implement a variety of appropriate alternative assessment methods that capture the achievements of students with disabilities and English-language learners who cannot be included in the large-scale surveys (or whatever other methods are used to assess the general student population). Figure 3-1 shows the general structure of the new paradigm NAEP with multiple methods for assessing students with disabilities and English-language learners not included in the standard assessments. A RESEARCH AGENDA The implementation of the ambitious goals that we have outlined above cannot be accomplished without an increased federal commitment to and funding for research that extends beyond NAEP and NCES to the U.S. Department of Education. Much of this research has applications well beyond NAEP, extending to assessments at the state, district, and even classroom levels. NAEP clearly cannot bear the full responsibility for this research effort, but the program should serve as a leader in pushing the research agenda forward. Specifically, NAEP, NCES, and the U.S. Department of Education should define a research agenda that includes: the determination of the most appropriate methods for assessing and providing accommodations to students with disabilities and English-language learners and the effects of changes in inclusion criteria and accommodations over time on trends in achievement results.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress FIGURE 3-1 Measures of student achievement, including new paradigm NAEP. NOTE: TIMSS = Third International Mathematics and Science Study; NELS = National Education Longitudinal Study; ECLS = Early Childhood Longitudinal Study. Methods for Assessing and Providing Accommodations The research agenda to be defined must include attention to issues and questions such as the following: The need for particular types of accommodations and the adequacy and appropriateness of the accommodations provided to various categories of students with disabilities and English-language learners. Research in this area must include work on the demands (linguistic and nonlinguistic) made by different assessments on students with disabilities and English-language learners. For example, the NAEP reading assessment may place very different demands on the language abilities of students than does the mathematics assessment. Research on accommodations must also include attention to how different groups of students respond to different kinds of accommodations (e.g., students with language and speech disorders versus students with specific learning disabilities, or recently arrived English-language learners versus those who have received instruction in English over several years). Work also should focus on ways in which the appropriateness of particular accommodations for particular groups of students can be determined. Attention should also be given to the examination of whether different accommodations can provide students with disabilities and English-language learners a fair opportunity to answer questions across the range of item difficulties being tested. The validity of different types of accommodations. A reasonable accommodation should provide both students with disabilities and English-language

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress learners access to tests through an adaptation of the assessment itself or a modification of the administration procedure that does not change the nature of the construct being measured. The accommodated scores should not be an irrelevant measure of the disability or the language limitation of examinees, but a reflection of what examinees know and are able to do. The feasibility and cost-effectiveness of particularly expensive accommodations such as translation. As discussed earlier in this chapter, the analysis of the recent use of translated versions of NAEP suggests that a number of nontrivial issues about the meaning of score data from translated assessments need to be resolved before the translation of instruments is adopted as a key accommodation method. Scaling and reporting. Research in this area should include attention to such questions as: What is the impact of accommodations on scores for students with disabilities and English-language learners? Does the NAEP scale accurately reflect results for respondents assessed under nonstandard conditions? Can the scores of students with disabilities and English-language learners be combined with scores of the general population for reporting NAEP results? If results for accommodated students cannot be reported on the NAEP scale, how might these results be best reported? Questions about access to the curriculum and opportunity to learn . Research in this area should include attention to such questions as: Do students with disabilities and English-language learners study the same curricula as other students? Is the content of the NAEP assessments appropriate for students with disabilities and English-language learners, given their educational experiences? Alternative assessment methods for describing the achievements of students with disabilities and English-language learners who cannot participate in NAEP's assessments of the general student population. To date, the focus of research efforts has been on strategies for including English-language learners and students with disabilities in large-scale survey assessments. However, the development of methods for assessing and reporting results for the diverse body of students not able to participate in the general assessments must also become a focus of research. Such research has the added benefit that it could provide the foundations for a generation of methodologies that are designed to assess student performance in ways that are appropriate for everybody. Effects of Changes in Inclusion Criteria and Accommodations As discussed previously, changes in inclusion criteria and in the availability of accommodations can potentially affect the measurement of trend results in two ways: (1) trends could be altered by the inclusion of students who previously would not have participated in NAEP but now are included in the national sample and (2) the availability of accommodations to students who previously would

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress have participated in NAEP without them may have unknown impacts on their performance on the assessment. Research currently being conducted as part of NAEP's explorations of enhanced participation and accommodations in the 1996 NAEP mathematics assessment will begin to address these issues. Comparisons of NAEP results—for the nation and for key subgroups—obtained from Sample 1 (old criteria, no accommodations) will be compared with those obtained from Sample 3 (new criteria, with accommodations). Equivalency of these two sets of results would indicate that the enhanced inclusion has no significant impact on overall NAEP scale score results and that trend lines could be continued with results from assessments administered with the new inclusion criteria and accommodations. If the two sets of results are not equivalent, then it may be necessary to continue to administer the assessment to both Sample 1 and Sample 3 in subsequent administrations or conduct other studies to gauge the effects of the use of new criteria and the provision of accommodations on trend lines. Results from this initial work will be forthcoming in fall 1998, but continued work of this nature will be required across NAEP's core subject-area assessments and over time to ensure that strategies for continuing trend lines are in place. MAJOR CONCLUSIONS AND RECOMMENDATIONS Conclusions Conclusion 3A. The participation and accommodation of students with disabilities and English-language learners are necessary if NAEP results are to be representative of the nation's students. There is currently a paucity of interpretable achievement data and accompanying contextual data on the performance and educational needs of these populations. Conclusion 3B. Enhanced participation of students with disabilities and English-language learners in NAEP depends on (1) the consistent application of well-defined criteria to identify these students and (2) accurate collection and reporting of information about them. Recommendations Recommendation 3A. NAEP should include sufficient numbers of students with disabilities and English-language learners in the large-scale assessment so that the results are representative of the nation and reliable subgroup information can be reported.

OCR for page 87
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Recommendation 3B. Criteria for identifying students with disabilities and English-language learners for inclusion in the large-scale survey need to be more clearly defined and consistently applied. Recommendation 3C. For those students who cannot participate in NAEP's standard large-scale surveys, appropriate, alternative methods should be devised for the ongoing collection of data on their achievement, educational opportunities, and instructional experiences. Recommendation 3D. In order to accomplish the committee's recommendations, the NAEP program should investigate the following: Methods for appropriately assessing, providing accommodations, and reporting on the achievements of students with disabilities and English-language learners, and Effects of changes in inclusion criteria and accommodations on trends in achievement results.