Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 18
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments 2 Characteristics of the Students, the Assessments, and Commonly Used Accommodations There are important educational, social, and practical reasons for including both students with disabilities and English language learners in educational assessments whenever possible. There are also laws requiring that these students be included in many kinds of testing, and these laws in many cases provide some guidance for determining how and when these students are to be tested. With more than 20 percent of the students enrolled in public schools in grades K through 12 identified as belonging to one or both of these groups (U.S. Department of Education, 2002; http://www.ncela.gwu.edu/askncela/01leps.htm), the nation clearly has an interest in monitoring their academic progress. A series of legal mandates, beginning in the 1960s, has spelled out the responsibilities of the nation and the states not only to educate these students and to address their specific needs, but also to collect data regarding many aspects of their schooling and achievement. This chapter provides an overview of the characteristics of each of these groups of students; the legal requirements that affect their participation in educational testing; the large-scale assessments, including the National Assessment of Educational Progress (NAEP), that are used to collect information about their academic progress; and the accommodations that are used in testing these students. This report addresses both students with disabilities and English language learners because members of both groups need accommodations and because many data collection issues have similar implications for both. However, the committee recognizes that the needs and characteristics of these two groups are in many ways very different; where relevant, we have attempted to identify the differences.
OCR for page 19
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments THE STUDENTS Students with Disabilities According to data collected for the 2000-2001 school year, students with disabilities who are ages 6 through 17 constitute 11.5 percent of the total student enrollment for prekindergarten through 12th grade (U.S. Department of Education, 2002, p. II-19). This percentage accounts for nearly 5.5 million children, the majority of whom are educated in the public school setting, rather than in schools devoted exclusively to serving students with disabilities. States report disability data in 13 categories. Table 2-1 provides information on the percentages of students in each of the categories enrolled in schools in the United States and outlying areas (American Samoa, Guam, Northern Marianas, Puerto Rico, Virgin Islands). Students with disabilities who qualify for special education are covered by the Individuals with Disabilities Education Act (IDEA). Under this law, these students must be provided with an individualized education plan (IEP), which spells out the goals for each student’s instruction and the kinds of services he or she needs, including the accommodations the student requires for standardized assessments. The student’s education must, by law, be tailored to his or her TABLE 2-1 Number and Percentage of All Students with Disabilities Ages 6 Through 17 Served Under the Individuals with Disabilities Education Act During the 2000-2001 School Year Disability Number Percentage Specific learning disabilities 2,748,569 50.0 Speech or language impairment 1,088,863 19.8 Mental retardation 545,465 9.9 Emotional disabilities 448,310 8.2 Multiple disabilities 106,926 1.9 Hearing impairments 66,092 1.2 Orthopedic impairments 68,253 1.2 Other health impairments 282,470 5.1 Visual impairments 24,033 0.4 Autism 74,166 1.3 Deaf-blindness 1,087 0.0 Traumatic brain injury 13,160 0.2 Developmental delay 28,935 0.5 All disabilities 5,496,329 NOTE: Percentage total does not add up to 100 percent because numbers were rounded. SOURCE: U.S. Department of Education (2002).
OCR for page 20
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments needs, and assessment of that student’s academic progress should reflect an understanding of those goals and of what material the student has had an opportunity to learn. Students who are considered to have a disability but who do not qualify for special education services are covered by regulations in Section 504 of the Rehabilitation Act of 1973 and the Americans with Disabilities Act of 1990. Such students may have temporary physical problems or a disability, such as one that confines them to a wheelchair, that does not call for a special education plan but does make accommodations necessary, or they may suffer from illnesses that have not progressed to the point at which they need special education. Some students with attention deficit disorder receive services under Section 504 regulations rather than through special education. In most states, the Section 504 Plan specifies the accommodations the student requires for standardized testing. Both IDEA and Section 504 regulations prohibit discrimination on the basis of a disability and therefore require that students who need testing accommodations be provided with them. The characteristics of the students who are classified as having a disability are extremely diverse and tend to vary along a continuum rather than manifesting themselves in tidy categories. Moreover, factors such as the nature of the special services that are available in the jurisdiction, the characteristics of the general education classrooms in the jurisdiction, as well as social and interpersonal factors, can affect the diagnosis of students’ needs. An earlier National Research Council committee (National Research Council, 1997a) noted specifically that the extent to which a child’s performance in school can be explained by intrinsic characteristics of the child, rather than the characteristics of the context in which that child is being educated, is difficult to discern. Commenting on the diversity of students with disabilities as a group, that committee further noted that they can meaningfully be described as a group only in the context of the rights they are accorded under the IDEA and other legislation (National Research Council, 1997a). This diversity complicates attempts to make generalizations about students with disabilities as a group, an issue we address in greater detail later in this report. This chapter also describes the IDEA and the other legislation that affects the education of these students. Additional detail about the process of identifying students with disabilities and determining appropriate accommodations for them appears in Chapter 4. English Language Learners Nearly 4.6 million students in U.S. public schools were designated English language learners during the 2000-2001 school year (http://www.ncela.gwu.edu/askncela/01leps.htm). This group of students is generally understood to include all those whose language skills in English are in need of development if they are to demonstrate their full academic potential. The total number of students
OCR for page 21
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments reported by states to be English language learners has nearly doubled during the past decade (Kindler, 2002). That growth has not been evenly distributed among the states, however. California, Florida, Texas, and New York, for example, have the largest numbers of English language learners, and have all seen growth in this group of students in the period from 1997-1998 to 2000-2001, but other states—which started out with smaller numbers—have seen more dramatic growth in that period. Georgia, for example, experienced a 113 percent increase in the number of enrolled students classified as English language learners. Other states have seen sizeable, though somewhat smaller, increases, including Mississippi with a 79 percent increase, Indiana with a 32 percent increase, Wisconsin with a 30 percent increase, North Carolina with a 27 percent increase, Ohio with an 18 percent increase, and Maryland with a 15 percent increase. A few states, most notably New Mexico, lost English language learners during that period (Kindler, 2002). Although the majority of English language learners are native Spanish speakers, states reported enrolling students who spoke more than 400 languages in the 2000-2001 school year. Table 2-2 provides estimates of the numbers of English language learners who report various languages as their first, or native language (http://www.ncela.gwu.edu/askncela/05toplangs.htm). As a group, English language learners bring with them to U.S. schools a diverse range of both English-language skills and previous academic experience. Some are fully literate in their native languages and ready to plunge into content appropriate to their age and grade level. Others are not, bringing very different needs to their classrooms. These students vary in terms of how long they have lived in the United States, how much exposure they have had to English in their previous schooling, the age at which they entered U.S. schools, the number of years they have been classified as English language learners and the nature of supports they may already have received, their prior academic performance, and socioeconomic characteristics (National Research Council, 1997b). Thus, as with students with disabilities, it is important to recognize that English language learners are not a uniform group. There is no federal legislation that lays out regulations and procedures for the education of English language learners in the way that the IDEA does for students with disabilities. Hence, both the means by which English language learners are identified and the services they are offered through the public schools vary considerably across jurisdictions. To determine which students need English instruction or other linguistic support in addition to their regular academic work, most states either use a test (commercially available assessments or assessments developed at the district level) or simply make informal evaluations of students’ ability to function and succeed in classrooms in which instruction is offered in English. However, in determining whether or not to include students in large-scale achievement testing programs, most states focus on the number of years students have been in the United States or the number of years students have been receiving
OCR for page 22
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 2-2 Most Common Language Groups for English Language Learners Language Estimated Number Estimated Percentage Spanish 3,598,451 79.00 Vietnamese 88,906 2.00 Hmong 70,768 1.60 Chinese, Cantonese 46,466 1.00 Korean 43,969 1.00 Haitian Creole 42,236 0.90 Arabic 41,279 0.90 Russian 37,157 0.80 Tagalog 34,133 0.70 Navajo 27,029 0.60 Khmer 26,815 0.60 Chinese, Mandarin 22,374 0.50 Portuguese 20,787 0.50 Urdu 18,649 0.40 Serbo-Croatian 17,163 0.40 Lao 15,549 0.30 Japanese 15,453 0.30 Chuukese 15,194 0.30 Chinese, unspecified 14,817 0.30 Chamorro 14,354 0.30 Marshallese 13,808 0.30 Punjabi 13,200 0.30 Armenian 13,044 0.30 Polish 11,847 0.30 French 11,328 0.20 Hindi 10,697 0.20 Native American, unspecified 10,174 0.20 Ukrainian 9,746 0.20 Pohnpeian 9,718 0.20 Farsi 9,670 0.20 Somali 9,230 0.20 Cherokee 9,229 0.20 Gujarati 7,943 0.20 Albanian 7,874 0.20 German 7,705 0.20 Yup’ik 7,678 0.20 Bengali 6,587 0.10 Romanian 5,898 0.10 IIocano 5,770 0.10 Other languages 152,082 3.50 Total 4,544,777 NOTE: Percentage total does not add up to 100 percent because numbers were rounded. SOURCE: Available: http://www.ncela.gwu.edu/askncela/05toplangs.htm.
OCR for page 23
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments linguistic support services (Council of Chief State School Officers, 2002). In Chapters 3 and 4, state policies regarding the inclusion of English language learners in large-scale testing programs and the provision of appropriate testing accommodations for them are discussed in greater detail. LEGAL REQUIREMENTS The rights of students with disabilities and English language learners are covered by a complex maze of federal laws. The Fourteenth Amendment to the U.S. Constitution provides the basis for much of the legislation concerning disadvantaged students’ rights to education. It guarantees protection from discrimination and provides for due process. Public schools are thus prohibited from denying students equal protection of the law or life, liberty, or property interests without due process. Title VI of the Civil Rights Act of 1964 prohibits discrimination on the basis of race, color, or national origin and has been interpreted as “requiring the inclusion of English language learners in testing” (Coleman in National Research Council, 2002a, p. 14). The reasoning behind this interpretation is that participation in testing is a benefit; categorically excluding a student from testing amounts to denying him or her a benefit, and it has the potential consequence of severely limiting that student’s future educational opportunities. The first piece of legislation to specifically address the educational needs of these groups was the Elementary and Secondary Education Act (ESEA) of 1965, which explicitly required that disadvantaged students be held by states to the same high standards that applied to other students (Taylor in National Research Council, 2000b, p. 14). Title I of this legislation, which has been renewed and modified numerous times since the 1960s, has called for these students to be assessed “appropriately” and for proper accommodations to be used to achieve this accountability. States must abide by these requirements in order to receive federal Title I funding, which is provided to support the education of disadvantaged children. Since the original passage of the ESEA, a number of other federal laws that relate to the testing of English language learners and students with disabilities have been passed, and they fall into two categories: laws that deal with fundamental student rights and laws that are related to a particular federal grant program. Laws in the first category provide that students who are in public or private schools that are recipients of federal funds are protected by guarantees related to appropriate test use provisions.1 The other category of laws, those that relate to federal grant programs, oper- 1 Such laws include the Fourteenth Amendment, Title VI of the Civil Rights Act of 1964, the Equal Educational Opportunities Act, Section 504 of the 1973 Rehabilitation Act, and Title II of the Americans with Disabilities Act of 1990.
OCR for page 24
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments ate somewhat differently. They have very specific requirements that do not give students rights to file legal claims, but instead set conditions for the award and use of federal funds around certain specified test use practices.2 The Individuals with Disabilities Education Act (P.L. 105-12) falls into the category of a grants program, since it provides funds to states to serve students with disabilities, but it is also a civil rights law that extends the constitutional right to equality of educational opportunity to students with disabilities who need special education. The laws that specifically affect each of the two groups that are the subject of this report are discussed in greater detail below. Students with Disabilities Individuals with Disabilities Education Act3 The IDEA is the primary federal law providing funding and policy guidance for the education of students with disabilities; its major policy goals have remained constant since the IDEA’s predecessor, P.L. 94-142, was enacted in 1975. The IDEA provides funds to states to serve students with disabilities who are in need of special education on the condition that the states ensure an appropriate education for them. As noted above, the IDEA is also a civil rights law extending the constitutional right to equality of educational opportunity to students with disabilities needing special education. The law sets out three basic requirements with which states and local districts must comply: All children who have disabilities and are in need of special education must be provided a free, appropriate public education. Each child’s education must be determined on an individualized basis and designed to meet his or her unique needs in the least restrictive environment. The rights of children and their families must be ensured and protected through procedural safeguards. The primary mechanism for ensuring that the educational objectives of the IDEA are met is the individualized education program (IEP), which must be prepared for each child identified as having a disability and being in need of special education. The IEP is a written statement that describes the child’s current 2 Laws that fall into this category are Titles I and VII of the 1994 ESEA, the Goals 2000: Educate America Act, and the No Child Left Behind Act. Title I of the 1994 ESEA serves disadvantaged, high-poverty students, while Title VII serves language-minority students. As noted above, Goals 2000 and No Child Left Behind promote standards-based reform efforts. 3 Text is adapted from National Research Council (1997a, pp. 46-47).
OCR for page 25
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments level of educational performance; the annual goals and short-term objectives that have been established for him or her; the specific educational and related support services to be provided, including instructional and testing accommodations; and procedures for evaluating progress on the stated goals and objectives. The center-piece of the law is Part B, which authorizes grants to states to support the education of students with disabilities and outlines the requirements that states and districts must meet as a condition of that funding. Section 504 of the Rehabilitation Act of 19734 Because the IDEA is essentially a federal grants program, state participation is voluntary and the act’s requirements are imposed on states and local districts only if they choose to accept the funding. All states are currently accepting IDEA funding. However, even without the IDEA, school districts would still have a legal obligation to serve students with disabilities because of two federal civil rights statutes: Section 504 of the Rehabilitation Act of 1973 and the Americans with Disabilities Act of 1990 (ADA). Section 504 prohibits discrimination solely on the basis of disability against persons who are otherwise qualified in federally assisted programs and activities. It applies to virtually all public schools, since the overwhelming majority receive some form of federal assistance. In the context of elementary and secondary education, Section 504 regulations require that local districts provide a free, appropriate public education to each school-age child, regardless of the nature or severity of the person’s disability. Whereas the IDEA addresses individuals with disabilities who need special education, Section 504 defines and protects a broader category of these individuals, whether or not they require special education programs or related services. So, for example, elementary and secondary students requiring only special accommodations but not special education are covered by Section 504. Section 504 requires the provision of reasonable accommodations for individuals with disabilities who are otherwise qualified to participate in an educational program or activity. Americans with Disabilities Act5 The Americans with Disabilities Act of 1990 is a comprehensive federal civil rights statute that provides a “national mandate to end discrimination against individuals with disabilities in private-sector employment, all public services and public accommodations, transportation, and telecommunications” (Hardman et 4 Text is adapted from National Research Council (1997a, pp. 47-50). 5 Text is adapted from National Research Council (1997b, pp. 50-51).
OCR for page 26
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments al., 1996, p. 13). The ADA also requires “reasonable accommodations,” a term that is being interpreted through case law. Perhaps the greatest impact of the ADA on the education of students with disabilities has been the increase in the availability of accommodations for persons in the private sector in employment, recreation, living arrangements, and mobility, which has led to a more comprehensive effort to prepare children with disabilities for greater adult participation in community settings. The ADA’s Title II mirrors the nondiscrimination provisions of Section 504. It extends civil rights protections for otherwise qualified persons with disabilities to include services, programs, and activities provided by “public entities,” which include state and local governments and their instrumentalities. Because of this provision, access to state and local programs must be provided regardless of whether the states receive federal funding. Thus, even public schools not covered by other federal laws governing special education must comply with the ADA. State Laws In addition to the federal laws governing the education of students with disabilities, all states and many local governments have enacted statutes and regulations designed to promote the rights of students with disabilities. Since states must have a plan in order to qualify for IDEA funds, all have enacted special education statutes that incorporate the major provisions of the IDEA. Some state laws, however, extend beyond the federal criteria for an appropriate education. The IEP Process6 The provisions of the federal IDEA and Section 504 and the requirements of most state special education laws require an individualized, appropriate education for students with disabilities. Two key assumptions about student evaluation and identification underlie these requirements. First, students are not eligible for coverage under the laws unless they have either been identified as being “disabled” and in need of special education or, under Section 504, are either defined as “disabled” or “regarded as being disabled.” Consequently, a process is needed to determine whether each individual is eligible for the procedural protections or services each law provides. Second, the laws require a process to evaluate each individual with a disability in order to identify the student’s capabilities and needs and the appropriate programs and services. The development of an IEP is the process that has been devised to meet these needs. In this process, teachers, other service providers, and parents work together 6 Text is adapted from National Research Council (1997a, pp. 56-57).
OCR for page 27
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments to define and document the program and services the student needs (Zettel, 1982). The IEP process includes both substantive protections governing a student’s educational program and procedural requirements that both foster the participation of parents in educational planning and provide an independent review mechanism if disputes arise between educators and the family over how or where to educate the student. Case Law The IDEA, Section 504, and the ADA are being further clarified through case law. Several recent cases, which are class action suits brought against states in which students were denied the accommodations specified in their IEPs, are relevant to this report. In Indiana (Rene v. Reed, 751 N.E.2d 736, Ind. Ct. App. 2001), a decision was rendered by the state appellate court that accommodations specified in an IEP need not be provided in a state assessment if they would affect the validity of test results. However, in a similar case in Oregon, state officials agreed to a settlement in which the state assumes the burden of proof for demonstrating the inappropriateness of an accommodation (Advocates for Special Kids (ASK) v. Oregon State Board of Education, Federal District Court, No. CV99-263 K1). In this case it was established that students with disabilities whose IEPs specify accommodations would receive those accommodations on statewide assessments unless the state of Oregon could prove that the accommodations would alter the construct being measured and thus invalidate the results. A case in California (Juleus Chapman et al. v. California Department of Education et al., No. C01-1780) was decided along similar lines. This case, which provides the basis for California’s policies on accommodations, made a distinction between the question of whether students have the right to take tests with accommodations provided for in their IEP or Section 504 Plans (the ruling affirmed this right), and the question of whether the resulting scores must be treated as equivalent to those from unaccommodated administrations (the court ruled that they need not be). These cases have brought national attention to the need for empirical evidence to document that certain accommodations are inappropriate for certain types of tests. That is, evidence is needed to evaluate whether providing an accommodation interferes with assessment of the targeted construct. We address this issue in detail in Chapters 5 and 6. English Language Learners For English language learners, Titles I and VII of the 1994 Elementary and Secondary Education Act have been key pieces of legislation. These provisions required that states report disaggregated achievement test results for both students with disabilities and English language learners so that the progress of each group could be monitored. Under these laws, school districts have what the law
OCR for page 28
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments calls an “affirmative obligation” to provide English language learners with equal access to educational programs so that students have the opportunity to become proficient in English and to achieve the high academic standards of their educational programs. School districts must ensure that their curricular and instructional programs for English language learners are recognized as educationally sound or otherwise vouched for as legitimate educational strategies, and that they are implemented effectively and monitored over time (and altered as needed) to ensure success. The fundamental legal requirement is that the achievement of all English language learners must be assessed as part of state testing programs, and that reasonable accommodations be provided for those who need them. The states are required to identify the languages spoken by students in their school systems and “make every effort to develop” assessments that can be used with these students (National Research Council, 2000b, p. 15). They are required to consider either accommodations or native language testing to obtain scores that are valid for their students, depending on the students’ needs and the instruction they have received. These requirements are qualified by the phrase “to the extent practicable” to allow states some leeway in addressing changing populations of nonnative English speakers and other practical concerns. While some flexibility in strategies is allowed, states are required to apply their policies regarding accommodations consistently across districts and schools. States are required to include all English language learners7 in assessment programs used for Title I purposes and must make a determination for each student of what form of testing, accommodations, or alternate language testing would yield the most valid and reliable results for that student. The content and performance standards against which English language learners are tested may not be less rigorous than those for other students, and English language learners must be tested at all of the grades included in the statewide testing system. The Title I legislation builds on the development over several decades of legal standards that have affected the schooling of English language learners, which began with Title VI of the 1964 Civil Rights Act, discussed above. A U.S. Supreme Court case, Lau v. Nichols (414 U.S. 563, 94 S. Ct. 786, 1974), later followed up on the principles of educational equality established there, holding that “there is no equality of treatment merely by providing students with the same facilities, textbooks, and curriculum; for students who do not understand English are effectively foreclosed from any meaningful education.” Also in 1974, Congress enacted the Equal Education Opportunities Act, judicial interpretations of which made more explicit what states and school districts must do to enable English language learners to participate “meaningfully” in the educational programs they offer. These standards were used in a 1981 7 The legislation uses the term LEP, limited English proficient, students.
OCR for page 30
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments While the law does not provide detailed guidelines for the testing of students with disabilities or English language learners, its requirements for data, as well as the high stakes it attaches to test results, add additional weight not only to the importance of accurate data but also to the particular challenges of obtaining valid and reliable results for these two groups of students. PROFESSIONAL MEASUREMENT STANDARDS In addition to the legal requirements that govern the testing of students with disabilities and English language learners, there are also standards regarding their testing that have been developed by the professional communities involved in educational measurement, psychology, and educational research. Development of the Standards for Educational and Psychological Testing (1999) was guided by members of the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). These standards, while not legally binding, are a widely respected guide to the best practices that have evolved over several decades. The most recent edition of the Standards devotes a chapter to the testing of individuals of diverse language backgrounds and a chapter to the testing of individuals with disabilities. They offer a number of specific guidelines for testing each group; readers are referred to Chapters 9 and 10 of the Standards for a detailed discussion of the relevant issues. LARGE-SCALE ASSESSMENTS Large-scale assessments are those given to large numbers of students for purposes that include program evaluation, establishing standards or requirements, and matching students to appropriate instructional programs. NAEP, a national assessment that yields the data on which the widely publicized Nation’s Report Cards are based, provides data on groups of students so that educational progress across the nation can be monitored. Many other assessments are used every year in states and districts. These assessments have a variety of purposes and use a variety of methods to obtain data. National Assessment of Educational Progress As mandated by Congress in 1969, NAEP surveys the educational accomplishments of students in the United States. The assessment monitors changes in achievement in different subject areas, providing a measure of students’ learning at critical points in their school experience. Results from the assessment inform national and state policy makers about student performance, assisting them in evaluating the conditions and progress of the nation’s education system (http://nces.ed.gov/nationsreportcard/about/).
OCR for page 31
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments NAEP includes two distinct assessment programs, referred to as “long-term trend NAEP” (or “trend NAEP”) and “main NAEP,” with different instrumentation, sampling, administration, and reporting practices. As the name implies, long-term trend NAEP is designed to document changes in academic performance over time, and thus the test items generally remain unchanged. It is administered to nationally representative samples of 9-, 13-, and 17-year-olds. Over the years, trends have been reported in subject areas such as reading, writing, mathematics, and science. For the next decade or so, NAEP’s plans call for reporting trends in the areas of reading and mathematics. By contrast, main NAEP test items reflect current thinking about what students know and can do in the NAEP subject areas. They are based on recently developed content and skill outlines in reading, writing, mathematics, science, U.S. history, world history, geography, civics, the arts, and foreign languages. Main NAEP assessments can take advantage of the latest advances in assessment methodology. Main NAEP results can be used to track short-term changes in performance. Main NAEP has two components: national NAEP and state NAEP. National NAEP tests nationally representative samples of students enrolled in public and nonpublic schools in grades 4, 8, and 12 in a variety of subject areas (U.S. Department of Education, National Center for Education Statistics and Office of Educational Research and Improvement, 2003). It reports information for the nation and specific geographic regions of the country. State NAEP assessments are administered to representative samples of students enrolled in public schools in the states (U.S. Department of Education, National Center for Education Statistics and Office of Educational Research and Improvement, 2003). State NAEP uses the same large-scale assessment materials as national NAEP. It is administered to grades 4 and 8 in reading, writing, mathematics, and science (although not always in both grades in each of these subjects). The No Child Left Behind Act of 2001 made participation in state NAEP mandatory and specified requirements for NAEP. Accordingly (http://nces.ed.gov/nationsreportcard/about/assessmentsched.asp): NAEP must administer reading and mathematics assessments for grades 4 and 8 every other year in all states. In addition, NAEP must test these subjects on a nationally representative basis at grade 12 at least as often as it has done in the past, or every four years. Provided funds are available, NAEP may conduct national and state assessments at grades 4, 8, and 12 in additional subject matter, including writing, science, history, geography, civics, economics, foreign languages, and arts. Until 2002, state and national NAEP were based on separate samples of students. Beginning with the 2002 assessments, a combined sample of schools
OCR for page 32
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments was selected for both state and national NAEP. It was thought that drawing a subset of schools from all of the state samples to produce national estimates would reduce the testing burden by decreasing the total number of schools participating in state and national NAEP. From this group of schools, representing 50 states, a subsample of students was identified as the national subset. Therefore, the national sample is a subset of the combined sample of students assessed in each participating state (http://nces.ed.gov/nationsreportcard/about/). NAEP differs fundamentally from many other testing programs in that its objective is to obtain accurate measures of academic achievement for groups of students rather than for individuals. To achieve this goal, NAEP uses complex sampling, scaling, and analysis procedures. NAEP’s current practice is to use a scale of 0 to 500 or 0 to 300 to summarize performance on the assessments. NAEP reports scores on this scale in a given subject area for the nation as a whole, for individual states, and for population subsets based on demographic and background characteristics. Results are tabulated over time to provide both long-term and short-term trend information. In addition to providing straightforward scale scores for each group, NAEP uses a process for defining what are called achievement levels as an alternative way of presenting the results. NAGB has established, by policy, definitions for three levels of student achievement: basic, proficient, and advanced (http://www.nagb.org). The achievement levels describe the range of performance that the National Assessment Governing Board believes should be demonstrated at each grade, and it reports the percentage of students at or above each achievement level. NAEP is intended to serve as a monitor of the educational progress of students in the United States. Although its results receive a fair amount of public attention, they have typically not been used for purposes that have significant consequences for groups or individuals, in part because they do not generate individual or school-level scores. However, a survey has shown that NAEP results are used for a variety of purposes, some of which are beyond those specified in the legislation governing it or envisioned by its developers (DeVito, 1996). These include: describing the status of the education system, describing student performance by demographic group, identifying the knowledge and skills over which students have (or do not have) mastery, supporting judgments about the adequacy of observed performance, arguing the success or failure of instructional content and strategies, discussing relationships between achievement and school and family variables, reinforcing the call for high academic standards and educational reform, and arguing for system and school accountability.
OCR for page 33
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments It is important to point out that not all of these uses are aligned with the original objectives for NAEP and that some may not be empirically supportable uses of the data. Chapter 5 discusses in detail the ways in which evidence of the validity of inferences that might be made about test scores can be gathered. The ways in which NAEP results are used are likely to change as a result of the No Child Left Behind Act. While the law does not make new requirements of NAEP (at one time there was a plan to use NAEP directly as a benchmark for state test results), it does mandate state participation in biennial NAEP assessments of fourth and eighth grade reading and math. It is expected that NAEP will serve as a rough benchmark for each state’s assessment results, in the sense that significant discrepancies between a state’s scores on its own assessment and on NAEP would signal the need for further investigation of that state’s data on student performance (National Assessment Governing Board, 2002a; Taylor, 2002). State Assessments As noted, the principal purpose of NAEP is to provide a snapshot of what groups of students across the nation (e.g., fourth graders, Hispanic fourth graders, fourth graders enrolled in urban schools, fourth graders who live in Illinois) have achieved in a variety of subjects. The assessments reflect broad-based consensus as to what are the key elements of each subject that students should be expected to know and be able to do in each tested grade. States and districts have sought similar information about their students’ mastery of the material deemed critical in their own standards and have developed state assessments that reflect state-level priorities with regard to subject matter. In addition, although some states use assessments with a structure similar to that of NAEP, states generally have a variety of objectives for their assessments that differ from the objectives for NAEP. In many states, for example, parents and others have consistently demanded that whatever testing their children participate in yield individual scores. Parents and others want the state and district assessments their children must take to tell them not just how well the school or district is doing but also how their own children are doing. For the most part, states and districts test their students in order to gain information for the following purposes: Accountability—providing evidence of the performance of teachers, administrators, schools, districts, or states, relative to established standards or benchmarks, or in comparison to others, or both. Decisions about students—providing data that are used in making important decisions about individual students, such as placement in academic programs, grade promotion, and graduation. Program evaluation—providing evidence of the outcome of a particular educational program in terms of student performance.
OCR for page 34
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments Tracking of long-term trends—providing evidence of changes in the performance of groups of students, such as those enrolled in a particular grade, school, or school district, those belonging to population subgroups, etc. Diagnosis—providing information about students’ strengths and weaknesses with regard to specific material or skills (such as proficiency in English), for use in improving teaching and learning. Regardless of the purpose of their assessment programs, states and districts are held accountable for the achievement of all of their students with disabilities and English language learners and thus are required to include them in many of the assessments they use, although attaining that goal has proved challenging. By contrast, however, NAEP has not included all students from these two groups in testing, nor is it currently planning to do so. For practical reasons, NAEP has limited both the kinds of students that can be included and the accommodations that can be offered. While the No Child Left Behind Act of 2001 does not call for direct linking of NAEP results with those of states, this discrepancy in inclusion policies is likely to prove a serious difficulty for a number of reasons. Chapter 3 addresses in greater detail the policies of both NAEP and the assessments used by states with regard to including and accommodating these students, and provides further discussion of the implications of these policies. ACCOMMODATIONS: ISSUES AND PRACTICE Many of the laws described earlier make reference to the need for students with disabilities and English language learners to receive appropriate accommodations when they are included in educational assessments. The accommodation should allow the student to demonstrate what he or she knows or can do in spite of his or her disability or limited fluency in English, without providing him or her with any other advantage. Perhaps the most straightforward example is the provision of a Braille version of a written test for a student with visual limitations. However, because the circumstances that can cause a student to need an accommodation when participating in assessments are so varied, the questions surrounding accommodations are frequently far less straightforward than this example. Consider a student who has a decoding disability that makes it difficult for him or her to respond to word problems on a mathematics assessment. It may take this student so long to decode the instructions and the text in the individual problems that he or she runs out of time before completing all of the items, even though this student’s mathematics skills are adequate for the tasks presented. Accommodations provided to a student in this situation may include the opportunity to hear the instructions and text read aloud or the provision of extra testing time. While the decision to provide this accommodation may be relatively straightforward, determining the amount of time that will compensate for the
OCR for page 35
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments decoding disability without giving the student an additional advantage is much more complicated. Furthermore, the identification of an appropriate accommodation depends on a clear definition of the construct(s) being measured on the assessment. In this example, if the mathematics assessment were intended to measure speed of response, the provision of extra time might interfere with measurement of the targeted construct(s). Issues associated with determining the appropriateness of accommodations are discussed in greater detail in Chapter 6. The assumption underlying the accommodation is that every student has a true level of competence in each tested subject area, and that a perfectly reliable and valid assessment will reveal that true performance. The accommodation is intended only to compensate for factors that prevent the student from demonstrating his or her true competence, and not to improve his or her performance beyond that level. However, in real-world testing situations, a variety of factors may influence any student’s performance on an assessment. There may be factors extraneous to the construct being tested that affect all students, not just those who need accommodation. Students without disabilities, for example, might also find that the time allowed for a test is too short, and their scores may improve if they are allowed extra time to complete their work. The critical questions to be answered are whether the scores from an accommodated assessment are comparable to scores from the unaccommodated assessment, and, furthermore, whether similar inferences can be made from the results of each. Mechanisms for evaluating the validity of scores from accommodated assessments, as well as research into those questions are discussed in later chapters. Here we explore the specific factors that affect the accommodation of students with disabilities and English language learners. Accommodations for Students with Disabilities Now that states are required by law to strive to include all students in their accountability programs, the need for testing accommodations has become even more acute.10 As was mentioned earlier, the principal guide to assessing students with disabilities is the IEP, developed by a team that includes both school staff and the parent(s) or guardian. In addition to describing the student’s current level of performance, defining the educational goals for the student, and describing the supports or services the student will need, the IEP also specifies how the student may participate in state and local assessments and any accommodations that must 10 States are also permitted to offer alternate assessments for some students (P.L. 107-10). Alternate assessments are mostly used for the small minority of students whose curriculum is significantly different from that of general education students and who are held to different sets of standards. Alternate assessments typically will not support the same inferences as the regular assessments or yield comparable results.
OCR for page 36
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments be provided for testing. The development of the IEP is guided by legal requirements, and the IEP is considered the authoritative statement of what is appropriate for the student. There are times when state accommodation policies may be in direct conflict with the recommendations made in a student’s IEP; for example, the IEP may require that the student receive the read-aloud accommodation on standardized assessments while the state may not permit that accommodation on all assessments (e.g., on assessments of reading comprehension). What is done in that situation varies from state to state; for example, states have various policies for reporting results of assessments in which accommodations that are not approved for the assessment were used (National Research Council, 2002a). In general, the specification of accommodations is to be based on the definition of the student’s needs and is to be consistent with accommodations that are used for instruction. There are several issues that can make both determining the appropriateness of accommodations and evaluating the results of accommodated assessments difficult. Disabilities can affect test-taking and scores in varying and sometimes unpredictable ways. For example, a student with a visual impairment may require a Braille version of a test as an accommodation. However, reading Braille can be time-intensive, since it is not possible to skim Braille text; thus, unless extra time is also provided, the student may not be able to demonstrate his or her true proficiency in the assessed subject area. Another difficulty is that disabilities can overlap with achievement constructs measured in assessments. For example, a student whose learning disability is in the area of mathematical computations may require a calculator as an accommodation. However, when an assessment is intended to measure computational skills, provision of a calculator would directly interfere with measurement of the intended construct. Thus both identification of appropriate accommodations and interpretation of accommodated scores depends on precise understanding of the nature of the student’s disability. The principal testing accommodations that are offered to students with disabilities are listed by category in Table 2-3. Accommodations for English Language Learners The requirements for including English language learners under the No Child Left Behind Act are, as we have seen, very similar to those for students with disabilities (although, as noted earlier, some increased flexibility in including English language learners has recently been written into the requirement). Although the requirements are similar, the nature of the challenge to educators is slightly different. First, there is no process like the IEP process that exists for students with disabilities to guide the diagnosis of students’ needs, the determination of how federal and other laws might apply to the individual student, or the identification of appropriate accommodations for the student. Nevertheless, English language learners need accommodations because lan-
OCR for page 37
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments TABLE 2-3 Testing Accommodations Offered to Students with Disabilities Setting Timing • Individual • Extended time • Small group • Flexible schedule • Study carrel • Frequent breaks during testing • Separate location • Frequent breaks on one subtest but not another • Special lighting • Adaptive or special furniture • Special acoustics • Minimal distractions environment Presentation Response • Audio tape • Mark in response booklet • Braille edition • Use of a Brailler • Large print • Tape record for later verbatim translation • Audio amplification devices, hearing aids • Noise buffers • Use of scribe • Prompts on tape • Word processor • Increased space between items • Communication device • Fewer items per page • Copying assistance between drafts • Simplify language in directions • Adaptive or special furniture • Highlight verbs in instructions by underlining • Dark or heavy raised lines • One complete sentence per line in reading passages • Pencil grips • Large diameter pencil • Key words or phrases in directions highlighted • Calculator • Sign directions to student • Abacus • Read directions to student • Arithmetic tables • Reread directions for each page of questions • Spelling dictionary • Multiple choice questions followed by answer down side with bubbles to right • Spell checker • Special acoustics • Clarify directions • Paper secured to work area with tape/magnets • Cues (arrows, stop signs) on answer form • Provide additional examples • Visual magnification devices • Templates to reduce visible print • Eliminate items that cannot be revised and estimate score Scheduling Other • Specific time of day • Special test preparation • Subtests in different order • On-task/focusing prompts • Best time for students • Others that do not fit into other categories • Over several days SOURCE: Adapted from Thurlow et al. (2002).
OCR for page 38
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments guage barriers have the potential to prevent them from demonstrating what they know and can do in several ways. Regardless of which academic area is being assessed, a test administered in English is, in part, a measure of English language skills (American Psychological Association et al., 1999). While using tests that avoid unnecessarily complex language can enable English language learners to demonstrate their skills with respect to the measured construct, distinguishing academic progress from language skills is difficult (e.g., in mathematics when word problems are used). Thus, the goal for accommodating English language learners is the same one that guides the accommodation of students with disabilities, that is, to obtain a more valid assessment of their skills with regard to the tested construct than is possible without the accommodation. At the same time, it is important to note that for students whose English proficiency is truly minimal, no accommodation can offer a reasonable adaptation of an assessment that is in English; in such cases, other means must be found to assess students’ academic strengths and needs (and the resulting scores are not likely to be comparable to those from the original assessment). Assessments can be translated, but a number of issues arise when this is done because the difficulty level and other characteristics of translated text can be significantly different from those of the original. The accommodations available for use with English language learners can present difficulties similar to those that arise in accommodating students with disabilities. They have the potential to provide tested students with an unrecognized advantage over other students or an unrecognized disadvantage. In both of these situations, accurate interpretations cannot be made from the resulting scores. For example, English language learners might be provided with a simplified-language version of a test when in fact other test takers might also score higher if they were offered this accommodation. Research has shown that use of simplified language can, in fact, offer all students, not just those with a language deficit, an advantage on some tests (Abedi, Hofstetter et al., 2001; Abedi and Lord, 2001). This suggests that language simplification may have removed factors that were construct-irrelevant for all test-takers. Alternatively, accommodations intended to compensate for deficits in English fluency may actually impede performance. For example, English language learners may actually be disadvantaged if they have not had experience with an accommodation they are offered, particularly if it is one that requires substantial time to use, such as offering the assessment in the native language when the language of instruction is English, or providing a glossary or bilingual dictionary. In these instances, test-takers’ time and attention are diverted from the test itself and their scores may be depressed (Abedi, Lord, Hofstetter, and Baker, 2000). Another challenge to administrators who need to assess English language learners is that, as noted above, while approximately 80 percent of the English language learners in U.S. schools are native Spanish speakers, the remaining 20 percent speak many different languages. States and districts with rapidly
OCR for page 39
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments changing immigrant populations face the need to develop programs that are flexible enough to handle many different languages. They may lack the resources to easily assess the language skills of students from each linguistic background, and they may not be able to afford to offer accommodations that rely entirely or in part on the student’s native language (e.g., offering a native-language version of the test, allowing students to respond in their native language, or offering a glossary or translated directions). The principal accommodations offered to English language learners are shown in Table 2-4. TABLE 2-4 Testing Accommodations for English Language Learners Setting Timing/Scheduling • Small group • Extended testing time (same day) • Individual administration • Frequent, extra, longer breaks • Separate location, study carrel • Time of day most beneficial to student • Preferential seating • Teacher facing student • Several (shorter) sessions • Testing over several days (some extended time) • Flexible scheduling (of subtests) Presentation Response • Oral reading of questions in English • Student dictates answer, uses scribe • Explanation of directions • Student response in native language • Translation of directions • Repetition of directions • Student marks answers in test booklet • Translation of test into native language • Person familiar with student administers test • Student types or uses machine • Audio cassette • Clarification of words (spelling, defining, explaining) • Highlighting key words • Oral reading of directions • Use of an interpreter (sight translator) • Bilingual version of the test • Oral reading of questions in native language • Simplified/sheltered English version of test • Use of place markers to maintain place Other • Use of bilingual word lists, dictionaries • Out-of-level testing • Use of brainstorming activities SOURCE: Rivera et al. (2000, p. 34).
OCR for page 40
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments SUMMARY This chapter has presented background information designed to illustrate several key points about the participation of students with disabilities and English language learners in large-scale assessments. First, students with disabilities and English language learners are an extremely diverse group. Including them in large-scale assessments so that their educational status and their needs will be addressed along with those of other students is not only important but a legal mandate for states. However, their diverse needs call for assessment approaches that are both flexible enough to evaluate what they know and can do and rigorous enough that the results can be safely compared with those from other assessments. Second, the legal requirements and professional standards regarding the assessment of these two groups of students provide considerable guidance to those responsible for setting assessment policy and developing assessment tools. They have established both the necessity for testing students with disabilities and English language learners and the key questions of fairness that arise when they are tested. These sources, however, offer relatively little guidance with regard to the practical difficulties associated with testing these two groups. As a consequence, there is considerable variation around the country in the way these students are assessed. Finally, a key tool used in assessing students in these two groups is accommodation. These measures, designed to overcome obstacles to testing that are irrelevant to the constructs being measured, have the potential to make it possible to obtain accurate results for students who could not be reliably assessed without them. Many different accommodations are in use around the country. The ways in which they are used, the variety in the ways they are applied, and the knowledge base about their effects on testing results are the focus of the remainder of the report.
Representative terms from entire chapter: