There is increasing pressure to include in assessment programs larger numbers of students with special language and learning needs. Several federal laws, for example, require the participation of all students in assessments used to gauge student performance. Those that affect English-language learners include Goals 2000, Title I (Helping Disadvantaged Children Meet High Standards) and Title VII (Bilingual Education) of the Improving America's Schools Act of 1994, and the reauthorization of the Office of Educational Research and Improvement (Title IX of Goals 2000). As Chapter 3 notes, English-language learners are also protected by Title VI of the Civil Rights Act of 1964; Title VI regulations forbid various forms of discrimination on the basis of national origin and limited English proficiency (Lau v. Nichols, 1974). These laws provide that "standards and assessments are to fully include English-language learners" and that "innovative ways of assessing student performance are encouraged, including modifications to existing instruments for English-language learners" (National Research Council and Institute of Medicine, 1997:132–134).
Given the limitations of time and resources, the committee has taken advantage of recent work in this area by another committee of the National Research Council (NRC) and the Institute of Medicine (IOM). The Committee on Developing a Research Agenda on the Education of Limited-English-Proficient and Bilingual Students considered in detail
how best to meet the academic and social needs of children who are English-language learners. That committee's report, Improving Schooling for Language-Minority Children: A Research Agenda (National Research Council and Institute of Medicine, 1997), devotes a portion of the discussion to the issue of assessing the language proficiency and subject-matter knowledge and skills of English-language learners. We draw on this earlier report throughout this chapter.
Population of English-Language Learners
Defining appropriate assessment policies for English-language learners is complicated by the great diversity of their language backgrounds, previous educational experience, length of time residing in the United States, and current instructional programs. The number of English-language learners in grades K-12 is large and has grown considerably over the last decade (National Research Council and Institute of Medicine, 1997:17–19). A survey conducted in 1991 (Fleischman and Hopstock, 1993) estimated the number of students classified as English-language learners at 2.3 million (or about 6 percent of the school population)—an increase of almost 1 million students over a similar survey in 1984.
About three-fourths of English-language learners in the United States have Spanish as their language background; 27 percent come from other language backgrounds. No other single language is spoken by more than 4 percent of English-language learners. Over half of English-language learners are in the early elementary grades (K-4). A large majority are from disadvantaged socioeconomic backgrounds.
Children who cannot participate meaningfully and equitably in English-only classrooms due to limited English proficiency are eligible by law for special instructional services. The type and range of such programs vary tremendously (see National Research Council and Institute of Medicine, 1997:19–21, for an overview). In addition, English-language learners enter the United States at different ages, and their home and community environments differ in the amount of English used. Some arrive in U.S. schools with limited or interrupted prior schooling. Once here, they may enter English-only instructional programs or bilingual programs; many shift between programs.
A number of methods are used to determine which students are English-language learners, to place these students in special instructional programs, and to monitor their progress. Improving Schooling for Language-Minority
Children reported that the use of language-proficiency tests in English was the most common method of identifying English-language learners. More than 80 percent of the affected school districts used such tests, either alone or in combination with other techniques. Large majorities of districts also used these tests to assign English-language learners to specific instructional services and to reclassify students after they had developed English proficiency. About half the districts also used achievement tests in English to identify and assign English-language learners; more than 70 percent used them for reclassification. Other methods include home language surveys, observations, interviews, and enrollment information. But because many states allow districts to choose their assessment methods (usually from a menu of state-approved instruments), there is wide variation in the ways these decisions are made (National Research Council and Institute of Medicine, 1997:115–116). Estimates of trends in the prevalence of English-language learners can be affected by these variations across jurisdictions and over time; variations in the practice of classifying students as English-language learners to increase funding can also affect these trends.
The appropriate use of these assessment methods helps ensure that English-language learners receive the services necessary for learning; their inappropriate use can result in faulty assignment of services—or no services at all—and can lead to the tracking of English-language learners in low-level classes, their retention in grade, and their failure to graduate. For example, researchers have found that lack of language proficiency can lead to low-track placements, particularly in middle and high school. Berman and colleagues (1992) document that it is not unusual for secondary schools to require that students demonstrate proficiency in English before they are given access to grade-level mathematics and science courses; students who are not fluent in English may be barred from regular classes or tracked into remedial or compensatory classes, where instruction proceeds at a slower pace.1
Assessing English-language learners who may also have learning disabilities is particularly problematic. A lack of appropriate instruments is exacerbated by the scarcity of personnel with expertise in evaluating both
Note the committee's finding that low-track placements are typically not educationally beneficial (Chapter 5).
linguistically and culturally diverse learners and students with disabilities.
The NRC and IOM report concludes that "most measures used [for assessing English-language proficiency] not only have been characterized by the measurement of decontextualized skills but also have set fairly low standards for language proficiency. Ultimately, English-language learners should be held to high standards for both English language and literacy, and should transition from special language measures to full participation in regularly administered assessments of English-language arts" (1997:118).
Testing English-Language Learners
The central dilemma regarding participation of English-language learners in large-scale assessment programs is that, when students are not proficient in the language of the assessment (English), their scores on a test given in English will not accurately reflect their knowledge of the subject being assessed (except for a test that measures only English proficiency). School officials typically decide first whether to exempt an English-language learner from an assessment altogether and, second, if the student is included, how to modify the test or testing procedures to measure the student's skills more accurately. Official policies about exempting or accommodating English-language learners in assessment programs vary widely from place to place. For example, surveys of statewide assessment systems suggest that states are in various stages of incorporating English-language learners into their statewide assessment programs.2 The following section describes some of the variation across jurisdictions in policies for exempting and accommodating English-language learners in large-scale achievement testing. It goes on to describe in more detail current assessment systems in one city—Philadelphia—and four states with large numbers of English-language learners—California, Florida, New York, and Texas.3
Exempting English-Language Learners Based on Language Proficiency
Recent data from the Council of Chief State School Officers' annual survey of state assessment programs (1998) indicate that nine states do not allow exemptions of English-language learners from one or more of their state assessments. When exemptions are allowed, amount of time in the United States (for 27 assessments) or amount of time in an English as a second language (ESL) program (for 18 assessments) are most often named as criteria. States report using formal (for 4 assessments) and informal (for 5 assessments) testing much less frequently to make exemption decisions. When exemption decisions are made at the local level, it is usually by local committees (26 reported instances) or school districts (25 reported instances) and less frequently by parents (9 reported instances). If exemption decisions are made at the local level, formal assessments tend to be used more frequently than at the state level (11 reported instances), but time in the United States (13) and time in an ESL program (16) are still the most frequently named criteria (pp. 334–338).
Decisions about exemption are thus often made on the basis of some indicator of English proficiency (see Box 9-1). Yet there is considerable variability from place to place in the criterion used and the basis for its choice. California, for example, recently tightened its policy regarding exemptions on the SAT-9, which is required each year for all students in grades 2 through 11. The new policy, which will allow far fewer exemptions, requires all English-language learners who have been in California for more than one year to take the SAT-9 in English. Children may also be assessed in their primary language, if such assessments exist.4
This policy is being contested. The superintendent of San Francisco's schools recently filed suit in federal court against California's test policy, arguing that the civil rights of students with limited English proficiency are being violated because they don't know enough English to read the test and show what they know about reading, writing, math, history, and science (Education Week, 1998). He argued that a longer time is required before a student is proficient enough to take the test in English—specifically, that students should be enrolled in the city's schools for 30 months
The director of bilingual education for the state reports that districts will probably use the Aprenda, the CTBS Espanol, or the SABE for this purpose (see Box 9-1).
BOX 9-1 Examples of Policies on Exemptions and Accommodations
Philadelphia: Exemptions: The SAT-9 and the Aprenda (a reading and math assessment in Spanish) are used to assess math and reading knowledge and skills. Level 1 English-language learners (those who are not literate in their native language) are generally exempt from the SAT-9 and are given the Aprenda. In some schools, the Aprenda is given to all students in bilingual programs. Levels 2 and 3 English-language learners take the SAT-9 with accommodations. Level 4 English-language learners (those who are almost ready to be mainstreamed) take the SAT-9 without accommodations.
Accommodations include extra time; multiple shortened test periods; simplification of directions; reading aloud of questions (for math and science only); translation of words or phrases on the spot (for math and science only); decoding of words upon request (but not for reading); use of gestures and nonverbal expressions to clarify directions and prompts; student use of graphic organizers and artwork, usually in combination with student's oral responses; testing in a separate room or small-group setting; use of a study carrel; and use of a word match glossary.
Philadelphia teachers reported that students generally reacted favorably. Some, however, said that "ungraded" English-language learners tested at their age-appropriate grade level, especially in middle and high school, were frustrated in spite of accommodations. Recommendations include allowing the use of bilingual dictionaries and electronic translators and even more time (interspersed with short breaks and over a series of days).
A few schools in Philadelphia are experimenting with portfolios, which may be used to assess math and reading knowledge and skills of Levels 1, 2, and 3 English-language learners.
Florida: The state requires district norm-referenced achievement tests at grades 4 and 8; Florida Writes, a writing assessment given at grades 4, 8, and 10; and the high school competency test, required for high school graduation. In the coming year, the state will replace the district norm-referenced achievement tests with the Florida comprehensive assessment test, a test of math and reading based on state standards.
Exemptions: The state suggests that English-language learners in an ap
proved limited-English-proficient program for fewer than two years may be exempted from the norm-referenced district testing programs and Florida Writes. Exempted students must be assessed through other means determined by school and district personnel. Students not receiving special language assistance services may not be exempted by virtue of their classification as English-language learners alone. For system accountability purposes, scores for students in the English as a second language program for less than two years are disaggregated but not used to identify critically low-performing schools.
Accommodations: Districts are required to provide accommodations for English-language learners who are currently receiving services in a program operated in accordance with an approved district plan for English-language learners, but the exact combination of accommodations to be offered to any particular student is individually determined, considering the needs of the student.
Texas: The Texas assessment of academic skills (TAAS) is used to monitor the progress of students in grades 3 to 8. Language-proficiency assessment committees at each school (a site administrator, bilingual educator, English as a second language educator, and a parent of a child currently enrolled) determine which assessment each child will take. On the basis of six criteria—literacy in English and/or Spanish; oral language proficiency in English and/or Spanish; academic program participation, language of instruction and planned language of assessments; number of years continuously enrolled in school; previous testing history; and level of academic achievement—the committee decides whether English-language learners are tested on the English TAAS, tested on the Spanish TAAS, or exempted and given an alternative assessment. The committee also makes program entry and exit decisions and monitors students' progress after special services are ended. Those entering U.S. schools in the 3rd grade or later are required to take the English TAAS after three years.
The TAAS scores, whether for the English or the Spanish TAAS, are used as base indicators in the accountability rating system. In the near future, scores on the reading proficiency test in English for students taking the Spanish TAAS or for those delayed in taking the English TAAS will be publicly reported. Schools are designated "exemplary," "recognized," "acceptable," and "low-performing" based on aggregated student scores, and rewards and sanctions are meted out on this basis.
or more before being required to take the test. The superintendent prevailed in the district court. It is unclear whether or how these requirements and disputes will be affected by a June 1998 California referendum that sharply limits bilingual education in the state's public schools.
A recent survey of state assessment programs for 1996–1997 (Council of Chief State School Officers, 1998) reported that only seven states do not permit accommodations in the administration of at least one assessment for English-language learners. The most common accommodations were giving tests to small groups (29 states), repeating of directions (28 states), allowing extra time (same day) (25 states), taking the test in a separate room (25 states) or alone in a study carrel (25 states), having a person familiar with the child's language and culture give the test (23 states), giving more time breaks (22 states), reading questions aloud in English (21 states), translating directions (19 states), extending the session over multiple days (15 states), simplifying directions (14 states), and using word lists or dictionaries (14 states). Ten states reported that they test students in a language other than English. Other accommodations included allowing student to respond in the native language to English questions, explaining directions, and oral reading of questions in the native language.
Other alternative assessment methodologies that have been suggested include using portfolios to collect a child's best work over time, developing computer-assisted assessments tailored to respond to language needs and content knowledge of students, extending scaffolding and sheltered instruction to assessments, dynamic assessment, and allowing English-language learners to display their knowledge using alternative forms of representation (e.g., showing math operations in numbers and knowledge of graphing in problem solving).
The survey indicated that 11 states currently have an alternative assessment in place for English-language learners in at least one assessment program. These range from a Spanish-language version of the Stanford 9 for students literate in Spanish, used in Arizona, to an option to local districts to determine their own alternative methods of assessment.
Tracking, Retention, and High School Graduation
No systematic information is available about whether or how English-language learners are accommodated in tests used for tracking or retention. In addition, we know very little about current assessment practices used to place English-language learners in Title I targeted assistance programs or other remedial programs, gifted and talented classes or advanced academic course work, or special education programs; there is an urgent need for research on these important questions.
A survey conducted in 1993–1994 (Rivera and Vincent, 1997) found that 17 states require students to pass one or more content-area tests to receive a standard high school diploma. States approach the testing of English-language learners in various ways (see Box 9-2). Eight states exempt them from the first administration of the test for a set period of time. For example, if the state first gives a graduation test in 10th grade, a newly arrived student who is judged to have insufficient English proficiency may be temporarily excused from taking the test. According to the survey, such deferrals are usually for six months to a year.
Eleven states permit accommodations, the most frequent being extra time, small-group administration, clarifying directions, flexible scheduling, and the use of dictionaries. Only two states, New York and New Mexico, give the test in students' native languages. Four states permit or require alternative assessments.
A major task in deciding how to assess English-language learners for purposes of high school graduation is to determine what level of English proficiency (if any) should be prerequisite to receiving a diploma. This decision will dictate whether graduation requirements in some domains can or should be assessed in the student's native language.
Deciding whether English proficiency should be a requirement for graduation—and, if so, what type and what level of English proficiency—could be considered a moral and policy decision about how society defines "basically educated." A more pragmatic view might see it as an economic decision about basic workplace requirements and improving students' employability. In other words, it is not essentially a psychometric decision. But it has enormous consequences for other psychometric activity, because any state or district that requires little or no English, or only conversational English, thus commits itself to developing bilingual and/or other language versions of all content tests. In most states, all children who do not graduate on time, including English-language learners
BOX 9-2 Examples of Graduation Testing Policies for English-Language Learners
Florida: Students who have been classified as limited-English-proficient and have been in an approved program for English-language learners for fewer than two years may be temporarily exempted from taking the high school competency test at the discretion of local school personnel, but such students cannot be awarded a standard high school diploma until the test is passed. English-language learners not receiving English-language learner services or former English-language learners may not be exempted. English-language learners who do not pass the competency test can return for a 13th year to study for sections of the test that were not passed; students may also opt for a certificate of completion in lieu of a high school diploma. According to state personnel, English-language learners generally pass the exit exam because most enter school in Florida in the early grades and are proficient in English by high school.
New York: In November 1997 the State Board of Regents approved revised high school graduation requirements, to be put in place for the class entering 9th grade in 2001. In four core subjects (English, mathematics, social studies, and science), students will be required to pass state Regents exams or approved alternative examinations (for special education), in addition to courses, to demonstrate achievement of the state standards.
English-language learners entering school in this country in 9th grade or later can take all required Regents examinations, except for the English examination, in their native language if available (Spanish, Haitian-Creole, Russian, Chinese, and Korean) if it is within three years of entering the United States.
and students with disabilities, may remain in school until age 21 or 22; thus English-language learners can retake the graduation test as their English-language proficiency increases. Box 9-3 explores important issues in this area using two hypothetical school systems.
Although graduation decisions are essentially different from promotion and tracking decisions, some of the considerations concerning relations between instructional methods and testing and concerning test presentation apply equally. As summarized in Improving Schooling for Language-Minority Children (National Research Council and Institute of Medicine, 1997), research on second-language learners suggests that:
However, all English-language learners must pass the English Regents exam, in English, to receive a diploma. New York state officials are concerned about the ability of two groups of English-language learners to pass the English Regents exam—students entering U.S. schools in the middle and secondary grades and those with limited or interrupted formal schooling at all grade levels. Thus the state is putting into place a comprehensive strategy to help ensure that they pass. This fourfold strategy includes: (1) a national and state search for effective program models and the dissemination of these models through a bilingual/English as a second language web site and funds to schools to help them prepare students to pass the Regents exams; (2) professional development for all teachers of English-language learners to help them integrate the English language arts standards into English as a second language and native-language arts instruction; (3) review of programs for English-language learners to ensure they are aligned with state standards, along with monitoring of school districts to ensure compliance with all regulations pertinent to English-language learners; and (4) other activities, including publications in the area of technology applications for bilingual and English-language learners, bilingual math glossaries, invitational statewide symposia on the education of adult English-language learners, and the identification and distribution of resources to provide increased after-school, weekend, and summer language arts academies for English-language learners.
Texas: Students must pass a high school exit test in reading, writing, and math (in English), taken in the 10th grade, to graduate. Governor George Bush is exploring the possibility of using the Texas assessment of academic skills (TAAS) at crucial grade levels to ensure that students have the requisite skills to proceed to the next grade level.
- Second-language learners can display a higher level of comprehension of texts that are read in a second language when discussing or when questioned about those texts in their first language than if all the discussion or testing activity occurs in the second language.
- Even at relatively low levels of proficiency in a second language, tasks that have been taught and practiced in that language may be performed better in the second than the first, stronger, language. This is particularly likely to be true for tasks that have relatively restricted formats, such as giving definitions, solving syllogisms, and making analogies.
- Transfer of knowledge and of skills from a first to a second language occurs, and it accounts for the relatively stronger academic performance
BOX 9-3 Graduation Requirements: Two Hypothetical Cases
The large city school district of New Colombia decides that it will require as a prerequisite to high school graduation that students (a) display sufficient control of English to function adequately in a job interview for an unskilled position in the restaurant or retail sector and (b) perform adequately on knowledge assessments in the domains of mathematics, U.S. government, and American literature. It also includes among its educational goals raising the high school graduation rates and decreasing disparity among language groups in rates of high school completion.
New Colombia adopts a language proficiency assessment designed to reflect conversational skill that requires one-on-one testing, using a scripted interaction protocol such as that developed for application of the guidelines developed by the American Council on the Teaching of Foreign Languages. Education officials recognize that test formats such as those used for the Test of English as a Foreign Language (TOEFL) reflect knowledge of vocabulary, formally taught grammar, and listening comprehension but do not relate to conversational/interactive proficiency, and furthermore recognize that TOEFL-style tests are more appropriate for those who have been explicitly taught a language than for those who have acquired it in a second-language setting. Teacher judgments are used to determine which students must take the conversation proficiency assessment, although New Colombia officials suspect that perhaps teacher judgment concerning conversational proficiency would be as good an indicator as teacher judgment concerning whose conversational fluency needs to be tested.
The American literature knowledge test presupposes having read works by Washington Irving, Mark Twain, Nathaniel Hawthorne, Willa Cather, F. Scott Fitzgerald, and Toni Morrison and being able to answer multiple-choice questions and provide short written answers to questions. New Colombia educators had a long debate about what form of access to these works of literature they considered essential, and what form of display of knowledge about them they considered appropriate for high school graduates. They eventually decided that English-language learners could choose to study these texts in English, or in translation, or in English with extensive footnotes providing interpretations and translations. They also decided that tests should be provided in booklets that included English and a translated version of all questions, and that students could respond to short-answer questions in any language or language mix. Provision of dual-language test versions is unlimited, upon student request.
In seven of the eight secondary schools in New Colombia, U.S. government
courses are provided in languages other than English, with one focus of these courses in some schools being to ensure knowledge of translation equivalents for key terms/concepts (e.g., "representation," "Senate," "bill," "gerrymander," "appeal," "reconciliation," "appropriation," "entitlement"). The graduation test requires knowledge of these terms in English, on the basis that such knowledge is essential to effective functioning in the domain of citizenship; thus, test booklets prepared for English-language learners that included translations of other test questions do not translate these key concepts, even in the other-language questions. Again, any student who requests a dual-language version of the test may get it.
The math test includes both straightforward calculation problems and problems involving analysis, and it requests that students display their work for both kinds of problems. Test scoring is sometimes difficult because the work displayed sometimes conforms to conventions or uses heuristics unfamiliar to the U.S. test monitors. Assuming that performance on the analysis problems would be heavily confounded by the level of comprehension of the text presenting the problem, New Colombia test writers decided that text should be provided in the native language as well as English. Bilingual presentation was considered important, since the student had been studying such problems in U.S. classrooms, thus some aspects of their formulation might be more familiar in English than in the native language. Students could opt for a dual-language version.
New Colombia encountered considerable support from the large language-minority groups for its approach to high-stakes testing for graduation, as well as many problems of ensuring comparability and fairness in its test materials. Since practical considerations limited the production of bilingual test materials to 17 languages, inevitably parents from the other 54 language groups represented in New Colombia objected, but as these groups aggregated to only 0.65 percent of the population, it was decided to ignore their demands. Protests from monolingual English-speaking parents started when it emerged that a higher percentage of students taking the bilingual versions of the test were passing compared to those taking the English-only version; English speakers pointed out that the bilingual tests gave students more information and more support than the equivalent English-only version.
North Brickley, a city with a somewhat smaller population of nonnative speakers of English than New Colombia, also considers English proficiency an important component of the skills needed for high school graduation. But North Brickley decides that English proficiency means the full array of English language skills—reading, writing, speaking and listening, and the capacity to carry out academic work in English. North Brickley also prioritizes the educational goal of high standards and ensuring high levels of competence of
its high school graduates, over the goal of increasing high school graduation rates. Thus, North Brickley rejects the New Colombia solution of allowing students to study and be tested on literature, government, and math with native-language support. Instead, a single, English-language test for each of these areas is developed, and students are told that answers provided in any language other than English will not be scored.
In consequence, North Brickley decides not to assess oral proficiency in English of nonnative speakers, reasoning that the written tests are more rigorous and demanding and that high academic performance depends more on literacy skills in English than on oral skills in any case. North Brickley's organization of its graduation requirement has the immediate consequence that graduation rates decline for nonnative speakers of English and that parents request transfer of their children out of bilingual classrooms and into English as a second language and English-only classrooms. Remediation efforts, introduced for English-language learners who fail the test on the first administration in response to parental and student demands, focuses on teaching English skills rather than on teaching subject matter, with the result that graduation rates on subsequent administrations rise only slightly.
Despite the insistence by the test administrators that performance in English is prerequisite to performance on the test, the results for the math subtest deviate strongly from the other subtests, in that nonnative speakers perform quite well. Some qualitative analysis of test-taking behavior indicates that these students are solving all the calculation problems in their native languages—Pascal, after all, did arithmetic in French and algebra in English—with considerably higher success rates than native English speakers. Because there are relatively few analytical problems on the test, this excellent performance on the calculations masks somewhat more mixed performance by the nonnative speakers on the problems requiring reading and analyzing a problem.
- in the second language of students who start the process of second-language learning with stronger language and literacy skills in the native language. But such transfer is not automatic; it occurs only when conditions for the emergence of the analogous second-language skills exist, and it can be aided by explicit support for the process of transfer.
- Bilinguals do not typically replicate their capacities across their two languages, unless the array of task demands in the two languages are similar. Thus, estimating total vocabulary of a bilingual requires, in effect, adding across independent vocabulary assessments in the two languages; and allowing a bilingual full opportunity to display knowledge may require allowing code-switching (the use of either or both languages during the same speech event). Similarly, assessing the full extent of the
- knowledge of a bilingual in any domain, which may or may not be a reasonable goal, may well necessitate some procedure for compiling knowledge across two languages.
If promotion and tracking decisions are meant to determine which available placement or treatment is most likely to benefit individual students, then it seems clear, given the relation between first-language accomplishments and likely performance in second-language settings, that first-language testing must play a role in these decisions for English-language learners. To predict whether a Spanish-speaking high school junior who has just arrived in the United States will be able to function in an advanced algebra/trigonometry class, it is much more important to know how much algebra she has mastered than to know how much English she speaks. And to predict whether a Haitian Creole 12-year-old will be able to move into secondary school after a year of elementary schooling in the United States, it is much more important to have available information about his literacy skills in Creole and/or French than to know how much English he speaks right now.
Psychometric and Measurement Issues
As we have stated throughout this report, the use of any assessment should meet the basic professional standards of validity and reliability (American Educational Research Association et al., 1985). But if a student is not proficient in the language of the test, her performance is likely to be affected by construct-irrelevant variance—that is, her test score is likely to underestimate her knowledge of the subject being tested. How can a student's content knowledge be validly assessed by someone who speaks only English if that student cannot readily express what she knows in English?
Some validity problems come from the "mainstream bias" of formal testing, including a norming bias (small numbers of English-language learners in the sample, making it potentially unrepresentative); from content bias (the test reflects the dominant-culture standards of language, knowledge, and behavior); and from linguistic and cultural biases affecting students' formal test performance (timed testing, difficulty with English vocabulary, and the great difficulty of determining what bilingual students know in their two languages) (National Research Council and Institute of Medicine, 1997:115).
When non-English versions of a test are used, other problems arise, including translation and score equivalence: Is the translated test comparable in content to the original? Can scores from the two different language versions be accurately compared? For example, would a score of 50 in one language be interpreted the same way as a score of 50 in another language? (American Educational Research Association et al., 1998:9-4).
"Every assessment is an assessment of language," the committee wrote in Improving Schooling for Language-Minority Children. "This is even more so given the advent of performance assessments requiring extensive comprehension and production of language."5 Research indicates that lack of proficiency in the language of the test can result in significant underestimation of the test taker's knowledge. A further problem is errors in the scoring of open-ended or performance-based measures. There is evidence that scorers may be influenced by linguistic features of students' answers unrelated to the content of the assessment. Thus, scorers may downgrade the performance of English-language learners unfairly, confounding the accuracy of the score (National Research Council and Institute of Medicine, 1997:120–122).
Research that can inform policy and guidelines for making decisions about exemptions, modifications, and accommodations in assessment procedures is urgently needed. A number of such research efforts are under way, including efforts to include more English-language learners in the National Assessment of Educational Progress (NAEP). 6 A recent report of the National Center for Educational Statistics (NCES) describes many of these current research efforts (Olson and Goldstein, 1997); these are summarized in the appendix to this chapter.
The research base regarding various strategies that can be used to enhance the participation of English-language learners in large-scale assessments
(National Research Council and Institute of Medicine, 1997; Olson and Goldstein, 1997) covers several areas:
Use of native-language assessments. Assessments can be developed in languages other than English, a strategy under active investigation. New York state, for example, will offer three of its four core subject Regents examinations in five languages in addition to English (Spanish, Haitian-Creole, Russian, Chinese, and Korean); the English exam must be taken in English.
A number of technical difficulties arise in attempting to create a comparable test in another language. Difficulties include "problems of regional and dialect difference, nonequivalence of vocabulary difficulty between the two languages, problems of incomplete language development and lack of literacy development in students' primary languages, and the extreme difficulty of defining a 'bilingual' equating sample (each new definition of a bilingual sample will demand a new statistical equating). Minimally, back-translation should be done to determine equivalent meaning, and ideally, psychometric validation should be undertaken as well" (National Research Council and Institute of Medicine, 1997:121).
Results of a recent NAEP field test of mathematics items illustrates the challenge of using native-language assessments (Anderson et al., 1996). "Spanish-language items were translations of English-version items. This research found substantial psychometric discrepancies in students' performance on the same test items across both languages, leading to the conclusion that the Spanish and English versions of many test items were not measuring the same underlying mathematical knowledge. This result may be attributable to a lack of equivalence between original and translated versions of test items and needs further investigation" (National Research Council and Institute of Medicine, 1997:122).
Implementation of non-English-language achievement tests as part of assessment systems needs to be accompanied by research establishing the validity and reliability of such assessments and their comparability to scores on related assessments in English.
Decreasing the English-language load through modification of items or instructions. This is difficult to do and research thus far is limited. "While some experts recommend reducing nonessential details and simplifying grammatical structures (Short, 1991), others claim that simplifying the
surface linguistic features will not necessarily make the text easier to understand (Saville-Troike, 1991). When Abedi et al. (1995) reduced the linguistic complexity of the NAEP mathematics test items in English, they reported only a modest and statistically unreliable effect in favor of the modified items for students at lower levels of English proficiency" (National Research Council and Institute of Medicine, 1997:122). Although this approach simplified sentence structure, semantic simplification might also be beneficial. One would need to decide, however, whether to simplify vocabulary directly related to the content being assessed, less related to the content, or both. Also, one would have to determine whether it is possible to simplify vocabulary without compromising conceptual complexity. Another approach to modifying text is to make the instructions more explicit.
Accommodations. The 1996 NAEP mathematics assessment included field trials of new criteria for including English-language learners in the assessment and the provision of accommodations. Results suggest that providing more accommodation options increased participation rates for English-language learners.
The diversity of the population of English-language learners is an important factor to consider in researching and designing accommodations. Researchers at the National Center for Research on Evaluation, Standards, and Student Testing have suggested a number of important background variables to examine with respect to their impact on test performance, including English-language proficiency, prior formal schooling, and length of time in the United States (Butler and Stevens, 1997). These researchers point out that two major obstacles to conducting research and to systematizing the procedures for including English-language learners are (1) inconsistencies in the definitions of English-language learners and (2) lack of agreement on common methods for measuring academic proficiency in a language. They argue that "any research on accommodations must begin by addressing these two issues" (p. 22).
A 1994 NCES conference on including English-language learners in assessments (National Center for Education Statistics, 1996) highlighted the need for a definition of English proficiency that could be consistently applied across states, districts, and schools (Olson and Goldstein, 1997). Related issues included the following:
- The best criterion to determine readiness to meaningfully participate in an English-language assessment is level of English literacy, rather than years in English-only instruction (or native-language instruction) or other background characteristics. This is because years in English-only instruction may not accurately predict English literacy, given variation in language acquisition due to individual factors and home, school, and community linguistic contexts.
- A measure of proficiency should not be limited to oral language proficiency, because such a measure is not sufficient to determine whether an English-language learner can meaningfully participate in a written assessment.
- Implementation of an approach that tailors testing to a student's English literacy level would require the development, validation, and adoption of a standard procedure to determine levels of English literacy. An empirically determined threshold level would indicate that the student could take the standard English assessment. Similar thresholds could be established for modified versions of the standard English assessment. An alternative would be to use current scores, including literacy subtests of language proficiency tests or reading/language arts scores on standardized achievement tests or other assessments.
- When a test is available in multiple languages, and when testing a student proficient in two or more languages for which the test is available, the student's relative language proficiencies should be determined, and the test generally should be administered in the test taker's most proficient language—unless the test is designed to determine proficiency in a certain language (American Educational Research Association et al., 1998).
- Generally, decisions about the language of assessment should take into account how much instruction in the native language students have received in the specific content to be assessed, i.e., reading or math (National Center for Education Statistics, 1996). However, because of the complexity of deciding which English-language learners are most appropriately assessed in their native language,7 this decision might be left up
- to states or local school committees who have deliberated and developed clear policy and assessment guidelines (as in Texas) for such decisions.
The NRC and IOM committee that wrote Improving Schooling for Language-Minority Children called for more research that would inform decision making about how to get valid test scores for English-language learners in large-scale assessments, particularly in developing guidelines for determining when they are ready to take the same tests as their English-proficient peers (1997:130).
The committee's findings and recommendations about English-language learners are reported in Chapter 12.
Including Students with Limited English Proficiency in Large-Scale Assessments: Summary of Current and Ongoing Research Activities
This appendix is a catalogue of the projects summarized and described in more detail in Olson and Goldstein (1997:44–58).
National Center for Education Statistics (NCES)
This research program proposes to examine issues around the participation of English-language learners in large-scale assessments, including the National Assessment of Educational Progress (NAEP). The following is a list of these activities.
- NCES Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students in NAEP. This report summarizes the results from a conference on the inclusion of English-language learners in assessments. Many issues are raised including possible modifications in NAEP administration procedures and the need to develop a set of guidelines and accommodations to promote increased participation of English-language learners in NAEP (National Center for Education Statistics, 1996).
- Working Paper on Assessing Students with Disabilities and Limited English Proficiency. This NCES working paper describes the current state of NCES policies that resulted in the exclusion of some students from NCES assessments, as well as concerns about data validity, assessment modifications, and the inclusion of English-language learners (Houser, 1995).
Office of Bilingual Education and Minority Languages Affairs (OBEMLA)
The Office of Bilingual Education and Minority Languages Affairs provides funding for research on the education of language minority students. Some of its activities include grants to various agencies and services, including the National Clearinghouse for Bilingual Education, which collects, analyzes, synthesizes, and disseminates information on studies of assessment and performance of English-language learners.
National Academy of Education (NAE)
Studies on the inclusion and accommodation of English-language learners have been carried out along with the NAE evaluation of NAEP. They focus on assessment development, exclusion decisions, and appropriate adaptations and accommodations for English-language learners. Several reports contain the results of these studies.
The Prospects Study
Previously mandated by Congress to evaluate Chapter 1 (now Title I), this study has been funded to examine English-language learners in the Title I program. It provides data collected from students, parents, and educators, on samples of students who were in 1st, 3rd, or 7th grade during the academic year 1991–1992 and were in schools with high concentrations of English-language learners (U.S. Department of Education, 1993, 1996).
Stanford Working Group
- This group makes recommendations regarding state-level assessments and Goals 2000. Some examples of recommendations include state development of assessments appropriate for English-language learners and the use of native-language and other alternative assessments (August et al., 1994).
- A working paper was prepared in 1994 both to help develop strategies for making appropriate and consistent decisions about assessment of English-language learners and to maximize their participation, particularly in NAEP (see National Center for Education Statistics, 1996).
Council of Chief State School Officers State Collaboration of Assessment and Student Standards (SCASS)
The purpose of the SCASS project is to develop guidelines for effective language learning and the assessment of English-language learners on statewide content standards for language and core content areas. SCASS has developed plans to improve assessments of language proficiency for selection and placement purposes; develop appropriate content assessments; conduct research on learning English; and work on the development of accommodations for English-language learners. A pilot study of the guidelines was planned for academic year 1996–1997 on accurate scoring for content knowledge.
National Center for Research on Evaluation, Standards and Student Testing (CRESST)
CRESST studies focus on the assessment of quality education. Its current studies of language issues look specifically at the linguistic features of NAEP items and how they may affect the performance of students with background in languages other than English (e.g., see Abedi et al., 1995; Shepard, 1996).
George Washington University Center for Equity and Excellence in Education (GWU/CEEE)
The goal of these research activities is to provide tools for policymakers, educators, and community members looking to develop strategies for designing and evaluating services for English-language learners. One report summarizes findings from a national survey of state assessment directors on the prevalence and use of assessments, modifications in assessments, and exemption of English-language learners from assessment programs (Rivera et al., 1997). Another report documents data from the GWU/CEEE and SSAP Council of Chief State School Officers/NCREL surveys on assessment policies for English-language learners in states requiring a test for high school graduation (Rivera and Vincent, 1997).
National Research Council/Institute of Medicine
A 1997 report of the National Research Council and the Institute of Medicine contributes to the understanding of the education of English-language learners, reviews methodologies used, discusses assessment issues, and makes recommendations regarding future research in order to inform policy and practice (National Research Council and Institute of Medicine, 1997). See also National Research Council (1999).
Abedi, J., C. Lord, and J. Plummer 1995 Language Background as a Variable in NAEP Mathematics Performance. National Center for Research on Evaluation, Standards, and Student Testing. Los Angeles: University of California at Los Angeles.
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education 1985 Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association.
1998 Draft Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association.
Anderson, N.E., E.F. Jenkins, and K.E. Miller 1996 NAEP Inclusion Criteria and Testing Accommodations. Findings from the NAEP 1995 Field Test in Mathematics. Washington, DC: Educational Testing Service.
August, D., K. Hakuta, and D. Pompa 1994 For all students: Limited English proficient students and Goals 2000. Occasional Papers in Bilingual Education 10(1994):4.
Berman, P., J. Chambers, P. Gandara, B. McLaughlin, C. Minicucci, B. Nelson, L. Olsen, and T. Parrish 1992 Meeting the Challenge of Language Diversity: An Evaluation of Programs for Pupils with Limited Proficiency in English. Vol. 1 [R-119/1: Executive Summary; Vol. 2 [R-119/2]: Findings and Conclusions; Vol. 3 [R-119/3]: Case Study Appendix. Berkeley, CA: BW Associates.
Butler, F.A., and R. Stevens 1997 Accommodation Strategies for English Language Learners on Large-Scale Assessments: Student Characteristics and Other Considerations. CSE Technical Report 448. National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles.
Council of Chief State School Officers 1998 Survey of State Student Assessment Programs. Washington, DC: Council of Chief State School Officers.
Durán, Richard P. 1989 Assessment and instruction of at-risk Hispanic students. Exceptional Children 56(2):154–158.
Education Week 1998 S.F. files suit over testing of LEP students. Education Week April 8.
Fleishman, Howard L., and Paul J. Hopstock 1993 Descriptive Study of Services to Limited English Proficient Students, Volume 1, Summary of Findings and Conclusions. Prepared for U.S. Department of Education. Arlington, VA: Development Associates, Inc.
Florida Department of Education 1993 November 1, 1993, Memorandum from Walter McCarroll to District Superintendents . Testing Limited English Proficiency Students. Tallahassee, Florida: The Florida Education Center
1996 December 4, 1996, Memorandum from David Mosrie to District Superintendents . Accommodations for Limited English Proficient Students in the Administration of Statewide Assessments. Tallahassee, Florida: The Florida Education Center.
Garcia, G.E. 1991 Factors influencing the English reading test performance of Spanish-speaking Hispanic children. Research Reading Quarterly 26(4):371–392.
Garcia, G.E., and P.D. Pearson 1994 Assessment and diversity. Review of Research in Education (20):337–391.
Houser, J. 1995 Assessing Students with Disabilities and Limited English Proficiency . Working Paper Number 95-13. Washington, DC: National Center for Education Statistics, Policy and Review Branch, Data Development Division.
Meisels, S. 1994 Designing meaningful measurements for early childhood. Pp. 202–222 in Diversity and Developmentally Appropriate Practices: Challenges for Early Childhood Education, B. Mallory and R. New, eds.. New York: Teachers College Press.
National Academy of Education 1993 Third Report to Congress on the evaluation of the NAEP Trial State Assessment. National Academy of Education.
1994 The Trial State Assessment: Prospects and Realities: Background Studies , George Bohrnstedt, ed. American Institutes for Research Palo Alto, CA: Armadillo Press.
1996 Quality and Utility: The 1994 Trial State Assessment in Reading, Background Studies. Stanford, CA: National Academy of Education.
National Center for Education Statistics 1994 A Study Design to Evaluate Strategies for the Inclusion of LEP Students in the NAEP State Trial Assessment. K. Hakuta and G. Valdes, eds. Office of Educational Research and Improvement, U.S. Department of Education, Washington, DC.
1996 Proceedings of the Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students in the National Assessment of Educational Progress (December 5–6, 1994), D. August and E. McArthur, eds. National Center for Education Statistics, Office of Educational Research and Improvement, U.S. Department of Education, Washington, DC.
National Research Council 1999 Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress, J. Pellegrino, L. Jones, and K. Mitchell, eds. Committee on the Evaluation of National and State Assessments of Educational Progress, Board on Testing and Assessment. Washington, DC: National Academy Press.
National Research Council and Institute of Medicine 1997 Improving Schooling for Language-Minority Children, D. August and K. Hakuta, eds. Board on Children, Youth, and Families. Washington, DC: National Academy Press.
New Standards 1995 Performance Standards. English Language Arts, Mathematics, Science, and Applied Learning. Volumes 1, 2, and 3. Consultation Drafts. Washington, DC: National Center for Education and the Economy.
New York State Education Department 1997 Proposal for Revising Graduation Requirements. November 14, 1997 Memorandum from Richard Mills to the Members of the Board of Regents.
1998 Regents Strategy for Intensive Instruction in the English Language for Limited English Proficient Students at the Secondary Level. January 6, 1998 Memorandum from James A. Kadamus to the Board of Regents.
Olson, J.F., and A.A. Goldstein 1997 The Inclusion of Students with Disabilities and Limited English Proficient Students in Large-Scale Assessments: A Summary of Recent Progress . NCES 97–482. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.
Philadelphia School District 1998 Reaching Higher. Consultation Draft. Philadelphia, Penn: School District of Philadelphia.
Rivera, C., and Carolyn Vincent 1997 High school graduation testing: Policies and practices in the assessment of English language learners. Educational Assessment 4(4):335–355.
Rivera, C., C. Vincent, A. Hafner, and M. LaCelle-Peterson 1997 Statewide Assessment Programs: Policies and Practices for the Inclusion of Limited English Proficient Students—Findings from a National Survey of State Assessment Directors. TM #026831. Washington, DC: ERIC Clearinghouse on Assessment and Evaluation. The Catholic University of America.
Saville-Troike, Muriel 1991 Teaching and Testing for Academic Achievement: The Role of Language Development. Focus, Occasional Papers in Bilingual Education, No. 4. Washington, DC: National Clearinghouse for Bilingual Education.
Shepard, L.A. 1996 Research Framework for Investigating Accommodations for Language Minority Students . Presentation made at CRESST Assessment Conference, UCLA.
Shepard, L., S.L. Kagan, and E. Wurtz 1998 Principles and Recommendations for Early Childhood Assessments. The National Education Goals Panel. Washington, D.C.: National Education Goals Panel.
Short, D. 1991 How to Integrate Language and Content Instruction: A Training Manual . Washington, DC: Center for Applied Linguistics.
Texas Education Agency 1996 Assessment System for Limited English Proficient Students Exempted from the Texas Assessment Program at Grades 3–8. Austin, Texas: Texas Education Agency.
1998 Assessment Requirements for Students of Limited English Proficiency in the 1997–98 State Testing Program. Memorandum to Texas Education Agency Administrators. Austin, Texas: Texas Education Agency.
Thurlow, M., K. Liu, S. Weiser, and Hamdy El Sawaf 1997 High School Graduation Requirements in the U.S. for Students with Limited English Proficiency. State Assessment Series Minnesota Report 13. Minneapolis, MN: National Center on Educational Outcomes.
U.S. Department of Education 1993 PROSPECTS: The Congressionally Mandated Study of Educational Growth and Opportunity, Interim Report: Language Minority and Limited English Proficient Students. Washington, DC: U.S. Department of Education.
1996 PROSPECTS: The Congressionally Mandated Study of Educational Growth and Opportunity, First Annual Report. Washington, DC: U.S. Department of Education.
Goals 2000, Educate America Act, 20 U.S.C. sections 5801 et seq.
Lau v. Nichols, 414 U.S. 563 (1974).
Title I, Elementary and Secondary Education Act, 20 U.S.C. sections 6301 et seq.
Title VI, Civil Rights Act of 1964, 42 U.S.C. sections 2000(e) et seq.
Title VII (Bilingual Education Programs), Improving America's Schools Act of 1994, P.L. 103–382.