Implications of the Committee’s Conclusions and Recommendations
IMPLICATIONS FOR NAEP
The work of this committee was stimulated by concerns about including students with disabilities and English language learners in the National Assessment of Educational Progress (NAEP) and about the effects that accommodations they receive may have on their performance. The National Center for Education Statistics, which initiated the study, asked the NRC to examine NAEP’s policies regarding the inclusion of students with disabilities and English language learners in its assessments, the use of accommodations for these students, and a variety of questions related to these issues. In response, the committee has focused much of its attention on reasons why NAEP officials should review their policies in both of these areas, compare them with those of the states, and work toward greater consistency between NAEP’s policies and those used in the states.
We have recommended that NAEP continue to pursue the goal of maximizing the participation of all students, and that NAEP officials be mindful of the requirements states must meet under the No Child Left Behind Act, recognizing that even though no formal comparisons of state scores to NAEP results are required, NAEP is frequently viewed as an informal benchmark for state results. We hope that this report makes evident that the committee believes it is very important that the sometimes stark differences between the policies in effect for NAEP and those used by states be recognized, and, to the extent possible, reduced.
At the same time, NAEP has in many ways been a model for the assessment community. Much groundbreaking research has been conducted by NAEP’s sponsors and others associated with the NAEP program, such as Educational
Testing Service (ETS). The developers of NAEP assessments have taken advantage of advances in educational measurement and have been early users of new item types and modes of assessment, such as constructed-response formats, items that require the use of calculators, items that require students to interact with objects such as atlases or scientific specimens, and the like. The NAEP Arts assessment, for example, has demonstrated the willingness of NAEP’s sponsors and assessment developers to explore new ways of gaining information about students’ knowledge and skills. With examples such as these, NAEP has motivated many states to consider and adopt more creative testing approaches than they had been using (National Research Council, 2000a).
Given this record, the committee believes that NAEP has a responsibility to take the lead as well on the complex issues surrounding the assessment of students with disabilities and English language learners. NAEP has published one study on the validity of accommodations that uses an external criterion variable (Weston), and several other studies are underway. We encourage NAEP to continue to pursue research that follows this model. More generally, we hope that the recommendations in this report will spur NAEP’s sponsors to set an example of forward-looking research and practice in these areas.
IMPLICATIONS FOR STATES
Although most of the committee’s messages have been directed to those who develop policy for NAEP and who are responsible for its technical soundness, the issues raised in this report have important implications for state and local assessment programs as well. Points that are relevant to states have been mentioned in several places, and the committee has offered one specific recommendation to states (Recommendation 4-5). However, we close the report with a discussion of the broader implications of our findings and conclusions for states because it is the states that are struggling most immediately with the technical challenges of assessing students with disabilities and English language learners.
The No Child Left Behind legislation has been the source of considerable urgency for states’ efforts to collect data about students’ academic achievement. As we have noted in earlier chapters, the requirement for assessing students and managing the resulting data under this law is steep. Yet previous NRC committees and others have already documented evidence that the collection and reporting of assessment results for these two groups were insufficient even before the No Child Left Behind Act took effect (e.g., National Research Council, 1997a, 1997b, 1999a; Thurlow et al., 2002), and the need for such data does not arise solely because of that law.
With regard to students with disabilities, an NRC committee specifically noted that large-scale studies of education issues frequently fail to include these students in their samples or include them in ways that are not systematic (National Research Council, 1997a, pp. 209-210). That committee called for data on how
students with disabilities compare with other students on many variables related to their schooling and educational achievement. They pointed out the need for research to investigate the relationship between accommodations and validity, not only to document the effects of various accommodations on test scores but also to develop criteria for deciding which accommodations will preserve the validity and usefulness of test results. This committee also noted the need for research to support the development of reliable and valid alternate assessments, and the equating and scaling of such alternatives.
There are even fewer data available for English language learners. For example, simply finding out how many fourth graders across the country are English language learners cannot readily be done with available data. National demographic data on limited English proficient students have been gathered for decades in the U.S. Department of Education’s Survey of the States’ Limited English Proficient Students and Available Education Programs and Services of Education. However, the quality of the data is uncertain because different states have used different criteria for the identification of students of limited English proficiency (summary reports available from National Clearinghouse on English Language Acquisition: http://www.ncela.gwu.edu). An NRC report on the schooling of language minority children identified specific research needs related to the assessment of these children (National Research Council, 1997b, pp. 128-131). It noted in particular the need for research that links assessment strategies to the knowledge base regarding language acquisition, and that can assist in the development of guidelines for when and how to assess these students, and for the development of accurate and consistent means of scoring the work of English language learners.
There is in general, as we have discussed, a lack of research findings that could be used to help policy makers, administrators, and others make decisions about when and how to offer accommodations to students with different kinds of needs. There is a particular need for research into the validity of interpretations made from accommodated scores, and we have described the direction we think this kind of research should take. We would encourage local education officials and schools to undertake such research, to take part in such research when it is conducted by others, and to provide data to researchers engaged in such research. We believe that such research is a crucial element in the effort to build an education system that strives to meet the needs of all students. In order that resources be adequately managed and that the needs of all students be recognized and addressed, those responsible for meeting these goals must have accurate information about all students.