. "4 Factors That Affect the Accuracy of NAEP's Estimates of Achievement." Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessment. Washington, DC: The National Academies Press, 2004.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments
COMPARABILITY OF NAEP SAMPLES ACROSS STATES
As was mentioned in Chapter 1, decision making about the identification of students with disabilities and English language learners, their inclusion in large-scale assessments, and the testing accommodations they need is guided by federal legislation (although far more detailed guidance is provided regarding students with disabilities than English language learners). It is up to states, however, to develop policies for complying with legislative requirements, and consequently the policies and the way they are interpreted vary from state to state, in some cases considerably. The variation in state policies has particular implications for NAEP. Decisions made at the state and local level affect NAEP’s results and the ways in which they can be interpreted.
For each administration NAEP officials identify a sample of students to participate in the assessment, and they provide guidelines for administering it. However, school-level officials influence the process in several ways. First, as they are developing the sample, NAEP officials make no attempt to identify students with disabilities or English language learners themselves; rather, the percentages of those students who end up in the sample reflect decisions that have already been made at the school level; these decisions are guided by state policies, which vary. Second, NAEP officials leave it to school-level staff, who are knowledgeable about students’ educational functioning levels, to determine whether selected students who have a disability or are English language learners can meaningfully participate. In general, this process is guided by the policy set forth in the NAEP 2003 Assessment Administrators’ Manual (U.S. Department of Education, National Center for Education Statistics, and Office of Educational Research and Improvement, 2003, pp. 4-19). Finally, NAEP officials provide lists of allowable accommodations for each of its assessments, but here as well it is school-level staff who decide which accommodations are appropriate for their students and which of those allowed in NAEP they are in a position to offer. Thus, differences in policies and procedures both among and within states can affect who participates in NAEP and the way in which students participate.
According to the most recent legislation, the purpose of NAEP is “to provide, in a timely manner, a fair and accurate measurement of student academic achievement and reporting of trends in such achievement in reading, mathematics, and other subject matter as specified in this section” (Section 303 of HR 3801). The legislation further indicates that the commissioner for education statistics shall:
(a) use a random sampling process which is consistent with relevant, widely accepted professional assessment standards and that produces data that are representative on a national and regional basis;
(b) conduct a national assessment and collect and report assessment data, including achievement data trends, in a valid and reliable manner on student academic achievement in public and private elementary schools and secondary schools at least once every two years, in grades 4 and 8 in reading and mathematics;