3
NAEP’s Influence on State Instructional and Assessment Programs
Currently, most states have established content standards and have developed assessment instruments designed to measure their students ’ mastery of these standards. These standards and assessment instruments vary greatly from state to state, however (Olson et al., in press), and this variation precludes equitable and credible comparisons of student performance using state assessment results (National Research Council, 1999c). Comparisons are made possible by state NAEP. While participation is voluntary, the majority of states have participated in state NAEP since its implementation.
State NAEP reports results in the same ways as national NAEP (see Appendix B), that is, through summaries of performance for the state as a whole and by demographic and background variables using scaled scores and achievement levels. States’ uses of these data and their reasons for participating were studied by DeVito (1997). One of the chief reasons states participate, according to DeVito, is to obtain an external reference point for comparing the results of their own assessments and to enable state-to-state and state-to-national comparisons. Moreover, states reported that they use the results to argue for more rigor in their curricula and standards, to examine curricular strengths and weaknesses relative to testing frameworks, and to study NAEP item formats as exemplars. Many states have adopted the NAEP models for standards-based reporting and use NAEP-like achievement levels.
The committee was interested in hearing firsthand discussion of the
uses states make of NAEP data. In soliciting participation for the workshop, the committee sought to identify NAEP-participating states that had experienced changes in educational or assessment policy. For example, California recently altered the state’s reading curriculum and teaching practices based in part on their students’ low reading performance on state NAEP (Jennings et al., 1997). The committee was interested in hearing about California’s experience. Other states identified as changing state policy due to NAEP performance were Delaware, Oregon, North Carolina, South Carolina, and Washington. Officials from Washington and South Carolina were invited and able to attend, and the director of assessment for North Carolina serves on the committee. The committee also sought regional representation in the participants. We identified Colorado, Connecticut, Louisiana, Maryland, and Texas, all states with extensive assessment programs in place. Of these states, officials from Colorado and Connecticut were able to attend.
As the purpose of this panel was to understand NAEP’s influence on state instructional and assessment programs, representatives from state assessment offices in Colorado, Connecticut, South Carolina, and Washington were asked to share their experiences on if, and how, state NAEP has affected educational policy, instructional practices, and curricular decisions. Their experiences with their state’s participation in state NAEP are included in this chapter; their comments on issues pertaining to district-level reporting of NAEP are incorporated into later chapters.
NAEP FRAMEWORKS GUIDE STATE ASSESSMENTS
A common theme voiced by the four state representatives on the panel was the utilization of NAEP frameworks as a resource during the development of their state curriculum standards and the design of state assessments. For example, Connecticut’s reading mastery test is built on aspects of the NAEP reading literacy frameworks. According to Peter Behuniak, Connecticut’s director of student assessment and testing, the reading comprehension component of the Connecticut Mastery Test in language arts “directly reflects the philosophy of the NAEP frameworks.” Connecticut’s reading components include all but the NAEP’s personal reflection stance.
NAEP’s mathematics frameworks influenced the development of the state of Washington’s mathematics standards. A high degree of alignment between state content standards and the NAEP content frameworks adds credibility to the state standards, according to the assessment directors, and
facilitates comparisons of student performance. When the results of NAEP and the state assessment are in accord, then the NAEP results lend validity to the state results. All three speakers spoke of the problems that arise when the standards are comparable, but comparisons of student performance on NAEP and state tests are not congruent (Chapter 4 addresses these types of comparisons in depth).
ACHIEVEMENT-LEVEL REPORTING IS INFORMATIVE
Speakers also commented on NAEP’s use of achievement levels to summarize performance. Don Watson, acting director of student assessment in Colorado’s Department of Education, commented that the achievement levels provide a clearer representation of achievement for the public than is possible with numerical scores. In fact, the achievement-level descriptors were so well received in Colorado that the use of similar descriptors was implemented within the state system.
Robert Silverman, Washington’s senior analyst for assessment, noted that reporting of NAEP performance by achievement levels had driven changes in his state’s policy as well. Results from a recent NAEP administration revealed that 60 percent of their students performed below the proficient level in reading. State legislators interpreted this finding as meaning that their students lacked essential reading skills and advocated for revisions in the state reading instruction and assessment program. Under the amended system, students take an oral reading test in second grade, which allows for early identification and remediation of reading problems. Low-performing students then receive an individualized reading program designed to improve their reading mastery.
NAEP INCLUSION PROCEDURES SERVE AS MODELS
In designing their assessment systems, states have used the NAEP model for inclusion of students with disabilities or limited English proficiency as a reference point in developing their own inclusion procedures. Watson commented that his state, Colorado, revised its policies on the basis of NAEP guidelines. The role of NAEP as the key indicator of academic achievement of all students across the country means that assessment results must include data gathered from students with disabilities and English-language learners. The accommodations and modifications imple-
mented by state NAEP include large print, Braille, and bilingual test versions, smaller testing settings, and untimed versions.
Although NAEP’s inclusion and accommodation policies have served as a model for states in modifying their inclusion and accommodation procedures for students with disabilities and those with limited English proficiency, it should also be noted that some states have broader inclusion policies than NAEP. The difference in state-developed and NAEP-established inclusion policies has caused some students to be included in state testing programs but precluded from participation in NAEP.
NAEP ITEM DESIGN IS INNOVATIVE
Speakers agreed that NAEP has been innovative in the design of test items. The release of NAEP items has been useful in guiding item development for their state assessment measures. For example, the use of performance assessments and constructed response questions in NAEP has led to the inclusion of similarly formatted questions in their state instruments. Furthermore, speakers acknowledged that, in many cases, the research involved in developing NAEP items has been more extensive than is possible within state research divisions. For this reason, speakers indicated that they feel quite comfortable using the NAEP design as a model in developing their state tests.
BACKGROUND AND CONTEXTUAL INFORMATION IS USEFUL
Panelists expressed appreciation for the background and contextual information provided by NAEP, as some states collect limited background information on students and school practices. The student questionnaires provide information beyond race/ethnicity and school attendance to include factors thought to influence academic performance, such as language spoken in the home, study and homework habits, and motivation toward school. The teacher questionnaires include a variety of information, such as the training of the teacher, kind of degree attained, number of years of teaching, the amount of control teachers have over instructional issues, and their instructional practice. This information is useful in studying the relationships between background or environmental factors and performance.
PROBLEMATIC ISSUES ASSOCIATED WITH STATE NAEP
The speakers agreed, and workshop participants concurred, that the time of year NAEP is administered is an issue. NAEP administrations occur between February and April, months when states schedule their assessments. The timing conflict has interfered with some states ’ participation in state NAEP (e.g., Illinois).
Another concern voiced by the speakers was the desire not to overtest students, since many states currently test students in fourth and eighth grade, as does NAEP. In one instance, Robert Silverman remarked that Washington modified their state testing sequence to accommodate NAEP’s schedule; they now assess students in third grade instead of fourth grade.
Speakers also stressed that the staff time commitment required to seek participation from schools is substantial. Some schools are reluctant to participate when they learn that scores for their school will not be provided, commenting that participation is not worth the time and effort required.