National Academies Press: OpenBook

Reporting District-Level NAEP Data: Summary of a Workshop (2000)

Chapter: 3 NAEP's Influence on State Instructional and Assessment Programs

« Previous: 2 Background
Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

3

NAEP’s Influence on State Instructional and Assessment Programs

Currently, most states have established content standards and have developed assessment instruments designed to measure their students ’ mastery of these standards. These standards and assessment instruments vary greatly from state to state, however (Olson et al., in press), and this variation precludes equitable and credible comparisons of student performance using state assessment results (National Research Council, 1999c). Comparisons are made possible by state NAEP. While participation is voluntary, the majority of states have participated in state NAEP since its implementation.

State NAEP reports results in the same ways as national NAEP (see Appendix B), that is, through summaries of performance for the state as a whole and by demographic and background variables using scaled scores and achievement levels. States’ uses of these data and their reasons for participating were studied by DeVito (1997). One of the chief reasons states participate, according to DeVito, is to obtain an external reference point for comparing the results of their own assessments and to enable state-to-state and state-to-national comparisons. Moreover, states reported that they use the results to argue for more rigor in their curricula and standards, to examine curricular strengths and weaknesses relative to testing frameworks, and to study NAEP item formats as exemplars. Many states have adopted the NAEP models for standards-based reporting and use NAEP-like achievement levels.

The committee was interested in hearing firsthand discussion of the

Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

uses states make of NAEP data. In soliciting participation for the workshop, the committee sought to identify NAEP-participating states that had experienced changes in educational or assessment policy. For example, California recently altered the state’s reading curriculum and teaching practices based in part on their students’ low reading performance on state NAEP (Jennings et al., 1997). The committee was interested in hearing about California’s experience. Other states identified as changing state policy due to NAEP performance were Delaware, Oregon, North Carolina, South Carolina, and Washington. Officials from Washington and South Carolina were invited and able to attend, and the director of assessment for North Carolina serves on the committee. The committee also sought regional representation in the participants. We identified Colorado, Connecticut, Louisiana, Maryland, and Texas, all states with extensive assessment programs in place. Of these states, officials from Colorado and Connecticut were able to attend.

As the purpose of this panel was to understand NAEP’s influence on state instructional and assessment programs, representatives from state assessment offices in Colorado, Connecticut, South Carolina, and Washington were asked to share their experiences on if, and how, state NAEP has affected educational policy, instructional practices, and curricular decisions. Their experiences with their state’s participation in state NAEP are included in this chapter; their comments on issues pertaining to district-level reporting of NAEP are incorporated into later chapters.

NAEP FRAMEWORKS GUIDE STATE ASSESSMENTS

A common theme voiced by the four state representatives on the panel was the utilization of NAEP frameworks as a resource during the development of their state curriculum standards and the design of state assessments. For example, Connecticut’s reading mastery test is built on aspects of the NAEP reading literacy frameworks. According to Peter Behuniak, Connecticut’s director of student assessment and testing, the reading comprehension component of the Connecticut Mastery Test in language arts “directly reflects the philosophy of the NAEP frameworks.” Connecticut’s reading components include all but the NAEP’s personal reflection stance.

NAEP’s mathematics frameworks influenced the development of the state of Washington’s mathematics standards. A high degree of alignment between state content standards and the NAEP content frameworks adds credibility to the state standards, according to the assessment directors, and

Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

facilitates comparisons of student performance. When the results of NAEP and the state assessment are in accord, then the NAEP results lend validity to the state results. All three speakers spoke of the problems that arise when the standards are comparable, but comparisons of student performance on NAEP and state tests are not congruent (Chapter 4 addresses these types of comparisons in depth).

ACHIEVEMENT-LEVEL REPORTING IS INFORMATIVE

Speakers also commented on NAEP’s use of achievement levels to summarize performance. Don Watson, acting director of student assessment in Colorado’s Department of Education, commented that the achievement levels provide a clearer representation of achievement for the public than is possible with numerical scores. In fact, the achievement-level descriptors were so well received in Colorado that the use of similar descriptors was implemented within the state system.

Robert Silverman, Washington’s senior analyst for assessment, noted that reporting of NAEP performance by achievement levels had driven changes in his state’s policy as well. Results from a recent NAEP administration revealed that 60 percent of their students performed below the proficient level in reading. State legislators interpreted this finding as meaning that their students lacked essential reading skills and advocated for revisions in the state reading instruction and assessment program. Under the amended system, students take an oral reading test in second grade, which allows for early identification and remediation of reading problems. Low-performing students then receive an individualized reading program designed to improve their reading mastery.

NAEP INCLUSION PROCEDURES SERVE AS MODELS

In designing their assessment systems, states have used the NAEP model for inclusion of students with disabilities or limited English proficiency as a reference point in developing their own inclusion procedures. Watson commented that his state, Colorado, revised its policies on the basis of NAEP guidelines. The role of NAEP as the key indicator of academic achievement of all students across the country means that assessment results must include data gathered from students with disabilities and English-language learners. The accommodations and modifications imple-

Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

mented by state NAEP include large print, Braille, and bilingual test versions, smaller testing settings, and untimed versions.

Although NAEP’s inclusion and accommodation policies have served as a model for states in modifying their inclusion and accommodation procedures for students with disabilities and those with limited English proficiency, it should also be noted that some states have broader inclusion policies than NAEP. The difference in state-developed and NAEP-established inclusion policies has caused some students to be included in state testing programs but precluded from participation in NAEP.

NAEP ITEM DESIGN IS INNOVATIVE

Speakers agreed that NAEP has been innovative in the design of test items. The release of NAEP items has been useful in guiding item development for their state assessment measures. For example, the use of performance assessments and constructed response questions in NAEP has led to the inclusion of similarly formatted questions in their state instruments. Furthermore, speakers acknowledged that, in many cases, the research involved in developing NAEP items has been more extensive than is possible within state research divisions. For this reason, speakers indicated that they feel quite comfortable using the NAEP design as a model in developing their state tests.

BACKGROUND AND CONTEXTUAL INFORMATION IS USEFUL

Panelists expressed appreciation for the background and contextual information provided by NAEP, as some states collect limited background information on students and school practices. The student questionnaires provide information beyond race/ethnicity and school attendance to include factors thought to influence academic performance, such as language spoken in the home, study and homework habits, and motivation toward school. The teacher questionnaires include a variety of information, such as the training of the teacher, kind of degree attained, number of years of teaching, the amount of control teachers have over instructional issues, and their instructional practice. This information is useful in studying the relationships between background or environmental factors and performance.

Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×

PROBLEMATIC ISSUES ASSOCIATED WITH STATE NAEP

The speakers agreed, and workshop participants concurred, that the time of year NAEP is administered is an issue. NAEP administrations occur between February and April, months when states schedule their assessments. The timing conflict has interfered with some states ’ participation in state NAEP (e.g., Illinois).

Another concern voiced by the speakers was the desire not to overtest students, since many states currently test students in fourth and eighth grade, as does NAEP. In one instance, Robert Silverman remarked that Washington modified their state testing sequence to accommodate NAEP’s schedule; they now assess students in third grade instead of fourth grade.

Speakers also stressed that the staff time commitment required to seek participation from schools is substantial. Some schools are reluctant to participate when they learn that scores for their school will not be provided, commenting that participation is not worth the time and effort required.

Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 14
Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 15
Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 16
Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 17
Suggested Citation:"3 NAEP's Influence on State Instructional and Assessment Programs." National Research Council. 2000. Reporting District-Level NAEP Data: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9768.
×
Page 18
Next: 4 Comparisons with National Benchmarks: Pros and Cons »
Reporting District-Level NAEP Data: Summary of a Workshop Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Assessment of Education Progress (NAEP) has earned a reputation as one of the nation's best measures of student achievement in key subject areas. Since its inception in 1969, NAEP has summarized academic performance for the nation as a whole and, beginning in 1990, for the individual states. Increasingly, NAEP results get the attention of the press, the public, and policy makers. With this increasing prominence have come calls for reporting NAEP results below the national and state levels. Some education leaders argue that NAEP can provide important and useful information to local educators and policy makers. They want NAEP to serve as a district-level indicator of educational progress and call for NAEP results to be summarized at the school district level.

Reporting District-Level NAEP Data explores with various stakeholders their interest in and perceptions regarding the likely impacts of district level reporting.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!