1

Introduction

The National Assessment of Education Progress (NAEP) has earned a reputation as one of the nation’s best measures of student achievement in key subject areas. Since its inception in 1969, NAEP has summarized academic performance for the nation as a whole and, beginning in 1990, for the individual states. Increasingly, NAEP results get the attention of the press, the public, and policy makers. With this increasing prominence have come calls for reporting NAEP results below the national and state levels. Some education leaders argue that NAEP can provide important and useful information to local educators and policy makers. They want NAEP to serve as a district-level indicator of educational progress and call for NAEP results to be summarized at the school district level.

At the same time, others have called for simpler, more intuitive and meaningful reporting of NAEP results. Advisers to the National Assessment Governing Board (NAGB) and the National Center for Educational Statistics (NCES) have proposed the use of market-basket reporting methods as one means to accomplish this. Market-basket reporting would allow results to be reported as percentages of items correct on sets of representative items. As part of their evaluation of NAEP, the National Research Council’s Committee on the Evaluation of National and State Assessments of Educational Progress stressed the need for clear and comprehensible reporting metrics that would simplify the interpretation of results and endorsed the concept of market-basket reporting for NAEP (National Re-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
REPORTING DISTRICT-LEVEL NAEP DATA: SUMMARY OF A WORKSHOP 1 Introduction The National Assessment of Education Progress (NAEP) has earned a reputation as one of the nation’s best measures of student achievement in key subject areas. Since its inception in 1969, NAEP has summarized academic performance for the nation as a whole and, beginning in 1990, for the individual states. Increasingly, NAEP results get the attention of the press, the public, and policy makers. With this increasing prominence have come calls for reporting NAEP results below the national and state levels. Some education leaders argue that NAEP can provide important and useful information to local educators and policy makers. They want NAEP to serve as a district-level indicator of educational progress and call for NAEP results to be summarized at the school district level. At the same time, others have called for simpler, more intuitive and meaningful reporting of NAEP results. Advisers to the National Assessment Governing Board (NAGB) and the National Center for Educational Statistics (NCES) have proposed the use of market-basket reporting methods as one means to accomplish this. Market-basket reporting would allow results to be reported as percentages of items correct on sets of representative items. As part of their evaluation of NAEP, the National Research Council’s Committee on the Evaluation of National and State Assessments of Educational Progress stressed the need for clear and comprehensible reporting metrics that would simplify the interpretation of results and endorsed the concept of market-basket reporting for NAEP (National Re-

OCR for page 1
REPORTING DISTRICT-LEVEL NAEP DATA: SUMMARY OF A WORKSHOP search Council, 1999a). Market-basket reporting would provide an easier-to-understand picture of students’ academic accomplishments. In pursuit of improved reporting and use of test results, NAEP’s stewards are exploring the feasibility and potential impact of district-level and market-basket reporting practices. Accordingly, at the request of the U.S. Department of Education, the National Research Council established the Committee on NAEP Reporting Practices to examine the feasibility and potential impact of district-level and market-basket reporting practices. Because these two topics are intertwined, the committee is examining them in tandem, focusing first on district-level reporting. During the course of the study, the committee is seeking to answer the following questions regarding district-level reporting for NAEP: What are the characteristics of district-level NAEP? If implemented, what information needs might it serve? What is the degree of interest in participating in district-level NAEP? What factors would influence interest? Would district-level NAEP pose any threats to the validity of inferences from national and state NAEP? What are the implications of district-level reporting for other state and local assessment programs? To begin to address these questions, the committee convened the Workshop on District-Level Reporting for NAEP on September 16 and 17, 1999. Although this workshop relates to one of the committee’s charges, it was not intended to bring closure on issues related to district-level NAEP. The committee’s work will continue with a workshop on market-basket reporting in February 2000, and joint consideration of the two issues will be taken up in the final report. WORKSHOP ON DISTRICT-LEVEL REPORTING The purpose of the National Research Council’s Workshop on District-Level Reporting for NAEP was to explore with various stakeholders their interest in and perceptions regarding the likely impacts of district-level reporting. NCES has, to date, had two experiences with district-level reporting. In 1996, NCES contacted several of the larger school districts in the country to gauge their interest in receiving district-level results. The

OCR for page 1
REPORTING DISTRICT-LEVEL NAEP DATA: SUMMARY OF A WORKSHOP data collected by these districts could potentially meet the requirements for district-level reporting through augmentation of the state NAEP samples, although there would be a fee associated with the augmentation procedures. In 1998, NCES identified several districts that met the sample size requirements “naturally” as a result of the state NAEP sampling procedures; these districts are referred to as the “naturally occurring districts.” Additional details on these experiences with district-level reporting appear in Chapter 2. The workshop consisted of four panels, each with a specific goal. The opening panel was designed to provide broad context for the two-days of workshop discussions. This panel explored the purposes that district-level reporting might serve, discussed who might use the results and how they might be used, and highlighted the key issues that should be considered. Panelists included two individuals who had earlier authored papers discussing the advantages and disadvantages of “below-state ” reporting for NAEP as well as representatives from the Council of Chief State School Officers and the Council of Great City Schools. A representative from a district that would qualify for receiving NAEP results served as the discussant. The second panel provided an opportunity for the committee and workshop participants to hear several state assessment directors discuss the impact that state NAEP has had on their state and local education policy, instruction, and assessment. The concerns that have been expressed regarding district-level reporting at previous committee meetings and elsewhere parallel those considered when state NAEP was implemented in 1990. Thus, the committee thought it would be useful to reflect on the lessons learned from state NAEP as they consider the likely impact of district-level reporting. The presenters focused on the ways in which state NAEP has affected their assessment and instructional programs, the types of comparisons made between NAEP results and state and local assessment results, and what happens when results from various assessments portray differing pictures of achievement. The states represented on this panel were Colorado, Connecticut, Nevada, and Washington. Steve Dunbar, a member of the National Research Council’s earlier Committee on the Evaluation of National and State Assessments of Educational Progress, served as the discussant. The third panel brought together district and state assessment directors to discuss their interests in district-level results, the types of information district-level NAEP would provide, the ways in which district-level results might be used, factors that would bear on their decisions to partici-

OCR for page 1
REPORTING DISTRICT-LEVEL NAEP DATA: SUMMARY OF A WORKSHOP pate, and issues regarding who should make participation and score release decisions. This panel had four subpanels: the first three subpanels included representatives from naturally occurring districts; each subpanel paired a district representative with a representative from the respective state assessment office, and each speaker addressed the issues in turn. The final subpanel consisted of representatives from districts that had expressed interest in receiving district-level NAEP data in 1996. Representatives from NAGB, NCES, and the contracting organizations that work on NAEP (the Educational Testing Service and Westat) sat on the final panel. This panel highlighted the technical issues related to sampling and scoring methodologies for district-level reporting and the policy issues related to participation and reporting decisions. Lauress Wise, also a member of the earlier Committee on the Evaluation of National and State Assessments of Educational Progress, served as the discussant. The workshop was structured so as to permit considerable discussion by presenters as well as participants, much of which is woven into this summary. Time was allotted for each speaker, and following each presentation, substantial time was devoted to open discussion. In preparation for the workshop, speakers were given sets of questions to address during their presentations and asked to supply written responses in advance1. Questions posed to the various panelists are included on the agenda, which appears in Appendix A. ORGANIZATION OF THIS REPORT The purpose of this summary is to capture the discussions and major points made during the workshop in order to assist NAEP’s stewards in their decision making about implementing below-state reporting and to provide information for those who would make decisions about whether or not to participate. The summary is organized as follows. Chapter 2 provides background information on NCES’s past experiences with reporting district-level results, along with a discussion of the benefits associated with and the concerns expressed about the implementation of state NAEP, since 1   Due to inclement weather (Hurricane Floyd), a number of participants were unable to attend the meeting, although some did participate via speaker phone. Their written responses were used in this summary.

OCR for page 1
REPORTING DISTRICT-LEVEL NAEP DATA: SUMMARY OF A WORKSHOP it is expected that these parallel issues related to district-level NAEP. Chapter 3 summarizes the information presented by Panel 2 speakers regarding the impact that state NAEP has had on state and local instruction and assessment programs. The next two chapters reflect the common themes that emerged from discussions at the workshop. There was considerable overlap in the nature of the comments made across the four workshop panels. Thus, instead of summarizing each panel’s discussions separately, we have organized these two chapters around the common issues raised during the workshop. Much of the discussion focused on issues related to comparing results from different districts. Chapter 4 is therefore devoted to the subject of interdistrict comparisons. Chapter 5 highlights participants’ comments regarding factors that bear on their interest in district-level data. Issues to consider and resolve are summarized in the Chapter 6. Appendix A contains the workshop agenda and list of participants; Appendix B contains general background information on NAEP.