6
Summing Up: Issues to Consider and Resolve
One of the intended goals of the workshop was to highlight factors that would affect states’ and districts’ interest in having district-level results. While no attempt was made to establish consensus about any of the suggestions that were made, several themes emerged from the interactions among workshop participants. The following discussion points out issues that need to be considered and resolved before implementation decisions can be made.
CLARIFY THE GOALS AND OBJECTIVES
The goals of district-level reporting for NAEP were not apparent to workshop participants. Some spoke of using tests for accountability purposes, pondering whether NAEP could be used in this way or not. They discussed the amount of testing currently done in their schools and stressed that new testing would need to be accompanied by new (and better) information. However, some had trouble identifying what new and better information might come from district-level NAEP data. Their comments might have been different, and perhaps more useful, if they had a clear idea of the purposes and objectives for district-level reporting. It would be helpful to have an explicit statement of the goals and objectives for district-level reporting combined with a logical argument for how the program is expected to achieve the desired outcomes.
DEVELOP SPECIFICATIONS
There were varying understandings among the participants about the nature and characteristics of district-level reporting. Participants talked about the possibility of receiving information that is currently part of national NAEP, commenting that the breadth of content areas and grades tested were attractive features. Some discussed receiving school-level or individual-level scores. Others generated their own assumptions in order to respond to the questions posed to them. One speaker stated his self-formulated assumptions then noted that if any of the assumptions proved to be inaccurate, “my testimony would change accordingly.”
Many speakers highlighted background data as being very useful and were intrigued by the prospect of seeing performance data broken down by background characteristics. Speakers were not aware that given the sample sizes required, such information is not likely to be provided. Their comments suggest the need for clear statements on the specifications for and constraints on district-level reporting.
EVALUATE COSTS AND BENEFITS
What would districts and states receive? When would they receive the information? How much would it cost? What benefits would be realized from the information? Workshop participants responded to questions about their interests in the program without having answers to these questions. Nonetheless, many said that their interest would depend on the answers. They need information on the types of reports to be prepared—a prototype report would be very useful for this—and the associated costs. Would there be varying levels of expense depending on the nature of the report?
Very important to the participants would be the time lag in receiving reports. Would they receive information in time to use it in their decision and policy making? Or would the time delays be such as to render the information useless? Are there options for reports that would require less preparation time?
In addition to monetary concerns, the costs in terms of time and effort on the part of teachers and students must also be considered. School systems already extensively test fourth and eighth graders. If more time is to be taken away from instruction for the purpose of additional testing, the benefits of the testing need to be laid out. Will additional testing amplify the information already provided? Or will the information be redundant
to that provided from current tests? Will the redundancy make it useful for external validation? Such information is important if NAEP’s stewards want to assess actual levels of interest.
EVALUATE PARTICIPATION LEVELS
Many speakers talked of the value of being able to make interdistrict comparisons based on districts with like characteristics. However, this use of the results rides on the assumption that sufficient numbers of districts will participate. Previous experiences with district-level reporting resulted in between 10 and 12 interested districts in 1996 and virtually no interested districts in 1998.
Meaningful comparisons, as defined by demographic, political, and other contextual variables that districts believe are important, depend on there being a wide variety of other districts with district-level reports. Having only a handful of districts that meet the sampling criteria may limit one of the most fundamental functions for district-level reporting, that is, having a carefully selected group of other districts against which to compare results. Thus, if making comparisons is the primary objective for receiving district-level reports, the targeted districts must feel secure in knowing that there are sister districts also completing the necessary procedures for receiving district-level results. The extent of participation will limit the ability to make the desired comparisons.
CONSIDER WAYS TO SERVE SMALL DISTRICTS
According to the sampling criteria, participation would be limited because many districts would not qualify to receive reports. However, if having district-level data turns out to be associated with educational improvements, should small districts be denied access to such an important program? The Third International Mathematics and Science Survey (TIMSS) has permitted a district consortium for the most recent assessment. Would such consortia be allowed for NAEP reporting, and would the reports be meaningful to the participants? If credible results are to be provided for all individual districts, would this necessitate implementing district NAEP as a census rather than a sample? These are issues that NAEP’s stewards need to address as they consider the goals, objectives, specifications, and components for district-level reporting.
CONSIDER THE IMPACT OF RAISING THE STAKES
A concern expressed when state NAEP was first implemented related to the increased stakes that would be associated with reporting data for smaller units. The message from several speakers (particularly district representatives) was that district-level reports would raise the stakes associated with NAEP. An evaluation of the effects of higher stakes, particularly as they relate to the types of inferences that may be made, would be important.