Summary
The National Assessment Governing Board has proposed a redesign of the National Assessment of Educational Progress (NAEP). A central premise of the document is that NAEP must be simplified; over time the assessment has been asked (National Assessment Governing Board, 1996:i; reprinted in appendix):
. . . to do more and more beyond its central purposes. Additions have been made without changing its basic design, making the National Assessment overly complex and costly.
The Governing Board seeks to initiate a process for streamlining NAEP's design in core subject areas, increasing the usefulness and timeliness of reports, and maintaining the accuracy, reliability, and validity of NAEP data.
This report comments on the May 1996 draft redesign plan, as requested by the U.S. Department of Education. It is part of the congressionally mandated evaluation of NAEP by the National Research Council (NRC). The NRC's Committee on the Evaluation of National and State Assessments of Educational Progress is charged with reviewing NAEP generally and evaluating the developmental state assessments, student performance levels, and the extent to which results are reasonable, valid, and informative to the public.
The committee concludes that the motivation for the redesign of NAEP is sound. During the past 25 years, policy concerns about educational opportunity, human resource needs, and school effectiveness have driven the program in varied and, in some cases, conflicting directions. Without changing NAEP's basic design, structural elements have been added and features have been changed in response to the diverse interests of the growing constituency for assessment in America's schools. We applaud the Governing Board for initiating this important redesign process and are pleased to see that the Commissioner of Education Statistics is hastening to address the issues raised in the redesign proposal.
The committee's chief conclusion is that the proposed redesign is at once too ambitious and not ambitious enough. It is too ambitious in the sense that it tries to be responsive to the interests of all kinds of users as well as to myriad criticisms of NAEP. The very degree of responsiveness militates against the overall goal of simplification and streamlining. At the same time, and more importantly, the current redesign proposal does not go far enough in addressing the root cause of the problems that motivated the redesign. We attribute much of the dilemma surrounding NAEP today as flowing from the multiplicity of purposes for this assessment program that has accrued over the years. What we do not see in the redesign proposal is a clear sense of priorities from which decisions about audience, information needs, measurement design and administration design would flow.
The committee also finds that the proposed changes in the NAEP design, administration, analysis, and reporting schemes are largely unspecified. Not only does the proposal not suggest choices—or at least recognize the need for choices—among many and varied design principles, it does not describe the process by which principles and plans would be implemented. The proposal does not specify mechanisms for deciding among conflicting program elements and ensuring the coherence and integrity of the assessment.
Finally, several aspects of the proposal involve suppositions about the future program for which there is a limited empirical base. Some of the proposal's basic premises are stated without data on feasibility or dis-
cussion of design tradeoffs. Proposed actions such as annual administrations are predicated on cost savings derived from simplification of NAEP 's measurement and administration designs. Whether or not a simplified assessment can be realized is as yet unknown. Should real cost savings not materialize, several redesign tenets would either have to be dropped or be saved at the expense of presently unspecified components of the program.
These considerations lead us to recommend that the National Assessment Governing Board and the National Center for Education Statistics, which has responsibility for developing the instruments and carrying out the National Assessment of Educational Progress, view the current redesign process as an interim solution. The implicit correlate is that a fundamental rethinking of NAEP is needed. Hence, care needs to be taken that decisions or choices made at this time do not compromise a more ambitious reconceptualization of the National Assessment of Educational Progress.