Evaluation of the Voluntary National Tests: Phase 1 (1999)

In his 1997 State of the Union address, President Clinton announced a federal initiative to develop tests of 4th-grade reading and 8th-grade mathematics that would provide reliable information about student performance at two key points in their educational careers. According to the U.S. Department of Education, the Voluntary National Tests (VNT) would create a catalyst for continued school improvement by focusing parental and community-wide attention on achievement and would become new tools to hold school systems accountable for their students' performance. The National Assessment Governing Board (NAGB) has responsibility for development of the VNT.

The tests would be voluntary because the federal government would prepare but not require them, and no individual, school, or group data would be reported to the federal government. Every effort would be made to include and accommodate students with disabilities and English-language learners in the testing program. The tests would provide sufficiently reliable information so all students—and their parents and teachers—would know where they stood in relation to high national standards and, in mathematics, also in relation to levels of achievement in other countries.

In order to provide maximum preparation and feedback to students, parents, and teachers, sample tests would be circulated in advance, and copies of the original tests would be returned with the original and correct answers marked. A major effort would be made to communicate test results clearly to students, parents, and teachers, and all test items would be released on the Internet just after each test was administered.

Congress recognized that a testing program of the scale and magnitude of the VNT initiative raises many important technical questions and requires quality control throughout development and implementation. In P.L. 105-78, Congress called on the National Research Council (NRC) to evaluate a series of technical issues pertaining to the validity of test items, the validity of proposed links between the VNT and the National Assessment of Educational Progress (NAEP), plans for the accommodation and inclusion of students with disabilities and English-language learners, plans for reporting test information to parents and the public, and potential uses of the tests. (Congress also requested two additional studies, one on the linkage and equivalency of tests and the other on appropriate test use.)

In accepting this charge, the NRC appointed us co-principal investigators. Working closely with NRC staff and consultants, under the auspices and oversight of the NRC's Board on Testing and Assessment, we have solicited a wide range of expert advice, conducted a number of data-gathering and analytical activities, and held three public workshops.

This report covers phase 1 of the evaluation (November 1997-July 1998) and focuses on three principal issues: test specifications and frameworks; preliminary evidence of the quality of test items; and plans for the pilot and field test studies, for inclusion and accommodation, and for reporting VNT results.

The VNT test specifications are appropriately based on NAEP frameworks and specifications, but they are incomplete. The close correspondence with NAEP builds on NAEP efforts to achieve a consensus on important reading and mathematical knowledge and skills and maximizes the prospects for linking VNT scores to NAEP achievement levels. However, the current test specifications lack information on test difficulty and accuracy targets and they are not yet sufficiently tied to the achievement-level descriptions that will be used in reporting. Some potential users also question the decision to test only in English.

We recommend that test difficulty and accuracy targets and additional information on the NAEP achievement-level descriptions be added to the test specifications. We also recommend that NAGB work to build a greater consensus for the test specifications to maximize participation by all school districts and states.

Because of significant time pressures, several item review and revision steps have been conducted simultaneously, and opportunities have been missed to incorporate feedback from individual steps. Yet relative to professional and scientific standards of test construction, the development of VNT items to date has been satisfactory, especially in light of the significant time pressures. NAGB and its consortium of contractors and subcontractors have made good progress toward the goal of developing a VNT item pool of adequate size and of known, high quality. While we cannot determine in advance whether that goal will be met, we find that the procedures and plans for item development and evaluation are sound. The hurried pace also prevented full development of an item tracking system.

The VNT test design presented some novel problems for which there are no ready solutions. For example, the compressed schedule did not permit the fundamental development work that would be required to ensure both inclusion and comparable validity of test scores for students who are English-language learners and students with disabilities.

In addition, the design of the tests and of their results has continued to evolve during the development process. For example, while the goal of reporting in terms of achievement levels has remained constant, there has as yet been no decision about the possibility of reporting scaled scores or ranges of scores as well. Indeed, some features of test design, such as test length, appear to have been determined administratively, ignoring possible implications for the validity or reliability of the test.

We recommend that NAGB allow more time for future test development cycles so that the different review activities can be performed sequentially rather than in parallel. We also recommend that NAGB and its contractor develop a more automated item-tracking system

so as to have timely information on survival rates and the need for additional items. Item development should be tracked by content and format categories and by link to achievement-level descriptions so that shortages of any particular type of item can be quickly identified.

The pilot and field test plans appear generally sound with respect to the number of items and forms to be included and the composition and size of the samples. More detail on plans for data analysis is needed and some aspects of the design, such as the use of hybrid forms, appear unnecessarily complex.

We recommend that NAGB and its contractor develop more specific plans for the analysis and use of both the pilot and field test data. These plans should include decision rules for item screening and accuracy targets for item parameter estimates, test equating, and linking. We also recommend that greater justification be supplied for some aspects of these plans, such as the use of hybrid forms, or that specific complexities be eliminated. NAGB should also prepare back-up plans in case item survival rates following the pilot test are significantly lower than anticipated.

Plans for including and accommodating students with disabilities and English-language learners are sketchy and do not yet break new ground with respect to maximizing the degree of inclusion and the validity of scores for all students.

We recommend that NAGB accelerate its plans and schedule for inclusion and accommodation of students with disabilities and limited English proficiency in order to increase both the participation of those student populations and to increase the comparability of VNT performance among student populations.

There are a number of potential issues in the reporting of test results to parents, students, and teachers that should be resolved as soon as possible, including: the adequacy of VNT items for reporting in relation to the NAEP achievement-level descriptions; mechanisms for communicating uncertainty in the results; and ways to accurately aggregate scores across student populations. We also question whether and how additional information might be provided to parents, students, and teachers for students found to be in the “below basic” category.

We recommend that NAGB accelerate its specification of procedures for reporting because reporting goals should drive most other aspects of test development. Specific consideration should be given to whether and how specific test items will be linked and used to illustrate the achievement-level descriptions. Attention should also be given to how measurement error and other sources of variation will be communicated to users, how scores will be aggregated, and whether information beyond achievement-level categories can be provided, particularly for students below the basic level of achievement.