Executive Summary

In his 1997 State of the Union address, President Clinton announced a federal initiative to develop tests of 4th-grade reading and 8th-grade mathematics that could be administered on a voluntary basis by states and school districts beginning in spring 1999. The principal purpose of the Voluntary National Tests (VNT) was to provide parents and teachers with systematic and reliable information about the verbal and quantitative skills that students have achieved at two key points in their educational careers. The Department of Education anticipated that this information would serve as a catalyst for continued school improvement, by focusing parental and community attention on achievement and by providing an additional tool to hold school systems accountable for their students' performance in relation to nationwide standards.

Initial plans for the VNT were laid out by the Department of Education. Subsequent negotiations between the administration and Congress transferred to the National Assessment Governing Board (NAGB, the governing body for the National Assessment of Educational Progress) exclusive authority to oversee the policies, direction, and guidelines for developing the VNT. Test development was allowed to continue, but pilot and field testing and operational use of the VNT were prohibited pending further consideration by Congress.

When Congress assigned NAGB oversight responsibility for the VNT, it also called on the National Research Council (NRC) to evaluate the technical adequacy of test materials. The NRC issued an interim letter report in July 1998 and a final report later that year.

The NRC is continuing its evaluation with a new committee to consider VNT development during the second year. This interim report describes findings and recommendations developed to date on the quality of VNT items, technical issues associated with screening the items and developing VNT test forms, and plans for inclusion and accommodation of students with disabilities and English language learners. A final report with additional discussion of these issues and also of VNT purpose and use will be issued in September 1999.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Evaluation of the Voluntary National Tests, Year 2: INTERIM REPORT Executive Summary In his 1997 State of the Union address, President Clinton announced a federal initiative to develop tests of 4th-grade reading and 8th-grade mathematics that could be administered on a voluntary basis by states and school districts beginning in spring 1999. The principal purpose of the Voluntary National Tests (VNT) was to provide parents and teachers with systematic and reliable information about the verbal and quantitative skills that students have achieved at two key points in their educational careers. The Department of Education anticipated that this information would serve as a catalyst for continued school improvement, by focusing parental and community attention on achievement and by providing an additional tool to hold school systems accountable for their students' performance in relation to nationwide standards. Initial plans for the VNT were laid out by the Department of Education. Subsequent negotiations between the administration and Congress transferred to the National Assessment Governing Board (NAGB, the governing body for the National Assessment of Educational Progress) exclusive authority to oversee the policies, direction, and guidelines for developing the VNT. Test development was allowed to continue, but pilot and field testing and operational use of the VNT were prohibited pending further consideration by Congress. When Congress assigned NAGB oversight responsibility for the VNT, it also called on the National Research Council (NRC) to evaluate the technical adequacy of test materials. The NRC issued an interim letter report in July 1998 and a final report later that year. The NRC is continuing its evaluation with a new committee to consider VNT development during the second year. This interim report describes findings and recommendations developed to date on the quality of VNT items, technical issues associated with screening the items and developing VNT test forms, and plans for inclusion and accommodation of students with disabilities and English language learners. A final report with additional discussion of these issues and also of VNT purpose and use will be issued in September 1999.

OCR for page 1
Evaluation of the Voluntary National Tests, Year 2: INTERIM REPORT ITEM QUALITY AND READINESS The committee examined the extent to which the VNT items are likely to provide useful information to parents, teachers, students, and others about whether students have mastered the knowledge and skills specified for basic, proficient, or advanced performance in 4th-grade reading or 8th-grade mathematics. The evaluation of the VNT items involved three key questions: Are the completed items judged to be as good as they can be prior to the collection and analysis of pilot test data? Are they likely to provide valid and reliable information for parents and teachers about students' reading or math skills? Does it seem likely that a sufficient number of additional items will be completed to a similar level of quality in time for inclusion in a spring 2000 pilot test? Based on the data from the committee's item quality rating panels and on other information provided to the committee by NAGB and its contractor, the committee reached several conclusions about current VNT item quality and about the item development process: The number of items at each stage is not always known, and relativelyfew items and passages have been through the development and reviewprocess and fully approved for use in the pilot test. The quality of the completed items is as good as a comparison sampleof released NAEP items. Item quality is significantly improved incomparison with the items reviewed in preliminary stages of developmenta year ago. For about half of the completed items, our experts had suggestionsfor minor edits, but the completed items are ready for pilot testing. Efforts by NAGB and its contractor to match VNT items to NAEP achievement-leveldescriptions have been helpful in ensuring a reasonable distributionof item difficulty for the pilot test item pool, but have not yet begun to addressthe need to ensure a match of item content to the descriptions of performance at each achievement level. Our efforts to match item content to the achievement-level descriptionsled to more concern with the achievement-level descriptions thanwith item content. The current descriptions do not provide a clearpicture of performance expectations within each reading stance ormathematics content strand. The descriptions also imply a hierarchyamong skills that does not appear reasonable to the committee. Given these conclusions, the committee offers the following recommendations: Recommendation 2.1: NAGB and its contractor should review item developmentplans and the schedule for item reviews to make sure that there willbe a sufficient number of items in each content and format category.The item tracking system should be expanded to include

OCR for page 1
Evaluation of the Voluntary National Tests, Year 2: INTERIM REPORT new items as soon as possible. Explicit assumptions about survivalrates should be formulated and used in targeting further item development.The review process, particularly NAGB's final review, should be acceleratedas much as feasible to allow time to respond to review recommendations. Recommendation 2.2: Specific issues identified by our item review,such as distractor quality, should be considered in further reviewof the VNT items by NAGB and its contractor. Recommendation 2.3: The contractor should continue to refine theachievement-level matching process to include the alignment of itemcontent to achievement-level descriptions, as well as the alignment of itemdifficulty to the achievement level cutpoints. Recommendation 2.4: The achievement-level descriptions should bereviewed for usefulness in describing specific knowledge and skillexpectations to teachers, parents, and others with responsibilityfor interpreting test scores and promoting student achievement. Thecommittee believes that basic, proficient, and advanced performanceshould be described for each knowledge (e.g., math content strand)or skill (e.g., reading stance) area. Revised descriptions shouldnot imply unintended hierarchies among the knowledge and skill areas. TECHNICAL ISSUES IN TEST DEVELOPMENT The VNT year 2 evaluation regarding technical issues in test development focuses on the extent to which the design for pilot testing will result in items that represent the content and achievement-level specifications, are free of bias, and support test form assembly; plans for the implementation of VNT pilot testing; plans for assembling field test forms likely to yield valid achievement-level results; and the technical adequacy of revised designs for field testing, equating, and linking. NAGB and its contractor have made progress in developing detailed plans for score reporting and for the design and analysis of pilot test data to screen items for inclusion in VNT forms. The committee reserves comment on difficult issues associated with the field test for our final report. Based on information reviewed thus far, we offer the following recommendations: Recommendation 3.1: Given that test items and answer sheets willbe provided to students, parents, and teachers, test forms shouldbe designed to support scoring using a straightforward, total correct-rawscore approach. Recommendation 3.2: Achievement-level reporting should be supplementedwith reporting using a standardized numeric scale, and confidencebands on this scale, rather than probabilistic statements, shouldbe provided regarding the likelihood of classification errors. Recommendation 3.3 An overlapping forms design (hybrid forms design)should be used for the pilot test, as planned. In addition, the contractorshould select the calibration procedure that is best suited to thefinal data collection design and in accord with software limitationsand should plan to conduct item-fit analyses.

OCR for page 1
Evaluation of the Voluntary National Tests, Year 2: INTERIM REPORT Recommendation 3.4: Information regarding expected item survivalrates from pilot to field test should be stated explicitly, and NAGBshould consider pilot testing additional constructed-response items,given the likelihood of greater rates of problems with these typesof items than with multiple-choice items. Recommendation 3.5: Detailed plans should be provided regarding itemanalyses of pilot test data. Specifically, methods for screeningmultiple choice items for appropriate statistics on each distractorshould be used. Additional specifications should be provided forthe ways in which differential item function analyses will be conductedand their results used in making decisions about the items. Recommendation 3.6 A target test information function should be decidedon and set. Although accuracy at all levels is important, accuracyat the lower boundaries of the basic and proficient levels appearsmost critical. Equivalent accuracy at the lower boundary of the advancedlevel may not be feasible with the current mix of items, and it maynot be desirable because the resulting test would be too difficultfor most students. INCLUSION AND ACCOMMODATION There are two key challenges to testing students with disabilities or limited English proficiency. The first challenge is to establish effective procedures for identifying and screening such students so they can appropriately be included in assessment programs. The second challenge is to identify and provide necessary accommodations to students with special needs while maintaining comparable test validity with that for the general population. After release of the NRC report on year 1 of VNT development, NAGB completed several activities related to inclusion and accommodation of students with disabilities and with limited English proficiency. They included preparation of two background papers, public hearings, and the summary of testimony from those hearings. At present, no specific recommendations or actions appear to have been made or taken on the basis of the hearings on inclusion and accommodation. Participation in the cognitive laboratories by students with disabilities and with limited English proficiency has been expanded, and the committee will be interested to learn how this information will be used in item and test development. At this time, the committee offers the following recommendations: Recommendation 4.1: NAGB should accelerate its plans and schedulefor inclusion and accommodation of students with disabilities andlimited English proficiency in order to increase the participationof both those student populations and to increase the comparabilityof VNT performance among student populations. Recommendation 4.2: NAGB should consider one or more additional accommodationsby expanding the pilot test that has been planned to include small-groupadministration and expanded time accommodations. This expansion shouldfocus on additional accommodations for English language learners,perhaps in the form of both a Spanish version and the use of English-Spanishdictionaries for the mathematics test and by examining group differencesas well as item-by-item differences in performance through differentialitem functioning analyses.

OCR for page 1
Evaluation of the Voluntary National Tests, Year 2: INTERIM REPORT Recommendation 4.3: NAGB should describe what additional accommodationsare being considered as part of the VNT planning and detail how thesepotential accommodations can provide useful information to studentswho will be included in the VNT by means of these accommodations. Overall, this interim report identifies a number of activities that need to be specified in more detail or accelerated to ensure the timely development of high-quality tests. The quality of completed items is good, but a good deal of item development work remains. Decisions about reporting and test difficulty and accuracy need to be made before the item pool can be evaluated fully, and a good deal more development and research on inclusion and accommodation issues is needed.

OCR for page 1
Evaluation of the Voluntary National Tests, Year 2: INTERIM REPORT This page in the original is blank.