5
Inclusion and Accommodation

In the November 1997 legislation that established the National Assessment Governing Board's responsibility for the development of the Voluntary National Tests, Congress required NAGB to make four determinations. The third of these is "whether the test development process and test items take into account the needs of disadvantaged, limited English proficient and disabled students" (P.L. 105-78:Sec. 307 (b)(3)). The same legislation called on the National Research Council "to evaluate whether the test items address the needs of disadvantaged, limited English proficient, and disabled students."

There are two key challenges to testing students with disabilities or limited English proficiency. The first is to establish effective procedures for identifying and screening such students, so they can appropriately be included in assessment programs. Federal law and state and local policy increasingly demand participation of these special populations in all education activities, both as a means of establishing the needs and progress of individual students and for purposes of system accountability. The second challenge is to identify and provide necessary accommodations (e.g., large-print type, extended time) to students with special needs while maintaining comparable test validity with that for the general population (see National Research Council, 1997, 1999e). That is, any accommodation should alter only the conditions of assessment without otherwise affecting the measurement of performance.

This issue is growing in importance, as is the number of students with disabilities or with limited English proficiency. Students with disabilities are now 12.3 percent of all students in elementary and secondary schools, and students with limited English proficiency are 5.5 percent of all students. For the latter category, however, it is important to remember that in some local jurisdictions the percentage may be as high as 60 percent. There is a need to be responsive and inclusive so that these increasing numbers of students can be optimally served by the nation's educational system. We concur with the observation in the earlier NRC report (National Research Council, 1999b:46): "The federal government has an important leadership role to play in subsidizing and demonstrating valid efforts to include these populations."



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report 5 Inclusion and Accommodation In the November 1997 legislation that established the National Assessment Governing Board's responsibility for the development of the Voluntary National Tests, Congress required NAGB to make four determinations. The third of these is "whether the test development process and test items take into account the needs of disadvantaged, limited English proficient and disabled students" (P.L. 105-78:Sec. 307 (b)(3)). The same legislation called on the National Research Council "to evaluate whether the test items address the needs of disadvantaged, limited English proficient, and disabled students." There are two key challenges to testing students with disabilities or limited English proficiency. The first is to establish effective procedures for identifying and screening such students, so they can appropriately be included in assessment programs. Federal law and state and local policy increasingly demand participation of these special populations in all education activities, both as a means of establishing the needs and progress of individual students and for purposes of system accountability. The second challenge is to identify and provide necessary accommodations (e.g., large-print type, extended time) to students with special needs while maintaining comparable test validity with that for the general population (see National Research Council, 1997, 1999e). That is, any accommodation should alter only the conditions of assessment without otherwise affecting the measurement of performance. This issue is growing in importance, as is the number of students with disabilities or with limited English proficiency. Students with disabilities are now 12.3 percent of all students in elementary and secondary schools, and students with limited English proficiency are 5.5 percent of all students. For the latter category, however, it is important to remember that in some local jurisdictions the percentage may be as high as 60 percent. There is a need to be responsive and inclusive so that these increasing numbers of students can be optimally served by the nation's educational system. We concur with the observation in the earlier NRC report (National Research Council, 1999b:46): "The federal government has an important leadership role to play in subsidizing and demonstrating valid efforts to include these populations."

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report In this chapter we first review the issue of inclusion and accommodation in the VNT, along with the findings and recommendations of the NRC on the year 1 evaluation of VNT development. We then review and assess the treatment of inclusion and accommodation in VNT development activities from fall 1998 through the August 1999 NAGB meeting. To accomplish these objectives, the committee reviewed the following documents: Voluntary National Test: Inclusions and Accommodations for Test Development. Policy Statement (National Assessment Governing Board, 1998a) Increasing Participation of Special Needs Students in NAEP: Results of the 1996 Research Study (Mazzeo, 1999) Cognitive Lab Report (American Institutes for Research, 1998d) Background Paper Reviewing Laws and Regulations, Current Practice, and Research Relevant to Inclusion and Accommodations for Students with Disabilities (American Institutes for Research, 1998b) Revised Inclusion and Accommodations Workplan (American Institutes for Research, 1998k) Summary of VNT Activities and Plans Related to the Assessment of Students with Disabilities and Students with Limited English Proficiency (American Institutes for Research, 1998m) List of Groups and Agencies with Interests in Issues Related to Inclusion and Accommodations for Students with Disabilities (American Institutes for Research, 1998h) List of Groups and Agencies with Interests in Issues Related to Inclusion and Accommodations for Students with Limited English Proficiency (American Institutes for Research, 1998i) Evaluation of the Voluntary National Tests, Phase I Report (National Research Council, 1999b) High Stakes: Testing for Tracking, Promotion, and Graduation (National Research Council, 1999d) Cognitive Lab Report: Lessons Learned (American Institutes for Research, 1999a) VNT: Issues Concerning Score Reporting for the Voluntary National Tests: Results of Parent and Teacher Focus Groups (American Institutes for Research, 1999p) Background Paper Reviewing Laws and Regulations, Current Practice, and Research Relevant to Inclusion and Accommodations for Students with Disabilities (American Institutes for Research, 1998a) Background Paper Reviewing Laws and Regulations, Current Practice, and Research Relevant to Inclusion and Accommodations for Students with Limited English Proficiency (American Institutes for Research, 1998c) Public Hearings and Written Testimony on Students with Disabilities and the Proposed Voluntary National Test: October-November 1998. Synthesis Report (National Assessment Governing Board, 1999b) Public Hearings and Written Testimony on Students with Limited English Proficiency and the Proposed Voluntary National Test: October-November 1998. Synthesis Report (National Assessment Governing Board, 1999c) Effects of Extended Time and Small-Group-Administration Accommodations Study 2 in Proposed Year 3-5 Research Plan (American Institutes for Research, 1999e) Improving Schooling for Language-Minority Children: A Research Agenda (National Research Council and Institute of Medicine, 1997) Preventing Reading Difficulties in Young Children (National Research Council, 1998b).

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report NAGB AND AIR ACTIVITIES Initial Plans In summer 1998 NAGB approved a policy for the inclusion and accommodation of students with disabilities and limited English proficiency in the pilot test of the VNT (National Assessment Governing Board, 1998a). The policy was based on four general principles: (1) that the current NAEP inclusion criteria would be used with the VNT pilot test, (2) that so far as feasible the testing accommodations used with NAEP would be provided to students who needed them in the VNT pilot, (3) that decisions to accommodate individual students would be made by knowledgeable school staff in consultation with the pilot test examiner, and (4) that records would be kept of any accommodations used by each student. The policy stated the inclusion criteria and listed the allowable accommodations for students with disabilities and for students with limited English proficiency at each grade level. NAGB revised this policy at its August 1999 meeting to include dual-language booklets (with questions in both English and Spanish) for the 8th-grade mathematics test. The policy adopted for the pilot test is based on research conducted in conjunction with the 1996 NAEP in mathematics and science. From its beginning around 1970 through the middle 1990s, NAEP assessments had been carried out without any kind of accommodation. Procedures for "inclusion" usually focused on the exclusion of some students from the assessments, rather than on universal participation. Since the mid-1990s, as the growth of special student populations and the importance of their participation in large-scale assessments have increasingly been recognized, NAEP has experimented with new, more inclusive participation and accommodation policies which are reflected in the NAGB principles for the VNT and the AIR planning documents. In 1996, there were essentially two "experimental" conditions in NAEP in addition to the then operational sample (S1): a sample of schools where the inclusion criteria were changed but no new accommodations were offered (S2) and a sample of schools where the new criteria were used and new accommodations (extra time, small-group administration, bilingual booklets, and large-print or Braille booklets) were offered (S3). The change in the inclusion criterion for students with disabilities had little effect on participation (slightly fewer grade 4 students participated). The change in inclusion criteria for limited-English-proficient students resulted in a significant (20%) drop in the participation of grade 4 students, but did not affect participation at grades 8 or 12. Comparison of the S2 and S3 samples showed that the additional accommodations did significantly increase participation, particularly for Spanish-language students (Mazzeo, presentation to NAGB, August 7, 1999).1 NAGB commissioned background reports from AIR on the inclusion and accommodation of students with disabilities and with limited English proficiency, but as of summer 1998 there had been little developmental work to address the special needs of these populations. The 584 participants in cognitive laboratory sessions included students with disabilities (32 in reading and 19 in math) and limited English proficiency (11 in reading and 12 in math), but their numbers were too small to provide substantial or reliable information about the potential participation of such students in the VNT (American Institutes for Research, 1998d). In a later document, American Institutes for Research (1998m:2) reports that "29 of the reading participants and 18 of the mathematics participants had some sort of special educational need other than gifted and talented. No systematic data were gathered, however, on the numbers of these students who were receiving special educational services guided by an individualized educational plan." 1   A more complete report of the result of this research is scheduled for release by the National Center for Education Statistics in fall 1999.

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report All of the other contractor documents (American Institutes for Research, 1998b, 1998h, 1998i, 1998k) involve proposals, but no specific development activities or plans aimed at the needs of disadvantaged students or those with limited English proficiency had been undertaken. NAGB's draft principles (labeled as a policy statement) reflect the NAEP experiments: depending on the test and population in question, accommodations for a pilot test may include large-print booklets, extended time, small-group administration, one-on-one administration, a scribe or computer to record answers, reading a test aloud by an examiner, other format or equipment modifications, or a bilingual dictionary if it is normally allowed by the school. However, the success of these policies in increasing participation has not yet been established (see above), nor have their effects on test performance and score comparability been validated. Year 2 Plans Since fall 1998, NAGB and AIR have continued information-gathering and planning activities related to inclusion and accommodation in the VNT. Although progress has been made, the committee finds that these activities have not fully implemented the earlier NRC recommendation for accelerated work on inclusion and accommodation. In its November 1998 report, American Institutes for Research (1998m) reviewed steps taken during item development to consider the needs of special students. They included passage review, plain language review, item editing, bias and sensitivity review, cognitive lab participation, and item analysis. Although these steps are appropriate and commendable, they do not focus specifically on the needs of students with disabilities or with limited English proficiency. We have already noted the small number of disabled students and English-language learners in the first year of cognitive laboratories. In fact, AIR's March 1999 report (American Institutes for Research, 1999a) on lessons learned in the first-year cognitive laboratories makes no reference to inclusion or accommodation issues or effects. Earlier, in commenting on plans for item analysis, American Institutes for Research (1998m:2) reported that, "even with the planned oversampling of Hispanic students [in the pilot test], it was not anticipated that sufficient numbers of either students with disabilities or students with limited English proficiency would have been included to permit DIF (differential item functioning) analysis based on these characteristics." AIR also suggested two enhancements of item development activities during year 2 (American Institutes for Research, 1998m:3): (1) "to review items against plain language guidelines that have been developed under the sponsorship of the Goals 2000 program and elsewhere" and (2) to include larger numbers of students with disabilities and limited English proficiency in the second year of cognitive laboratories. The committee has been informed that both of these proposals have been adopted. In particular, an effort is being made to include students with disabilities and with limited English proficiency among the nine students in each item trial in this year's cognitive laboratories. The AIR planning document also reports that, in connection with its exploration of reporting and test use, focus groups with parents and teachers included (American Institutes for Research, 1998m:5): "one composed of special education teachers, one comprising parents of special education students, and one involving teachers of students with limited English proficiency." A report of the findings of these focus groups was completed in April of this year (American Institutes for Research, 1999p). It consisted of a general description of parents' and teachers' comments regarding draft score reporting materials, additional desired score reporting materials, information desired by teachers to help them increase the academic achievement levels of their students, and ways teachers could communicate with parents about increasing student achievement. However, no mention was made of the special needs of

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report students with disabilities and limited English proficient students in reporting proposed VNT score information. This lack of attention to the problems of inclusion and accommodation is somewhat puzzling in the light of AIR's background documents (American Institutes for Research, 1998a, 1998b, 1998c). In the committee's judgment, those three reports are thorough, competent, and cogent, and they would have provided an ample basis for additional development activities to address the needs of students with disabilities and with limited English proficiency. An AIR planning document (American Institutes for Research, 1998m:5) also acknowledged the issue: Following the release of the NRC report on appropriate test use in September 1998, the board undertook a series of public hearings in which it invited citizens concerned with the education and assessment of students with disabilities or limited English proficiency to comment on the findings of the NRC report and the evolving plans for addressing the needs of these students in the VNT. Invitations to the hearings were developed on the basis of lists of relevant groups and agencies that were assembled by AIR (American Institutes for Research, 1998h, 1998i). The hearings were held during October and November 1998 in several large cities: Washington, DC; Atlanta, GA; New York, NY; Chicago, IL; Austin, TX; and Los Angeles, CA. Witnesses were asked to respond to a series of questions about inclusion and accommodation in large-scale assessment, including the proposed VNT. For example, in the case of students with disabilities (National Assessment Governing Board 1999b:1-3), there were eight queries: What should be the criteria for including a student with disabilities in standard administrations of large-scale tests? What, if any, should be the criteria for exempting a student with disabilities from standard administrations of large-scale tests? What examples exist of such criteria that have been empirically determined and/or validated? What accommodations in the administration of the test should be considered for students with disabilities and according to what criteria should those accommodations be provided? What adaptations (i.e., changes or special versions) to the test should be considered and under what circumstances should they be provided? What validation evidence is sufficient to conclude that results from testing with accommodations/adaptations are comparable to results from testing without accommodations/adaptations? Until such validity is established, what cautions about interpretation, if any, should accompany individual test results? The Voluntary National Test is intended to provide individual student results. What are the pros and cons of permitting aggregate results to be reported for students with disabilities within a school, district, or state? In setting performance standards on a test for interpreting test results, what, if any, considerations should be given with regard to implications for students with disabilities? What specific criteria should be considered in reviewing test items as they are being developed to optimize the inclusion of students with disabilities? What other specific considerations should be taken into account in test development? What other issues related to inclusion and accommodations in testing for students with disabilities should be considered by the Governing Board for the Voluntary National Tests in order to ensure fair and equitable test administration for all students?

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report Parallel questions were also posed with reference to students with limited English proficiency (see National Assessment Governing Board, 1999e). In addition, witnesses who addressed the testing of students with limited English proficiency were asked: ''What are the technical and policy pros and cons of testing students with limited English proficiency in the student's native language and under what circumstance, if any, might this be considered as an appropriate adaptation in large-scale testing?" (National Assessment Governing Board, 1999c:3-5). The testimony at these two series of hearings was recorded and summarized in two reports (National Assessment Governing Board, 1999b, 1999c). While the hearings may have been useful in providing a platform for a number of interested groups and agencies-and signaling the interest of NAGB in the views of those groups and agencies-the committee does not find that the hearing summaries add substantial new information or ideas that could affect the design of the VNT. AIR has proposed a study of the effect of two particular accommodations-extended time and small-group administration-on VNT performance, which would be carried out in connection with the pilot test (American Institutes for Research, 1999e). These two accommodations are common, and, thus, the committee considers it important to understand their effects on test performance and validity. This proposal appeared in a proposed year 2 research plan, but that plan was abandoned because of the congressional ban on pilot testing during 1999. The proposal was modified for year 3 of development, in connection with the year 2000 pilot test (American Institutes for Research, 1999e:8-9), which states: It is important to determine the impact that extra time has on student performance and whether only examinees with disabilities or limited English proficiency benefit from having extra time, or the degree to which the scores of other examinees also would improve with such accommodations. In the case of small-group administration, the research proposal focuses on students with individual education plans (IEPs) and is designed to estimate the effects of extended time and small-group administration, but is not designed to compare the performance of students with IEPs to that of other students under the experimental conditions. Each study will be based on a single pilot test form. The first, extended-time study is proposed for a supplementary sample of large schools in order to minimize the cost of obtaining 150 observations in each cell of the design. The experimental design crosses the three student groups by two times of administration: standard time (45 minutes) and twice the standard time (90 minutes). The second study will use a combination of schools in the main pilot sample and a supplementary sample of schools. There are five cells, crossing standard administration, embedded small groups, and pull-out small groups with standard time limits and double time limits (excluding the double-time condition with standard administration). In both studies, it should be possible to assess the speededness of the tests, as well as differences in performance between groups of students and experimental conditions. The analysis of the first study is also proposed to include analyses of differential item functioning among students with disabilities, students with limited English proficiency, and students who are not in these special groups. Since there are only 300 students in each of the three groups, the committee is concerned that these analyses may not have sufficient statistical power, relative to the design of the main pilot sample. However, the committee does agree that it will be valuable, first, to assess the influence of these two accommodations on VNT performance.

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report CONCLUSIONS AND RECOMMENDATIONS Year 1 Report In its year 1 VNT evaluation report, the National Research Council (1999b:46) noted: Because of the compressed schedule in the early phases of VNT development, along with the desire to achieve close correspondence between the VNT and NAEP, the NAGB plans and the AIR background paper on students with disabilities both focus on recent NAEP practices for inclusion and accommodation, rather than taking a broader, more proactive stance. We believe the federal government has an important leadership role to play in subsidizing and demonstrating valid efforts to include these populations. . . . The procedures discussed in the draft documents are intended to increase participation and provide valid assessments for all students, but they essentially involve retrofitting established assessment instruments and procedures to special populations of students; another approach would be to design and develop assessments from the beginning that are accessible to and valid for all students. . . . Unless extensive development work is done with students with disabilities and with limited English proficiency, it would be unreasonable to expect that the VNT will be valid for use with these student population. Both of these populations are heterogeneous, e.g., in primary language, level of proficiency in English, and specific type of disability. Moreover, they differ from the majority of students, not only in ways that affect test-taking directly-e.g., those that can be accommodated through additional time or assistive devices-but also in styles of learning and in levels of motivation or anxiety. Such differences are very likely to reduce the validity and comparability of test performance. The Committee on Appropriate Test Use identified two important ways in which inclusion and accommodation can be improved (National Research Council, 1999d). First, the focus should be on inclusion and accommodation issues throughout item and test development, so a test is designed from the ground up to be accessible and comparable among special populations. For example, oversampling of students with disabilities and with limited English proficiency in the course of pilot testing will provide sufficient numbers of cases in major subgroups of these students to permit key statistical analyses. Second, test developers should explore the use of new technologies-such as computer-based adaptive testing for students who need extra time-that show promise for substantially reducing or eliminating irrelevant performance differentials between students who require accommodation and other students. The report recognized, however, that development work of this kind is just beginning, and there are presently few exemplars of it. The NRC year 1 evaluation report concluded (National Research Council, 1999b:47): The statement of principles and the AIR planning documents provide a limited basis for evaluation of provisions for inclusion and accommodation in the VNT-and no specific basis to address the quality of item development relative to the needs of those students. . . . A major opportunity for improved large-scale assessment is being lost in NAGB's conservative approach to inclusion and accommodation in the VNT. It thus recommended and explained (National Research Council, 1999b:47): NAGB should accelerate its plans and schedule for inclusion and accommodation of students with disabilities and limited English proficiency in order to increase the participation of both those student populations and to increase the comparability of VNT performance among student populations. . . . This recommendation requires prompt action because so much of the development work in the first round of the VNT has already been completed. We have already noted the modest attention to students with special needs in the cognitive laboratory sessions. In the pilot test, NAGB plans to identify students with disabilities and with limited English proficiency and with the types of accommodations that have been provided. However, there are no provisions in the design to ensure that there will be sufficient numbers of these students-such as students requiring specific types of accommodation-to support reliable DIF analyses. We think that it would be feasible to include

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report larger numbers of such students in the pilot and field tests, for example, by increasing sampling fractions of such students within schools. Moreover, there appears to be no plan to translate the 8th-grade mathematics test into Spanish (or any other language), a decision that is likely to affect participation in the VNT by major school districts. There has been some discussion of a Spanish translation after the field test, but this would be too late for the item analyses needed to construct comparable English and Spanish forms. Next Steps After release of the NRC report on year 1 of VNT development, NAGB completed several activities related to inclusion and accommodation of students with disabilities and with limited English proficiency. They included preparation of two background papers, public hearings, and the summary of testimony from those hearings. However, these activities have not yet carried the development process much beyond its state in the summer of 1998, when the report found (National Research Council, 1999b:47) "a limited basis for evaluation of provisions for inclusion and accommodation in the VNT-and no specific basis to address the quality of item development relative to the needs of those students." We applaud AIR's proposal to evaluate the effects of two common accommodations on VNT performance among students with disabilities and with limited English proficiency in the pilot test. Consistent with recommendations of both the NRC year 1 evaluation report and the committee's interim report, plans for the pilot test were changed to include provision of a dual-language translation of the 8th-grade mathematics test. In addition, proposed research on extra time and small-group or one-on-one administration in conjunction with the pilot test were approved at the August 1999 NAGB meeting. With respect to the provision of dual-language booklets, while language simplification methodology has been used in the test development process, little attention has been paid to other language issues regarding the VNT mathematics test (e.g., whether translation of existing questions into Spanish and other languages will produce comparable items or whether there are methods to reduce the reading level of mathematics items). Participation in the cognitive laboratories by students with disabilities and with limited English proficiency has been expanded, and the committee will be interested to learn how this information will be used in item and test development. The committee is concerned about implementation of the principles of inclusion and accommodation that were adopted by the National Assessment Governing Board (1998a) for the pilot test. Given the modest subsequent development activity, the committee can only assume that these principles are likely to be applied in the field test and under operational conditions. However, the committee is concerned that it may be more difficult to provide these accommodations on a large scale than under the operational conditions of NAEP, where they are now standard procedure. RECOMMENDATION 5.1 NAGB should accelerate its plans, research, and schedule for inclusion and accommodation of students with disabilities and limited English proficiency in order to increase the participation of both those student populations in numbers representative of their numbers in the student population. Although the condensed schedule of test development during year 1 provided a substantial rationale for limited attention to issues of inclusion and accommodation, the extension of the development schedule for another year prior to pilot testing provided extra time for more intensive consideration of these issues. That window is more than half closed, and there are modest signs of progress: the effort to include more students in cognitive labs, the intent to apply guidelines for simplified language, and the AIR proposal to study effects of accommodation in the pilot test.

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report We offer additional recommendations to underscore our concern for the urgency and NAGB's need to expand inclusion and accommodation efforts for the VNT. RECOMMENDATION 5.2 NAGB should consider expanding the accommodation research planned in conjunction with the pilot test to include a systematic analysis of the use and effect of dual-language booklets. Additional accommodations for English-language learners, in the forms of both a Spanish-only translation of the mathematics test and the use of English-Spanish and English-other languages dictionaries for the mathematics test, should also be considered for the pilot test. Because of the large number of limited-English-proficient students in some areas and the fact that a large percentage of those students speak Spanish, the committee applauds NAGB's decision to provide a side-by-side Spanish translation of the 8th-grade mathematics test. Providing this dual-language testing accommodation and the related accommodations we propose in Recommendation 5.2 will allow more limited-English-proficient students to demonstrate what they know and can do in mathematics. In addition it is important not to lose sight of related VNT longer term issues and needs. For example, while perhaps not feasible in the pilot test, the concern to build tests "from the ground up" rather than translate them after they have been developed in English is important to consider for the field test. Certainly more direct approaches to second-language item development should not be ruled out for test development purposes. There also needs to be an ongoing research plan beyond the pilot test that continually promises to advance the inclusion objective for students with disabilities and who have limited English proficiency. The issue of the VNT 4th-grade reading test is clearly a sensitive one. On one hand are the existing NAEP guidelines and tradition regarding the English-only nature of this measure. On the other hand is the desire to include as many students as possible and to provide necessary and reasonable accommodations to accomplish inclusion. Thus, it is important as a basic premise that NAGB clearly articulate the constructs to be measured by the VNT 4th-grade reading test so that reasonable accommodations can be considered to fulfill the goal of maximum inclusion. RECOMMENDATION 5.3 NAGB should clarify the reading constructs (e.g., reading proficiency, reading proficiency in English, etc.) being measured by the 4th-grade reading test prior to the field test and then address what accommodations would not invalidate assessment of these constructs. In particular, NAGB should clarify when reading competency could be assessed in a student's primary or native language if it is not English. This issue needs to be addressed in the very near future. It is especially important in light of the grade level of the reading test (4th), the current NAEP rules for inclusion of English-language learners on the basis of years of English-reading school experience (3 years), and the weight of the evidence as shown in two recent reports (National Research Council and Institute of Medicine, 1997; National Research Council, 1998b) that document the greater likelihood of students developing strong English literacy skills if they first learn to read and write in their primary (native) language. There remains a critical need to obtain statistical information on the efficacy of different accommodation for students with disabilities and those who have limited English proficiency. Some studies and analyses will likely require oversampling of students within specific special populations. This should begin with the pilot test version as planned.

OCR for page 58
Evaluation of the Voluntary National Tests, Year 2: Final Report RECOMMENDATION 5.4 The National Assessment Governing Board should assess the effects of various accommodations for limited-English-proficient students and students with disabilities at both the item and total test score levels. To do so will require oversampling in the pilot and field tests. Finally, it is important for NAGB to provide a clear and concise list of accommodations for administrative use with the VNT, and they should be conveyed in user-friendly language. Given that VNT and NAEP scores may not be comparable, it might not be necessary to strictly emulate the standard NAEP accommodations. Additionally, the role of individualized educational plans should be defined within the context of these suggested VNT accommodations. It is particularly important that the procedures intended for operational administration of the VNT be in place for the field test, as data from the field test will provide both normative information for use in reporting operational results and the basis for linking performance on the VNT test forms to the NAEP achievement levels. RECOMMENDATION 5.5 The National Assessment Governing Board should provide a clear, concise, and detailed list of accommodations for the VNT for students with disabilities or limited English proficiency for use on the VNT field test. We think these issues should be addressed in a systematic manner-one that will gather valid, scientific evidence that can be used to improve the inclusiveness and validity of testing students with learning disabilities and students with limited English proficiency. Our specific recommendations suggest a step-by-step approach to learning more about testing these groups. They should provide new evidence that will improve testing guidelines or suggest additional lines of research to improve practice.