Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 12
Evaluation of the Voluntary National Tests, Year 2: Final Report 2 Purpose and Use In its statement of purpose, released on June 23, 1999, the National Assessment Governing Board (1999e:1) responded to the congressional request that it: Determine and clearly articulate the purpose and intended use of my proposed federally sponsored national test. Such report shall also include-(A) a definition of the meaning of the term 'voluntary' in regards to the administration of my national test; and (B) a description of the achievement levels and reporting methods to be used in grading any national test. Although NAGB has had exclusive authority to oversee the policies, directions, and guidelines for developing the VNT since November 1997, this report was the board's first general statement about the overall purposes of the Voluntary National Tests. Thus, we carefully reviewed the process leading to the report, as well as its content. To accomplish this, we reviewed the following documents: The Voluntary National Test: Purpose, Intended Use, Definition of Voluntary and Reporting (National Assessment Governing Board, 1999e) Overview: Determining the Purpose, Intended Use, Definition of the Term Voluntary, and Reporting for the Proposed Voluntary National Test (National Assessment Governing Board, 1999a) Public Hearings and Written Testimony on the Purpose, Intended Use, Definition of the Term ''Voluntary," and Reporting of the Proposed Voluntary National Test. Synthesis Report. April (National Assessment Governing Board, 1999d) High Stakes: Testing for Tracking, Promotion, and Graduation (National Research Council, 1999d) The National Assessment Governing Board and Voluntary National Tests: A Status Report (Guerra, 1998).
OCR for page 13
Evaluation of the Voluntary National Tests, Year 2: Final Report THE NAGB PROCESS AND REPORT Two Scenarios The National Assessment Governing Board did not take a position for or against the VNT, but simply responded to the conditions of the congressional request. By March 1999 NAGB had prepared a set of background materials that offered two competing scenarios for the purpose and use of the VNT: a "public policy model" and an "individual decision model" (National Assessment Governing Board, 1999e:Appendix); see Appendix A for NAGB's descriptions of the two scenarios. Congress set a deadline of September 30, 1999, for NAGB's report, but the board responded 3 months early in order to give Congress sufficient time to use the report in its deliberations about the future of the VNT. The public policy and individual decision models had several elements in common: overall purpose; no federal requirement, sanction, or reward for participation; no federal receipt, scoring, tabulation, or reporting of VNT data; extensive feedback of individual performance data with results reported in terms of NAEP achievement levels; no ties between the VNT and a specific curriculum, teaching method, or approach; no intent to diagnose specific learning problems or English language proficiency; no use as the sole criterion in high stakes decisions about individual students; and no evaluation of instructional practices or programs, or of school effectiveness. In addition, some issues were left open in NAGB's scenarios for decisions under either policy model: whether (after development by NAGB) the VNT would be under the operational control of a federal agency or would be marketed by one or more commercial test publishers and who would pay for the VNT after its initial development and administration (which is to be funded by the federal government). Distinctions between the public policy model and the individual decision model included the definition of "voluntary," the intended use of the tests, some details of the reporting plan, and several aspects of test administration. Under the public policy model, public or private school authorities would "volunteer" on behalf of their students. Depending on state and local law and policy, public schools might volunteer at the state or district level. Parents might ''opt out" of testing if permitted by public or private school authorities. In contrast, under the individual decision model, parents (alone) would decide whether a student participated in the VNT. Under the public policy model, parents, students, and authorized educators would receive reports of VNT performance; under the individual decision model, only parents and students would receive reports. Some norm-referenced information1 (at the national level) would supplement achievement-level reporting under either model, but under the individual decision model, there would be no comparisons at the class, school, district, or state level. In addition, under the public policy model, individual data could be compiled by state or local participants, subject to technical guidance on reporting from a central source. Such institutional participants "will bear responsibility for using resulting data in valid, appropriate ways" (National Assessment Governing Board, 1999e:Appendix). 1 A norm-referenced test is used to ascertain an individual's status with respect to the performance of other individuals on the test (Popham, 1990:26).
OCR for page 14
Evaluation of the Voluntary National Tests, Year 2: Final Report Use of VNT results by persons other than students and parents could be far more extensive under the public policy model than in the individual decision model. In the individual model, "follow-up with school/teacher is up to the parent (National Assessment Governing Board, 1999e:Appendix)." In the public policy model, parent and teacher follow-up would be recommended but determined at the state, district, or school level; results could also be used to make decisions about individual students, or they could be compared with student performance on state or local tests. Educational authorities might wish to use test results as "an external anchor to their state tests" (National Assessment Governing Board, 1999e:Appendix) or, possibly, as part of school accountability systems. Finally, there would be a number of operational differences between the testing programs under the public policy model and the individual decision model, for example, in dissemination strategies, training of relevant school personnel, and systems for reporting and responding to parental queries. Public Comment Process NAGB's descriptions of possible purposes, uses, and implementation strategies for the VNT provided a framework for public comment in a series of open hearings. More than 2,000 invitations were issued for hearings in Atlanta, GA, Washington, DC, Chicago, IL, and San Francisco, CA, from March 29 to April 12, 1999; 51 people and organizational representatives provided oral or written testimony in response to the invitations. About two-thirds of responses came from school systems or educational organizations, while the remainder came from research organizations, testing companies, corporations, foundations, or other national organizations (National Assessment Governing Board, 1999d). After these hearings, NAGB prepared a draft report for review at its May 1999 meeting, and the draft was released for comment by mail, on NAGB's Internet web site and in the Federal Register. A final hearing with state and district testing officers was held on June 12, 1999, at the annual Large Scale Assessment Conference of the Council of Chief State School Officers, and a revised report was approved at a special meeting of the board on June 23, 1999. Although the weak public response to NAGB's alternative scenarios was disappointing-and possibly indicative of a decline in public interest in the VNT-the committee finds that the report reflects an effort by NAGB to bring a wide range of views into its deliberations in a limited time frame. NAGB Proposals NAGB's report responds to each of the congressional charges: purpose, intended use, definition of voluntary, and reporting issues. In addition, it comments on other issues of implementation and on the linkage of the VNT to the NAEP. The board suggests to Congress and the President that the following statement of purpose be considered (National Assessment Governing Board, 1999e:5): To measure individual student achievement in 4th grade reading and 8th grade mathematics, based on the content and rigorous performance standards of the National Assessment of Educational Progress (NAEP), as set by the National Assessment Governing Board (NAGB). In recommending this statement of purpose, NAGB pointed to the importance of grades 4, 8, and 12 in schooling and to the NAEP's focus on achievement at those grade levels. Proficiency in reading by grade 4 and in mathematics by grade 8 is generally regarded as fundamental to academic success. Since the NAEP provides scores only for population groups-nationally or for states-the VNT would be the first national test to provide individual scores consistent with the NAEP frameworks and achievement levels. However, NAGB noted its sensitivity to fears of federal encroachment on state and local
OCR for page 15
Evaluation of the Voluntary National Tests, Year 2: Final Report control of schooling, including the emergence of a national curriculum. The report thus argues for "checks and balances in the governance and operation of the voluntary national testing program to address these reasonable concerns" (National Assessment Governing Board, 1999e:6). Concern with federal intrusion is strongly expressed in NAGB's definition of voluntary in the Voluntary National Tests (National Assessment Governing Board, 1999e:7): The federal government shall not require participation by any state, school district, public or private school, organization, or individual in voluntary national tests. The federal government also shall not make participation in voluntary national tests a specified condition for receiving federal funds or require participants to report voluntary national test results to the federal government. Because the federal role in determining who will participate is so limited by this definition, the report stated that the definition of voluntary must accommodate "a wide range of diversity of governance authority," and it suggests an array of levels at which the decision to participate in the VNT could be made, consistent with state and local law and policy (National Assessment Governing Board, 1999e:8): Public and private school authorities should be afforded the option to participate in the voluntary national tests. For public schools, state and/or local law and policy should determine whether the initial decision to participate is made at the state level or at the local district level. Where state law or policy provides that the initial decision be made at the state level, and the state decides not to participate, school districts should be afforded the opportunity to decide whether to participate, to the extent permitted by state and local law and policy. For private schools, the decision to participate should be made by the appropriate governing authority. Parents may have their children excused from testing as determined by state and local law and policy in the case of public schools. In the case of private schools, parents may have their children excused from testing as determined by the policy of the appropriate governing authority. Parents whose schools are not participating but want their children to take the voluntary national tests should have access to the tests either through a qualified individual or testing organization before the tests are released to the public or through dissemination procedures at no or minimal cost (e.g., public libraries and the Internet) after the tests are released to the public. Clearly, this definition spreads responsibility for test-taking broadly across potentially interested parties-but it explicitly excludes federal authorities. States, school districts, private schools, or parents could all opt for the VNT, while parents could opt out, when permitted by local policy. The report suggests to Congress and the President that they consider the following statement of intended use of the Voluntary National Tests (National Assessment Governing Board, 1999e:9): To provide information to parents, students, and authorized educators about the achievement of the individual student in relation to the content and the rigorous performance standards for the National Assessment, as set by the National Assessment Governing Board for 4th grade reading and 8th grade mathematics. In its rationale, NAGB focuses on the measurement of individual student achievement in relation to NAEP performance standards, consistent with its preceding statement of purpose, and it identifies the audience for VNT results as "Parents, students, and authorized educators." The report suggests several ways in which the VNT could affect future educational progress, for example (National Assessment Governing Board, 1999e:10): (1) parents could become more involved with the child's education, (2) students could study hard and learn more, (3) teachers could work more to emphasize important skills and knowledge in the subjects tested without narrowing or limiting their curricula, and (4) parents, students, and teachers could have a means for better communication about the child's achievement. However, NAGB recognizes that the achievement of such outcomes would depend "on local effort,
OCR for page 16
Evaluation of the Voluntary National Tests, Year 2: Final Report resources, skill, and persistence" (National Assessment Governing Board, 1999e:10) and thus would not automatically follow from the VNT or any other testing program. The report also noted that VNT results would probably be used in appropriate ways other than its enumerated uses of individual test scores, "at the discretion of the participating state, district, or private school authorities, who would be responsible for following appropriate technical standards and validation procedures" (National Assessment Governing Board, 1999e:10). The report notes that the VNT, like NAEP, does not reflect an approved curriculum, instructional method, or approach; that it is not designed "to diagnose specific learning problems or English language proficiency;" and that-consistent with professional testing standards (citing standards of the American Educational Research Association et al. [in press] and the report of the National Research Council [1999c]-the VNT "should not be used as the sole criterion in making high stakes decisions (e.g., placement or promotion) about individual students'' (National Assessment Governing Board, 1999e: 11). However, NAGB did not accept the recommendation of the National Research Council (1999c) that the VNT not be used for any high stakes decision about individual students. Rather, the report explains (National Assessment Governing Board, 1999e:11): It is conceivable that circumstances could exist in which state and/or school district authorities would find information from the voluntary national test useful, in concert with other valid information, in making "high stakes" decisions about individual students. Therefore, it seems prudent not to foreclose such use absolutely. The Governing Board recommends that participating states, districts, and schools should be afforded the discretion to decide whether to use the voluntary national test for high stakes purposes and the responsibility for ascertaining that such uses are valid. NAGB recommends extensive feedback of information about individual student performance and about the test itself in the context of the performance standards (achievement levels of basic, proficient, and advanced) determined by NAGB (National Assessment Governing Board, 1999e:11): Consistent with the purpose and intended use of the voluntary national tests, the National Assessment Governing Board suggests that results of the voluntary national tests be provided separately for each student. Parents, student, and authorized educators (those with direct responsibility for the education of the student) should receive the test results report for the student. Test results for the student should be reported according to the performance standards for the National Assessment of Educational Progress (NAEP). These are the NAEP achievement levels: Basic, Proficient, and Advanced. All test questions, student answers, and an answer key should be returned with the test results; it will be clear which questions were answered correctly and which were not. The achievement levels should be explained and illustrated in light of the questions on the test. Also, based on the nationally representative sample of students who participated in the national tryout of the test the year before, the percent of students nationally at each achievement level should be provided with the report. The report recognizes a likely demand for aggregate reporting of test results, "for the nation as a whole or by state, district, school, or classroom," but it states that no compilations should be provided automatically by the VNT program, "since the purpose and use of the testing program are directed at individual student level results" (National Assessment Governing Board, 1999e:11). Thus, participants who want aggregate results "should be permitted to obtain and compile the data at their own cost, but they will bear the full responsibility for using the data in appropriate ways and for ensuring that the uses they make of the data are valid" (National Assessment Governing Board, 1999e:11). NAGB states that it would develop and provide guidelines for "appropriate and valid" compilation and reporting of VNT results and that its guidelines would "explicitly require full and clear disclosure about exclusions and/or absences from testing, so that results and comparisons would be accurately portrayed" (National Assessment Governing Board, 1999e:12).
OCR for page 17
Evaluation of the Voluntary National Tests, Year 2: Final Report ASSESSMENT AND RECOMMENDATIONS Individuals who are already familiar with the proposal for Voluntary National Tests will find little that is new in NAGB's report on purpose, intended use, meaning of the term voluntary, and reporting. Rather, it codifies several elements of the proposal as initially offered by the Clinton administration and modified when NAGB assumed control of test development: a focus on individual performance in reading in the 4th grade and mathematics in the 8th grade; the effort to link VNT content, standards, and reporting to the National Assessment of Educational Progress; extensive feedback of test results to individual students; voluntary participation by states or local public and private school authorities; and a clearly defined prohibition of federal participation in the VNT program, beyond its support of test development and, possibly, operational costs. The NAGB report also articulates two significant modifications of the VNT design: permitting individual parents to opt for the VNT program, even if their children's schools do not participate, and use of norm-referenced, as well as criterion-referenced,2 score reports to individual students. The committee finds one significant element lacking, however, both in the original proposal for the VNT and in the NAGB report: evidence to support the belief that the VNT, if implemented, would in fact have favorable effects. The proposal for the Voluntary National Tests, as articulated by the Clinton administration and revised by the National Assessment Governing Board, assumes positive educational effects of individual achievement testing based on high standards, as embodied in NAEP. President Clinton argued that "good tests will show us who needs help, what changes in teaching to make, and which schools need to improve" (National Research Council, 1999e). Among other potential benefits suggested in early discussion of the VNT was the belief that information provided by the VNT would lead to greater parental involvement-a positive outcome, given that the relationship between parental involvement and student achievement is one of the more consistent findings of educational research. Yet critics have argued that information from the VNT would simply perpetuate beliefs of failing students and their parents about the students' inability to learn and might not lead to any improvement in educational processes at all. The NAGB report on VNT purpose and use takes an important first step in delineating what the VNT is being designed to do: "to measure individual student achievement in 4th-grade reading and 8th grade mathematics, based on the content and rigorous performance standards of the National Assessment of Educational Progress" (National Assessment Governing Board, 1999e:5). The report includes a vignette illustrating possible outcomes from implementation of the program and then states: "The story emphasizes that, while having widely recognized standards and assessments can provide focus for planning and a common language for students, parents, and teachers, what is most important is what parents, students, and educators actually do with that knowledge." Much more dialogue is needed, however, to articulate the range of ways in which the information provided by the VNT might or might not lead to positive changes. 2 A criterion-referenced test is used to ascertain an individual's status with respect to a defined assessment domain. Whereas a norm-referenced test references performance to a norm group, a criterion-referenced test references performance to a defined set of criterion behaviors, such as a specific type of reading skill (i.e., the ability to infer the main idea of a reading passage) or a particular mathematics skill (i.e., the ability to solve word problems based on two or more arithmetic operations) (Popham. 1990:27).
OCR for page 18
Evaluation of the Voluntary National Tests, Year 2: Final Report The revised test standards (American Educational Research Association et al., 1999), just recently approved, place greater emphasis on the need to consider consequences of test use. In this regard, AIR has proposed an evaluation of the VNT on instruction, parental involvement, and student motivation to accompany implementation of the VNT (American Institutes for Research, 1999e). The committee supports the idea of studies of the consequences of the VNT, based in part on evidence from research on the consequences of other similar assessments (see also National Research Council, 1999c). In this regard, the committee believes that a good starting point for consideration of potential effects is from existing research on the impact of state assessments, many of which use standards-based reporting and bear other similarities-in design and purpose-to the proposed VNT. The committee believes high priority should be given to articulation of the benefits to be derived from the VNT, as well as to a search for evidence of the likelihood that such benefits would be realized. That evidence may be found through a review of similar programs of educational initiatives and in continuing evaluation of the VNT, if and when implemented. RECOMMENDATION 2.1 High priority should be given to the articulation of the potential educational effects of the VNT and to the development of a program of research and evaluation to determine whether and how the VNT contributes to improved educational outcomes. On the whole, the recommendations of the National Assessment Governing Board follow its public policy model, rather than the individual decision model. That is, students may be "volunteered" by state or local school authorities or by private schools. Score reports are to go to "authorized educators," as well as to students and their parents. Institutional participants have the option of aggregating data, subject to guidelines to be developed later. Thus, it may be possible to report and compare scores at the classroom, school district, or state levels, which would not be possible under the individual decision model. In addition to the intended use, as described above, NAGB does not limit or proscribe other uses of VNT scores by institutional participants (National Assessment Governing Board, 1999e:10): The Governing Board does not assume that uses of data from voluntary national tests beyond the intended we described above are necessarily inappropriate or should be prohibited to states, districts, and private schools. Any such additional use of voluntary national test data would be done at the discretion of the participating state, district, or private school authorities, who would be responsible for following appropriate technical standards and validation procedures. In particular, NAGB states: The voluntary national tests we not intended to be used as the sole criterion in making "high stakes" decisions about individual students. . . . The primary consideration is that, when making decisions on matters such as promotion, retention, or placement, scores from large-scale assessments should only be used in combination with other information about student achievement. This recommendation is consistent with professional standards for test use and with the NRC report on high-stakes testing (National Research Council, 1999d); however, the Board proposes no mechanism for enforcing its recommendation. The NAGB report also ignores several other standards of appropriate test use when it adds, "it is conceivable that circumstances could exist in which state and/or school district authorities would find information from the voluntary national test useful, in concert with other valid information, in making
OCR for page 19
Evaluation of the Voluntary National Tests, Year 2: Final Report 'high stakes' decisions about individual students" (National Assessment Governing Board, 1999e:11). NAGB's report does not suggest what those circumstances might be. The NRC report enumerated an extensive list of the circumstances under which the VNT, as proposed, might be used to make high stakes decisions about individuals, and it found that in each case use of the VNT would be invalid or otherwise unethical (National Research Council 1999c:Ch. 10). For example, there is no provision-such as an alternative form-for students to retake the VNT, and rapid public release of test items and answers would surely preclude retaking the then-current form of the test. Yet the opportunity to retake a high-stakes examination is one of the basic criteria of appropriate test use in decisions about individuals. In addition, the explicit and deliberate absence of alignment of the VNT (or its model, NAEP) with existing curricula would appear to invalidate its use either as a criterion for promotion or as a guide to placement. The committee is left without any information about the circumstances under which NAGB envisions that state or school district authorities could appropriately use VNT information to make high-stakes decisions about individual students. The NAGB report is also silent on what would be done if the VNT were used inappropriately to make high-stakes decisions. NAGB adopted its public policy model (slightly modified) with respect to participation in the VNT and uses of the resulting information. Yet the focus of the VNT-both in terms of purpose and intended use-is on the achievement of the individual student. The proposed approach is problematic because of its effort to combine individual and public action. It attempts to inform, but does not protect or empower parents or students. Consider, in contrast, the role of the primary institutional participants-state, local, and private school authorities. In NAGB's report, they are insulated from federal intrusions and from tangible responsibility for appropriate use of information from the VNT in decisions about individual students, despite NAGB's expressed concern about some potential misuses of the tests. In effect, NAGB (or its operational successor) would be no more responsible for appropriate use of the VNT than any commercial publisher is for any existing test. Although there are instances in which findings from the National Assessment of Educational Progress (the other major testing program for which NAGB is responsible) have been misused, the potential for serious misuse is much greater in the case of the VNT, with potential significant consequences for individual students, teachers, and schools. Thus, NAGB should rethink its positions with respect to prevention and control of potentially inappropriate uses of the VNT. RECOMMENDATION 2.2 The National Assessment Governing Board should develop explicit and detailed guidelines, practices, and enforcement mechanisms for the appropriate use of the Voluntary National Tests relative to high-stakes decisions about individual students or about teachers, classrooms, schools, or other educational units. Those guidelines should illustrate uses of the VNT relative to high-stakes decisions that are inappropriate and explicitly state the potential consequences of such inappropriate uses. For example, one of the recommendations of the 1999 NRC report on appropriate test use (National Research Council, 1999c) was that test producers adopt "truth in labeling," much as food and pharmaceutical producers are now required to do. NAGB has asked its prime contractor, the American Institutes for Research, to develop and test materials for reporting scores and other information about the VNT to students and parents. However, none of the materials for parents or teachers that have been presented to the committee bear on inappropriate uses of the test information or on what parents or teachers can do to prevent misuse of the tests. Just as NAGB provides school authorities with broad license to use information from the VNT to make decisions about individual students, it also provides a similar license for the use of a variety of
OCR for page 20
Evaluation of the Voluntary National Tests, Year 2: Final Report aggregations of test score-at the classroom, school, district, or state level. While NAGB does not plan for the automatic aggregation of VNT score data, it accepts that many participants (e.g., schools, districts, states) may wish to aggregate results. The NAGB report states that it "would develop and provide guidelines and criteria for use by states, districts, and schools for compiling and reporting the data from the voluntary national tests in ways that are appropriate and valid" (National Assessment Governing Board, 1999e:11), and it insists that the users could be responsible for the appropriate uses of aggregate data. However, the committee has not seen a framework or outline for such guidelines. Apart from requiring "full and clear disclosure about exclusions and/or absences from testing" (National Assessment Governing Board, 1999e:12), the report does not indicate concern about potential misuses of aggregate data-such as those pertaining to evaluation of "instructional practices, programs, or school effectiveness''-that were noted in its draft VNT scenarios. Again, the report provides no indication of what NAGB would do in the event of misuse of aggregated VNT information. The committee believes this is a major deficit in the NAGB report: RECOMMENDATION 2.3 The National Assessment Governing Board should develop explicit and detailed guidelines, practices, and enforcement mechanisms for the appropriate compilation and use of aggregate data from administrations of the Voluntary National Tests relative to high-stakes decisions about teachers, classrooms, schools, or other educational units. Most of the NAGB report and the committee's assessment deal with some of the more technical and educational issues surrounding the VNT, yet there are a formidable number of practical issues that must be addressed in the near future. These include issues of funding, namely: Who will pay for the VNT and delivery, and who will administer, score, and report results? Will it be a federally funded program or turned over to the commercial sector for development, administration, and reporting? Will there be central delivery management and oversight or a free-market model with many firms licensed to carry out this function (see Guerra, 1998)? Will there be funding stages (from the federal government to "volunteers"), or will there be continued federal funding? Who will be responsible for test security? Who will provide technical support and answer questions about test use and test results? In developing proposals for delivery systems and funding, NAGB can determine the extent of national interest in participation under each of the scenarios it considers. These and related issues need to be considered now because they have a direct bearing on potential participation and the timing of decisions about participation. There is a clear need for NAGB to begin to consider information gathering regarding potential test administration (i.e., operational) issues. For users to decide whether to participate, they will need to know about funding and operational delivery systems (e.g., procedures for administration, scoring, reporting, etc.). Congress will almost certainly want to know about the scope of VNT use. RECOMMENDATION 2.4 The National Assessment Governing Board should continue to develop plans for how the VNT would operate. Specifically, it should develop proposals for operational delivery systems for the VNT and for funding ongoing development and delivery costs so that potential users can make decisions about their participation, based on the costs as well as the potential educational value of the VNT.
Representative terms from entire chapter: