Evaluation of the Voluntary National Tests, Year 2: Final Report (1999)

In its statement of purpose, released on June 23, 1999, the National Assessment Governing Board (1999e:1) responded to the congressional request that it:

Although NAGB has had exclusive authority to oversee the policies, directions, and guidelines for developing the VNT since November 1997, this report was the board's first general statement about the overall purposes of the Voluntary National Tests. Thus, we carefully reviewed the process leading to the report, as well as its content. To accomplish this, we reviewed the following documents:

• The Voluntary National Test: Purpose, Intended Use, Definition of Voluntary and Reporting (National Assessment Governing Board, 1999e)

• Overview: Determining the Purpose, Intended Use, Definition of the Term Voluntary, and Reporting for the Proposed Voluntary National Test (National Assessment Governing Board, 1999a)

• Public Hearings and Written Testimony on the Purpose, Intended Use, Definition of the Term ''Voluntary," and Reporting of the Proposed Voluntary National Test. Synthesis Report. April (National Assessment Governing Board, 1999d)

• High Stakes: Testing for Tracking, Promotion, and Graduation (National Research Council, 1999d)

• The National Assessment Governing Board and Voluntary National Tests: A Status Report (Guerra, 1998).

The National Assessment Governing Board did not take a position for or against the VNT, but simply responded to the conditions of the congressional request. By March 1999 NAGB had prepared a set of background materials that offered two competing scenarios for the purpose and use of the VNT: a "public policy model" and an "individual decision model" (National Assessment Governing Board, 1999e:Appendix); see Appendix A for NAGB's descriptions of the two scenarios. Congress set a deadline of September 30, 1999, for NAGB's report, but the board responded 3 months early in order to give Congress sufficient time to use the report in its deliberations about the future of the VNT.

The public policy and individual decision models had several elements in common:

• overall purpose;

• no federal requirement, sanction, or reward for participation;

• no federal receipt, scoring, tabulation, or reporting of VNT data;

• extensive feedback of individual performance data with results reported in terms of NAEP achievement levels;

• no ties between the VNT and a specific curriculum, teaching method, or approach;

• no intent to diagnose specific learning problems or English language proficiency;

• no use as the sole criterion in high stakes decisions about individual students; and

• no evaluation of instructional practices or programs, or of school effectiveness.

In addition, some issues were left open in NAGB's scenarios for decisions under either policy model: whether (after development by NAGB) the VNT would be under the operational control of a federal agency or would be marketed by one or more commercial test publishers and who would pay for the VNT after its initial development and administration (which is to be funded by the federal government).

Distinctions between the public policy model and the individual decision model included the definition of "voluntary," the intended use of the tests, some details of the reporting plan, and several aspects of test administration. Under the public policy model, public or private school authorities would "volunteer" on behalf of their students. Depending on state and local law and policy, public schools might volunteer at the state or district level. Parents might ''opt out" of testing if permitted by public or private school authorities. In contrast, under the individual decision model, parents (alone) would decide whether a student participated in the VNT.

Under the public policy model, parents, students, and authorized educators would receive reports of VNT performance; under the individual decision model, only parents and students would receive reports. Some norm-referenced information¹ (at the national level) would supplement achievement-level reporting under either model, but under the individual decision model, there would be no comparisons at the class, school, district, or state level. In addition, under the public policy model, individual data could be compiled by state or local participants, subject to technical guidance on reporting from a central source. Such institutional participants "will bear responsibility for using resulting data in valid, appropriate ways" (National Assessment Governing Board, 1999e:Appendix).

Use of VNT results by persons other than students and parents could be far more extensive under the public policy model than in the individual decision model. In the individual model, "follow-up with school/teacher is up to the parent (National Assessment Governing Board, 1999e:Appendix)." In the public policy model, parent and teacher follow-up would be recommended but determined at the state, district, or school level; results could also be used to make decisions about individual students, or they could be compared with student performance on state or local tests. Educational authorities might wish to use test results as "an external anchor to their state tests" (National Assessment Governing Board, 1999e:Appendix) or, possibly, as part of school accountability systems.

Finally, there would be a number of operational differences between the testing programs under the public policy model and the individual decision model, for example, in dissemination strategies, training of relevant school personnel, and systems for reporting and responding to parental queries.

NAGB's descriptions of possible purposes, uses, and implementation strategies for the VNT provided a framework for public comment in a series of open hearings. More than 2,000 invitations were issued for hearings in Atlanta, GA, Washington, DC, Chicago, IL, and San Francisco, CA, from March 29 to April 12, 1999; 51 people and organizational representatives provided oral or written testimony in response to the invitations. About two-thirds of responses came from school systems or educational organizations, while the remainder came from research organizations, testing companies, corporations, foundations, or other national organizations (National Assessment Governing Board, 1999d). After these hearings, NAGB prepared a draft report for review at its May 1999 meeting, and the draft was released for comment by mail, on NAGB's Internet web site and in the Federal Register. A final hearing with state and district testing officers was held on June 12, 1999, at the annual Large Scale Assessment Conference of the Council of Chief State School Officers, and a revised report was approved at a special meeting of the board on June 23, 1999. Although the weak public response to NAGB's alternative scenarios was disappointing-and possibly indicative of a decline in public interest in the VNT-the committee finds that the report reflects an effort by NAGB to bring a wide range of views into its deliberations in a limited time frame.

NAGB's report responds to each of the congressional charges: purpose, intended use, definition of voluntary, and reporting issues. In addition, it comments on other issues of implementation and on the linkage of the VNT to the NAEP. The board suggests to Congress and the President that the following statement of purpose be considered (National Assessment Governing Board, 1999e:5):

In recommending this statement of purpose, NAGB pointed to the importance of grades 4, 8, and 12 in schooling and to the NAEP's focus on achievement at those grade levels. Proficiency in reading by grade 4 and in mathematics by grade 8 is generally regarded as fundamental to academic success. Since the NAEP provides scores only for population groups-nationally or for states-the VNT would be the first national test to provide individual scores consistent with the NAEP frameworks and achievement levels. However, NAGB noted its sensitivity to fears of federal encroachment on state and local

control of schooling, including the emergence of a national curriculum. The report thus argues for "checks and balances in the governance and operation of the voluntary national testing program to address these reasonable concerns" (National Assessment Governing Board, 1999e:6).

Concern with federal intrusion is strongly expressed in NAGB's definition of voluntary in the Voluntary National Tests (National Assessment Governing Board, 1999e:7):

Because the federal role in determining who will participate is so limited by this definition, the report stated that the definition of voluntary must accommodate "a wide range of diversity of governance authority," and it suggests an array of levels at which the decision to participate in the VNT could be made, consistent with state and local law and policy (National Assessment Governing Board, 1999e:8):

Clearly, this definition spreads responsibility for test-taking broadly across potentially interested parties-but it explicitly excludes federal authorities. States, school districts, private schools, or parents could all opt for the VNT, while parents could opt out, when permitted by local policy.

The report suggests to Congress and the President that they consider the following statement of intended use of the Voluntary National Tests (National Assessment Governing Board, 1999e:9):

In its rationale, NAGB focuses on the measurement of individual student achievement in relation to NAEP performance standards, consistent with its preceding statement of purpose, and it identifies the audience for VNT results as "Parents, students, and authorized educators." The report suggests several ways in which the VNT could affect future educational progress, for example (National Assessment Governing Board, 1999e:10):

However, NAGB recognizes that the achievement of such outcomes would depend "on local effort,

resources, skill, and persistence" (National Assessment Governing Board, 1999e:10) and thus would not automatically follow from the VNT or any other testing program.

The report also noted that VNT results would probably be used in appropriate ways other than its enumerated uses of individual test scores, "at the discretion of the participating state, district, or private school authorities, who would be responsible for following appropriate technical standards and validation procedures" (National Assessment Governing Board, 1999e:10). The report notes that the VNT, like NAEP, does not reflect an approved curriculum, instructional method, or approach; that it is not designed "to diagnose specific learning problems or English language proficiency;" and that-consistent with professional testing standards (citing standards of the American Educational Research Association et al. [in press] and the report of the National Research Council [1999c]-the VNT "should not be used as the sole criterion in making high stakes decisions (e.g., placement or promotion) about individual students'' (National Assessment Governing Board, 1999e: 11).

However, NAGB did not accept the recommendation of the National Research Council (1999c) that the VNT not be used for any high stakes decision about individual students. Rather, the report explains (National Assessment Governing Board, 1999e:11):

NAGB recommends extensive feedback of information about individual student performance and about the test itself in the context of the performance standards (achievement levels of basic, proficient, and advanced) determined by NAGB (National Assessment Governing Board, 1999e:11):

The report recognizes a likely demand for aggregate reporting of test results, "for the nation as a whole or by state, district, school, or classroom," but it states that no compilations should be provided automatically by the VNT program, "since the purpose and use of the testing program are directed at individual student level results" (National Assessment Governing Board, 1999e:11). Thus, participants who want aggregate results "should be permitted to obtain and compile the data at their own cost, but they will bear the full responsibility for using the data in appropriate ways and for ensuring that the uses they make of the data are valid" (National Assessment Governing Board, 1999e:11). NAGB states that it would develop and provide guidelines for "appropriate and valid" compilation and reporting of VNT results and that its guidelines would "explicitly require full and clear disclosure about exclusions and/or absences from testing, so that results and comparisons would be accurately portrayed" (National Assessment Governing Board, 1999e:12).

Individuals who are already familiar with the proposal for Voluntary National Tests will find little that is new in NAGB's report on purpose, intended use, meaning of the term voluntary, and reporting. Rather, it codifies several elements of the proposal as initially offered by the Clinton administration and modified when NAGB assumed control of test development:

• a focus on individual performance in reading in the 4th grade and mathematics in the 8th grade;

• the effort to link VNT content, standards, and reporting to the National Assessment of Educational Progress;

• extensive feedback of test results to individual students;

• voluntary participation by states or local public and private school authorities; and

• a clearly defined prohibition of federal participation in the VNT program, beyond its support of test development and, possibly, operational costs.

The NAGB report also articulates two significant modifications of the VNT design: permitting individual parents to opt for the VNT program, even if their children's schools do not participate, and use of norm-referenced, as well as criterion-referenced,² score reports to individual students.

The committee finds one significant element lacking, however, both in the original proposal for the VNT and in the NAGB report: evidence to support the belief that the VNT, if implemented, would in fact have favorable effects. The proposal for the Voluntary National Tests, as articulated by the Clinton administration and revised by the National Assessment Governing Board, assumes positive educational effects of individual achievement testing based on high standards, as embodied in NAEP. President Clinton argued that "good tests will show us who needs help, what changes in teaching to make, and which schools need to improve" (National Research Council, 1999e). Among other potential benefits suggested in early discussion of the VNT was the belief that information provided by the VNT would lead to greater parental involvement-a positive outcome, given that the relationship between parental involvement and student achievement is one of the more consistent findings of educational research. Yet critics have argued that information from the VNT would simply perpetuate beliefs of failing students and their parents about the students' inability to learn and might not lead to any improvement in educational processes at all.

The NAGB report on VNT purpose and use takes an important first step in delineating what the VNT is being designed to do: "to measure individual student achievement in 4th-grade reading and 8th grade mathematics, based on the content and rigorous performance standards of the National Assessment of Educational Progress" (National Assessment Governing Board, 1999e:5). The report includes a vignette illustrating possible outcomes from implementation of the program and then states: "The story emphasizes that, while having widely recognized standards and assessments can provide focus for planning and a common language for students, parents, and teachers, what is most important is what parents, students, and educators actually do with that knowledge." Much more dialogue is needed, however, to articulate the range of ways in which the information provided by the VNT might or might not lead to positive changes.

The revised test standards (American Educational Research Association et al., 1999), just recently approved, place greater emphasis on the need to consider consequences of test use. In this regard, AIR has proposed an evaluation of the VNT on instruction, parental involvement, and student motivation to accompany implementation of the VNT (American Institutes for Research, 1999e). The committee supports the idea of studies of the consequences of the VNT, based in part on evidence from research on the consequences of other similar assessments (see also National Research Council, 1999c). In this regard, the committee believes that a good starting point for consideration of potential effects is from existing research on the impact of state assessments, many of which use standards-based reporting and bear other similarities-in design and purpose-to the proposed VNT.

The committee believes high priority should be given to articulation of the benefits to be derived from the VNT, as well as to a search for evidence of the likelihood that such benefits would be realized. That evidence may be found through a review of similar programs of educational initiatives and in continuing evaluation of the VNT, if and when implemented.

On the whole, the recommendations of the National Assessment Governing Board follow its public policy model, rather than the individual decision model. That is, students may be "volunteered" by state or local school authorities or by private schools. Score reports are to go to "authorized educators," as well as to students and their parents. Institutional participants have the option of aggregating data, subject to guidelines to be developed later. Thus, it may be possible to report and compare scores at the classroom, school district, or state levels, which would not be possible under the individual decision model.

In addition to the intended use, as described above, NAGB does not limit or proscribe other uses of VNT scores by institutional participants (National Assessment Governing Board, 1999e:10):

In particular, NAGB states:

This recommendation is consistent with professional standards for test use and with the NRC report on high-stakes testing (National Research Council, 1999d); however, the Board proposes no mechanism for enforcing its recommendation.

The NAGB report also ignores several other standards of appropriate test use when it adds, "it is conceivable that circumstances could exist in which state and/or school district authorities would find information from the voluntary national test useful, in concert with other valid information, in making

'high stakes' decisions about individual students" (National Assessment Governing Board, 1999e:11). NAGB's report does not suggest what those circumstances might be. The NRC report enumerated an extensive list of the circumstances under which the VNT, as proposed, might be used to make high stakes decisions about individuals, and it found that in each case use of the VNT would be invalid or otherwise unethical (National Research Council 1999c:Ch. 10). For example, there is no provision-such as an alternative form-for students to retake the VNT, and rapid public release of test items and answers would surely preclude retaking the then-current form of the test. Yet the opportunity to retake a high-stakes examination is one of the basic criteria of appropriate test use in decisions about individuals. In addition, the explicit and deliberate absence of alignment of the VNT (or its model, NAEP) with existing curricula would appear to invalidate its use either as a criterion for promotion or as a guide to placement. The committee is left without any information about the circumstances under which NAGB envisions that state or school district authorities could appropriately use VNT information to make high-stakes decisions about individual students. The NAGB report is also silent on what would be done if the VNT were used inappropriately to make high-stakes decisions.

NAGB adopted its public policy model (slightly modified) with respect to participation in the VNT and uses of the resulting information. Yet the focus of the VNT-both in terms of purpose and intended use-is on the achievement of the individual student. The proposed approach is problematic because of its effort to combine individual and public action. It attempts to inform, but does not protect or empower parents or students. Consider, in contrast, the role of the primary institutional participants-state, local, and private school authorities. In NAGB's report, they are insulated from federal intrusions and from tangible responsibility for appropriate use of information from the VNT in decisions about individual students, despite NAGB's expressed concern about some potential misuses of the tests. In effect, NAGB (or its operational successor) would be no more responsible for appropriate use of the VNT than any commercial publisher is for any existing test. Although there are instances in which findings from the National Assessment of Educational Progress (the other major testing program for which NAGB is responsible) have been misused, the potential for serious misuse is much greater in the case of the VNT, with potential significant consequences for individual students, teachers, and schools. Thus, NAGB should rethink its positions with respect to prevention and control of potentially inappropriate uses of the VNT.

For example, one of the recommendations of the 1999 NRC report on appropriate test use (National Research Council, 1999c) was that test producers adopt "truth in labeling," much as food and pharmaceutical producers are now required to do. NAGB has asked its prime contractor, the American Institutes for Research, to develop and test materials for reporting scores and other information about the VNT to students and parents. However, none of the materials for parents or teachers that have been presented to the committee bear on inappropriate uses of the test information or on what parents or teachers can do to prevent misuse of the tests.

Just as NAGB provides school authorities with broad license to use information from the VNT to make decisions about individual students, it also provides a similar license for the use of a variety of

aggregations of test score-at the classroom, school, district, or state level. While NAGB does not plan for the automatic aggregation of VNT score data, it accepts that many participants (e.g., schools, districts, states) may wish to aggregate results. The NAGB report states that it "would develop and provide guidelines and criteria for use by states, districts, and schools for compiling and reporting the data from the voluntary national tests in ways that are appropriate and valid" (National Assessment Governing Board, 1999e:11), and it insists that the users could be responsible for the appropriate uses of aggregate data. However, the committee has not seen a framework or outline for such guidelines. Apart from requiring "full and clear disclosure about exclusions and/or absences from testing" (National Assessment Governing Board, 1999e:12), the report does not indicate concern about potential misuses of aggregate data-such as those pertaining to evaluation of "instructional practices, programs, or school effectiveness''-that were noted in its draft VNT scenarios. Again, the report provides no indication of what NAGB would do in the event of misuse of aggregated VNT information. The committee believes this is a major deficit in the NAGB report:

Most of the NAGB report and the committee's assessment deal with some of the more technical and educational issues surrounding the VNT, yet there are a formidable number of practical issues that must be addressed in the near future. These include issues of funding, namely: Who will pay for the VNT and delivery, and who will administer, score, and report results? Will it be a federally funded program or turned over to the commercial sector for development, administration, and reporting? Will there be central delivery management and oversight or a free-market model with many firms licensed to carry out this function (see Guerra, 1998)? Will there be funding stages (from the federal government to "volunteers"), or will there be continued federal funding? Who will be responsible for test security? Who will provide technical support and answer questions about test use and test results? In developing proposals for delivery systems and funding, NAGB can determine the extent of national interest in participation under each of the scenarios it considers.

These and related issues need to be considered now because they have a direct bearing on potential participation and the timing of decisions about participation. There is a clear need for NAGB to begin to consider information gathering regarding potential test administration (i.e., operational) issues. For users to decide whether to participate, they will need to know about funding and operational delivery systems (e.g., procedures for administration, scoring, reporting, etc.). Congress will almost certainly want to know about the scope of VNT use.