materials under consideration, but the time available was insufficient to thoroughly document reviewer work.

At the second test site, a State Systemic Initiative coordinator brought together some 30 reviewers (including science and math educators, scientists, and mathematicians) from across the state for a day and a half to learn how to review and select science and math instructional materials.

Before beginning the review, the facilitator began the review training by discussing examples of review comments that cited evidence either effectively or ineffectively. Subsequently the reviewers were asked to generate their own definitions for the review criteria specified by the tool. This took nearly two hours to reach consensus.

The reviewers then divided into 8 groups and, using 18 standards, conducted a mock review of one middle school science unit that was in use in a number of school districts in the state. Because the unit did not meet the two content standards, several reviewers expressed concern that the standards-based review would undermine the use of the unit that had been chosen by their school district. Expressing satisfaction with the process as a whole, the reviewers said they viewed the process as one they could use to select instructional materials, despite concerns about the time involved.

At this site, a group of nine research scientists and four teachers reviewed a popular advanced placement biology textbook. The outreach director of a university program served as facilitator. No reviewer training was provided. The standards, review documents, and instructional materials were mailed to each participant in advance of the meeting. Each reviewer was instructed to spend no more than five hours reviewing the high school text.

The group discussion revealed some confusion about the task purpose. One reviewer asked, ''Are we reviewing the materials, the instrument, or the standard"? Over half of the submitted review forms did not mention the standard used. Interestingly, all the scientists judged the materials as having completely met the standards, while all the teachers stated that the materials met the standards incompletely.

As a result of the second round of field tests, the guide was modified and amplified as described in the following paragraphs. Experience with the diverse review situations in which the tool was used suggested that the guide