Committee's response: Redesign the tool so that evaluation and selection can be carried out by either the same group or two different groups.
Discussions with the field-test groups revealed that although there had been considerable work by others to develop evaluation tools for science instructional materials, no one had undertaken the task of guiding states and districts for the purpose of carrying out a standards-based selection process for these materials.
Committee's response: Include in the training guide advice on organizing and carrying out evaluation and selection, designed for the school district facilitators of these processes.
The Committee modified the prototype tool according to the elements listed above and added a guide that included the requisite training for reviewers. The modified tool was used in a second round of field tests. This included discussion meetings and review activities at three new sites, described below. During this round of field tests, the groups could choose the materials to be reviewed and the Committee's facilitator (the study director) experimented with training methods. Therefore, each field test in this round had unique features. This testing provided an opportunity to learn more about how the tool could be used in a variety of situations, allowed an evaluation of the addition of training to the procedure, and informed subsequent revisions to the guide.
The first test site of the second round was based on a one-day meeting of four groups, each consisting of two teachers and two scientists. A district science coordinator convened the groups to consider whether an elementary science unit currently in use in the district was aligned with state science education standards. The teachers in the group had taught the unit and were therefore familiar with the materials. The scientists also knew the materials because they had assisted the teachers one-on-one in understanding the materials and using them in the classroom.
The reviewers spent much of the time discussing the standards that they had been assigned to consider. All four groups emphasized that the standards against which the materials were to be judged were overly broad. Three of the four groups completed a review of the