The field trials demonstrated that many evaluation team members are not sufficiently familiar with the applicable standards to carry out the review tasks without training. Moreover, some members of evaluation teams are not inclined to refer consistently to the standards, preferring to make judgments based on their own views of what should be included in the instructional materials. Training must therefore include a description of the applicable standards, the way they were developed, and why it is important to base evaluations on the standards. The goal of this training is to assure that all evaluators accept the applicable standards as the basis for their judgments.
Another lesson learned from the field trials concerns the priorities given to different aspects of the review materials. In the absence of training, some reviewers made no priorities among the several criteria being considered. There was, in some instances, resistance to the idea that the quality of the scientific content and pedagogical approach must take priority over all other criteria (e.g., quality of pictures and diagrams, teacher aids, cost, or applicability to a bilingual school setting). In such cases, the relative quality of the materials became secondary. Apparently, current practice does not always give precedence to these two critical matters.
Participants in the field trials consistently found that the time required to complete the review was too long. This was true even though the Committee was attentive to this issue in the earliest version of the tool, and at each iteration attempted to streamline the process. It was common for the review of an individual material to take between two and four hours, even when both the pertinent grade level and relevant standards were restricted. In an actual evaluation process, for example, six different materials might be under consideration, requiring between 12 and 24 hours of work. To this, the time required for training and for follow-up discussions by the evaluation team must be added. Subsequently, evaluation of another set of materials may be required for a different grade level or a different set of standards. The total time required is a difficult assignment for classroom teachers and working scientists, except perhaps when the task is carried out during vacation time. In that case, compensation should be provided (Tyson-Bernstein, 1988). This is a serious issue because a thorough, thoughtful review with reference to standards is, by its nature, a lengthy process. The Committee considered some strategies to help ameliorate this problem. The most promising strategies included limiting the review materials to materials judged acceptable by the NSRC (NSRC, 1996, 1998) or Project