After its study of current selection practices, an investigation of other efforts to develop evaluation tools, and some practical experience in carrying out evaluations, the Committee designed its process for developing and testing an evaluation tool. It formulated a shared set of principles on which the tool would be based, including a goal of fulfilling needs not met by other organizations' efforts. The Committee then constructed a prototype tool and subjected it to an iterative process that cycled experiences from field tests and focus groups back to the Committee to inform the modifications made in subsequent drafts.
The Committee established the following general principles as the basis for its design of a prototype evaluation tool.
1. The evaluation tool should fulfill needs not met by other instruments. The Committee identified unmet needs from its analysis of the review tools available for instructional materials. We found, for example, that the National Science Foundation's (NSF) Framework for Review is designed for materials that cover programs of a year or more of classroom work, and that it addresses only briefly the question of whether the materials under review are likely to lead to student learning and understanding (NSF, 1997). The latter question was therefore selected for emphasis in our prototype tool. The National Science Resources Center's (NSRC) Evaluation Criteria for Science Curriculum Materials (NSRC, 1998) does not ask reviewers to evaluate materials against specific Standards or Benchmarks, which the Committee deems necessary. The Project 2061 review tools require highly trained evaluators and weeks of effort (Roseman et al., 1997), and they are not feasible for many local school districts with limited time, funds, and expertise. Moreover, none of these tools articulate a process that encompasses both evaluation and selection processes.
2. The evaluation tool should assume that a set of standards and a curriculum program or framework will inform the work of evaluators in appraising the effectiveness of instructional materials. Evaluation of science instructional materials is a formidable task. A set of standards and a curriculum program or framework documents the school district's expectations for science education and serves as an important reference for the evaluation. Moreover, the existence of such policies implies an established