2061 (AAAS, forthcoming a, b, c); setting aside materials that are plainly inadequate; or selecting a limited number of materials to be reviewed based on information acquired from other states or school districts. However, any such narrowing of the field to be reviewed should be employed with caution. Considering the magnitude of the instructional materials investment and the societal costs of failure to educate students successfully, adequate resources — including time — to accomplish the selection of the best possible instructional materials must be provided. Developing the capacity of the reviewers and paying attention to local standards for student learning are responsibilities that are too important to be evaded.
The field-test teams' comments underscore the diversity of opinion, experience, goals, and standards that exist in the 50 states and the thousands of school systems. Moreover, comments and reactions to the tool were different depending on where in the K-12 years the instructional materials were designed to be used.
This report should be considered a beginning. The Center for Science, Mathematics, and Engineering Education plans to continue the work begun by this Committee by disseminating this report and encouraging its use. It is expected that wide application will reveal additional desirable modifications to the guide and tool. The Committee envisions that the tool will be regularly revised in response to experience and ongoing learning research.
The Committee recognized an inherent difficulty in trying to determine whether a particular instructional material is "good." The definition of "good" must include an assessment of the match between the instructional material and the applicable standards, learning goals, and pedagogical approaches. The critical question is whether the material will increase the likelihood that students will attain the knowledge defined by the standards and goals. That is, will the material be effective? Here the Committee found itself on uncertain ground, and evaluation teams will have similar experiences. There is no adequate body of research on this topic. There is, of course, a literature that evaluates pedagogical approaches and what children are capable of learning and understanding at different ages (NRC, 1999b). But on the question of the specific attributes of effective materials, little is known.
Conventional analysis of teaching effectiveness is based primarily on student performance on standardized