2 A Review of National Efforts to Evaluate Instructional Materials

Acknowledging the diversity of ways schools select instructional materials and the variety of national, state, and local standards, the Committee sought to design a practical and flexible evaluation approach that can be useful across the nation.

The Committee began its work by examining related national efforts and familiarizing itself with the nature of the task and relevant issues. The Committee then established a set of key principles to guide the development of an evaluation tool. This chapter of the report describes these aspects of the Committee's work. Chapter 3 ''The Development of a Guide for Evaluating Instructional Materials" describes the adopted principles and the process the Committee used to develop and test an evaluation tool. The tool itself is presented in Chapter 4 as the "Guide to Selecting Instructional Materials."

The Committee studied several national efforts to provide states and localities with guidance in the review and selection of K-12 science instructional materials. These efforts are summarized below. Chapter 5, Contact Information, and the references provide information on how to access these tools.

PROJECT 2061

The American Association for the Advancement of Science, through its Project 2061, has developed a tool for evaluating how well instructional materials are likely to contribute to the attainment of specific learning goals in science, mathematics, and technology (Roseman et al., 1997; Roseman, 1997a,b). Working closely with scientists, mathematicians, educators, and curriculum developers, the project staff formulated and tested a procedure for analyzing curriculum materials that attends to both content alignment with standards and instructional design. Experience with this tool indicates that



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
Selecting Instructional Materials: A Guide for K-12 Science 2 A Review of National Efforts to Evaluate Instructional Materials Acknowledging the diversity of ways schools select instructional materials and the variety of national, state, and local standards, the Committee sought to design a practical and flexible evaluation approach that can be useful across the nation. The Committee began its work by examining related national efforts and familiarizing itself with the nature of the task and relevant issues. The Committee then established a set of key principles to guide the development of an evaluation tool. This chapter of the report describes these aspects of the Committee's work. Chapter 3 ''The Development of a Guide for Evaluating Instructional Materials" describes the adopted principles and the process the Committee used to develop and test an evaluation tool. The tool itself is presented in Chapter 4 as the "Guide to Selecting Instructional Materials." The Committee studied several national efforts to provide states and localities with guidance in the review and selection of K-12 science instructional materials. These efforts are summarized below. Chapter 5, Contact Information, and the references provide information on how to access these tools. PROJECT 2061 The American Association for the Advancement of Science, through its Project 2061, has developed a tool for evaluating how well instructional materials are likely to contribute to the attainment of specific learning goals in science, mathematics, and technology (Roseman et al., 1997; Roseman, 1997a,b). Working closely with scientists, mathematicians, educators, and curriculum developers, the project staff formulated and tested a procedure for analyzing curriculum materials that attends to both content alignment with standards and instructional design. Experience with this tool indicates that

OCR for page 17
Selecting Instructional Materials: A Guide for K-12 Science it provides consistent results from one reviewer to another (Kulm and Grier, 1998). The procedure uses research-based criteria; requires extensive training (four days are recommended to train evaluators to minimum competency); demands evidence-based arguments to support all judgments; and involves two review teams in the examination of each material and subsequent reconciliation of differences. The procedure can be applied to a variety of K-12 materials, ranging from those that cover a few weeks to several years of classroom programs (Roseman et al., 1997; Roseman, 1997a,b). Although the procedure has been developed for use with the learning goals in Benchmarks (AAAS, 1993) and the Standards (NRC, 1996), it is applicable to state or district curriculum frameworks if the learning goals are clearly articulated. Project 2061 used a four-step evaluation process: identification of learning goals, content analysis, instructional analysis, and summary report (AAAS, forthcoming c). The process clustered its evaluation questions on content analysis into three groups: accuracy (examined by scientists and to be published in Science Books and Films); alignment with standards; and coherence. Material found to be aligned with standards is then subjected to the instructional analysis to determine the likelihood of students learning the specific benchmarks and standards that serve as a basis for the analysis. The seven clusters of evaluation questions (each benchmark-specific) on instructional analysis are: providing a sense of purpose; taking account of student ideas; engaging students with phenomena; developing and using scientific ideas; promoting student thinking about phenomena, experiences, and knowledge; assessing progress; and enhancing the learning environment. Also see "Instructional Analysis" in Chapter 5. Project 2061 has applied its analysis procedure to middle school science textbooks and will publish a report titled Middle Grades Science Textbooks: A Benchmarks-based Evaluation in fall of 1999 (AAAS, forthcoming b). A report on the evaluation of middle grades mathematics programs has also been published and is available on the Project 2061 website (See Contact Information).

OCR for page 17
Selecting Instructional Materials: A Guide for K-12 Science THE NATIONAL SCIENCE RESOURCES CENTER The National Science Resources Center (NSRC), co-sponsored by the National Academy of Sciences and the Smithsonian Institution, collects and disseminates information about exemplary teaching resources; develops and disseminates curriculum materials; and sponsors outreach activities in the form of leadership development and technical assistance to help school districts develop and sustain hands-on science programs. In 1988, the NSRC published its first compilation of critically reviewed elementary school curriculum materials for teaching science (NSRC, 1988). This was updated in 1996 with the publication of Resources for Teaching Elementary School Science (NSRC, 1996). In the latter volume, the materials include core instructional materials, supplementary materials (such as activity-centered units) and activity books, as well as the Science and Technology for Children instructional materials produced by the NSRC itself. According to the preface, these materials were reviewed for their consonance with principles advocated in the Standards, particularly those that "emphasize student inquiry, teaching for understanding, and the inclusion of science as a core subject in every grade level, starting in kindergarten." Information on obtaining the NSRC evaluation tool used for the review is available in "Contact Information" at the end of the book. The NSRC has released an additional volume Resources for Teaching Middle School Science (NSRC, 1998)). The preface states that all the curriculum materials listed are "standards-based," that is, a panel of teachers and scientists found them to meet the NSRC's evaluation criteria for middle school science curriculum materials (NSRC, 1998). THE NATIONAL SCIENCE FOUNDATION'S FRAMEWORK FOR REVIEW For many years, the National Science Foundation (NSF) has supported the development of instructional materials for K-12 science education. The NSF's goal has been to develop high-quality materials that have potential for national impact. In 1996 the NSF developed and implemented its Framework for Review to help it answer two questions: (1) What are the characteristics of the portfolio of comprehensive instructional materials for middle school science developed with NSF funds? and (2) How sufficiently do these materials provide for a comprehensive program for middle

OCR for page 17
Selecting Instructional Materials: A Guide for K-12 Science school science consistent with national standards? The study and its results were published, and information on obtaining the report and Review Framework (NSF, 1997) is available online. The NSF designed its review framework as a peer review exercise for use by NSF evaluation panels composed of scientists, science and technology educators, and science teachers. It requires written responses as well as an overall numerical rating on a five-point scale. Because no materials were evaluated by more than one panel during the NSF evaluation, no conclusions can be drawn about the reliability of the instrument or the process. The framework is designed to review recent NSF-supported middle school curriculum materials that contain a year or more of course materials. Major criteria to be addressed by framework users are: Is the science content correct? How well do the materials provide for conceptual growth in science? How well do the materials align with the Standards? Notably, the NSF framework addresses only briefly the question of whether the materials under review are likely to lead to student learning and understanding. It does ask whether the materials provide guidance to teachers, suggestions for appropriate instructional strategies, ideas for a variety of assessment activities, suggestions for implementation, and whether they accommodate student diversity. THE U.S. DEPARTMENT OF EDUCATION The U.S. Department of Education (DoEd) established an expert panel in 1996 to develop a process for identifying promising and exemplary programs (including instructional materials) in science and mathematics. The panel established criteria and trained teams of reviewers to evaluate instructional materials that publishers voluntarily submitted. The criteria were: The program's learning goals are challenging, clear, and appropriate for the intended student population. The program's content is aligned with its learning goals and is accurate and appropriate for the intended student population. The program's instructional design is appropriate, engaging, and motivating for the intended student population. The program's assessment system is appropriate and designed to provide accurate information about student learning and to guide teachers' instructional decisions.

OCR for page 17
Selecting Instructional Materials: A Guide for K-12 Science The program can be successfully implemented, adopted, or adapted in multiple educational settings. The program's learning goals reflect the vision promoted in national standards in science education. The program addresses important individual and societal needs. The program's assessment system helps teachers select or modify activities to meet learning needs. The detailed criteria are available on the DoEd's website and can be used by state or local review teams for evaluation guidance (DoEd, 1997c). The DoEd plans to publish lists of programs that meet its criteria at the promising or exemplary levels. The designation of exemplary will require evidence of effectiveness and success. CENTER FOR SCIENCE, MATHEMATICS, AND ENGINEERING EDUCATION The National Research Council's Center for Science, Mathematics, and Engineering Education (CMSEE) has published other reports that encourage the thoughtful selection of instructional materials aligned with standards. CMSEE's Committee on Science Education K-12 and the Mathematical Sciences Education Board will jointly publish Designing Mathematics or Science Curriculum Programs: A Guide for Using Mathematics and Science Education Standards (NRC, 1999a). As stated in that report, its purpose "is to assist those who are responsible for making decisions about curriculum with the process of improving the coherence of mathematics and science curriculum programs." The report focuses on ways states and local school districts can develop, or adopt and transform standards into a logical, grade-by-grade curriculum. It recognizes that one aspect of curriculum development must be the selection of appropriate instructional materials, materials that can support the grade-by-grade goals for student learning according to the pedagogical approaches embodied in the curriculum. Moreover, it emphasizes that the process of developing a curriculum program must be flexible while it considers the interplay between the curriculum itself and the instructional materials available to support the curriculum. Such flexibility is required to assure coherence throughout K-12 instruction, while limiting the need for individual states or school districts to undertake the challenging and costly job of developing new instructional materials to match its chosen curriculum program. The report discusses some general guidelines for the work of selecting instructional materials and emphasizes the

OCR for page 17
Selecting Instructional Materials: A Guide for K-12 Science need to assure scientific validity and alignment with content and pedagogical standards. It then refers to instruments available for structuring the tasks, including instruments under development (such as the one presented in this report) or already published (as summarized above). In 1998 the National Academy of Sciences published Teaching About Evolution and the Nature of Science, which includes a section on the evaluation of instructional materials with respect to the inclusion and alignment of material on evolution.