National Academies Press: OpenBook

Selecting Instructional Materials: A Guide for K-12 Science (1999)

Chapter: 2 A Review of National Efforts to Evaluate Instructional Materials

« Previous: 1 The Relationship of Instructional Materials to Achieving K-12 Science Standards
Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×

2 A Review of National Efforts to Evaluate Instructional Materials

Acknowledging the diversity of ways schools select instructional materials and the variety of national, state, and local standards, the Committee sought to design a practical and flexible evaluation approach that can be useful across the nation.

The Committee began its work by examining related national efforts and familiarizing itself with the nature of the task and relevant issues. The Committee then established a set of key principles to guide the development of an evaluation tool. This chapter of the report describes these aspects of the Committee's work. Chapter 3 ''The Development of a Guide for Evaluating Instructional Materials" describes the adopted principles and the process the Committee used to develop and test an evaluation tool. The tool itself is presented in Chapter 4 as the "Guide to Selecting Instructional Materials."

The Committee studied several national efforts to provide states and localities with guidance in the review and selection of K-12 science instructional materials. These efforts are summarized below. Chapter 5, Contact Information, and the references provide information on how to access these tools.

PROJECT 2061

The American Association for the Advancement of Science, through its Project 2061, has developed a tool for evaluating how well instructional materials are likely to contribute to the attainment of specific learning goals in science, mathematics, and technology (Roseman et al., 1997; Roseman, 1997a,b). Working closely with scientists, mathematicians, educators, and curriculum developers, the project staff formulated and tested a procedure for analyzing curriculum materials that attends to both content alignment with standards and instructional design. Experience with this tool indicates that

Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×

it provides consistent results from one reviewer to another (Kulm and Grier, 1998). The procedure

  1. uses research-based criteria;

  2. requires extensive training (four days are recommended to train evaluators to minimum competency);

  3. demands evidence-based arguments to support all judgments; and

  4. involves two review teams in the examination of each material and subsequent reconciliation of differences.

The procedure can be applied to a variety of K-12 materials, ranging from those that cover a few weeks to several years of classroom programs (Roseman et al., 1997; Roseman, 1997a,b). Although the procedure has been developed for use with the learning goals in Benchmarks (AAAS, 1993) and the Standards (NRC, 1996), it is applicable to state or district curriculum frameworks if the learning goals are clearly articulated.

Project 2061 used a four-step evaluation process: identification of learning goals, content analysis, instructional analysis, and summary report (AAAS, forthcoming c). The process clustered its evaluation questions on content analysis into three groups:

  1. accuracy (examined by scientists and to be published in Science Books and Films);

  2. alignment with standards; and

  3. coherence.

Material found to be aligned with standards is then subjected to the instructional analysis to determine the likelihood of students learning the specific benchmarks and standards that serve as a basis for the analysis. The seven clusters of evaluation questions (each benchmark-specific) on instructional analysis are:

  1. providing a sense of purpose;

  2. taking account of student ideas;

  3. engaging students with phenomena;

  4. developing and using scientific ideas;

  5. promoting student thinking about phenomena, experiences, and knowledge;

  6. assessing progress; and

  7. enhancing the learning environment.

Also see "Instructional Analysis" in Chapter 5.

Project 2061 has applied its analysis procedure to middle school science textbooks and will publish a report titled Middle Grades Science Textbooks: A Benchmarks-based Evaluation in fall of 1999 (AAAS, forthcoming b). A report on the evaluation of middle grades mathematics programs has also been published and is available on the Project 2061 website (See Contact Information).

Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×

THE NATIONAL SCIENCE RESOURCES CENTER

The National Science Resources Center (NSRC), co-sponsored by the National Academy of Sciences and the Smithsonian Institution, collects and disseminates information about exemplary teaching resources; develops and disseminates curriculum materials; and sponsors outreach activities in the form of leadership development and technical assistance to help school districts develop and sustain hands-on science programs.

In 1988, the NSRC published its first compilation of critically reviewed elementary school curriculum materials for teaching science (NSRC, 1988). This was updated in 1996 with the publication of Resources for Teaching Elementary School Science (NSRC, 1996). In the latter volume, the materials include core instructional materials, supplementary materials (such as activity-centered units) and activity books, as well as the Science and Technology for Children instructional materials produced by the NSRC itself. According to the preface, these materials were reviewed for their consonance with principles advocated in the Standards, particularly those that "emphasize student inquiry, teaching for understanding, and the inclusion of science as a core subject in every grade level, starting in kindergarten." Information on obtaining the NSRC evaluation tool used for the review is available in "Contact Information" at the end of the book.

The NSRC has released an additional volume Resources for Teaching Middle School Science (NSRC, 1998)). The preface states that all the curriculum materials listed are "standards-based," that is, a panel of teachers and scientists found them to meet the NSRC's evaluation criteria for middle school science curriculum materials (NSRC, 1998).

THE NATIONAL SCIENCE FOUNDATION'S FRAMEWORK FOR REVIEW

For many years, the National Science Foundation (NSF) has supported the development of instructional materials for K-12 science education. The NSF's goal has been to develop high-quality materials that have potential for national impact. In 1996 the NSF developed and implemented its Framework for Review to help it answer two questions: (1) What are the characteristics of the portfolio of comprehensive instructional materials for middle school science developed with NSF funds? and (2) How sufficiently do these materials provide for a comprehensive program for middle

Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×

school science consistent with national standards? The study and its results were published, and information on obtaining the report and Review Framework (NSF, 1997) is available online.

The NSF designed its review framework as a peer review exercise for use by NSF evaluation panels composed of scientists, science and technology educators, and science teachers. It requires written responses as well as an overall numerical rating on a five-point scale. Because no materials were evaluated by more than one panel during the NSF evaluation, no conclusions can be drawn about the reliability of the instrument or the process.

The framework is designed to review recent NSF-supported middle school curriculum materials that contain a year or more of course materials. Major criteria to be addressed by framework users are:

  • Is the science content correct?

  • How well do the materials provide for conceptual growth in science?

  • How well do the materials align with the Standards?

Notably, the NSF framework addresses only briefly the question of whether the materials under review are likely to lead to student learning and understanding. It does ask whether the materials provide guidance to teachers, suggestions for appropriate instructional strategies, ideas for a variety of assessment activities, suggestions for implementation, and whether they accommodate student diversity.

THE U.S. DEPARTMENT OF EDUCATION

The U.S. Department of Education (DoEd) established an expert panel in 1996 to develop a process for identifying promising and exemplary programs (including instructional materials) in science and mathematics. The panel established criteria and trained teams of reviewers to evaluate instructional materials that publishers voluntarily submitted. The criteria were:

  1. The program's learning goals are challenging, clear, and appropriate for the intended student population.

  2. The program's content is aligned with its learning goals and is accurate and appropriate for the intended student population.

  3. The program's instructional design is appropriate, engaging, and motivating for the intended student population.

  4. The program's assessment system is appropriate and designed to provide accurate information about student learning and to guide teachers' instructional decisions.

Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
  1. The program can be successfully implemented, adopted, or adapted in multiple educational settings.

  2. The program's learning goals reflect the vision promoted in national standards in science education.

  3. The program addresses important individual and societal needs.

  4. The program's assessment system helps teachers select or modify activities to meet learning needs.

The detailed criteria are available on the DoEd's website and can be used by state or local review teams for evaluation guidance (DoEd, 1997c). The DoEd plans to publish lists of programs that meet its criteria at the promising or exemplary levels. The designation of exemplary will require evidence of effectiveness and success.

CENTER FOR SCIENCE, MATHEMATICS, AND ENGINEERING EDUCATION

The National Research Council's Center for Science, Mathematics, and Engineering Education (CMSEE) has published other reports that encourage the thoughtful selection of instructional materials aligned with standards. CMSEE's Committee on Science Education K-12 and the Mathematical Sciences Education Board will jointly publish Designing Mathematics or Science Curriculum Programs: A Guide for Using Mathematics and Science Education Standards (NRC, 1999a). As stated in that report, its purpose "is to assist those who are responsible for making decisions about curriculum with the process of improving the coherence of mathematics and science curriculum programs." The report focuses on ways states and local school districts can develop, or adopt and transform standards into a logical, grade-by-grade curriculum. It recognizes that one aspect of curriculum development must be the selection of appropriate instructional materials, materials that can support the grade-by-grade goals for student learning according to the pedagogical approaches embodied in the curriculum. Moreover, it emphasizes that the process of developing a curriculum program must be flexible while it considers the interplay between the curriculum itself and the instructional materials available to support the curriculum. Such flexibility is required to assure coherence throughout K-12 instruction, while limiting the need for individual states or school districts to undertake the challenging and costly job of developing new instructional materials to match its chosen curriculum program. The report discusses some general guidelines for the work of selecting instructional materials and emphasizes the

Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×

need to assure scientific validity and alignment with content and pedagogical standards. It then refers to instruments available for structuring the tasks, including instruments under development (such as the one presented in this report) or already published (as summarized above).

In 1998 the National Academy of Sciences published Teaching About Evolution and the Nature of Science, which includes a section on the evaluation of instructional materials with respect to the inclusion and alignment of material on evolution.

Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
Page 17
Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
Page 18
Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
Page 19
Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
Page 20
Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
Page 21
Suggested Citation:"2 A Review of National Efforts to Evaluate Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.
×
Page 22
Next: 3 The Development of a Guide for Evaluating Instructional Materials »
Selecting Instructional Materials: A Guide for K-12 Science Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Science Education Standards set broad content goals for teaching grades K-12. For science teaching programs to achieve these goals—indeed, for science teaching to be most effective—teachers and students need textbooks, lab kits, videos, and other materials that are clear, accurate, and help students achieve the goals set by the standards. Selecting Instructional Materials provides a rigorously field-tested procedure to help education decisionmakers evaluate and choose materials for the science classroom. The recommended procedure is unique, adaptable to local needs, and realistic given the time and money limitations typical to school districts.

This volume includes a guide outlining the entire process for school district facilitators, and provides review instruments for each step. It critically reviews the current selection process for science teaching materials—in the 20 states where the state board of education sets forth a recommended list and in the 30 states where materials are selected entirely by local decisionmakers.

Selecting Instructional Materials explores how purchasing decisions are influenced by parent attitudes, political considerations, and the marketing skills of those who produce and sell science teaching materials. It will be indispensable to state and local education decisionmakers, science program administrators and teachers, and science education advocates.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!