evidence. The review forms ask reviewers to provide evidence, including examples, explanations, and references, to back up their judgments. Clear and complete citations will be vital to the selection process. Examples of good and poor evidence help reviewers understand what is meant (See "Citing Evidence" in Chapter 5).

Practice by doing a mock review. Practice is essential before beginning the actual review. Without it, the first review will become the training experience. For the mock review, choose a sample of high-quality instructional materials of sufficient complexity to simulate actual review challenges. For example, this could be a four-to eight-week module that contains a teacher's guide, student materials, assessments, and optional supplements.

Provide copies of Forms 1-5 of the review forms and "Directions to Reviewers." Depending on your schedule, you can answer any questions the reviewers have on the entire process before they begin their mock review, or you can proceed to answer questions one step at a time.

Have reviewers discuss their results after each section. To resolve concerns, refer back to the worksheets, such as "Definitions of Criteria" and "Directions to Reviewers," and to reference materials. Some calibration of the review process is desirable, and the discussion will be quite helpful to the reviewers. However, complete agreement on how to apply the criteria is not desirable. Diverse individual reviews, guided by the standards and backed up with evidence, will produce the most comprehensive, useful results.

Plan for reflection and evaluation of the process. Convene the reviewers at the end of the process to discuss its benefits and drawbacks and how to improve the process. Sometimes the review participants do not participate in the selection process that follows. If this is so, provide them with information about the future uses of their reviews.

Constraints and Cautions

If training for this process is minimal (for reasons of time or budget, or both, for example), reviewers will be likely to produce widely varying reviews and recommendations, and the result will be a prolonged and possibly confusing selection process.

If your standards document is sketchy, the reviewers may not have sufficient information to understand exactly what students are supposed to know or do to meet the standards. This will make review results highly variable. In some cases, the official assessments, used in the region provide more information about what students should know or be able to do. But using assessments as a substitute for stan-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement