Section II
Recommended Processes And Tools



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science Section II Recommended Processes And Tools

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science This page in the original is blank.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science 4 Guide to Selecting Instructional Materials INTRODUCTION TO THE GUIDE The instructional materials used in K-12 science classes provide the basis for what students can learn and what teachers should teach. The process used to select those materials is critical to providing students and teachers with a solid foundation for achievement and successful teaching. This guide is designed to help school personnel review and select science instructional materials. Specifically, this guide will be most useful to anyone appointed to facilitate the process — for example, a district or state science program administrator, a science department head, or a school principal. The facilitator will work with both the review and selection teams and eventually will seek approval from a school board, advisory board, or principal. In some cases, individual schools or teachers may work alone to review and select materials; in other cases, communities and states may review and recommend materials for adoption lists. Since the applicable policies and logistical arrangements are highly variable, this guide cannot address all situations. Rather, the guide is based on principles and processes that individuals, committees, and communities may adapt for their unique circumstances and needs. The review process is designed to be more open-ended than most and to rely heavily on the professional judgments of the reviewers rather than scales, formulas, and averages. As such, it is similar to the type of review used by scientists to evaluate each other's scientific work. This may be perceived to be a drawback because this type of review will be new to most reviewers of instructional materials. In addition, in order to produce a reliable review, reviewers will need to be versed in the standards, to have experience teaching the grade levels for which materials are being considered, and to have the knowledge and understanding of science as described in national standards. In the

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science end, the experience of carrying out the kind of rigorous review that is common in the scientific world, requiring so much background, will be valuable on many levels. It will provide a significant professional growth experience for many reviewers, help develop a local capacity to select and implement a strong science program successfully, and contribute to developing leadership among local science educators. Assumptions This guide and the process it advocates are based on four key assumptions: The selection of instructional materials can be carried out either for a comprehensive science program or for a small part of such a program. The process in this guide can be used equally well for a variety of selection needs: selecting materials for a multiyear program (for example, K-5, 6-8, or 9-12); meeting a specific goal (such as identifying instructional materials for a new ninth grade physics course); or selecting a single unit of study for part of a year. The review of instructional materials, which precedes selection, will be based on standards; that is, specific student learning goals. Applying standards to the process makes student learning of important concepts and skills a key factor in making selection decisions. It is also assumed that local policies will determine the source of the standards to be used — national, state, or local. A curriculum framework (see box) is in place that is based on standards and describes a scope and sequence for student learning. It also is assumed that the selection process involves decisions about which instructional materials are most likely to help students achieve the learning goals given in the framework. At least two people will review each instructional material, and a group including both experienced teachers and scientists will collaborate in the review process. Experienced teachers contribute their knowledge of how children learn, how to manage a classroom learning environment, and the particular challenges of the local

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science student population. Scientists can contribute their broad knowledge of science content and scientific inquiry and can be particularly helpful in reviewing the importance of the content and its accuracy. "Curriculum framework," as used here, means the design for a science program. Frameworks can be official documents representing a mandate approved at the state, county, or district level or can be working documents, useful for sketching out proposed components of a multigrade science program. Many frameworks are published as a matrix of topics and grades or grade ranges. A review and selection process can identify resources for each cell in the matrix. Review and Selection Process Overview The review and selection process in this guide differs from some other processes in that it has been designed to rely on the individual and collective judgments of the reviewers, not on checklists, scales, or rubrics. The judgments are based on standards, and incorporate evidence about how likely it is that students will learn through use of the materials. The final products include a review team summary report and recommendations to the decision-making body. Provision is made for consideration of the costs of the materials and reviewer opinions about the need for teacher professional development. These processes are designed to be flexible to suit various purposes, timelines, and available resources. The review process generates information about the quality of instruction units — the building blocks of a complete science curriculum. The selection of a collection of materials should not be viewed as the equivalent of constructing a multiyear curriculum program. For more information about constructing a complete science program, see Designs for Science Literacy (AAAS, forthcoming a) and Designing Mathematics or Science Curriculum Programs: A Guide for Using Mathematics and Science Education Standards (NRC, 1999a). The review and selection process presented here is written as a guide for the person responsible for organizing and carrying out the task, the facilitator. The complete process is made up of five steps: A Facilitator Plans the Review. Training Reviewers. Carrying Out the Review of Materials. Selecting Materials. Evaluating the Process and Results. There are consequences for omitting any of the parts, some of which are discussed in sections entitled "Constraints and Cautions." If, over time, the entire process is implemented, and increasing numbers of teachers and community members have an opportunity to participate, the local capacity to select effective instructional materials will be greatly enhanced.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science STEP 1: A FACILITATOR PLANS THE REVIEW As the facilitator, you should begin planning at least a year before final instructional material selections are scheduled to be made. During this planning time, you will be gathering data about the effectiveness of existing science education programs, becoming familiar or reacquainted with state and local policies concerning instructional materials selection, and constructing an action plan and budget. In the process, you will be contacting school personnel and community members for information and opinions, as well as building awareness of the existing program and the possible need for changes. Recommended Process Policy information. Compliance with policy is necessary to gain final administrative approval and access to funds for new instructional materials. For example, you will need to know whether your state produces lists of materials from which you must select materials and when state and local funds will become available. Information about deadlines can be especially important in budget planning and for avoiding unnecessary delays. Find out how flexible the policies and regulations are and the consequences of not conforming to policy. Take advantage of the Internet, conferences, and publications to stay current. If your local plans and needs conflict with state policies or regulations, you have time to build administrative and community support for solutions. Find out about policy waivers and the recent history of how many have been granted. Talk to local administrators about the options available and your concerns in order to gauge their support. Make sure you know the history of local selection practices. Budget planning. Each review situation will have unique policies and resources for completing the review. At a minimum, develop a budget for two days of training prior to the actual review — one to understand the process and define the criteria and one to do a mock review. In order to make a rough estimate of the time that will be required to do the review, use the following guideline taken from field-test experience: three hours per reviewer (use a minimum of two reviewers) to carry out a review using three standards on one piece of instructional material that is designed to support about eight weeks of the school curriculum. These minimal time recommendations assume that: some community scientists are already informed of and involved in

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science science curriculum planning, implementation, and evaluation, and therefore are comfortable working with school personnel; your potential reviewers have a reasonably deep understanding of standards; and professional development for science teachers and ongoing community outreach has developed a broad common understanding of effective science education programs and practices. If the preceding capabilities are not available, you will need more resources for as many capacity-building activities as possible. Be sure that you have provided funds for the staff required for the extensive preparation and facilitation of the review and selection processes. Also plan for the time and associated costs required for community outreach activities. Obtaining and organizing the materials to be reviewed can be very time consuming. Your budget should adequately allow for this task and any shipping or storage fees that may be necessary. Coordination with other science education initiatives. Contact those persons responsible for curriculum and instruction inside and outside your immediate program. Use their advice to compile a broad account of local science education efforts, including a history of recent professional development in science, sources of current funding, and projects and programs in science teaching and learning that are under way or planned. Research new science education initiatives being discussed or to be launched soon in the region or state. Coordination with the plans and proposals of others involved in science education in your area may enable you to share resources for recruiting and training reviewers, developing community support for the science program, and planning for the successful implementation of the new program. Become familiar with the processes used and lessons learned by colleagues in other disciplines who have recently completed instructional materials selection. Make a written summary of these findings. These will be useful later in training reviewers and making presentations to administrators and community groups. Data collection. Compile and analyze evidence on current student achievement in science, teacher opinions on what is working, elements of the science program in need of revision, and community perceptions of the science program. An anonymous survey of the materials that teachers are actually using may be necessary, since the curriculum prescribed by current policies may not be the one that has been implemented in the classroom. A survey of parents and students will not

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science only collect useful data, but also increase interest in the review, selection, and approval processes. In addition to the basic reporting of standardized test scores, a study of the item analyses can provide useful data on student achievement. This information is usually provided along with the overall scores to school administrators. Professional development in how to interpret and apply the test-item analysis information is useful for principals and teachers, who are then better prepared to provide information on student achievement. In regions that disaggregate the test scores in a number of ways — by gender, race, courses, or classrooms — it is possible to further pinpoint needs that should be taken into consideration in selecting instructional materials. Another source of data related to student science achievement is enrollment data in upper-level science courses, in which students enroll by choice or by meeting prerequisites. Improvements in the science courses should show a trend to increased demand and enrollment for advanced courses, as well as an increased participation of currently underrepresented minorities. The information collected before the review will help influence final selection decisions and provide compelling background information in support of your recommendations during the approval process. Identification and involvement of community stakeholders. Support from influential members of the community will be critical when recommendations for the ultimate selections need to be approved and when the new materials are introduced into schools. Selected local scientists and engineers from industry, faculty of local colleges and universities from both the education and science departments, and leaders of science education programs can be made members of an advisory board, along with teachers, students, and parents. Some members of this board may become reviewers and trainers. Participation in the advisory board and in the review and selection process will help educate community members about the curriculum, standards, classroom needs, and available instructional materials. Involve the community in learning about the science program through district, school-level, and community activities such as open house events, community meetings, and newsletters. Educate participants about program goals and the science standards and gather opinions and suggestions. Keep community members informed through periodic updates using all of the news media available in your community.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science Recruit reviewers. Choose highly qualified people whose judgments can be trusted to help increase student achievement in science. Selection criteria should include science content knowledge, demonstrated knowledge of effective teaching practices, and depth of knowledge of science standards. Individuals who have participated in professional development in science will have a common base of experience. Recruitment will be enhanced by including a description of the training to be provided and the professional growth benefits of participation. Science subject matter knowledge is the most fundamental requirement for reviewers. Teachers often will have acquired this knowledge through classes and experience outside the district's professional development program. Therefore, be sure to collect background information on all potential reviewers, including their college majors, previous experience, and summer internships, through an application process. To identify a pool of potential teacher reviewers, obtain information on participants in past professional development for science teachers. This may also be a useful exercise for identifying scientists and university faculty who could serve as reviewers. Community advisory groups and partnership activities may also yield potential reviewers, such as practicing scientists and engineers. By all means, try to identify those who have had experience working with school personnel. Consider requesting information from each potential reviewer on possible conflicts of interest and sources of bias, such as participation in professional development sponsored by publishers, past and present consultant agreements, or experience in publisher field tests. Reviewers need not necessarily be excluded because of these activities: when the team convenes, possible biases and conflicts of interest should be declared by each individual and that information then used to avoid potential problems. Build the capacity of the reviewers. The success of your review and selection process depends on the depth of knowledge of the reviewers — of science subject matter, standards, and effective science teaching. Invest as much as possible in building this knowledge and experience. These professional growth opportunities need not be limited to the reviewers. Wider participation will not only build capacity to review new materials but, more broadly, to accept and implement them. Resources outside your immediate locale can help you build the necessary capacity. Various organizations provide leadership development opportunities,

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science many focused on improving science and mathematics education. For example, Project 2061 offers extensive training in the review of instructional materials, which makes for excellent facilitator preparation even when not possible for all reviewers. National or state organizations may offer professional development on the Standards, Benchmarks, and state standards. Universities may offer seminars on how children learn and the efficacy of various assessment strategies. Partnership programs with local science and technology organizations can provide important information on current scientific knowledge and practices. Pilot-test materials. If there is sufficient lead time (at least six months), plan to have reviewers and others actually use materials in their classrooms. This is particularly valuable when innovative instructional strategies are represented in the materials or when the materials use new technology. Provide training and support for the use of the materials to help ensure that the pilot is a fair test of the quality of the instructional materials. Initially, pilot teachers will be strongly biased by their experiences — good or bad — with the new instructional materials. Sufficient time and frequent opportunities to discuss their experiences with others can moderate the effects of this bias on the review and selection processes. Constraints and Cautions If you are short on time, use the policy information and science program effectiveness data that you have on hand. Depend on existing and experienced advisory bodies and educators who are interested in science. Because short timelines are unlikely to produce much of a change from the status quo, consider seeking approval for a postponement of the deadline, if necessary. If you are short on money, give existing advisory boards preparation tasks or at least seek their help in finding resources. If policy will allow, consider confining the scope of the instructional material review to those areas identified as most in need of improvement. If you cannot recruit reviewers according to the criteria suggested here, plan to spend more time in training the reviewers. Sometimes members of the review and selection team are political appointees, a situation helpful in gaining eventual approval of the instructional materials recommended. Adequate training will be even more important in developing a common understanding of the task and a common background knowledge about science program goals, if the members of your team have an uneven knowledge about science education standards, effective instruction, and local policies. If the community lacks knowledge about your science program, consider

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science who is most likely to affect the selection process and then target your outreach efforts to them. If your community has contradictory ideas about the need for science program improvement, do not skimp on this initial step of preparation. A well-planned and well-executed review process ultimately can be annulled by lack of community support. Schedule frequent progress discussions with other administrators to obtain their advice and commitment as well. If you arrange for publisher representatives to make presentations to reviewers, try to provide a level playing field for large and small publishers. Give all presenters a common format to follow and forbid the offering or accepting of gifts (which is usually prohibited by local policy anyway). Remember that reviewers can be inappropriately influenced by these presentations, even if they involve only an overview of the program and its components. Caution reviewers to look for evidence to support the claims made by the publishers. To save time and money, a common impulse is to narrow the field of materials to be reviewed by some kind of prescreening. Various scenarios were examined during the development of this guide, and each carries some risk of undermining a valid review process. The most promising current resources for prescreening are those reviews of science materials published by organizations that have made a large investment in developing both detailed review criteria and the reviewer expertise. Most notably, Project 2061 is producing in-depth reviews based on its Bench-marks for Science Literacy (AAAS, forthcoming c). These reviews compare materials according to various criteria and are available on the Internet (See Chapter 5 ''Resources for Training"). The National Science Resources Center has produced two books of recommended instructional resources, one for elementary school science, and one for middle school science (NSRC, 1996, 1998). The criteria used are provided as appendixes in both books, with the full text available on the Internet (See Chapter 5 "Resources for Training"). Another source of middle school science review information is the Ohio Systemic Initiative (Ohio Systemic Initiative, 1998). If there is community-wide agreement on the success of some elements of the current science program (e.g., high student achievement and teacher satisfaction), it may be possible to keep those elements in place and focus the review on revising only those parts of the curriculum to be changed. A publisher's claims of standards addressed or recommended grade levels should be viewed with suspicion. Only a careful review will reveal the degree to which the content of

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science This page in the original is blank.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science This page in the original is blank.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science This page in the original is blank.

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science

OCR for page 39
Selecting Instructional Materials: A Guide for K-12 Science This page in the original is blank.