National Academies Press: OpenBook
« Previous: Part III - Bibliography
Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×

7
Background and Methodology

Karen S. Hollweg

One of the main goals of this project was to produce a bibliography of the available literature and completed research regarding the influence of the National Science Education Standards (NSES). The purpose of this chapter is to document the methodology used in creating the bibliography (published in Chapter 8) and to give an overview of the information contained in it.

There has been an increasing interest internationally in using research evidence to inform the development of policy and practice. Researchers in the United Kingdom at the Cochrane Collaboration, the Centre for Reviews and Dissemination in York, and the Institute of Education at the University of London have become known for their high-quality systematic reviews of research relevant to education. In 2001, the Evidence Informed Policy and Practice Information and Co-ordination Centre (EPPI-Centre) at the University of London wrote a Review Group Manual to guide the work of individuals interested in participating in their production and dissemination of systematic reviews in education. The manual, available online at http://eppi.ioe.ac.uk, and the Framework, presented as Figure 1-1 in Chapter 1 and first published in Investigating the Influence of Standards (National Research Council [NRC], 2002), served as starting points for this project. The Steering Committee and staff drew on both documents as they designed the literature search and the guidelines for the work of the commissioned authors, described below.

THE LITERATURE SEARCH

The project Steering Committee and staff wanted to locate as much of the research that addressed the charge as possible and at the same time avoid bias in the search. To make the search as rigorous, exhaustive, and replicable as possible given the limited resources available, two basic strategies were employed: (1) electronic searches of bibliographic databases, journals, and federally funded agencies and institutions, and (2) searches of Web sites of numerous organizations and agencies actively involved in science education research and analysis of standards-based policies and practices.

First, the NRC library staff performed the research using the following databases: ERIC, NTIS, PAIS, PsycINFO, and Sociological Abstracts. In simultaneous searches of these databases, the librarian created a large base set consisting of documents produced between 1993 and 2001 by the journals, federal agencies, and

Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×

BOX 7-1 Literature Search Targets

Journals

  • American Educational Research Journal

  • Educational Evaluation & Policy Analysis

  • Educational Leadership

  • Educational Researcher

  • Harvard Educational Review

  • International Journal of Science Education

  • Journal for Research in Mathematics Education

  • Journal of Research in Science Teaching

  • Journal of Science Education & Technology

  • Journal of Science Teacher Education

  • Journal of Teacher Education

  • Phi Delta Kappan

  • Research in Science & Technological Education

  • Research in Science Education

  • Review of Educational Research

  • Review of Research in Education

  • School Science & Mathematics

  • Science Educator

  • Science Scope

  • Science Teacher

  • Scientia Paedogogica Experimentalis

  • Teachers College Record

  • Teaching & Teacher Education

Federal Agencies

  • National Education Goals Panel

  • National Science Foundation

  • U.S. Department of Education

    • Office of Adult and Vocational Education

    • Office of Educational Research and Improvement

    • Office of Elementary and Secondary Education

    • Planning and Evaluation Services

Organizations

  • AAAS/Project 2061

  • American Federation of Teachers

  • American Institutes for Research (AIR)

  • Brookings Institution

  • Carnegie Corporation

  • Carnegie Foundation

  • Consortium for Policy Research in Education (CPRE)

  • Cosmos

  • Council of Chief State School Officers (CCSSO)

  • Education Commission of the States

  • Education Development Center (EDC)

  • Eisenhower National Clearinghouse (ENC)

  • ERIC Clearinghouse (ERIC)

  • Fund for Improvement of Education

  • Hoover Institution/Stanford

  • Horizon Research, Inc.

  • Inverness Research Associates

  • National Center for Research on Evaluation, Standards, and Student Testing (CRESST)

  • The National Commission on Teaching and America’s Future

  • National Institute for Science Education

  • RAND Corporation

  • Research Triangle Institute (RTI)

  • SRI

  • TERC

  • Thomas B. Fordham Foundation

  • The Urban Institute

  • Westat

  • Wisconsin Center for Education Research

organizations listed in Box 7-1. This base set was then cross-searched using the keywords in Box 7-2. Both of these lists were generated by a combination of suggestions from members of the Steering Committee, the Committee on Science Education K-12, staff, and others consulted by staff. The goal was to search multiple sources representing the full range of large and small entities involved in standards-based science education work and to include the work of groups having different philosophical and political perspectives.

Keyword searches were supplemented by “free text” searches—that is, looking through titles and abstracts for key words and phrases. To prevent exclusion of potentially useful studies, the searches were intentionally overinclusive (e.g., including full text, rather than just titles and abstracts) and encompassed everything from January 1993 (the year in which the Benchmarks were published) through October 2001.

Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×

BOX 7-2 Key Words and Phrases Used to Identify Studies

AAAS 2061

AAAS Benchmarks

accountability + science

assessment + science

association + science education

Benchmarks + science

benchmarks for science literacy

business + science education

classroom assessment + science

college entrance + science

college placement + science

curriculum + science

district assessment + science

district curriculum + science

district standards + science

Education standards + science

industry + science education

instructional materials + science

instructional materials development + science

local systemic change

local systemic change initiative

local systemic initiative

LSC

materials selection + science

National Educational Goals Panel

National Science Education Standards

national standards + science

NBPTS + science

policy + science

policy makers + science education

policymakers + science education

politicians + science education

preservice + science

Professional association + science education

professional development + science

professional organization + science education

public + science education

RSI

rural systemic initiative

Science education

SSI

standards + education + science

standards-based + science

standards-based reform + science

state assessment + science

state curriculum + science

state standards + science

state systemic initiative

statewide systemic initiative

student learning + science

teacher certification + science

teacher development + science

teacher preparation + science

teachers + science

teaching + science

teaching credential + science

teaching practice + science

text + science

textbook + science

textbooks + science

urban systemic initiative

USI

Knowing that research regarding the NSES and Benchmarks was ongoing, the project staff also attempted to collect “gray” or “fugitive” literature that had not yet been published in journals or other hard copy formats. The primary strategy for this search was accessing and scanning items posted on the Web sites of the agencies and organizations listed in Box 7-1. In addition, science education researchers and officials responsible for managing government-funded research and evaluation programs were contacted and asked to suggest additional sources of material for consideration.

When duplicates were deleted, these searches resulted in several hundred items concerning the NSES and the Benchmarks for Science Literacy.1

1  

Subsequently in this chapter, reference to the National Science Education Standards is meant to imply both the NSES and the Benchmarks. The two are not distinguished because of their overlap.

Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×

FIGURE 7-1 Bibliography worksheet.

IDENTIFYING ITEMS FOR INCLUSION

The next step was to identify the items from this large collection that would provide evidence to address the research question: What does the research tell us about the influence of the NSES on various facets of the educational system, on opportunities for all students to learn, and on student learning? Explicit criteria for inclusion were defined and applied to each study to verify that the study actually addressed the research question. Only studies that met inclusion criteria were to be included in the bibliography and provided to the commissioned authors.

To reduce bias in this process, a Bibliography Worksheet was created that defined explicit criteria for inclusion (Figure 7-1). Full documents were obtained for all items that included reference to the NSES or Benchmarks and one or more other key words used in the search, and a copy of the inclusion criteria chart was

Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×

attached to each. Either the project director, program officer, or a project intern (graduate student) scanned each document, noted the study design (i.e., criteria 2) and component of the system addressed (criteria 3) by highlighting the pertinent section(s) of the Worksheet, and categorized each document/study as I (meeting the criteria—documents that met at least one of the criteria in 2 and in 3), II (questionable, unclear), or III (not meeting criteria—for documents that did not meet at least one of the criteria in 2 and in 3). The project director reviewed all IIs and assigned them to the I or III categories, erring on the side of overinclusion to prevent exclusion of potentially useful studies. Many of the items categorized as III were fact sheets and classroom activities keyed to the NSES as opposed to studies that assess the NSES as a means of inducing change or that focused on outcomes of standards-based interventions (see criteria 2). The resulting 245 items included implementation or outcome studies that focused on one or more of the elements in the Framework shown on the Bibliography Worksheet.

Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×

THE ANNOTATIONS

Each of the commissioned authors was sent copies of the papers that the staff had categorized as relevant to that author’s topic. In addition, staff assigned to each author the responsibility for annotating a number of papers. In general, the paper was assigned to the author for whom the study was most relevant, but studies addressing multiple components of the education system were distributed to equalize the load among authors.

The commissioned authors agreed to evaluate the bibliographic entries relevant to their topics and to write each annotation to include the following:

  1. A statement regarding the nature of the work, whether the paper describes conceptual or experimental research, and the type(s) of data used by the researcher(s)

  2. The overall purpose of the paper, including methods the researchers used to collect and evaluate that data;

  3. The methodological rigor of the research enterprise;

  4. The inferences that were drawn;

  5. A statement regarding the findings in terms of the areas of influence listed in the inclusion criteria.

Authors were encouraged to add other studies with which they were familiar to the original set of 245 items identified so that the project could provide a more comprehensive bibliography to the field.

WHAT’S IN THE BIBLIOGRAPHY

The next chapter contains the entire bibliography for the project, including (1) all 245 items identified through the literature search and processed using the inclusion criteria, (2) additional studies that were either published after the search or added by the authors, and (3) references that are cited in this publication for background, but that do not provide research evidence regarding the influence of the NSES.

Annotations are included for the research studies that authors discuss in their review papers and that ground their arguments and conclusions. In cases where a series of studies are included, the most recent one is annotated and earlier ones are mentioned in that annotation. While all annotations have been written using the same guidelines (as noted above), they vary in style and length due to the fact that many different people wrote them. The authors’ rationale explaining how studies were singled out for inclusion in their reviews is contained within each author’s paper and is not part of the bibliography.

Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×
Page 121
Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×
Page 122
Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×
Page 123
Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×
Page 124
Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×
Page 125
Suggested Citation:"7. Background and Methodology." National Research Council. 2003. What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/10618.
×
Page 126
Next: 8. Annotated Bibliography »
What Is the Influence of the National Science Education Standards?: Reviewing the Evidence, A Workshop Summary Get This Book
×
Buy Paperback | $47.00 Buy Ebook | $37.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 2001, with support from National Science Foundation, the National Research Council began a review of the evidence concerning whether or not the National Science Education Standards have had an impact on the science education enterprise to date, and if so, what that impact has been. This publication represents the second phase of a three-phase effort by the National Research Council to answer that broad and very important question.

Phase I began in 1999 and was completed in 2001, with publication of Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education (National Research Council, 2002). That report provided organizing principles for the design, conduct, and interpretation of research regarding the influence of national standards. The Framework developed in Phase I was used to structure the current review of research that is reported here.

Phase II began in mid-2001, involved a thorough search and review of the research literature on the influence of the NSES, and concludes with this publication, which summarizes the proceedings of a workshop conducted on May 10, 2002, in Washington, DC.

Phase III will provide input, collected in 2002, from science educators, administrators at all levels, and other practitioners and policy makers regarding their views of the NSES, the ways and extent to which the NSES are influencing their work and the systems that support science education, and what next steps are needed.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!