3
Building the Knowledge Base

Even if the quality of discrete education research studies is outstanding, if the field is not able to forge connections among them, it will amass a multitude of studies that cannot support inferences about the generalizability of findings nor sustain the long-term theory building that drives scientific progress. Forging connections among studies will enable the field to be more than the sum of its parts. Lacking the infrastructure or the professional norms to engage in efforts to connect and integrate theories and data across investigations, the scientific study of educational phenomena will be fragmented (as some currently are; see Lagemann, 2000). The progression of scientific knowledge in education (and other scientific fields) is neither linear nor predictable, but it can be facilitated by explicit efforts to promote the accumulation of research-based knowledge.

FRAMING CONCEPTS

Two of our five workshops had considerable bearing on the committee’s deliberations on this topic. The first of these workshops was designed to consider a range of strategies that might foster the growth of a cumulative knowledge base in education, including the development of common measures of key constructs, data sharing and replication, and ways of taking stock of what has been learned. During this first workshop, scholars from a range of fields also explored how the nature of scientific knowledge itself influences knowledge building. An understanding of the core nature of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 36
Advancing Scientific Research in Education 3 Building the Knowledge Base Even if the quality of discrete education research studies is outstanding, if the field is not able to forge connections among them, it will amass a multitude of studies that cannot support inferences about the generalizability of findings nor sustain the long-term theory building that drives scientific progress. Forging connections among studies will enable the field to be more than the sum of its parts. Lacking the infrastructure or the professional norms to engage in efforts to connect and integrate theories and data across investigations, the scientific study of educational phenomena will be fragmented (as some currently are; see Lagemann, 2000). The progression of scientific knowledge in education (and other scientific fields) is neither linear nor predictable, but it can be facilitated by explicit efforts to promote the accumulation of research-based knowledge. FRAMING CONCEPTS Two of our five workshops had considerable bearing on the committee’s deliberations on this topic. The first of these workshops was designed to consider a range of strategies that might foster the growth of a cumulative knowledge base in education, including the development of common measures of key constructs, data sharing and replication, and ways of taking stock of what has been learned. During this first workshop, scholars from a range of fields also explored how the nature of scientific knowledge itself influences knowledge building. An understanding of the core nature of

OCR for page 36
Advancing Scientific Research in Education education research and how it is similar and different from other fields and disciplines is an important foundation for the recommendations in this chapter. An analysis of some of conceptual ideas discussed at this workshop—such as the relationship of education research to educational practice, the context dependence of findings, and its public character—is presented in Appendix B. Extending this wide-ranging day-and-a-half of dialogue, a second workshop examined in greater depth the role of education research journals in both promoting high-quality and a more integrated knowledge base. A passage from Scientific Research in Education helps to clarify how we view the idea of knowledge accumulation by linking the key concepts of theory and generalizability: “It is the long-term goal of much of science to generate theories that can offer stable explanations for phenomena that generalize beyond the particular…. Science generates cumulative knowledge by building on, refining, and occasionally replacing, theoretical understanding” (National Research Council, 2002, p. 3). Theoretical constructs provide an organizing conception to which research inquiries relate, creating a common point of reference that can facilitate efforts to make sense of the wide diversity of studies and findings in education research. They give meaning to data, drive the selection of questions and methods, and provide the foundation for large-scale data collection efforts. For example, in the National Assessment of Educational Progress, designing the noncognitive data collection (that is, background data on test-takers, schools, teachers, and instruction) around a coherent set of strong theories would result in a more streamlined data set and promote more coherent lines of investigations focused on well-articulated theoretical models (Grissmer, 2003). Theory also drives replications, a powerful tool for establishing generalizaiblity. Replicating an investigation with comparable subjects and conditions to see if similar results are achieved is essential for being able to generalize to more people and settings than are represented in a single study (Shadish, Cook, and Campbell, 2002; Cronbach et al., 1980; Cronbach, 1982) and to clarify the boundaries of prevailing theories. Replication involves applying the same conditions to multiple cases, as well as replicating the design and including cases that are sufficiently different to justify the generalization of results and theories. In many of the natural and physical sciences, there are professional norms that encourage data sharing—a practice that enables verifications and replications and provides opportunities for investigating new questions,

OCR for page 36
Advancing Scientific Research in Education forging interdisciplinary links, and developing and validating measures. In addition, there are multiple experiments focused on the same issue from different perspectives. New technology has allowed changes in how research is conducted in these fields. For example, in modern astronomy, natural observation has been transformed through the collection and processing of images of phenomena—images and information that are available to multiple investigators simultaneously (Knorr-Cetina, 1999). Consortiums of scientists work on data sets housed in centralized databases. Similarly, in physics, there are projects in which researchers, linked through various technologies, work on problems at the same time in locations throughout the world, searching for new phenomena. Working with data on solar systems is different from working with data on human beings. Research involving humans brings with it fundamental moral and legal responsibilities to protect their rights. These protections shape, and sometimes constrain, data collection, data sharing, and data use.1 For this and other reasons, education researchers rarely replicate their work or that of their peers or reanalyze the findings of others using secondary data. But it is important to recognize that research is not a covert activity, and individual investigators have a professional obligation to contribute to the advancement of their field. For example, when secondary analyses of large-scale databases and data collections have been conducted, they have proven fruitful; see Box 3-1 for an example that shows the value of secondary analysis of existing data. And maximizing data sharing while ensuring the confidentiality of research participants—topics we take up in some detail in the next section—is yet another way in which the field can work collaboratively to advance scientific understanding and progress. MECHANISMS FOR BUILDING THE KNOWLEDGE BASE We see three points of leverage for encouraging knowledge accumulation in education research: professional associations, scholarly journals, and infrastructure supports like data banks. Efforts like data sharing should be supported by professional norms that are developed through sanctions and rewards and reinforced informally by the community of researchers 1   Many of the other barriers to data sharing, however, are common among all sciences (National Research Council, 1985).

OCR for page 36
Advancing Scientific Research in Education (Schneider, 2003). The recommendations in this chapter target these leverage points and call for formal efforts to encourage knowledge accumulation in education research. They serve to focus attention on the development of theories and explicit attempts to reanalyze, replicate, and test the boundaries of those theories with empirical inquiries. Although we focus on institutions in the recommendations, it is critical to acknowledge that all members of the education research communities need to be involved in contributing to and using these infrastructure supports. The tools themselves will not promote the broader goal of scientific progress; their active and effective use by the field will. A recent National Research Council committee captures this idea in discussing publication and data sharing in particular (2003a, p. 4): Community standards for sharing publication-related data and materials should flow from the general principle that the publication of scientific information is intended to move science forward. More specifically, the act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings. An author’s obligation is not only to release data and materials to enable others to verify or replicate published findings … but also to provide them in a form on which other scientists can build with further research. All members of the scientific community—whether working in academia, government, or a commercial enterprise—have equal responsibility for upholding community standards as participants in the publication system, and all should be equally able to derive benefits from it. It is in this spirit that the committee makes the following recommendations. Recommendation 5: Professional associations involved in education research should develop explicit ethical standards for data sharing. Data are the foundation of scientific inquiry, and sharing them among peers is one direct way to facilitate the transparency, accountability, and scholarly communication so vital to scientific advancement (National Research Council, 2002). Sharing data among investigators working on common areas of inquiry enables the production of knowledge that can only be derived from merging, comparing, combining, reanalyzing, or integrating data. Indeed, the potential benefits of data sharing are many, because the practice enables the direct interplay between data and theory at every stage of development in a line of inquiry. In education research, data sharing can

OCR for page 36
Advancing Scientific Research in Education BOX 3-1 The Value of Data Sharing In describing his experience gaining access to and using the infamous Coleman data, workshop speaker Ron Ehrenberg provided a vivid example of the importance of data sharing in education research, as well as the need to prepare adequate documentation of data to enable further analyses. In the early 1990s, Ehrenberg and his colleague Dominic Brewer became interested in the effects on student achievement of school districts’ efforts to aggressively increase their hiring of underrepresented minority faculty. The underlying assumption of these efforts was that minority teachers would be more effective teachers of minority students because they could serve as role models, had higher expectations for these students, and would provide more positive feedback to them. In order to accomplish their hiring objectives, many school districts were providing financial incentives for their older, experienced teachers to retire, even in the face of a declining pool of minorities seeking to enter careers in education and evidence that new minority teachers were under-performing on teacher certification exams compared with new white teachers. Ehrenberg and Brewer wanted to test the results of trading off teacher experience, educational preparation, and academic ability for teacher race and ethnicity on both minority and non-minority students. The ideal data set for this purpose would contain information on teacher and student characteristics, including teachers’ educational background, test scores, and experience. The data would be individual in nature so that students could be matched to specific teachers, and they would be longitudinal in order to permit examination of the relationship between students’ test scores and teacher characteristics during the time period. While no individual-level data sets from a contemporary period provided the information required for the analysis, Ehrenberg and Brewer determined that they could use the school-level data from the Equal Opportunity Survey (EOS) database (the data on which the 1966 Coleman report, On Equality of Educational Opportunity, was based) if they were willing to make one crucial assumption; because these data were not longitudinal in nature, they needed to assume that differences in test score performance between two grades in a school at a point in time were reasonable proxies for how much students would learn if they remained at the school for both grades. Despite such limitations, Ehrenberg and Brewer wanted to use an existing data set rather than engage in an original data collection effort due to the immediacy of the question.

OCR for page 36
Advancing Scientific Research in Education The EOS data are considered one of the most important social science data sets of their era. However, Ehrenberg reported that these data are not well archived—the data he received from the National Archives were poorly documented, and data records were missing entirely for more than 5,800 of the original 50,000 teachers surveyed. As a result, he had to overcome several methodological issues in order to conduct his analysis. Ehrenberg’s and Brewer’s (1995) findings included evidence that teacher’s academic ability correlated with higher student achievement, and that a higher percentage of black teachers in a school correlated with greater academic achievement for black students but lower achievement for white students. Recognizing that these findings had considerable policy implications, Ehrenberg and Brewer cautioned that policy decisions should not be made based on evidence from one study alone. Indeed, they subsequently used data from the National Education Longitudinal Study of 1988 (NELS) to re-examine their research question (Ehrenberg, Goldhaber, and Brewer, 1995). As with the EOS data, the NELS data were not perfectly suited to their analysis, but they provided several advantages over the EOS data. Although they did not include measures of teacher academic ability, they came from a more contemporary period, permitting Ehrenberg and Brewer to extend their analysis to consider the matching of teachers and students not only by race but also by gender and ethnicity. They found that, for the most part, the match of teachers and students by race, gender, and ethnicity did not affect how much students learned. However, they did find evidence that racial matching sometimes influenced teachers’ subjective evaluations of students, which might in turn influence students’ aspirations and the tracks on which they are put. While the findings from these data sets are not entirely consistent, Ehrenberg argued, there is enough commonality to suggest the need to train teachers to be effective with all population groups, regardless of race or ethnicity. Without the availability of these data sets and techniques for adapting data to new questions, this research would not have been possible in such a timely fashion. Workshop: Understanding and Promoting Knowledge Accumulation in Education Research: Tools and Strategies June 30-July 1, 2003 Transcript available at: http://www7.nationalacademies.org/core/ Key Speaker: Ronald Ehrenberg, Cornell University

OCR for page 36
Advancing Scientific Research in Education facilitate the verification of results obtained by allowing other researchers to reproduce them, enable replications that test the boundaries of theories and help articulate their generalizability, promote the development of validated measures, and provide opportunities (and often cost savings) to pursue new questions and directions. Thus, sharing data can facilitate growth in new areas of inquiry by allowing groups of researchers to consider how others’ data, measures, and constructs reinforce, call into question, extend, or refute their own, enabling collaborative thinking and advances in investigating phenomena—such as learning processes—that require in-depth contextual analysis and differentiation. One mechanism for encouraging and facilitating data-sharing and knowledge accumulation is the development of ethical standards for data sharing in the professional and scientific associations that represent education researchers and related social scientists (and reinforcement of those standards in the publishing policies of their journals; see Recommendation 6). The American Psychological Association (APA), the American Anthropological Association (AAA), and the American Sociological Association (ASA), all have such standards; they vary in strength but encourage the same concept. For example, one part of the ASA code reads: “Sociologists make their data available after completion of the project or its major publications, except where proprietary agreements with employers, contractors or clients preclude such accessibility” (American Sociological Association, 1999, p. 16). And a part of the AAA code states: “The AAA supports the sharing of research data and encourages ethnographers to consider preserving field notes, tapes, videos, etc. as a resource accessible to others for future study. Ethnographers should inform participants of the intent to preserve the data and make it accessible as well as the precautions to be undertaken in the handling of the data” (American Anthropological Association, 2004). Notably, however, the last revision to the ethics code at the American Educational Research Association (AERA) was in 2000, and it did not result in the addition of such data-sharing elements. We urge the AERA to adopt such a standard, and we encourage other such associations to continue to revisit their standards to promote the maximum ethical use of data. How to craft and implement such a policy in a way that protects the fundamental right to information privacy is a matter of some complexity (National Research Council, 2000). Any data-sharing policy must ensure that privacy and confidentiality safeguards are in place to protect data on study participants and groups of participants; that the rights of, and pro-

OCR for page 36
Advancing Scientific Research in Education tections for, authors are considered; and that standards are appropriate for the different types of research conducted under the umbrella of education research. A host of federal laws and regulations govern the collection of education data in relation to privacy and confidentiality issues. Key privacy laws include those that govern all research involving human participants (e.g., the Privacy Act of 1974) and those pertaining to education research involving human participants in particular (e.g., the Protection of Pupil Rights Act and the Federal Educational Rights and Privacy Act). Additionally, the Education Sciences Reform Act governs confidentiality issues in education research, and the so-called “common rule” from the code of federal regulations (CFR, Title 45, part 5) provides a regulatory framework for the protection of human participants in education and other fields. Each of these statutes and regulations affects the collection of data from human research participants in education research, and therefore are likely to apply to the sharing of data for further analysis as well. Ensuring compliance with federal laws and regulations governing human research participant protections and safeguards is mainly the responsibility of institutional review boards (IRB). Every institution has its own IRB. Every IRB has its own policy on data collections and data sharing, and no single interpretation or set of guidelines has emerged. The historical focus of IRBs on informed consent and notification policies to protect the rights of human research participants is complex and widely variable (National Research Council, 2000), and their experience considering and encouraging data sharing while protecting these rights limited. These complexities notwithstanding, there are IRBs that recognize and promote the ethical sharing of data that can be used as models to facilitate its practice more broadly (see, e.g., University of Pittsburgh IRB Manual). Conceptually, maintaining confidentiality requires attention to three relevant questions: Who has access to data? What are threats to confidentiality? What are techniques for protecting confidentiality? (Bradburn, 2003). On the question of who has access to the data, typically there is a restricted set of people who have full access to data and have promised to protect the confidentiality of the subjects. A key question is how this “umbrella of confidentiality” can be extended while ensuring that the responsibilities of confidentiality are upheld. With respect to the second question, a common threat to confidentiality occurs any time a researcher adds data to an existing data set, because the addition of, for example, geographic information to a person file increases the potential to identify individuals and compli-

OCR for page 36
Advancing Scientific Research in Education cates efforts to maintain confidentiality. Another source of threats stems from law enforcement. The Patriot Act, for example, provides for government access to any individual data collected by a federal agency, even education statistics, with few exceptions, and data can be subpoenaed for various reasons (Bradburn, 2003). Finally, threats to maintaining confidentiality are typically more pronounced with the sharing of qualitative data. Developing strategies for maintaining confidentiality with such data will require broad and creative effort, but is just as important a goal to work toward as the sharing of quantitative data. In promoting broader data sharing while ensuring privacy and confidentiality safeguards, the field can learn many valuable lessons from the experience of federal statistical agencies (e.g., the National Center for Education Statistics [NCES] and the U.S. Census Bureau) and work to adapt and extend them to different types of data. There are two main strategies for protecting confidentiality: restricting access and altering data (Bradburn, 2003). For example, agencies may restrict access so that it is permitted only in data enclaves. This is the model used at the U.S. Census Bureau, which supports a number of research data centers around the country. There are very strict controls over access, but the centers permit a lot of research that would be otherwise impossible to conduct. Another model is used at NCES. Both public-use data files and restricted-use data files are available: the public-use files contain anonymous versions of individually identifiable data; restricted-use files contain more detailed, individually identifiable data, although direct identifiers such as Social Security numbers and addresses are removed from all analysis files. Access to public-use data is open to any interested party. Restricted-use files are available only to qualified researchers for approved statistical purposes. Researchers apply for a license and are loaned a data file, and their use of the file is subject to the terms of a license agreement. A strategy of altering data can also be an effective way to maximize access to data while maintaining confidentiality of research participants. Ways of altering data include such strategies as holding identification data in a separate file, and creating synthetic data sets (Bradburn, 2003). NCES currently has more than 100 public-use microdata files with data from the late 1980s through the present available on its web site. Data files from the 1960s and 1970s are also available through an archive. Researchers can manipulate the data in a number of these files using the NCES Data Analysis System. In addition, NCES has developed a set of tools to

OCR for page 36
Advancing Scientific Research in Education facilitate access to the data, including several on-line tools. It is possible to look up basic data on individual schools and to compare schools in a variety of ways, such as by having a similar characteristic (e.g., size; rural, suburban, or urban; etc.). NCES also provides a set of prerun tables that permit individuals to cut the tables in different directions, such as getting student scores by several different variables (Seastrom, 2003). These policies avail education researchers of opportunities to mine the extensive data sets collected by NCES, and do so in a way that protects the privacy and confidentiality of research participants. Recommendation 6: Education research journals should require authors to make relevant data available to other researchers as a condition of publication and to ensure that applicable ethical standards are upheld. Norms for data sharing in many of the physical and natural sciences are reinforced in their professional and disciplinary journals. In journals such as Science and Nature, once an article is published, the author has to make the data available to those who wish to replicate the results. In both of those publications, authors are required to provide their data arrayed in identified files that directly correspond to results reported in the tables and figures in the manuscript. The Proceedings of the National Academy of Sciences has a similar policy: the journal will also house the data if appropriate data repositories do not exist. If authors of published articles refuse to comply with requests for their data, the journal bars them from future publishing. Similar traditions in the social and behavioral sciences and education journals are not as well established. There are exceptions, however. The American Economic Review, for example, has a policy similar to those in the Nature and Science. In addition, journals published by the APA require authors to provide the data relevant to their articles to competent researchers. There are problems implementing these sorts of policies, as well as issues associated with promoting replication and reanalysis even when such policies do exist. For example, although APA’s data-sharing policy had been in effect for 25 years, staff estimates that less than one-tenth of a percent (0.001) of available data are actually shared as the policy envisions (VandenBos, 2003). In our view, this low rate of use among psychologists reflects a host of cultural barriers and institutional incentives structures that work against data sharing that also exist in education research. Data sharing directly

OCR for page 36
Advancing Scientific Research in Education involves two parties: the person or organization that originally collected the data and the person or organization that would like to make use of an existing data source.2 Disincentives exist for both (Natriello, 2003). For individual researchers who have engaged in data collection, it is not always clear how to prepare data to share, a problem that has intensified as data sets have become more complex. A further disincentive along this line is that the process of preparing data to be shared requires considerable effort (and therefore cost) beyond what is required for preparing data for analysis and publication. A lack of standards for the preparation and archiving of public data sets leads to institutions and individuals having to reinvent the wheel in determining how to go about the process of data preparation. Finally, researchers may be reluctant to share their data because when they do so, they lose exclusivity for reporting on the data. Researchers often want to hold their data as long as possible so that they have the advantage in analyzing them before the larger community gains access to them (Natriello, 2003). There are also disincentives for researchers who want to use existing data. Such efforts often require researchers to expend considerable effort (and therefore cost) in order to overcome technical barriers. Some researchers do not view the work as truly their own if they were not involved in the data collection. Finally, publication outlets can serve as a significant disincentive for conducting work using existing data. Peer-reviewed journals as well as tenure and promotion policies both tend to value publications that strike off in new directions or provide novel ideas and to devalue studies that build on previous work (Schneider, 2003; Natriello, 2003). These and related issues will need to be addressed formally to mitigate such disincentives. For example, universities and departments could work to change tenure and promotion decisions so that they recognize efforts that promote data sharing as valuable intellectual contributions to the field; a singular focus on rewarding researchers for publishing in a small set of elite journals may not serve the field in the long run. Associations could lead an effort to develop standards and protocols for preparing data to be shared. And federal funding agencies could encourage the sharing of data by outlining the conditions under which it is permissible in the informa- 2   Other parties have a role in, and may present conflicting interests with respect to, data-sharing (e.g., the public, federal funding agencies); see National Research Council (1985) for a discussion.

OCR for page 36
Advancing Scientific Research in Education tion provided to its grantees. Individual education researchers can also facilitate these efforts in small ways through their everyday work. Recommendation 7: Professional associations and journals should work in concert with funding agencies to create an infrastructure that takes advantage of technology to facilitate data sharing and knowledge accumulation in education research. Advances in modern technology open avenues for the development of research infrastructure that facilitate the building of the knowledge base in education unimaginable just a few decades ago (National Research Council, 1985). In this context, we comment on the value of data repositories, registries of studies undertaken, bibliographic indexes of published studies, digitization of journal content, and open access models. A host of existing data housing and data repository sites exist across many of the social and behavioral sciences, as well as in education research. The Inter-university Consortium for Political and Social Research (http://www.icpsr.umich.edu/org/index.html), for example, maintains and provides access to a vast archive of social science data, including education-related data, for research and instruction, and offers training in quantitative methods to facilitate effective data use. It was established in 1962, includes over 500 member colleges and universities, and encourages all social scientists to contribute to its data resources. The archive includes subparts, such as the National Archive of Criminal Justice Data, the Health and Medical Archive, and the International Archive of Education Data. The Henry A. Murray Research Center is a data archive dedicated to the study of lives over time. It is unique in that it holds many longitudinal studies and includes not only quantitative data, but also qualitative materials, such as case histories, open-ended interviews, and audio and video clips. Data repositories can be powerful tools in the pursuit of scientific understanding in education. For example, they can be used to facilitate an accumulated knowledge base by encouraging the continued development of measures of key concepts in education research. In the context of dominant theories, these concepts and variables can be validated and revisited as investigators engage in new investigations over time, greatly facilitating integration across studies. In our view, attention is needed on how to encourage greater use of existing repositories as well as the possibility of developing new ones for facilitating data sharing and knowledge accumulation in education research.

OCR for page 36
Advancing Scientific Research in Education Developing a single bibliographic index of education research studies—similar to the Educational Research Information Center (ERIC)—is another essential component of a technology-enabled infrastructure for education research. A fully elaborated resource not only should provide access to the full text of research documents and efficient and powerful searching tools for information retrieval, but also should be buttressed by a comprehensive indexing system that includes standardized definitions of key terms in an electronic thesaurus and clear standards for required descriptors of articles. The process of developing such an indexing system requires specialized expertise and would be a multiyear undertaking. A multiyear effort spearheaded by the National Library of Medicine to develop a system for the biomedical sciences links to the keywords of major medical journals (Rothstein, 2003). It is neither possible nor desirable to mandate single definitions of concepts for all of education research. Indeed, many changes in terminology are made in light of improved understanding of an issue. Overly restrictive standardization can straightjacket researchers and, ironically, stunt efforts to advance knowledge accumulation. Yet, it would be valuable to keep up-to-date definitions of major constructs in education through partnerships of associations like the AERA and federal agencies like the Institute of Education Studies (IES) that track, for example, a set of basic terms used in their journals and databases. Such efforts should recognize the importance of, and take into account, the testing of ideas and revision of measures and definitions over time and across different contexts. For example, definitions of measures could include relevant clarifications and differentiations that might arise when considering different student populations. When feasible, there should be links to appropriate data repositories. Ongoing discussions to promote such standards for methodological terms in particular within the International Campbell Collaboration3 can provide helpful guidance. In implementing this recommendation, we also see the potential for an international effort that can bring together existing expertise, as well as for promoting policies that index terms across international contexts (e.g., relating the terms “elementary school” and “primary school”). 3   The international Campbell Collaboration is a nonprofit organization that aims to help people make well-informed decisions about the effects of interventions in the social, behavioral and educational arenas. Its objectives are to prepare, maintain and disseminate systematic reviews of studies of interventions (see www.campbellcollaboration.org).

OCR for page 36
Advancing Scientific Research in Education Another mechanism to pursue as a long-term goal for facilitating knowledge accumulation is the establishment of a register of all education studies undertaken. Such a register would be different from a bibliographic database of all published studies (such as ERIC). One rationale for this kind of resource stems from the problem of publication bias. Publication bias, sometimes called the “file drawer problem,” is the tendency for researchers to publish results that have positive results or effects and not to publish findings that have null, negative, or not statistically significant results (Iyengar and Greenhouse, 1988). Thus, efforts to synthesize existing work on a particular topic—including meta-analysis and the reviews currently being conducted by the What Works Clearinghouse—are at risk of being biased because of the systematic loss of statistically insignificant or negative study results or of being incomplete because studies are not available in published journals. In either case, systematic reviewing is unnecessarily laborious and expensive because hand searching and other ways of sifting through the “fugitive” literature are required. Another rationale for creating a single resource for researchers and consumers of research that contains information about ongoing work is that it could be a valuable tool for facilitating communication and collaboration among investigators working on similar issues and problems and for expanding access to relevant research among consumers of research. Creating such a register of all studies undertaken is an ambitious undertaking. It would require navigating some thorny intellectual territory regarding what should be included in such a register: the vast quantity and divergent nature of the kinds of inquiries that might reasonably be considered education research will make the development of clear standards and protocols a difficult and probably contentious task. Clinicaltrials.gov, a federal effort to register trials of serious and life-threatening diseases, is an excellent model for one kind of study. A proposed effort to expand this register to include all initiated clinical trials in the near future offers initial lessons from which the education research communities could learn and begin their own dialogue about how to approach a similar undertaking. Journals, too, can use technology to support similar aims at relatively low cost. Nearly 80 percent of scholarly journals are now available on-line (Willinsky, 2003). Indeed, the question is not if journals should be digitized but when, how, and for what purpose. Creating databases that house every article published by journals is technically feasible, and it offers a number of ways to promote quality and coherence in education research. For example, it enables the compilation of collections of related work to

OCR for page 36
Advancing Scientific Research in Education BOX 3-2 The Value of Digital Journal Content Gary Natriello, executive editor of the Teachers College Record (a peer-reviewed journal of Columbia University), has led an effort to digitize the journal’s content and leverage the opportunities afforded by digitization for packaging material and opening channels for communication in ways that facilitate the building of a knowledge base over time. He described the innovative uses of TCR content enabled by the creation of TC Record On-line (www.tcrecord.org). Natriello described digitization as providing the journal choices about how to bundle material in articles, series, or compilations. It also enables different formats and lengths, and innovative ways to represent articles. He said that it also offers the opportunity for two-way interaction. For example, the TC Record On-line publishes community discussion, in which they invite people to comment on articles and on themes and content and place that commentary on their home page. In addition, the editorial team has edited collections of works, linked related materials across decades or longer, tracked the progress of a series of articles that developed over time, followed illustrate the progression of a line of inquiry over time. Authors’ names can be searched to identify potential reviewers by quickly allowing editors to view their publication records on the topic of a pending manuscript. Indeed, the power of digitized content is that it is flexible—innovative ideas for packaging and relating studies can be tried and studied at relatively low cost. Opening access through the Internet has important potential benefits to consider in the context of building the knowledge base: it can promote scholarly communication by allowing on-line dialogues on topics of interest, including critiques and reviews of published articles. See Box 3-2 for an example of how one education research journal has done so. Finally, allowing consumers free or low-cost access to journals on-line in education can serve a very important function in helping to engage multiple audiences and extending the reach of these publications. Open access is a movement in publishing that innovatively seeks ways to provide access to users of content at no charge. Some journals, such as the AERA publica-

OCR for page 36
Advancing Scientific Research in Education an author through his or her career, and “seized” teachable moments. Natriello also described plans for the next generation of the journal, in which they will convene groups of scholars around particular content areas, facilitate the group’s interaction, and summarize and make available the results of that interaction to simulate an on-line consensus conference. Another strategy TC Record On-line has attempted in order to facilitate greater knowledge accumulation is making data sets available on-line. Natriello told the group that the journal has offered to do that for authors for nearly five years, and no one has done so to date. The TC Record On-line team believes that their strategy for encouraging submission of data needs to change, so that they illustrate the process and power of this option by posting examples, paying people to do so, and exploring the use of other such incentives. Workshop: The Role of Journals in Developing the Education Research Knowledge Base November 11, 2003 Transcript available at: http://www7.nationalacademies.org/core/ Key Speaker: Gary Natriello, Teachers College, Columbia University tion Educational Researcher, are free immediately and available around the world. Another example is the delayed open access provided by Teachers College Record and the New England Journal of Medicine, which open access six months after initial print publication. Lessons are beginning to emerge from the early experiences of several major journals that have experimented with this idea, and momentum is building to expand access to scientific findings in peer-reviewed journals. Indeed, the National Institutes of Health (NIH) recently invited comment on a proposed new requirement for NIH-funded researchers to provide articles accepted for publication to the agency so it can make them freely available six months after publication (see http://grants1.nih.gov/grants/guide/notice-files/NOT-OD-04-064.html). A major issue with opening access this way has to do with the financial losses that are likely to accrue to publishers. Some people have suggested that the losses stemming from declines in paid subscriptions be offset by generating revenue from authors as a condition of publication. An example of an on-

OCR for page 36
Advancing Scientific Research in Education BOX 3-3 Using Technology and Expanding Access to Facilitate Research Communication John Willinsky described an effort called the Public Knowledge Project (http://www.pkp.ubc.ca/) that he has spearheaded to expand access to research through free (open access) online tools. He argued forcefully that a crucial dimension of the quality of research knowledge has to do with its circulation. Claims of research quality, he posited, are reduced by anything that unduly restricts the circulation of research, and this is becoming more a salient issue since access is declining in many institutions due to the high cost of journals. Willinsky provided a demonstration of the journal management and publishing system that he has developed through the Public Knowledge Project. The Open Journal Systems was designed on three principles: it is free, it reduces the amount of work that is normally involved in editing by automating many of the processes, and it improves the scholarly and public quality of the research published. The premise is that access to tools that reduce publishing costs would provide an incentive to journals to make more of their work open to readers. The system is free to download and installs on any server. Editors and authors only need to do word processing and use fill-in web forms to use it. Open Journal Systems helps to manage the review process by allowing authors to upload their work, including supplementary files and data sets; by enabling reviewers to access these items and provide review comments all on an online basis; and by allowing editors to do their editing where they wish: in airport kiosks or lounges—anywhere there is a web line open access content management tool is described in Box 3-3 and shows how such tools can benefit researchers and consumers alike. Developing and fully exploiting each of these mechanisms will require care in ensuring that such infrastructure development and standardization be approached in a way that continues to encourage critique, questioning, exploration, and reconceptualizations. Actions that lead to constraining standardization will stifle innovation and crowd out talent. In our view, the field would benefit from the kind of careful foundational development called for in these recommendations by enabling advances across fields, investigators, and studies.

OCR for page 36
Advancing Scientific Research in Education browser. Willinsky pointed out that this feature allows international teams of editors to begin to participate in journal publishing, which has historically been difficult due to the need for centralization. As he described it, one innovation of this system is that it provides journals using it with a research support tool to accompany each published article. The research support tool provides readers an “information context” that supports the public’s ability to interpret and use relevant research-based information. He argued that the public has an interest in research when it is directly concerned with what they are doing. They have the motivation, he argued, but not sufficient context to interpret the research. Thus, the research support tool identifies, for example, that an article is peer reviewed; it also allows readers to readily locate related studies in the Educational Research Information Center database, to help them see that “no study is an island unto itself.” The system also includes links to other relevant resources associated with the author’s keywords, such as FirstGov (www.first.gov), a website that taps into U.S. government websites as well as the websites of all states. Willinsky informed the group that the tool is currently being tested with policy makers in Ottawa as a way of assessing how open access publishing can increase research’s impact on policy. Workshop: The Role of Journals in Developing the Education Research Knowledge Base November 11, 2003 Transcript available at: http://www7.nationalacademies.org/core/ Key Speaker: John Willinsky, University of British Columbia Building infrastructure takes time. No single journal, scientific association, or federal agency could build these tools alone. We call on the major institutional players in education research—the AERA and the IES (especially ERIC)—to provide the leadership and resources to explore and encourage these kinds of efforts and to set priorities for implementation in light of resource availability over time. We also see a role for the major philanthropic foundations that support education research to make these ideas a reality. It will take the combined efforts of these institutions to marshal the resources that will be required.

OCR for page 36
Advancing Scientific Research in Education Recommendation 8: Education research journals should develop and implement policies requiring structured abstracts. Abstracts are short summaries that accompany full-length articles, and they can also appear independently to summarize presentations given at meetings. Abstracts serve a number of purposes. One of the most important is to allow other researchers and consumers of research to easily obtain information about the key elements of published studies. Such information enables researchers working on an issue to find related work, enhancing the likelihood that relevant bodies of work can be identified and interpreted to promote an accumulated knowledge base over time. In a similar way, structured abstracts would also promote better utilization of education research. A recent article published in Educational Researcher (Mosteller, Nave, and Miech, 2004) made this case convincingly, offering a prototype structure for consideration by the education research communities and associated journals. Proposed elements of structured abstracts include background/context; purpose/objective/research question/focus of study; setting; populations/participants/research subjects; intervention/program/ practice; research design; data collection and analysis; findings and results; and conclusions and recommendations (p. 32). It is important to recognize that following a sequential format in structured abstracts may not be appropriate for all kinds of education research (e.g., the goal of some anthropological work is to generate a question, rather than to address a question or test a hypothesis). Including a common set of information in these abstracts, such as those described in the Mosteller et al. article, is the key. Beyond the generalized benefits the field could realize from structured abstracts, the early experience of several recent efforts to synthesize high-quality education research on topics of importance to policy and practice has provided an additional rationale and sense of urgency for implementing a policy of structured abstracts in education research journals. The International Campbell Collaboration and the federally funded What Works Clearinghouse are both developing systematic, rigorous, and transparent processes for summarizing research findings and ways to communicate those summaries to educators who can most benefit from them. Many of the technical challenges associated with creating these resources stem from problems in accessing and summarizing articles from scholarly journals to synthesize social science research. Structured abstracts would help alleviate these problems. Workshop speakers with expertise in systematic reviewing argued that to date, the process of systematic reviewing has been hampered by the very

OCR for page 36
Advancing Scientific Research in Education poor quality of abstracts and, in some cases, very misleading titles of articles. This problem is compounded by the fact that electronic abstracts are often less useful than the actual hard copies: for example, the Educational Resources Information Center—the primary database for education research articles—reduces authors’ abstracts, often resulting in their missing some of the crucial information needed for systematic review information retrieval and analysis (Rothstein, 2003; Sebba, 2003). The way in which current abstracts fail most often is in providing an adequate description of the characteristics of the sample (Sebba, 2003). For example, a study may involve only ethnic minorities or elementary school students, but the abstract does not specify the age group or the race/ ethnicity of the sample used in the study. A requirement for structured abstracts, coupled with greater attention to the quality of abstracts by editors and publishers, would go a long way in producing abstracts that are readable, well organized, brief, and self-contained and that facilitate the systematic review process (Hartley, 1997). Developing standards for structured abstracts should be done internationally, so that terms that vary across borders can be defined and referenced. In sum, abstracts are critical to the information retrieval process for developing systematic reviews and meta-analyses to sift through and to identify the universe of relevant research. When these abstracts fail to contain basic information about the study objective, sample strategy, research design, and other key features of research, any searching process becomes intensely laborious, slowing the work considerably. Furthermore, missing relevant articles has the potential to bias the summary results by skewing the sample of studies selected for review. Implementation of a standardized format for abstracts in journals is a relatively easy yet powerful change for journals to make to facilitate such reviews (Ad Hoc Working Group for Critical Appraisal of the Medical Literature, 1987). CONCLUSION Most codes of ethics that specify professional norms and expectations for social scientists include standards for ways in which individual investigators are responsible for contributing to their field as a whole. Many of these standards relate directly to the kinds of efforts we recommend be taken to build a knowledge base in education research that accumulates over time through better and more frequent communications and data sharing. Therefore, most of what we call for is not new: rather, they are policies and practices that have been neglected; they deserve renewed attention.