Design Principles for Fostering Science in a Federal Education Research Agency
The federal government has an important and legitimate role in supporting research as a public good, including research in education (e.g., National Research Council, 1999d; President’s Committee of Advisors on Science and Technology, 1997; Geweke and Straf, 1999). The federal government’s role in education research dates back to the middle of the nineteenth century, when the U.S. Department of Education was established1 to collect statistics and provide exemplary models for the nation’s schools. Then as now, the nation recognized the value of centrally generated education research that should be made available to all states, districts, and schools. In the absence of a federal leadership role, knowledge gained by one state or district that might be relevant to others would not likely be widely distributed, as individual states tend to undervalue the benefits that would accrue to others. Moreover, many scientific studies contrast alternative education approaches or models, and important comparisons are frequently made across states, districts, and schools. The federal government is also the natural place to collect and make data widely available on education performance, costs, processes, inputs, and their interrelationships.
Assuming a legitimate federal role in education research, this chapter addresses the question: How should a federal education research agency be designed to foster scientific research in education, given the complexities of the practice of education, the stringencies of the scientific principles, and the wide range of legitimate research designs?
While our focus is on design principles for a single agency, we point out that education research of national interest has historically been supported by several offices in the U.S. Department of Education, by other agencies of the federal government, and by private organizations (e.g., foundations). A federal agency is only one part of this larger enterprise, but it occupies a central place within it. Indeed, while the committee makes a number of suggestions for one agency to lead the scientific enterprise, we recognize that some of the tasks might best be conducted in partnership with other agencies or nongovernmental organizations, and we encourage the exploration of such options. Within this broader context of scientific research in education, this chapter takes up the specific issue of how a federal research agency might be designed to best fulfill its role in the scientific enterprise.
Our approach in this chapter is forward looking. Throughout, we speak of a generic agency because the committee wanted to free its deliberations from exclusive consideration of the current incumbent, the U.S. Department of Education’s Office of Educational Research and Improvement (OERI). Although this report is in part intended to help policy makers think about the pending reauthorization of OERI, the committee was not charged with, nor did it conduct, an evaluation of OERI. Rather, we relied on data we have collected from a sampling of federal social science research agencies and programs—including OERI—about how they support their science missions.2 In short, while we reiterate that we did not evaluate OERI, we clearly could not avoid learning about it or discussing it, especially in a comparative way, to address our charge effectively. Thus,
throughout this chapter we refer to OERI and other agencies, most often comparing various aspects of funding and operations among them.
We also relied on information the committee gathered at a workshop it sponsored in March 2001 that featured panels of senior officials from these and other agencies as well as knowledgeable experts about the federal role. The participants discussed the federal role in education research and related social sciences across several agencies with an eye toward the future of a federal education research agency (again, OERI was one of several agencies represented and discussed). This event is summarized in a workshop report (see National Research Council, 2001d).
Based on the information gathered at the workshop and through subsequent data collection, our guiding principles of science, the features of education that influence the conduct of research, and the nature of scientific progression, we develop six design principles around the notion of creating a scientific culture. We argue throughout this report that science itself is supported through the norms and mores of the scientific community, and we believe that cultivating these values within a research agency is the key to its success. We also note that decades of organizational fixes at the current agency have arguably not done much to improve its culture and, consequently, its reputation.
Our focus on a scientific culture within an agency stems from the recognition that an agency in many ways reflects the field it supports, and vice versa. An agency’s success requires a strong group of scholars, and the broader community depends in part on a vibrant federal presence. Thus, our design principles emphasize the role of researchers to lead and staff the agency, to serve on advisory boards, to help synthesize the current state of knowledge, and to act as peer reviewers of proposals and programs of research. The principles also recognize the role of the agency in building the professional capacity of the field.
Other themes in this report are embedded in the design principles as well. For example, we take up the issue of research ethics—an influential aspect of the education research enterprise (see Chapter 4)—from the perspective of the federal regulations that govern them. We also argue for flexible decision-making authority in the agency to accommodate the dynamic nature of scientific progress and opportunity (see Chapters 2 and 3). And we suggest the agency attempt to enhance part of its research port-
folio by finding ways to bring it closer to the complexities of educational practice (see Chapter 4).
It is important to recognize the difference between this focus on developing a scientific culture and the focus on research methods in H.R. 48753—the bill that at least in part led to this report—and several related debates about the future of OERI. The language in the bill contains many of the key concepts we treat in this report, including systematic data collection, experimentation, rigorous reasoning, replication, and peer review. However, attempting to boost the scientific basis of federally funded education research by mandating a list of “valid” scientific methods is a problematic strategy. The inclusion of a list of methods—regardless of how they are applied in particular situations—erroneously assumes that science is mechanistic and thus can be prescribed. We have shown that science adheres to a set of common principles but its application depends greatly on the particulars of a given situation and the objects of inquiry. The definitions also make clear distinctions between quantitative and qualitative methods, implying that these two types of research approaches are fundamentally different; we argue the opposite. Furthermore, the use of definitions of methods as a tool for improvement fails to recognize the crucial role of theory and, as we emphasize, a strong, self-regulated, skeptical community of researchers that pushes the boundaries of knowledge. It is in this spirit that we focus on scientific culture in approaching the design of a federal education research agency.
The committee recognizes an inherent dilemma in designing an agency to support scientific research in education. Scientific education research is often grounded in the practical problems of teaching, learning, and schooling and their varied contexts. Therefore, it is important to engage practitioners in the functions of the agency and to link research to the design and management of federal education programs. However, as we describe below, history has shown that a close bureaucratic relationship between research and educational programming in a federal agency can overwhelm the research function. Thus, we attempt to clarify the proper roles of researchers, practitioners, and politicians to ensure the needs of these com-
To view the text of the bill, go to http://thomas.loc.gov/ and search for H.R. 4875 in the 106th Congress.
munities are met and their strengths maximized. We believe strongly that the responsibility for the success of the agency and the broader research effort it supports lies not solely with federal policy makers, but is shared among all those who have a stake in education and education research.
Another dilemma has to do with the composition of the education research community itself. As we argue earlier in this report, we believe that the vast diversity that characterizes the field of education research is both a great strength and a troubling weakness. The variation in epistemological paradigms, methodological tools, and professional training lends the enterprise intellectual vitality. This same variation, however, is a source of cultural divisions among subfields that fosters isolation and impedes scientific consensus building and progress. In short, the “community” of scientists in education is really an eclectic mix of scholars with different norms and different standards of evidence. While we talk about the scientific community and its role in a federal agency as if it were a unified, easily identifiable group, the reality is more complex. The talent pool in education research is shaped by a number of structural, historical, and cultural variables, and parsing them out requires careful analysis. Thus, in the discussion that follows, we attempt to highlight issues that may be relevant in the implementation of the design principles vis-à-vis the field.
Our vision is that the fundamental mission of a federal education research agency would be to promote and protect the integrity of scientific research in education with the goal of generating knowledge that can inform policy and practice decisions.4 To achieve this mission, the agency needs to develop and nurture a scientific culture, and to do so, it must have an infrastructure, supported by sufficient resources, that enables a cadre of experienced staff to make decisions flexibly and to interact continuously
with the field it supports. We develop six design principles from these core ideas:
Staff the agency with people skilled in science, leadership, and management.
Create structures to guide agenda, inform funding decisions, and monitor work.
Insulate the agency from inappropriate political interference.
Develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to policy and practice.
Adequately fund the agency.
Invest in research infrastructure.
The rest of this chapter elaborates these principles and provides suggestions for specific mechanisms that could be implemented to support them. We stress that these suggestions do not reflect a view that there is one “model” that dictates the specific design features of any federal research agency. Indeed, the U.S. federal research enterprise is characterized by a range of structures and processes that is effective in respected agencies across the federal government (National Research Council, 2001d; Mathtech, 1996).
DESIGN PRINCIPLE 1
Staff the Agency with People Skilled in Science, Leadership, and Management
We begin with leadership and staffing deliberately: a scientific culture begins (and ends) with competent people. Attracting and retaining an adequate number of qualified leaders and staff is so critical to a healthy federal education research agency that we believe without it, little else matters. There is no substitute for leadership and human capacity.
The leaders of the agency are of paramount importance. All federal agency leaders need leadership and management skills and, for this agency, the leaders—political appointees and career officials alike—must be respected educational researchers with strong scientific credentials. The culture of an organization emanates from its leaders; without research
experience at the top levels, the norms and mores of science will likely not take hold.
Similarly, the agency’s research staff should have extensive education research experience. They must have knowledge of relevant content as well as be able to recognize scientifically rigorous design, theory, data collection strategies, and analysis techniques. The agency should employ a mix of research staff, where promising junior scholars work alongside senior research staff to infuse new ideas into the agency’s work and to develop future senior staff of the agency. Providing ongoing professional development opportunities for research staff is also critical to allow continuing and sustained interaction with the broader research community.
How can a federal education research agency attract and retain such human resources to develop and maintain a scientific culture that fosters and protects the integrity of scientifically rigorous research in education? This is a difficult question.
In keeping with the range of hiring strategies of several existing federal research agencies,5 a federal education research agency should have the authority to pursue multiple approaches to develop its leadership and staff. Developing a core of permanent staff offers the benefit of increasing institutional knowledge and long-term relationships within the government and the research field. Short-term assignments can serve the dual purpose of updating the agency with new ideas from the field and acquainting university faculty and other researchers with the operations, needs, and
accomplishments of the agency. For similar reasons, the appointment of postdoctoral fellows can be beneficial. These appointments also help build the capacity of future leaders in the research field.
Another way for a federal education research agency to cultivate scientific norms in its staff is to engage in collaborative research efforts with other agencies to encourage interaction with staff who have traditions of supporting related scientific work. Such collaborations can enrich the breadth and depth of staff knowledge, and they also offer the benefit of developing cutting-edge, interdisciplinary research programs across federal agencies (National Research Council, 1999d). There are several current interagency efforts in education research from which lessons could be learned: examples include the Interagency Education Research Initiative, a partnership of OERI, NICHD, and NSF aimed at understanding how to scale up promising education practices, and a joint OERI-NICHD initiative focused on understanding how best to help bilingual students learn to read in English.
Although these policy tools can help attract and retain top staff, staffing will depend heavily on related issues, such as funding, reputation, and leadership. For example, to the extent that education research is underfunded relative to other opportunities available to researchers (see Design Principle 5 below), top talent will likely go elsewhere, both in the field and in the agency. The early 1980s provides a lesson. With significant federal budget cuts in education and social science research, researchers migrated, especially to the health and national defense fields, many never to return. These funding deficiencies have affected OERI’s ability to attract and retain a cadre of capable staff despite having hiring authority similar to NSF and NIH. Staff levels were reduced drastically in the 1980s as a result of deep federal budget cuts, but even as its funding began to climb in the 1990s, the agency again lost 25 percent of its staff, including some of the most capable and experienced individuals (Vinovskis, 2000).
The reputation of an agency and its leadership will also affect staffing. Developing a good reputation, of course, is not a simple matter of policy fixes. Adequate funding will help, but if the agency suffers from a poor reputation, its leaders will have to be creative about staffing possibilities and may need to convince a critical mass of researchers to serve at the same time. In this vein, the development of a scientific culture is critical; initial
appointments have to be highly talented researchers who, in addition to being offered very attractive positions, should be encouraged to view such federal service as an important way to strengthen their profession.
DESIGN PRINCIPLE 2
Create Structures to Guide Agenda, Inform Funding Decisions, and Monitor Work
To accomplish its core tasks, a federal education research agency must be supported by a coherent system of governance. While we do not offer a comprehensive plan, we do believe that two essential elements of such a structure have the highest probability of cultivating scientific norms both inside and outside the agency: a high-level governing board and standing peer review panels of top-flight scientists.
Governing boards are common management and oversight tools for federal research agencies. We believe that particular attention to aspects of the board’s composition and responsibilities can further the development of a scientific culture within the agency as well as foster interactions with other stakeholders. We suggest that the agency operate under the general direction of a high-level governing board, drawn from leaders in education policy and practice, education research, business, and both political parties. The diversity of the governing board will allow the many different cultures associated with each representative group to learn from one another as they work toward common goals. Many research agencies currently have some kind of a governing or advisory board (e.g., the National Science Board of the NSF and OERI’s National Educational Research Policy and Priorities Board [NERPPB]). Such a board should provide advice to senior leadership, recommend research directions, help to safeguard the independence of the agency, provide critical links to practice and policy communities, and reinforce scientific norms inside the agency.
A key task of this board would be to develop the research agenda. No matter how strong its science, if the agency does not carefully develop and maintain its agenda in close collaboration with researchers, policy makers, and practitioners and in alignment with available resources, it will fail to
meet its mission. The challenges facing American education—low achievement levels, cost, growing numbers of second-language learners—are very real and demand serious investment. For education research to play a role, it is imperative that a federal education research agency have clear, longterm priorities for generating knowledge and promoting its transfer and utilization to engage the field in a collaborative effort toward understanding core issues in education.
The agency might also include an agenda-setting committee, chaired by a distinguished practitioner, which would work with the board on the research agenda. Representatives from the scientific community should serve to help identify areas that warrant further research on the basis of the state of scientific development; this task may involve identifying areas of research that are ripe for immediate testing, or that require more basic descriptive research to generate scientific hypotheses or assertions. Representatives from the practice communities should serve to articulate the high-priority issues for educational improvement from a practical perspective. And representatives from the policy communities should serve to articulate short and enduring policy issues, as well as the feasibility of moving in new directions as recommended by researchers and practitioners.
Another role for a governing board would be to report to Congress and the nation on the agency’s progress toward clearly stated and commonly shared goals. In the spirit of the 1993 Government Performance and Results Act (GPRA), the agency should be accountable for research results: knowledge generation and dissemination. Since research results are difficult to quantify, federal research agencies have struggled to comply with this law. A recent report (National Research Council, 1999b) provides some guidance on how to assess the outcomes of research for GPRA reporting. Plans and measures should be developed according to the character of the research program and acknowledge (as we do in this report) that the progression of science is jagged and often unexpected.
Standing Peer Review Panels
Peer review is the single most typically used mechanism for nurturing a scientific culture within and outside federal research agencies, and one that should play a feature role in a federal education research agency. In
the ideal, peer review is both a process by which scientific work is assessed and funded and a product in that it provides a venue for the scientific culture of self-regulation (Chubin and Hackett, 1990) we describe throughout this report. The process works on several levels. First, by involving a group of active researchers, the current state of the art is introduced and used in judging proposed research. Second, especially when used as a feedback mechanism for the field (National Research Council, 2001d), the review process itself encourages the development of an active community of scientists working together on education problems: the process of reviewing proposals and communicating feedback fosters the development of common standards of quality and other scientific norms in the field over time. Third, the peer-review process acts as a buffer against outside political pressures to choose certain proposals or fund certain researchers regardless of scientific merit.
A wide variety of peer review structures—ad hoc review committees, standing panels, mixture of outside and panel evaluations, and the like— can work. Indeed, the current federal system is characterized by this diversity of approaches (U.S. Government Accounting Office, 1999). For example, NIH uses both standing “study sections” and ad hoc review groups, while NSF and OERI only use ad hoc panels of reviewers for each competition. In contrast, the Office of Naval Research does not use panels of peer reviewers to fund proposals but rather regards its staff as peers able to make such decisions internally (National Research Council, 2001d).
We believe that a federal education research agency ought to use standing review panels akin to the NIH study sections as its primary peer review vehicle. We envision these standing panels as providing continuity in overseeing research programs (see Design Principle 4). This suggestion reinforces the recommendations of several other groups (that are studying OERI in particular), including the RAND panels (see http://www.rand.org/multi/achievementforall/) and the NERPPB in its policy statements (National Educational Research Policy and Priorities Board, 2000). When researchers join peer review panels that have a life over a period of years, panel members strengthen their knowledge and the panel as a whole develops an integrated, communal expertise. Members then communicate this knowledge to their colleagues through their review of proposals and interaction with colleagues, and they also demand that, in the proposals
they review, the research community is up to date on the most recent synthesis of knowledge.
We caution that peer review is not a perfect quality assurance mechanism, and it can go wrong. History has shown that it can be a conservative instrument for selecting research proposals for funding, and can stifle innovation as a result (Cicchetti, 1991). To mitigate these problems, the agency’s standing panels should have rotating membership terms to ensure that fresh perspectives are regularly replenished.
Whatever the structure and management of peer review panels, their successful implementation will require diligence and care with respect to their composition. For peer review to work well, the choice of peers is vital. By peers, we mean scientists conducting research in an education area that substantially overlaps with that of the proposals under review and, importantly, who can think beyond their own line of work. Since the topical areas under any particular competition can be quite vast, sometimes it is not possible to achieve total representation of topical areas (August and Muraskin, 1999). Therefore, it is critical to ensure the peers can think broadly. The goal is to assemble reviewers who have both the substantive and methodological knowledge to make science-based judgment about the merits of competing proposals and the state of current understanding. As a result, we believe that policy makers and practitioners should not have responsibility for judging the scientific merit of research proposals; they should have opportunities for ongoing collaborations in the agency, but not as part of peer review panels (see “Governing Board,” above). This makes clear that agency staff must be adept at selecting and attracting appropriate individuals and achieving balance in groups of peer reviewers.
Furthermore, engaging the scientific community in the course of research planning and progress through peer review depends critically on an ample talent pool of peers. In the short term, one important consideration is the need to engage a range of perspectives relevant to the work. Agency leaders have to ensure that the collective membership of these panels does not bring a narrow view to the work. At the same time, the choice of peers must maximize the intellectual caliber and scientific expertise of the group. An overemphasis on ensuring broad content area, epistemological, and methodological representation can backfire if such considerations outweigh the overarching need to engage top scientific talent
in the effort. In sum, assembling the right group is a finely nuanced task. Ultimately, the long-term viability of standing panels or other mechanisms for peer review in scientific education research will depend on sustained attention to building the capacity of the field itself (see Design Principle 6 below).
We have focused thus far on the issues of peers and peer review from the perspective of a federal agency. However, the responsibility to assemble high-quality panels in the short term and to enhance the profession in the long term does not rest solely with the federal government. Indeed, we believe the community of researchers plays the most critical role in making peer review work. It is the professional responsibility of scientists to participate in efforts that promote scientific collaboration, consultation, and critique. A federal agency is a natural place to engage in that work. The future of the field—and the federal agency that supports it—will depend in no small part on finding new ways to harness the scholarly potential of its diverse perspectives.
DESIGN PRINCIPLE 3
Insulate the Agency from Inappropriate Political Interference
A federal education research agency must be designed to prevent inappropriate political criteria from entering into the agency’s agenda for research, its choice of research studies, its selection of grantees, and its scientific norms. Ensuring that political interference is minimal will foster a scientific culture, protect the scientific process, and prevent research from being sacrificed to the policy passions and practice fads of the day. While we are agnostic about where in the federal government an education research agency should reside, it must have a large degree of independence from partisan politics from both the executive and legislative branches of government.
We want to be clear that buffering the agency from politics in the U.S. system cannot, and should not, be total. However desirable the autonomy of the agency might be from a scientific perspective, its research agenda must be responsive to the needs of decision makers in education. Although research should not be driven only by the needs of the moment—say, school-based management one year, charter schools the next, standards and
accountability the year after—proper attention must be paid to political concerns.
Nonetheless, there are specific kinds of political interference from which a federal education research agency must be insulated. They include micromanagement of decision making, the distortion of the research agenda to be solely short-run, and the use of the agency as a tool to promote a particular policy or position—problems that occur with some frequency across research agencies (Vinovskis, 2000; National Research Council, 1992). To protect the agency from these influences, we suggest that it have independent authority for hiring, disbursal of funds, and publishing. We also urge that agency staff be trusted to make decisions based on their best judgments—informed by frequent interaction with the field—about scientific opportunity. In addition, we believe that the head of the agency should serve a fixed term that spans political administrations. Finally, a consistent fiscal resource commitment would help protect the agency from partisan budget decisions (see Design Principle 5).
Budgetary discretion is a particularly important area in light of the federal government’s funding patterns for previous education research agencies (the National Institute of Education and OERI) over the last few decades. Two trends are noteworthy. First, in the Behavioral and Social Research Program at the National Institute on Aging (NIA), the Child Development and Behavior Branch at the NICHD, and the Social, Behavioral, and Economic Sciences and Education and Human Resources Directorates at NSF, staff had the freedom to develop programs and solicit proposals for a significant proportion of their fiscal 2000 research budgets. By contrast, how OERI’s roughly $130 million research budget will be spent is largely determined by requirements in its current authorizing statute. This legislation requires that at least 25 percent of the annual appropriation for research fund field-initiated studies and at least 15 percent fund research centers.6 Our review of a sample of research agency authorizing statutes showed that no other research agency is subject to such legal requirements
about its use of funds. Furthermore, at our public workshop in March 2001 (National Research Council, 2001d), we heard repeatedly—from three former assistant secretaries, a branch chief at another agency who works with OERI on an interagency research program, and several OERI staff— that a lack of flexibility was a significant problem in OERI. A federal education research agency must have the discretion to invest the bulk of its appropriations in its scientific research agenda to effectively manage evolving research programs. We believe that Congress should have the ability to review the outcomes of the research (see Design Principle 2) and to make appropriations decisions based on performance over time. It should not, however, require that funds be allocated through specific mechanisms or earmark funds for unreviewed projects.
Second, the current federal education research agency, OERI, includes several large nonresearch, service-oriented programs such that its research mission is compromised (National Research Council, 1992). The original intent of including the “I” in OERI (that is, the school improvement function) was to forge a close relationship between cutting-edge research and program funds aimed at improving schools. While in the abstract this idea made good sense, a school improvement agenda can overwhelm the agency’s fiscal and intellectual capacity to focus on its core research mission. While total funding (in 2000 dollars) for OERI has increased nearly tenfold between 1980 and 2000, the percentage of its budget that funds its core research mission fell sharply in the early 1980s and has since remained at roughly 15 percent. The lion’s share of its monies has funded service-oriented programs to states, school districts, and schools to implement “research-based” reform. This trend is also evident at NSF’s Education and Human Resources (EHR) Directorate, which also houses a mix of education reform programs and research. Since 1980, EHR’s budget has risen substantially (from $163 million in fiscal 1980 to $691 million in fiscal 2000 [in 2000 dollars]), but the proportion of its total appropriation that funds research has been meager, ranging from 2.2 to 7.7 percent.7
The trends at these two hybrid organizations strongly suggest that the research function of a federal education research agency should be organizationally separate from an educational improvement mission, leaving the latter to a parallel entity with its own budget. A measure of bureaucratic distance between these two functions is also desirable because it would be difficult to develop a common culture in an education research agency given the appropriate differences between research and program administration.
These potential benefits notwithstanding, it is essential that the research agency forge close links with an improvement entity to foster the integration of research-based insights into the design and implementation of service-oriented education programs at all levels. We leave open the question of what form such an educational improvement organization should take (e.g., regional institutes), but do make some suggestions about the infrastructure needed to better connect research and practice (see Design Principle 6).
DESIGN PRINCIPLE 4
Develop a Focused and Balanced Portfolio of Research That Addresses Short-, Medium-, and Long-Term Issues of Importance to Policy and Practice
Scientific research must focus on the challenges confronting education—increasing achievement for all learners, teaching children science in classrooms where 5 to 15 different languages are spoken, creating opportunities where access to rigorous education has been blocked, and other pressing, difficult problems facing educators. While these needs and pressures will be strong, the agency is sure to fail if it attempts to produce “quick solutions to poorly understood problems” (National Research Council, 1992, p. viii).
A federal education research agency must have the freedom to go beyond the short-term view and make long-term investments in promising lines of research that have reasonable probability of helping to solve important practical problems and generating new and refined theoretical frameworks. It must have the freedom to address topics that may not accord with political opinion or administration policy. More generally, the research agenda must be aligned to reflect an understanding of its develop-
mental phase. This assertion does not suggest that education researchers lower their standards for rigor, but rather that resources and investigations need to be targeted according to a frank assessment of the formative developmental character of the field. Thus, the agency should support a balanced research portfolio with the goal of building theoretical frameworks to undergird the research enterprise and the long-term capacity of research programs to anticipate, as well as address, the pressing needs of policy and practice. Striking the right balance is the key to a successful research portfolio.
In Chapter 2 we show that science-based knowledge accumulates when a field is organized around, and works toward understanding of, a particular question, theory, or problem. With some notable exceptions, the current education research enterprise is highly fragmented (Lagemann, 2000; Vinovskis, 2000; National Research Council, 1992). A new agency must lead the field by focusing scientific research on clearly defined, long-term, strategically focused programs of research with the goal of producing cumulative findings about pressing problems in education (see http://www.rand.org/multi/achievementforall/; National Research Council, 1999d). Moreover, the research portfolio should be use-inspired (Stokes, 1997), including a mix of fundamental science and applied questions; projects with short-, mid-, and long-term horizons; and a variety of research types and methods.
To achieve this balance, we suggest that the agency develop constellations of related research projects, or programs. For example, schools under externally mandated accountability requirements are searching for curricula and teaching methods to boost all students’ science achievement on a variety of outcomes (e.g., tests, course-taking, grades, and other formative measures). A program of research focused on this challenge might support short-term syntheses of what is known, mid-term evaluations of promising programs, and long-term studies of the acquisition and development of science competence.
The development of research programs holds promise for several reasons. First, in areas where there has been sustained support for research (like the example of early reading skills we highlight in Chapter 2), there is a clear progression of education research findings. For example, the Office of Naval Research began funding studies on advanced educational technology
in 1969, and it is only in the last few years—30 years later—that this investment has yielded applicable results (National Research Council, 2001d). Second, establishing programs with a long-term view focuses the agenda and develops an infrastructure for research to progressively move toward scientific consensus.
While long-term research should be an important part of an agency’s programs, that portfolio should also include shorter range challenges. One way of addressing immediate challenges is by summarizing existing research related to a particular topic or problem. Indeed, research syntheses, consensus panels, literature reviews, and other kinds of summary statements about a body of work are important because conflicting evidence across individual studies in education is a major source of frustration among education policy makers and practitioners (see e.g., Sroufe, 1997). Studies that reach different conclusions about the same topic are commonplace, not only in education research, but also in many sciences (e.g., experimental ecology) and many fields (e.g., public health) (National Research Council, 2001d). But this fact does little to placate policy makers and practitioners who often must make decisions or take action on the basis of available information.
Perhaps more importantly, a federal education research agency should systematically conduct research syntheses as part of their program work in order to build scientific consensus by supporting thorough, balanced reviews of research selected from studies that meet scientific criteria. As we describe in Chapter 2, these syntheses provide a mechanism for the accumulation of research-based knowledge. Statistical syntheses (e.g., Glass and Smith, 1978; Hedges and Olkin, 1983; Hedges, Laine, and Greenwald, 1994) provide a means of aggregating across studies. Such syntheses depend on statistical sampling and design quality standards and attend to scientific principles of reproducibility, transparency in standards of evidence, estimation of the role of chance and variation, and the wide availability of findings to invite professional critique (peer review).
At times, however, even statistical syntheses produce conflicting evidence (as was the case with class-size reduction effects; see Chapter 3). Thus, the agency should also support a complementary synthesis method— consensus panels that synthesize bodies of work. Such panels bring together scholars with diverse perspectives and experiences on an issue. They are
charged with amassing the best knowledge and wisdom in an area and using clear standards of evidence for selecting and interpreting studies. Such panels must also adhere to scientific principles of transparency of method, assessing uncertainty, and subjecting findings to the skeptical eye of the broader scientific community. They can be a natural parallel approach with statistical syntheses. Such groups of experts working together can propel science forward by forcing groups of scholars with different perspectives and expertise to confront one another in a healthy, scholarly debate toward the advancement of theoretical, methodological, and empirical understanding.
The extent to which such reviews are conducted in federal research agencies varies considerably. The Child Development and Behavior Branch at NICHD and the Behavioral and Social Research Program at NIA both produce research reviews annually; OERI and the Social, Behavioral, and Economic Sciences and Education and Human Resources Directorates at NSF have no formal mechanism for review and synthesis. We believe that the fact that federal agencies (and foundations) that support education research typically do not view synthesis as their primary responsibility must change if knowledge in education is to grow substantially. Indeed, in the absence of regular efforts to synthesize knowledge, several new entities whose sole aim is to synthesize what is known about a particular topic, problem, or intervention have been created in recent years or are in their planning stages (e.g., Campbell Collaboration [see Box 2-1], Education Quality Institute, and a potential new center of the Pew Charitable Trusts).
Finally, these programs ought to include investments in the scientific study of effective modes of dissemination and implementation of the research. We view the critical issue of research utilization as not only a role for a federal education research agency, but also as an area much in need of sustained scientific study itself.
In addition to developing coherent programs, infusing new, cutting-edge, innovative lines of inquiry into a research portfolio also should be an important function of a federal research agency. To this end, the agency should support a healthy program of field-initiated studies outside of the scope of its pre-defined programs.
DESIGN PRINCIPLE 5
Adequately Fund the Agency
The call for higher levels of funding for education research is hardly new (National Research Council, 2001d, 1999d; President’s Committee of Advisers on Science and Technology, 1997; Shavelson and Berliner, 1988; Vinovskis, 2000; Fuhrman, 2001; Schmidt, 2001; Forgione, 2001). We include it among our design principles for this reason: although we did not conduct an analysis of the quality of federally funded education research, we agree with those who came before us that funding has not historically been aligned with the intended scope of the education research enterprise. Given our assumption that the agenda of a federal education research agency will be roughly comparable to what it has been in the past, coupled with the obvious recommendation that resources be aligned with the scope and expectations of the enterprise, it follows that we recommend increased appropriations to ensure that the agency can adequately meet its mandate.
For background on these recommendations, we briefly review available data and literature related to the federal investment in education research. Unfortunately, no reliable estimates exist of the total investment in education research (Morrill, 1998; National Research Council, 1992). It is difficult even to ascertain how much is invested at the federal level due to problems categorizing research projects across the many agencies and sub-agencies that fund education research. In 1997 the President’s Committee of Advisors on Science and Technology published a report that attempted to sum the federal investment across agencies based on 1995 dollars. It found that less than one-tenth of 1 percent (<0.001) of the total amount the U.S. spent on K-12 education was invested in research. By contrast, it reported that 23 percent of the amount spent on prescription and nonprescription medication was invested in drug development and testing. Similarly, the National Research Council (1999d) concluded that compared to other knowledge-dependent fields, there has been remarkably little invested in the systematic, scientific study of education.
Further evidence of the inadequacy of funding comes from a comparative assessment of research funding in federal agencies. The data we collected from OERI and four similar agencies and organizations helped us gauge (in a rough sense) the ratios of funding level to the scope of the agenda. Comparing the breadth of research agendas alongside annual
funding in this way, we find a stark contrast between OERI and other federal social science research outfits across the federal government. The substantial and long-term investment made by NICHD in early reading research, for example, has reaped a significant return for the agency and the nation. NICHD has invested a total of $100 million over 30 years specifically to better understand phonological awareness and related early reading competencies. It was only through this substantial, sustained investment that, in conjunction with significant funding and intellectual contributions by other federal agencies (e.g., the Office of Special Education Programs in the U.S. Department of Education) and other countries, the research in this relatively focused area has been able to grow.
By contrast, the scope of OERI’s research mandate is sweeping and its funding level modest. Its 1994 reauthorization established five institutes within the agency that roughly sketched its agenda. These five institutes include such broad categories as student achievement, students at risk of educational failure, education policy, early childhood, and postsecondary and life-long learning, and fund research through a range of mechanisms (e.g., centers, field-initiated studies). Each of these institutes spans academic subject areas (e.g., reading, mathematics, science, history) and in many cases educational levels (e.g., student achievement, policy, and at risk all span pre-K through adult learning). This categorization is a reasonable way to parse the field, and we would expect any federal education research agency to cover a similar breadth of content. However, it is unreasonable to expect that robust, research-based knowledge could grow out of them given the fact that roughly $130 million per year (fiscal 2000 level) must cover this broad scope.
Over the course of its history, the primary research agency has had roughly the same agenda but large differences in funding levels. A 1992 National Research Council report charted the precipitous drop in funding for the National Institute of Education (NIE, predecessor agency to OERI) and OERI between 1973 and 1991. In 1973, NIE’s total budget was $136 million ($527.5 million in 2000 constant dollars). By 1991, only $78.4 million ($99.1 million in 2000 constant dollars) of OERI’s budget was allocated to research. This substantial drop in funding occurred with no commensurate change in the scope of its agenda. The report argued that OERI’s limited resources had been spread “so thinly that mediocrity was almost assured. Only a few lines of research have been sustained for
the time they needed to bring them to fruition” (National Research Council, 1992, p. 3). To put this mismatch in dollars into perspective, the Tennessee STAR study (see Box 3-3), a single investigation in a single state that spanned 4 years, cost $10 million over its lifetime.
Although total funding at OERI (adjusted for inflation) has risen substantially, nearly all of the increase has funded service-oriented programs with only tenuous connections to research. Since 1990, there has been a slight rebound in total education research funding, with the fiscal 2000 level at approximately $130 million (including nonresearch activities, OERI’s fiscal budget exceeds $800 million). We view this trend as positive, but believe that given the current breadth of the education research agenda, future increases will be necessary.
In sum, we believe that if a federal education research agency is to have an agenda at least as ambitious in scope as its predecessors’, its funding must be higher than these agencies have had in the past. At the risk of overstating the obvious, we wish to make clear that this is not a call to simply “throw more money at research.” Money alone will not ensure the creation and accumulation of high-quality science-based knowledge in education— or any other field. Increases in funding must be targeted to important problems and attract the best researchers in the country to work on them. Thus, funding should increase as other design principles are institutionalized in a federal education research agency. In particular, steady growth in funding for research should occur as parallel investments are made in human resources in the agency and in the field of education research more globally, the topic we take up next.
DESIGN PRINCIPLE 6
Invest in Research Infrastructure
The infrastructure of any organization is the basic underlying system that determines how it functions to meet its mission. Research infrastructure includes a wide range of supports, but most commonly refers to scientists in the field (people), the tools those scientists have to conduct their work (instrumentation and methods), and the resources those scientists need (time, money, and access to research participants). We believe it is essential for a federal education research agency to consistently invest part of its annual appropriations in infrastructure-building programs.
Specifically, we believe funding is particularly critical in three areas: the education research community; data development, information sharing, and access; and links with the practice and policy communities.
Community of Education Researchers
A federal agency must play a role in nurturing the community of education researchers. The greater the field’s capacity to conduct high-quality scientific research in education and to monitor and maintain high scientific standards, the greater is the likelihood the agency will succeed in its mission. Our focus, consistent with the theme of developing a scientific culture in the agency, is on nurturing scientific norms in the field as a whole.
Historians tracing the field of education research have noted its failure “…to develop a strong, self-regulating professional community” (Lagemann, 2000, p. ix) over a long period of time. We argue throughout this report that the role of the community of scientists in enforcing scientific principles and engaging in professional, skeptical debate about a reasonably well-defined corpus of scientific work is paramount to the success of the enterprise. The complexity of education, and the attendant scope of the research effort, has to date hindered the ability of groups of scholars to form such a community with common intellectual focus.
The organization of programs of research within a federal education research agency (see Design Principle 4 above) would provide a natural springboard for the development of such communities (see http://www.rand.org/multi/achievementforall). The strategic focus of such programs and the standing panels that guide them can provide a common language and set of goals to coalesce groups of peer investigators. In addition, the agency should create incentives for those whom it funds to publish their research and syntheses in peer-reviewed journals.8 Such incentives might include a requirement for progress reports to include evidence of peer-reviewed journal publications, final reports to be in the form of a series of journal articles, and evaluations of new proposals that take into
consideration the publication record of the principal investigator and other key personnel.
A federal education research agency cannot develop and maintain these communities alone. It can leverage its investment in human resources through partnerships with other federal agencies, scholarly professional associations, colleges and universities (especially schools of education), journal publishers, and others. These partnerships could lay the foundation for broad-based efforts aimed at various parts of the system that interact with the education research profession. For example, partnerships with journal publishers and professional associations could lead to the development and monitoring of standards for journal publications and professional meetings. Collaborations with professional associations might feature training and fellowship programs for young scholars (e.g., the Statistics Institute at the annual meeting of the American Educational Research Association [AERA], funded jointly by OERI and the National Center for Education Statistics [NCES], or the AERA Research Grants Program funded by NSF, OERI, NCES, and AERA [Shavelson, 1991] to support dissertation and field-initiated research studies and to place research fellows at NSF and NCES). The agency could also forge links with schools of education, schools of arts and sciences, and other university departments to develop strategies for training and supporting future scientists in education research.
The training of education researchers is a long-term undertaking. As we discuss in Chapter 1, current scholarship in education is generated by investigators trained in schools of education as well as in, for example, psychology, history, economics, sociology, mathematics, biology, and public policy departments. In schools of education, students often pursue nonresearch-oriented goals (e.g., school administration) and may therefore reach the graduate level without any research training. In a related vein, publication standards and peer review also vary considerably in education journals. These complex structural issues will require careful study and innovative approaches to address them effectively.
Data Development, Sharing, and Access
The advancement of scientific knowledge is facilitated when investigators work with the same set of variables and theoretical constructs. Ideally,
the field uses a common set of constructs across research studies to enable replication in different contexts and to better understand the extent to which findings from one study can be extended in other situations. This common core would facilitate understanding of how variables and relationships between variables change over time; if the construct changes, there is no basis for comparison from one time to another. In education, this base has been difficult to establish. As we argue in Chapter 4, there is little consensus about the goals of education, which has presented the community with the challenge of making sense of findings from multiple studies on similar topics but based on different measures. Weak theoretical understanding (see Chapter 5) is another reason why such constructs have not yet been fully developed in education.
A federal education research agency is a logical central place to develop and maintain databases that house these common variables. With the emergence of new technologies for data collection, management, and analysis, such an agency, perhaps in collaboration with an education statistics agency (like the current NCES) and as theory is strengthened, could develop the capacity to maintain data systems on major issues that provide rich information about educational achievement, processes, costs, institutions, policies, and services on an ongoing basis. The system could draw on the extensive resources already available through NCES, the Organisation for Economic Co-operation and Development (OECD), and NSF and develop a system based on a common conceptual frame that links these data in a coherent way.9 For similar reasons, the agency should encourage and facilitate data sharing among its grantees while ensuring privacy and other ethical standards are met.
A key role for a federal education research agency in developing the data infrastructure for scientific education research is by facilitating access to research participants (e.g., students, teachers, administrators, policy makers) and sites (e.g., classrooms, schools, state legislatures). This access is essential to the viability of education research and its potential as a tool for improv-
ing education, but researchers have difficulty gaining access to these sources of data for at least two reasons. First, educational practitioners (especially teachers) typically do not see education research as useful to their day-to-day work (Weiss, 1995). This indifference often means that school officials are unwilling to commit the resources (which is usually a substantial amount of time) required to engage in research efforts. A second reason arises out of federal rules and regulations regarding research ethics. Data access for education research involves legitimate concerns about protecting research participants—particularly young students—from inappropriate actions in the name of research. Protections for human research participants, including participants in education research, have been in effect in the United States since 1974 (now codified in Title 45 Part 46 of the U.S. Code of Federal Regulations). The primary protective mechanism outlined in these federal regulations are institutional review boards (IRBs), oversight groups that review all federally funded research involving human participants to ensure their ethical treatment.
It is important to recognize that education research, including evaluation studies, rarely presents any true risk to the participant so long as care is taken to protect identities and that researchers understand and are responsive to the needs of individual participants. Explicit exemptions outlined in the U.S. code (see Box 6-1) make this clear. Tom Puglisi, the former Director of Human Subject Protections in the federal Office for Human Research Protections, summed up the intent of current law most succinctly by stating that “much social and behavioral research is exempt from the regulations governing research” (Puglisi, 2001, p. 34).
In addition to this core statute, there are at least two other laws (the Family Education Rights and Privacy Act and the Protection of Pupil Rights Amendment) and U.S. Department of Education policy (developed by the agency’s Family Policy Compliance Office) that govern access to education data. In combination, these rules have been variously interpreted and implemented, often creating confusion and erecting unnecessary barriers to conducting scientific research that typically poses “minimal risk” to students. To add to this already maze-like array of statutes, regulations, and policies, a recently passed amendment (Parental Freedom of Information Amendment) to the U.S. House of Representatives version of the pending “No Child Left Behind” legislation, would compound the situa-
tion further. Ironically, this amendment would undermine the $30 million evaluation program proposed in the same bill.
Research ethics is a complex area that the committee did not have the time nor the expertise to consider fully.10 The committee believes that the basic principles that underlie these regulations and govern the ethical conduct of research involving human participants must be upheld; however, we do see bureaucratic problems and inconsistencies in the way these principles have been implemented. A federal education research agency will need to address these issues as a vital part of its investment in building infrastructure. If ethical access to data on students cannot be achieved, scientific progress will be seriously hindered. We suggest that the agency, in collaboration with other federal agencies conducting science-based education research and other interested groups (e.g., social science research associations, research ethicists) invest some of its resources to work toward the dual goals of scientific access to data and protection of individuals. Without ethical access to research participants and sites, the mission of the agency cannot be met.
Links to Practice and Policy Communities
We argue above that the practice and policy communities must be engaged in the work of the agency to develop its research agenda. We also call for regular syntheses of research findings to inform practitioners and policy makers about the cumulative knowledge that scientific education research has generated. Here, we suggest a third connection to practice and policy communities, based on the premise that field-based education research that adheres to scientific principles (see Chapter 3) and attends to the features of education (see Chapter 4) will be significantly strengthened by an infrastructure that bridges the gap between researchers and practitioners.
We wish to be clear that we are not calling for the agency to develop a dissemination network to “translate” research into practice. The transla-
Research activities in which the only involvement of human subjects will be in one or more of the following categories are exempt from this policy:
tion of research findings into practice is not a straightforward affair, and indeed, many have rejected this common metaphor outright (see e.g., Willinsky, 2001). The effect of social science on practice is typically indirect, affecting change incrementally through “knowledge creep” (Weiss, 1980, 1991a, 1999). The scholarly literature on research utilization also suggests that local application of knowledge is a long-term process that involves changes in practitioners’ beliefs, as well as in their procedural skill for implementing the knowledge (Weiss, 1991b, 1999). And how to spark
exempt under paragraph (b)(2) of this section, if: (i) the human subjects are elected or appointed public officials or candidates for public office; or (ii) federal statute(s) require(s) without exception that the confidentiality of the personally identifiable information will be maintained throughout the research and thereafter.
SOURCE: Code of Federal Regulations. Title 45-Public Welfare, Part 46-Protection of Human Subjects, pp. 107-108. Washington, DC: U.S. Government Printing Office.
large-scale change in the U.S. education system—research-based or otherwise—is not well understood (Elmore, 1996).
Two recent reports have drawn on these and related literatures to suggest fundamentally new ways of organizing the education research enterprise. The first, Improving Student Learning: A Strategic Plan for Education Research and Its Utilization (National Research Council, 1999d), makes the case that education research would have a stronger impact on practice if it were supported by an infrastructure that promoted ongoing collaborations
among researchers, practitioners, and policy makers. A second phase of this Strategic Education Research Partnership is currently focused on how to take this idea and build a place—and the enabling strategies, incentives, and infrastructure—to allow these partnerships to flourish. The second, a report of the National Academy of Education (1999), made a similar argument that the prevailing model of research implementation—moving from basic research to development to large-scale implementation of programs—is based on simplistic assumptions about the nature of education and education research. The report concluded that a more productive perspective would view research production and research understanding as part of the same process, also suggesting the need for better partnerships between researchers and educators. Both reports, therefore, simultaneously urge the supply of, and the demand for, education research.
Although the critical issue of research utilization is beyond the scope of the committee’s charge (although we do believe that more research on the topic is very much needed), we focus here on the benefits to scientific inquiry that these collaborative models envision. We suggest that a federal education research agency invest in an infrastructure that builds connections between researchers and practitioners because we see the potential to enhance the research itself. Sustained collaborations between researchers and practitioners could strengthen field-based scientific education research by incrementally infusing a deeper knowledge of the complexities of educational practice into theory building, empirical testing, and methods development in a number of ways. First, situating the research in the messiness of day-to-day educational environments would enable closer attention to context, which we argue is essential to recognize and treat in scientific research. This infrastructure would also establish mutual trust and working relationships that could offer long-term, facilitated access to research participants and sites, and so protect against research being abandoned (as we describe in Chapter 4) when the dynamic conditions surrounding education inevitably shift (e.g., changes in school leadership). Furthermore, strategically and appropriately engaging the knowledge of practitioners’ craft throughout the research process can provide relevant insights that otherwise might be missed.
There are a few examples of such models in practice (e.g., Consortium on Chicago School Reform, http://www.consortium-chicago.org), but this
kind of infrastructure building is fundamentally new. We suggest that an agency support such partnerships carefully and incrementally. There are not only significant structural and cultural barriers to forging these partnerships, but there is also the potential for them to be unproductive. The nature of their work requires practitioners to be driven by immediate crises of the day. These needs could skew the research to be too short-term and tactical in nature to contribute substantially to science-based knowledge. Similarly, there may also be tradeoffs between traditional views of scientific quality and the utility of the work for practice (National Research Council, 2001d). Thus, we urge that the development of these collaborations should include explicit plans for studying their effectiveness and improving them over time.
We believe that clear and consistent focus on translating these design principles into action will promote a strong scientific culture within an agency and strengthen the federal role in education research. For those who know the history of NIE or OERI, many of the principles will strike a familiar chord. For those who don’t, many of them will seem self-evident. However hackneyed or intuitive, we believe they are the crux of the matter. Too often “reform” efforts of the past have focused on changing the existing agency’s organizational structure without adequately grappling with the core issues related to building an infrastructure that supports a scientific community and fosters scientific norms within the agency. Arguably, not since the early days of NIE has the primary agency in the federal government charged with education research had the basic tools to develop a scientific culture and to achieve its mission. Although the details may shift, the principles we propose are intended to stand as guideposts for a federal agency charged with support of scientific education research regardless of the particular situation of the existing federal infrastructure at any given point in time.