Research and Education Reform: Roles for the Office of Educational Research and Improvement (1992)

We have briefly reviewed the need for education reform and the difficulties in achieving it (Chapter 1). We have discussed the various roles of education research and some of its contributions (Chapter 2). We have examined in detail the structure, programs, operations, and funding of the Office of Educational Research and Improvement (OERI) (Chapter 3). Finally, we presented our appraisal of the agency and the environment in which it operates (Chapter 4). In this chapter we first summarize our conclusions and then present our detailed recommendations for OERI.

Education research has been used and useful in far more ways than is commonly thought. Research has contributed new perspectives and conceptualizations of learning, teaching, and schools. It has guided the development of innovative curricula, teaching strategies, special programs, and technologies. Evaluations have assessed the consequences of programs, practices, and policies. And statistics have monitored the status of the nation's schools and warned of impending problems.

Despite the contributions of education research, education reform is difficult to accomplish successfully. Authentic and sustained reform requires the confluence of many elements—appropriate school organization, improved curricula and pedagogy, more public understanding and community support, better teacher education, supportive policies, and the mobilization of adequate resources. In addition, the contributions of research to practice often have not been direct and predictable; rather they have slowly percolated through established policies and practices. Numerous supply-

side and demand-side problems impair the linkages between the research and practice.

In answer to the question—How can the Office of Educational Research and Improvement better contribute to the improvement of schools?—the committee has reached a two-part answer: by strengthening the agency's ability to support and achieve benefits from traditional approaches to R&D and by expanding the agency's functions to encourage establishment of learning communities—collaborative relationships between teachers, administrators, and researchers in the pursuit of new understandings and practices that will improve education. The traditional approaches and this new one should not be pursued separately. The fostering of learning communities will need the insights and innovations of the traditional approaches. In turn, the traditional approaches will function better if informed by the needs, craft knowledge, and inquiries of teachers and administrators.

OERI can foster learning communities through support for research, development, demonstrations, and evaluations aimed at encouraging the needed collaborations; through obtaining diverse input into the planning of research and development; and through wide dissemination of its work. However, OERI cannot singlehandedly encourage learning communities. Leadership and support must also come from those who educate teachers, administrators, and researchers; from those who hire them, manage their work environment, and structure the incentives and rewards for outstanding performance; and from national, state, and local policy makers who establish the broad parameters that affect both.

The mission of OERI is inherently difficult, and the committee does not expect that to change. The disagreements and conflicts over education are endless. Probing the mysteries of human learning is not easy. Linking research with practice remains a challenge. And improving schools is always difficult.

But OERI is also faced with many problems that are not inherent in its mission or responsibilities. If these problems are eliminated or reduced, the agency could be more effective. Frequent changes in leadership have caused organizational instability, false starts, abandoned efforts, and unfulfilled agendas. Having the head of any research agency serve at the will of a high political appointee creates the appearance, if not the reality, of politicization. So does requiring a research agency to submit its reports for clearance by a politically controlled public affairs office. Congressional actions have also weakened OERI. In addition to substantial budget cuts through most of the 1980s, set-asides in the appropriations have almost eliminated field-initiated research, and mandated studies have occasionally been politically skewed.

Fragmentation within OERI, and between it and other federal agencies, has resulted in agenda setting with little benefit from what the others have learned and accomplished. The paucity of sustained research has often

limited the advance of understanding. The paucity of sustained development efforts has resulted in many innovations that are less effective and more expensive than necessary. Inadequate mechanisms for quality control and accumulation of results have forced practitioners and policy makers to wade through large literatures with little guidance as to what is valid, important, and widely applicable. Weak links with teachers, administrators, and policy makers have often limited researchers' knowledge about the realities of schools and public policy making and denied practitioners the benefits of R&D. Inadequate funding has contributed to most of these problems and undermined OERI's capacity to deal effectively with them.

Along with these conclusions, four caveats should be kept in mind when considering our recommendations. First, much that needs to be done about U.S. education is beyond the responsibilities and authority of OERI. OERI's mission is to expand understanding and assist in the improvement of education. It has no authority over teacher education institutions, state education agencies, school boards, district administrators, principals, teachers, or parents. The agency also is tightly constrained in the extent to which it can promote or induce change. The role of OERI (and its predecessor, the National Institute of Education [NIE]) has always been limited to generating new knowledge, developing new techniques and approaches, disseminating information about both, and assisting interested parties to apply the education research and development.

Second, no two or three of the committee's recommendations, by themselves, are likely to make a big difference. The committee has not found an isolated fatal flaw in OERI. Rather, it has found an agency with a very difficult role, severely constrained resources, and a number of organizational and functional weaknesses. OERI needs to be rebuilt, not merely repaired.

Third, even if OERI is rebuilt, it cannot fulfill the unreasonable expectations that have often been placed on education R&D. OERI cannot provide quick answers to long-standing and complex questions. It generally cannot find simple, universal solutions for problems embedded in diverse causes. And it cannot, alone, reform education in the United States. What OERI will be able to do is respond more fully to needs for education R&D, support the sustained work that is necessary for scientific and technological advances, improve quality control, and better assist teachers and administrators in a quest for improved education.

Fourth, unless there is a substantial increase in OERI's budget, no amount of leadership and organizational change can accomplish what needs to be done. Our recommendations would require a large increase in the agency's budget. If such an increase is not provided, the agency's mission and responsibilities should be dramatically narrowed.

Our recommendations are organized into four groups and numbered

accordingly: (A) mission, governance, and agenda; (B) organization and functions; (C) operations; and (D) funding. The committee has made rough estimates of the additional resources needed for most of the recommendations. These estimates are based on the committee's judgments about the intensity of effort that is desirable and its knowledge of the staffing levels used in other federal research agencies to conduct similar activities. Unless otherwise indicated, all estimates are for permanent staff and for annual program expenditures in 1991 constant dollars. The committee has not estimated the increases that will be needed for support services personnel or for the staff and expenses budget, though increases will obviously be needed for both.

The legislative authorization for OERI directs the agency to ''provide leadership in the conduct and support of scientific inquiry into the educational process.'' That mission is accompanied by a substantial list of specific responsibilities. The authorization also outlines the composition and duties of OERI's 15-member advisory council and prescribes a mechanism for generating and publishing priorities for research.

Our first seven recommendations apply to OERI's mission statement, leadership structure, and mechanisms for setting research priorities. They simplify the mission statement, replace the advisory council with a more representative policy-making board, establish a new mechanism for setting priorities, specify the board's responsibilities for monitoring and reporting on the federal system for education R&D, and provide for a balance in various forms of needed research and development.

OERI's current mission statement includes a long list of responsibilities for the agency, which drives the agency to try to be everything to everybody. As a result, OERI has spread its resources so thin that there is little chance of fulfilling any of the responsibilities well. The recommended mission statement does not preclude any of the agency's current responsibilities, but it does direct the agency to focus more narrowly if its budget is not sufficient for credible work in all areas or if the subsequently discussed governance and agenda-setting processes suggest doing so.

We have added to the mission statement a reference to expanding fundamental knowledge and understanding in education. Basic research is critical to progress in research (see Chapter 2), and OERI has a long history of giving it short shrift. The Office of Education, in the late 1960s, and NIE, in the early 1970s, provided substantial support for basic research, but OERI has provided very little funding for it: $1.9 million in 1989 (the last years for which data are available), or 5.5 percent of OERI's R&D budget, a far smaller percentage for basic research than prevails in other major federal research agencies.

We have retained from the current mission statement a reference to promoting excellence and equity in education because both are in keeping with the goals of this country. We have also retained a reference to monitoring the state of education because it is essential for knowing what is actually achieved.

OERI's current mission statement does not explicitly indicate that the agency should work closely with the major groups that are importantly involved in, or affected by, education. Although OERI (and NIE) has not been oblivious to this need, at various times it has operated with less participation from some of these groups than would be optimal. Each of the groups included in our recommended mission statement has unique perspectives and expertise that are essential for reforming U.S. education: researchers apply the theories and methods of various sciences in developing new knowledge; educators and parents are the main agents of a child's education; employers are a key "consumer" of education; policy makers determine the distribution of public resources and set the broad parameters for the conduct of education; and students are both the targets of education and key actors in them.

The perspectives and expertise of all these groups needs to be infused into OERI's work. That will not be easy under any circumstances, but it is a critical challenge that must be met by the agency. Some of the committee's subsequent recommendations should help with the task.

This recommendation would not require changes in OERI's staffing or funding levels.

OERI supports many kinds of education R&D activities. Each makes different contributions, and a mix of them is necessary to fulfill its mandate.

Over the years, the distribution of these activities has skewed away from basic research, field-initiated research, and long-term efforts. This change has been the result of declining budgets, congressionally specified set-asides for specified activities, and decisions of OERI and NIE administrators. Although there is overlap among the three categories of basic research, field-initiated research, and long-term sustained efforts—much basic research is field-initiated, some basic research is part of long-term sustained efforts, and some field-initiated research is part of long-term sustained efforts—each needs to be recognized for its contribution.

In 1989 OERI spent only 5.5 percent of its R&D budget on basic research; in the same year, the Agriculture Research Service spent 46.6 percent of its R&D budget on basic research, NIH spent 59.8 percent, and NSF spent 93.5 percent. Basic research explores the fundamentals of the studied phenomena, generates new views of reality, and proposes new visions of the achievable.

Since 1988 only about 3 percent of OERI's R&D budget has been available for field-initiated research; in comparison, NIH uses about 56 percent and NSF about 94 percent of their budgets, respectively, for field-initiated research. This research permits scholars and others throughout the country to propose topics for research rather than only to respond to announcements prepared by the agency. It also helps train the next generation of researchers by providing assistantships for graduate students throughout the country. Field-initiated research has been a key contributor to scientific and technical advances in all fields.

OERI and NIE have often been criticized for failing to provide sustained support for R&D activities. The actual record is varied, and the criticism is only partly justified, but more sustained efforts are needed. Long-term efforts are needed for advances in basic research as well as for development of new applications. Major advances in the natural sciences and technology applications often have taken a decade or longer.

Without substantially enhanced programs of basic research, field-initiated research, and long-term sustained efforts, OERI will be a feeble partner in the nation's quest for substantial education reform. Research and development require long-term investments.

The staff and budget implications of this recommendation are discussed under Recommendations B-2 and B-3.

Nominees for the directorship should be distinguished researchers, proven managers, and persons with substantial knowledge of U.S. education. The

director would be responsible for implementing the agenda developed by a policy-making board specified in Recommendation A-4. Under the delegated authority of the secretary, the appointee would also guide and manage all the agency's activities.

NIE had six Senate-confirmed directors over a 13-year period, and since it was dissolved into OERI in 1985 there have been three confirmed assistant secretaries of the latter agency. During the same 19-year period, there have been six directors of NSF, five at the National Institutes of Health (NIH), and three at the Agriculture Research Service. The rapid turnover at OERI has been dysfunctional to an agency that needs sustained leadership in planning for, investing in, and supporting the long-term efforts that are required for major scientific and technological advances.

There are several precedents for 4- or 6-year terms of office in federal research agencies, though none of them is perfectly analogous to OERI's situation (see Chapter 4, "Governance"). These arrangements have been used to assure sustained professional management and to minimize the opportunities for politicization. They cannot assure either—appointees are still free to quit and both the President and Congress retain discretion over agency budgets. In addition, there are some potential problems with terms of office for the top administrator. They make the administrator less accountable to the executive branch, and an incompetent could linger for years unless there is a precaution against this, such as specifying removal by the President with the consent of the Senate. Nevertheless, we have specified the longest term of office that can reasonably be supported by precedent. We think the potential benefits outweigh the potential risks. Four-year terms are more common and provide each President with an appointment opportunity. When they have been used there often is a tradition of reappointments. Given OERI's history, we fear that there would be few reappointments and 4 years is a short period in the management of R&D.

It should be noted that a fixed term of office does not give the director unlimited discretionary powers. On the contrary, he or she would still be subject to many of the normal management processes and to the full budgetary process that involves the Secretary of Education, the Office of Management and Budget, the Executive Office, and Congress.

This recommendation would not require additional staffing or funding for OERI.

OERI currently has a National Advisory Council on Educational Research and Improvement. It is composed of 15 members serving 3-year terms, appointed by the President with the advice and consent of the Senate. Congressional legislation requires the council to be "broadly representative" of several groups. For at least the past 3 years, however, the council has apparently had no active education researchers or other social scientists among its members. The council is also widely considered to have little influence.

NIE had a policy board. Under both Republican and Democratic administrations in the 1970s, it was widely considered competent and hard-working, but was unable to help NIE gain the support of educators, the public, or Congress. In the early 1980s the board was considered less distinguished, more politicized, and less effective.

Three options for OERI's future have been analyzed (see Chapter 4, "Governance"). There are potential advantages and disadvantages to policy-making boards, advisory councils, and administration of the agency without either. With OERI's history of controversy, constant charges of politicization, and fragmentation, bringing focus and stability to the agency is a bigger job than any one person is likely to manage. We have chosen a policy-making board because persons of accomplishment are more likely to agree to serve on such a board and because a policy-making board is likely to be more influential. Our analysis indicates, however, that policy boards for federal agencies have little unilateral power—they must win the respect and support of the President and Congress.

The board we propose would differ significantly from the boards of NIE. It would be larger and more diverse, helping to ensure that its members understand the views of the major groups concerned with education. It would be limited to people who have already proven their ability to make important contributions to research on education or to excellence in education, thus ensuring competence and some common understandings during its deliberations. In addition, as indicated in the next recommendation, the board would not set OERI's agenda on the basis of its member's own predilections, but rather would distill priorities from the needs and capabilities of the country after wide consultation with those concerned about education.

The board we propose is modeled closely on the National Science Board of the National Science Foundation. Most observers believe that this board has served its agency well. But there are no assurances that the same structure will work effectively for OERI: the National Science Board has the advantage of making decisions about astronomy, geodesy, and computer engineering, topics about which only a small number of people have any knowledge; OERI's board would be making decisions related to education,

a topic with which almost everyone has prolonged experience and strong opinions. Though there are reasons to be uncertain about the efficacy of the proposed board, the committee thinks it is the best option available for stabilizing and guiding OERI. Its task will not be easy, and its success is not guaranteed, but the alternatives are less promising.

The board members should be appointed by the President, with the advice and counsel of the board, for 6-year staggered terms. For initial appointments to the board, the President should seek advice from professional organizations of those who are involved with education and research on educational issues. The board should be led by a chair, elected by the board members from among their own ranks, for a 2-year renewable term. The OERI board should monitor the health, needs, and accomplishments of OERI's R&D and all federally sponsored education R&D; report periodically to the President and Congress about both; and guide the agenda setting of the agency.

This recommendation will not require additional staff or funding.

The current authorization for OERI requires the Secretary of Education to publish proposed research priorities in the Federal Register every 2 years, allowing 60 days for public comment. This process results in the establishment of a new set of priorities every 2 years, which can interfere with the continuity and stability needed for many education R&D activities.

The committee proposes long-term plans, with a limited biennial update. Development of 5-, 10-, and 15-year national research plans have been standard practice at NIH for many years. It would be desirable that the agenda-setting process begin after the director has served for about 1 year—assuring his or her familiarity with the realities of the agency.

OERI's agenda for education R&D must reflect the priority needs of researchers, teachers, administrators, parents, students, employers, and policy makers. Publication in the Federal Register is far from sufficient outreach to these groups. OERI did hold some meetings before competing the laboratories and centers in 1985 and 1990, but several of its other programs have not had the benefit of such outreach.

The agenda-setting process must also reflect the capabilities of the education R&D enterprise. Unrealistic objectives of quick fixes to complex problems or universal solutions to problems with multiple causes serve only to disappoint researchers and potential users of their work. Without the

integration of needs and capabilities, the productivity, effectiveness, and applicability of the education R&D will suffer.

The board should not impose its own priorities, but rather it should design, oversee, and participate in a process that distills priorities from the needs and capabilities of the country. Preparing priorities that fairly reflect diverse views of all the groups concerned with education will not be easy. Education has been the subject of fierce loyalties and long-standing conflicts. Facts and values frequently clash; needs and capabilities seldom coincide. The priority-setting process will be messy and at times confusing, but only through such a process can there be an agenda that truly reflects the country's needs.

In the context of the previous and following recommendations, this one would require OERI to have about three additional staff persons and $100,000 annually to support the board's priority-setting process.

For most of the history of OERI and NIE, the board's (or council's) report has not portrayed the extent and nature of federal involvement in education R&D nor reflected on the progress and contributions of research and development to the improvement of education. This has limited its ability to identify needs, chart a course for OERI in light of what other agencies are doing, and illustrate the utility of proposed work. This lack has also deprived the agency of institutional memory, hindered researchers' and practitioner's efforts to build on what has been learned by past efforts, and denied policy makers information on the extent of progress.

OERI and other federal agencies involved in education R&D have done a poor job of describing the contributions of their work (see Chapter 4). There is a need to synthesize—succinctly and saliently—what has been learned from education research, how it has extended prior knowledge, the implications for practice and school reform, the development and assistance activities that have used the research, and the effects of those efforts. This information is important not only for OERI, but for researchers, teachers, administrators, other agencies engaged in education R&D, the President, and Congress.

This recommendation directs OERI's policy-making board to prepare a more ambitious report than is currently the case. The resources needed for some of the data collection are discussed in Recommendation A-7. Additional resources will be needed to analyze the collected data, to assemble

information on the accomplishments of federally supported education R&D, and to prepare a written report of that information. This would require about two additional staff members and about $25,000 annually.

There has rarely been comprehensive information on the programmatic activities and funding levels of education R&D throughout the federal government. For two decades NSF has been responsible for monitoring the extent and nature of R&D in the United States: it collects and reports federal R&D expenditures by agency, fields of science and engineering, and budget functions, as well as several other categories, yet none of these data provide even a rough picture of federal support for education R&D.

In addition to the Department of Education, several other federal departments and agencies sponsor education R&D. NSF, the Department of Health and Human Services, and the Department of Defense have long-standing and substantial involvements in this work. Reasonably comprehensive and current information on federal agencies' education R&D activities is needed to identify existing priorities and unmet needs. The committee has not analyzed which agency would be best suited to collect the needed information nor what specific data should be collected. Determination of those matters should be made during consultations among OMB, NSF, FCCSET, and OERI.

This recommendation will probably require one additional staff person at OERI to coordinate preparation for the consultations and follow-up from them, for a period of 1 or 2 years. The agencies that undertake the data collection will also need some increase in staffing and budget, but if the work can be melded into another data collection effort, only a fraction of one person's time and $25,000–$50,000 for additional expenses will probably be needed.

OERI currently has six offices: the Office of the Assistant Secretary; the Office of Research, which administers the 25 R&D centers, the Educational Resources Information System (ERIC) and field-initiated studies; Programs for the Improvement of Practice, which includes the ten regional laboratories, the Program Effectiveness Panel and the National Diffusion Network

(PEP/NDN); the National Center for Education Statistics (NCES); Fund for the Improvement and Reform of Schools and Teaching (FIRST); and Library Programs.

The committee recommends several changes in the organization and functioning of OERI to address major deficiencies in the current organizational structure and provide a mechanism for improved integration of research, development, demonstration, dissemination, and school assistance activities.

The reorganization would result in seven to nine major units: an Office of the Director, three to five R&D directorates, NCES, Library Programs, and a Reform Assistance Directorate (see Figure 5-1). Each R&D directorate would include a program of field-initiated research and institutional mechanisms for sustained and integrated programs of basic research, applied research, and development. The Reform Assistance Directorate would be responsible for supporting reform efforts at the state and local level. This unit would include regional Reform Assistance Laboratories, an expanded PEP/NDN, a modified FIRST Program, and a national electronic network system that would incorporate the current ERIC system. It would also support schools of education, state departments of education, and local districts in efforts to make teachers and principals full partners in the quest for school reform. The ten recommendations in this section detail our proposed organizational structure and many changes in the operations and functions of OERI.

This recommendation is directed at improving three key deficiencies of OERI: fragmentation of its activities; little collaboration between researchers, teachers, and administrators; and the lack of integrated assistance for school reform.

Over time, OERI and NIE have been organized both programmatically (with R&D units for basic skills, equal educational opportunity, and career education) and functionally (such as by research, development, and dissemination). OERI currently is organized partly in each of those ways, with an Office of Research, NCES, Programs for the Improvement of Practice, FIRST, and Library Programs. Some scholars have concluded that prior reorganizations of OERI and NIE had little effect. The committee does not disagree with those conclusions, but believes that the rapidly changing leadership and dwindling resources that accompanied prior reorganizations foredoomed them to failure.

Under the proposed reorganization, much of OERI's research and a substantial portion of its development activities would be organized within three, four, or five R&D directorates. Each directorate would coordinate R&D centers, field-initiated research programs, special studies, and linkages with the Reform Assistance Division.

The committee is not prepared to specify what the feel of the directorates should be, other than they should be enduring and important problem areas, as determined by our recommended agenda-setting process. One possible organizational arrangement might include, four R&D directorates: the social context of education; preparation for schooling; organization and administration of schools; and teaching, learning, and assessment. Another possibility would be to focus the directorates on various subject areas: reading and writing; mathematics and science; and social studies, humanities, and the arts. A third possibility might be three directorates, on preparation for schooling, inner-city schools, and rural schools.

The primary responsibilities of the Reform Assistance Directorate (RAD) would be to provide reform assistance to parents, schools, districts, states, Congress, professional education associations, teacher development institutions, commercial publishers, and employers. It would do so through a broad range of dissemination, liaison, technical assistance, and support activities. This unit would administer OERI's publication activities, the regional laboratories, PEP/NDN, FIRST, and ERIC, with important modifications to all but the first. The regional laboratories would become Reform Assistance Laboratories (see Recommendation B-4); PEP/NDN would be expanded (see Recommendation B-5); the FIRST programs would be altered (see Recommendation B-6); and ERIC would be incorporated in a national electronic information system (see Recommendation B-7).

The RAD would also establish outreach, liaison, and collaboration with parents, professional development institutions, professional education associations, commercial publishers, employers, and policy makers. Each group has important roles to play in the reform of education in the United States. OERI should have at least one professional staff member assigned primarily to each group. Those people would be responsible for bringing the relevant needs and contributions of these groups to the attention of appropriate persons and units in OERI, as well as bringing to the attention of these groups the various resources and contributions of OERI.

NCES would remain a separate unit, operating much as it does currently. There would, however be corrections to the severe staffing shortage (see Recommendation B-8).

Better linkages are needed among all the directorates and NCES than has previously been the case among the offices in OERI (and NIE). Likewise, measures should be taken to improve linkages between OERI and the

institutions it funds and among those institutions. This is an important administrative challenge that must be met by the agency.

The committee believes that this and following organizational recommendations will better coordinate disparate research activities, better focus the efforts on solving major education problems, better apply the expertise of researchers, teachers and administrators to these problems, and improve the technical assistance offered in support of education reform. The goal is to build an R&D system based on a new conceptualization of the relationship between research and practice, so that practice seeks the wisdom of research, and research effectively contributes to practice.

As with any organizational structure, this one has potential disadvantages. One is that evolving problems that do not fit into the established R&D directorates may go unaddressed. A second is that the Reform Assistance Directorate will have to serve, in effect, many masters, which is never easy.

Most aspects of this recommendation are elaborated in subsequent recommendations, and the consequences for OERI's staff and budget are discussed there. Approximately nine additional staff (including clerical help) would be needed in order to assign one staff member for liaison with each of the following group: parents, teacher development institutions, professional associations of educators, commercial publishers, employers, and policy makers.

The agency probably would benefit from the services of an organizational consultant when planning and executing the reorganization. We have not estimated that one-time cost.

OERI invests much less of its R&D budget in field-initiated research than other federal agencies with major research responsibilities. It funded only 12 new field-initiated proposals in each of the last 2 years. The lack of support for field-initiated research deprives the field of the ideas and efforts of many of the best and brightest researchers throughout the country. Those who are not on the winning team for a center have little opportunity for research support from OERI.

Each R&D directorate should operate a substantial program of field-initiated research that is consistent with its focus and coordinated with its other R&D activities. The program would support both basic and applied research, emphasizing sustained efforts. Embedding the field-initiated research in the directorates should focus the work on national priorities, help accumulate findings across studies, and bring findings to bear on practice.

The committee suggests that about 250 new field-initiated studies be funded by OERI each year. They should generally be supported for 3 years, at an average of about $200,000 per year, with opportunities for renewal when further meritorious work is likely. The committee believes OERI would have no problem attracting several hundred promising proposals if it makes substantial funding available for multiyear periods and if it actively solicits proposals from scholars in all relevant disciplines.

This recommendation would require about 22 additional staff members to administer the expanded field-initiated research programs. Since OERI currently has only $1.3 million for field-initiated research, an additional $148.7 million would be required annually, after a 3-year phase-in period.

The centers were created to assemble a critical mass of multidisciplinary investigators to engage in large-scale, long-term, cumulative efforts to improve education. The centers have always varied in the activities undertaken. Some have conducted considerable basic research; others have focused on applied research. Some have done development work; others have not. A few have carried their development work into demonstration and evaluation phases; most have not. Over the past decade, as support for the centers has declined dramatically, their activities have shifted primarily to small-scale 2–4 year studies on related topics. The centers' past history has produced some major contributions, and it is important to build on those successes.

The committee recommends several changes to expand their contributions in the future. First, the centers should undertake considerable more basic research than they currently do. Second, the centers should engage in more sustained efforts of applied research, development, and demonstration, aimed at nurturing new methods, approaches, and tools to full maturity. Such efforts require long-term institutional support. Third, innovative methods, programs, and processes developed by the centers should be subject to a quality assurance process before wide-scale distribution. One possible mechanism is provided through the Program Evaluation Panel (see below). In the past the centers and laboratories have been allowed to engage in widespread dissemination of their innovations without any outside assessment of their merit. The Reform Assistance Directorate would be responsible for nation-wide dissemination of the centers' completed innovative developments, to help ensure that dissemination of the innovations is coordinated with other reform assistance efforts.

Most of the centers now receive less than $1 million in annual support from OERI. The committee cannot imagine a robust R&D center operating

at much less than $3 million annually in core funding. Without that level of support, there will not be the critical mass, diversity of expertise, and scale of operation that are needed to tackle the difficult research problems and development efforts that confront the nation's schools. Some of the proposed R&D directorates in OERI might have only one large center. Others would have several. A center might engage in both research and development or might focus exclusively on one or the other.

The committee is concerned about the current practice of the centers' having to compete every 5 years, regardless of the importance and quality of their work. In several respects the competitions have been a success, but the 5-year cycles are inconsistent with the need for the repeated iterations of research, development, demonstration, and evaluation, which often require a decade or more.

The centers generally did not compete until the 1980s, and there is ample precedent in other federal agencies for negotiated renewals. One option would be to have competitions only every 10 or 15 years. During the interim period, accountability would be achieved by continuous monitoring and feedback, by periodic formal evaluations, and by basing a portion of the federal contribution at each 5-year renewal on the evaluations. A panel of researchers, developers, teachers, and administrators could be assembled in the fourth year to review internal and external evaluations of all the centers and judge their performance. OERI would adjust its contribution to each in response to the judgments; termination would be used only in cases of inadequate performance.

This recommendation would require an increase in the budget of OERI, but no increase in staffing. Assuming OERI has four R&D directorates, each with one to five centers and average funding levels of $4 million per center, the annual cost would be at least $48 million, or $26.4 million more than current funding for centers. (The fiscal year 1991 appropriation provides $20.6 million for centers, and an additional $1.0 million of other money is used to fund one center.)

Like the centers, the laboratories have varied in the activities they have undertaken. A few laboratories have conducted substantial applied research, but most have performed only a limited amount. Some have engaged in ambitious and sustained development efforts, while others have not, and only a few have vigorously conducted evaluations.

Research on change processes and experience with those processes repeatedly has shown that personal contacts are important in facilitating com-

plex change. School reform is seldom achieved merely by the adoption of a new curriculum or teaching approach; rather, it involves a process of organizational development, including needs assessment, goal setting, the adoption of an integrated set of changes, continuous monitoring, and subsequent adaptation.

The laboratories are in a good position, with some modifications, to facilitate school reform. Laboratories have established contacts with most state education agencies, they are well known and respected by many of the school districts in their regions, and most have considerable experience in providing technical assistance. Although much assistance has been provided through the adoption of a discrete curriculum or approach, laboratories have increasingly assisted schools in achieving broad-based changes.

In fostering reforms, the RALs should also strive to help state education agencies, intermediate education agencies, school districts and schools to become learning communities, with the desire and skills needed to incorporate research findings and practitioner wisdom into improvements. Whatever the success in meeting today's reform needs, new challenges will always arise in the future.

The committee foresees at least five modes of operation by which the RALs would foster education reform and the creation of self-sustained learning communities. First, RALs would conduct a wide range of activities: applied research, development, demonstrations, evaluations, dissemination, state policy assistance, and technical assistance for the purposes of facilitating school reform. As with the centers, the RALs' innovative methods, programs, and processes should be subject to a quality assurance review—such as those conducted by the Program Evaluation Panel—before wide-scale distribution. Second, the RALs would assign a few liaison and assistance staff to each state in their respective regions. These staff would apprise state and intermediate service agency officials of the resources available through all of OERI, inform the RALs of the states' needs and interests, and provide some direct services with the support of the full RAL staff. Third, the RALs would scout for exemplary practices developed in the field by teachers and administrators and help them in further refinements, evaluations, and preparation of applications for PEP certification. Fourth, the RALs would provide short- and intermediate-term technical assistance to districts and schools through various of means, including personal visits by RAL staff, training of trainers, telephone assistance, and mailed materials. In this mode they would subsume the responsibilities of the NDN state facilitators. Fifth, the RALs would conduct long-term assistance at a few sites, with a RAL staff member, backed by the full RAL, spending 1 year or more providing intensive technical assistance.

The committee believes that direct services by RALS to the states,

intermediate service agencies, and local school districts should be partially reimbursed, on a sliding scale (adjusted for the ability to pay). Partial reimbursement signifies a partnership between the federal government, the states, and the local districts that is important for the RALs' success. It will help to remind the RAL staff that they must address both local and regional concerns, allow the RALs to provide more services than otherwise possible, and help ensure that the recipients of RAL services make good use of them. The extent of cost sharing might be phased in, starting at a relatively low percentage while the RALs establish a track record that builds up demand for their services.

The committee questions the advisability of competitions for RALs. The laboratories competed in 1985 and 1990, and only one of the 19 incumbents was unseated in those two rounds of competitive bidding. (In 1985 OERI awarded planning grants to 21 institutions to encourage new bidders; no incumbent was unseated.) The competitions require considerable time from both the OERI staff and the laboratory personnel. Competition for the renewals of the laboratory contracts was a reasonable idea, but it resulted in only one new provider and it is not clear that it improved the performance of the incumbents.

An alternative way to try to ensure accountability from RALs would be for OERI to monitor them regularly and provide feedback, periodically evaluate them formally with expert consultants, and base a portion of the federal contribution to each RAL on these reviews. Assuming the RALs operate with 5-year contracts, a panel of researchers, developers, teachers, and administrators could be assembled in the fourth year to review internal and external evaluations of all RALs and judge their performance. OERI would adjust its contribution to each RAL in response to the judgments; termination would be used for significant noncompliance with the contractual terms.

This recommendation would require additions to OERI's staffing and budget. OERI currently assigns a staff member to monitor each laboratory. The monitors devote only about 50 percent of their time to these responsibilities. With state liaison and assistance responsibilities added to the RALS, the monitoring duties would expand, requiring almost full-time effort; this would require about seven additional staff (including clerical support).

The needed increase in budget depends on the number of state liaison and assistance staff and the extent of cost sharing with the state. Assuming one to four RAL staff are assigned to each state, depending on its geographic size and population, and assuming that the states would pay one-third of staff salaries for the first 5 years, the extra salaries and benefits would cost OERI approximately $5 million. If the states provided the office space and basic support services, the cost to OERI for the travel, communications, and other direct expenses would be about $4.5 million. The core functions of

the RALs would probably cost about the same as the laboratories' current activities, and a modest expansion would be possible from the cost-sharing provisions.

PEP certifies the effectiveness of innovative programs and processes. NDN provides the developers of certified programs and processes with grants for dissemination activities and funds state facilitators to assist local schools in choosing and adopting these innovations. PEP and NDN have provided useful services, but those services have been limited (see Chapter 3). PEP's certifications standards are flawed, and NDN does not support refinements and evaluations of promising programs and processes. In addition, NDN has focused on facilitating the adoption of innovations, but the reform of schools requires far more.

The initial PEP certification process has been criticized by some as being too lenient and by others as being too rigid. Innovations that gain initial certification have often been field tested in only a few sites, assessed on only a few outcomes, and judged with no follow-up after termination of the ''treatment.'' PEP makes little effort to assess possible disadvantages of innovations, and lower standards are used for recertification 6 years later. Yet developers find the process expensive, difficult, and time consuming.

The committee concludes that the PEP standards are reasonable for identifying an innovation as "promising," with the exception of their current failure to require reporting of identified program weaknesses. But for recertification, the standards should be more stringent. The evaluation should include a sample of dissemination sites to document broad generality of the innovation. It should require follow-up assessment of effects at least 2 years after completion of the "treatment," to ensure that the effects are not fleeting. And it should be conducted or monitored by PEP, or another competent body that is independent of the developers, to further assure high quality evaluation.

Such evaluations will be expensive and are not likely to be undertaken without new sources of funding. Schools that adopt innovations are often uninterested in externally managed assessments, identifying appropriate control groups can be difficult, and following a mobile student population for several years is never easy. But without high-quality and credible evaluations,

school districts will never be able to choose wisely among available innovations.

Many NDN innovations have been the subject of a limited development process and rushed into dissemination. This is understandable because of the widespread lack of sustained support for education R&D activities, but it is dysfunctional. Given what is well known about the commercialization of technology, many of these programs could benefit from further cycles of research, development, testing, and refinement, aimed at maximizing their impact, expanding their effectiveness in a wide range of settings, improving their potential for dissemination and adoption, and minimizing their costs.

To meet the needs for further R&D and rigorous evaluation of promising innovations, every year NDN would identify five or ten NDN innovations that seem to have the most promise of improving education in the United States. Each would be funded for long-term additional research and development and for evaluation in accordance with the standards proposed above for PEP recertification.

An important part of NDN is the provision of funding to the developers of innovations so that they can disseminate the programs and provide training to adopting schools. These grants now average about $75,000 per year, for a 4-year period, with a good chance of renewal for another 4-year period. This level of funding is substantially less than is needed to make new programs widely known across the country and to support their adoption. In addition, because the original PEP certification is good for 6 years, decisions about NDN renewal are made before recertification, which deprives NDN of a reassessment of the merit of the program or process before making those decisions. The committee concludes that the average dissemination grants should be twice the current annual level and awarded for 5-year periods and that recertification should be required before awarding renewal grants for another 3-year period.

New curricula, teaching approaches, and technology can be important for school reform, but they are seldom sufficient by themselves. Effective reform requires coordinated changes across subject areas and grade levels and in the organization, management, and operations of schools. A few of the innovative processes disseminated by NDN are aimed at facilitating such changes, but most of the disseminated innovations have a much narrower focus. As a consequence, the committee suggests above that the duties of the NDN state facilitators be assumed by the state liaison and assistance staff of the Reform Assistance Laboratories. Because the NDN state facilitators have already generally forged good linkages with schools throughout the state, the RALs might well want to incorporate these people into their new state staffs.

This recommendation will require an increase in the NDN budget and staff. Assuming that seven innovations are selected each year to receive

extra R&D and evaluation assistance and that the support is for 5-year periods with an average of $200,000 per year, after 5 years of phasing in the annual cost would be $7.0 million. Doubling the dissemination grants to developers would require an additional $7 million annually and should be implemented immediately. Elimination of the NDN state facilitators would yield annual savings of $6.4 million, so the net increase would be about $7.6 million annually. Only about two additional staff persons would be needed to administer this program because the staff who currently administer the state facilitator program could be reassigned.

The FIRST office manages a number of individually authorized programs designed to support local school-based reforms that are expected to have national significance (see Chapter 3). The committee finds the goals of these programs commendable, but the means used to achieve them inadequate. FIRST projects have been funded for 1- to 3-year periods and for total amounts generally between $50,000 and $150,000; "reforms of national significance" simply cannot be developed and adequately evaluated under these conditions. In addition, most of the activities supported by these programs are not education R&D. Using the NSF definitions, only $3.5 million of the $46 million budget supports research or research-based development. Neither the legislation nor the administrative guidelines specify that the proposals under these programs make use of relevant research.

The committee does not intend to imply that good reform ideas are derived only from research. We do believe, however, that the federal government should only sponsor the development of ideas that have been tested against applicable bodies of research and use the research to guide the development and refinement phases. The FIRST office's field-initiated improvement programs should exemplify collaboration between research and practice. These programs should be research based and involve teachers and principals in the development process. In turn, funding for these programs should be sustained at levels that are adequate for the tasks. The committee suggests that they be funded for 3- to 6-year periods, at levels of $100,000 to $300,000 per year.

This recommendation would not require additional expenditures because the committee proposes that the current level of funding be used to support fewer projects at higher levels of support. A reduction in staff by one or

two would be possible because of the smaller number of projects, reducing monitoring responsibilities.

Electronic communication technologies offer new opportunities for linking research to practice that should be promoted by OERI. Many of them require only that users have access to a $800 microcomputer, with a modem and a telephone line. A national electronic network could serve at least three purposes: expand access to information and data resources, facilitate input by practitioners into the research process, and foster sharing of resources and expertise among practitioners.

Such a network would allow researchers, teachers, administrators, and parents to have access to the major resources of the federal education research enterprise. Those resources could include not only ERIC, but the electronic card catalog of the Department of Education's Research Library; information on the research, development, dissemination, and technical assistance activities of the laboratories and centers; the applications and other descriptive materials on all NDN programs; the databases of NCES; NSF's forthcoming database on its mathematics and science education projects; and similar resources of other federal agencies involved in education research and development. Computerized "expert systems" might also be developed to provide advice to researchers and practitioners on various matters.

Such a network would also facilitate practitioners' participation in the research enterprise. "E-mail," "file transfers," and "a synchronous computer conferencing'' would allow geographically dispersed teachers to assist in planning studies, reviewing proposals, and discussion of preliminary results—without having to travel. Outstanding teachers and administrators are often reluctant to engage in national activities when school is in session because of the burdens caused by their absences.

Electronic networking would also allow teachers and school administrators to share ideas and feedback among themselves through "electronic bulletin boards." For instance, a bulletin board might be established for each NDN program, allowing users and potential users of the program to post queries, tips, and warnings, and to answer the posted queries.

The Reform Assistance Directorate should foster development of a national electronic network in several ways. NSF's National Research and Education Network, which currently serves universities, and commercial networks should be explored as the telecommunications backbone of the network. Resources that would be of substantial use to researchers, teach-

ers, administrators, and parents should be identified and prepared for access through the network. Standardized information infrastructures, good documentation, and user-friendly "interfaces" should be developed so that even occasional users can have access the network and its resources. Demonstration grants should be provided to schools to purchase needed equipment, train staff in use of the network, and provide assistance to the end users. Library Programs funds should be used for demonstrations in public libraries so that the network resources will be readily available to all concerned with education.

ERIC would be one of the resources available through the network. Several modest enhancements should be made to ERIC even before the proposed network could be operational. ERIC should better coordinate the selection of journals to be indexed and assure that key journals in the social sciences are covered. Most of the major social science journals publish important research on education issues, and some of them are not currently being indexed. ERIC should continue efforts to flag key documents, publicize the presence of flags more widely, and take steps to assure that the flagging process is not politicized. Many users are overwhelmed by the large number of citations they receive in response to a search, and they need help in identifying those that represent the most thorough and objective treatments of the subject. Current efforts to provide full-text coverage of key documents and articles should also be expanded: currently, the results of ERIC searches are citations with brief abstracts, and these are of very limited to use to all except those with access to a university or other large library. Lastly, ERIC's efforts to provide access to the international literature should be augmented so that U.S. scholars and educators can benefit from the research and practice in other countries.

This recommendation will require an increase in the staffing and budget of OERI. Over the next 2 or 3 years, approximately three staff and $1 million will be needed to explore alternative options for the electronic network. The implementation phase will undoubtedly require additional staff and funding. The suggested enhancements to ERIC will require about six additional staff and $1.0 million more annually. The staff positions would allow four coordinators—for journal indexing, the flagging of key documents and articles, full-text initiatives, and international acquisitions and collaborations—and two clerical support persons. The budget increase would permit clearinghouse indexing of all key social science journals, expanded and more careful flagging of key documents and articles, the purchase of rights to the full text of key materials, and the augmented international efforts.

The recent history of NCES is a story of success. In 1986 the National Research Council concluded that the center's operations and products suffered from so many problems that the center should be abolished unless several specified improvements were undertaken. The administration, Congress, the center administrator, and the field all faced the challenge head on. Five years later, NCES is widely perceived as a scientifically responsible agency, responsive to many needs, and an important source of statistics and assessment data.

Recent legislation mandated that NCES be headed by a commissioner appointed by the President, with the advice and consent of the Senate, for a 4-year term. NCES also has complete control over its reports. The committee commends these arrangements, which it believes will permit the center to maintain its independence and integrity.

The committee is gravely concerned, however, about the staffing levels of the center. Over the past few years the budgets and the number of projects have tripled, while staff levels have remained essentially unchanged. NCES now has a much lower rate of staff to program dollars than comparable federal statistics agencies (see Chapter 4). With many new data collection projects mandated in the past 3 years, staff have been focusing most of their time on starting these activities; the monitoring of contracts, verification of data, and preparation of reports have all been delayed. Delays in the release of NCES data and reports have begun, and they are likely to become much more severe in the near future.

The committee has not undertaken a detailed staffing analysis of NCES, but it believes that the available data suggest the need for a large and prompt increase in staffing. The exact number could easily be disputed. Our best guess is that current staff should be about doubled to carry out all new responsibilities added over the past few years. This recommendation would require approximately 138 additional staff positions.

The committee is convinced that widespread school reform will require partnerships between researchers and practitioners. Each has much to contribute to the quest. Researchers can provide breadth and depth of inquiry

and rigor of investigation: elaborate new theories, conduct carefully controlled experiments, study programs and practices in multiple sites, and prepare national indicators of educational progress. Practitioners have an intimate and holistic understanding of the realities of schooling: they accumulate craft wisdom from daily experiences. Among their ranks are exemplars of good practice and effectiveness whose "magic" needs to be understood and conveyed to others. Practitioners are also the ultimate implementers of most reform strategies.

Successful school reform usually depends on involving teachers and principals in defining needs, analyzing options, planning and coordinating changes, and monitoring their implementation. Good teaching is a complex activity that does not lend itself to compulsory or mechanical adoption. New curricula, teaching approaches, and technology can be valuable, but only in the hands of enthusiastic, coordinated, and committed teachers.

There are several important roles OERI can play in assisting local school reform. One role is through support of research and development efforts that will inform school improvement efforts and provide useful tools for them. Another is to ensure that the results are known by those who might use them. These have been the traditional roles pursued by OERI and NIE for the past two decades. There is, however, a need for another role in which OERI encourages and fosters partnerships among researchers, practitioners, and policy makers, where each becomes involved in disciplined inquiry and each contributes to the learning of the others, so that all come to understand education better and be more effective in contributing to its improvement.

OERI could encourage such partnerships in several ways. It could fund new approaches to conveying each group's needs to the other. It could support development of ways to better share the expertise of each group with the other. And it could support innovative collaborations where each group works with the other on their respective responsibilities.

OERI's efforts will have to be supported with leadership at the national, state, and local levels. For instance, school districts will have to provide release-time for teachers; teacher education institutions will have to experiment with substantial changes in their programs; and policy makers and researchers will have to take the time to listen and communicate with practitioners much more effectively than they have in the past.

This recommendation will require more staff and funding for OERI. We suggest a program of R&D and demonstration with perhaps 100 new awards a year, in amounts ranging from $25,000 to $300,000 a year, for 2-to 6-year periods. This program would require approximately $65 million dollars a year (after a 6-year phase-in period). About 15 additional staff would be needed. We propose a higher staff ratio for this program because it involves lending support to teachers, administrators, and researchers in seldom-undertaken activities.

The long-term prospects for any field of research depend critically on long-range capacity building. NSF and NIH maintain vigorous programs for building human resources for science; OERI has rarely supported such efforts.

Some progress can be made in this direction if the committee's other recommendations are implemented. Revitalized programs of R&D at OERI would foster intellectual excitement and provide expanded opportunities. For example, a decade ago there were no AIDS researchers; today there are thousands, including many of the best biomedical researchers in the country. They were attracted by the importance of the problem, the intellectual challenge, and the financial support that became available. When established researchers entered a new field, students followed, and a new research area quickly became the focus of intense investigation.

Scholars from other disciplines who do not see how their work might be relevant to education research must be persuaded that OERI welcomes their insights and input: economists, sociologists, anthropologists, and political scientists have tools and lines of inquiry that are directly applicable to problems in education, as do neurobiologists studying the development and functioning of the brain; and mathematicians, natural scientists, and scholars in the humanities and arts obviously have a wealth of knowledge about the subjects taught in school. OERI needs to actively solicit proposals for research from them, as well as offer dissertation funds for graduate students in the sciences, humanities, and arts who choose to do research on education issues.

Twenty percent of school-age youth are racial or ethnic minorities, and they suffer disproportionately from the severest educational problems. Fourteen percent of U.S. adults are minorities. Only 10 percent of the U.S. members of the American Educational Research Association, who report their major responsibilities to be research, evaluation, or development, are minorities (Patricia Majors, personnel communication, 1992). In addition, only 10 percent of social scientists, and 6 percent of psychologists in the United States are minority (National Science Board, 1989). People hold many different views about the importance of race in the conduct of science and education. This committee believes it is important to have more minorities trained to participate fully in education R&D activities. OERI and NIE used to have modest minority fellowship programs, but none was offered from 1983 to 1990. In 1991 OERI funded a small program at one institution, with $150,000 to support up to 10 graduate student internships.

Both NIH and NSF support programs aimed at developing more minority and women researchers. As women now earn 58 percent of the doctorates in education and 46 percent of the doctorates in the social sciences and now comprise 49 percent of the membership of the American Educational Research Association, the training of racial and ethnic minority researchers should be given priority.

A solid program of dissertation grants for graduate students in all the social sciences would provide about 100 2-year grants of $10,000 per year, requiring about three additional staff and $2 million annually, phased in over 2 years. These grants would be incentives for dissertation research on issues relevant to education and therefore not have to cover the full expenses of graduate students. A viable program of minority fellowships would provide about 100 3-year grants, averaging $23,000 per year, to cover living expenses at the same level provided by major NSF and Department of Education fellowship programs, plus full payment of tuition and fees. This program would require another five staff members and $6.9 million annually, phased in over 3 years.

OERI depends on the Department of Education for hiring and other personnel matters, for procurement, and for report clearance, and OERI has experienced various problems with all three. It has had limited success in recruiting highly qualified personnel. Its proposal review process has commendably involved both researchers and practitioners, but they have been allowed to make decisions in areas outside of their respective expertise. And despite its research and development in areas of considerable dispute, OERI has rarely used a formal process for developing consensus.

The four recommendations in this section address these problems. They would give OERI its own personnel authority, its own contracts and grants office, and sole responsibility for its publications; encourage the agency to recruit top-flight personnel and create an intellectually attractive work environment for them; modify the peer review of proposals for grants and contracts; and have the agency use a consensus development process to determine the implications of important and controversial bodies of research and evaluation. Each of these should help the agency better fulfill its mission.

Many individuals and institutions that have dealt with the Department of Education's contracts and grants office report inordinate delays, a lack of responsiveness to legitimate needs, and incompetence. Several OERI staff have echoed those complaints and voiced similar ones about the Depart-

ment's personnel office. Such complaints are not uncommon at other federal agencies, but the vehemence of the criticism of OERI does appear unusual. In addition, many of the critics have had the benefit of experience with other federal agencies and claim the problems are more severe at the Department of Education. Providing OERI with its own contracts and grants office would not guarantee improved service, but it would put that possibility within the control of the agency.

Similar complaints of delays and nonresponsiveness have been frequently voiced about the Department of Education's personnel office. In addition, independent staffing authority would give OERI its own salary and expenses budget as well as its own personnel office. NCES has had its budget and number of projects tripled over the past 5 years with virtually no increase in its staffing slots. That represents a serious failure of the current salaries and expense budget process. Independent staffing authority will not guarantee avoidance of this in the future, but it will make disparities between staffing levels and agency responsibilities more obvious to those involved in the budget process.

There is at least one disadvantage in giving OERI its own personnel and procurement offices. As a small organization, OERI will not be able to acquire the depth or breadth of expertise that is possible at the departmental level. Indeed, because of this, one knowledgeable administrator in OERI has expressed doubts that the quality of service would be markedly improved. All things considered, however, the committee believes OERI should have its own authority for staffing, contracts, and grants.

All OERI reports, except those of NCES, must now be cleared by each assistant secretary of the Department of Education who has jurisdiction over matters discussed in the report. They also must be cleared by the Editorial Policy Division of the department's Office of Public Affairs. The criteria for approval include "consistency with ED's mission and goals" and "conformity with legislation, regulations, and policy" (see Chapter 3), and there have been many allegations of ideologically motivated delays and modifications, resulting from this review process. The committee has not attempted to verify these allegations, but thinks that the current clearance procedures are inappropriate for a research agency, whose work should be characterized by the highest standards of objectivity.

In place of the Department of Education's clearance procedure, OERI should establish a rigorous review and revision process for all reports that are formally issued under its imprimatur. The report review procedures currently used by NCES might be a good model for the rest of OERI to follow.

Independent staffing authority, contracts and grants authority, and reports authority are common among the larger research agencies, such as the Agriculture Research Service, NSF and NIH (with a few exceptions for

reports) and have been granted to some relatively small federal statistics units. The Bureau of Labor Statistics within the Department of Labor has all three authorities. Recent legislation offered the former two for NCES, but the then Secretary of Education decided against implementation.

This recommendation will require additional staff for OERI, but there could be some offsetting reductions in departmental staff. No budget increase would be needed. The committee cannot estimate the number of staff needed for the personnel office and the contracts and grants office; the Office of Personnel Management should do that. OERI would need about four additional staff for its own rigorous report clearance process. OERI currently issues about 160 reports a year, and staff also prepare many other professional papers and speeches in their official capacities.

OERI can provide effective leadership in education R&D only if it is staffed with high-quality personnel. A cadre of professionals whose qualifications are known and respected is essential to develop a partnership between the agency and the field. NIE recruited many outstanding researchers during its first years, but many left as its budget dwindled, and for years OERI and NIE have held little attraction for top scholars. Although OERI has retained some excellent staff members and recruited some others, many observers within the agency and outside believe it does not have enough.

If this committee's various recommendations are implemented, OERI should have a better chance of attracting the kind of personnel needed. It is particularly important to secure excellence at the top levels—the director and associate directors—both because of the responsibilities of those positions and because such persons serve as magnets for attracting outstanding mid-level professionals.

Another mechanism that would help bring top talent to OERI is to implement a ''rotators" program, bringing top researchers, developers, teachers, and administrators to the agency, for significant periods of time, involving them in the planning and the execution of OERI's work. Such a program would strengthen the agency, make it less parochial, enrich the intellectual environment, and improve OERI's relationships with the fields of research and practice.

There is widespread opinion in OERI, by staff and several managers, that the agency has provided very limited opportunities for staff to engage in professional activities such as attendance at professional meetings and advanced training seminars. This is a serious problem in a research agency. Staff need to be on the cutting edge of new developments when helping to develop the research agendas, soliciting research proposals, evaluating pro-

posals, monitoring the work, and assessing its importance after completion. A perennially tight salaries and expenses budget is blamed for the problem.

This recommendation will not require additional staff positions or program budget. The rotators would fill existing staff slots.

NIE relied primarily on researchers to review proposals. OERI has considerably increased the use of teachers, administrators, policy makers, and others. The inclusion of practitioners as reviewers is commendable—and not unprecedented in federal agencies—but changes should be made in the procedures.

Administrators and teachers are generally not qualified to judge the technical adequacy of proposed research or evaluations. Similarly, researchers are generally not qualified to judge the programmatic promise of proposals, except those in their areas of substantive expertise. OERI has had both types of experts judge both the technical and programmatic merit of proposals, and all their ratings have been averaged. Proposals for basic research should be reviewed only by researchers, because, by definition, basic research is directed at probing fundamental understandings and is not expected to have direct programmatic application. Proposals for applied research, development, and technical assistance should be reviewed by panels with a broad range of expertise. Even then, however, given the nature of OERI's work, we think a minimum of one-third should be researchers—and a higher proportion when the proposed activities primarily involve research.

In addition, the review process should be structured to ensure that only reviewers with technical research expertise make judgments about scientific and technical merit, that only reviewers with the appropriate programmatic expertise make judgments about those matters, and that proposals that are judged to be technically or programmatically inadequate are not recommended for funding.

This recommendation would not affect the staffing or budget needed by OERI.

There are disputes over many education issues, including the interpretation of research results. Teachers, administrators, parents, journalists, and

policy makers repeatedly state they have difficulty determining what is true in a large body of literature with apparently conflicting results. Is bilingual education or English as a second language (ESL) instruction most effective for students with limited English proficiency? For which students and under what conditions? Is the phonics or the whole-language approach superior for teaching beginning reading skills? Superior in what respects and inferior in what respects? Is homogeneous or heterogeneous grouping best—for the slower students and for the faster students? These and other contested issues have been subject to considerable research, but few practitioners know, or could themselves assess, what the weight of evidence says about each.

Although OERI (and NIE) has long sponsored and conducted reviews of research and evaluation literature—some of which have been high-quality scholarly works—reviews differ from the consensus process that we are advocating. Reviews are usually the product of one or two people; a consensus process typically relies on a panel of 10–15 people with diverse expertise and perspectives, who can be expected to raise more questions, apply a wider spectrum of knowledge and skill, and arrive at a broader set of insights. After considering the available evidence, they reach agreements about what is known from the evidence and what is not yet well established and issue a public statement reporting their judgments and rationales. Sometimes they also indicate needed lines of future research. The findings of a consensus development process are likely to be more credible to teachers and administrators than a review because a substantial number of prominent people must agree on the findings.

This recommendation would require some additional staffing and funding. If contracted out, each consensus development process would require about $400,000–$900,000 and perhaps 0.2 staff for liaison and monitoring over 12–24 months. If conducted in house, each would require about 1.5–2.5 full-time staff for 12–24 months and about $100,000 of program expenses. Given all the controversy in education, there could easily be two or three topics a year that would benefit from this scrutiny.

OERI's budget was $380 million dollars in fiscal 1991, of which most was spent on Library Programs and development work that is not based on research. Under the narrow definition of R&D used by NSF and OMB, the agency invested only an estimated $58 million in research and development work. By a broader definition that includes routine statistics collection and dissemination of R&D, the agency spent $126 million. Neither of these amounts is adequate for the mission of OERI and the education needs of the country.

The committee recommends increases in OERI's budget for more basic and applied research, more research-based development, laboratory staff with state liaison responsibilities, a minority fellowship program, more extensive refinement and evaluation of promising innovations, and consensus conferences to reach findings about important and contested bodies of research and evaluation. If that increase is not forthcoming, the mission and activities of the agency should be significantly narrowed.

The funding of OERI (and NIE) declined dramatically over the past two decades while its mission remained largely unchanged. As a consequence, the agency has carried on a broad array of activities with little depth and little chance of making substantial contributions. Over the same period, the need for significant reform of the nation's schools has gained prominence. There is now major public support for the six National Education Goals, but little agreement about how to achieve them. Other needs for research and development have long gone unfilled. Basic research on many education issues is almost nonexistent. Promising innovations have gone without rigorous evaluation. And there has been little effort to help teachers be full partners in the quest for school change.

A substantial increase in the budget and staffing of OERI is required if it is to play an important role in filling these needs and assisting the nation with effective education reform. In trying to determine how much money is needed for a given field of research, three lines of analysis are often brought to bear: a comparison of R&D investment in various fields; computation of likely rates of return on investments; and examination of what activities are important to undertake.

Federal expenditures for education R&D are much lower than federal expenditures for R&D in agriculture, transportation, health, and other major fields, not only in terms of dollars expended but also as a percentage of all federal expenditures in each field and as a percentage of national expenditures in each field (see Chapter 3). By this line of analysis, education R&D is significantly underfunded by the federal government.

In 1986 the Office of Technology Assessment completed a review of the return-on-investment approach for guiding scientific research and concluded that, ''while valid conceptually, [it] does not provide a useful practical guide to improving Federal research decision making (U.S. Office of Technology Assessment, 1986:9)." The applicable return on investment is not dollars, but rather a broad spectrum of social benefits that are difficult to measure and assign dollar values.

The social benefits of improved education include both the personal

satisfactions of a good education and the social contribution of education to "human capital." The productivity of any economy depends on its human capital—the knowledge, skills, and habits that employees bring to their jobs. If education R&D improves education's contributions to human capital, the returns to society will be in the form of increased economic productivity, improved competitiveness in world trade, and a heightened standard of living in the United States. Although it is not possible to apply returns-on-investment analysis to research investments, it is obvious that a small percentage increase in the productivity of the nation's $350 billion education enterprise could yield substantial cost savings, or alternatively, increased outputs in thousands of schools, and the benefits are likely to be reaped for many years.

The third line of analysis for deciding on the level of investment in a field of R&D is to list all the things that are considered important to do and price them. This report has noted many serious problems in the nation's schools for which the nation should be diligently seeking solutions, as well as needs and opportunities for R&D that have resulted from recent work. For example, progress in understanding reading difficulties is incomplete; there is much dissatisfaction with traditional means of measuring student learning, and several proposed alternatives need careful exploration; international comparisons have yielded attention-getting results but require several refinements to be fair gauges; intriguing new teaching approaches are in early stages of development and need much more work; several promising curriculum innovations have not yet been rigorously evaluated; and efforts to restructure schools offer the promise of large and sustained gains, but need further development and evaluation.

Another need for education R&D results from the current ferment in educational policy at both the state and federal level. New ideas are being advanced and implemented with little or no research foundation, such as proposals about school choice, national testing, and student mentors. There is a need for these ideas to be rigorously studied so that their full potential can be realized and assessed. Without well-designed and careful evaluation, new ideas will be judged by press releases and public opinion, rather than on the basis of rigorous evidence.

The committee's prior recommendations are based on the problems and opportunities identified in this report. For most we have estimated the cost and number of additional staff needed. Together our recommendations will require an addition of about $267 million annually for OERI's program budget. About 214 more staff will be needed for expanded program operations, several more for OERI's own personnel and procurement offices, and whatever additional number is required to provide support services for the enlarged staff (accounting and payroll, supplies, mail, computer support,

etc.). Some of the additional staff will have considerable liaison or monitoring responsibilities, which require more travel and communications costs than those for other employees; therefore, the OERI staff and expenses budget will need to be increased somewhat more than would be expected by the number of added staff.

Several of the above recommendations could be implemented within 1 year; others will have to be phased in over 3–6 years. The new governance, agenda setting, and organizational changes should be implemented soon. Funding of the new multiyear grant programs should be phased in over the number of years for which the grants are to be awarded. A large portion of the additional staff should be brought on board as quickly as practical to meet the critical needs of NCES and the forthcoming needs of the reorganization.

The committee recognizes that its funding recommendations would require a large expenditure. Some people will simply dismiss it as too expensive. The committee sees it as a critical investment in the nation's future. Without the investment, and concomitant efforts at state and local levels, the country is not likely to come close to meeting the national education goals.

Over the first 6 years, our recommendations will cost the nation approximately $1.3 billion in additional expenditures. Over the same period, the nation will spend about $1,500 billion on elementary and secondary education in this country. It is clear that this added investment in R&D will be paid back many times over if it improves the effectiveness or efficiency of our education system by even 1 percent. It also should be noted that even with full implementation of all our recommendations, federal investment in education R&D will still be significantly less than federal investment in agriculture, transportation, or health.

Our committee cannot address the future of OERI much beyond the next 5 or 6 years. Recommendations A-4, A-5, and A-6 establish a mechanism for monitoring the health and needs of OERI's R&D and that of the entire federal government. This monitoring should be supplemented with a strategic planning process that anticipates the needs for education R&D 5, 10, and 15 years into the future and develops a strategy for effectively meeting those needs.

It should be noted that any research enterprise is much like an airplane accelerating down a runway. In the early stages, there is much noise and vibration, but the plane just rolls along the tarmac. Only when it has gained enough speed do the aerodynamic forces exceed the gravitational ones, allowing the plane to lift off. The committee believes that its recommendations, if implemented, will accelerate the education R&D enterprise in this country, but lift-off, moving the enterprise into a new mode of functioning, is several years away.

Some of our recommendations require little, if any, additional funding. Given the federal budget difficulties, there will be the temptation to implement those and postpone the others requiring larger expenditures. We believe that would be a serious mistake.

The past and present strategy of sprinkling NIE's and OERI's limited funds across an enormous array of disparate activities has not proven to be an effective national strategy for education R&D. Basic research has almost dried up. The sustained efforts that are so important for progress in science and technology have been limited. Promising innovations are disseminated without the iterations of R&D that are needed to maximize results and minimize costs across diverse situations, and without evaluation of their long-term effects.

Important successes have been achieved, but for every accomplishment of OERI and NIE, teachers, researchers, and policy makers can cite at least ten unfulfilled needs that are fully within the agency's mission. And even if all the problems in organization and functioning identified in this report are remedied, the agency could not possibly fulfill its responsibilities with its current funding levels.

There are at least two alternatives for paring back the breadth of the agency's activities. First, OERI could focus exclusively on dissemination and technical assistance activities. Some people suggest that the biggest bang for the buck, at least in terms of school improvement in the short term, would come from putting the existing research knowledge base into wider use through enhanced dissemination and technical assistance. This approach, however is like eating one's seed corn, and would become self-defeating within a few years.

Second, the agency could invest the entire amount of its limited funds in research and statistics and a few highly targeted development activities, eliminating all dissemination and technical assistance. Several programs of high-quality research can be pursued at much less expense than widespread programs of dissemination and improvement. As these programs bear fruit, there would be increased motivation for dissemination, and perhaps the states or for-profit companies would move to meet the need, although history does not suggest this is very likely.

On several occasions OERI and NIE have tried to concentrate their limited resources, but they have always come under heavy pressure to do the opposite—from researchers of all the excluded fields of inquiry, from curriculum associations of all the excluded subject areas, and from policy makers concerned with all the excluded issues. Only an act of Congress

would be likely to ensure a narrowing of focus. And such legislation would alienate so many participants in education that the travails of the agency might well worsen.

We emphasize that neither of these reductions to OERI's activities is consistent with the widespread, pressing, and diverse needs for education reform. They are not in the best interest of the agency, the federal government, or the nation's education system. Yet it would be more honest to drastically cut OERI's mission than to pretend that it can fulfill that mission without significant increases in funding.

OERI needs to be rebuilt for its complex and difficult mission. The agency has made some valuable contributions, but it has been seriously compromised by governance, organizational, operational, and funding difficulties.

Our study has led us to conclude that the agency can best fulfill its mission by strengthening its traditional roles and by supporting a research enterprise that is a powerful and enduring partner for learning communities—communities comprised of researchers, teachers, administrators, policy makers and parents.

School reform is rarely achieved by mechanically plugging in innovations. Rather it is a developmental and systemic process that requires new learning on the part of both researchers, practitioners, and policy makers.

We recommend major changes in the governance of OERI to improve the agenda setting of the agency, balance its portfolio of work, and bring stability to the top management. Our recommendations for reorganization would expand, focus, and integrate much of the research and development work; strengthen PEP and NDN; correct a critical shortage of staff in NCES; coordinate, extend and supplement the various dissemination and technical assistance efforts; strengthen quality assurance; and initiate efforts to encourage and foster the above mentioned learning communities. Our recommendations about operations would give OERI more control over staffing, contracts, grants and reporting; strengthen its peer review process; and enhance the quality of the staff.

We think these changes will make a difference, an important difference, if they are sustained for a substantial period of time. The story of OERI and its predecessor does not offer much hope for implementation and maintenance of these changes, but conditions are changing. Never before has education received so much attention simultaneously from the administration, Congress, and the governors. Despite the differences of opinion on many issues, there is widespread agreement that business cannot continue as usual in most of the nation's schools, that more than marginal improve-

ments are needed, and that the nation's diverse resources must be mobilized to meet this challenge. Our recommendations for OERI are in full accord with that agreement.

Our recommendations will not be easy to implement, they will not be painless, and they will not yield immediate results. But we think they are a wise long-term investment in the future of the country.