Page 307

10—
Issues Related to the
Research Infrastructure

This chapter departs from the discussion of specific research areas in Chapters 2 through 9 to examine issues related to the infrastructure within which the research is conducted. The following issues are addressed:

Issues about process

 

Agenda setting and the development of Requests for Proposals (RFPs)

 

Review of research proposals

 

Consensus development and accumulation of results

 

Dissemination

Cross-cutting issues

 

Basic versus applied research

 

The funding of research centers versus the funding of field-initiated studies

 

Lack of expertise in the agencies

 

Insufficient or incompetent inclusion of language variables in surveys

 

Need for collaboration and coordination

 

Limited availability of funds

As context for this discussion of infrastructure issues, Appendices A-C present the findings of a comprehensive study designed to consolidate for the first time information on the history, the numerous organizations and programs, and the specific activities that comprise the infrastructure for research on English-language learners and bilingual education. The information gathered in the course of that study served as the basis for the review of infrastructure issues in this



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 307
Page 307 10— Issues Related to the Research Infrastructure This chapter departs from the discussion of specific research areas in Chapters 2 through 9 to examine issues related to the infrastructure within which the research is conducted. The following issues are addressed: • Issues about process   — Agenda setting and the development of Requests for Proposals (RFPs)   — Review of research proposals   — Consensus development and accumulation of results   — Dissemination • Cross-cutting issues   — Basic versus applied research   — The funding of research centers versus the funding of field-initiated studies   — Lack of expertise in the agencies   — Insufficient or incompetent inclusion of language variables in surveys   — Need for collaboration and coordination   — Limited availability of funds As context for this discussion of infrastructure issues, Appendices A-C present the findings of a comprehensive study designed to consolidate for the first time information on the history, the numerous organizations and programs, and the specific activities that comprise the infrastructure for research on English-language learners and bilingual education. The information gathered in the course of that study served as the basis for the review of infrastructure issues in this

OCR for page 307
Page 308 chapter. The reader is referred to Appendix A for an explanation of the study approach, which included extensive review of the literature and various background documents, as well as interviews with key personnel at both the federal and state levels. Earlier chapters of this report have assessed the state of knowledge about the linguistic, cognitive, and social development of English-language learners and about the programs and teachers that educate them and data collected on them; they have also offered observations on the quality of the research. This chapter assesses the infrastructure that produced much of that research and identifies the characteristics that seem to have facilitated or inhibited good research. Our principal judgment, resting largely on the reviews included in previous chapters, is that the infrastructure has often failed to produce the high-quality and relevant research needed, this despite a great expansion of research on LEP issues in the past 15 years and the strenuous and skilled efforts of many researchers and agency officials. The effectiveness of the infrastructure has been strongly influenced by some factors we cannot hope to change, such as the politics of bilingual education. But we can recommend changes in organization, procedures, and allocation of resources that might improve the infrastructure, and changes in training that might strengthen the skills of the people within that infrastructure in the future. The final section of this chapter, then, presents a set of recommendations for addressing the issues listed above, and thereby improving the infrastructure for research on English-language learners and bilingual education. Issues About Process Agenda Setting and the Development of Requests for Proposals RFPs Federal research funds for the study of education have always been very modest and unpredictable. Thus, the possibilities for rational agenda setting are constrained. Agenda setting in education research is always tentative; the major players are always changing; and the process is always vulnerable to interruption, undue haste, politics, and controversy. Even during periods when funding has been fairly level, as with the laboratories and centers, the agenda-setting process has been haphazard, sometimes mandated by Congress, sometimes left to internal agency staff, sometimes involving extensive participation by practitioners and other stakeholders, and sometimes left largely to the discretion of research center directors. Congressional mandates relevant to agenda setting are of two sorts: substantive and procedural. An example of a substantive agenda provided by Congress is the 1978 reauthorization of the Bilingual Education Act, which specified eight areas of research to be conducted by the new Title VII: studies to determine and evaluate effective models for bilingual-bicultural programs; studies to determine language acquisition characteristics and the most effective method of teaching

OCR for page 307
Page 309 English within the context of a bilingual-bicultural program to students who have language proficiencies other than English; a 5-year longitudinal study to measure the effect of this title on the education of students who have language proficiencies other than English; studies to determine the most effective and reliable methods of identification of students who should be entitled to services under this title; the operation of a clearinghouse on information for bilingual education, which would collect, analyze, and disseminate information about bilingual education and related programs; studies to determine the most effective methods of teaching reading to children and adults who have language proficiencies other than English; studies to determine the effectiveness of teacher training preservice and inservice programs funded under this title; and studies to determine the critical cultural characteristics of selected groups of individuals assisted under this title for purposes of teaching about culture in the program. A National Center for Education Statistics (NCES) official reminded us that many of their studies are mandated by law. Of course, when we say that the agenda began with Congress, the question really goes back to who inserted the mandate in the bill and argued it through committees and in some cases the administration. The 1978 research agenda was fashioned by a planning committee from within the Department of Education (Rudolph Troike, personal communication). In the case of evaluation, which is a form of research, an agency's agenda is often shaped by Congress in a piecemeal fashion. Such is the case with the Planning and Evaluation Service (PES) in the Office of the Under Secretary of Education or the evaluation group in the Administration for Children, Youth, and Families in the Department of Health and Human Services (DHHS). Many of their studies are mandated by law or otherwise initiated from outside the agency. When an evaluation unit's mission is to serve a variety of programs, it makes little sense to talk about coherent agenda setting within the unit. PES, for example, evaluates the programs for the Office of the Under Secretary, the Office of Elementary and Secondary Education, and other offices. Often evaluations are specified in legislation; at other times they are requested by the agency that administers a program. In the case of the evaluation group in the Administration for Children, Youth, and Families, their main task is evaluating Head Start, and various aspects of that ongoing evaluation task are specified in law. Agenda setting is thus reactive, although some agencies are by their nature more reactive than others. Yet even these groups strive to bring some coherence to their activities. In the case of PES, one official (Valena Plisko, personal communication) said that having the recently promulgated Department of Education strategic plan has been helpful in setting an evaluation agenda. In the case of the evaluation group in the Administration for Children, Youth, and Families, in 1995 they began to develop a more coherent agenda to link their various studies and build on those done in the past (Michael Lopez, personal communication). One piece that was recommended by an outside panel and not mandated was a

OCR for page 307
Page 310 study of bilingual Head Start programs. In addition, in response to a request by the Head Start Office in the Administration for Children, Youth, and Families, the National Research Council (NRC), under the auspices of the Board on Children, Youth, and Families, convened a series of meetings—the Head Start Roundtable—to provide a systematic analysis of research needs relevant to the changing context Head Start faces as it moves into its fourth decade. The report issued as a result of these meetings (National Research Council, 1996) explicitly addresses issues of ethnic and cultural diversity. However, agencies' internal plans are always vulnerable to interruption as a result of outside demands and internal pressures, so previous blue-ribbon efforts of the Administration for Children, Youth, and Families have not succeeded very well. The second form of Congressional input into agenda setting involves the establishment of required procedures for agenda setting by research agencies that receive funds. One example is the Individuals with Disabilities Education Act, which requires extensive consultation with stakeholders, including adult learners with disabilities and the parents of school students with disabilities. Another example is the recent reauthorization of the Office of Educational Research and Improvement (OERI) in the Goals 2000 legislation, which established the National Educational Research Policy and Priorities Board. This act instructed OERI to coordinate research and required the development of a National Research Priorities Plan for the Department of Education's research efforts. While agendas may be developed internally by research agencies, the word ''agenda" is perhaps too broad here. Kingdon (1984:205) makes a distinction between agenda setting and "alternative specification," a distinction akin to that made by the military between strategies and tactics. The White House and Congress, adjudicating the relative claims of politics, principles, priorities, and public opinion, are more likely to be involved in establishing the larger agenda. In the case of the education of English-language learners, agenda setting could establish the urgency for research on these students and their education, as in the 1970s, or it could raise questions about the appropriateness of various kinds of programming and call for counter-research, as in the 1980s, or it could demote the issue to a more silent priority, as seems to be the case in the 1990s. Sometimes the research agencies have a role in this larger agenda setting. They are often called upon to testify to Congress about their main objectives and what they recommend as major emphases (Graham, cited in Kaestle, 1992). When centers or laboratories are recompeted en masse, the agencies play a key role in proposing a new roster of research concerns to Congress or the administration. When OERI recompeted virtually all of its centers in 1989-1990, its planning process spanned 1988-1989, involving public meetings with researchers, practitioners, and policymakers. Tentative agenda priorities were published for public comment. Then, under the new Assistant Secretary, Christopher Cross, a blue-ribbon panel reconsidered the priorities and confirmed them. RFPs proceeded from this agenda for the set of OERI centers that have just completed their

OCR for page 307
Page 311 work. Similarly, in the recent recompetition for the regional laboratories, extensive public hearings were held, and public comment was sought for the recent recompetition of the centers and for the field-initiated research. In these agenda-setting exercises, one can see the impact of the American tradition of public control in education. In the case of OERI, the client of education research is often seen as the practitioners—the teachers and administrators of schools. In the case of the Office of Special Education and Rehabilitative Services, the client is more often seen as a student with disabilities or the parents of a student. The Individuals with Disabilities Education Act requires broad constituent input into research agendas by such clients. These stakeholders are involved at the early stages, identifying strategic targets for research, according to an official (Louis Danielson, personal communication). This is a different view of the field from that which prevails in scientific agencies, which in thinking of the field think of the researchers themselves, not the clients of the research (i.e., practitioners or students). Once the broad agenda has been established, the initiative usually passes to the research agencies, in consultation with researchers in the field, to determine what sorts of studies and what specific studies should be done—what Kingdon (1984) calls the specification of alternatives. These detailed agendas can be done well or poorly, depending on the mix of procedures and personnel that constitute the infrastructure. At the Office of Bilingual Education and Minority Languages Affairs (OBEMLA) and other agencies, the failure to maintain an active relationship with the field, for example through the use of standing panels, is exacerbated by the lack of qualified researchers on the agency staff, sometimes resulting in poorly drafted RFPs and Grant Announcements. In cases where an agency, with the support of Congress, trusts the researchers in the field to initiate useful research without a great deal of specification from the government, agenda setting is of less importance. The National Institutes of Health (NIH) provides a striking contrast to education research agencies in this regard. In the National Institute of Child Health and Human Development, for example, about 90 percent of the research funds are devoted to field-initiated research. There are occasional Congressional mandates and targeted requests for applications on particular topics, but for the most part the agenda is left open and is determined, in effect, by the aggregate of projects successfully proposed from the field. Education research is in a double bind. There has been little faith (and little money) in field-initiated studies, so an agenda is needed to guide research efforts; yet the infrastructure is unstable and ineffective in building agendas. The task of creating an overarching agenda for research in the Department of Education has now fallen to OERI and the new National Educational Research Policy and Priorities Board. Time will tell how successful that effort will be, but given the instability and fragmentation that have characterized the past, something of this nature is needed. At lower levels of specification, the record has been mixed. Both insiders

OCR for page 307
Page 312 and outsiders believe that OBEMLA's agenda-setting attempts have been weakened by politicization, lack of leadership, turnover of key personnel, and lack of a conceptual plan. As for the specification of alternatives in setting agendas for research centers devoted to LEP issues, the record is also mixed. In the opinion of Amado Padilla (personal communication), codirector of the winning proposal for the second OERI language center in 1985, the RFP was quite good. In response to the RFP, Padilla says, his planning group started brainstorming, trying to match up existing researchers and ideas with the research and development specified in the RFP. This is typical of the specification of alternatives by prospective OERI research center planning groups. They try to balance the demands of the RFP with their view of the field and its problems, plus their judgments about who does good research, plus pressures to get the work out in a relatively short time and to recognize various constituencies with an interest in the domain being studied. In the case of the third OERI language center, the National Center for Research on Cultural Diversity and Second Language Learning, its codirector believes the coherence of its proposal came from ideas that emerged in the group's proposal planning sessions, notably a Vygotskyan psychological bent and an interest in ethnographic studies of multilingual situations (Barry McLaughlin, personal communication). The group wanted to get some finished work out rapidly, according to McLaughlin, and in response to the grant announcement felt a need to include as many language groups and geographic areas as possible, a complication of working in the area of language diversity. Also, there was a great deal of emphasis on the word "national" in the center's title. They did not have the latitude to select a few things to do well; rather, they felt they were expected to conduct research representative of the whole field of practice in the education of English-language learners. As a result, they included some research already in progress, plus a relatively large number of new, small projects, resulting in a more diverse repertoire of small-scale studies than might have resulted had they felt neither of the above pressures. Research on English-language learners and their education is an extreme case of the problems faced by education research at the federal level, and education research is an extreme case of the problems faced by most federal research agencies. In education, and on LEP issues in particular, the procedures and agencies are unstable, the funds are sparse, the agency personnel are often untrained in research, and the topics are controversial. Thus, agenda-setting efforts are ad hoc, reactive, fragmented, and political. The need is for commitment to a more stable, longer-term agenda that will survive more than the usual 2 years served by agency heads. To effect such agenda development, responsibility needs to be located at a high level within the Department of Education; moreover, participants need to distinguish among different levels of agenda setting—the basic program versus the specification of alternatives versus the selection of

OCR for page 307
Page 313 research projects—and then determine which stakeholders are appropriately included in each. Presumably, practitioners and politicians should have more input at the more general level of establishing overall goals for research, appropriate to the needs of the field as well as influenced by the high-pay-off areas where research is likely to make significant progress. Researchers and agency officials with deep research knowledge should have more to say about the specification of alternatives and should have exclusive discretion to judge the technical merits of individual projects. At the same time, despite the appropriateness of having different players at different levels of the agenda-setting process, the whole process should have coherence. There should be enough feedback and accountability so that the research findings relate in helpful ways to the larger agenda, and the agencies providing the funds should have an ongoing, affirmative responsibility to monitor the whole enterprise, relating research findings to the larger mission of the agenda. Review of Research Proposals Procedures for the review of proposals for research projects funded by the federal government differ considerably depending on the agency, on what kind of research is involved, and on whether the proposal is individually submitted or part of a center's work. Agencies that fund work largely by contract, such as PES, rely on internal staff reviews. This seems to stem from various considerations. First, there are not very many competitors for large-scale contract evaluation research, so it might be difficult to find knowledgeable outside reviewers without a conflict of interest (John Chapman, personal communication). Also, contracts are tied more closely than grants to the specifications laid down by the agency; there is less emphasis on creativity and more on technical capacity. All of these research-related activities, however, may soon be subject to new routines and standards. The National Educational Research Policy and Priorities Board is currently overseeing the development of standards for the evaluation and conduct of activities carried out by OERI, including the review and selection of proposals and the monitoring of grants awarded. The classic process of external peer review, often cited as an enviable model by leading education scholars, is exemplified by NIH and the National Science Foundation (NSF). When budgets are sizable and much of the research is initiated by individual proposals from the field, as in the cases of NIH and NSF, agencies develop the capacity to maintain two features missing in Department of Education research agencies: standing panels of experts from the field and research administrators in the agency with substantial research expertise. We return to the latter in the discussion below on expertise. As for the standing panels, National Institute for Mental Health panel members have terms of 3 to 4 years, and they meet three times a year to review proposals. Their proposal rankings are expressed in a ranking system, on the basis of scientific merit. There

OCR for page 307
Page 314 is also a national advisory council that includes both scientists and citizen members, but our source at the institute told us that the "role of the council differs across institutes," and at the National Institute for Mental Health, the council generally supports panel judgments of merit (Mary Ellen Oliveri, personal communication). At the National Institute of Child Health and Human Development, the research administrator we interviewed said the peer review system is "very robust" (Norman Krasnegor, personal communication). A "highly educated and technically sophisticated group of scientists" comes to Washington three times a year to review applications. Their national advisory council looks at applications to examine policy implications or to handle appeals. At NSF, all research grants of over $50,000 must be subjected to peer review. Some of this is done by mail, by ad hoc reviewers; some of it is done by standing panels, as at NIH. In the Department of Education agencies that do substantial amounts of research on language issues—OERI and OBEMLA—there is no tradition of standing panels. According to some department staff (Edward Fuentes and Joseph Conaty, personal communication), competitions of the same type are infrequent in the Department of Education, and thus standing panels might not work as well as at NIH and NSF. Sometimes there has been thorough peer review that has been well regarded in the field; often the review process has gotten worse marks (Kaestle, 1992). The suspicion by some that education is a weak field in which things do not get done well cannot be adjudicated since so many other adverse conditions prevail: too little budget, too much leadership turnover, and too little proportion of the budget in field-initiated research, plus the fragmentation of competing research paradigms. With the new National Educational Research Policy and Priorities Board, the new research institutes, and a commitment to spending 20 percent of future funds on field-initiated research, OERI may have the opportunity to build more of a tradition of standing panels. At the Office of Special Education and Rehabilitative Services, also in the Department of Education, the research funds have been (until now) somewhat more stable; there all reviews are done by ad hoc panels of experts familiar with the particular population or disability being researched, and until now, all of these panels have been brought to Washington for face-to-face meetings. But the vulnerability of these arrangements, as with education researchers' hopes for the new OERI institutes, is abundantly clear in the present political climate. Since most research in education, including that on LEP issues, is conducted in research centers, the processes by which projects are chosen within centers are of considerable interest; however, they are more loosely governed and variable than the processes for judging the merit of field-initiated proposals or proposals for initial funding of the centers themselves. Of course, the initial roster of projects accompanies the proposal for establishing the center, and those proposals receive intense scrutiny from the agencies and from peer reviewers. These

OCR for page 307
Page 315 reviews are watched intensely by researchers in the field, who occasionally protest decisions and question the fairness of the selection process. This process, however, chooses among the centers proposed; it does not in general inquire about the possible alternatives to individual projects. In any case, the process by which researchers are included in the center proposals is not an open one. It is a matter of planning groups attempting to respond to RFPs and Grant Announcements, judging what researchers are doing good work relevant to the center's mission and who is available. Often these decisions are influenced by geographical considerations and networks of acquaintances. No doubt this often results in excellent work by groups of high competence, but the process by which the funding reaches individual researchers is strikingly different from that for peer-reviewed field-initiated proposals. Consensus Development and the Accumulation of Results It is widely charged that education research seldom adds up to much, that it is too equivocal to inform practice. Some people argue that in social science research, results are necessarily "messier" than in the physical and biological sciences. Nonetheless, education researchers have, for the most part, not done a good job of accumulating evidence and building upon past research. Emerson Elliott (cited in Kaestle, 1992:15), the recently retired Commissioner of Statistics at NCES, came to the National Institute of Education (NIE) in 1972 after working on health issues at the Office of Management and Budget. At the health institutes, he said, "there was a strong sense that there was science, and that it was cumulating to something." Education research was "discredited'' by the lack of such a conviction. One center director said that when he sent off his final report to NIE, "I don't think they even opened the boxes" (Amado Padilla, personal communication). This situation has not universally characterized LEP-related research. During 1990-1992, OBEMLA funded three symposia. The first, in September 1990, focused on topics including demographics, issues of method and pedagogy, language teaching and learning, early childhood education issues, assessment, and LEP exceptional issues (see OBEMLA, 1990). The second symposium (September 1991) addressed evaluation and measurement issues (see OBEMLA, 1992). The third (August 1992) addressed middle and high school issues (see OBEMLA, 1993). Compendia of the research papers were published and widely distributed. But instances of such synthesizing activities are outweighed in the historical record by complaints of inattention to results. There are two related problems: one is whether the agencies do anything with the research they have funded (read it, understand it, critique it, synthesize it, disseminate it); the second is whether researchers in the field have a sense of evidence being amassed, of new directions and questions coming from completed research, and of relatively secure knowledge accumulating. No doubt there is some of this cumulative process in education

OCR for page 307
Page 316 research, including work on LEP issues. Nonetheless, the frequent complaints about ignored reports and lack of synthesis are symptomatic of the weak infrastructure of education research in general and of research on English-language learners in particular. The politicization of the issues and rapid turnover of leadership in the research agencies exacerbate the problem of building a cumulative knowledge base. Key figures in the agencies and the centers are very aware of this problem, and there has been much discussion of it in the past few years. The Center for Research on the Education of Children Placed at Risk (CRESPAR) holds an annual symposium at the American Association for Education Research to take stock of results, and they periodically produce a volume of papers that reviews research from their center and elsewhere on a particular topic. In addition, they began a peer-reviewed journal in 1996 called the Journal of Education for Students Placed at Risk, which is now published by Lawrence Erlbaum Associates (Robert Slavin, personal communication). And Judith Anderson (personal communication), former acting director of the At-Risk Institute, which funds CRESPAR, says they are developing institute-wide guidelines for the synthesis and dissemination of research. The Administration for Children, Youth, and Families is conducting a review of past work in preparation for the production of a more coherent research agenda for the future (Michael Lopez, personal communication). At NCES, Edith McArthur (personal communication) reports, they have branched out from their now traditional annual reports, The Condition of Education and The Digest of Educational Statistics, to include special reports on focused topics such as urban youth, the education of Hispanic students, and high school dropouts. There is also a set of reports to various audiences for each large data set they produce. Eugene Garcia (personal communication), until recently the head of OBEMLA, reminded our interviewer that the Department of Education cosponsored the NRC effort that produced this report, which itself was a form of literature review, consensus development, and agenda building. The successful proposal for the new Southwest Educational Development Laboratory displays the capacity of the current research infrastructure to inform new projects of past results. It proposed convening experts in the field before designing applied research on bilingual programs. It also proposed keeping abreast of research and program developments through journals, conferences, and electronic communication, looking to research agencies such as the National Center for Research on Cultural Diversity and Second Language Learning (now the Center for Research on Education, Diversity, and Excellence) and the Center for Applied Linguistics as sources of new knowledge and synthesis of ongoing research (Southwest Educational Development Laboratory, 1995). While the above efforts can yield an ongoing, informal sense of what has been learned and what research is needed, there are more formal procedures for exploring and stating consensus in complex areas of research. The procedures often cited come from NIH. Our source at the National Institute of Child Health

OCR for page 307
Page 317 and Human Development explained that there is a special office in NIH responsible for conferences on consensus formation. It is "a very elaborate process that takes several years" (Norman Krasnegor, personal communication). Experts confer, examining the literature in a given area and exploring and debating the levels of certainty about research findings according to an established procedure. Both Christopher Cross (cited in Kaestle, 1992), as Assistant Secretary of OERI, and Joseph Conaty (personal communication), as Director of Research in OERI, have recommended the NIH model for education research. Conaty commissioned some papers in preparation for a budget request to pursue consensus development on the health institutes model. However, as he said, "if you have no discretionary money, you can't do consensus panels," and he never got funding specifically for the purpose, despite repeated requests. Dissemination If there are two issues that make education researchers and research administrators grimace, it is coordination of research efforts and dissemination of results—not because they do not want to do these things, but because no one seems to have clear answers about effective ways of doing them. We speak to the coordination issue below; here we look at the dissemination issue. The issue of dissemination will not go away. The old linear model of research and development—some people do research, others develop materials from it, and others distribute it and train practitioners how to use it—is regularly criticized. Over the past two decades, pressures have mounted to involve practitioners in the agenda setting and conduct of research and to have researchers involved in thinking about links to practice from the start of their work. For better or worse (worse, we think), governments support very little basic research on language-minority issues and bilingualism. Most research, therefore, is intended from the start to reach some conclusions directly relevant to policy and practice—whether case studies of best practices or statistical studies of large programs, evaluation work, or data gathering about relevant populations and their experiences. Making the links takes more than good intentions and more than traditional dissemination modes; it takes imagination, high priority, and resources. Imagine, for example, how many education research agencies would have the resources to adopt the approach of the health institutes. Mary Ellen Oliveri of the National Institute of Mental Health (personal communication) reports that the health institutes concentrate much more on basic research than do the education agencies, so their dissemination efforts are generally through traditional academic venues—journals and conferences. But when special reports are needed for policy purposes, they typically are based on the deliberations of large numbers of outside scientists (Mary Ellen Oliveri, personal communication). Similarly, Norman Krasnegor of the National Institute of Child Health and Human Development

OCR for page 307
Page 332 of OERI in 1994, and some of these concerns are being addressed for the department as a whole by the National Educational Research Policy and Priorities Board, which has developed and approved standards for peer review. Several principles are important for quality control across all topics of education research. Scholars from outside the department should participate in the review of proposals; in the building of consensus regarding what is known and what areas of investigation are most promising; in determining what combination of methodologies is best suited to the tasks defined; and in the developing Requests for Proposals, particularly ensuring that they set clear criteria for quality. In addition, some aspects of peer review relate specifically to research on LEP issues. First, the need to include English-language learners in research whenever doing so would affect the scientific quality of the inferences drawn implies the value of having an expert on LEP issues on any peer review committee dealing with large-scale research on students in general. Second, the need to investigate English-language learners within a larger theoretical framework implies the value of including expert researchers on the larger contextual and conceptual issues (for example, literacy development or learning in content areas), along with those on English-language learners per se. Research agencies in the Department of Education have infrequently used standing panels. More often they have used ad hoc raters, sometimes with face-to-face meetings and sometimes not. There are exceptions; NCES has standing panels for its survey projects. But the practice has been spotty over the years in the department. The result has been less expertise to support decision making in the agencies, less-supportive relationships from the research community, and less contact between agency specialists and researchers on a collegial basis. Some researchers in the LEP area believe that some poor-quality research has resulted from ineffective peer review. They believe there have been problems with the composition of review panels, with respect to both the mix of department staff and outside experts and the mix of researchers and nonresearchers. In addition, some have suggested that funding of projects through centers sometimes shields poor-quality projects from rigorous review. Inclusion of Language-Minority and LEP Variables and Expansion of Language Concerns in Agency Research Programs 10-4. Coordinating committees should encourage research agencies to include LEP variables in data gathering and to conduct research focused specifically on English-language learners. The two groups recommended above—the standing advisory committee and the informal discussion group—should encourage increased attention to language issues in surveys and other data gathering. They should also encourage increased attention to language issues as a substantive focus in research, not only that sponsored by the department, but also that sponsored by agencies outside of the department that deal with

OCR for page 307
Page 333 childhood and education. In doing so, they should distinguish among three different and desirable forms of inclusion: • The incorporation of English-language learners into studies from which they are now excluded in order to obtain better population estimates. Adaptations of assessment instruments and procedures might be required to ensure inclusion. • The disaggregation of data by LEP status, where possible and appropriate, in reporting and analyzing the data. This might help in understanding English-language learners in particular or in illuminating the status and experiences of students generally in some way. Oversampling might be necessary for such disaggregation. • In funding of education research more generally, a requirement in Requests for Proposals to include English-language learners as subjects in research on a wide variety of topics where the language dimensions may earlier have been ignored. Perhaps necessary at some point would be a systematic inquiry by Congress into the extent of exclusion of English-language learners from federally funded research, followed by Congressional action if the situation should warrant. This action might include incentives for more work in this area. For some years there has been an attitude, reflected in the research infrastructure on English-language learner and bilingual education concerns, that agencies would leave those concerns to OBEMLA. One reason for this reluctance to initiate research and data gathering on English-language learners is that the inclusion of complex language variables in studies is expensive and difficult. Another is that the subject matter is politically controversial. Thus OBEMLA long had the role of persuading other agencies to incorporate language variables in their survey work and of providing the funds for doing so from the Title VII budget. This helped raise the visibility of language issues, but was not a good long-term solution. Too often the language component was added late in the process or with too little expertise, so the items added were not defined in useful ways. The demography of language diversity in this country suggests that students' language abilities and histories will continue to be important variables for us to understand in studying education, and as noted above, studies that exclude part of the population from participation because of language are skewed scientifically. Therefore, it is important for agencies to develop the expertise and the incentives needed to include language variables routinely and competently in their education research. It is also true that agencies outside of the Department of Education, such as the National Science Foundation and the Administration for Children, Youth, and Families, that deal with education issues seldom focus on the language diversity of students. It is important that they be persuaded to give more agenda priority to such issues.

OCR for page 307
Page 334 Integration of the Work of the Regional Laboratories into the Research Program 10-5. The above-recommended Advisory Committee on Research on English-language Learners should ensure that staff from the three new regional laboratories specializing in language diversity participate in the department's research coordination activities. The directors of these laboratories should work with each other, with the above-recommended advisory committee, and with the above-recommended informal discussion group to ensure regular communication and collaboration, from the setting of agendas to the synthesis and dissemination of research. Historically, the regional laboratories have operated more or less independently of the research and development centers. This has led to recurring complaints about fuzziness in the differentiation of the missions of the two groups. Two features of the present situation deepen our concern for the importance of integrating the work of the laboratories into an overarching agenda and collaborative network. First, three of the newly contracted regional laboratories have specialized missions involving English-language learner and multicultural issues, so there is a new opportunity for expanded and coordinated research work in these areas. Second, however, the laboratories are supervised by the Office of Reform Assistance and Dissemination (ORAD), not by the same institute that supervises the new Center for Meeting the Educational Needs of a Diverse Student Population. Substantive Research Expertise Within the Agencies 10-6. Agencies in the Department of Education that have substantial responsibility for research on language-minority and LEP issues, such as OBEMLA, PES, and OERI (including the institutes, NCES, and ORAD), should allocate resources to train current staff and recruit staff with solid research experience so that there is substantive research expertise on English-language learners within the agencies. Agencies with incidental but important contact with such issues should find means to get the consultative expertise they need in a timely fashion. 10-7. The same key agencies should budget and implement internal senior research fellowships for scholars with expertise in LEP issues for periods of 6 to 12 months. These scholars would be involved in the ongoing research funding issues of the agency while engaging in some research of their own. As noted earlier, many researchers and agency personnel bemoan the lack of research specialists within the agencies, both in general and specifically with regard to language-minority and LEP issues. We emphasize the OBEMLA's research staff and procedures need to be strengthened if it is to play a role in research management. There is currently little faith in the office's research

OCR for page 307
Page 335 capacity, a judgment that the committee unhappily shares. If the institute structure of OERI is to thrive, the institute staff must have more depth in the areas each institute covers. Typically, one person is given all responsibilities regarding English-language learners for a given office. Offices need more staff capacity to address complex language issues. One way to accomplish this would be through the employment of excepted personnel. Another would be through the Society for Research in Child Development Executive Branch Fellowships program. A third would be through training of existing staff on LEP issues, or conversely, training of OBEMLA staff on research issues. Accumulation and Dissemination of Research and Data 10-8. The National Educational Research Policy and Priorities Board should charge the new Department of Education Advisory Committee on Research on English-language Learners with the development of a comprehensive system for integrating the review and synthesis of new knowledge into the dissemination and agenda-setting processes for LEP research. The committee could, for example, hold periodic meetings to assess the state of knowledge; it could adopt and sponsor consensus exercises such as those employed by the National Institutes of Health; and it could hold annual research symposia, as OBEMLA has done in the past. Through the board, the committee could consult the educators who use research results to develop priorities for the further accumulation of needed knowledge. The committee might also consider supporting the establishment of one or more additional juried research journals, with attention to achieving the most neutral or catholic stance on methodological and policy issues. But most important, the committee, working under the board, must be the locus of a coherent process of knowledge accumulation, from the genesis of research in agenda setting to the dual problematic processes arising from the conduct of good research: developing consensus on new knowledge and relating is to practice. Research is cumulative enterprise that depends on a tradition giving impetus to new studies. The usual process by which new knowledge is reviewed and archived is the publication of research findings in peer-reviewed journals; but in the field of language-minority and LEP issues, the political nature of the field has distorted even that process. The Bilingual Research Journal, published by the National Association for Bilingual Education, and READ Perspectives, published by Research in English Acquisition and Development, Inc., both maintain editorial review boards of credible researchers. Yet each is eyed with suspicion by the other political camp, and many serious scholars are discouraged from submitting their work to such publications. Potential contributors may believe that judgments on their work depend on political orientation, as well as on the disciplinary orientation of the reviewers.

OCR for page 307
Page 336 Moreover, many studies, particularly those funded through government contracts, never appear in the standard venues of publication and dissemination. Thus, the insertion of such work in the accumulating knowledge base—the process of archiving and reflecting upon the results—is left to the authors themselves, the funding agency, or the National Clearinghouse for Bilingual Education. Many of the investigators in charge of these studies do not work in a setting where publications are rewarded; thus relying on them to archive and disseminate on their own initiative is not effective. The agencies, to put it simply, have a very poor record of accumulating, synthesizing, reflecting upon, and disseminating research results from the studies they have funded, on LEP issues in particular and on educational research in general. The National Clearinghouse for Bilingual Education, funded by OBEMLA, is charged with being a broker between research and practice, providing information on effective practice to the field. As a part of the Department of Education's technical assistance and information network, the clearinghouse should continue to play a role in solving this problem, but it is not integrated into any coherent system for planning, evaluating, and disseminating research results. Ideally, the building of a successful cumulative knowledge base can result only from a dynamic and coherent process that establishes priorities, funds projects, selects researchers, monitors research, coordinates work sponsored by different agencies, reviews and synthesizes results, disseminates new knowledge, and establishes new priorities. That this will not happen by itself in the infrastructure as currently configured is obvious from past performance. Resources are thin, and this challenging task is not anyone's clear-cut responsibility. The Next Generation of Researchers 10-9. Research agencies should devote a substantial portion of their funds for research on minority-language and LEP issues to doctoral dissertation competitions and postdoctoral fellowships. 10-10. Congress should restore substantial funding of Title VII fellowships for doctoral training,4 but allocate the grants to individuals studying LEP issues in any graduate department, rather than to those in programs in bilingual education per se. Although we do not have systematic data on the issue, there is considerable concern among senior researchers and agency officials that insufficient talent exists at present, or in training, to accomplish the needed research in the 4Funding for the Title VII fellowships was discontinued in the fiscal year 1996 budget, but through a reprogramming request, funding has been continued for current fellows only. The budget for fiscal year 1997 also did not appropriate funds for fellowships, and a reprogramming request is under review.

OCR for page 307
Page 337 language-minority and LEP areas. Through its work in doctoral training, dissertation support, and postdoctoral fellowships, the Spencer Foundation recognizes insufficient research talent as a general problem in education research. Its efforts have been crucial to attracting talented young people to work in education research, and some of them have worked on bilingual education and language-minority issues. Furthermore, in states with large language-minority populations, such as California, support programs (e.g., the Language Minority Research Institute) and training programs (e.g., that at the University of California at Santa Barbara) have helped to train and support researchers working on these issues. But federal research agencies, and others as well, also need to give attention to the problem of the future of the research corps. And the issue has special urgency for research relating to LEP issues because the area is politically charged, which may deter talented researchers from choosing it as a focus of their studies. Models for the needed support abound. The Office of Special Education and Rehabilitative Services, for example, divides its considerable field-initiated studies funds among doctoral dissertation support, postdoctoral fellowships, and senior research grants, and various institutes of the National Institutes of Health support training programs. Title VII fellowships are a special case. They have been the major source of funding to develop research talent in bilingual education. Although the purpose of the fellowship program is to develop faculty for teacher training programs, the attainment of a doctorate, a teaching position at a university, and tenure at a university necessarily involves Title VII recipients in the conduct and use of research on LEP issues, and many of the active researchers in this area have been recipients of Title VII fellowships. On the other hand, Title VII fellowships tend to be restricted to students in schools of education and within these schools to students in bilingual/bicultural training programs. Many researchers in bilingual education received their degrees in educational areas outside of bilingual education and in disciplinary fields outside of schools of education, such as psychology, anthropology, and linguistics. Those researchers typically do not have access to Title VII fellowships. It might be more judicious and productive to award fellowships on an individual basis rather than to institutions, so that a broader range of students can have access to such support. Cultural Versus Structural Change 10-11. All parties involved in developing the infrastructure for research on LEP issues should be aware that part of the problem is the need to escape a past history of interagency competition and mutual suspicion. The infrastructure is composed of attitudes as well as institutions. The previous recommendations require resources, recruiting, and new institutional arrangements. But they are largely structural; if they are to work well, they must be accompanied by changing attitudes. When research budgets are

OCR for page 307
Page 338 low, issues are politically charged, agendas are volatile, leadership is constantly changing, and leaders believe they must continually reinvent their agencies, a vicious circle of low morale, low expertise, low performance, and low respect infuses the federal education research enterprise, and where good work is done in the agencies, it is done under great stress and without much reward. It is inevitable and understandable that under such circumstances, competitive and defensive attitudes are common. Even if new resources are forthcoming and new structures mandated, it will take an act of collective will to build effective collaboration across federal research agencies and between those agencies and their two ''fields"—the field of educators who need to be involved in the agenda formulation, the conduct, and the uses of the research, as well as the academic field of researchers who work on LEP issues. Roles for State Education Agencies and Foundations Infrastructure Needs Regarding State Education Agencies 10-12. States should place some emphasis on the concerns expressed in recommendation 10-4 above—to include English-language learners in data gathering, to disaggregate the data by language status where possible in reporting, and more generally to be alert to the potential enrichment of research designs by attending to language issues. Specifically, states should collaborate with experts in institutions of higher education and district and school staff to learn more about the following areas: • The incorporation of English-language learners into state assessment programs. Issues to be addressed would include how to decide which students get which assessments, as well as the development of alternative assessments for students unable to take the standard ones. • The development of standard procedures for determining the English and native-language proficiency of English-language learners, best program placements, and the point at which these students should be exited from special programming. • The development and evaluation of various theoretically driven models of instruction. • The development of curricula that would enable English-language learners to meet high standards. • The development and evaluation of teacher education programs and certification examinations for mainstream teachers who work with English-language learners and for teachers who work in English as a second language and bilingual education programs. 10-13. The Department of Education should consider providing financial support for some of these collaborative activities.

OCR for page 307
Page 339 10-14. State universities in states with large numbers of English-language learners should consider establishing research and technical assistance programs to support faculty and students with interest in these issues. The University of California's Linguistic Minority Research Institute is an example of a state-wide research support network for language-minority and LEP issues. Some states with large English-language learner populations are very conscious of the educational issues surrounding such students. Others, perhaps with smaller but growing numbers of English-language learners, are less active in gathering data and developing programs for these students. Our survey disclosed that even in the states with the highest concentrations of English-language learners, little research on LEP issues is conducted under the auspices of the state departments of education. However, there is a great deal of potential here for contribution to the research effort, because states conduct program evaluation and assessment of students and school-level performance, and they collect descriptive information on schools, teachers, and students. Infrastructure Needs Regarding Research Support from Foundations 10-15. Foundations concerned with education research and reform should encourage grantees, where appropriate, to place some emphasis on the concerns expressed in recommendation 10-4 above—to include English-language learners in data gathering, to disaggregate the data by language status where possible in reporting, and more generally to be alert to the potential enrichment and generalizability of research designs by attending to language issues. 10-16. Foundations can facilitate a more coherent research agenda on LEP issues by setting up and supporting communication—ongoing networks or conferences—among people who do not otherwise work together. The work that led to this report is the kind of reflection and synthesis that can result from such support. Foundations have the independence and the resources to be catalysts for research, brokers for tough-minded stock taking, and sponsors of research synthesis and agenda setting (as in the case of the present report, which was funded by a combination of foundation and federal funds). The Spencer Foundation's central role in sponsoring basic and applied research, as well as in supporting the recruitment and training of the next generation of education researchers, has been discussed above. With this notable exception, foundations interested in education tend to emphasize action, reform, and the development of effective educational programs, not research per se. However, because research on learning and its contexts is often intertwined with such activities, these foundations can foster excellent research and at the same time press researchers to relate their work to the world of practice. In our informal survey of the foundations most interested

OCR for page 307
Page 340 in education, we did not find as robust an interest in language issues as we had hoped. Perhaps this report may inspire shifts of emphasis in some agendas or suggest ways in which foundation-sponsored work can address language-minority and LEP issues without much additional cost. References American Association for the Advancement of Science 1996 Research and Development FY 1997. AAAS Report XXI. Washington, DC: American Association for the Advancement of Science. American Educational Research Association 1996 The grass is greener: Department of Defense Centers. Research Policy Notes (Dec. 1995-Jan. 1996):3-4. Atkinson, R.C., and G.B. Jackson, eds. 1992 Research and Education Reform: Roles for the Office of Educational Research and Improvement. Committee on the Federal Role in Education Research, National Research Council. Washington, DC: National Academy Press. Donmoyer, R. 1996 Educational research in an era of paradigm proliferation: What's a journal editor to do? Education Researcher 25(2):19-25. Elliott, E. 1996 Briefing Paper on Peer Review for Consultants to the National Education Research Policy and Priorities Board. Office of Educational Research and Improvement, National Educational Research Policy and Priorities Board. Washington, DC: U.S. Department of Education. Kaestle, C.F. 1992 Everybody's Been to Fourth Grade: An Oral History of Federal R&D in Education. Wisconsin Center for Education Research, University of Wisconsin-Madison. Kiesler, S.B., and C. Turner, eds., 1977 Fundamental Research and the Process of Education. Committee on Fundamental Research Relevant to Education, National Research Council. Washington, DC: National Academy Press. Kingdon, J.W. 1984 Agendas, Alternatives, and Public Policies. New York: Harper-Collins. National Council on Educational Research 1978 Fourth Annual Report. National Institute of Education. Washington, DC: U.S. Department of Education. National Research Council 1996 Beyond the Blueprint. Directions for Research on Head Start's Families. Roundtable on Head Start Research, Board on Children, Youth, and Families, Institute of Medicine and National Research Council. Washington, DC: National Academy Press. Norwood, J.L. 1995 Organizing to Count: Change in the Federal Statistical System. Washington, DC: Urban Institute Press. Office of Bilingual Education and Minority Languages Affairs 1990 Proceedings of the First Research Symposium on Limited English Proficient Students, September. U.S. Department of Education.

OCR for page 307
Page 341 1992 Proceedings of the Second Research Symposium on Limited English Proficient Students, August. U.S. Department of Education. 1993 Proceedings of the Third Research Symposium on Limited English Proficient Students, September. U.S. Department of Education. Office of Science and Technology Policy 1991 By the Year 2000: First in the World. Prepared by the Committee on Education and Human Resources, Federal Coordinating Council for Science, Engineering and Technology, James D. Watkins, Secretary of Energy, Chair. Washington, DC: Office of Science and Technology Policy. Riley, R.W. 1995 Letter from the U.S. Secretary of Education, to the Honorable John Porter, Chairman, Subcommittee on Labor, Health and Human Services, and Education, Committee on Appropriations, House of Representatives, September 18. Southwest Educational Development Laboratory 1995 Task 7: Specialty Area Development/Language and Cultural Diversity. Proposal prepared by Southwest Educational Development Laboratory, Austin, Texas. U.S. General Accounting Office 1994 Peer Review: Reforms Needed to Ensure Fairness in Federal Agency Grant Selection. Report to the Chairman, Committee on Governmental Affairs, U.S. Senate. GAO/PEMD-94-1. Washington, DC: U.S. General Accounting Office. Vinovskis, M.A. 1993 Analysis of the Quality of Research and Development at the OERI Research and Development Centers and the OERI Regional Educational Laboratories. Unpublished report, Office of Educational Research and Improvement, June. U.S. Department of Education, Washington, DC. 1995 Changing Views of the Federal Role in Educational Statistics and Research. Office of Educational Research and Improvement, Preliminary draft, September. U.S. Department of Education, Washington, DC.

OCR for page 307