International Education and Foreign Languages: Keys to Securing America's Future (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Part II Key Areas of Concern I n the preceding chapters, the committee briefly reviewed the legislative and funding history of the Title VI and Fulbright-Hays (Title VI/FH) programs and information about the demand for foreign language and area expertise. We also considered the role that the Title VI/FH programs play in relation to other federal programs with a foreign language or area studies emphasis. On the basis of these reviews, we came to the conclusion that the Title VI/FH programs play a unique and vital role in the array of federal programs aimed at addressing foreign language and area knowledge needs. Specifically, their role is to address a broad set of needs for foreign language, international, and area expertise, rather than to respond to the current demand for government expertise. That conclusion forms the con- text for our discussion of several of the key areas identified in our charge. This part provides an overview of the evidence that pertains to the ef- fectiveness of the Title VI/FH programs in addressing each of the eight key area specified in the congressional request. In conducting our review, the committee first developed a conceptual model. Before discussing the types of information used by the committee in conducting its review, a word about the committee’s model and the available evidence is warranted (see Ap- pendix B for further detail, including schematics of the conceptual model). Our conceptual model clearly illustrates the complexity of the interactions among programs and that multiple programs—directly or indirectly—con- tribute to most of the areas specified in the committee’s charge. In general, therefore, the committee’s analysis of the programs’ performance related to each of the eight key areas addresses the programs as a whole. Although some of the chapters highlight specific programs given their particular rel- 83 

84 INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES evance to the key area being discussed, we do not compare the performance of specific programs to one another or specify the relative contribution of individual programs to the overall Title VI/FH goals. Indeed, this would not have been possible with the evidence available. The conceptual model also articulates the differences among impacts, outcomes, outputs, activities, and inputs. In determining the effectiveness of a program, one should aim to assess impacts and outcomes or the ac- complishment of program or social objectives that can be attributed to the program. However, in the case of the committee’s review, and indeed in the case of much program decision making, very little evidence that would en- able an assessment of outcomes is available. In addition, because the eight key areas identified in the committee’s charge include both outcomes (e.g., increasing the numbers of underrepresented minorities in international ser- vice, reducing shortages of foreign language and area experts) and outputs (e.g., conducting outreach), in some cases the program’s performance in one area might affect its performance in another. For example, effective out- reach to K-12 could contribute to increased infusion of a foreign language and area studies dimension throughout the education system. The report is organized to present the available evidence—largely evidence related to outputs and activities—for each of the key areas in turn. The committee recognizes, however, that just as the programs interrelate to one another to achieve the specific objectives reflected in the key areas, the key areas themselves are interrelated and one might contribute to another. In conducting our review, the committee utilized information from a variety of sources. Although no single source provided sufficient evidence to draw reliable conclusions and the combined evidence is not optimal, the combined sources allow some conclusions to be drawn about the programs. Specifically, the committee • reviewed all extant evaluations; • reviewed program monitoring data, selected grant applications, historical financial data, and written comments from experts and officials; • commissioned papers and targeted analyses; • conducted public meetings to get input from subject matter experts, officials, and others and met with grantees in Washington and officials from the U.S. Department of Education (ED) with responsibilities for these programs; and • conducted site visits to eight universities: Georgetown University, George Washington University, Indiana University, New York University, Ohio State University, San Diego State University, the University of Califor- nia, Los Angeles, and the University of Wisconsin-Madison. The committee reviewed 11 studies that provide relatively current in- 

KEY AREAS OF CONCERN 85 formation about the Title VI/FH programs. Two other important studies were not considered by the committee because they examined the programs almost 30 years ago. In addition to the completed evaluations, a couple of ongoing activities will eventually provide information about some pro- grams. About half of the evaluation studies of the various Title VI/FH programs were performed by people or entities that were themselves receiv- ing funds under Title VI/FH. In a few cases, studies relied on self-reporting on the success of the programs; in other words, administrators were asked to rank the effectiveness of their own efforts, which may produce biased results. Several studies had various methodological issues, including lack of a comparison group (control groups would be infeasible) or low response rates. Descriptions of these studies, with their shortcomings, are provided in Appendix B. Given the reality that these are the only studies available, the committee reports their results with specific needed caveats when ap- propriate. In some cases, a study is relevant to more than one key area, and specific findings are discussed in different chapters. Recognizing the limitations of the available evaluation evidence, however, the committee recommends in the final chapter that ED invest in more rigorously designed program evaluations. The committee also reviewed program monitoring data and histori- cal funding data. ED provided direct access to complete the Evaluation of Exchange, Language, International and Area Studies (EELIAS) database as of March 2006. The database is used for grant monitoring purposes; for example, reports are used to decide on continuation awards for multiyear grants. ED has also used the database to collect data needed for its national performance measures. The committee hoped that the database, which in- cludes multiyear data for some programs, would also be a readily available program evaluation tool. We found, however, that while the database has the potential to collect comprehensive information, methodological issues with how it was structured and implemented limit its usefulness beyond individual grant monitoring; in fact, the database itself became a focus for our review. Whenever possible, we have presented relevant information from the database with necessary caveats. The final chapter also discusses in more detail the limitations of the system and the committee’s observa- tions about its limitations.   Studies by McDonnell, Berryman, and Scott and published by the Rand Corporation in 1981 and 1983 are cited in the references. The study addressed the four programs that had been established at the time: National Resource Centers (NRC), Foreign Language and Area Studies Fellowships, International Research and Studies, and the Undergraduate International Studies and Foreign Language Program.   This includes a study of Middle East Studies Centers by the Social Science Research Coun- cil. ED is also in the planning stages of awarding a contract to evaluate the International Research and Studies Program. 

86 INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES The committee also invited individuals and organizations working, par- ticipating in, or otherwise affected by or interested in Title VI/FH programs to present information to the committee about the programs’ strengths and weaknesses. The committee also solicited input through several public meetings that included comments by people expert in the field and met with Language Resource Centers and NRC directors in Washington, DC. Some of the information presented was based on evidence, whereas some of it was based on their opinion or anecdotes. Over the course of the study, the com- mittee received written comments from 17 individuals representing numer- ous organizations and grantees and heard from numerous others. The line between evaluation studies and public input may be blurred in some places. The same studies mentioned above were often cited by the individuals who made presentations or submitted information to the committee. The committee commissioned targeted analyses for areas not addressed at all in existing evaluations: producing relevant instructional materials, advancing uses of new technology, and increasing efforts aimed at participa- tion of minorities in international programs. Although the review provided insights into what is being produced by Title VI grantees in these areas, it did not include necessary information to assess the quality of the materials produced or the extent to which the materials are being used; this would have required a more systematic evaluation. Finally, the committee gathered qualitative evidence through eight site visits to universities with center programs. The site visits were designed in part to address the shortcomings of other available sources in terms of providing an understanding of how the major programs in this complex array of programs operate in practice. All of the universities had at least one NRC; several also had Language Resource Center grants, Centers for International Business Education and Research grants, or both. Although not a representative sample, the universities chosen provide a general pic- ture of the diversity of grantees and provide qualitative examples of how the center programs are implemented. In some cases, there was unanimity across the universities visited on particular points; these points are high- lighted in the text of relevant chapters. In other cases, the committee felt that an example from the site visits illustrates a conclusion drawn from the combined evidence; in these cases, an illustration is provided in the form of a text box in the chapter. In addition to drawing on all of the sources of evidence listed above,   See Appendix B for a list of the individuals who submitted written comments as well as agendas for the committee’s public meetings. Written comments are available at http://www7. nationalacademies.org/cfe/International_Education_Programs.html.   During the course of the committee’s review, an evaluation of the Institute for International Public Policy Program commissioned by the United Negro College Fund Special Programs Corporation was completed and included in the committee’s commissioned analysis. 

KEY AREAS OF CONCERN 87 the committee drew on its own expertise and lessons from relevant related literature to establish findings and draw conclusions. Although no single type of evidence provides a sufficient basis on which to draw conclusions about the program, the committee thinks that the combined weight of the various sources provides sufficient basis to support the conclusions drawn. In many cases, however, the evidence was too limited to support recommen- dations about specific programs or particular key areas. Clearly additional rigorously designed program evaluations and a better designed program monitoring system would improve future reviews. This issue is discussed in Chapter 11. The chapters that follow provide the committee’s discussion of each of the eight key areas specified in the congressional request for this study.