|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 83
Part II
Key Areas of Concern
I
n the preceding chapters, the committee briefly reviewed the legislative
and funding history of the Title VI and Fulbright-Hays (Title VI/FH)
programs and information about the demand for foreign language and
area expertise. We also considered the role that the Title VI/FH programs
play in relation to other federal programs with a foreign language or area
studies emphasis. On the basis of these reviews, we came to the conclusion
that the Title VI/FH programs play a unique and vital role in the array of
federal programs aimed at addressing foreign language and area knowledge
needs. Specifically, their role is to address a broad set of needs for foreign
language, international, and area expertise, rather than to respond to the
current demand for government expertise. That conclusion forms the con-
text for our discussion of several of the key areas identified in our charge.
This part provides an overview of the evidence that pertains to the ef-
fectiveness of the Title VI/FH programs in addressing each of the eight key
area specified in the congressional request. In conducting our review, the
committee first developed a conceptual model. Before discussing the types of
information used by the committee in conducting its review, a word about
the committee’s model and the available evidence is warranted (see Ap-
pendix B for further detail, including schematics of the conceptual model).
Our conceptual model clearly illustrates the complexity of the interactions
among programs and that multiple programs—directly or indirectly—con-
tribute to most of the areas specified in the committee’s charge. In general,
therefore, the committee’s analysis of the programs’ performance related to
each of the eight key areas addresses the programs as a whole. Although
some of the chapters highlight specific programs given their particular rel-
OCR for page 84
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
evance to the key area being discussed, we do not compare the performance
of specific programs to one another or specify the relative contribution of
individual programs to the overall Title VI/FH goals. Indeed, this would
not have been possible with the evidence available.
The conceptual model also articulates the differences among impacts,
outcomes, outputs, activities, and inputs. In determining the effectiveness
of a program, one should aim to assess impacts and outcomes or the ac-
complishment of program or social objectives that can be attributed to the
program. However, in the case of the committee’s review, and indeed in the
case of much program decision making, very little evidence that would en-
able an assessment of outcomes is available. In addition, because the eight
key areas identified in the committee’s charge include both outcomes (e.g.,
increasing the numbers of underrepresented minorities in international ser-
vice, reducing shortages of foreign language and area experts) and outputs
(e.g., conducting outreach), in some cases the program’s performance in one
area might affect its performance in another. For example, effective out-
reach to K-12 could contribute to increased infusion of a foreign language
and area studies dimension throughout the education system. The report
is organized to present the available evidence—largely evidence related to
outputs and activities—for each of the key areas in turn. The committee
recognizes, however, that just as the programs interrelate to one another
to achieve the specific objectives reflected in the key areas, the key areas
themselves are interrelated and one might contribute to another.
In conducting our review, the committee utilized information from a
variety of sources. Although no single source provided sufficient evidence
to draw reliable conclusions and the combined evidence is not optimal, the
combined sources allow some conclusions to be drawn about the programs.
Specifically, the committee
• reviewed all extant evaluations;
• reviewed program monitoring data, selected grant applications,
historical financial data, and written comments from experts and officials;
• commissioned papers and targeted analyses;
• conducted public meetings to get input from subject matter experts,
officials, and others and met with grantees in Washington and officials
from the U.S. Department of Education (ED) with responsibilities for these
programs; and
• conducted site visits to eight universities: Georgetown University,
George Washington University, Indiana University, New York University,
Ohio State University, San Diego State University, the University of Califor-
nia, Los Angeles, and the University of Wisconsin-Madison.
The committee reviewed 11 studies that provide relatively current in-
OCR for page 85
KEY AREAS OF CONCERN
formation about the Title VI/FH programs. Two other important studies
were not considered by the committee because they examined the programs
almost 30 years ago.1 In addition to the completed evaluations, a couple
of ongoing activities will eventually provide information about some pro-
grams.2 About half of the evaluation studies of the various Title VI/FH
programs were performed by people or entities that were themselves receiv-
ing funds under Title VI/FH. In a few cases, studies relied on self-reporting
on the success of the programs; in other words, administrators were asked
to rank the effectiveness of their own efforts, which may produce biased
results. Several studies had various methodological issues, including lack of
a comparison group (control groups would be infeasible) or low response
rates. Descriptions of these studies, with their shortcomings, are provided
in Appendix B. Given the reality that these are the only studies available,
the committee reports their results with specific needed caveats when ap-
propriate. In some cases, a study is relevant to more than one key area,
and specific findings are discussed in different chapters. Recognizing the
limitations of the available evaluation evidence, however, the committee
recommends in the final chapter that ED invest in more rigorously designed
program evaluations.
The committee also reviewed program monitoring data and histori-
cal funding data. ED provided direct access to complete the Evaluation of
Exchange, Language, International and Area Studies (EELIAS) database as
of March 2006. The database is used for grant monitoring purposes; for
example, reports are used to decide on continuation awards for multiyear
grants. ED has also used the database to collect data needed for its national
performance measures. The committee hoped that the database, which in-
cludes multiyear data for some programs, would also be a readily available
program evaluation tool. We found, however, that while the database has
the potential to collect comprehensive information, methodological issues
with how it was structured and implemented limit its usefulness beyond
individual grant monitoring; in fact, the database itself became a focus for
our review. Whenever possible, we have presented relevant information
from the database with necessary caveats. The final chapter also discusses
in more detail the limitations of the system and the committee’s observa-
tions about its limitations.
1 Studies by McDonnell, Berryman, and Scott and published by the Rand Corporation in
1981 and 1983 are cited in the references. The study addressed the four programs that had
been established at the time: National Resource Centers (NRC), Foreign Language and Area
Studies Fellowships, International Research and Studies, and the Undergraduate International
Studies and Foreign Language Program.
2 This includes a study of Middle East Studies Centers by the Social Science Research Coun-
cil. ED is also in the planning stages of awarding a contract to evaluate the International
Research and Studies Program.
OCR for page 86
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
The committee also invited individuals and organizations working, par-
ticipating in, or otherwise affected by or interested in Title VI/FH programs
to present information to the committee about the programs’ strengths
and weaknesses. The committee also solicited input through several public
meetings that included comments by people expert in the field and met with
Language Resource Centers and NRC directors in Washington, DC. Some
of the information presented was based on evidence, whereas some of it was
based on their opinion or anecdotes. Over the course of the study, the com-
mittee received written comments from 17 individuals representing numer-
ous organizations and grantees and heard from numerous others.3 The line
between evaluation studies and public input may be blurred in some places.
The same studies mentioned above were often cited by the individuals who
made presentations or submitted information to the committee.
The committee commissioned targeted analyses for areas not addressed
at all in existing evaluations: producing relevant instructional materials,
advancing uses of new technology, and increasing efforts aimed at participa-
tion of minorities in international programs.4 Although the review provided
insights into what is being produced by Title VI grantees in these areas, it
did not include necessary information to assess the quality of the materials
produced or the extent to which the materials are being used; this would
have required a more systematic evaluation.
Finally, the committee gathered qualitative evidence through eight site
visits to universities with center programs. The site visits were designed
in part to address the shortcomings of other available sources in terms of
providing an understanding of how the major programs in this complex
array of programs operate in practice. All of the universities had at least
one NRC; several also had Language Resource Center grants, Centers for
International Business Education and Research grants, or both. Although
not a representative sample, the universities chosen provide a general pic-
ture of the diversity of grantees and provide qualitative examples of how
the center programs are implemented. In some cases, there was unanimity
across the universities visited on particular points; these points are high-
lighted in the text of relevant chapters. In other cases, the committee felt
that an example from the site visits illustrates a conclusion drawn from the
combined evidence; in these cases, an illustration is provided in the form
of a text box in the chapter.
In addition to drawing on all of the sources of evidence listed above,
3 SeeAppendix B for a list of the individuals who submitted written comments as well as
agendas for the committee’s public meetings. Written comments are available at http://www7.
nationalacademies.org/cfe/International_Education_Programs.html.
4 During the course of the committee’s review, an evaluation of the Institute for International
Public Policy Program commissioned by the United Negro College Fund Special Programs
Corporation was completed and included in the committee’s commissioned analysis.
OCR for page 87
KEY AREAS OF CONCERN
the committee drew on its own expertise and lessons from relevant related
literature to establish findings and draw conclusions. Although no single
type of evidence provides a sufficient basis on which to draw conclusions
about the program, the committee thinks that the combined weight of the
various sources provides sufficient basis to support the conclusions drawn.
In many cases, however, the evidence was too limited to support recommen-
dations about specific programs or particular key areas. Clearly additional
rigorously designed program evaluations and a better designed program
monitoring system would improve future reviews. This issue is discussed in
Chapter 11. The chapters that follow provide the committee’s discussion
of each of the eight key areas specified in the congressional request for this
study.
OCR for page 88
