2

The FSI Testing Context

This chapter provides an overview of the FSI testing context. It discusses how languages are used in the Foreign Service (the target language use domain of the test), how FSI assesses the language proficiency of Foreign Service officers and other Foreign Service personnel, and how the results of those assessments are used in the State Department.

MANDATE FOR ASSESSING FOREIGN LANGUAGE PROFICIENCY

Foreign Service officers are posted to nearly every country in the world. As of December 2018, the U.S. State Department was operating 170 embassies and 107 consulates or missions to international organizations. In some countries, such as Brazil and China, the United States operates an embassy and several consulates.

Most U.S. embassies include eight job categories that require the greatest use of foreign language. The job categories and their associated language uses are summarized in Table 2-1.

Foreign Service officers are expected to be able to function effectively and professionally in these capacities. Accordingly, their language proficiency is expected to be adequate to perform in the local language across the job categories. For this reason, foreign language proficiency is a central feature in the professional development of U.S. diplomats and is required for many Foreign Service officers. This requirement was established in the Foreign Service Act of 1980, as amended, and has been incorporated into high-stakes personnel decisions relating to tenure, promotion, overseas

TABLE 2-1 Summary of U.S. Embassy Job Categories and Language Uses

^a The broad language uses for chiefs of mission apply generally to career Foreign Service officers serving as ambassadors, although language proficiency is not a prerequisite for an appointment. Career Foreign Service officers who serve as chiefs of mission usually acquire relevant language skills during the course of their careers. However, most political appointees serving as ambassadors receive little or even no language training immediately prior to assignment as a chief of mission and conduct business in English or through a translator or interpreter.

postings, and incentive pay. The law directs the Secretary of State to establish foreign language proficiency requirements though it does not prescribe how the secretary should define or measure proficiency (Foreign Service Act of 1980, as amended, Section 702, 22 U.S.C 4022 et seq.):

LANGUAGE NEEDS AND TRAINING OF FOREIGN SERVICE OFFICERS

FSI’s School of Language Studies provides intensive language training on a full-time basis for Foreign Service officers to develop their language proficiency. The FSI school is also used by other U.S. government agencies that assign personnel abroad, including the Agency for International Development and the U.S. Department of Defense. The school provides instruction in more than 65 languages. The length of training depends on the difficulty of the language for English speakers, ranging from 24 to 30 weeks (Spanish, French) to 88 weeks (Arabic, Japanese).

Every year, FSI surveys State Department employees who completed FSI language training during the previous fiscal year and who are currently serving in a language-designated position. The survey asks them how they are using their language skills in their work and how well FSI language training prepared them to do so.

Aggregated results from the 2012 to 2016 surveys show a need to frequently perform a wide range of activities in the local language related to their jobs. As reported on the surveys, the most commonly used language activities include

• socializing both informally and in business settings,
• understanding meeting discussions and social conversations,
• understanding job-related documents,
• understanding broadcast and print media,
• communicating over the telephone and through e-mail,
• interviewing to elicit information,
• making presentations,
• writing social correspondence,
• giving instructions or explaining procedures, and
• monitoring and interacting using social media.

In some situations, locally employed staff who are native speakers of the local language can assist, but in regular everyday settings some level of language proficiency by the Foreign Service officers is essential. Although not addressed in the survey, it is likely that Foreign Service officers carry out these various tasks with some assistance from language supports, including dictionary and translation apps.

The goal of the language training is to prepare Foreign Service officers to participate effectively in this wide range of language activities. The focus is on a level of professional language proficiency that would allow Foreign Service officers to carry out any of the formal or informal job activities associated with language-designated positions in embassies and consulates. Box 2-1 illustrates the range of these activities with examples of

anonymized responses to a question on the Annual Language Impact Survey that asked respondents to describe a memorable time when they used their language skills effectively on the job.

CURRENT FSI TESTING

Overview

In fiscal 2018, FSI directly administered 3,364 tests in 63 languages to Foreign Service officers and other government agency personnel, and it outsourced 802 tests for external candidates for limited career appointments in consular affairs at overseas posts. This test volume is generally as it has been in recent years, although the volume has decreased since peaking at 5,729 in 2011. The number of languages tested in fiscal 2018 also was slightly lower than had been usual over the past decade, when approximately 80 languages were tested each year.

About two-thirds of the tests are in five widely used languages: Arabic (260), French (583), Mandarin Chinese (271), Russian (208), and Spanish (1,071). The tests in the remaining languages are given to far fewer people, including 35 languages with 10 or fewer test takers. (All data are for fiscal 2018 for in-house tests.)

Across all languages tested, FSI’s assessment of language proficiency relies primarily on in-person tests of speaking and reading, which have evolved from an approach first developed by the agency in the 1950s. Test scores are reported on a five-point scale (1 to 5) and defined using skill-level descriptions.¹ These descriptions were developed by the Interagency Language Roundtable (ILR), a group that coordinates second language training, acquisition, and testing approaches across the U.S. government. At the time of this report, the skill-level descriptions in the ILR framework were being revised.

The typical goal for language training is for Foreign Service officers to score at ILR level 3 in both speaking and reading (referred to as “3/3”), with this level of language proficiency intended to enable the kinds of job-related tasks the officers will encounter. Box 2-2 provides the ILR descriptions for level 3 reading and speaking, which are the focus of the FSI assessment. There are similar descriptions for listening, writing, translation, interpretation, and intercultural communication.

___________________

¹ The full skill-level descriptions for the ILR scale include a 0-level for no proficiency, “plus” levels for levels 0–4, and examples to elaborate the descriptions.

Test Uses and Decisions

Scores on the FSI test are used to make many types of decisions about test takers. For example, many job postings to other countries are contingent on test scores. Foreign Service or other government agency personnel who go through the language training program before they leave to take up a posting in another country typically take FSI’s language assessment at the end of the language training period. However, only about half of the tests administered each year are directly related to training. Non-training-related tests are taken by officers who want to add or update a score for retention or for promotion purposes or to have a current score on file so that they may apply for a posting in a country where that language is spoken.

The State Department provides requirements and incentives for personnel in language-designated positions to achieve a level 3 in both speaking and reading. Job assignments typically are contingent on achieving those target scores, although employees who do not reach the target ILR level of proficiency can take up their assignments while continuing to work to achieve the required proficiency.

In addition to the language score requirements associated with specific postings, employees receive incentive pay for their demonstrated proficiency in certain priority languages: 5 percent of salary for a 2/2 rating, 10 percent for a 3/3 rating, and 15 percent for a 4/4 rating. For all Foreign Service officers, scores below 4/4 expire after 5 years.

Components of the FSI Test

In all languages, the current FSI language proficiency assessment consists of a speaking test and a reading test. Listening is not tested separately but is incorporated in the speaking test. Although there are some variations, what follows is a general description of the FSI assessment.

Speaking Test

The speaking test has three parts:

1. Social conversation. The test taker introduces him or herself and discusses with the testing team topics such as daily life situations, and if proficiency allows, more complex topics, such as social, political, and current events.
2. Work-related statement and exchange. The test taker selects a general topic from a set of topics that are loosely aligned with the Foreign Service career tracks, such as consular affairs, diplomatic security, environment/science/technology/health care, international

3. development, management, political/military affairs, or public diplomacy. The test taker has 5 minutes to prepare an introductory statement on the topic. After the introductory statement, the test taker engages with the tester in the exchange part of the conversation.

4. Interview. This component of the test is an information gathering and reporting exercise. The test taker selects a topic from a category that aligns with the Foreign Service career tracks. Without preparation, the test taker begins interviewing the tester on that topic, in the language that is being tested. The test taker asks questions and listens to the responses until he or she feels that enough information has been collected. The test taker reports, in English, what was said immediately after the tester’s response to each question.

Two aspects of the speaking test were changed in 2015. The social conversation now includes a gradual warm-up aimed at putting the test taker at ease, and a longer presentation task was replaced with a work-related exchange focusing more on an interactive dialogue.

Reading Test

The reading test consists of two tasks:

1. Reading for gist. This component is a carefully timed diagnostic test during which the testing team estimates the test-taker’s working level in reading. The test taker is given six paragraphs of varying difficulty, with 6 minutes to identify the subject matter and the general meaning of as many passages as possible. Test takers are instructed that the task is like reading the newspaper—skimming and scanning documents for information.
2. Reading in depth. The outcome of reading for gist determines the level of difficulty of the text for the reading in depth portion of the test. Here, the test taker reads two to three longer articles in the target language and then reports, in English, on the main ideas, the supporting details, and information that generally explains the meaning of the text. The test taker is given 12 minutes to read each text. The objective is not to provide a direct translation of the text but instead for the test taker to use his or her own words to report as much information as possible from the text.

In 2018, the preparation time for the reading in-depth task was extended from 7 to 12 minutes to reassure test takers that they should focus on comprehension and not speed of reading.

Test Administration and Scoring

FSI strives for its test administration and scoring procedures to be consistent across all languages tested. Most FSI tests are conducted in person, by digital video conference, or by speakerphone. Although the preferred mode is in person, video conferencing has increased in recent years and is now used for about 20 percent of test takers, and testing by speakerphone is around 10 percent.

The speaking and reading tests each last about 1 hour. Test takers can start with either portion, and the two portions can be separated by an optional 5-minute break.

The test is administered to individual test takers by a tester and an examiner. The tester is the rater who interacts with the test taker in the language of the test. The examiner interacts with the test taker in English to administer the test, provide instructions, and monitor the timing of each task. FSI’s goal is for testers and examiners to be unfamiliar with test takers. This goal is relatively easily accomplished in high-volume languages, for which the language school has full-time testers and multiple instructors. However, for languages with fewer learners and thus fewer instructors, the tester may have also been the test-taker’s language teacher in the early phases of language training.

In contrast with other agencies that use the ILR framework for language proficiency testing, FSI does not align specific reading texts with individual ILR levels. Based on the FSI’s belief that it is possible to show a range of proficiency when reading a specific text, there are three general categories of FSI reading texts that roughly correspond to the proficiency ranges of ILR levels 1 to 2 (A-level texts), 2-plus to 3 (B-level texts), and 3-plus to 5 (C-level texts). FSI’s testing protocol is adaptive in that it involves an initial determination of the working level of the test taker and includes the flexibility to move up or down from that initial level.

The FSI scoring approach also differs from other agencies. Scoring is an interactive, deliberative, and consensus-based procedure involving the tester and examiner (Hart-Gonzalez, 1994). An overall ILR proficiency-level rating is determined holistically, with the tester and examiner reaching an initial tentative consensus, based on their overall judgment of the performance. They then consider the test-taker’s strengths and weaknesses related to five factors: comprehension, ability to organize thoughts, grammar, vocabulary, and fluency. As part of this consideration, the tester and

the examiner separately estimate quantitative values for the five factors, which are added together to create an “index” score on a 120-point scale. This index score is used as a check on the holistic rating on the ILR scale and to confirm the consensus between the tester and the examiner, possibly leading to some adjustment in the consensus ILR scale score. Although listening is not considered explicitly or reported separately, listening skills are obviously required to perform well on the speaking test and are reflected in the comprehension factor. The scoring sequence—from initial holistic rating to the five-factor derivation of an index score and then to comparison of the separate index scores with the initial holistic rating—is repeated two times, once for the speaking test and once for the reading test.

If test takers are dissatisfied with their test results, they can ask for their scores to be reviewed within 30 days of their test. They can generally retake the test after 6 months.