National Academies Press: OpenBook

A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute (2020)

Chapter: 7 Balancing Evaluation and the Implementation of New Approaches

« Previous: 6 Evaluating Validity in the FSI Context
Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×

7

Balancing Evaluation and the Implementation of New Approaches

Throughout this report, the committee notes a number of specific options related to language assessment that FSI may want to explore, given the research literature on language assessment. The committee was specifically asked by FSI not to provide recommendations about FSI’s language assessment program since the agency is responsible for determining its own program. Thus, in this concluding chapter the committee addresses the basic choice about the balance between evaluation and the implementation of new approaches that are relevant to FSI in determining how to proceed. Decisions about how to set this balance will influence all aspects of the development of FSI’s assessment program.

BASIC CONSIDERATIONS

At the heart of the FSI’s choice about how to strengthen its testing program lies a decision about the balance between (1) conducting an evaluation to understand how the current program is working and could be changed in light of a principled approach to assessment, and (2) beginning the implementation of new approaches. Evaluation and implementation are both necessary: evaluation of the current program without implementation of new approaches to bring improvements will have no effect, and implementation of new approaches without a full evaluation of the current test could be very harmful to the current program. However, given limited time and resources, it is important to decide the relative attention to give to each.

Through this report, the committee addresses both evaluation, through

Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×

the presentation of a principled approach to assessment, and implementation, through the presentation of new approaches to assessment. For evaluation, the report stresses the importance of understanding the target language use domain as a foundation for both the design and the validation of a testing program, and it briefly describes the available techniques for developing that understanding (Chapter 3). Furthermore, the report addresses the role the understanding of language plays in undergirding a testing program and the importance of understanding its sociocultural and institutional contexts. The report also stresses the importance of evaluating the validity of the uses of the test in light of the target language use domain and key details related to the test (Chapter 6). For implementation, the report suggests possible changes to the current test that might reflect identified goals for strengthening the current test, on the basis of what FSI currently knows about the strengths and weaknesses of the test or information that would result from an evaluation of the test (Chapter 4). These arguments and discussions raise the essential question about whether to emphasize—at this time—further evaluation to better understand the test and how well it is working or initial steps toward implementing plausible changes.

For FSI’s decision about the relative attention to give to each, it will be important to consider how well FSI’s current language assessment practices address the language proficiency needed by Foreign Service officers. The committee’s limited understanding of the language proficiency needed by Foreign Service officers and the current language assessment suggests that there are certainly points of commonality. FSI’s assessment is clearly different from a language assessment that might be used in other settings—such as certifying the language abilities of medical professionals or admitting graduate students to a course of study—and the distinctive aspects of FSI’s assessment appear to reflect the language tasks of Foreign Service officers. However, the committee had insufficient evidence about the nature of the language proficiency needed and the alignment of FSI’s assessment to those abilities to draw any conclusions about how close the alignment is. The committee’s discussion of some possible changes to the current test highlights a number of ways that the coverage of the language proficiency of Foreign Service officers may be limited in capturing all important aspects of their language-related tasks, but the committee has no information about the relative importance of these omitted aspects of Foreign Service tasks.

One of the key issues for FSI to consider is whether it has sufficient information to draw firm conclusions about the degree of alignment between the aspects of language proficiency measured by the test and the aspects that affect the performance of key Foreign Service tasks. If the available information is not sufficient to draw firm conclusions, then obtaining better information about the alignment is particularly important. However, if there is already good information about the degree of alignment, then

Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×

that information can help guide the consideration of changes to the current language assessment program.

The alignment between the language proficiency demonstrated by the current test and the language proficiency needed by Foreign Service officers is only one example of a key piece of evidence needed by a language assessment program to consider possible changes; it is captured in the first example validity claim discussed in Chapter 6. The other example validity claims discussed in that chapter suggest other instances of the tradeoff between evaluation and the implementation of new approaches. What is already known about the scoring process, the interpretation of the scores, and the relative benefits of the use of the scores? In each case, there could be very limited information, suggesting the importance of evaluation to improve understanding, or there could already be sufficient information to suggest that the test should be strengthened in some particular way or that there are no clear weaknesses.

One way to find a good balance between an evaluation of the current test and beginning implementation of new approaches to assessment is to consider the examples of validity evidence discussed in Chapter 6 and the best practices for testing programs recommended by the professional standards. For example:

If the answer to either of these questions is “no,” then it makes sense to place more weight on the evaluation side, that is to first gather evidence to better understand how the current program is working. If the answer to these questions is “yes,” then there is probably already sufficient information to suggest particular ways that the test could be strengthened.

SOME CONSIDERATIONS ON THE EVALUATION SIDE

To the extent that FSI chooses to emphasize the evaluation side of the evaluation-implementation tradeoff, there are a number of important considerations. The discussions in Chapters 3 and 6 point toward a number of concrete questions that FSI could usefully further investigate related to the target language use domain and the different validity claims related to the current test. In addition, the possible changes discussed in Chapter 4 could each become a topic of evaluation, as an early step toward implementation. The exploration of new testing approaches on an experimental basis allows a testing program to better understand the tradeoffs of a change

Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×

before any major decision to implement those approaches for an entire assessment program.

Beyond the specific evaluation questions themselves, there are questions about the institutional structure that supports evaluation research at FSI and provides an environment that fosters continuous improvement. Many assessment programs incorporate regular input from researchers into the operation of their program. This input can include two different elements. First, technical advisory groups are often used to provide an assessment program with regular opportunities for discussion of technical options with outside researchers who become familiar with the program’s context and constraints during their service as advisors. Second, assessment programs also sometimes provide opportunities for researchers to work in-house as visiting researchers or interns to conduct research related to the program, such as conducting validity studies. Both of these routes allow assessment programs to receive new ideas from experts who come to understand the testing program and can provide tailored, useful advice. It is likely that there are constraints related to privacy and international security issues that could limit sharing data and publishing research on FSI outcomes, but it is possible that these constraints can be addressed with techniques to anonymize and share limited data for research. There also are costs associated with these activities, but many ongoing testing programs decide that these costs are outweighed by the long-term benefits of receiving regular input from outside researchers.

SOME CONSIDERATIONS ON THE IMPLEMENTATION SIDE

There are two salient constraints in the FSI testing program that are likely to strongly influence the consideration of possible new approaches. In the context of FSI’s current testing program, these constraints appear to be fixed. However, it is worth considering the possibility that these constraints may be more flexible than currently presumed.

The first constraint relates to the policy that all languages should be assessed using the same approach. The fairness concerns that provide the foundation for this policy are understandable, but the comparability of results from the testing process is what actually matters for fairness, not an identical testing procedure. If the fairness issue can be addressed, it may be possible to consider using different testing approaches across languages.

It is worth noting that FSI’s current assessment program already involves some limited variation in assessment procedures. The most prominent of these variations is the possibility for a test taker to interact with the evaluators over the phone or with a video conference rather than in person. In addition, in cases where only one assessor is available in a particular language, the assessor used for the test can be the test-taker’s instructor. The

Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×

general finding in the literature is that both of these variations can have an effect on assessment outcomes. However, in the context of FSI’s assessment program, these variations are accepted as providing scores that are sufficiently comparable to those provided in a standard in-person assessment with an assessor who does not already know the test taker.

The reason to consider using different approaches to assess different languages is the practical implications of the number of test takers. Some assessment techniques—such as technology-mediated approaches—have relatively high development costs but relatively low administration costs per person. Thus, such techniques may be cost-effective only for relatively high-volume tests. In considering possible new language testing approaches, FSI needs to decide whether the practical limitations that might prevent the use of some approaches for the low-frequency languages should automatically disqualify their use for all languages.

The second constraint relates to the role of the Interagency Language Roundtable (ILR) framework. As FSI considers possible changes to its language assessments, it may want to consider options—such as the use of multiple measures—that may be awkward to score directly in terms of the ILR framework. However, the use of the ILR framework for coordination of personnel policies across government agencies does not need to be interpreted as a constraint requiring the use of ILR skill-level descriptions for all aspects of FSI scoring. As detailed in Chapter 5, whatever assessment approaches may be developed can always be mapped to the ILR framework for the purposes of final scoring and the determination of language proficiency.

The committee appreciates that FSI faces complicated choices about possible changes to its language proficiency testing, and the agency’s interest in exploring the many aspects of modern language testing is commendable. The committee hopes that this report’s discussion of the research in the field contributes to FSI’s forward-looking decision process.

Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×

This page intentionally left blank.

Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×
Page 79
Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×
Page 80
Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×
Page 81
Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×
Page 82
Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×
Page 83
Suggested Citation:"7 Balancing Evaluation and the Implementation of New Approaches." National Academies of Sciences, Engineering, and Medicine. 2020. A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute. Washington, DC: The National Academies Press. doi: 10.17226/25748.
×
Page 84
Next: References »
A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The United States is formally represented around the world by approximately 14,000 Foreign Service officers and other personnel in the U.S. Department of State. Roughly one-third of them are required to be proficient in the local languages of the countries to which they are posted. To achieve this language proficiency for its staff, the State Department's Foreign Service Institute (FSI) provides intensive language instruction and assesses the proficiency of personnel before they are posted to a foreign country. The requirement for language proficiency is established in law and is incorporated in personnel decisions related to job placement, promotion, retention, and pay.

A Principled Approach to Language Assessment: Considerations for the U.S. Foreign Service Institute evaluates the different approaches that exist to assess foreign language proficiency that FSI could potentially use. This report considers the key assessment approaches in the research literature that are appropriate for language testing, including, but not limited to, assessments that use task-based or performance-based approaches, adaptive online test administration, and portfolios.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!