National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"1 Introduction." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

1
Introduction

Under the Immigration and Nationality Act, applicants for U.S. citizenship are tested on their knowledge of English and of U.S. history and government. While the number of people who apply for citizenship fluctuates from year to year, over the past decade there has been an average of about 800,000 applicants per year, most of whom end up taking the tests. Roughly 80 percent of the applicants become U.S. citizens by passing the tests and fulfilling other requirements specified in the legislation (U.S. Department of Homeland Security, 2003).

The U.S. Citizenship and Immigration Services (USCIS), an agency in the U.S. Department of Homeland Security, is responsible for the design and administration of the tests. In 2001, the agency began the process of redesigning the testing program due to concerns that the current testing procedure may not be sufficiently uniform, and that test content may not be appropriate to determine if applicants have a meaningful knowledge of English or of U.S. history and government (U.S. Commission on Immigration Reform, 1997).1

The naturalization tests affect large numbers of people, and any changes will be controversial. The process of redesigning the tests must therefore be based on the highest professional testing standards and a clear understanding of the impact of any changes. For these reasons, USCIS approached the Board on Testing and Assessment (BOTA) of the National Research Council to request independent advice about the redesign project.

COMMITTEE CHARGE

The Committee on the U.S. Naturalization Test Redesign was established by the board in April 2004 to conduct a multiyear study under the following charge:

The Committee on the U.S. Naturalization Test Redesign has been established to provide guidance that will help the U.S. Citizenship and Immigration Services (USCIS) optimize the validity, reliability, and fairness of the redesigned naturalization tests of English and U.S. history and government and to provide an overall assessment of that validity, reliability, and fairness before the nationwide implementation of the tests in 2006. The committee will review the ongoing design decisions of the project with USCIS, examine pertinent data about the possible options, discuss technical issues with USCIS as they arise during development, provide guidance drawn from the related research literature, and consider the impact of constraints and trade-offs on the appropriateness of the

1  

More background about the naturalization test redesign is provided on the USCIS web page: http://uscis.gov/graphics/services/natz/NatzTesting.pdf.

 

Suggested Citation:"1 Introduction." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

decisions that affect the validity, reliability, and fairness of the resulting tests. In its final report, the committee will evaluate the validity, reliability, and fairness of the tests as a means for certifying the knowledge and understanding of English and U.S. history and government of applicants for naturalization as required by the Immigration and Nationality Act. The committee’s conclusions and recommendations will take into account the practical, financial, legal, and political constraints that limit both the types of tests that USCIS can propose and the length and cost of the process for developing those tests.

Some of the issues to be considered include content and skill frameworks, test item development, test scoring and administration procedures, the comparability of test forms, study materials, performance standards, disability exemptions, and due consideration that takes into account the applicant’s age or background.

This report is the committee’s response to the first part of this charge. The committee’s recommendations are made to help optimize the validity, reliability, and fairness of the redesigned tests. The committee’s final report will respond to the second part of the charge by providing an assessment of the processes used to develop and evaluate the new testing program before its nationwide implementation.

COMMITTEE’S APPROACH

In making the recommendations contained in this report, the committee seeks to provide the formative guidance requested in the charge without becoming directly involved in specific decisions related to the test redesign in a way that would impair our objectivity about those decisions. The intent is to preserve the committee’s ability to serve an evaluative role in relation to the redesign project. This strategy will allow the committee to write a final report that provides an objective and credible assessment of the technical quality of the redesigned tests, as called for in the second part of the charge.

When tests are used to make high-stakes decisions about individuals, it is especially important that they meet high standards for validity, reliability, and fairness (National Research Council, 1999a).2 The naturalization tests have serious consequences for a large number of people, and the committee concurs with the observation made in its statement of work that “any changes will be controversial and must be based on the highest professional testing standards and a clear understanding of the impact of the new tests.” USCIS and MetriTech, its testing contractor, have provided the committee with information about the efforts to date to redesign the naturalization tests and to revise the procedures for their administration. The redesign team has explained the activities carried out thus far to determine the content and skills to be tested in history and government and in English proficiency, to pilot test new item formats for the English

2  

The committee recognizes that, technically, what need to be valid and fair are the interpretations and uses of test scores, not the test itself. Similarly, a test is not reliable in general, but rather, reliable for a certain population. However, for ease of communication, throughout this report we use the shorthand of referring to the validity, reliability, and fairness of the new tests, with the above meanings in mind.

Suggested Citation:"1 Introduction." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

test and plan for further research and pilot testing, to develop a study guide, and to seek stakeholder input.

The committee recognizes that these efforts represent a substantial investment in the redesign of the tests, and they begin to address some of the shortcomings of the current testing system. When explaining the shortcomings of the current system and stating the purpose for the redesign, USCIS has referred to the objectives of having a more standardized testing process and more meaningful test content. These problems need to be addressed in a way that meets professional testing standards; it is not sufficient to create a new testing program that is simply better than the status quo or the current process. We also recognize that any testing program is developed in a real-world context, constrained by existing legislation, budget, timeline, staff, and other resources. While test development often occurs in the context of limited resources, the goal nonetheless must be to follow a legitimate process that satisfies professional norms for testing practice. If current resources are insufficient to do so, then allocation of additional resources may need to be considered.

In short, the committee is concerned that the redesign program currently lacks the necessary advisory structure for making important decisions and a coherent, overarching research and test development design that will lead to a scientifically and politically defensible testing program. Our recommendations are intended to provide helpful guidance that, if acted on in the near term, will increase the eventual validity, reliability, and fairness of the new tests, as well as public support for them. The recommendations in this report may appear to add extra work and delay the timeline for test development, but the committee thinks that, ultimately, time and resources will be saved by taking the technically sound approach described in this report, thereby avoiding more costly pitfalls and setbacks further along in the redesign process.

GUIDE TO THE REPORT

The chapters that follow address two major sets of issues, under the headings of structure and process. By structure, we mean the mechanism to oversee the redesign effort and provide advice through boards or advisory committees, arranged by expertise and function. Not only would a sound structure bring in a variety of expert opinions to address complex issues, but it would also enhance the credibility of the project to stakeholders and the public. Process refers to the research design and sequence of test development activities, based on best practices in testing, to ensure that the resulting decisions about naturalization applicants are valid, reliable, and fair. While the exact nature of research and test development activities varies depending on a test’s purpose and the context in which it is developed, professional norms of good testing practice require that a series of general procedures be in place when developing high-stakes, standardized tests.

Although we make a distinction between structure and process in our discussion, we want to emphasize that because the two are integral to each other, they need to be designed and implemented jointly. Even the most well-considered research and development process cannot be sustained without an appropriate advisory and oversight structure. Alternatively, that structure would be an empty shell without a sound, overarching plan for research and development.

Suggested Citation:"1 Introduction." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 4
Suggested Citation:"1 Introduction." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 5
Suggested Citation:"1 Introduction." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 6
Next: 2 Structure of the Redesign Program »
Redesigning the U.S. Naturalization Tests: Interim Report Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!