The Bicentennial Census: New Directions for Methodology in 1990: 30th Anniversary Edition (2015)

8

Measuring the Completeness of the 1990 Census

RECAPITULATION OF MAJOR ISSUES IN COVERAGE EVALUATION

Chapter 4 described the methods used to evaluate the population counts in past censuses and appraised the quality of the various evaluation procedures. There is no need to repeat the detailed information of Chapter 4 in discussing the methods planned for 1990, but it is useful to summarize the main features of the evaluation methods. We shall also repeat key comments on this subject from the panel’s interim report (National Research Council, 1984).

The methods used by the Census Bureau, or suggested by others for use in evaluating coverage of decennial censuses, can be grouped into four types:

1. Pre- or post-enumeration surveys, such as the 1980 Post-Enumeration Program (PEP);
2. Reverse record checks;
3. Matching with administrative records, including multiple and composite list techniques; and
4. Demographic analyses.

We later suggest a fifth method for coverage evaluation, which we call systematic observation. Systematic observation is a close relative of ethnographic studies, or resident observation.

Starting with the 1950 census, the Census Bureau’s evaluation of coverage concentrated on the first and fourth methods above. A reverse record

check study was carried out in 1960 but its quality was judged too poor for it to be used. (By contrast, this procedure has been judged successful in Canada and considerable reliance has been placed on it.) Administrative list matching has been used for special studies relating to coverage evaluation, but not for the production of overall estimates of net undercount.

There are known weaknesses to each of these methods, at least in the way they have been carried out in the past. Through 1970, the Census Bureau’s judgment was that demographic analysis provided the best estimates of undercoverage, and these estimates were generally used in discussions of the undercount. Subsequent events, particularly the large level of presumed undocumented immigration, caused the Census Bureau to anticipate that this would no longer be true in 1980 and to rely on the PEP for coverage evaluation of the 1980 census.

Demographic analysis relies on estimates of populations independent of the current census, using such information as annual figures on births and deaths, immigration and emigration, and past census data. In earlier uses of the method, it was recognized that the net immigration statistics were somewhat shaky, but it was felt that a moderate error in this component could be tolerated without an important effect on the total estimate. However, by 1980, the uncertainty regarding the number of undocumented aliens in the United States changed perceptions of the accuracy of the independent population figures. New importance was attached to questions about the general quality of data on immigration and emigration. For the 1980 census, demographic analysis initially showed a net overcoverage of the white population, a result that the Census Bureau staff and most other analysts considered unlikely. The PEP and other survey-related procedures have the advantage over demographic analysis of providing subnational data, although cost constraints severely limit the number of areas for which separate estimates can be produced. (It is probably unrealistic to assume that reliable estimates will be available for more than at most 100 subareas of the United States.)

The most recent Census Bureau statements indicate that the Census Bureau intends for PEP-type surveys to be the basic evaluation tool in 1990. Demographic analysis will be continued, primarily for use in checking the reasonableness of the survey results for aggregate sex-age-race groups. The panel considers the Census Bureau to be acting prematurely in making a decision at this time for the evaluation method for 1990, particularly in light of the improvements that may be possible in other methods of coverage evaluation, as well as in the PEP. These possibilities are discussed in a later section of this chapter. We first repeat several assessments of evaluation methods from the panel’s interim report:

1. Each of the various methods currently used in the United States and other countries to measure the completeness of census coverage is

2. subject to serious limitations, including biases, in measuring the coverage of various population groups.

3. There is at present no reason to expect a breakthrough in the methodology of coverage evaluation before 1990. However, some significant improvements are possible, expected, and important.
4. There is, at this time, very little information on the quality of subnational estimates of coverage derived from any of the currently used evaluation programs.

These assessments are not meant to discourage evaluation efforts, but to encourage the Census Bureau to continue to explore methods of reducing the levels of uncertainty. One other general point about evaluation should be made. Information about the quality of the national census count is important in its own right. However, its value would be considerably increased if it could be used to modify population counts in subnational geographic areas. In Chapter 7 we have identified research whose successful completion might make it possible to use evaluation results for subnational adjustments. For such modifications to be of greatest use, the evaluation results should be known soon after the census is completed. While the accuracy of the evaluation methodology and ability to provide subnational estimates should be given the first priority, the ability to produce data quickly should also be an important criterion in choosing the evaluation methodology for 1990.

APPRAISAL OF CENSUS BUREAU PRETEST PLANS FOR COVERAGE EVALUATION

We begin by describing the current coverage evaluation research and testing program of the Census Bureau and the panel’s views toward these programs as expressed and updated from its interim report. Then follows a description and assessment of a recent Census Bureau position paper, by Kirk Wolter, on plans for coverage evaluation and adjustment in the 1990 decennial census.

Current Program for Testing and Research of Coverage Evaluation

1985 Pretest of Post-Enumeration Survey Methodology

The Census Bureau experienced a number of problems in conducting the 1980 Post-Enumeration Program, and it is planning a pretest in 1985 on post-enumeration survey (PES) methodology (Hogan, 1984a:Appendix A) to try to explore ways of overcoming some or all of these difficulties. The pretest involves selecting a sample of 200 blocks in Tampa, Florida, one

of the two 1985 pretest sites. The blocks will be completely relisted and matched to the pretest census records. The matching will be a two-way computer match between the sample and the census listings, which will enable the Census Bureau to estimate the overcount as well as the undercount. Nonmatches will be followed up using many different sources, for example, telephone directories, the post office, local welfare rolls, etc.

Problem areas that the Census Bureau identified for research are:

1. Computer matching;
2. Balancing the undercount with the overcount;
3. Evaluating the overcount;
4. Nonresponse research;
5. Alternate questionnaire design;
6. Rules on whether the current or the listed resident should be enumerated;¹
7. The use of the Post-Enumeration Program to benchmark other evaluation methods of interest;
8. Homogeneous domains and their effect on block sampling; and
9. Limited follow-up.

Originally the Census Bureau hoped to obtain information in the 1985 pretest about each of the problem areas listed above. Because many of them cannot be tested independently, the panel was concerned that the pretest might be unable to produce meaningful results for specific areas. There was some indication that the Census Bureau had not identified methods and criteria for the evaluation of some of the components of this test. Furthermore, the likely sample size was too small to identify the differences in alternative methods of estimating the net undercount because, in total, the undercount would probably be substantially less than 5 percent. Therefore, in our interim report, we recommended narrowing the scope of the 1985 pretest. The panel believed that priorities for the post-enumeration survey pretest should be based on an error profile of the Post-Enumeration Program in 1980, and the most promising improvements investigated. As a result of the panel’s recommendations, the Census Bureau decided to focus its pretest of Post-Enumeration Survey Methodology on the areas of computer matching and nonresponse.

__________________

¹ Rules on whether the current or the listed resident should be enumerated in the PES refer to the problem of movers and whether new residents or the residents listed as present on Census Day are counted.

Research Study on Hard-to-Count Groups

In this pretest, which is to run simultaneously with the Tampa post-enumeration survey, a sample from various administrative lists of males ages 18-40 and children under age 10 who live on 1985 PES blocks will be drawn in order to examine an administrative records matching approach to coverage evaluation of hard-to-count groups. The people found on these various lists will be matched to the 1985 pretest census and post-enumeration survey lists to see if they were included in either (no composite list will be created). People who do not match to either will be followed up for verification of address and other information. However, no tracing to determine the status of cases not living in the sample block at the time of the census will be done. The major objective is to determine if administrative records matching is feasible as a technique for improving coverage of the post-enumeration survey. The feasibility of this approach will be measured by the number of individuals located who were missed by the census and the PES, the political sensitivity raised by this operation, and the accessibility of the various list sources. The following administrative lists were initially under consideration:

1. The 1983 Internal Revenue Service Individual Master File;
2. Unemployment records;
3. Immigration and Naturalization Service files;
4. Job Training Partnership Act files;
5. Draft registration files;
6. Driver’s license files; and
7. Other lists, for example, police blotters or records of local hospital admissions.

Since this pretest will not form a composite list, there will be no testing of this important component of administrative list-based coverage evaluation programs. Many of the lists proposed for use (e.g., police blotters and unemployment records) have been tried previously with poor results (see Bureau of the Census, 1976a:2-8) and also pose problems of duplicates. In addition, the possible nonrepresentativeness of a composite list formed from these administrative lists will have to be accommodated if dual- or triple-system estimation with either the census list, or the census and post-enumeration survey lists, is contemplated. For these reasons, the panel recommended in the interim report against proceeding with this pretest until these difficulties were resolved. However, the panel is in favor of continued non-field-test research of this methodology. For example, the panel believes that research is needed for assessing the relative advantages of various alternative approaches to estimation of coverage of the total population using several administrative lists (see the discussion in Appendix 4.1).

As a result of the panel’s interim report recommendations, the Census Bureau focused its attention on a more limited number of administrative lists. Otherwise, the hard-to-count study is proceeding as described above.

The Forward Trace Study

The Census Bureau designed the Forward Trace Study (Hogan, 1984a: Appendix C) to test various methods for tracing people from their 1980 census address to their current address. The purpose is to determine which tracing method would be most effective for use in a reverse record check.

The success of the reverse record check in Canada has suggested the use of a similar procedure in the United States. A major difference between the United States and Canada in the application of this technique is the 10-year time span between censuses in the United States, compared with 5 years in Canada. This time difference increases the difficulty in tracing people from the previous census to their present residence. The Forward Trace Study principally addresses this time effect.

The Forward Trace Study began in October 1981 when a sample was selected from the 1980 census supplemented by a sample of missed persons derived from the 1980 PEP. A third sample of immigrants was added later. Unfortunately, problems arose in obtaining the fourth sample of intercensal births, due to the sensitivity of records for out-of-wedlock births and adoptions. The approximate sample sizes for the four subsamples are:

Three different tracing methods are being investigated: (1) periodic tracing with periodic personal contact, (2) periodic tracing with initial personal contact, and (3) tracing only at the end of the period. The three different tracing procedures will be compared for cost and completeness, especially for hard-toenumerate groups. One concern is that the people for whom the first tracing method is used may become sensitized to the census, and therefore may be enumerated with greater or lesser frequency than the general population. The extent of this sensitization would have to be well estimated in order to reliably estimate the degree of underenumeration from such a system.

The panel feels that the Forward Trace Study is likely to yield useful information as to the feasibility of using a reverse record check to evaluate the completeness of coverage of the 1990 decennial census, and therefore should be completed.

Description and Critique of the Wolter Paper

In October 1984, Kirk Wolter of the Census Bureau presented a position paper that represented both a change and a narrowing of focus for the research and testing of methods of coverage evaluation and adjustment for the 1990 census. Adjustment-related issues are discussed in Chapter 7. Here we discuss the issues related to coverage evaluation. In his paper, Wolter offered the possibility of major modifications to the Post-Enumeration Program used in 1980. In addition, he outlined the basis for the decision on whether to release adjusted data, at what time adjusted data might be released, and to what level of geographic detail adjusted data might be presented. We summarize this paper and the panel’s reaction to it.

1. Wolter suggested that the Census Bureau might use an independent survey, rather than the Current Population Survey (CPS), which was used in 1980, as the survey of the population of the United States for the Post-Enumeration Program in 1990.

There are many advantages to the use of an independent survey. Restrictions come with the use of the Current Population Survey, including the sampling design, the timing of the survey, the type of interviewing and follow-up used, the questions asked, etc. A survey dedicated to coverage evaluation will give the Census Bureau the opportunity to consider many possibilities, including: (1) the use of administrative records in frame development, (2) the use of a compact area sample design, and (3) the use of more intensive interviewing and follow-up techniques to reduce nonresponse. However, these freedoms bring with them certain disadvantages. The methodology underlying the Current Population Survey is well-tested. The interviewers are skilled at their jobs (it is suggested below that an independent survey use, wherever possible, Current Population Survey interviewers), and the frame development is well-understood. Moreover, the Current Population Survey, already budgeted, would avoid the possibly substantial costs entailed in developing and running a new sample survey of the United States.

2. The paper suggests that this independent sample be made up of compact area clusters, unlike the sampling design of the Current Population Survey.

The advantages of a sample of compact area clusters (such as entire city blocks) grow primarily from the ability to concentrate the enumeration and matching efforts on these small, geographically compact areas. Thus, two-way matching between the sample survey and the census records can

be contemplated. The inability to perform a two-way match was one of the major problems of the 1980 PEP program. In addition, small-area estimates of net undercount could be used in model development and validation with compact area clusters. An added possibility is the use of national and local administrative records in the same regions, also for purposes of model development and validation.

There are also disadvantages to this proposal. The measurement of undercoverage may not be an ideal application for a highly clustered sample design. If undercoverage is extremely homogeneous within clusters, the effective sample size achieved by clustering could be well below that of the 1980 Post-Enumeration Program, even though the same number of individuals was sampled.

On balance the panel favors proposals (i) and (ii) of Wolter’s paper, that is, the use of an independent survey, which samples compact area clusters, for use in the 1990 Post-Enumeration Program, particularly if subsequent testing shows the intracluster correlations to have only a moderate impact on the effective sample size.

3. Wolter strongly puts forward the post-enumeration survey as the key element of the 1990 coverage evaluation program, to the exclusion of methods such as administrative records, reverse record checks, and systematic observation.

Wolter presents many arguments for the discontinuance of research on a coverage evaluation program based on a reverse record check. The reasons given are: (1) the Census Bureau has little experience in running reverse record checks; (2) the program in 1990 would have to be experimental, since it would be the first time this method was used on this scale; (3) there have been problems maintaining current addresses for the sample created; (4) unexpected difficulties have arisen in acquiring birth records from the states because of the sensitive nature of these records; (5) the Forward Trace Study is as yet incomplete; and (6) all indications are that a reverse record check would be more expensive than a post-enumeration survey.

Throughout this report, one of the major themes has been the need for the Census Bureau, in its research and testing programs, to focus on priority areas, to the exclusion of less promising ideas. There are advantages to the narrowing of efforts, and coverage evaluation is certainly an area in which some narrowing is needed. Only in this way can the Census Bureau develop the expertise and assurance needed to implement successful coverage evaluation techniques. However, in this instance, the panel feels that the focusing is premature. The panel is of the opinion that the available information comparing the various approaches to coverage evaluation is inconclusive

as to the relative merits of these approaches. More information needs to be gathered before strong directions can be recommended. The panel has recommended a substantial amount of winnowing down elsewhere in the decennial census research and testing program to accommodate a liberal approach to research and testing here.

The objections Wolter presents to further investigation of the use of reverse record checks are not compelling. An experimental reverse record check was a part of the 1960 coverage evaluation program in the United States, and Canada’s experience cannot be disregarded. Furthermore, experimental programs can and should be used during the 1990 census so as to be ready for the census in the year 2000. Also, the serious problems associated with reverse record checks do not seem to be any more serious than those posed by the use of a post-enumeration survey.

As mentioned above, the post-enumeration survey has special problems with respect to certain populations. Reverse record checks, administrative list methods, and systematic observation are real possibilities for measuring undercount for these groups. The panel feels that the exclusive reliance on a post-enumeration survey methodology for coverage evaluation in 1990 is, at this time, premature.

4. Wolter emphasizes the necessity for the development of a fast and accurate matching algorithm whether or not the 1990 PES is to be used for adjustment or coverage evaluation.

Record matching forms an essential part of most of the existing workable coverage evaluation methodologies at this point in time, including a PES or a reverse record check. The panel is in full agreement with the spirit of the above statement emphasizing the importance of the development of matching capabilities. Much of the research ongoing at the Census Bureau to expedite the matching process is devoted to the development of algorithms for computer matching. The panel applauds these efforts.

Wolter’s paper bases a great deal of the adjustment decision on the successful development of a fast matching algorithm. To quote Wolter (1984:6, emphasis in original):

The strength of this statement necessitates some quantification of what a fast and accurate matching algorithm actually is capable of doing. Once this quantification has been made, if it then appears likely that fast and accurate matching will not be possible for 1990, we encourage the Census Bureau to investigate and develop possible fallback procedures that could then be considered for use.

5. Finally, as a first step in the process toward a decision on adjustment, Wolter calls for summarization of current evaluation studies from the 1980 decennial census.

The Census Bureau has completed a number of studies based on the 1980 census that, when summarized, promise to provide useful information pertaining to coverage evaluation and possible adjustment of future censuses. There are a number of other studies as yet uncompleted or unreported that would also yield important information on strategies for coverage evaluation. For example, the Census/CPS/IRS Match Study provides a three-way match that could be used to form estimates for certain subgroups of the population. Estimates using this three-way match might have smaller variance and possibly smaller bias than estimates using the two-way match performed in the PEP. Other studies, for example, the Demographic Analysis of National PEP Estimates, Local Area Estimation Research, and the Exploratory Analysis of PEP Data (Hogan, 1984a), have direct implications for the feasibility of adjustment procedures.

The panel supports Wolter in urging that the above summarization be prepared and that the Census Bureau allocate sufficient staff resources to this task. However, the panel is also concerned that important studies from the 1980 evaluation program may not be completed or fully documented. The results have potential implications with respect to the effective design of other field tests currently being planned. The panel has an overall concern that the history of tests completed by the Census Bureau has not always been available to help in the design and consideration of new tests.

THE 1990 DEMOGRAPHIC ANALYSIS PROGRAM: POSSIBLE IMPROVEMENTS

Demographic analysis requires data from sources, independent of the current census, to estimate the number of persons in a given age-race-sex category. The corresponding number recorded in the census can be evaluated by comparison with the demographic approximation. The simplest form of such analysis is illustrated by the construction of the estimated number of white females ages 20-24 in 1990 from:

1. The number of white female births from April 1, 1966, to April 1, 1970;
2. The number of white female immigrants from April 1, 1966, to April 1, 1990, whose age on arrival would place them into the target age group as of April 1, 1990;
3. The number of deaths prior to April 1, 1990, occurring in the United States to all white female residents born in the time period April 1, 1966, to April 1, 1970; and
4. The number of white female emigrants born during the target period. The group includes both persons born in the United States and those who migrated there.

The number of births is determined from birth registration data adjusted for the estimated proportion of underregistration; the number of deaths is the registered number; and the number of legal immigrants is derived from Immigration and Naturalization Service statistics. The number of emigrants is unknown and is estimated from a variety of fragmentary information, mostly from immigration data of other countries and cohort analysis of consecutive censuses.

This basic form of calculation is applicable only to persons born in 1935 or later, because well-founded estimates of completeness of registration of births begin in 1935. Other forms of analysis have been used for cohorts born before 1935. For persons over age 65, Medicare files provide reliable data on the size of that population. For persons between 45 and 64 for the 1980 census, and between 55 and 64 for the 1990 census, more complex procedures attempt to estimate the true size of a cohort at each census date by pooling information about the number of persons recorded in the cohort in several censuses, making allowance for the estimated differential overall completeness of different censuses, for broadly similar but

systematically evolving patterns of age misreporting, and for differential undercounts by age.

As indicated in Chapter 4, the main weaknesses of demographic analysis are the following:

1. No subnational estimates of undercount are available (and it is the geographically differential undercounting that leads to possible inequities in apportionment and fund allocation);
2. No estimate of the undercount for Hispanics can be constructed because Hispanic groups, until very recently, have not been identified in birth and death registrations and are not identified in immigration records;
3. It is necessary to use relatively crude and largely unverifiable methodology in estimating emigration;
4. There are no sufficiently accurate estimates available on the number of illegal immigrants; and
5. There are no available estimates for the reliability of the various component estimates.

Points (1) and (2) above, unlike (3) and (4), limit the available detail provided by demographic analysis but do not affect the reliability of the resulting estimates. Notwithstanding (3), (4), and (5), the method is generally thought to have provided better national estimates of undercounts by age, sex, and a limited breakdown of race for the censuses of 1950, 1960, and 1970 than did the post-enumeration surveys.

Demographic analysis was found less useful in evaluating the completeness of the coverage of the 1980 census. This is attributed primarily to the large number of unrecorded immigrants who are thought to have entered the United States during the 1970s (see Appendix 8.1). Another problem with the application of demographic analysis to the 1980 census is that the methodology of treating race, particularly Hispanics, was changed. This change created difficult problems of consistency with other data sources, including earlier censuses. The unknown number of emigrants continued to be a problem in 1980. Nevertheless, demographic analysis remained useful for those groups less affected by these shortcomings, particularly blacks. For blacks it is believed that demographic analysis provided a reasonable measure of undercount by age; however, it failed for whites and failed to provide estimates for Hispanics. Thus, it no longer provided reasonable measures of the differential undercount by race.

A useful modification of the procedure seems to be to apply demographic analysis separately to persons born in the United States and to the foreign born, provided the reliability of reporting of country of birth is high enough. One advantage is the potential availability of good national

estimates by age and sex for the native white and native black populations, at least up to age 55 (i.e., for persons born in 1935 or later). The only estimates of international migration needed for this group are allowances for the movements of persons born in the United States. A portion of this movement could be inferred from immigration information from other countries. Estimates of emigration could also be derived, as a by-product, from a reverse record check, if one is carried out in conjunction with the 1990 census, or perhaps by a multiplicity-sampling approach incorporated in the Current Population Survey. If this modification is successful, the resulting demographic estimates could presumably be used to check the results of a reverse record check or a PES, or could be used as a benchmark for those methods as they relate to persons born in the United States.

The Census Bureau should investigate the value of the native-born approach to modifying demographic analysis. Of course, an analysis of the quality of the information on reported place of birth would be required. The value of PES information on place of birth should also be investigated.

THE REVERSE RECORD CHECK PROGRAM: CONSIDERATIONS FOR 1990

A reverse record check methodology has been used by Statistics Canada since 1961 in its assessment of the completeness of the coverage of its censuses. This procedure is described in Chapter 4, so we summarize only the basic methodology here.

A reverse record check is an evaluation program in which a sample of the population is drawn from a frame created prior to the census and traced forward to the time of the census. The proportion of the sample that is determined through tracing to be residing in the United States on Census day provides an estimate of the total population. Usually, the sample is a combination of samples from the following four lists: (1) the previous census, (2) births in the intercensal period, (3) immigrants from the intercensal period, and (4) people missed in the previous census as determined from the previous coverage evaluation program. This technique has not been used extensively in the United States.

Compared to post-enumeration surveys of the kind conducted by the United States to evaluate its censuses, the reverse record check seems to offer several advantages:

1. Unlike the “do it again, but better” method, it does not rely on the assumption that the post-enumeration survey might succeed very much better where the census failed. And unlike the “do it again, but independently” method, it does not have to rely on the unverifiable and unlikely assumption that the events of being missed by the census and by the post-enumeration survey are independent.
2. The coverage error estimates do not depend in a major way on matching errors—a significant point of vulnerability of “do it again, but independently” methods of the type carried out after the 1980 census and planned for the 1990 census.
3. The reliability of the 1980 census coverage evaluation is significantly affected by nonresponse in the post-enumeration survey. There is no nonresponse in the reverse record check per se. There is an analogous category of tracing failed—but here again, the reverse record check has some advantages in that the tracing of a small number of residual cases can be (as it is in Canada) carried out over several months, as opposed to the tight time schedule of the field work of post-enumeration surveys.
4. Imputation in the post-enumeration survey cannot be validated. By contrast, inputting for tracing failed cases can be partially assessed by reference to independent control totals. Indeed, the reverse record check provides an estimate of the number of persons who died since the previous census—a verifiable number. After matching with the census, the method also provides an estimate of the number of persons enumerated in the census—another verifiable number.
5. The reverse record check provides a direct estimate of the number of emigrants since the last census, which can be used to overcome one of the significant data gaps of demographic estimation—both to evaluate the current census and as a benchmark for its intercensal population estimation.

One problem for the reverse record check method is the lack of records for undocumented aliens, so that they cannot be represented in the reverse record check sample. Another significant disadvantage of the reverse record check is the need for the tracing operation. However, with a 5-year gap between censuses, the 5 percent tracing failed rate achieved in Canada compares favorably with the over 8 percent imputation needed in the 1980 evaluation program used in the United States.

The panel believes the Census Bureau’s experimental initiative called the “Forward Trace Study” may provide some information as to ways of overcoming the problem posed by the 10-year intervals between censuses in the United States. As discussed above, the Forward Trace Study is test-

ing three modes of tracing a sample of individuals counted in the previous census, a sample of individuals missed in the previous census, and a sample of intercensal immigrants. The outcome of this study may help determine an effective method for tracing people in the United States. As indicated in Recommendation 8.1, the panel is concerned that a reverse record check be given more attention as a potential coverage evaluation methodology in 1990. Assuming that Recommendation 8.1 is persuasive and the decision is made to proceed in 1990 with a reverse record check in either a testing mode or as a primary coverage evaluation program, it is then necessary to know very soon which of the three versions of tracing will be used. If it happens that either of the methods for more intensive tracing in the Forward Trace Study wins out over tracing at the end of the period, the intensive tracing must begin by 1986 in order to benefit from the shortened period between contacts. Therefore the samples need to be drawn by 1986.

THE 1990 POST-ENUMERATION PROGRAM: POSSIBLE IMPROVEMENTS

Recent Census Bureau reports indicate that a type of post-enumeration survey will be the predominant component of the coverage evaluation effort in the 1990 decennial census, as it was in 1980. Assuming this and given the weaknesses of the 1980 version of this program outlined in Chapter 4, what possibilities are there for improvement in the Post-Enumeration Program for 1990?

There are two purposes for which a post-enumeration survey might be used. The first is to evaluate coverage, for example, to identify subgroups of the population, by state and major city, that were disproportionately missed by the census. The second is to use the results for purposes of adjusting the population counts of states, major cities, and smaller geographic regions. These two purposes of coverage evaluation and adjustment overlap to a considerable degree. It is this second purpose, adjustment, on which we concentrate. We consider possible areas for improvement to the techniques of the 1980 Post-Enumeration Program; however, any improvements to the Post-Enumeration Program as a potential adjustment program are clearly improvements to it as a coverage evaluation program.

We organize this section as follows. First, we provide a description of the general procedure used in the 1980 Post-Enumeration Program. Then

we identify the features of the Post-Enumeration Program in which worthwhile gains appear to be possible. For each feature identified, possible approaches for improvement are discussed.

The 1980 Post-Enumeration Program

As a coverage evaluation program, the 1980 Post-Enumeration Program was useful in identifying demographic subsets of the population, by state and major city, that were disproportionately missed by the census. For example, the 1980 Post-Enumeration Program indicated that, nationally, blacks and nonblack Hispanics were missed more frequently than whites. In addition, the PEP provided considerable information about erroneous enumerations, duplications, and incorrectly geocoded addresses, which indicated limitations of the decennial census methodology (see Wolter, 1983; Cowan and Bettin, 1982). Thus, the Census Bureau derived a substantial amount of information on the quality of the 1980 decennial census dataset as well as information about which populations to direct its energies to for coverage improvement in 1990. In this sense, the 1980 Post-Enumeration Program can be seen as a continuation of, and improvement on, methods used for coverage evaluation in the 1950, 1960, and 1970 decennial censuses.

Chapter 4 contains a detailed description of the 1980 Post-Enumeration Program. However, for convenience, we repeat the overall strategy here. The basic idea was to recount independently a sample of households, and subsequently match individuals included in the two enumerations to determine those missed by the census but included in the recount. An estimation model, often referred to as capture-recapture, or dual-system estimation, was then applied to supplement the direct coverage estimates by adding an estimate of the number of individuals missed by the census. The Current Population Survey was the enumeration system used to perform this recount in 1980, and in this context was called the P sample. Although the sampling frame for the Current Population Survey is not independent of the decennial census, it undergoes sufficient changes over the intercensal period so that the listing of addresses used is fairly distinct from that of the decennial census (see Bureau of the Census, 1978a). This along with the independence of surveying operations in the Current Population Survey and the census helps promote the desired independence. The P sample included about 185,000 persons each for April and August 1980.

It was possible to search the census files for matches of individuals enumerated in the Current Population Survey. However, the search had to be restricted to a limited geographic area. Thus, a person counted by the census within the “wrong” area (as per Current Population Survey definitions and operations) appeared at the conclusion of this match as if he or

she were missed by the census. Due to the sampling design of the Current Population Survey, which did not make use of compact area clusters, it was essentially impossible to search the Current Population Survey files for individuals counted in the census. (This would then be a two-way match.) As a result, there was no mechanism in the P sample, by itself, for checking the validity of census enumerations. Invalid or erroneous census enumerations include not only improperly geocoded census addresses, but also curbstoning, individuals who should not have been included in the census, such as foreign visitors and people who were born after Census Day, duplicate enumerations, etc. The need to measure the frequency of these problems gave rise to a second sample, this time a sample of 100,000 individuals from the decennial census itself, called the E sample. The latter sample was used partly to “balance out” from the P sample the contribution of persons included in the census but at the wrong address and partly to estimate the number of persons erroneously included in the census—in order to derive, with dual-system estimation, net under- or overcount estimates. We note that there may be less need for the E sample in 1990 due to the possibility (mentioned above) of the use of a sample of geographically compact clusters for the PES, since in that case two-way matching may be feasible.

Improving the 1980 Post-Enumeration Program Methodology

We have identified four aspects of the 1980 Post-Enumeration Program that might benefit from special attention, although we do not necessarily have unambiguous recommendations to offer in every instance:

1. Reduction in the level of survey nonresponse;
2. Reduction in the percentage of unresolved matches;
3. Improvement in methods to balance the local undercount with the overcount; and
4. Estimation of the degree of independence between survey and census.

Reduction in the Level of Survey Nonresponse

Like any sample survey, the survey used for the Post-Enumeration Program will suffer from an imperfect sampling frame and interview refusals. As mentioned in Chapter 4, over 4 percent of the Current Population Survey interviews in April 1980 were refusals. Even when an interview is conducted, a lack of detailed information on address or to a lesser extent age, sex, and race for a record can create situations in which the status of a match with the census is unclear. The resulting problem of a large percentage of unresolved matches is addressed in the next section. Here we are

concerned with people in the sample for the PEP, or who would have been in the sample had the sampling frame been complete, for whom no information was collected. This is a central issue since there is a possibility that the same types of people who are missed in the census are either missing from the PEP sampling frame or refuse to cooperate with the PEP interviewer.

It would, obviously, be highly desirable to decrease the rate of refusal. In 1980, the Census Bureau used the April and August Current Population Survey samples as the P sample of PEP and utilized essentially the same Current Population Survey procedures as for other months. Therefore, one possibility for reducing refusal, assuming that the Current Population Survey is again used for the PEP, is to employ more intensive follow-up than is usually done, perhaps after the end of the regular survey week of the CPS. The possibility of making cooperation legally required should also be explored. This approach may introduce a discontinuity into the time series of employment and unemployment estimates, although this risk might be reduced by appropriate measures. An alternative, currently under consideration by the Census Bureau and discussed above in the critique of the paper by Kirk Wolter, is the use of a separate survey for coverage evaluation. Should a separate survey be used, it might still be highly desirable to employ experienced Current Population Survey interviewers during the non-CPS weeks.

Reduction in the Percentage of Unresolved Matches

In the 1980 Post-Enumeration Program, after completing a Current Population Survey interview, Census Bureau staff geocoded the address of each sample residence to determine the enumeration district in which that residence should have been placed in the 1980 decennial census. Then that (and only that) enumeration district was searched clerically for a name-address-race-sex-age combination that matched, according to defined criteria, a record from the Current Population Survey. Each Current Population Survey interview was categorized in one of three ways: matched with the census, not matched with the census, and match status unresolved. This last group is the most troublesome, at least if one assumes that errors involving the first two categories are well controlled. These cases can easily give rise to very significant matching errors, and hence errors in the estimated undercount.

When the April 1980 Current Population Survey was used, matching status could not be determined for approximately 8.5 percent of cases. This was due to a variety of causes, especially incomplete responses, response errors in either the census or the CPS, refusals to respond to the Current Population Survey, and ambiguities related to addresses (particularly in rural areas). Use of the August Current Population Survey resulted in

over 10 percent of cases with unresolved matching status, larger than the April CPS presumably because of the problems introduced by mobility (see Wolter, 1983). In order to derive estimates of the number of persons missed, a match status had to be imputed to the unresolved cases. Over 30 percent of the April imputations resulted from an incomplete follow-up interview of CPS interviewees who were not initially matched with the census.² Depending on the method of imputation used (combined with some other factors), the Census Bureau generated 12 different sets of PEP estimates of the undercount for states and major cities. These estimates appeared sensitive to the method of imputation (and other factors) used (see the discussion in Chapter 4).

Improvements to the geographic system in 1990 may be helpful in reducing the number of unresolved matches. The Census Bureau’s TIGER (Topologically Integrated Geographic Encoding and Referencing) system, currently under development, could very well represent a substantial improvement over previous geographic systems, and could be in place by 1990. However, any resulting benefits from this new system would be dependent, to a large extent, on the quality of the responses that are to be coded. There are also efforts by the Census Bureau to avoid the necessity of geocoding, by treating the address as an alphanumeric response which can then be used to block the census dataset for matching in ways that do not require knowledge of the precise enumeration district of the individual.

Fractional matching is an idea that could be explored as an alternative method of inputting match status to cases for which the match status was unresolved. Assume that the likelihoods outlined in Appendix 4.2 from the Fellegi-Sunter mathematical model for matching are stored and available for the cases that are left unresolved, along with the several likely matches for these unresolved cases. It is conceivable that a function of these likelihoods could be developed empirically that would impute to each unmatched post-enumeration survey record (by computer algorithm and suitable personal follow-up) a fractional match status in such a manner that the sum of these fractions is equal to the unknown number of matched cases. Fractional matching is therefore merely a model relating match status to likelihoods from some model, for example, the Fellegi-Sunter model. Assuming a computer matching success rate of 60-70 percent—perhaps an optimistic rate—this, without clerical assistance, would result in a massive imputation of match status. Given the substantial impact on undercoverage estimates of imputing match status to only about 8 percent of cases, such a major increase in the reliance on imputation cannot be recommended on the basis of our current state of knowledge. However, there is the possibility of using fractional matching solely for those cases for which the match

__________________

²From conversations with Robert Fay, III.

status is either very likely or very unlikely, leaving the remainder for clerical follow-up. Finally, its use to impute match status to the residual number after clerical follow-up of unresolved cases should be explored.

Another suggestion that has been made is for the Census Bureau to subsample the unresolved cases in order to concentrate efforts on them. There are two possible applications of this idea. The first is to sample from all cases unmatched by the computer algorithm. Not all members of the panel favor this idea. The use of sampling of all the matches left unresolved by the computer algorithm would result in estimates of undercoverage with substantially increased variances for important subpopulations and subnational regions. The second notion is to sample after a first-stage personal follow-up of unmatched cases has been attempted. The advantages of this approach parallel those discussed in Chapter 6 on sampling for nonresponse follow-up. The full panel endorses this idea.

We point out that the use of especially intensive interviewing, discussed in the previous section, should improve the reporting of identifying information and hence might reduce the problem of individuals with unresolved match status. Finally, the use of computer matching might permit an extension of the area of search within the census file for each PES sample case, as well as the use of more matching variables and more advanced methodology. These improvements may well result in a significant reduction of the nonmatch rate.

Improvement in Methods to Balance Local Undercount with Overcount³

In the 1980 version of the Post-Enumeration Program, the E sample was used to estimate the genuine overcount of the census. It was also used to offset the Current Population Survey sample cases that could not be matched to the census within the local area to which matching was restricted, often as a result of faulty determination of census geography, or indistinct addresses.

As described above in Chapter 4, the form of the dual-system estimate used in the 1980 Post-Enumeration Program for a particular demographic stratum was as follows (see Cowan and Fay, 1984):

where N_T is an estimate of the total population, n_P is the weighted sample total of the number of persons in the P sample, m is the weighted number of

__________________

³The terms undercount and overcount are understood here to mean gross undercount and gross overcount.

persons who are in both the census and the P sample, nc is the census count of persons, e is the weighted number of persons who were census erroneous enumerations from the E sample, g is the weighted number of persons in incorrectly geocoded housing units in the census from the E sample, d is the weighted number of duplicate counts in the census from the E sample, and i is the count from the census of field-related imputations.

The four subtracted quantities are therefore: (e) people who were counted in the census who should not have been, for example, people born after Census Day; (g) people who were counted in the census but placed in the wrong area and therefore, given the blocking used in the clerical match, were incapable of being matched; (d) people who were counted in the census more than once in the same enumeration district; and (i) people who were imputed into the census, for example, people for whom no questionnaire was returned or residences that were imputed to be occupied. An error for these four quantities substantially less than the magnitudes being measured is necessary for a reasonable estimate of those missed in the census, since the magnitudes of the quantities being added and subtracted is of the same order as that of the undercount. From Cowan and Fay (1984) we have the national percentage rates for the above four quantities:

The question thus becomes how can this balancing of the undercount and the overcount be reduced or eliminated from the Post-Enumeration Program estimation process?

The Census Bureau has recently advanced one possibility, discussed above, for avoiding the necessity of balancing (see Wolter, 1984). The idea is to use an independent survey in the Post-Enumeration Program in place of the Current Population Survey. The independent survey would sample geographically compact clusters and check for both over- and underenumeration in the same clusters using a two-way match. With a two-way match: (1) census duplicates are easier to find by checking post-enumeration survey records that match with more than one census record and (2) census erroneous enumerations and misgeocoded records can be estimated from an examination of census records that do not match to any post-enumeration survey records. This avoids the need for intricate assumptions of balancing errors. In addition, local area estimates of net undercount could be exploited in model development (through the use of local area characteristics as auxiliary variables) as well as in model validation (by comparing for a subsample of small areas the direct and model-

based estimates of undercount). As indicated above, the panel is in favor of this proposal.

Estimation of the Degree of Independence Between Survey and Census

A major and untested assumption of the 1980 Post-Enumeration Program is that, for each person, the events of being included in the census and the Current Population Survey are independent. However, there is evidence supporting the belief that many of the types of individuals missed in the census are also missed disproportionately by the Current Population Survey and, for that matter, by any type of household survey technique. For example, the CPS estimates of young males, particularly blacks, are consistently below the corresponding demographic estimates. The ethnographic study sponsored by the Census Bureau provided additional evidence of this phenomenon (see Valentine and Valentine, 1971). This lack of independence of inclusion in the evaluation survey and the census may be particularly likely for persons with tenuous or irregular household connections, for undocumented aliens, and for other groups who have reason to avoid visibility of any sort. For these people, the frequency of being missed by both the census and the survey may be substantially different than would be indicated if these events could be regarded as probabilistically independent. (Equivalently, the probability of inclusion in the census, given inclusion in the post-enumeration survey, may not equal the unconditional probability of inclusion in the census.) Thus k, the parameter mentioned in Chapter 4, may be substantially different from 1.

As mentioned in Chapter 4, the Census Bureau makes use of a stratified dual-system estimator, that is, the population is first stratified using certain demographic characteristics, then the dual-system estimator is applied separately within strata. This serves two purposes. First, the k’s for each subtable formed with the stratification may all be closer to 1 than for the unstratified case. (However, the dependence is still likely to be substantial.)

Second, the stratification helps keep the probabilities of inclusion constant within strata. This is another assumption often used in the model underlying dual-system estimation. These two assumptions—independence of inclusion probabilities for the two lists, and the equality of the inclusion probabilities within one list (within strata)—both need to be carefully studied. These assumptions are at least partially confounded. Thus, any study of the degree of validity or robustness of the independence assumption will be enhanced by simultaneously studying the degree of validity or robustness of the equality assumption. It is possible to examine the individuals missed by the census to see whether they differ with respect to various covariates not used in the determination of the strata. The extent of

the differences would then be a test for equality of inclusion probabilities. However, if the individuals examined are only those caught by the post-enumeration survey, the results may be affected by any nonindependence between census and post-enumeration survey. Therefore, an important factor is the gathering of information for individuals missed by both the census and the post-enumeration survey.

A method that should be tested for its potential to count some of the individuals who typically escape the counting method used by censuses and surveys is the reverse record check. Systematic observation, discussed below, should also be tested for this purpose. Within the context of the Post-Enumeration Program itself, an approach that deals with certain aspects of this exceptionally difficult problem is triple system estimation (see Marks et al., 1977, and Appendix 4.1). In this approach, the independence assumption for two lists is often replaced by an assumption of conditional independence involving three lists. Validation of the assumption of conditional independence would be needed. Unfortunately, there is at present no “third” system with a reasonably complete coverage of the population of the United States. A union of suitably selected administrative records might be envisaged, but various problems, outlined in Chapter 4, make this possibility appear unlikely for the immediate future.

Ericksen and Kadane (1985) and Fellegi (1985) emphasize the sensitivity of dual-system estimation to the assumption of the independence of inclusion frequencies. Ericksen and Kadane (1985) propose a method that may be applicable to some special groups. They argue that, for blacks in the 1970 census, the probability of inclusion in the census, given inclusion in the post-enumeration survey was not equal to the probability of inclusion in the census, as the assumption of independence would indicate, but was instead greater than twice the probability of inclusion in the census. The method used assumed that the demographic estimate of the national black undercount was correct for 1970. The general applicability of this approach is limited since the estimation of k would require knowledge of the total population—which is the end objective in wishing to estimate k in the first place. Furthermore, Fellegi (1985) argues that the numerical stability of their estimate is not good. Nevertheless, the panel supports the call of Ericksen and Kadane for further research to understand the degree of dependence that exists for various subpopulations and for various lists or surveys, for example, how k depends on the list, in addition to the census, that is used and on the population studied.

1. Reduction of post-enumeration survey nonresponse;
2. Reduction of unresolved matches between records for individuals listed in the post-enumeration survey and the decennial census;
3. Validation of the assumptions and/or development of alternative methodologies with respect to netting-out of overcounts and undercounts with reference to the place of enumeration; and
4. Investigation of alternatives to the assumption that the inclusion of individuals in the post-enumeration survey is unrelated to their inclusion in the decennial census and the estimation of the strength of this relation.

Some Remaining Considerations

Below we consider two remaining problem areas of the 1980 PEP program, timeliness and variance estimation, and discuss the current approach of the Census Bureau to their resolution. The panel has no recommendations to offer here other than endorsing the efforts of the Census Bureau.

Timeliness. One of the most important aspects of a potential adjustment program, resulting from the current deadlines for reapportionment and redistricting, is the timeliness of the program. In 1980, even preliminary estimates were not available from the Post-Enumeration Program until late in 1981. Apart from other considerations, this factor alone caused the results to be unusable for some purposes of adjustment. There is consequently a substantial interest in speeding up the Post-Enumeration Program process, without compromising its quality. In fact, one of the key elements now under investigation by the Census Bureau, and mentioned prominently in the position paper by Kirk Wolter, is testing of the operational feasibility of an adjustment by December 31, 1990. The possibility of meeting such a deadline would be enhanced by the use of a pre-enumeration survey and extended use of automation, both under consideration by the Census Bureau.

Use of a pre-enumeration survey. In order to boost the total sample size, the 1980 Post-Enumeration Program made use of the April and August Current Population Surveys, which served as more or less independent post-enumeration surveys. It has been suggested that earlier months of the Current Population Survey could be used, which would be ready for matching at the time of the decennial census. The January through March Current Population Surveys would be possibilities, with March having the additional advantage of containing a wealth of characteristics information that could be used for purposes of content evaluation and possibly modeling. Even if the Current Population Survey is not used in the 1990 PEP, the timing of the PEP will involve similar considerations.

The advantage of the use of a pre-enumeration survey is the possibility of having the survey files ready and waiting for the creation of the decennial census files. Even so, an appreciable fraction of the matching could not be done until personal follow-up was completed.

A possible disadvantage of a pre-enumeration survey is the potential sensitization of the population to the decennial census. As a result of the survey experience, the pre-enumeration survey interviewees would probably be more aware of the upcoming decennial census than the general population, and this may affect their actions regarding inclusion in the census. (It is not clear whether this is likely to lead to a greater or lesser desire to be enumerated.) However, sensitization is also possible with the use of a post-enumeration survey, since the taking of the census may affect cooperation with the survey. Sensitization could be reduced by the use of a survey that either precedes or succeeds the decennial census by a longer time period, say, 1 or 2 years. However, the panel has strong reservations about that idea. As the time period between survey and census lengthens, population mobility, deaths, etc., are likely to increase problems of accuracy.

The relative trade-off between a possible sensitization of the population versus the early preparation of pre-enumeration survey files to be matched to the census is at this time unknown. This is an area in which research is needed; it is a major part of the Census Bureau’s 1986 pretest program for coverage evaluation methodologies.

Automation. There are currently a number of field tests planned by the Census Bureau to determine the most effective use of new automation technologies for information collection, transfer, storage, and retrieval (see Chapter 3). To date, these tests have concentrated on the roles of collection offices, processing offices, and logistics. Of key importance from the point of view of coverage evaluation are attempts to generate, very early on, machine-readable records of the basic identification of enumerated persons and households, adequate for computer matching.

In order to exploit the potential existence, at an early date, of both census and post-enumeration survey records in machine-readable form, effective computer matching algorithms have to be developed. In 1980, the matching was done clerically, a slow process that limited the search to one enumeration district for each CPS record. In trying to improve the timeliness of Post-Enumeration Program estimates, the Census Bureau (see Wolter, 1984) is placing a great deal of emphasis on its ability to develop software for automated matching.

The algorithm used by the Census Bureau (see Kelley, 1984a; Jaro, 1985) for computer matching was discussed in Chapter 4 on matching procedures. We are not recommending any modifications to that basic strategy. However, we do have one suggestion that the Census Bureau may

wish to investigate further. The idea is to utilize computers to assist the clerical matching. A large proportion of cases unresolved by the computer matching algorithm take the form of records in one or the other of the two files having a multiplicity of possible matching cases in the other file, but with inadequate evidence to make a unique status assignment by computer. Such cases can be presented to clerks on display terminals in a split-screen fashion for visual inspection and decision. Some proportion of cases will still remain unresolved because the reported information is inadequate for match status determination. However, the efficiency and speed of dealing with clerically resolvable cases should be greatly enhanced. (A recent paper indicates that the Census Bureau is already planning something quite close to this; see Jaro, 1985).

Automation will have a much greater impact on matching operations at the Census Bureau than merely speeding up the processing. For example, it might allow the possibility of searching a wider geographic area for a matching record, and hence lessen reliance on the need for finely balancing local over- and undercounts.

Estimation of variance due to matching. Should a 1990 version of the Post-Enumeration Program be used to adjust the population counts, it will be important to derive estimates of the error attributable to various causes, including matching. We concentrate here on the estimation of the variance of the matching process.

Matching can be considered to consist of three phases: an initial computer match, a subsequent clerical operation to resolve the more difficult cases, and imputation for cases whose match status could not otherwise be resolved. Given two files to be matched, the very nature of a computer algorithm is such that, conditional upon the files and the computer algorithm, there is no variance. Of course stochastic response errors in both the census and the post-enumeration survey will undoubtedly induce some matching variance. The estimation of this variance is technically feasible but would probably introduce serious operational difficulties when superimposed on the other rigorous requirements of a coverage evaluation program. The rough magnitude of this variance might, however, be estimated using some intercensal experiments.

For the component of the matching that is done clerically, a combination of designs involving interpenetration of a sample of matching clerks, together with some rematching, can readily be established. The design can be fully analogous to the estimation of interviewer and response variance (see Hansen et al., 1971; Fellegi, 1964).

SYSTEMATIC OBSERVER METHODOLOGY: CONSIDERATIONS FOR 1990

It is generally recognized that a serious undercount problem exists for some members of poor minority groups living in large central cities. There are also indications that in these areas the largest number of individuals are missed through incomplete reporting of household members rather than through failure to enumerate the households themselves. In particular, demographic studies using sex ratios seem to indicate that a disproportionate number of adult black males are missed by the census. Other studies suggest that it is unrealistic to expect improved traditional interview or self-enumeration procedures to increase substantially the coverage of such individuals. Finally, it is also for such individuals and for such areas that the Census Bureau has experienced the greatest difficulties in using matching to estimate undercount. These general perceptions, in conjunction with the resident observer study of Charles and Betty Valentine described in Chapter 5, provide the motivation for this section, which outlines a research program aimed at finding out who is missed and at developing a procedure to estimate the number of individuals missed.

In 1972, the Census Bureau asked the Advisory Committee on Problems of Census Enumeration of the National Research Council to assess the Valentine study. The committee reviewed the study and suggested that the Census Bureau continue to support such studies. The Census Bureau contacted additional anthropologists and undertook to support graduate student participant observer studies. For a number of reasons, including personnel problems, none of the studies was completely successful. All of them took the form of Census Bureau support for graduate students in a graduate program at a university.

We believe in the potential of a trained individual, through normal, day-to-day encounters, to become aware of people in his or her neighborhood who would be difficult to enumerate through typical census procedures. There is a major difference between an effort of this kind and the anthropological studies such as the Valentine study. In the Valentine study, a considerable amount of personal information, such as sources of income and personal relationships, was obtained (and kept confidential, of course). In the type of study proposed in this section, the only information obtained would be name, age, race, sex, and address. This difference in degree of invasiveness might prevent the occurrence of the problems experienced in resident observer studies conducted since the Valentine study. The proposed study makes use of a type of enumeration similar to that used in Casual Count, described in Chapter 5. The different objectives of the proposed study require the use of a term different from the anthropological one of resident observers. We have adopted the term systematic observers.

Research activity on systematic observation can be coordinated with pretests being conducted for the 1990 census, but such research is not restricted to pretest activities. Consistent with the two terms resident observation and systematic observation, we envision two possible types of studies. In resident observation, similar to the Valentine study, anthropologists work in an area on an essentially full-time basis for a considerable period of time. In such studies, highly trained professionals attempt to identify the reasons for noncompliance and misreporting as well as to quantify the magnitude of the problem. The identification of the reason for noncompliance, especially with respect to different population subgroups, is vital for understanding how coverage improvement and coverage evaluation might be improved to help minimize the problem of differential undercoverage. Observers of this type could be placed in a number of different types of localities. Brooks (1974) suggested that research could profitably be conducted in the following types of study areas:

1. A Mexican-American community in the Southwest;
2. A transplanted Appalachian community in the urban north central region;
3. An urban black community in the north central region;
4. A northeastern Puerto Rican community;
5. A northeastern black urban community;
6. A Navajo reservation;
7. A black southern rural community;
8. A white ethnic community; and
9. A white or mixed southern urban area.

Resident observer studies might provide information leading to the development of alternative data collection procedures.

The second type of activity, called systematic observation, would employ less highly trained professionals. The observers would live in the area and become familiar enough with the residents to make reliable reports on the number of persons in each of a small number of households at a particular date, as well as their name, age, sex, and race. It is conjectured that this activity would require only a fraction of an employee’s time. By initiating several systematic observer studies at the earliest possible time, the following questions can be investigated:

1. How difficult is it to recruit, train, and position systematic observers?
2. How long must systematic observers reside in an area before they can provide reliable data on residents?
3. How large an area (number of households) can a systematic observer be expected to provide reliable data for?

4. What procedures can be used to validate the quality of the data provided by the systematic observers?
5. Are different procedures required in different types of areas?
6. Can problems of perceived invasion of privacy be overcome?

Naturally, the results of a research program are unknown at the present time. However, to make clear the nature of our objectives, we outline a possible scenario for the use of systematic observers. The first step in the process would be the delineation of the area of study. This would include, but would not necessarily be limited to, the low-income areas of large central cities. The Census Bureau would draw an area sample of segments, each containing, say, 20 housing units. The Census Bureau would then recruit a full-time Census Bureau employee to live in or near each sample segment for a period of, say, 1 year beginning at least 9 months prior to Census Day for the 1990 census. The individuals recruited would be employees of the Census Bureau, and as such would be sworn to uphold the confidentiality of the information collected, and would be subject to fines and imprisonment for any betrayal of that responsibility. In those areas in which the Census Bureau had offices, the individuals could spend part of their time as office employees of the Census Bureau. A condition of their employment would be that they live in the study area and that they become knowledgeable about the nature and composition of households in the area assigned to then. Living within the area, they would identify themselves as employees of the Census Bureau and would explain that part of their job is to become familiar with the community. As full-time employees of the Census Bureau, they would be instructed in procedures for data collection and in the techniques of systematic observation. At some point in the census procedure, presumably a few weeks before Census Day, the systematic observers would prepare a listing of households in their designated area, and indicate the household composition.

The need for the systematic observers to identify themselves as employees of the Census Bureau raises an important question as to whether the local area will be sensitized to the decennial census when it is taken, that is, whether the individuals residing in the area will be counted more or less well than the population in general. The proposed studies should attempt to measure the extent of any such sensitization.

The systematic observers could be used in other aspects of the census operation. For example, they might be used as enumerators or supervisors in the general area, but at some distance from the area segment for which

they had primary responsibility. The area segment for which the systematic enumerator reported household composition would be enumerated in the census by a different census enumerator operating under an independent supervisor.

The data collected in the regular census enumeration would be matched against the data collected by the systematic observer. Because the original study area segments were randomly chosen, it would be possible to construct an estimator of the net number missed by race, age, and sex. It would also be possible to make estimates of household composition for the population. Some details of the sampling calculations underlying this statement are contained in Appendix 8.2.

It must be stressed that the ethical and public relations dimensions of such an operation are the most problematic and must be considered with great care, since there is the possibility of these types of studies being perceived as an invasion of privacy. The authors of the Valentine report, of ethnographic and anthropological studies such as Tally’s Corner (Liebow, 1967), and of internal memos of the Census Bureau conclude that the ethical problems are not insurmountable. Moreover, the resident observer studies indicate that a person whose avowed interest is the study of the community will be tolerated by that community. Some of these issues were addressed in an October 1974 memo by Harold Nisselson, chair of the 1980 Census Coverage Committee. The basic feeling of the coverage committee was that such studies, while sensitive, can be defended as being responsible scientific studies. They can be designed in a manner such that little or no disruption of the activities of the members of the community need occur, and such that information has a minimal chance of being disclosed. As mentioned above in Chapter 5, the possibility of using focus groups should be considered here, both to assess the ethical and public relations dimensions of systematic observation, as well as to help understand ways in which these studies may be made more effective.

Systematic observer studies are expensive, but the total cost of including in the census a broad sample of the type described might be comparable to many of the activities used in the 1980 census to increase coverage. The cost of a large systematic observer study is also on the same order of magnitude as the post-enumeration studies being considered as a part of a census evaluation and adjustment program. Some rough cost considerations are also contained in Appendix 8.2.

APPENDIX 8.1
THE POPULATION OF ILLEGAL ALIENS: METHODS AND ESTIMATES

Over the past 10 years or so, concern with the number of illegal migrants, and particularly with those coming from Mexico, has been accompanied by a plethora of estimates of their numbers. In most cases the interest is in estimating the stock of illegal migrants at some point in time. There are a few examples, however, of attempts to estimate yearly flows. A sufficiently long series of estimates of the latter, in combination with appropriate information on survival patterns (determined by mortality and return migration), could yield an estimate of the stock of migrants. The estimates for illegal aliens residing in the United States have ranged from as little as 600,000 during the mid-1970s (Robinson, 1980) to a high of about 8.2 million around 1975 (Lesko and Associates, 1975). In a recent study, Warren and Passel (1983) provided a lower bound for the estimates of illegal migrants by estimating all those who were counted during the 1980 census. Their final figure of about 2 million is reasonably close to other estimates of total illegal migrants residing in the United States during the 1970s. Thus, Lancaster and Scheuren (1978) obtained a figure for the age range 18-44 in 1973 of about 4 million as the midpoint of a subjective confidence interval with 1.4 and 5.72 million as extremes. Bean et al. (1983) calculated that the correct figure for Mexicans in 1980 should be not less than 1.5 million and no more than 3.8 million (though the lower bound depends heavily on numerous assumptions), whereas Korns (1979) calculated an estimate of about 2 million.

Table 8.1 classifies the available estimates according to a combination of characteristics. The first one is the quantity being estimated, stocks or flows. Estimates of stocks that are derived from original estimates of flows are classified as being part of the latter’s set. The second characteristic is the type of information source used. We distinguish three autonomous sources—special surveys of migrants (or of migrants’ families), apprehension data, and departure and arrival records—that can be used in combination with conventional sources such as censuses, surveys, and vital statistics of the country of origin of the migrants or of the United States or both.

Finally, the third characteristic is the type of method used. Partially dependent on the data sources, the methods of estimation can be classified as being direct, indirect, and residual. A direct method is one that permits the calculation of migrants based on a direct count of the population of interest. For example, an estimate obtained by surveying households in Mexican localities and probing into the number of relatives residing in the United States is a direct estimate (CENIET, 1981). This differs from indirect estimates such as those obtained by applying an estimated ratio of success-

TABLE 8.1 Types and Magnitudes of Estimates of Illegal Migrants

^aEstimated flows can and frequently have been converted into estimates of stocks by assuming an initial population of migrants and patterns and levels of survivorship.

SOURCE: Palloni (1985).

ful to unsuccessful illegal entries for a fixed period of time. In this case, the desired quantity—flow of illegal entries—is obtained only after a secondary quantity is estimated or assumed.

An estimate obtained through a residual method involves the accurate estimation of two quantities, with the difference taken to be a measure of the number of illegal persons residing in the United States. For example, the application of expected death rates to a base population yields an expected number of deaths, which is then compared to the observed number. Differences between the two are then used to calculate the size of a population contributing to the death records but not to the exposure (Robinson, 1980). Analogous procedures have been used through the matching of different sources of information such as the United States census, the Current Population Survey, and a variety of administrative records (Lancaster and Scheuren, 1978). In all cases, the estimate obtained relies on the prior estimation of two quantities. Errors in these estimates are potentially large, since they depend on a difference between two separate estimates, both of which are subject to errors.

It is important to notice that some of the methods and estimates described above and in Table 8.1 are not necessarily tailored to the measurement of illegal migration but to the fine tuning of estimates of net migrants who are not counted in official statistics (missed by the census, for ex-

ample), whereas others are only geared to the production of estimates of flows or stocks of illegal migrants. While the former subpopulation group possibly contributes to census undercount, the second escapes altogether from migration records and may affect the accuracy of methods to evaluate census coverage.

As methods of improved census coverage are refined, the magnitude of the net undercount may become, in absolute terms, quite insignificant. However, the differentials in undercount by geographic areas, ethnic groups, sex, and age may be more resistant to elimination. One of the factors that contributes to differential undercount is the differential composition of the resident population in terms of their legal status in combination with differential success in counting each of them. If the population that has entered illegally is more difficult to enumerate, the differential undercount could be reduced by applying improved procedures to harder-to-enumerate areas (groups) with high concentrations of illegal residents. If, however, harder-to-count groups are equally drawn from the legal and illegal resident population, a focus on areas (groups) with heavy concentrations of illegal migrants will not necessarily reduce the differential undercount. However, even in the latter case, some methods of census coverage evaluation (e.g., demographic methods and reverse record checks) are affected by errors that vary directly with concentration of illegal residents, since migration records and statistics do not include illegal migrants. This is one of the reasons for the importance of developing new data sources to obtain estimates of flows of illegal migrants entering during the intercensal period. In a recent report, Hill (1985) has explored the feasibility of a variety of indirect procedures to estimate illegal immigration from collected (but not regularly processed) data or from new data that could be collected in relatively simple and economic ways using an infrastructure already in existence. These methods are highly dependent on assumptions regarding the distribution of illegal migrants by duration of stay and the process through which they are removed from a “current” population of illegal migrants. However, since they require relatively simple information, they are worth pursuing. The same applies to other methods to estimate the changes in the size of the population of illegal migrants, methods that in one way or another have been used in other disciplines (window, tagging, and indicator relationships) to estimate hidden populations.

APPENDIX 8.2
ESTIMATES OF VARIANCE AND COST FOR A LARGE SYSTEMATIC OBSERVER STUDY

Some very crude estimates of variance and cost are developed in this appendix to demonstrate the order of magnitude of costs that might be involved in a large systematic observer study. Because it would be desirable to concentrate systematic observers in problem areas such as central cities, we have based estimates of variability and costs on areas that have higher miss rates than are observed in the general population.

The Valentines reported a 17 percent undercount for the inner city neighborhood they observed, but a 5 percent figure for the 20-30 percent of the population that is most difficult to enumerate represents a figure that will give a conservative estimate of the accuracy per unit cost. If we assume that a systematic observer is responsible for an area containing about 40 people, that 50 percent of the time 4 people are not counted as part of the census, and that 50 percent of the time no people are missed, the coefficient of variation for the miss rate for the population of sampling units would be 100 percent. Based on these assumptions, a study containing 400 resident observers would yield a coefficient of variation for the sample mean miss rate of about 5 percent.

If the cost per systematic observer were $12,000, the direct cost of the systematic observers for 400 sampling units would be approximately 4.8 million dollars. This cost is based on the assumption that systematic observers are paid approximately $24,000 per year, including fringe benefits (GS-7). The cost calculations assume that observation of a sample unit requires the equivalent of one-half person per year. If we assume that supervision and processing costs are equal to direct field costs, a study utilizing 400 systematic observers would cost on the order of $10 million.

This page intentionally left blank.