Page 41
7
Calibration of the Long Form to ACS Output
Although there could certainly be additional uses of the output from the American Community Survey over time—some of which are described earlier—in the short term the ACS output is primarily intended as a (more timely) substitute for the decennial census long form. Given this, it is important to determine the effects that would be expected when switching from the long-form estimates to those from the ACS on various applications of long-form data. To analyze this, the Census Bureau is developing a “calibration” model of the long form in 2000 to the hypothesized full implementation of the ACS in 2000, based on ACS data collections prior to full implementation (described later), and the long-form output in 2000. The development of this calibration model will be challenging, since not only will individual-level matching of the long form to the ACS not be possible in 2000 due to the designs of the long form and the ACS data collections, but also the ACS sample sizes prior to full implementation will be substantially smaller than for the full implementation. This calibration model, in addition to clarifying ACS/long-form differences, will be used to understand the dynamics of change in various subject-matter areas (e.g., income, employment, health, welfare, education) between 2000 and later years in the decade by providing an analogue to a full implementation of the ACS in 2000.
Related to the issue of the effect on long-form applications of switching from the long form to the ACS, users have a need to understand the quality of output from the ACS (or any major data collection), since this understanding assists in the utilization of the estimates. Users need to have estimates of bias (often relative to some gold standard), variance (and associated confidence
Page 42
intervals),1 and other summary error measurements of ACS at various levels of geographic aggregation and over time to best understand its utility for various applications. This type of information is useful, for example, in the development of models in which ACS information is combined with information from other sources, as discussed in Chapter 2. In addition to informing users about the quality of the ACS information, an evaluation of the quality of ACS output would help direct Census Bureau efforts toward improving the ACS over time. To understand the quality of the ACS output, a variety of evaluation methods will need to be identified. This issue was not directly addressed at the workshop, except that it was pointed out that, with calibrated long-form data, examining mean square errors for the ACS relative to the long form, possibly at some temporal or geographically aggregated level within various categories, could provide substantial information about the bias of the ACS. This was an approach taken in the panel study of small-area estimates of poverty. The remainder of this chapter focuses on the ACS/long-form calibration model.
The ACS data collections prior to full implementation in 2003 involve several steps. The Census Bureau is now in the field in 31 comparison sites, chosen on the basis of expected differences between the two data collection schemes, to examine ACS/long-form differences. This data collection began in 1999 and will continue through 2001. The Census Bureau plans on using these data to support site-specific analysis of the ACS/long-form differences. In addition, an annual 700,000 household national ACS sample will be collected; it started in 2000 and will end in 2002. In 2000, by design, no housing units will receive both the long form and the ACS questionnaire. The 31 comparison site data collection is designed to understand the factors that are associated with ACS/long-form differences. Then the calibration model will use these factors as covariates, along with the long-form responses, in models fitted using the annual 700,000 ACS sample collected from 2000-2002. One problem with this basic approach is that due to the size of the 2000-2002 ACS sample, it may be difficult to model the ACS/long-form differences in very small areas.
Charles Alexander stated that the Census Bureau needs substantial assistance in calibrating the long form to the ACS. It needs help both in understanding whether and how to do the calibration and in understanding the proper role of the calibrated numbers. In selecting a model to calibrate the 2000 long form to the ACS, methods similar to those used for small-area estimation using either the 1990 postenumeration survey or the 2000 integrated coverage measurement plan are under consideration. One important
1 The variance estimates from the ACS will be complicated by the various weighting schemes discussed in Chapter 5.
Page 43
difference is that the ACS problem has a large number of dependent variables of interest, rather than just the single one of undercoverage. He mentioned that there was some recent interest in comparing the results of various microsimulation models using long form and ACS information, and he supported more efforts in this direction to help understand the implications of this shift from the long form to the ACS.
RESEARCH DIRECTIONS
In his presentation, Jay Breidt pointed out that as the long form is the recognized standard for various economic and demographic analyses from 2000 and the plan is to use the ACS to replace the long form after 2000, it is critical to understand ACS/long-form differences that are due to methodological changes. He noted parallels between this calibration problem and that of remote sensing. In remote sensing, one is comparing the differences between a map based on remote sensing and ground truth, both of which are subject to measurement error, features on the map are also subject to substantial sampling variance, and there are some covariates to assist in understanding differences between ground truth and the map. There is also stratification, clustering, nonresponse, mode and instrument differences, a temporal displacement problem, and spatial displacement, all of which make the analogy relatively useful.
The calibration problem has several complexities. Who are the potential users of calibrated data? At the local level, the quality of the comparison is not that crucial, since one is looking for major changes. Certainly, those trying to draw inferences at larger levels of aggregation, possibly as input to models, could make use of a calibration of this sort. The data items most likely to be of interest are the long-form variables, both in a univariate and a multivariate sense. The interesting geographic domains are likely to include various moderate levels of aggregation of interest that have correspondence with census geography, since different users need data aggregated in different ways.
The idea of making use of a form of the model proposed for use in 1990 with the postenumeration survey is a leap, since that methodology is essentially univariate. This is a concern, because that methodology might not extend in a natural way to a multivariate setting. (A more natural analogy is with missing data models suggested in Clogg et al. [1991] and in Schafer [1997].) Furthermore, any sort of regression-type methodology, in which one uses point estimates, could lead to trouble, since point predictions are too smooth.
A possible alternative is to view this as a missing data problem in which one has the missing data structure illustrated in Table 7-1.
As this table indicates, the long form, with about 17 million housing units, overlaps with (roughly) 117,000 ACS housing units in 2001 and 2002
Page 44
|
Housing Units |
||||
|
Sample Included in |
Long Form |
ACS 2000 |
ACS 2001 |
ACS 2002 |
|
Long form alone |
16,433 |
|||
|
ACS 2000 alone |
700 |
|||
|
Long form and ACS 2001 |
117 |
117 |
||
|
ACS 2001 alone |
583 |
|||
|
Long form and ACS 2002 |
117 |
117 |
||
|
ACS 2002 alone |
583 |
|||
|
Total housing units |
16,667 |
700 |
700 |
700 |
and with an additional 583,000 unmatched ACS housing units; in addition there are 700,000 unmatched ACS housing units in 2000. Development of the calibration model could be considered as a long-form missing data problem in which the long-form data for the 700,000 housing units are all missing, or one could think of it as an ACS missing data problem, in which one has 700,000 of the 3 million ACS questionnaires for 2000, and all of the associated long-form responses are also missing.2 Considering this as a missing data problem leads one to consider some sort of imputation approach at some level of geographic aggregation, attempting to mimic the joint distributional properties of the long-form and ACS responses. It would be useful to attempt this at the lowest possible level of geographic aggregation, so that users could have flexibility in aggregating the estimates. This approach can get complicated very quickly, but one might consider constructing a semiformal model through the formation of imputation classes, possibly guided by the results of the study of the 31 comparison sites. As is typical, one might use hot-deck or some distance-function matching algorithm to form a complete data set. One might also use multiple imputation to provide an assessment of the uncertainty.
2 The inability to carry out individual-level matching is unfortunate, though individual matching could be done with a 1- or 2-year lag given the current ACS design for 2000. This type of comparison forces one to model not just the differences between the data sources, but also the temporal differences. Ignoring the ability to link across years, which is problematic, one is restricted to models that only use marginal information. Carrying out such an imputation at the level of the individual household would not capture changes in the long-form and ACS sampling frames.
Page 45
For the development of a formal model, one might begin with the following real or hypothesized data inputs (and accompanying notation): the partial ACS implementation, denoted by a, for 2000-2002, the long-form data, L2000, for 2000, the hypothesized ACS under a full implementation in 2000, A2000, the hypothesized ACS responded to by the entire country, α, and the hypothesized long form responded to by the entire country, λ2000. The calibration problem is to predict A2000 given L2000 and a. This could be addressed, at least in theory, using Bayes' theorem, by computing the posterior distribution, (i.e., the probability distribution of the full ACS implementation in 2000 conditional on the long form and the partial ACS implementation). That conditional probability can be broken down into five factors: (1) the probability density associated with the long-form sampling model, p(L2000/λ2000); (2) the probability density associated with long-form responses given ACS responses, p(λ2000/α), which is a crucial, problematic element of the calibration model; (3) and (4) the probability densities associated with the ACS sampling models, p(A2000/a, α) and p(a/α); and (5) the ACS spatiotemporal model, p(α2000, α2001, α2002), which describes how the ACS would measure various quantities in various regions across time. (This model is analogous to that used by the Panel on Estimates of Poverty for Small Geographic Areas described above.)
This is an extremely complicated modeling exercise, although parts of it are relatively straightforward. For example, the sampling probabilities for the ACS and the long form are known. The last factor, the ACS multivariate response structure, would be very difficult to address, particularly since there is no information about how responses at the level of individual housing units change over time. Therefore, one would have to accomplish the modeling of this factor at some higher level of aggregation. The fitting of such a model would likely require computational methods, such as Markov chain Monte Carlo sampling, to estimate posterior means, posterior variances, and posterior quantiles and to replicate posterior predictions: that is, make multiple imputations.
There are at least two possibilities for the structure of the resulting imputed dataset. The first is to impute an ACS record for each long-form record, which would permit direct comparison of estimates at low levels of aggregation. There would also be the opportunity of making direct comparisons at the level of individual households. A second possibility would be to create a pseudo-ACS 2000 data set by augmenting the 700,000 ACS household records to get to the full ACS implementation sample size. This approach might require some weights to reflect information on the long form, and one might need replicates to capture variability, but it would support longitudinal analysis.
It is clear that long-form/ACS comparisons are extremely important. In making these comparisons, one wants to avoid confounding methodological
Page 46
change with true change. Response to the ACS and other factors might be quite dynamic, and if the basic mechanism underlying differences between the ACS and long form is not well understood, the calibration model might not be measuring what one wants in assessing differences, for example, between 2008 and 2000. Finally, whatever is done, uncertainty measures are needed, and the entire process needs to be thoroughly documented.
FINAL POINTS
Some proposed calibration models demonstrated the complexities faced by the Census Bureau in developing such a model to link the long form and the ACS. There is a need to understand small-area time dynamics in various ACS responses and to understand the causes of ACS and long-form discrepancies. Both of these suffer from a lack of information at the level of the individual household. The modeling of ACS and long-form discrepancies might need to be performed at a somewhat higher level of geographic aggregation, and some simple synthetic-type assumptions would then be used to “bring down” these estimates to lower geographic levels.
