Click for next page ( 80


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 79
4 American Community Survey PUT FORWARD AS A POTENTIAL REPLACEMENT for the decen- nial census long form in 2010, the American Community Sur- vey (AC S) is a maj or household survey anticipated to include 250,000 housing units each month. The hope underlying the ACS is that, when fully operational, the survey will provide continuous infor- mation on demographic characteristics, social welfare participation, ect- ucation en c! health status, commuting patterns, distribution anc! fre- quency of crime, and other important attributes of the population of the Unitecl States. Until now, the equivalent of the ACS has only been concluctect on a limited scale. Pilot data collection of the ACS began in selectee! test sites in 1996 a geographic base that reached 31 sites by 1999 and the resulting data have fee! into reports of the feasibility of quality data col- lection.~ In conjunction with the 2000 census, a larger-scale prototype Come of the 31 test sites are blocks of adjacent counties, but most are single county sites. Hence, in the dialogue that has emerged regarding the ACS, "31 test sites" and "31 counties" are used fairly interchangeably even though the test sites span 36 counties (U.S. Census Bureau, 2003a). 79

OCR for page 79
80 PLANNING THE 2010 CENSUS ACS began operations, involving 700,000 households per year. Data for this survey, known as the Census 2000 Supplementary Survey (C2SS), were first collected in 2000, anct data collection continucct at this level in 2001 anc! 2002.2 Original plans called for the ACS to begin full fielc! implementation in 2003, a schedule that would support publication of small-area esti- mates in 2008. However, congressional stalemate on the budget for fis- cal year 2003 clelayoc! full implementation by at least 1 year; moreover, the fiscal year 2003 budget totals approved by Senate appropriators fell well short of the funcis neeclec! for full ACS deployment. Funding in support of full fielct implementation in 2004 anct 2005 is unclear at this timed In this chapter, the panel offers its interim assessment on the ACS. As we will describe in detail, some benefits of the ACS are nearly in- ctisputable, key among them the increased timeliness of the data rel- ative to traditional long-form census estimates. But the new data re- source brings with it new challenges in evaluation anct estimation, anct the Census Bureau neecis to bolster the case for the ACS by provicI- ing stakeholclers with information on the ways in which ACS infor- mation shoulc! be usec! in a variety of contexts. In orcler for the ACS to replace the long form, it must be clemonstratect that the ACS can acloquately meet all of the unique functions of long-form social anc! cle- mographic clata. The panel is supportive of full implementation of the ACS. However, we recognize that much remains to be clone in articu- lating the strengths and the weaknesses the challenges as well as the new opportunities of the ACS as a replacement for the long form. CONDUCTING THE ACS When the ACS is fully fielclecl, it will use as its sampling frame the same Master Aciciress File (MAF) usec! by the decennial census. The 2The Census Bureau refers to the latter two data collections as SSO1 and SS02 the 2001 and 2002 Supplementary Surveys respectively. 3 Under the funding levels provided by the Bush administration s proposed budget for fiscal year 2004, questionnaire mailing for a full-scale ACS could begin during the fourth quarter July-September) of 2004. Field work for follow-up would be deferred until after September 2004, pushing the considerable expense of field interviewing into the fiscal 2005 budget process. Prior to the fourth quarter mailing, data would continue to be collected in the 31 test sites and at the C2SS levels (Lowenthal, 2003a>.

OCR for page 79
AMERICAN COMMUNITYSURVEY 81 annual sample of housing units chosen for participation in the survey will be cliviclec! into monthly mailout panels, anc! each month's panel will be a systematic sample across the complete acictress list. Thus, it is intenclec! that each month's sample will be a representative sample (approximately 480) of the population of each area of the United States. However, this simplified version of the sample selection process will be complicated by alterations similar to practices currently usect in the decennial census long form, including oversampling of small geographic areas. The ACS is intenclec! to be aciministerec! primarily via mailout/ mailback. However, the proposed ACS techniques to follow up with households that clo not return the mail form cliffer from decennial census practice. All mail nonresponclents will be initially followoc! up by computer-assistecl telephone interviewing (CATI) during the month following questionnaire mailout, if there is an available phone number. After CATI follow-up, a random one-thirct of the remaining nonresponclents will be clesignatec! for follow-up by fielc! enumerators. The precise nature of this sequential follow-up process remains to be cleterminecl; there are tentative plans to sample areas with low mail anct telephone response rates at a higher fraction rather than a strict one-thircl random sample; this oversampling may help to make sample variances more comparable across areas. The stagewise nature of ACS follow-up leacis to another important design feature, which is that all of the information collected in a given month will be usect as inputs for that month's estimates. That is, a particular month's estimates may include mailback responses from the present month's systematic sample of housing units but will also include completed telephone anc! personal interviews from 1 anc! 2 months prior, respectively. This design choice is advantageous in that it simplifies data processing anc! production loac! there is no neec! to wait until month t + 2 for final resolution of all the housing units chosen in month t before processing responses already submittecl. But it cloes raise complex methodological challenges, including the choice of weighting methods to aciciress unit nonresponse. While the size of this survey will make some direct small-area es- timates possible, the estimates for areas under 65,000 population typ- ically will be produced by aggregating information over either 3 or 5 years, depending on the size of the area. At this time, moving averages

OCR for page 79
82 PLANNING THE 2010 CENSUS are planned to be used for these aggregate-year estimates, though other possibilities coulc! be consiclerec! in the future. It is the neect for a 5-year winclow to produce cletailect small-area es- timates that puts a firm constraint on the ciate of full ACS deployment. The initial plans for full deployment in 2003 would produce small-area estimates in 2008, allowing some time for the new ACS figures to gain acceptance as a long-form replacement. Hence, to match the long-form data production schedule of the 2000 census, the absolute cleaciline for full implementation of the ACS is 2007, which would permit the pub- lishing of national estimates analogous to those from the long form in 2012. STRENGTHS OF THE ACS A great strength of the ACS relative to other national household surveys is its large sample size, which allows it to provide small-area information on the American population population characteristic profiles for counties, cities, anct other local areas. Over a 5-year period, the survey's sample size will approximate that of the census long form, supporting the production of estimates for small anct nonstanclarct geographical areas, such as school districts anc! traffic analysis zones. In acictition anct again given the large sample size information will be available for population groups defined by factors other than ge- ography, including racial and ethnic groups, age classes, occupational groups, anct educational anct health categories. (Tabulations can also be prepared for subpopulations with some combination of these characteristics.) While the census long form can only provide these small area pro- files in once-per-clecacle snapshots, the ACS collects information con- tinuously throughout the clecacle. Therefore, the ACS has the impor- tant advantage of providing estimates of the intercensal dynamics of small-area changes in the many variables listed above. Such estimates have been almost nonexistent up to now anct can provide important in- formation for policy initiatives en c! public anc! private planning. The increased timeliness of the ACS estimates relative to census long form estimates is a very substantial benefit. ACS data products are at most 3.5 years out of ciate when released; census long form data procI- ucts are never less than 2 to 2.5 years out of clate anct can be as much as 12.5 years out of ciate. Presently, using census data to develop lower

OCR for page 79
AMERICAN COMMUNITYSURVEY 83 bouncis on the amount of year-to-year change that occurs for various estimates for example, poverty rates involves examining census-to- census differences and dividing by 10; this annual change may be mea- surec! directly uncler the ACS. The ACS may eventually permit researchers to develop an inte- gratec! framework for more accurate small-area estimation, perhaps combining one or more waves of ACS data with results from aciminis- trative records, other household surveys, anc! the short-form decennial census. This broacler perspective views the ACS as a supplement to the social anc! demographic information currently collected by existing surveys anct administrative records systems. There are a variety of synergies that can be imagined between ACS anc! household surveys such as the Current Population Survey, jointly using each to improve the information collected by the other. Relative to the decennial census, the prime advantage of a full- flecigect ACS to the Census Bureau is the resulting prospect of a short-form-only census. Though the census long form was only aciministerect to a 1-in-6 sample in the 2000 census, the operational burclen is tremendous; completed long forms constitute a mountain of paper, and each form must be unstapled (running the risk of pages being mishancilecI) before processing. There are also reasonable arguments that the ACS may provide more accurate information than the census long form. ACS data woulc! be collected uncler more controlled circumstances by more experienced interviewers. Moreover, by spreading the clemanct on respondents to provide cletailec! personal anc! household information over the clecacle, the ACS may also be less susceptible to flaws anct inaccuracies that may arise from nonresponse in a once-a-clecacle measurement. During the 2000 census, concern over the perceived intrusiveness of the long form questions was well publicized, leacling to the conjecture albeit one that has not been empirically documented that this concern may have negatively impacted response rates on long form questions anti, accorclingly, hurt the accuracy of long-form clata. COSTS OF THE ACS The great advantages of the ACS timeliness anc! accuracy must be offset against the costs of implementing the program. Given that it cannot "piggyback" on some of the infrastructure proviclec! by the

OCR for page 79
84 PLANNING THE 2010 CENSUS decennial census, one might assume that the ACS coulc! cost more than the marginal cost of the long form it is replacing. However, the Cen- sus Bureau has argucct that operational efficiencies will make a short- form-only census complemented by ACS a less expensive option than a mixect long-anct-short-form traditional census. In congressional tes- timony on May 13, 2003, Census Bureau director C. Louis Kincan- non commented that "our current estimates indicate that three com- ponents of the 2010 Census LACS, MAF/TIGER Enhancements, anc! early planning/testing] will cost approximately $11.2 billion. However, if we change course right now en c! revert to a traditional census, the cost will increase to more than $12 billion anct perhaps much more."4 In its original presentation of its 2010 census strategy, the Census Bureau argucct that most of the acictitional costs of ACS can be paid for through the associated greater efficiencies in the 2010 census. Ac- corcling to the bureau, these savings would result by eliminating the collection anct processing of long-form information cluring the clecen- nial census, through improvement of MAF/TIGER, anc! through use of hanct-helct data collection crevices to facilitate fielct follow-up of mail nonresponclents. As the panel notec! in its letter report (National Re- search Council, 2001c), we have not seen validation of this claim based on empirical evidence anc! suggest that a fuller cost-benefit analysis of the ACS would help bolster the case for the survey. ACS INFORMATION AS A REPLACEMENT FOR LONG-FORM INFORMATION Our basic theme in this report is the importance of integration within the census process, and in that spirit the panel urges the Census Bureau to make a stronger case for the ACS and its role in the broader census context. At the most basic level, the case for the ACS as a replacement for the census long form is an easy one the ACS's content is patterned on the long form, so the ACS will succeed in collecting the same set of data items as the long form. That information will, moreover, be collected 4The remarks are quoted from the director's prepared testimony before the U.S. House Subcommittee on Technology, Information Policy, Intergovernmental Rela- tions, and the Census at a hearing on the ACS's potential to replace the census long form in 2010.

OCR for page 79
AMERICAN COMMUNITYSURVEY 85 anc! released on a much more timely basis than the census long form a significant benefit, anc! an improvement over the long form. But two basic questions remain to be fully answered in bolstering the case for the ACS. First, is the ACS able to satisfy all of the neects currently acictressect by long-form ciata? This larger question can in turn be cliviclec! into at least two aspects. The first stems from the fact that ACS estimates will for all but the largest population or geographic groups be based on averages across multiple years of data. Hence, the question arises: are there applications using the census long form for which substitu- tion of a moving average-type estimate from the ACS would be inap- propriate? The seconct subquestion is how well ACS estimates match other estimates of the same phenomena not only how ACS finclings compare with census long-form results but also how ACS estimates compare with other survey measures. (We will briefly discuss another aspect of the question whether the ACS can provide specific break- clowns en c! analyses, to the same extent that the long form cloes in a later section on the group quarters population.) The seconct fundamental question of interest is: What is the quality of ACS estimates anc! data relative to the census long form? Specifically, what can be said about error both bias anct variance in data collected through the ACS, and how does that compare with the census long form? It is also important to consider the level anct possible patterning of ACS unclercoverage, just as it is important to analyze the same with respect to the census long form.5 In the following sections, we offer some initial comments on these two basic questions. The panel recognizes that there are no absolutes in weighing the prospective ACS against the census long form that all options involve tracle-offs of both costs anct benefits, anct that the ACS cannot reasonably be expected to be better than the census long form in every respect (ancl vice versa). We are optimistic that increased Cen- sus Bureau attention to informing data users anc! stakeholclers (whether long-term users of the long-form data or newcomers) about the unique 5For this discussion, we make the simplifying assumption that survey undercover- age, relative to the census, may be considered a component of nonresponse. (This is a simplifying assumption because some natural causes of survey undercoverage, like any incompleteness of the operational ACS address list, are of course not exclusively due to missing data.)

OCR for page 79
86 PLANNING THE 2010 CENSUS features anc! challenges of working with ACS data will build a stronger case for the survey. ESTIMATION USING THE ACS Adequacy of Moving Averages as Point Estimates A basic concern regarding the American Community Survey as a re- placement for the census long form is whether ACS estimates -which, particularly for small areas or groups, would be moving averages of mul- tiple years' data points can take the place of fixed point-in-time esti- mates. Obviously, ACS estimates have one clear advantage in that those fixect point-in-time estimates could, for the census long form, refer to a point as much as 12 years ago. More to the point, though, the con- cern is whether funct allocation formulas or other public anct private planning neecis for demographic data can be aciciressec! using a combi- nation of data from multiple years. The Census Bureau has issucc! a ctraft report that attempts to acictress users' concerns about this shift (Alexancler, 2002), en c! Zaslavsky anc! Schirm (1998, 2002) outline the advantages and disadvantages a locality may experience through use of either a moving average or a clirect (census) estimate. The crux of the debate on this point is that a moving average is a smoother! estimate; by averaging a particular time periocl's data obser- vation with those within a particular time winclow, the resulting esti- mate is meant to follow the general trend of the series but not be as extreme as any of the inctiviclual points. The ramifications of this basic concept emerge when moving average estimates are entered into sensi- tive allocation formulas or compared against strict eligibility cutoffs. A smoothed estimate may mask or smooth over an individual year drop in level of need, thus keeping the locality eligible for benefits; conversely, it may also mask incliviclual-year spikes in activity and thus disqualify an area from benefits. It is clear that the use of smoothed estimates is neither uniformly advantageous nor ctisacivantageous to a locality; what is not clear is how often major discrepancies may occur in practice. One basic conceptual answer to this conundrum is to not use mov- ing averages and instead use sample-basecl estimates from individual years. These estimates would be unbiased in terms of probability but

OCR for page 79
AMERICAN COMMUNITYSURVEY 87 coulc! be highly variable; this woulc! affect aspects of formula grants such as "holcI-harmless" provisions.6 A relater! worry that has been expressed about moving averages is that, by incorporating estimates from other time periocls, the estimates for a given time perioc! coulc! be substantially biasec! anc! will not truly reflect the conditions for that given time period. The outstanding em- pirical question is assessing the bias that may result from averaging over 3 years of data comparer! to 5, en c! trying to weigh the magnitude of that bias against the bias associated with using a long-form estimate that is up to 12 years olcI. Intuitively, it is sensible that, when examining data series in which change is substantial between census years, moving av- erage estimates woulc! be preferable to seriously ciatec! estimates. When there is little change through the clecacle, there should be little ctiffer- ence between the two estimates. However, since this is an empirical question, the Census Bureau should carry out research that helps to evaluate this tracle-off. The continuous measurement properties of the ACS give it unique advantages over the decennial snapshots available from the census long form, but they also raise a final, related point of concern regarding mov- ~ ~ . . . . ng averages. ~~ eat Issue IS assessing year-to-year cad range In a c ata series. It is incorrect to use annual estimates baser! on moving averages over several years when assessing change since some of the clata are from overlapping time periods en c! hence identical. At the least, the results will yielct incorrect estimates of the variance of the estimates of change. Therefore, users shoulc! be cautioned about this aspect of the use of moving averages. Along the same lines, moving averages present the same types of problems when they are usec! as clepenclent variables in various statistical models, in particular time series models, anct in some regression models. Therefore, the Census Bureau coulc! bolster the case for the ACS anct potentially help relieve users' concerns if it would pro- cluce a user's guicle that details the statistical uses for which moving averages are and are not intenclecl, the problems they pose to users, and . . . means to overcome them. 6A "hold-harmless" provision in a funding formula is one that limits the amount by which an allocation can change from one year to another; for instance, under a 70 per- cent hold-harmless level, a unit's allocation may only decrease by up to 30 percent. In a hold-harmless situation, an unusually volatile observation one year due to increased variability could mean that the unit's allocation may remain out of true alignment for several cycles due to the amount of allocation automatically carried over.

OCR for page 79
88 PLANNING THE 2010 CENSUS Though the prospect of using moving averages rather than long- form census estimates cloes raise legitimate concerns anc! will have im- pact on users, the panel judges that the benefit of more timely infor- mation collected by the ACS outweighs these concerns. That said, it is important that the Census Bureau strive to minimize the impact of the change anc! work to eclucate users anc! stakeholclers about the nature of the change. Comparing ACS/C2SS to the Census Long Form Thus far, we have outliner! from conceptual anc! theoretical perspec- tives the issues surrounding the acloquacy of ACS estimates to replace the long form. It is also natural to aciciress the question from a more pragmatic point of view: the ACS anct the census long form purport to measure the same basic phenomena, but clo the resulting data from both series actually tell the same story? Comparisons of how the ACS or C2SS estimates match census long-form estimates implicitly treat the census long-form data as an effective "golc! stanciarcI" a questionable assumption at best, given that it discounts the various (anct sometimes substantial) sources of error to which the long form is subject. First, the long-form data for small areas are subject to substantial sampling error. In aciclition, as mentioned above, the long form is also subject to nonresponse, and for some sample items, the amount of item nonresponse for the long form in the 2000 census was extremely high (National Research Council, 2001a). Love (2002) has iclentifiect a number of sources of differences be- tween the ACS (or C2SS) anc! long-form census estimates that com- plicate any direct comparison. These include: different reference clates; different mocles of follow-up of nonresponse; different criteria usec! to clecicle if a response is acceptable; different edit anct imputation tech- niques; different methods for data capture anc! processing; differences as to whether or not proxy interviews are accepted (they are not ac- ceptec! by ACS but they are accepted for the decennial census); cliffer- ences as to who is an eligible respondent; anct different weighting pro- ceclures usect to acictress nonresponse anct sampling (e.g., the weighting of the long-form estimates to the 100 percent data). The reference pe- rioct associated with a question item is of particular interest for ACS

OCR for page 79
AMERICAN COMMUNITYSURVEY 89 estimates, since annual averages will be the average of responses cor- responcling to 12 different reference periods, clepencling on when the questionnaire was applied There are also differences in the target popu- lation. For example, the ACS cloes not currently include group quarters in its survey, but the census cloes. Work on comparing the ACS (test sites) anct C2SS estimates to cen- sus long-form estimates has been initiated by the Census Bureau. To ciate, what is known is that there are some large differences; generally, these differences can be explained by the amount of sampling error in the two surveys (U.S. General Accounting Office, 2002a). However, examination of complete-sample C2SS data suggests large differences for the number of housing units lacking complete plumbing facilities anct for the number of unpaid family workers. Also, at the state level, a large number of C2SS estimates clifferec! from the long-form estimates by at least 10 percent, including the number of workers commuting us- ing public transportation, the number of households with income above $200,000, the number of housing units lacking complete plumbing fa- cilities, anct the number of renter-occupiect units with gross monthly rent of $1,000 to $1,499. The Census Bureau neecis to complete this analysis, including the contribution of sampling variance, for all years of data collection, anct attempt to identify the sources of differences other than sampling error. A priority of this analysis should be responses related to resiclency, but all responses should be examined. QUALITY OF ACS ESTIMATES The error associated with ACS data may be clecomposect into sam- pling error (sample variance) anct nonsampling error, the latter of which can be further separated into error clue to nonresponse anc! measure- ment error clue to various causes. At the most basic level, sampling error in the ACS will be slightly larger than that for the long form because the total ACS sample size over a 5-year period will be slightly smaller than that for the census long form. On its own, this difference is unlikely to have a substantial impact on users. However, sampling error clue to initial mail and CATI nonresponse is widely variable and could be appreciable in some small

OCR for page 79
92 PLANNING THE 2010 CENSUS responses on both the short anc! long forms. For example, for age, the census imputation rate was 3.6 percent, whereas for the C2SS it was 2.4 percent. Salvo anct Lobo (2002) report that the allocation rate ~ 11 1 1 ~ ~ 1 _ ~ ~ ~ ~ ~ ~ - tessent1ally the same as the Item Imputation rate' tor the ~t)t)t) Ale In Bronx County was typically much higher in the 1990 long form than in the 2000 ACS, ancI, further, that this difference was strongly relater! to the lower quality of fielct data collection for census long-form informa- tion in comparison to the ACS. The U.S. General Accounting Office (2002a) reports on preliminary work carried out by the Census Bureau for long-form items in which the imputation rates were slightly higher than for the 2000 C2SS. The Census Bureau intends to extenct the analysis of imputation rates to all long-form items in the near future. Since there was no content follow-up for the 2000 census long form, it is very reasonable to expect that the gap for long-form items will be even more pronounced than the observer! difference for short-form items. We point out that there are other differences in administration between the ACS (C2SS) anc! the census long anc! short forms (e.g., the ACS uses CATI en cl CAPI) that complicate this comparison, some of which are cliscussec! below. However, it seems correct to anticipate that the ACS clata will be founct to be subject to less item nonresponse for long-form information than the census. Quality of Imputed Responses Rates of unit anct item nonresponse are only partially informative as measures of the ultimate error clue to nonresponse. This is because the imputation en cl weighting routines that the Census Bureau uses to treat item en c! unit nonresponse (anc! survey unclercoverage) can off- set some of the information loss, clepencling on the extent to which the various assumptions used to support the imputation methods hold (e.g., responses missing at ranclom). Therefore, measures of the qual- ity of imputations are an important additional measure of the impact of Item anc unit nonresponse. This impact could be measured using either a reinterview survey or through matching to a more reliable source of data (possibly aclmin- istrative records or highly reliable household surveys). Both of these approaches are problematic. Reinterview surveys of appreciable sample size are expensive and require high-quality interviewing to elicit higher

OCR for page 79
AMERICAN COMMUNITYSURVEY 93 quality responses than proviclec! earlier. Matching studies are limited by the availability of higher quality, comparable information a cliffi- cult stanclarct to meet. The Census Bureau is in the process of carrying out a matching study comparing C2SS responses to those for the 2000 census short form, on the unclerstancting that both sets of responses are subject to error.10 Some interesting work on responses to race anc! ethnicity questions has been carried out (Bennett anct Griffin, 20021. A less satisfying vari- ant of this analysis coulc! still be carried out for small geographic aggre- gates, for example, comparing census anc! ACS frequencies anc! means for responses at the tract level, which would overcome the inability to match incliviclual long-form responses. Some of this work is being con- cluctect by the Census Bureau anct is ctiscussect below. Historically, there were matching studies of census responses to Internal Revenue Ser- vice (IRS) anct Current Population Survey (CPS) clata for earlier cen- suses (Bureau of the Census, 1964, 1975b),l1 en c! excellent reinterview studies were clone in the 1970s anct 1980s (Bureau of the Census, 1970, 1975a). Also, limited research on the quality of the imputations for 1990 were carried out by Thibaucleau (2002), but comparable work has not been carried out for 2000. Measurement Error Measurement error consists of differences between the response that was intenclec! by the survey designers given a householcl's char- acteristics anct the response that was actually captured. Possible contributors to measurement error include: misunclerstancling of a question by the responclent, collecting data for the wrong time periocl, responding in the wrong units, transposing digits, making errors in capturing the response, intentional lying by either the respondent or the fielc! enumerator, anc! so on. fondue to the design of the C2SS specifically, the provision that the same respon- dent would not receive both the census long form and the C2SS this matching is only feasible for characteristics on the census short form. ~~Confidentiality concerns in the 1980s and 1990s led the IRS to restrict access to data, even for statistical purposes, thus precluding further census matching studies in recent decades. More recently, the IRS has facilitated limited administrative records research by the Census Bureau using IRS data with appropriate safeguards.

OCR for page 79
94 PLANNING THE 2010 CENSUS it is reasonable to assume that, generally, the measurement error in ACS will be either comparable to, or very possibly somewhat less than, that for the census long form. The reason behind this argument follows from ACS design specifications: the ACS interviewing staff will be more experienced than short-term census enumerators, anct ACS interviewers are forbiciclen to use proxy respondents. One challenge in comparing measurement error between the ACS anct the census long form is reconciling the different definitions of res- iclence in the two systems.l2 These definitions are both valic! anc! cle- fensible, anct each may have particular advantages in different contexts, but their basic differences complicate comparison. Moreover, the ACS stages data collection over 3 months, anct this may incluce error clue to temporary vacancies and frequent moving. For analytic purposes, the moving time window of the ACS may present difficulties in inter- preting quantities like income. Each interview's snapshot is intended to capture a responclent's income for the 12 months preceding the in- terview, as opposed to a fixed April-to-April reference frame; this may . . . . camp. 1cate time series comparisons. TOPICS FOR FURTHER RESEARCH AND DESIGN CONSIDERATION A substantial agenda of outstanding operational anc! methoclologi- cal issues would have to be acictressect in a fully operational ACS. Some of these issues shoulc! be tackled in the near future in orcler to gener- ate the maximum benefits from use of the ACS as part of an integrated framework of estimates. Voluntary versus Mandatory Response The law governing conduct of the census imposes penalties on "whoever, being over eighteen years of age, refuses or willfully neglects Tithe census attempts to capture "usual residence" the location where respon- dents usually live or spend most of their time. By comparison, the ACS captures "cur- rent residence," the place where the respondent is at the time of the interview. More precisely, the ACS uses a "Two Month Rule"; any respondent at a sampled household unit who has been living at the location for more than 2 months is considered a current resident (U.S. Census Bureau, 2003a>. This can create differences for migrant workers or "snowbird" retirees who live for lengthy periods in different areas of the country.

OCR for page 79
AMERICAN COMMUNITYSURVEY 95 . . . to answer, to the best of his knowledge, any of the questions on any schedule submitted to him in connection with any census or survey" enablectin other parts of the census cocle (13 USC ~ 241(a)~.l3 In aciclition, it is a crime to willingly give false answers to such censuses or surveys (13 USC ~ 241 (b) ~ . Accordingly, census mailings in 2000 as in previous years prominently featured notices that "your response is required by law." The Census Bureau has argucc! that the ACS is intenclec! to replace the mandatory census long form anct, hence, the ACS should be con- cluctec! on the same mandatory basis as the census. The General Ac- counting Office has concurred with the bureau that it has statutory authority to conduct the ACS and that it has the authority to require responses (U.S. General Accounting Office, 2002b). The distinction between voluntary anct mandatory conduct is a significant one because it is believed that the "requirecl by law" verbiage on census forms plays . . . a rot e In raising response rates. However, early congressional discussion of the nature en c! content of the ACS led inctiviclual members of Congress to suggest that the ACS be concluctec! on a voluntary basis. Accordingly, the Census Bu- reau is conducting part of the 2003 Supplementary Survey (the proto- type ACS) on a voluntary basis; this test includes replacing "requirecl by law" verbiage with a more generic appeal (U.S. Census Bureau, 2003a). The response rates, including item nonresponse rates, on the voluntary surveys will be compared with a control group receiving manciatory- response questionniares, as well as to the 2001 anct 2002 Supplementary Surveys. The Census Bureau anticipates being able to report initial re- sults of this test to Congress in August 2003, anct the basic question of mandatory response is an important one to have settled early. Interaction with Intercensal Population Estimates and Demographic Analysis Programs One high-priority research area shoulc! be the development of mocI- els that combine information from other sources household surveys, administrative records, census data, anc! so forth with ACS informa- tion. One prominent example of this is the interplay of estimates from However, the census code does provide that respondents cannot be compelled to disclose their religious beliefs or affiliation (13 USC 241 Achy.

OCR for page 79
96 PLANNING THE 2010 CENSUS ACS and the population estimates program from the Census Bureau. At this point, it is planned that estimates from the ACS are to be con- trolled to postcensal population estimates. However, this should not be considered a one-way street. It is also possible for ACS to be used to provide the population estimates program with improved estimates of internal and external migration, fertility, household size, and vacancy status. The resulting improved population estimates could then be used as improved marginal totals to which to control ACS estimates. Fur- ther, the ACS also provides direct information on population size, and a joint estimate from population estimates and from the ACS is con- ceivable. The Census Bureau needs to carry out research on how the ACS can be used to improve intercensal population estimates. Further- more, the Census Bureau needs to examine how existing household surveys could change their posts/ratification practices (controlling to- tals by age, race, and sex) given the collection of ACS data. The potential for ACS to provide improved estimates of internal and external migration also suggests the importance of exploring the potential interactions between the ACS and population estimates derived by demographic analysis. Demographic analysis uses aggre- gate data on birth, death, immigration, and emigration to produce population estimates by ace. sex. and race. 1 1 1 1 ____~_ ---- -- r---~-~ Demographic analysis was a key benchmark used to evaluate coverage in the 2000 census, but it has significant limitations. First, estimates of immigration and emigration particularly those of illegal immigration are inherently difficult to produce with precision. Second, existing administrative records used to generate demographic analysis counts facilitate only the most basic racial comparisons white and black but do not permit direct estimation of Hispanics and other groups. The Census Bureau should consider ways in which the ACS might inform demographic analysis estimates, including more refined estimators of birth among the foreign-born population and of internal migration. Other possibilities for instance, using ACS and household sur- vey information jointly in regression models to provide improved esti- mates of the frequency of crime or unemployment could also be ad- dressed as a research topic.14 Another high-priority research area would . . . ~ .. . 14The use of models that combine information from other sources has implications for the sample designs of the major household surveys and is a future research topic

OCR for page 79
AMERICAN COMMUNITYSURVEY 97 be identification of better procedures for weighting anc! imputation, to aciciress nonresponse anc! unclercoverage in the AC S; the hope woulc! be to develop procedures that are, in a sense, optimized for ACS survey data, anc! not simply borrowoc! from procedures usec! on the decennial census long form. . . Group Quarters The intent of the census long form is to provide information on characteristics of the entire population. This means not only the popu- lation resicting in housing units but also those living in group quarters- such places as college dormitories, military barracks, prisons, anc! mecli- cal anct nursing facilities. Nonresponse to the census long form anct the neec! to impute for nonresponse may detract from the overall reliability of census long-form data, but those data clo at least allow users to make some inference about the group quarters population. Accorclingly, the complete elimination of the census long form anc! the possible loss of data on the group quarters population is an obvious concern of some census stakeholclers. In its ciraft operational plan, the Census Bureau has inclicatec! that the ACS will be aciministerect to a 2.5 percent sample from the bureau's group quarters roster (U.S. Census Bureau, 2003a). It remains to be cleterminect how acloquate this may be for monitoring this important population group, especially for small levels of geography. SUMMARY AND ASSESSMENT In 1995, a previous Committee on National Statistics (CNSTAT) panel related to the decennial census offerec! its comments on an idea "which the Census Bureau has recently been investigating:" to drop the long form from the census and substitute a con- tinuous measurement survey that is, a large monthly sur- vey of perhaps 200,000 to 500,000 households. By averag- ing the results of the monthly surveys over a period of 3 to 5 years, more timely long-form-type clata, accurate enough of great potential interest. Use of these models and connections to external programs such as the ACS may permit other household surveys to reallocate sample to areas in which estimates are less reliable.

OCR for page 79
98 PLANNING THE 2010 CENSUS for use in relatively small geographic areas, could be pro- clucecI.... In its preliminary work, the Census Bureau has spec- ulatec! that the costs of the new continuous measurement survey over a clecacle could be roughly offset by the cost savings from ciropping the long form from the census anc! by other cost reductions that might be achieved in inter- censal operations.... Although we believe that the proposer! continuous measurement system deserves serious evaluation, we conclude that much work remains to develop credible estimates of its net costs anc! to answer many other funcia- mental questions about data quality, the use of small-area estimates based on cumulated data, how continuous mea- surement could be integrated with existing household surveys, anc! its advantages compared with other means of providing more frequent small-area estimates. In our judgment, it will not be possible to complete this work in time to consider the use of continuous measurement in place of the long form for the 2000 census (National Research Council, 1995:91. Eight years later, faced with the task of offering advice on making the vision of continuous measurement a reality in the 2010 census, the similarity between the arguments then and now is uncanny. Similar, too, are the points of concern; the current panel is harct-pressect to improve upon the basic summary of concerns outliner! by our predecessors. We are, however, much more sanguine that a compelling case can be macle for the ACS and that it is a viable long-form replacement in the 2010 census. In summary, the panel appreciates the enormous potential benefit of the ACS of having a program for continuous measurement of social anct demographic variables of key national interest. The ACS presents a unique source of timely information that coulc! be extremely useful to public and private planning and that could be used to support more effective and targeted fund allocation. The potential benefits of the ACS are self-evident and require little salesmanship. However, what cloes require fuller justification is how these benefits offset the costs of

OCR for page 79
AMERICAN COMMUNITYSURVEY 99 the program and, more fundamentally, how the program works as a true long-form replacement. The panel is optimistic that such a compelling case can be macle, though it will take continucct evaluation work and research. Recommendation ACS-1: The Census Bureau should carry out more research to understand the differences between and relative quality of ACS estimates and long- form estimates, with particular attention to measure- ment error and error from nonresponse and imputation. The Census Bureau must work on ways to effectively communicate and articulate those findings to interested stakeholders, particularly potential end users of the data. The fact that the Census Bureau has not clone more in comparing the data collected from the 31 ACS test sites, the C2SS, and the 2001 and 2002 Supplementary Surveys with the data collected by the 2000 census long form is disappointing. Such analyses would help assess the quality of ACS data and would be helpful in making the argument for transition from the long form to the ACS. This deficiency is probably clue to limited analytic resources at the Census Bureau and creates an argument for "farming out" this analysis to outside researchers. Fur- thermore, since access to local information is very useful in interpret- ing the results, the Census Bureau should explore whether local experts might be interested in assisting in this effort. Recommendation ACS-2: The Census Bureau should make ACS data available (protecting confidentiality) to analysts in the 31 ACS test sites to facilitate the comparison of ACS and census long-form estimates as a means of assessing the quality of ACS data as a replacement for census long-form data. Again, with appropriate safeguards, the Census Bureau should re- lease ACS data to the broader research community for evaluation purposes. Recommendation ACS-3: The Census Bureau should is- sue a user's guide that details the statistical implications

OCR for page 79
100 PLANNING THE 2010 CENSUS of the difference between point-in-time and moving av- erage estimates for various uses. Part of a fuller justification of the ACS necessarily involves a cost- benefit assessment: enumeration of all benefits and costs, measurement or postulation of the benefits and costs, and comparison with costs and benefits Including data collection and processing) of the status quo approach (the census long form). The panel acknowledges that it is difficult to put a price tag on the value of more timely data, but coming to terms with cost-benefit tracle-offs is an important part of assessing the program. Estimates of the possible error in ACS and long-form estimates as a function of the clateciness of the data need to be factored into any comparison. This can be clone by adding them to estimates of mean squared error. Such comparisons will be somewhat approximate in several respects, but the resulting assessments will be more reflective of the relative utility of these two sets of estimates. ACS Funding The panel looks forwarc! to further discussion on the methoclolog- ical challenges associated with the ACS but, at this particular time, our most fundamental recommendations regarding the ACS must be very pragmatic in nature. In our letter report (National Research Coun- cil, 2001c), we strongly urged the Census Bureau to make contingency planning a focus of its planning efforts, with particular attention to the funding levels for the ACS. The difficulty of securing fiscal year 2003 funcling for the anticipated full launch of the ACS underscores the im- portance of that recommendation. Implementation of the ACS woulc! allow the 2010 census to consist only of the short-form questionnaire, a design feature that is too crit- ical and too wicle-reaching to leave unresolved until late in the clecacle. The short-form-only census would facilitate broacler Internet clata collection and the use of MCDs to collect respondent ciata; it woulc! reduce the data collection effort and simplify use of multilanguage forms. A late reemergence of the need for long-form clata collection woulc! remove any efficiencies the Census Bureau hac! clevelopec! from its s treamline ct cle sign .

OCR for page 79
AMERICAN COMMUNITYSURVEY 101 Funding for the ACS is, of course, not a decision of the Census Bureau but of Congress. Accorclingly, in building a compelling case for the ACS, the Census Bureau neects to work in concert with congres- sional liaison. The importance of making a decision on general 2010 census structure within the next 2 years early in the clecacle must be emphasized; the role of the ACS in that structure must be articulated. Furthermore, it must be stressed that support for the ACS cannot be erratic; major changes in sample size over the course of the program could severely compromise use of the ACS as a vital component of a coorclinatec! set of estimates. The panel is encouraged by statements in a recent hearing on the ACS that indicate that congressional authorizers are aware of the importance of making a clear decision regarding ACS funcling. Specifically, at a May 13,2003, hearing on the ACS, Represen- tative Adam Putnam (R-FL), the chairman of the House Subcommittee on Technology, Information Policy, Intergovernmental Relations, anc! the Census, commented in his opening statement: I am also very aware that we are rapidly approaching the point where the Census Bureau neects to know one way or the other if there will be a long form in the 2010 census or will the ACS be the new survey tool. It's funclamen- tal to a successful 2010 Census that we let the Census Bu- reau know as soon as possible how the Congress expects the Census to be concluctecI. I'm hopeful that we can con- tinue to work together to resolve these final remaining is- sues, anc! that Congress can make a final determination on full functing for the ACS in the near future. Given our panel's charge, the most basic question we face is whether the ACS is a satisfactory replacement for the census long form (anc! therefore something that shoulc! be the foundation of 2010 census planning as it has become). We recognize that significant estimation and weighting challenges must be aclclressecl; the survey's costs, bene- fits, and uses must also be clearly articulated in order to convince users anc! stakeholclers of the surveys' effectiveness. However, we clo not see any looming flaw so large in magnitude that full ACS implementation should be set asicle. We therefore encourage full congressional funding of the ACS. It is important, though, that Congress recognize that funding of the ACS

OCR for page 79
102 PLANNING THE 2010 CENSUS should be viewoc! as a long-term commitment. As Representative Put- nam noted in his comments, it is important that Congress send a clear signal (whatever it clecicles) regarding the ACS. The benefits of the ACS will be jeoparclizec! if the survey program is faced with oscillating bucI- get commitments; cuts in functing (anct with them reductions in sample size) will impair the overall quality of the survey, with first and most pronounced impact on the ability to produce estimates for small geo- graphic areas and population groups. Contingency Planning In the meantime, the Census Bureau must begin contingency plan- ning to be prepared should support for the ACS not be forthcoming. Some possibilities include: reinstitution of the long form in 2010; implementation of a 1-year ACS (e.g., like the C2SS) to run simulta- neously but not buncilect with the census; greatly increasing the sample size en c! revising the content of the Current Population Survey; or greater use of administrative records supplemented with other survey ciata. The costs and benefits of these various approaches need to be clevelopect and presented for review so that decisions on the ACS can be fully informed. Also, planning neecis to be started on the most likely of these or other contingencies so that the bureau is well preparecl.l5 Recommendation ACS-4: The Census Bureau should identify the costs and benefits of various approaches to collecting characteristics information should support for the full ACS not be forthcoming. These costs and benefits should be presented for review so that decisions on the ACS and its alternatives can be fully informed. Tithe Office of Inspector General of the Census Bureau's parent agency, the U.S. Department of Commerce, has expressed similar concerns. "If the Bureau does not receive sustained ACS funding throughout the decade, it may be unable to eliminate the long form for 2010"; consequently, the Census Bureau's planning for 2010 should "include a contingency plan for use of the long form" (U.S. Department of Commerce, Office of Inspector General, 2002:iv).