The American Community Survey (ACS), after a decade of testing, is a reality: the first set of ACS data products, released in August–November 2006, reports on the social, demographic, economic, and housing characteristics in 2005 of cities, counties, and other areas with 65,000 or more people. With the advent of the ACS, there will no longer be a long-form sample as part of the decennial census.
The Census Bureau asked a panel of the Committee on National Statistics to assess the usability of ACS data. The report advises users on making the transition from the long-form sample to the ACS. It identifies areas for research and development by the Census Bureau so that the ACS can realize its full potential to improve the nation’s information on people and communities.
THE ACS IN BRIEF
The ACS has similar content on people and housing as the 2000 long-form questionnaire, but its design is different. It is a continuing monthly survey, in which sampled housing unit addresses receive a questionnaire each month, cumulating to about 2 million responding households each year and 10–11 million over 5 years. ACS data products are period estimates that average 12, 36, and 60 months of data, respectively, for 1-year, 3-year, and 5-year periods. In contrast, the 2000 long-form sample of over 16 million responding households pertained to a fixed time—Census Day, April 1.
The ACS has three major benefits compared with the long-form sample:
The first benefit is timeliness: ACS data products are released 8–10 months, instead of 2 years, after data collection.
The second, and an even more important, benefit is frequency: ACS data products are updated every year instead of every 10 years, which will make it possible in many areas to track trends in such important population characteristics as educational attainment, employment, poverty, diversity, and others.
A third benefit is higher quality of the data in terms of completeness of response to the survey items: the much more complete response to the ACS compared with the 2000 long-form sample is achieved by the use of computer-assisted telephone and personal interviewing of households that do not respond by mail. The ACS interviewers are experienced and highly trained in contrast to the lightly trained temporary enumerators that were used for nonresponse follow-up in the 2000 census. In addition, ACS telephone interviewers contact mail respondents to obtain answers to missing items, a step not done in 2000.
A weakness of the ACS compared with the long-form sample is the significantly larger margins of error in ACS estimates, even when cumulated over 5 years. The primary reason is the much smaller sample size of the ACS. Another important reason is the greater variation in the ACS sample weights resulting from the subsampling for field interviewing of households not responding by mail or telephone. Also, the postcensal population and housing estimates used as survey controls are less effective than the full census controls used with the long-form sample: they are subject to unmeasured estimation error, they are applied at a less detailed level than the census controls, and they are not directly related to the ACS in the way that the census controls are related to the long-form sample.
The larger ACS sampling errors are a particular problem for small cities, counties, and other governmental jurisdictions; they also apply to small neighborhoods in large cities, but neighborhoods can often be combined satisfactorily into larger user-defined areas for analysis. For small areas for which 1-year period estimates are not available or sufficiently precise, users must learn to work with 3-year and 5-year period estimates, which are very different from point-in-time estimates.
The census long-form sample was heavily used by federal, state, and local government agencies, researchers, the private sector, the media, and the public. The ACS continuous design will initially challenge many such users in adapting their applications based on the long-form sample to the
new data. Yet this design also provides the platform for important new applications that the long-form sample could not support.
This summary provides the panel’s general guidelines for using the ACS for such applications as fund allocation, program planning by federal, state, and local governments, transportation modeling, private-sector decision making, research on population and housing trends, and general public understanding. It then presents the panel’s recommendations to the Census Bureau for investment in the ACS and in user education and outreach that will be necessary to make the most effective use of the new data.
GENERAL GUIDELINES FOR ACS USE
The panel encourages users to follow the general guidelines below in working with the ACS period estimates.
Always examine margins of error before drawing conclusions from a set of estimates.
Review the available information about nonsampling errors for estimates of interest and use this information in interpreting findings from the ACS.
Carefully consider the pros and cons of alternative strategies for extracting value from ACS 5-year period estimates for very small areas, such as aggregating small-area estimates into estimates for larger, user-defined areas.
When using ACS data to estimate shares of some total, compare estimates among areas or population groups, or assess trends over time, use ACS estimates that pertain to the same time period (1-year, 3-year, or 5-year) for all geographic areas or population groups that are being compared. Do not use a mixture of different period estimates.
When analyzing trends over time for an area or population group, use ACS 1-year period estimates whenever they are available and sufficiently precise for the purpose of interest and be cognizant of changes in geographic area boundaries that may affect comparability. Keep in mind that the sampling error for the estimate of the difference between pairs of 1-year period estimates will be larger than the sampling error of either estimate.
If only 3-year or 5-year period estimates are available or sufficiently precise, use them with care for analyzing trends over time for an area or population group. In general, avoid analyses of changes over time that are based on overlapping period estimates (for example, 5-year period estimates for 2010–2014 and 2011–2015).
Take advantage of the availability of 1-year and 3-year period estimates for public use microdata areas, which include about 100,000 people, to assist with analyses for smaller areas.
Take care to label ACS estimates, including those for 1 year, 3 years, and 5 years, as period estimates.
Use ACS 3- and 5-year period estimates for income, housing value, and housing costs with care. To compensate for the differing time periods for which dollar amounts are collected, those amounts are adjusted to a common calendar year by the change in the national consumer price index. This inflation adjustment expresses all of the reported dollar amounts in a comparable manner with regard to purchasing power as of the most recent calendar year in the period. However, the resulting estimates should not be interpreted as current-year estimates.
Use care in comparing ACS estimates with estimates from other data sources, including the 2000 long-form sample and other surveys, and be cognizant of the differences that could affect the comparisons. Such differences may include population coverage, sample size and design, reference periods, residence rules, and interview modes.
The panel strongly supports the ACS, but it does not underestimate the challenges facing the Census Bureau, which must produce a flood of data products every year, or the challenges facing the user community. The continuous ACS design will ultimately support not only current applications, but also new applications requiring innovative data products. However, there will be a learning curve. For a successful transition that leads to the full use of the ACS, the panel makes five overarching recommendations (identified by chapter number) to the Census Bureau on investment in the ACS, increasing the precision of ACS estimates, a user education and outreach program, priorities for research and development, and looking to the future.
Recommendation 7-1: The Census Bureau should continue to make sufficient funding of the ACS one of its top priorities. It should seek adequate funding on a continuing basis, not only for data collection and production, but also for ongoing programs of methodological research and evaluation and user outreach and education.
Recommendation 4-4: The Census Bureau should identify potential ways to increase the precision of ACS estimates for small geographic ar-
eas, particularly small governmental jurisdictions, through reallocation of the sample and through increases in the overall sample size. Cost savings should be sought to support such increases, although increases that could significantly improve the precision of estimates will require additional funding from Congress. Sample reallocation should also be considered to minimize anomalies across areas (for example, jurisdictions with very similar populations that fall into different sampling rate categories).
Recommendation 7-2: The Census Bureau should develop a comprehensive program of user education, outreach, and feedback for the ACS. Two goals of the program should be (1) to educate users in the basics of the ACS, how it differs from the census long-form sample and other data sources, and appropriate methods to use the data; and (2) to develop paths for systematic feedback from users to improve the training materials, identify potential problems with the data, and suggest ways to improve data products and documentation to maximize the utility of the data and facilitate data use.
Recommendation 7-9: The Census Bureau should assign priority to the following topics for research and development: sample size and allocation; the Master Address File (MAF); population controls; residence rules; estimates of change with multiyear averages; comparisons with other surveys and administrative records; and the development of automated tools for data quality review of ACS products.
Recommendation 7-10: As part of its research and development program for the ACS, the Census Bureau should dedicate a portion of resources to pursue innovative, longer term projects. While short-term research and development must focus on the ACS as a replacement for the census long-form sample, research must also address how the ACS can improve the nation’s information on population and housing in ways that were not possible with the long-form sample and may not even be envisioned today.
The panel’s additional recommendations to the Census Bureau address areas for research and development for the ACS, including the sample frame, data collection for housing units, sampling and data collection for group quarters, data products, data quality review, period estimation, and survey operations. These are followed by recommendations that address user education and outreach and data quality monitoring and improvement.
Sample Frame (Master Address File)
Recommendation 4-1: Given the centrality of the MAF to the ACS, the Census Bureau should ensure that adequate resources are provided to maintain the highest possible completeness and accuracy of MAF address information on a continuous basis.
Recommendation 4-2: The Census Bureau should plan now for programs to follow the 2010 census to ensure that the MAF is updated on a continuous basis more completely than is being done prior to 2010. These programs should include not only the current updates from the Delivery Sequence File and the Community Address Updating System, but also such initiatives as continuing local review, the use of ACS field interviewers to investigate address problems, and the use of address information from the Census Bureau’s e-StARS database of linked administrative records.
Recommendation 4-3: The Census Bureau should support a continuing research program on the quality of the MAF and the cost-effectiveness of the various operations that are designed to update the MAF. This program should include periodic field checks on MAF addresses, comparisons with housing unit estimates for specific areas, comparisons with the e-StARS database, and comparisons with the results of the 2009 complete block canvass that will be used to prepare the 2010 census MAF. The program should also include studies of methods to improve the listing of small multiunit addresses in urban areas, characteristics of duplicate housing units, and characteristics of undeliverable mail addresses. In addition, the program should examine the effectiveness of the Community Address Updating System and explore ways to improve its performance.
Data Collection for Housing Units
Recommendation 4-5: The Census Bureau should conduct experimental research on the effects of the different data collection modes used in the ACS—mailout-mailback, computer-assisted telephone interviewing (CATI), and computer-assisted personal interviewing (CAPI)—on ACS estimates and, when possible, on response errors for questionnaire items. In addition, the Census Bureau should assess how different patterns of responding by mail, CATI, and CAPI among population groups and geographic areas affect comparisons of ACS estimates and inform data users of consequential differences.
Recommendation 4-6: The Census Bureau should conduct experiments to determine the extent to which ACS respondents give different answers to the decennial census usual residence rule and the ACS 2-month residence rule and the extent to which they apply the specific ACS residence rules (for example, reporting commuter workers at the family residence, applying the 2-month rule prospectively). To help clarify residence according to the census and ACS concepts, the experimental questionnaire should ask about other residences at which respondents spend time. The Census Bureau should assess the implications of the experimental results for ACS population estimates for different geographic areas and population groups. Depending on the results, the Census Bureau should consider appropriate changes in the ACS questionnaire instructions on residence or in the residence rules themselves.
Sampling and Data Collection for Group Quarters
Recommendation 4-7: The Census Bureau should discuss with data users their requirements for detailed information from the ACS for residents of institutions and other types of group quarters, particularly at the local level. The discussions should assess benefits against costs, and the results should be used to determine any changes to the group quarters component of the ACS—for example, the possible deletion of institutions from the ACS universe—that would be cost-beneficial for users and stakeholders.
Data Products—Confidentiality Protection
Recommendation 4-8: Because of the potential value of month of data collection for analysis of the ACS public use microdata samples, the Census Bureau should revisit its decision to omit this variable as a confidentiality protection measure. If further research determines that including exact month of data collection would significantly increase disclosure risk, the Census Bureau might consider perturbing the month of data collection or taking other steps to protect confidentiality. Similarly, the Census Bureau should consider developing selected summary tables that identify the season of collection (such as summer or winter) for geographic areas for which such information would be useful.
Recommendation 4-9: The Census Bureau should undertake research to develop confidentiality protection rules and procedures for tabulations from the ACS that recognize the protection afforded to respondents by pooling the data over many months. Whenever possible, the
Census Bureau should prefer confidentiality protection procedures that preserve the ability to aggregate smaller geographic areas into larger, user-defined areas.
Data Products—Collapsing Cells for Large Sampling Errors
Recommendation 4-10: The Census Bureau should monitor the extent of collapsing of cells that is performed in different tables to meet minimum precision standards of 1-year and 3-year period tabulations from the ACS and assess the implications for comparisons among geographic areas and over time. After sufficient information has been gleaned about the extent of data collapsing, and its impact on users, the Census Bureau, in consultation with data users, should assess whether its collapsing rules are sound or should be modified for one or more subject areas.
Data Products—Inflation Adjustments
Recommendation 4-11: The Census Bureau should provide users with a full explanation of its inflation adjustment procedures and their effects on multiyear ACS estimates of income, housing costs, and housing value. It should consult with users about other kinds of income and housing amount adjustments they may need and conduct research on appropriate estimation methods (for example, methods to produce latest-year amounts from multiyear averages). It should consider publishing selected multiyear averages in nominal dollars as well as inflation-adjusted dollars.
Data Products—Tabulation Specifications
Recommendation 4-12: If some or all group quarters residents continue to be included in the ACS, the Census Bureau should consult with users regarding the most useful population universe for tabulations, which, depending on the table, could be the entire population, the household and group quarters populations separately, or the noninstitutional and institutional populations separately.
Recommendation 4-13: The Census Bureau should consider expanding the geographic areas for ACS tabulations in order to afford users greater flexibility for aggregating small areas into larger user-defined areas. Two possibilities to investigate are to lower the population threshold for 1-year period estimates to, say, 50,000, and to produce 3-year (and possibly 1-year) period estimates for user-defined statistical
subareas of large cities (aggregations of census tracts or block groups) and counties (aggregations of places and towns).
Data Quality Review
Recommendation 4-14: The Census Bureau should increase its research and development on automated tools and standardized procedures to facilitate timely review and quality control of the large volume of ACS data products.
Recommendation 5-1: The Census Bureau should conduct an in-depth review of the weighting scheme used for producing ACS 1-year period estimates and assess a range of alternative schemes that might improve the quality of the estimates.
Recommendation 5-2: The Census Bureau should evaluate the quality of the postcensal housing unit estimates and the MAF sampling frame in relation to one another. In the light of this evaluation, the Census Bureau should assess the suitability of the current housing unit control factor adjustment and modify it as necessary.
The Census Bureau should attempt to identify areas in which improvements can be made to the postcensal housing unit estimates and to the MAF sampling frame. In particular, it should investigate an integrated approach for developing the postcensal housing unit estimates and for continuously updating the MAF that would benefit both and reduce the variability in the housing unit control factor.
Recommendation 5-3: As a high priority, the Census Bureau should undertake research to evaluate the effect of the postcensal population controls on ACS estimates and to examine alternative methods of making the adjustment that may be superior to the one currently used (including dispensing with the population controls entirely). The Census Bureau should make users aware in ACS documentation that biases in the ACS estimates caused by errors in the population controls are not reflected in the margins of error reported with the estimates and should conduct research to examine the effects of these errors on ACS estimates.
The Census Bureau should also give priority to research on ways to improve the postcensal population estimates at the county level, including estimates of internal migration and international immigration and the classification of race and ethnicity.
Recommendation 6-1: The Census Bureau should conduct research to examine the bias and variance properties of the planned multiyear weighting scheme and compare these properties with those of some alternative schemes.
Recommendation 6-2: The Census Bureau should consult users about the utility of the currently proposed multiyear period estimates—particularly for estimates of totals—for areas that change markedly in population size. It should investigate whether there are other forms of estimates that could be produced and would better serve user needs.
User Education and Outreach
Recommendation 7-3: As an integral part of its education, outreach, and feedback program for the ACS, the Census Bureau should establish a dedicated ACS user staff. That staff should partner with organizations that will assist end users, including the State Data Center network as a key partner and many other organizations and groups. The staff should work with the media to help them understand ACS data so that they can explain and showcase the value of the data to communities in an effective and accurate way.
Recommendation 7-4: The Census Bureau should establish an ongoing advisory group of experienced data users with whom to interact about user education materials, web site design, table content, and other aspects of the data products and education and outreach program for the ACS.
Data Quality Monitoring and Improvement
Recommendation 7-5: The Census Bureau, in collaboration with user education partners, should carry out research on ways to facilitate understanding of the quality measures provided on the ACS web site. The Census Bureau and its partners should also consider what additional quality indicators—for example, some of the indicators presented at a finer level of geographic detail—would be useful to provide for the 2005 ACS and subsequent 1-year period estimates and what indicators to provide for the 3-year and 5-year period ACS estimates when those become available.
Recommendation 7-6: The Census Bureau, in consultation with data users and statistical methodologists, should evaluate its presentation of sampling errors of estimates that are published on the ACS web site
and its descriptions of methods for computing approximate estimates of sampling errors for estimates for which sampling errors are not published. Steps should be identified to improve the usability and ease of comprehension of information on sampling errors.
Recommendation 7-7: The Census Bureau should develop and publish an ongoing quality profile for the ACS to inform users of the survey’s data quality, to guide the development of a continuing program of data quality assessments, and to identify areas for survey improvement. The Census Bureau should seek input from users on priority topics for assessment and design reports that they would find to be useful additions to the technical reports.
Recommendation 7-8: The Census Bureau should continue to seek funding with which to implement methods panels (large samples of households) for experimentation with questionnaire design, question wording, residence rules, data collection mode, and other features of the ACS. The methods panels should be conducted annually so that the survey can be kept current in meeting data needs and collecting responses in the most efficient and effective ways.