The final step in the production and release of tabulations and other ACS data products is review by subject matter analysts to be sure there are no obvious errors or anomalies in the data. Each year the ACS processing staff and subject matter analysts must complete the entire process of preparing and reviewing data products within the span of a few months. In contrast, the preparation and review of data products from the long-form sample typically required well over a year to complete.
The volume of estimates to be reviewed each year led the Census Bureau to develop automated tools to facilitate the work of the staff. One tool, ART II, was developed in 2005 as an improved version of a similar tool (ART) used in 2003–2004. This tool automates the process of identifying statistically significant differences in estimates from one year to the next and facilitates other aspects of the review process. Other tools enable analysts and managers to track the process of review for tabulations and PUMS (U.S. Census Bureau, 2006:13-11).
We support continued efforts by the Census Bureau to automate and standardize the review process for ACS products, including not only the final review, but also review at earlier stages, such as when imputations for missing data and weighting adjustments are applied to the data records. As the time approaches when 1-year, 3-year, and 5-year period estimates must be provided for thousands of geographic areas every year (including 5-year estimates for over 200,000 individual block groups), the immensity of the review task threatens to overwhelm the analyst staff. They will run the risk of inadvertently releasing poor-quality data unless they receive a high level of technical assistance.
The Census Bureau recently identified prerelease review of demographic data, including from the ACS and other household surveys, as an important problem that merits research (Bell, 2006:10). The panel urges the Census Bureau to not only continue, but also to step up its investment of resources for automated tools, standardized protocols, and other means to facilitate an appropriate level of review of ACS data products that will ensure a high standard of quality before they are released each year. Consulting with computer software development firms and with computer scientists in academia may generate useful ideas and identify existing automated tools that are relevant to the Census Bureau’s needs (see National Research Council, 2003b).
Recommendation 4-14: The Census Bureau should increase its research and development on automated tools and standardized procedures to facilitate timely review and quality control of the large volume of ACS data products.