Cover Image

Not for Sale

View/Hide Left Panel
Click for next page ( 32

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 31
CHAPTER 4 Using ACS Data Once a data user has obtained the ACS data that are needed for the specific analysis, she or he will need to consider the special issues that affect the ACS data. The issues described in this sec- tion will affect how analyses are done and how analysis results are interpreted and reported. As discussed above, the Census Bureau's migration to ACS adds some complexity to common data uses, but also introduces the ability to perform new and better data analyses. This section begins with a discussion of ACS data quality, focusing on non-sampling errors, bias, and other issues that could affect how well an ACS estimate reflects the actual population. The issues identified in this section can help data users understand their results better, and can help explain why unexpected results may be occurring. The issues discussed in Section 4.1 are related to the ACS data quality and accuracy. Sections 4.2 and 4.3 describe two Census Bureau data processing issues that affect how users will need to work with and interpret the ACS data. Section 4.2 considers the effects of the Census Bureau's ACS data accumulation over time and geography, and the use of one-, three-, and five-year aver- ages. Section 4.3 discusses potential data use and analysis challenges introduced by data disclo- sure limitations. These first three sections outline many of the key issues that data users will need to be aware of to design and implement ACS analyses. The following three sections describe issues related to how analysts actually perform their analyses. Section 4.4 describes the need to consider the effects of sampling error on the ACS estimates. Section 4.5 describes the issues analysts will need to consider in comparing ACS results with Census 2000 results. Finally, Section 4.6 outlines the implications and opportunities of ACS's frequent data releases. 4.1 Accuracy of ACS Data A key objective for the Census Bureau in migrating from decennial census Long Form data collection to the continuous data collection approach of ACS was to improve the quality of the data collected by improving the ways the data are collected and processed. To evaluate whether this objective is being achieved, the Census Bureau and other researchers have evaluated quality measures for the initial ACS effort and have compared the early ACS results to the decennial cen- sus Long Form. 4.1.1 Census Bureau Evaluation of ACS The Census Bureau has published 11 reports discussing ACS data quality issues based on the test site data and the C2SS experiment. The 11 Census Bureau reports are published under the title, "Meeting 21st Century Demographic Data Needs--Implementing the American Community 31

OCR for page 31
32 A Guidebook for Using American Community Survey Data for Transportation Planning Survey," and are made available at The indi- vidual reports are Report 1: Demonstrating Operational Feasibility (issued July 2001); Report 2: Demonstrating Survey Quality (issued May 2002); Report 3: Testing the Use of Voluntary Methods (issued December 2003); Report 4: Comparing General Demographic and Housing Characteristics with Census 2000 (issued May 2004); Report 5: Comparing Economic Characteristics with Census 2000 (issued May 2004); Report 6: The 2001-2002 Operational Feasibility Report of the American Community Survey (issued May 2004); Report 7: Comparing Quality Measures: The American Community Survey's Three-Year Averages and Census 2000's Long Form Sample Estimates (issued June 2004); Report 8: Comparison of the American Community Survey Three-Year Averages and the Census Sample for a Sample of Counties and Tracts (issued June 2004); Report 9: Comparing Social Characteristics with Census 2000 (issued June 2004); Report 10: Comparing Selected Physical and Financial Characteristics of Housing with the Census 2000 (issued July 2004); and Report 11: Testing Voluntary Methods--Additional Results (issued December 2004) These reports are summarized throughout the remainder of Section 4. Data Quality Measures Measuring how accurately a survey like ACS captures the attributes of the survey sample and the population from which it is drawn is very difficult because in order to do so one would need to know the true characteristics of the population (in which case, the survey would not be a very useful effort). Therefore, survey researchers try to detect clues to potential problems in different survey components. In surveys, non-sampling error can result from a variety of problems, including Coverage errors, Reporting errors, Non-response error, and Processing and coding errors. As discussed below, some of these errors lend themselves to quantitative analyses, so that indicators can be used to assess the presence and degree of these non-sampling errors. Coverage Rates Survey coverage refers to how closely the sampling frame covers the target population. Coverage error occurs If housing units that belong to the target population are excluded (called under-coverage), If housing units that belong to the target population are counted more than once (over-cover- age), or If out-of-scope housing units (i.e., those not in the target population) are included in the sampling frame (over-coverage). The sample completeness rate indicates how well a target population is covered by a survey's sample population. This rate is calculated by dividing the survey's weighted population estimates, without non-response or coverage error adjustments, by the independently derived population estimates or counts. Unit Response Rates Unit response rates measure the degree of participation of sampled housing units in the survey. Non-response due to inability or unwillingness of housing units to participate can cause bias if the characteristics of non-respondents are different from those of respondents.

OCR for page 31
Using ACS Data 33 Table 4.1. Comparison of C2SS and census 2000 population item imputation rates. Percent of Eligible Items Variable Census 2000 Imputation C2SS Imputation Relationship 2.2 % 1.5 % Gender 1.0 % 0.5 % Age 3.6 % 2.4 % Hispanic Origin 4.2 % 3.6 % Race 3.9 % 2.4 % Source: United States Census Bureau, 2002. Item Non-Response Item non-response occurs when a given respondent does not provide answers for one or more items on the questionnaire. Robust methods for reducing item non- response were employed through different ACS phases. For mail responses, the automated clerical review and the follow-up operations contribute to reducing item non-response. During the CATI and CAPI procedures, the fact that a response is received to every question by the auto- mated instrument before the next question is asked reduces item non-response significantly even when "don't know" responses are allowed. After all data collection phases, items that were still missing were obtained by borrowing the data from respondents with similar characteristics, a process known as imputation or item allocation. ACS Data Quality The Census Bureau's first assessment of the potential data quality of ACS was the assessment of the accuracy and timeliness of the C2SS data that is reported in the second report of the Census Bureau evaluation series, available at Downloads/Report02.pdf. In this report, Census Bureau experts and managers concluded When implemented, the ACS will improve survey quality compared to the decennial census Long Form. That is, some increase in sampling error will occur due to smaller sample sizes in any given year. However, timeliness will greatly improve, and non-sampling error should be reduced by the use of permanent, highly trained field staff.13 The report evaluated C2SS on the basis of unit non-response, item non-response, sample completeness, control of processing/measurement errors, and sampling errors. Unit non-response rates for C2SS were found to be quite low (and lower than other Census Bureau surveys), but statistically significant differences in the response rates were found between census tracts with different dominant racial/ethnic groups. Tracts with 75 percent or more of the population reporting a race or ethnicity of African American/black or American Indian/Alaskan Indian had statistically lower response rates than tracts that were similarly dominated by a pop- ulation reporting to be white. In terms of item non-response, the C2SS imputation rates for basic demographic items were significantly lower than for the decennial census. Significant differences in the imputation requirements were found for several key population variables, as shown in Table 4.1. The C2SS sample completeness was evaluated in relation to Census 2000, and was compared to the sample completeness ratio for the 1990 Census Long Form in relation to 1990 decennial counts (sample completeness measures for the year 2000 Long Form were not yet available at the writing of the report). The percent of the population represented in the C2SS sample was slightly higher than for the 1990 Long Form sample. 13 U.S. Census Bureau, Meeting 21st Century Demographic Data Needs Implementing the American Community Survey: Report 2: Demonstrating Survey Quality (May 2002), p. 7.

OCR for page 31
34 A Guidebook for Using American Community Survey Data for Transportation Planning The researchers believe that through the ongoing monitoring of ACS quality measures, improvements in unit and item response rates can be realized in the future. In addition, improvements in the MAF that are underway and will continue as part of the ACS program will lead to improvements in sample completeness. It was not possible for the researchers to fully evaluate the potential processing and measure- ment errors. However, they do note several procedures that help to reduce these errors and have been implemented under the ACS quality assurance program. Sampling error was the one quality measure that the analysts determined would be adversely affected by ACS. Data users have concluded that the higher sampling error of ACS will have a significant impact on the usefulness of the data. With a sampling rate of 3 million housing units per year, data accumulated over five years will correspond to a sample size of less than three- fourths of the roughly 16.7 percent sampling rate achieved with the Long Form Survey. Considering the effect of sample size alone on the standard error of the estimates and assum- ing a constant sampling rate of 2.5 percent, the ACS estimates will have a standard error equal to 2.8 times, 1.6 times, and 1.25 times that of the Long Form for annual estimates, three- and five-year moving averages, respectively.14 By examining the ACS test site data, the Census Bureau researchers drew the following conclusions: While the targeted levels of sampling error for single year estimates were met overall, differ- entials in levels of mail response for some population groups indicate that sampling error is dis- proportionately higher, suggesting the need for design changes.15 Even with improved survey follow-up procedures to address the problem of differential response to the initial mail surveys, the authors concluded that The ACS five-year averages are expected to have somewhat higher [relative standard error levels] than corresponding Census 2000 Long Form estimates... The premise of the ACS design is that this moderate increase in [standard errors] for a five-year average is worthwhile in order to obtain regular updates of the estimates throughout the decade, and to obtain what is expected to be a generally lower level of non-sampling error.16 The best assessments of actual ACS (as opposed to C2SS) non-sampling error are the Census Bureau's Accuracy of the Data reports, which are updated annually and available at www.census. gov/acs/www/UseData/Accuracy/Accuracy1.htm), and Report 7 of the Census Bureau's ACS evaluation series, which is available at Report07.pdf. The evaluation report compares Census 2000 data quality measures to the same ACS (1999- 2001) data quality measures at the county and census tract level for the ACS test sites. To analyze the differences at smaller geographic breakdowns, census tracts within the ACS test sites were divided into five groups: 1. County population less than 100,000; 2. County population between 100,000 and 1 million, with tract population less than 4,000; 3. County population between 100,000 and 1 million and tract population greater than 4,000; 14 Ronald Eash, Impacts of Sample Sizes in the ACS, presented at TRB Census Data for Transportation Planning: Planning for the Future Conference, May 12, 2005. 15 U.S. Census Bureau, Meeting 21st Century Demographic Data Needs Implementing the American Community Survey: Report 2: Demonstrating Survey Quality (May 2002), p. 27. 16 U.S. Census Bureau, Meeting 21st Century Demographic Data Needs Implementing the American Community Survey: Report 2: Demonstrating Survey Quality (May 2002), p. 29.

OCR for page 31
Using ACS Data 35 Table 4.2. Comparison of quality measures at the county level. Characteristic ACS Census 2000 Self-Response Rate 55.3% 68.1% Total Housing Unit Non-Response 4.4% 9.7% Occupied Housing Unit Non-Response Rate 5.2% 8.7% Allocation Rates Population Item Total Allocation Rates 6.5% 11.2% Occupied Housing Unit Total Allocation Rates 7.7% 15.8% Vacant Housing Unit Total Allocation Rates 23.2% 19.8% Population and Occupied Housing Unit Total Allocation Rates 6.9% 12.8% Sample Completeness Rates Housing Sample Completeness 92.9% 90.3% Household Population Sample Completeness 90.4% 91.1% Source: United States Census Bureau, 2004. 4. County population greater than 1 million and tract population less than 4,000; and 5. County population greater than 1 million and tract population greater than 4,000.17 Table 4.2 summarizes some of the key quality measures compared in the ACS and Census 2000 at the county level. The figures shown reflect the Census Bureau's weighted definitions of response and completion rates. Based on their evaluation of all of these items, the authors concluded The quality measures suggest that the ACS multiyear averages are at least as good as the esti- mates from the Long Form. When we also consider the enhanced timeliness of information from the ACS, the superiority of reengineering the 2010 Census over retaining traditional methods is clear.18 ACS Unit Response The self-response rate for Census 2000 was 68.1 percent while ACS was lower at 55.3 percent. This means that Census 2000 respondents were more likely to mail back their questionnaires than were ACS respondents. The authors note . . . the higher census Long Form self-response rates mean that the success of the census depended less on follow-up operations than did the success of the ACS. This was an expected result--past experience has consistently indicated that the census will produce mail return rates of between 10 to 20 percentage points higher than other similar operations, even decennial tests.19 The decennial census benefits from a large advertising and public relations campaign, and, therefore, has much higher visibility. The authors also point out, "Census 2000 used questionnaires in languages other than English, especially in Spanish, which would have increased self-response rates in linguistically isolated areas--the ACS used English questionnaires only." 17 Tracts with population less than 500 were discarded for this study; there are about 590 such tracts in the coun- try. The average tract population in the United States (65,000 tracts) is about 4,300. 18 U.S. Census Bureau, Meeting 21st Century Demographic Data Needs Implementing the American Community Survey: Report 7: Comparing Quality Measures: The American Community Survey's Three-Year Averages and Cen- sus 2000's Long Form Sample Estimates (June 2004), p. vii. 19 U.S. Census Bureau, Meeting 21st Century Demographic Data Needs Implementing the American Community Survey: Report 7: Comparing Quality Measures: The American Community Survey's Three-Year Averages and Cen- sus 2000's Long Form Sample Estimates (June 2004), p. 15.

OCR for page 31
36 A Guidebook for Using American Community Survey Data for Transportation Planning Similar statistically different self-response rates were found for each tract group that was analyzed. Despite the lower initial return rate, the non-response rates for total housing units and occupied housing units were lower in the ACS than in Census 2000. At the tract level, ACS also consistently showed lower rates. ACS Sample Completeness Rates The sample completeness rate indicates how well a target population is covered by a survey's sample population. Rates greater than 100 would indicate over coverage of the population, and rates less than 100 would indicate under coverage. Both efforts failed to include the whole universe in their samples. The housing unit sample completeness rate for ACS was reported to be 92.9 percent compared to 90.3 percent for Census 2000, while the household population sample completeness rates were 90.4 percent and 91.1 percent respectively. ACS Item Response Rates and Item Allocation (Imputation) The reported total allocation rates in Table 4.2 are the weighted averages of the item allocation rates for the individual corre- sponding variables. Both Census 2000 and ACS allocate (impute) responses when items are left blank or responses are out of range. For all population items (54 responses) and occupied hous- ing unit items (29 responses), Census 2000 had higher allocation rates than ACS. Both for pop- ulation and occupied housing unit responses, the ACS allocation/imputation rate was about five percent lower than the Census 2000 rate, with a similar trend across the five tract groups. The differences in the vacant housing unit items (12 responses) are most likely the result of issues related to the comparability of the two estimates. The lower ACS imputation rates are a strong indication that the quality of ACS data compares favorably with census Long Form data. The lower (improved) levels of item non-response for ACS can be seen in Table 4.2 in the pre- vious section. While the reduced need for item allocation is very good news for data users, as noted in the previous section, the item allocation procedures used by the Census Bureau are still limited by the individual sequencing of these allocations. Although individual transportation-related items show reasonable allocation rates, many of the household items, when combined with person items, show unusual results. This is likely to result from the Census Bureau's practice of processing the allocations of household items and person items separately without any cross-referencing. ACS Operational Quality Measures Reports 1 and 6 of the Census Bureau's evaluation series (available at and www/Downloads/Report06.pdf) reviewed the operational feasibility of ACS. In Report 1, Cen- sus Bureau staff reviewed the outcome of the C2SS and the 1999 and 2000 ACS test site deploy- ment to evaluate ACS from an operational standpoint. The key findings of this effort were 1. Implementing the ACS should improve the year 2010 decennial census; and 2. The successful implementation of the C2SS during 2000 demonstrated that full implementa- tion of the ACS is operationally feasible. According to the report: By having only a Short Form in 2010, the Census Bureau can more sharply focus on its consti- tutional mandates--to fully enumerate the population to apportion the House of Representatives. The ACS development program--supported by a complete and accurate address system--will simplify the decennial design, resulting in improved coverage in 2010.20 The researchers also report that C2SS achieved the quality standards, budgets, and schedules that the Census Bureau had established. The C2SS effort came in slightly under budget, and most of the workload issues identified with the effort were attributed to the fact that the C2SS was 20 U.S. Census Bureau, Meeting 21st Century Demographic Data Needs Implementing the American Community Survey: Report 1: Demonstrating Operational Feasibility (July 2001), p. 7.