PART III
Education, Outreach, and Future Development



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 225
Using the American Community Survey: Benefits and Challenges PART III Education, Outreach, and Future Development

OCR for page 225
Using the American Community Survey: Benefits and Challenges This page intentionally left blank.

OCR for page 225
Using the American Community Survey: Benefits and Challenges 7 Important Next Steps The full implementation of data collection for the American Community Survey (ACS) in 2005–2006 is a historic event for the nation’s statistical system. Based on over 10 years of research and development by the Census Bureau, the ACS is intended to replace the decennial census long-form sample as a source of regularly updated demographic and socioeconomic information on the population and housing of states, counties, cities, and other governmental and statistical areas. The panel’s assessment is that the ACS will deliver on its promise to provide more timely, frequent, and complete information than the longform sample. Given the survey’s continuous design, however, ACS estimates are not the same as the long-form-sample estimates for a point in time (Census Day, April 1); instead, they represent annually updated period estimates based on 12 months of data and (once sufficient years of data are accumulated) 36 and 60 months of data. Only 60-month estimates (5-year period estimates) will be available for the smallest areas. ACS estimates also have significantly higher sampling errors than the corresponding longform-sample estimates, a feature of particular concern for the smallest areas (small counties, cities, towns, villages, American Indian and Alaska Native areas, and school districts, as well as census tracts and block groups). While the ACS continuous design presents challenges to users, it also affords opportunities to develop applications that go far beyond what was possible with the long-form sample. Some innovative uses of the ACS will be easier to implement than others. In the same vein, some uses of the ACS to replace long-form-sample data will be easier to implement than others.

OCR for page 225
Using the American Community Survey: Benefits and Challenges Overall, there is no doubt that the ACS can be of great benefit to many users, not only in the short term, but also over time as the survey is improved and new measures and applications are developed. To achieve these goals will require sustained and even expanded resources and effort on a continuing basis, not only for collection and production of the ACS data, but also for user education and outreach and methodological research and evaluation. The Census Bureau should seek adequate funding for the ACS as a top priority. The panel hopes that the user community will express its support and that Congress will provide the needed annual funding as the ACS comes fully online. Recommendation 7-1: The Census Bureau should continue to make sufficient funding of the ACS one of its top priorities. It should seek adequate funding on a continuing basis, not only for data collection and production, but also for ongoing programs of methodological research and evaluation and user outreach and education. The Census Bureau has already devoted considerable resources to methodological research and data product design as part of the developmental work for the ACS over the past 10 years. Yet this work cannot stop with full implementation. On the contrary, the sheer volume of estimates means that the full ACS is in many ways brand new to the Census Bureau and the user community, even though the ACS concept and test data have been around for a period of years. Now is therefore the time to expand the resources for evaluation of the full production ACS and for methodological research and experimentation to improve the survey to reflect the evaluation results. Now is also the time to significantly expand the resources to educate and receive feedback from users, as over the next few years they experience for the first time the full panoply of 1-year, 3-year, and 5-year period data products from the ACS. The purpose of this chapter is to outline this needed effort so that the ACS can evolve to meet its full potential. The chapter starts by describing an education program that is needed to inform users about what the ACS is, how to use its data products, and how interactions between the Census Bureau and the ACS user community can mutually benefit the ACS. The next section reviews the requirements for continued monitoring of basic indicators of data quality. The third section outlines areas for research and evaluation so that the ACS design, data collection, and estimation procedures can be continually improved and users can be more fully informed regarding sampling and nonsampling errors in the data. This section indicates the panel’s priorities for where limited resources can be most usefully directed in the next few years. The final section briefly describes a vision of what the ACS could become as it not only supports applications that

OCR for page 225
Using the American Community Survey: Benefits and Challenges previously used the census long-form sample, but also provides the basis for improving and expanding the information that is available to understand and plan for the nation’s growing, diverse communities. 7-A EDUCATING DATA USERS ABOUT THE ACS The overriding priority for small-area data users is to adjust their perspective from having long-form-sample point-in-time estimates available once every 10 years to having 1-year, 3-year, and 5-year ACS period estimates available annually for geographic and statistical areas depending on their population size. With some exceptions, notably in the housing and transportation communities, the panel found relatively little preparation for this change on the part of data users. No doubt a key reason has been limited resources. In addition, it is often hard to imagine how to use very different kinds of data that are not yet available for most areas. Chapters 2 and 3 are intended to help users understand the key features of the ACS and to provide guidance for using the data for a range of applications, but much more work remains to be done. While the Census Bureau has tried to facilitate the transition from the long-form sample to the ACS, the fact is that the full implementation of the ACS will be a sea change for data users. Appropriate reorientation on the part of users will not occur as a result of issuing new documentation or a new web site, essential as those elements of a data dissemination plan are. Appropriate reorientation will occur only as a result of a comprehensive education effort that is based on a plan to provide a set of best practices for data use that are well illustrated, using examples that are meaningful and that clearly explain period estimates and their differences from the pointin-time estimates that are commonly provided by other data sources. The plan must also provide for systematic feedback from users that can help the Census Bureau refine and tailor the education program to user needs. Such feedback should also benefit the Census Bureau by identifying potential problems with the data to follow up and improvements to data products that would facilitate data use. The education and outreach plan, of which key elements are outlined below, is designed primarily for users who expect to make repeated, multiple uses of the ACS data and who will therefore need to learn about the survey in some detail. However, there are also first-time users, infrequent users, and users who lack resources for participating in educational programs (for example, users in many small governmental jurisdictions). These users need to find key estimates easily and not have to master the complexities of the data. The Census Bureau, in cooperation with the network of organizations that it enlists as partners for education and outreach (see Section 7-A.3 below), should identify ways to help these users. Being

OCR for page 225
Using the American Community Survey: Benefits and Challenges proactive in this regard will increase the value of the ACS and reduce the likelihood that users will fail to take advantage of ACS data because they find them too complex. Approaches to help occasional users will likely also help experienced users. One approach is for the Census Bureau to work with organizations that have a mission to assist data users, such as State Data Centers, to help them develop simple data products and explanatory materials that are specifically designed for occasional or novice users. In addition, the Census Bureau itself could develop additional data products for the first-time, occasional, or resource-constrained user. These products could consist of simple tabulations that meet commonly accepted standards of precision. Similarly, simple tabulations of year-to-year change might be provided whenever there has been a significant increase or decrease in a key estimate, such as the poverty rate. The goal should be to make these products as transparent and accessible as possible, including giving them a special and prominent location on the Census Bureau’s ACS web site that contains a link to the rest of the site for users who want more information. 7-A.1 Key Elements of the Education Strategy The program of ACS education should have two major components. The first component should aim to provide a foundation of the basics about the ACS and methods to use the data appropriately. Users should be helped to grasp the key elements that make the ACS different from the long-form sample, the most important of which are the change from point-in-time to period estimates of characteristics, the increase in the size of sampling errors, and the opportunities and challenges that will arise with annually updated data for different time periods. After introducing data users to ACS concepts, the goal should be to educate them about the new perspectives they need to have and the new techniques they need to employ in order to make effective use of the data. The second component should aim to create paths for outreach to and feedback from users that enable the Census Bureau to engage in a continuous dialogue regarding questions and issues that need to be addressed. At this stage of the program, no one, including the Census Bureau, can anticipate all of the questions and issues that will arise from the data user community. The Census Bureau will have an opportunity to accumulate a critical mass of user reactions to the 1-year and 3-year period estimates that should permit the staff to become more responsive to data users before the first 5-year period estimates are released in fall 2010. This will occur only if an adequate mechanism is in place to deliver feedback to the Census Bureau.

OCR for page 225
Using the American Community Survey: Benefits and Challenges 7-A.2 Providing a Foundation for the Basics The Census Bureau’s ACS web site (http://www.census.gov/acs/www) provides a great deal of information on all aspects of the ACS, including access to Using Data from the 2005 American Community Survey, a 31-page guidebook for users; the Guide to the ACS Data Products, an online tool for learning more about the various kinds of tables and other data products; the ACS Data User Training Guide, a set of PowerPoint presentations; and a voluminous ACS Design and Methodology document, which explains ACS operations from sampling to data release and includes facsimiles of the ACS questionnaires (U.S. Census Bureau, 2006). While helpful and necessary, these materials are not sufficient by themselves for educating data users about the ACS. To build a foundation of knowledge that is meaningful for those who apply the data in their work, the Census Bureau needs to develop userfriendly application-oriented documentation and metadata, including sample applications that can be presented in paper form and on the web in the form of online tutorials. This type of documentation differs markedly from the provision of technical information. Both types of documentation are needed. Two core features of the Census Bureau’s application-oriented documentation should be, first, to provide key information to assist in the transfer to the ACS from the census long-form sample and, second, to describe methods and best practices to apply the ACS small-area data on the socioeconomic characteristics of the population for a variety of applications. Consultation with major user groups should yield instructive applications for large cities, smaller governments, rural places, transportation interest groups, and other groups that serve the data user community. One benefit of developing these kinds of examples is to enable data users, including key intermediaries, to assist the Census Bureau to establish standards and best practices for using ACS data. A recent publication, developed with input from Census Bureau staff and data users, takes this approach (Taueber, 2006). It is aimed at helping community planners access, interpret, and report on the ACS data for their areas. The Census Bureau’s consultations should include a wide range of users, including state governments, local governments (including regional and local councils of governments), not-for-profit agencies, academic researchers, the private sector, and the media. Within those sectors, applications should be developed for users with different focal interests: transportation, education, health, social services, criminal justice, economic development, and the environment. Applications should represent an assortment of typical uses: policy development, program planning, budgeting, site selection, fund allocation, and outcomes monitoring.

OCR for page 225
Using the American Community Survey: Benefits and Challenges Basic kinds of information that users need to understand in order to make the transition from the long-form sample to the ACS, including the implications of differences for data quality and utility, include: Differences between the long-form-sample and ACS questionnaires: the format of the questionnaires, the application of residence concepts, the reference periods for questions, and the wording of questions. Differences between the long-form-sample and ACS data collection processes: an understanding of how the ACS data are collected, using different modes, from a series of monthly samples supporting annually updated 1-year, 3-year, and 5-year period estimates for different levels of geography. Differences between the long-form sample and the ACS in the accuracy and geographic specificity of population and housing unit controls that are applied to the estimates. How to compare ACS estimates to the 2000 census long-form sample (and other surveys) in light of differences between them—in particular, how to make comparisons for the 2005 ACS estimates, which pertain to households only and do not include group quarters. Information relevant to methods and practices for using the ACS that users need to understand include: The provision of data from the ACS, including: the various formats for obtaining data, the geographic levels of data availability, and the trade-offs between different data products. Data access needs to be emphasized, via the American FactFinder web portal, data on CDs and DVDs, and data available from the Census Bureau’s FTP sites. The sampling error of estimates for 12-month, 36-month, and 60-month intervals and how to interpret variability. How to interpret multiyear period estimates. How to gauge change over time using multiyear estimates and how to conduct comparisons across areas. Special issues for small-area data, focusing on strategies to increase precision at a small-area level, such as combining information across time and geography.

OCR for page 225
Using the American Community Survey: Benefits and Challenges 7-A.3 Building a Network for Education, Outreach, and Feedback In order to encourage widespread and informed use of the ACS data, the Census Bureau needs to expand its infrastructure in two ways. First, it needs to establish a headquarters ACS users’ staff devoted to education and outreach who would cultivate a network of trained intermediaries to assist with providing a basic ACS education to users. Second, it should form a small informal advisory group of experienced data users that meets with the ACS user staff on a regular basis in person and by conference call. The group would be a key point of contact for considering ideas to improve data products, educational materials, user outreach, and related topics. Once a network of trained intermediaries is established, it will enable the development of a full-fledged system of regular feedback that can make the ACS education and training program—and appropriate uses of the data—grow and prosper. Feedback in the early years of implementation will assist the Census Bureau to adapt the training program to better meet user needs. In the longer term, user feedback should be a valuable source of ideas for modifying and improving the ACS to serve a wider range of applications and provide an increased return on investment in the data collection. The Census Bureau is already reaching out to federal agencies to train and assist them in using the ACS. It has established a Federal Agency Information Program (see http://www.census.gov/acs/www/SBasics/fed.htm) through which Census Bureau staff members are available to make presentations to agency staff, provide assistance on specific applications of the ACS, such as in funding formulas, and prepare special data tabulations on a cost-reimbursable basis. The Census Bureau’s work with federal agencies should be helpful to other users, such as state and local governments that interact with those agencies. To develop an adequate network of intermediaries, however, the Census Bureau (including its regional office staff) should reach out to many organizations outside the federal government. An adequate return on the investment in the ACS can only be achieved if the small-area data are used to the widest extent possible. There are many organizations that the Census Bureau should strive to include in its network: the State Data Centers (SDCs, http://www.census.gov/sdc/www/); the Federal State Cooperative Program for Population Estimates (FSCPE, http://www.census.gov/population/www/coop/fscpe.html); the Census Information Centers (CICs, http://www.census.gov/clo/www/cic.html); college and university research institutes and data centers; professional associations (for example, the American Statistical Association, the Population Association of America, the American

OCR for page 225
Using the American Community Survey: Benefits and Challenges Planning Association, the Association of Public Data Users, the American Association of Public Opinion Research, the Transportation Research Board, the American Library Association); major state and local government organizations (for example, the U.S. Conference of Mayors, the National Governors Association, the National Association of State Legislatures, the National Association of Counties, the National Association of Towns and Townships, the Association of Metropolitan Planning Organizations); local and regional councils of governments and planning agencies (for example, groups concerned with regional transportation plans and environmental issues); not-for-profit groups (local chapters of the United Way, Red Cross, United Hospital Fund); the media (for example, Investigative Reporters and Editors, Inc., and that organization’s computer-assisted reporting program); and other for-profit and not-for-profit groups, such as market research professionals, associations of health professionals, social service agencies, and the variety of groups that serve special populations, such as the disabled, farm workers, veterans, and immigrants. The involvement of the SDC network is of critical importance since the SDCs will be on the front line of answering information requests for 2010 census and ACS data. The SDCs must be able to effectively present ACS data and assist data users, including making the data understandable to users with a very wide range of experience and expertise. The SDC steering committee is already focusing its efforts on “training the trainers,” so that SDCs have sufficient knowledge to then train the entire network of 1,800 organizations and general data users that they serve. In addition, individual SDCs have already developed helpful explanatory materials for the 2005 ACS data products (see, for example, “Ten Things to Know About the American Community Survey, 2005 Edition,” prepared by the Missouri Census Data Center).1 To move this initiative forward, the Census Bureau should support and encourage local hands-on workshops on the applications of ACS data. At least some of these workshops should be done in coordination with SDC affiliates so that best practices are provided for local users. These workshops can be the basis of an education network that, once established, can serve as an efficient information-sharing mechanism between the Census Bureau and the data user community. In general, training courses can be developed at many different levels and for many different groups. Some can be in the form of tutorials, to be 1 Available at http://mcdc2.missouri.edu/pub/data/acs2005/Ten_things_to_know.shtml.

OCR for page 225
Using the American Community Survey: Benefits and Challenges presented at special workshops or sessions at the annual meetings of many key groups that are already part of the census network as listed above. 7-A.4 Working with the Media ACS training and education should be adapted to the needs of media organizations. The media need to become a partner in explaining why the ACS is important and how the data can best be used. The media should welcome this partnership, since more frequently updated information will give them opportunities for many new stories over the decade on such topics as immigration, domestic migration, education patterns, and other topics of public concern. The need for an active media education and partnership program is evident from the press coverage of the August 15, 2006, release of 2005 ACS data for political and statistical areas with 65,000 or more total population. This initial release provided information on age, sex, race, ethnicity, ancestry, place of birth, citizenship, year of immigration, residence last year, language spoken at home, education, disability, marital status, fertility, veteran status, and whether grandparents are caring for grandchildren in their home. A review by panel staff of 57 articles in 44 newspapers around the country published August 15-16, 2006, that used the new ACS data found that interest in the data was high but understanding of them and how to use them was often poor. The ACS was sometimes confused with other programs, such as the census and the population estimates, and understanding of how to compare the 2005 data with the 2000 long-form sample and other sources was limited (see Box 7-1). The Census Bureau should conduct extensive analyses of news coverage of the 2005 ACS and revise and enhance its user education program and documentation accordingly, not only for the media, but also for other data users. As a top priority, guidance about comparisons of estimates from the 2005 data with the 2000 long-form sample and other sources (including the population estimates program) is clearly needed. Indeed, guidance on comparisons with the 2000 long-form sample will continue to be critical because the ACS cannot itself serve as a comparison source for estimates of change, particularly for small areas, until more years of data are released and analyzed. 7-A.5 Recommendations on User Education, Outreach, and Feedback As with any major initiative, creating an education and outreach system to accompany the ACS will involve a significant commitment of resources from the Census Bureau and its affiliates, and from the user community as well. The ACS represents a substantial increase in the volume of informa-

OCR for page 225
Using the American Community Survey: Benefits and Challenges ferences between the housing unit controls that are used for the ACS and the MAF could contribute importantly to MAF evaluation and improvement in its coverage (see Reese, 2007). In particular, identification of large differences, positive and negative, could provide the basis for targeted field evaluations to determine reasons for discrepancies and suggest methods to improve the MAF in areas with particular kinds of address problems, such as small multiunit structures (see Recommendations 4-1, 4-2, 4-3, and 5-2). Ideally, research and development on the MAF would proceed on a continuous basis after 2010 so that the ACS MAF is as kept as up to date and accurate as possible. Population Controls Another critically important area for assessment is the accuracy and application of the population control adjustments to the survey weights. The adjustments may adversely affect the accuracy of estimates for some kinds of areas, such as those experiencing seasonal population fluctuations or rapid population growth or decline. They also will not capture differential rates of population growth in small areas within estimation areas (large counties and groups of small counties). ACS estimates produced with population controls for a census year will likely differ—sometimes substantially—from the census counts for many areas, producing discontinuities in time series of ACS estimates. Full evaluation of the current procedures for producing the controls, as well as of alternative procedures that are under development (see Section 7-D.4 below and Recommendation 5-3), requires 2010 census counts for comparative assessment. However, work can proceed now to design the evaluation program. Moreover, it could be helpful to conduct more extensive analyses that compare the 1999 population estimates with the 2000 census counts (see Section 5-D). In addition, analysis should be conducted, beginning with the 2005 ACS data, of how much difference the controls make to the ACS survey weights and to identify systematic patterns of large upward and downward adjustments that merit investigation. Also, research should be conducted to assess the effects of errors in the population controls on ACS estimates of characteristics, and users should be made aware of the results. Evaluation of the population controls requires research that should be low cost, although given the many responsibilities of Census Bureau staff, the Bureau may want to arrange for outside researchers to work collaboratively with Bureau analysts. Additional resources will be required for work to improve the methods for producing the population (and housing) controls on the basis of evaluation results and to implement new methods on a production basis. However, the costs can be spread over several Census Bureau programs, not just the ACS, given the many uses of the population estimates.

OCR for page 225
Using the American Community Survey: Benefits and Challenges Residence Rules For many purposes, including comparisons with the 2010 census and with the annual population estimates, it is critical to conduct research to understand the implementation of the 2-month residence concept in the ACS and its effects on estimates for geographic areas and population groups. Experiments should be included in the ACS methods panels to determine how respondents interpret the 2-month residence rule in deciding whom to include and not include on the questionnaire and how their responses differ when they are asked to apply the census usual residence rule (see Recommendation 4-6). Such research could identify needed changes to question wording and instructions for reporting residence that would make reporting more consistent with the rules. The Census Bureau plans—and the panel supports—a program of annual methods panels, so that there should be little additional cost of the recommended research. Estimates of Change A major focus for many data users in using the ACS is to examine estimates of change—from the preceding year, from the last census—for geographic areas and population groups of interest. The ACS provides successive 1-year and (once the necessary data are accumulated) 3-year and 5-year period estimates, but not direct estimates of change. As discussed in Chapters 3 and 6, using period estimates to track trends over time, particularly the 3-year and 5-year estimates, is not straightforward and the interpretation may often be unclear. Users will need specific, detailed guidance on how to work with the period estimates for time-trend analyses if they are not to be frustrated in their use of the ACS. Comparisons with Other Data Sources It is important that the Census Bureau periodically compare selected ACS estimates with the corresponding estimates from other surveys and administrative records—for example, comparing ACS estimates of income and employment with those from the CPS and the IRS Statistics of Income, or comparing ACS estimates of housing characteristics with those from the American Housing Survey and administrative records. The Census Bureau established a precedent for this kind of work when it performed a large number of aggregate comparisons between estimates from the Census 2000 Supplementary Survey and the 2000 long-form sample; these comparisons helped establish the validity of the ACS (see Section 2-B). It is often difficult to develop valid comparisons given that data sources differ in details of definitions, data collection operations, and other features. Moreover, analysts cannot assume that a particular comparison source is a gold standard of truth, as all data sets contain errors. Nonetheless, when well executed, aggregate comparisons can document differences in estimates and suggest reasons for differences. In turn, these findings can stimulate further research on which data source—the ACS or another—appears to be

OCR for page 225
Using the American Community Survey: Benefits and Challenges more accurate and ways to improve the ACS, the other source, or both. In addition, such comparisons are important to establish face validity of the ACS for users who have long relied on other data sources. Automated Tools and Standardized, Well-documented Procedures for Data Quality Review While the Census Bureau has made strides in this area, it should conduct further testing and implementation of tools and procedures that can facilitate careful and timely review of the quality of ACS estimates by Bureau analysts (see Recommendation 4-14). When multiple estimates—1-year, 3-year, and 5-year period estimates for geographic areas and population groups—begin to pour out of the data collection and processing system (beginning in 2008 for 3-year period estimates and 2010 for 5-year period estimates), the Census Bureau must be in a position to cope with them. Users will expect the Census Bureau to keep to its announced schedule of releasing all estimates within 8–10 months of the end of data collection and, at the same time, to minimize obvious errors in the estimates (for example, assigning a group quarters to an incorrect geographic location or misaligning the decimal place in coding income). Having the best automated tools and documented procedures possible will be essential to enable the Census Bureau’s analysts to do a good job of data quality review under tight time schedules and constrained staff resources. 7-C.3.b Other Areas for ACS Research and Evaluation In addition to the seven high-priority topics discussed in Section 7-C.3.a above, the panel believes that six other areas are important to include in the ACS research program. Work in these areas should move forward to the extent that resources permit. Four of the six areas involve research and consultation with users that, if possible, would be useful to complete in time to make decisions on whether to change certain features of the ACS 3-year and 5-year period data products before these products are first released. The required research in each of these four areas could be largely based on low-cost simulations of the advantages and disadvantages of alternative approaches: Determination of the universe for the survey—specifically, whether to drop some or all group quarters from the ACS to save resources, and, if some or all group quarters are retained, which tables to present for the total population, household population, and group quarters population to be most useful for users (see Recommendations 4-7 and 4-12). Refinement of confidentiality protection procedures for 3-year and 5-year period estimates to recognize the protection afforded by

OCR for page 225
Using the American Community Survey: Benefits and Challenges averaging over 36 and 60 months of data and consideration of including month of interview in the PUMS (see Recommendations 4-8 and 4-9). Assessment of the inflation adjustments for 1-year, 3-year, and 5-year period income and housing value and cost estimates to determine if the current procedures best serve the needs of users and the costs and benefits of alternative procedures, including no adjustments at all (see Recommendation 4-11). In addition, guidance should be developed to help users interpret ACS dollar estimates with the current inflation adjustment procedures, and provisions should be made to provide unadjusted estimates to users who need them them. Determination of geographic areas for publication (see Recommendation 4-13): Does it makes sense—considering user needs, feasibility, and effects on precision of estimates—to reduce the population threshold for 1-year period estimates from 65,000 to 50,000 and to develop and publish 3-year (and possibly 1-year) period estimates for components of PUMAs? The other two areas that would benefit from research involve data collection modes and weighting adjustments: Experimentation on the response effects of the different data collection modes used in the ACS—mailout-mailback, CATI, and CAPI (see Recommendation 4-5). This topic is important because of the large proportion of responses that the ACS obtains from CAPI or CATI and not the original mailout mode and the likelihood that mode of collection differentially affects responses. Mode effect experiments could be included in an ACS methods panel. Assessment of the effects of the various steps in the weighting process for producing 1-year period estimates (that is, the steps other than the housing unit and population controls) (see Recommendation 5-1). Although not as important as research on the population and housing controls, analysis of the other weighting steps could be useful to identify possible ways to simplify the process and modify one or more steps to improve the precision and accuracy of the ACS estimates. The quality profile outline provided in Section 7-C.1 above lists other topics for research and evaluation in addition to those the panel specifically addressed. Although these topics were not singled out by the panel, they should not be lost sight of when the Census Bureau is allocating research resources. In particular, two related topics that warrant investigation

OCR for page 225
Using the American Community Survey: Benefits and Challenges whenever resources become available are methods to impute missing data responses and methods to include the variability from item imputation in addition to sampling error in estimating the standard errors of ACS estimates. Item nonresponse is less of a problem than it was in the 2000 long-form sample, but the effects of the imputation procedures should still be investigated. The Census Bureau should also investigate the utility of using more sophisticated imputation methods than those currently being employed. It should evaluate alternative methods for including the variability from item imputation in the estimates of the sampling errors for estimates from the ACS (see Bell [2006] for research on including imputation in variances for estimates from the 2000 census). Finally, it will be important for the Census Bureau to have a process for periodically reviewing its research and evaluation priorities and adjusting them as appropriate. It may be that an area thought to be of pressing concern appears less so upon initial investigation, whereas an area that was not high priority to begin with becomes of increasing concern for uses of the data. Close consultation with users and monitoring of ACS data quality will help the Census Bureau keep its research and evaluation program on track. The Census Bureau will also need to periodically reevaluate its research priorities in light of available funding and staff. The Census Bureau should plan its research and evaluation program from the beginning to involve both intramural projects by its own staff and extramural work by outside researchers. In this way, it can better ensure that there are always highly qualified researchers actively assessing the ACS even if in-house staff are pulled away on production and other priorities. 7-C.3.c Recommendation for Research Priorities Recommendation 7-9: The Census Bureau should assign priority to the following topics for research and development: sample size and allocation; the MAF; population controls; residence rules; estimates of change with multiyear averages; comparisons with other surveys and administrative records; and the development of automated tools for data quality review of ACS products. 7-D A VISION FOR THE FUTURE At the present time, the ACS is viewed by the Census Bureau and data users primarily as a replacement for the long-form sample. While the panel agrees with that thrust in the short term, neither the Census Bureau nor the user community should lose sight of the vast potential for the ACS to

OCR for page 225
Using the American Community Survey: Benefits and Challenges contribute to new and improved measurement over the longer term. The continuous design of the ACS, which may initially challenge users to adapt their long-form-sample-based applications to the new data, provides the platform for developing important innovative applications for the future. There are at least five ways in which the ACS could contribute to new and improved measurement. They involve (1) more timely and accurate measures of key indicators for small geographic areas by combining information from the ACS, other surveys, and administrative records; (2) measures of seasonal population fluctuations and multiple residences; (3) cost-effective, up-to-date data collection for rare populations; (4) improved population estimates; and (5) improved estimates from other household surveys (other surveys may also help improve the ACS). 7-D.1 Small-Area Estimates The planned ACS estimates for geographic areas involve accumulating and averaging 12, 36, and 60 months of data, depending on population size. For small counties, cities, and other areas for which 3-year or 5-year period estimates are provided, many users would very likely prefer continuously updated 1-year estimates for the latest year rather than estimates that represent an average over a longer period of time. Modern small-area estimation methods that borrow information across time, geography, and data sources could be used to develop indirect 1-year period estimates for key indicators, such as poverty, unemployment, food stamp participation, and others, for all counties and cities (not just those with fewer than 65,000 people). Statistical models could use data from the ACS and relevant administrative records to generate indirect estimates that would likely improve on the direct ACS estimates in precision, accuracy, and currency. Depending on the availability of administrative records, the indirect estimates might lag the latest release of the period estimates, although models could possibly be developed to project the indirect estimates forward 1 or 2 years to represent the latest year. Small-area estimation models that use the ACS could also incorporate estimates from other surveys when those surveys are believed to provide estimates of higher quality than the ACS estimates. For example, the Current Population Survey (CPS) very likely provides more accurate measures of labor force, employment, and unemployment status than the ACS (see Section 2-B.2.e). The CPS includes a more detailed set of questions and has other design features, such as a fixed reference week for measurement, to reduce nonsampling error.4 Although the CPS sample size, even when accumulated for 12 months, does not support precise estimates for subnational 4 See http://www.bls.gov/cps/cps_over.htm#overview.

OCR for page 225
Using the American Community Survey: Benefits and Challenges areas except for a few large states, it could play an important role in small-area model-based estimates from the ACS by providing controls so that the ACS estimates reflect the best available national and regional estimates. A substantial amount of work needs to be carried out to make indirect estimation a reality for the ACS. The Census Bureau has already taken some important initial steps (see Bell, 2006; Chand and Alexander, 1997; Huang and Bell, 2005). In addition, the Census Bureau’s Small Area Income and Poverty Estimates (SAIPE) models and its Small Area Health Insurance Estimates (SAHIE) models are closely related to the models that might be worthwhile to develop for the ACS, as are the models used for the Bureau of Labor Statistics’ Local Area Unemployment Statistics (LAUS) program.5 The SAIPE models of poverty and median income for states, counties, and school districts currently use data from the CPS Annual Social and Economic Supplement (CPS ASEC), federal income tax records, food stamp records, the 2000 census long-form sample, and census-based population estimates. Census Bureau researchers have conducted work on the potential for school lunch program records, earned income tax credit records, and Medicaid records to improve the SAIPE models. The SAHIE models of health insurance coverage for states and counties currently use data from the CPS ASEC, federal income tax records, food stamp records, Medicaid records, and census-based population estimates. The LAUS models of employment and unemployment for states and a few other large areas currently use data from the monthly CPS (current and historical estimates); the monthly Current Employment Statistics (CES) program, which surveys a large number of nonfarm business establishments; and state unemployment insurance (UI) records. The LAUS estimates for smaller areas, such as counties and cities, are constructed through a building-block approach that uses data from the CPS, the CES program, state UI systems, and the 2000 census long-form sample. Presumably, the inclusion of the ACS in all of these models, which are designed to improve the CPS estimates, could result in small-area estimates that are more precise than the current model-based estimates. As noted above, models could also be developed to improve the ACS direct estimates by producing more precise small-area estimates that represent a current (or recent) time period instead of averages over a longer time period. Three caveats are in order. First, it is not clear how strong a predictive model can be developed that would improve on the ACS period estimates for many of the characteristics of interest. Second, the effort required to 5 For SAIPE, see National Research Council, 2000a, 2000b; http://www.census.gov/hhes/www/saipe/saipe.html; for SAHIE, see http://www.census.gov/hhes/www/sahie/index.html; for LAUS, see http://www.bls.gov/lau/lauov.htm.

OCR for page 225
Using the American Community Survey: Benefits and Challenges generate a set of indirect estimates for one characteristic, say, the poverty rate, may not provide much information for the development of indirect estimates for another characteristic, say, employment. Third, if a multi-variate approach is used to exploit the correlations that exist among ACS estimates, the complexity of the modeling task is greatly increased. Consequently, a program to develop a large number of indirect estimates would take substantial time and resources. Yet the payoffs could be great from selected indirect estimates that are continuously updated for such purposes as fund allocation. 7-D.2 Seasonal and Multiple Residences The long-form sample could not provide information on seasonal fluctuations in population, which characterize many localities, because it was conducted at a point in time and asked only about the location of the respondent’s usual residence. In contrast, the ACS is conducted continuously and asks respondents to employ a 2-month residence rule. The current data processing and estimation system for the ACS ignores the month-by-month information, producing instead period estimates for 1, 3, and 5 years that are controlled to census-based population and housing unit estimates as of July 1 of a specific year. However, the Census Bureau’s use of monthly data to produce pre– and post–Hurricane Katrina and Rita profiles for affected areas in the Gulf Coast demonstrates that it could be not only feasible, but also very valuable to produce such profiles for other areas. To investigate the feasibility of producing part-year data for specified areas on a regular basis, the Census Bureau should conduct research on the extent to which the ACS monthly data exhibit significant seasonal variations in total population and key characteristics for localities expected to have such variations. It would be important to inform this analysis from the results of the test recommended by this panel and the Panel on Residence Rules in the Decennial Census on how respondents record their residence using the ACS 2-month rule compared with the census usual residence rule. This test may identify responses that do not accord with the 2-month rule that can be ameliorated by changes in question wording and instructions for the ACS. The outcome of research on seasonal residence could be special data products for areas that have significant seasonal fluctuations, which would represent a major addition to the stock of useful information for them. One problem concerns sample size, given that seasonal change may be evident only for small areas. To the extent that seasonal patterns tend to be repeated each year, it would be possible, and likely essential, to combine multiple years of data in order to produce sufficiently precise estimates of part-year populations for affected areas.

OCR for page 225
Using the American Community Survey: Benefits and Challenges In addition to estimates of seasonal population fluctuations, the ACS could be a vehicle for information about multiple residences more generally—for example, people with weekday and weekend homes or students away at college or boarding school. Questions that may be needed to improve reporting of residence using the 2-month rule, such as whether a household member has another residence, could also provide useful information on multiple residences. Such information would be valuable not only for planning and research, but also for designing coverage improvement programs for the decennial census. 7-D.3 Surveying Rare Populations The census long-form sample has historically provided the basis for follow-up surveys for specific, relatively small, or “rare,” populations, such as scientists and engineers and low-income minorities. By using the long-form sample to identify a population of interest for follow-up after the census, targeted postcensus surveys could be more cost-effective than nontargeted stand-alone surveys, which require much larger sample sizes to capture enough cases of the rare population of interest. The ACS can similarly provide the basis for sampling a small targeted population by serving as the initial screener to identify specific households or persons for interview. (ACS data can also be used to identify areas with a higher percentage of the target population for selecting a sample, using more current data than the long-form sample.) The ACS has the advantage that it can be used for this purpose more often than once a decade, although care will need to be taken to minimize respondent burden and provide for informed consent for any follow-on survey. There is a procedure for identifying and testing new questions to be included in the ACS, which could potentially expand its use as a screener. For example, a question on field of bachelor’s degree is planned for testing in the 2007 Methods Panel. If the question is added to the ACS, it will be used to target a sample of people in science and engineering fields to support the work of the National Science Foundation. Of course, there is a limit on how many questions can be added to the ACS without an adverse effect on response rates and public perception of the survey, unless some questions can be identified for deletion. Moreover, all ACS questions are mandatory, which makes it incumbent on the Census Bureau to consider the response burden of any new questions very carefully. 7-D.4 Improving Population Estimates There is a pressing need for the Census Bureau to conduct research on methods to improve the estimates of population by age, sex, race, and ethnicity that are used as controls for the ACS and serve so many other

OCR for page 225
Using the American Community Survey: Benefits and Challenges important purposes, such as providing factors for fund allocation formulas, controls for other household surveys, and denominators for vital rates. Information from the ACS on place of birth, citizenship, and year of immigration is already used to generate estimates of net migration from abroad for the population estimates program, and the Census Bureau is interested in examining other components of the estimates that might benefit from ACS information. For example, ACS estimates might supplement IRS tax records to estimate internal migration at the county level and perhaps for smaller geographies. The ACS could also possibly improve the population estimates and its own coverage of population and housing through linkages with the Census Bureau’s E-StARS program (see Section 4-A.4). The E-StARS Master Address Auxiliary File could be used to improve the MAF, which would in turn improve the ACS coverage of housing units. (At present, the MAF provides input to E-StARS, but there is no feedback loop back to the MAF.) Going a next step, ACS estimates of occupancy rates and persons per household could possibly be used with an improved MAF count to generate an alternative set of population estimates to compare with the estimates that are produced from the current component method (see Section 5-C). Yet another approach is to use E-StARS to provide population controls for subcounty areas within the framework of the existing population estimates. The Census Bureau has begun work along these lines, which should be pursued. Critical to making progress toward improved population estimates is for the Census Bureau to design and conduct an extensive evaluation program of alternative estimation methods and data sources in conjunction with the 2010 census. In planning and evaluating its research, the Census Bureau should involve knowledgeable users and producers of population estimates, such as the members of the Federal State Cooperative Program for Population Estimates. 7-D.5 Improving Survey Estimates Most items on the ACS questionnaire are covered in other household surveys, often in much more detail. For example, as noted in Section 7-D.1 above, the monthly CPS, which provides the nation’s official measure of unemployment, includes additional questions about work status beyond those used in the ACS to determine each respondent’s labor force situation. Other surveys that overlap with the ACS include the American Housing Survey, the CPS Annual Social and Economic Supplement, the Survey of Income and Program Participation, the National Health Interview Survey, and the National Household Travel Survey. These other surveys not only obtain extensive information about their primary topic, but also typically include a large number of additional variables for use in analysis. However, they rarely provide state, or substate, estimates.

OCR for page 225
Using the American Community Survey: Benefits and Challenges A critical question for the future of federal statistics is in what ways the ACS can contribute to and in what ways it can borrow strength from the other major national household surveys. At this early stage of implementation of the ACS, it would be foolish to think about dropping or curtailing another survey because its content overlaps with the ACS. Instead, what is needed is in-depth research to compare estimates, determine the strengths and weaknesses of each, and develop methods to improve both the ACS and other surveys. Each will undoubtedly continue to have an important role to play—the ACS primarily by providing small-area estimates and other household surveys primarily by supporting rich, multivariate policy analysis and basic social science research. The challenge will be to integrate the ACS and other surveys in ways that strengthen them all. One way in which the ACS can help other household surveys involves the MAF sampling frame. Assuming that the advent of the ACS will lead to continuous updating and improvement of the MAF (see Section 4-A.4), it should be possible to update the sampling frames for other surveys more than once a decade. Indeed, the Census Bureau plans to adopt the MAF as the sampling frame for its other household surveys. In addition, responses to the ACS could be used to identify population groups of interest for oversampling in other surveys. With regard to improved estimates for overlapping content items, the ACS could likely help other surveys—and vice versa—in several ways. For example, if research establishes that ACS estimates of a particular item are comparable with those for that item in another survey, the ACS could provide valuable controls for the other survey. But if research establishes that the ACS estimates are less accurate than those from another survey, estimates from the other survey might be used to calibrate the ACS estimates for small subgroups in some simple model-based way. For key items of national importance, it might become possible to use the ACS, other surveys, and administrative records to develop the best estimates for the nation, states, and, possibly, substate areas. These estimates could be published as independent time series, similar to the estimates of gross domestic product, which draw on many data sources. 7-D.6 Recommendation for Future Research and Development Recommendation 7-10: As part of its research and development program for the ACS, the Census Bureau should dedicate a portion of resources to pursue innovative, longer term projects. While short-term research and development must focus on the ACS as a replacement for the census long-form sample, research must also address how the ACS can improve the nation’s information on population and housing in ways that were not possible with the long-form sample and may not even be envisioned today.