The American Community Survey (ACS) was born of a desire to implement a continuous survey as a replacement for the “long-form” questionnaire that was used to enumerate a sample of the U.S. population as part of the 1960-2000 censuses. The goal for the ACS was twofold: (1) to produce continuously updated social and economic information on a more timely basis than the once-every-decade census and (2) to reduce the burden imposed on the constitutionally mandated complete census head count by the necessity to collect the long-form questions as well.
Like the census long form, the ACS collects detailed characteristics data from a sample of the total population, which consists of both persons residing in housing units and those living in group quarters (GQ). The Census Bureau defines a group quarters facility as “a place where people live or stay, in a group living arrangement, that is owned or managed by an entity or organization providing housing and/or services for the residents” (U.S. Census Bureau, 2008a). Group quarters are further defined as institutional (e.g., correctional facilities, nursing homes) and noninstitutional (e.g., military housing, college dormitories).
Although the ACS collects social and economic data on the characteristics of the GQ population, the GQ sample size is not large enough to produce accurate estimates of the characteristics of the GQ population for small geographic areas, and such estimates are not published. Given that less than 3 percent of the U.S. population resides in GQ facilities, a first reaction might be to downplay concern about this problem. A closer examination, however, reveals that the GQ estimates can play an important role in ACS estimates of totals and characteristics for the combined household and GQ population, especially in
geographic areas that are small in population size. In approximately 4.5 percent of places1 in the nation, over 10 percent of the total population resides in group quarters, and 1.3 percent of places have over 25 percent of their population in group quarters. The lack of accurate data about the GQ population can adversely affect the ACS estimates produced for such places and, generally, for small geographic areas, especially because GQ residents tend to be systematically different from the household population in the communities where they live. Moreover, the fact that many geographic areas have none of their GQ facilities included in the ACS sample can substantially alter the characteristics of the total population year by year, even in small communities where GQ facilities represent only a small proportion of the total population.
The U.S. Census Bureau requested the Committee on National Statistics at the National Academy of Sciences/National Research Council to convene a panel to conduct an in-depth review of the statistical methodology for measuring the GQ population in the ACS. The panel was to consider user needs for ACS data on various components of the GQ population and, in light of user needs and considerations of operational feasibility and compatibility with the treatment of the household population in the ACS, recommend alternatives to the current sample design, weighting procedures, and other methodological features that can make the GQ data from the ACS more useful for users of small-area data.
THE PROBLEM
Difficulties associated with measuring the GQ population are not limited to the ACS. The accurate classification and enumeration of the GQ population has also been an ongoing concern for the decennial census (National Research Council, 2004). The operational challenges associated with collecting data from nonhousehold populations are similar in the ACS and the census. However, the fact that the ACS must rely on a sample of what is a small and very diverse population, combined with limited funding available for survey operations, makes the ACS GQ sampling, data collection, weighting, and estimation procedures more complex and the estimates more susceptible to problems stemming from these limitations. The concerns are magnified in small areas, particularly in terms of detrimental effects on the total population estimates produced for small areas. The reasons for this are among the main topics of this report.
One of the methodological features that adversely affect ACS estimates for a large number of small geographic areas is that GQ populations are sampled at the state level, without controlling for their distribution at substate levels of
1The Census Bureau defines a “place” as a concentration of population either legally bounded as an incorporated place, such as a city or town, or delineated for statistical purposes as a census-designated place.
geography. As a result, the ACS GQ sample is ill-suited, not only for estimating the characteristics of GQ residents for many substate areas but also for producing accurate estimates of the combined total housing and GQ population for these areas. Yet these adverse effects are masked because detailed characteristics data are not published separately for the GQ population. Moreover, due to the relatively small overall ACS sample size, a substantial number of counties and smaller geographic areas with actual existing GQ populations within their boundaries have no GQ residents represented in the sample. This is often the case even after the sample is cumulated over a 5-year period for the 5-year data release intended to provide data for geographic areas as small as census tracts and block groups. Because the GQ population is weighted only to state-level controls, there can be large effects on the ACS estimates of the total population, not only in those areas for which there is no sample but also in areas that do have sample cases. The reason is that the GQ sample in those areas is overweighted in order to account for the missing sample in other areas. Again, the effects are masked.
VALUE OF GQ DATA IN THE ACS
A fundamental question for the panel was whether there is a demonstrated and sufficiently compelling need for collecting data on residents of group quarters as part of the ACS. Although the panel was not charged with a formal cost-benefit analysis of continuing to include GQ data in the ACS, the panel’s deliberations were conducted being mindful of the costs associated with the GQ data collection and of the need for realistic assumptions about ACS funding levels going forward. The panel communicated extensively with the data user community throughout its work and engaged consultants to examine the requirements for characteristics of the group quarters populations at the national, state, and local levels. In the end, the panel concluded that the GQ data in the ACS fulfill an important data user need and, indeed, appear to be required by statutes at the federal and state levels for administrating and funding a variety of programs based on the number and characteristics of the resident population. Data users, especially those with an interest in small geographic areas, depend on accurate data on the characteristics of the total population, which are based on combining the household and GQ data. Anything less does not do justice to the spirit in which the ACS was created to produce accurate and reliable estimates of the characteristics of the entire U.S. population, including in small geographic areas. At the same time, however, it is imperative to develop solutions to improve the quality of the GQ data as a component of the total population estimates. Currently, the quality of the GQ data compromises the ACS estimates of the characteristics of the total population to such an extent that, without significant improvements to the GQ data, the goals for the ACS would have to be reconsidered.
The panel recognizes the underlying challenge that large sample sizes are needed to provide accurate estimates of rare populations, such as residents of group quarters, at small levels of geography. From a methodological perspective, the most straightforward solution to address the shortcomings discussed would be to increase the sample size to the level necessary to enable the Census Bureau to produce high-quality estimates of the characteristics of GQ residents, including at small levels of geography. This could involve increasing the sample size for at least a subset of GQ types, particularly those that are most likely to change quickly. However, there are alternatives to a design-based solution (i.e., one that changes the sample size and design), such as modeling or imputing some of the GQ data, and these could also have the potential of improving the GQ estimates significantly at lower cost.
The Census Bureau already has conducted research on methodological issues for group quarters; in particular, it has investigated imputation methods for improving the ACS estimates of GQ residents and total population for small geographic areas, and plans to implement a new imputation method for the GQ data that it collects in 2012 and beyond. However, continued research is needed, not only to improve the chosen imputation procedure but also to investigate other promising methods for bolstering the GQ data in the ACS. The optimal method—considering both costs and data quality—could involve a combination of changes to the sample design, data collection procedures, and weighting and estimation strategies for group quarters.
Recommendation 3-1: Data on the characteristics of the total population fulfill an important need, particularly for small geographic areas. The Census Bureau should identify ways of improving the group quarters estimates from the American Community Survey as input to estimates of total population characteristics for small geographic areas.
OPTIMIZING GQ DATA IN THE ACS
The panel’s recommendations for changes to the survey design, data collection procedures, and weighting and estimation strategies for the GQ component of the ACS are summarized below. The recent release of the 2010 census data represents a unique opportunity to evaluate different options because the decennial enumeration provides up-to-date basic demographic information on all group quarters for all levels of geography for purposes of comparison.
Sampling Frame Development and Maintenance
The American Community Survey sampling frame for both housing units and group quarters is based on the Master Address File (MAF), which is the Census Bureau’s inventory of known living quarters. The quality of the list
is perhaps the single most important aspect of any list-based data collection approach, because it serves as the foundation on which all other elements of the survey depend, from sample selection to the development of controls used to produce the final estimates. In the case of the ACS, maintaining an up-to-date inventory of GQ facilities has proven to be a major challenge.
The designation of certain addresses as GQ facilities is inherently difficult. Even when that is accomplished, the MAF is most complete only immediately subsequent to each decennial census. The procedures that have been used to update the MAF between censuses are not complete with regard to additions, deletions, and modifications of addresses, so that errors creep into the MAF over a decade. The procedures for updating GQ addresses are particularly problematic. Although the Census Bureau’s ACS Office supplements the MAF-based sampling frame for some types of group quarters with input from other sources, such as the Federal Bureau of Prisons and military liaisons, the updating process does not take full advantage of sources of information that are available within various Census Bureau units and from federal, state, and local partners.
To increase the accuracy and efficiency of the GQ address updating operations, the panel recommends the following:
Recommendation 4-1: The Census Bureau should give high priority to developing a detailed and systematic operational plan, with clear timelines and evaluation benchmarks, for a group quarters (GQ) address updating system. This should include a plan for greater information sharing and more efficient information flow between different Census Bureau divisions and programs to improve the inventory of group quarters in the Master Address File (MAF). The updating process for the MAF should include not only the additional information that is acquired by the American Community Survey Office on some types of group quarters but also information that is potentially available from other sources, including
- the Census Bureau’s Population Estimates Program (PEP), which obtains updated information on group quarters from state demographic offices, with varied success—PEP staff should follow up with every state to obtain information on changes to their GQ inventories, and the Census Bureau should develop procedures to ensure that the information is incorporated into the MAF updating process;
- Census Bureau divisions that develop frames for sampling particular GQ types for other federal agencies; and
- other federal agencies that may have information on particular types of group quarters.
In part because of the difficulties related to maintaining a current inventory of GQ facilities, the ACS sampling frame for group quarters contains a
relatively high percentage of cases that are either ineligible or eligible but unoccupied at the time of the data collection. This sampling frame is particularly inefficient in the case of some types of group quarters that change frequently and are sometimes difficult to distinguish from households—for example, a large house that is converted into a group home for people needing care. According to the current sample design and procedures, if a sampled unit is in the “wrong” sample (a GQ facility turns up in the housing unit sample or vice versa), an interview is not conducted. This design reduces the effective sample size for estimation and wastes data collection resources. To increase the efficiency of the sampling frame, the panel makes the following recommendations:
Recommendation 4-2: The Census Bureau should evaluate, by comparison with the 2010 census and other data sources, the reasons for the relatively high rates of ineligible and eligible but unoccupied group quarters (GQ) facilities in the American Community Survey sample and determine whether there are practical ways to reduce these rates for all or some GQ types. The evaluation should take into account the costs associated with determining that a facility is ineligible or unoccupied and how these costs would change if, for some GQ types, additional in-house research is performed before a case is sent to the field.
Recommendation 4-3: To increase effective sample size by more efficiently targeting resources, the Census Bureau should consider combining the American Community Survey (ACS) sampling frame for some types of group quarters (GQ) with the housing unit sampling frame and, in tandem, modifying its data collection procedures to enable field representatives to collect data from all cases—housing unit and group quarters—in the combined sample. Additional research will be needed to determine which GQ types are best suited for integration with the housing unit sample, but the GQ types that are especially difficult to update and that are most similar to housing units may be the best candidates. These group quarters could continue to be included in the GQ universe of the ACS for purposes of weighting and estimation.
Recommendation 4-4: For group quarters (GQ) types that are not integrated into the housing unit sampling frame, the Census Bureau should develop improved and expanded procedures that enable more efficient, realtime use of status updates received from field representatives. An operations plan needs to be constructed that allows new GQ facilities to be added to the Master Address File and changes in the status of existing addresses to be reported. The Census Bureau should also continue to pursue the development of procedures that will allow for more efficient updating of the hous-
ing unit sample with cases that have been converted from group quarters to housing units.
Sample Allocation and Selection
The current sample design is not optimized for substate geographies. Because the GQ sampling and estimation procedures are controlled at the state level, substate estimates may be highly variable. To address this serious problem, the Census Bureau needs to investigate possibilities for exercising more control over the allocation rates at the substate level. This may be possible, for example, by making adjustments to the sample design to pay more individualized attention to some small jurisdictions. The Census Bureau will need to determine an allocation that optimizes data quality and collection costs.
Recommendation 5-1: The Census Bureau should conduct a formal evaluation of sample redesign strategies that would make it possible to control the American Community Survey group quarters sample allocation at the substate level. The evaluation should focus on identifying options that can improve the precision of the estimates at the state and substate levels without substantially increasing the costs of the data collection.
The ACS GQ sample consists of a subsample of small group quarters, defined by the Census Bureau as facilities that are expected to have 15 or fewer residents based on the information available from the sampling frame, and a sample of large group quarters, defined as facilities with more than 15 residents. The ACS uses a probability proportional to size GQ sample design. The sample of small group quarters in a state is proportional to the number of small group quarters in the frame for that state. The sample of large group quarters is proportional to the expected number of residents in large group quarters in the state. There are concerns that not only is information about the existence of GQ facilities outdated on the Master Address File, but also information about the number of expected residents in a GQ facility becomes quickly dated, which adversely affects the efficiency of the sample.
Recommendation 5-2: The Census Bureau should monitor the accuracy of the measures of size used in the probability proportional to size group quarters (GQ) sample design in the American Community Survey and should assess the resources allocated for updating the GQ sampling frame in the context of how the measures-of-size information available from the sampling frame affects the effectiveness of the sample design.
Given that some GQ facilities can be very large relative to the size of the household population in a geographic area, the implications of redesigning the sample to capture more of them in the sample with certainty (as is done in surveys of other unequally distributed entities) should also be evaluated.
Recommendation 5-3: The Census Bureau should assess whether useful strategies could be learned from other surveys that incorporate a must-take stratum of large units in the sample design and evaluate these strategies for possible use in the sample design for group quarters in the American Community Survey.
The residents of large group quarters are subsampled in groups of 10. This means that although the ACS sample does not include a sufficiently large number of group quarters overall for accurate estimates for small geographic areas, some large GQ facilities can have multiple groups of 10 residents in the sample. This strategy is less costly in terms of data collection operations than including a larger number of group quarters in the sample with fewer residents in each, but it may be statistically inefficient because group quarters provide housing and services to people with similar needs and circumstances, and the intraclass correlations within group quarters are naturally high for many variables.
Recommendation 5-4: The Census Bureau should expand on the research it initiated to determine the optimal cluster size for subsampling residents in large group quarters (GQ) in the American Community Survey, estimating intraclass correlations for different variables, and factoring in facility-level and person-level costs using a variety of approaches. The analysis should address whether the same subsample size is efficient for each GQ type and whether the size of the subsample per facility should be reduced.
Weighting and Estimation
As is the case with the sample design, the current weighting and estimation procedures for GQ residents in the ACS are not optimized for small-area estimates. In addition, the inadequate updating of GQ information following the decennial census may adversely affect the PEP estimates, which are used as auxiliary data to adjust the sample estimates (in other words, as “controls”) in the ACS at the state level by type of group quarters. The panel makes the following recommendations regarding the PEP controls:
Recommendation 6-1: The Census Bureau should conduct an evaluation of the 2010 American Community Survey estimates of the group quarters (GQ) population against the 2010 census counts at all levels of geography for which the Census Bureau’s Population Estimates Program (PEP) prepares such estimates. This research should estimate bias and imprecision
by GQ type and seek to identify ways to improve the PEP estimates of group quarters.
Recommendation 6-2: Depending on the outcome of the evaluation discussed in Recommendation 6-1, the Census Bureau should evaluate the relative advantages and disadvantages of developing control totals for group quarters (GQ) residents in the American Community Survey by demographic characteristics (age, sex, race, ethnicity) at the state level, possibly in addition to the control totals that are currently implemented by GQ type. The Census Bureau should also evaluate the possibility of using population controls only for the GQ types for which reliable controls are available. Finally, the Census Bureau should evaluate whether data from outside sources that are currently used to provide updates for the sampling frame could also be used for controls.
The state-based sample design of the ACS is not an efficient vehicle for providing substate estimates of the GQ population, and the estimates can be especially error prone in small areas where the GQ population represents a large portion of the total population. Many small areas are missing group quarters in the sample entirely. Statistical alternatives for producing improved GQ estimates could include indirect estimates. A variety of options exist, including the strategy currently being researched by the Census Bureau, which would involve the use of data from in-sample GQ facilities to impute person records for group quarters that are not in sample. Other options are described in the report, and the panel encourages the Census Bureau to pursue research designed to evaluate whether statistical methods of this type can be developed to improve the estimates.
Recommendation 6-3: The Census Bureau should evaluate statistical methods, such as indirect estimation, for producing group quarters estimates for counties in which group quarters are known to exist based on the American Community Survey sampling frame but are not included in the sample.
The advantage of the imputation method in particular is that it emulates the ACS data capture approach and enables the “modeled data” to be folded directly into estimates not only of the total population counts but also of the population characteristics. As such, this is a promising approach that should be evaluated and then could be continuously improved, even after an initial approach is implemented in the ACS for data collected in 2012. The panel recommends some refinements and additional research related to this plan.
Recommendation 6-4: The Census Bureau’s research on imputing group quarters (GQ) person records in the American Community Survey should further investigate the possibility of using a donor selection procedure
that deemphasizes geographic proximity in relation to matching by GQ type, trying out alternatives to the proposed sequence of collapsing the combinations of geography and GQ type. The possibility of using a cluster approach to donor selection should be reevaluated using clusters formed for this purpose based on GQ data from the 2010 census. The Census Bureau should also expand its simulation study of imputation methods to include a sufficiently large number of samples capable of revealing significant differences between the imputation-based and the design-based estimates.
In addition to a smaller-than-optimal sample size of GQ residents in the ACS, several questionnaire items exhibit very high item nonresponse rates for some group quarters (GQ) types. For example, income is missing at very high levels for residents of nursing homes, as well as “other institutional” and “other noninstitutional facilities.” A strategy of omitting selected items for some GQ types could be preferable to including data that are so heavily underreported. One promising approach to accomplishing this goal is for the Census Bureau to abandon the tradition of using the same questionnaire for very disparate populations.
Recommendation 6-5: The Census Bureau should evaluate the possibility of customizing by group quarters (GQ) type the American Community Survey questionnaire for the GQ population with the goal of reducing item imputation rates, improving data quality, and reducing the burden on the GQ respondents who are required to answer questions that are not applicable to their circumstances. Changes to consider should include omitting or revising some of the questions on the GQ questionnaire for some types of group quarters.
LOOKING TO THE FUTURE
The process of improving estimates of the group quarters and total populations for small geographic areas in the American Community Survey will need to involve not only continued research and development by the Census Bureau but also regular feedback from data users. It is the panel’s observation that data users are not yet familiar with the properties of the 5-year ACS estimates for small geographic areas and the limited information that can be provided specifically for GQ residents. If the strategies recommended by the panel and the Census Bureau’s research in the near term do not lead to cost-effective ways of improving the ACS estimates for small areas to the satisfaction of data users, then the role of the ACS in providing information about the total population— including residents of group quarters—will need to be rethought.
