Page 34
6
Sample and Questionnaire Design
The development of the ACS raises a number of issues concerning sample and questionnaire design, both for the ACS and for current household surveys. The potential uses of the ACS that involve sample or questionnaire design include: (1) modifying the ACS questionnaire to enable its use as a screener for oversampling selected populations (and for asking them further questions) to support current household surveys; (2) providing information for oversampling areas in other household surveys, which in turn could be used to make the sample design for these surveys more efficient (e.g., oversampling areas associated with higher variances for the National Crime Victimization Survey, possibly through use of variables associated with areas having more criminal activity); (3) using the ACS to help determine when redesigns of household surveys are needed and to support those redesigns;1 and (4) using responses to the ACS questionnaire to effectively increase the sample size of current household surveys through regression-type modeling, with the possible redesign of those surveys as a result of this modeling. With this last possibility, assume an ACS question that, when aggregated, has a given correlation (e.g., at the state level) with an aggregate output of a household survey of interest. Then, using a regression-type model to combine information, how much could the variances of estimates from the household survey be reduced? How would one change the
1 By survey redesign, we mean at least a new sample size, a new sample allocation, and reselection of primary sampling units and households within primary sampling units.
Page 35
design of the household survey as a result of such modeling? For example, if such a model reduced the variances differentially by area, the sample design could be modified to concentrate samples in those states for which the model was less effective.
These proposed uses of ACS responses bring up several policy issues, of which one is key: What will be the process for adding questions of interest to other federal agencies to the ACS questionnaire? Other important questions include: Who will decide the priorities and allocate the costs of these additional questions? What is the effect of added questions on the quality of response for the remainder of the ACS questionnaire? The policy for determining ACS content after 2002 will require the interaction of the Office of Management and Budget (OMB), Congress, and other interested and involved agencies.
Denise Lewis of the Census Bureau described the National Crime Victimization Survey (NCVS) within the context of the broad issues under discussion, especially with respect to sample and questionnaire design. The NCVS2 is a household survey that collects data on the amount and types of crime in the United States and measures the incidence of personal crimes of violence and theft and other household crimes, such as burglary and motor vehicle theft. The Bureau of Justice Statistics uses the NCVS to publish annual estimates of the nation's crime rate for various demographic groups.
The potential benefits of the ACS acting in concert with the NCVS lie in four areas: (1) improving the effectiveness of the data collection by increasing the use of CATI; (2) improved weights and improved control totals for use at the state level; (3) use of output from the ACS (assuming a limited number of crime-specific questions were added) and the NCVS in statistical models for developing improved state estimates, possibly even making many more state estimates reliable enough for release (analogous to the modeling accomplished for small-area estimates of poverty); and (4) as a screening device, possibly including use of the ACS to screen for crime victims, to screen for non-telephone households, and to screen for rare events. Before proceeding in these directions, several important questions need to be addressed. First, what is the likelihood of adding questions to the ACS? Second, what priority does the production of state-specific estimates have? Third, what are the cost implications of using the ACS as a screening mechanism?
2 The sample design for the NCVS is a stratified, multistage cluster sample that collects information on all persons 12 and over in about 60,000 housing units. Each sample consists of six rotations. Sample units in a given sample rotation are interviewed once every 6 months for 3 years. Each rotation is further divided into six panels, and each subsequent panel is interviewed in successive months, so one-sixth of a rotation is interviewed each month during a 6-month period.
Page 36
Alexander focused on the policy issues that need to be addressed to make full use of the ACS as discussed. First, with regard to adding a question to the ACS questionnaire, the content currently consists only of questions that are required by law. For a separate section of questions that are voluntary, one hopes that the addition of such a section would not adversely affect the quality of the required responses. Second, if an agency is planning on using its survey as a follow-up to the ACS, a problem might arise if the ACS responses are considered confidential under the privacy provisions of Title XIII.3 Third, the charge for additional questions is yet to be worked out. Many of the agencies interested in making use of the ACS do not have a great deal of discretionary funds for this purpose. Finally, there are a number of pragmatic, more focused questions, such as how quickly the ACS should pick up new construction. Some users may need this to be done more expeditiously than others. Answering these questions will require the interactions of several parties, including OMB, Congress, statistical agencies, and user groups.
RESEARCH DIRECTIONS
The main purpose of Lynn Weidman's presentation was to raise some operational questions and discuss these in terms of what the Census Bureau would like to do. The Census Bureau manages many household surveys, which are redesigned after each decennial census to take advantage of the latest available data. A hope is to use the ACS for help in redesigning these surveys more frequently. Prominent examples of these surveys are the Survey of Income and Program Participation (SIPP), which asks about income, employment, and participation in various governmental aid programs; the National Crime Victimization Survey, discussed above; the Current Population Survey, which deals mainly with employment but has many supplements dealing with a wide range of subjects (particularly each March's demographic supplement); the Consumer Expenditure Survey, which details how people spend their money; and the National Health Interview Survey, which requests information on a wide variety of health questions.
In the past, new construction would be added to the address list in each primary sampling unit (PSU) at the time of the decennial census. The hope now is to more continuously update the master address file. The ACS is also considered for use as either a screener or to oversample areas that have a large percentage of people with certain characteristics, and as a source of covariate
3 Title XIII of the United States Code provides detailed regulations concerning the activities of the U.S. Census Bureau.
Page 37
information for regression models to reduce the variance of household surveys. Weidman discussed each of these in turn.
Oversampling Groups
Household surveys often have different coefficient-of-variation stipulations for specified demographic groups. Two ways of achieving better performance for subgroups is oversampling of areas and screening persons and housing units to include more persons with those characteristics in the sample. For various household surveys there is potential interest in oversampling young children, the elderly, high- or low-income groups, and racial groups. Until now, one could use the decennial census to identify areas that could be oversampled to collect data targeted to these groups. As the decade progresses, those areas are less and less useful for this purpose. The ACS will be collecting information that could be used to retarget the sample.
This approach raises several problems. One problem is the size of geographic aggregation at which this process would operate. The larger the area, the greater the travel for household survey interviewers if they are using CAPI. It is not clear at what geographic levels ACS will be informative. Certainly, oversampling individual blocks will not be feasible with ACS. For higher levels of geographic aggregation, one might need to accumulate data over several years to inform targeting, but then the information becomes somewhat dated.
A Screening Tool
With respect to screening, the current situation is analogous to targeting, since one now uses the decennial census, even when it is less and less current. The demands raised by screening might require more sampling (or oversampling) since one has to find a match to the characteristic(s) of interest. However, screening is more efficient, since one gets the households or people one wants to interview. The greatest problem is that one is now asking people to answer two completely different questionnaires, the ACS questionnaire and the follow-on questionnaire, at the same time. The ACS questionnaire alone takes 30-60 minutes to complete. As a result, the Census Bureau has to worry about low response rates for the ACS and especially any follow-on survey. The issue of nonresponse, possibly involving the groups of interest, also complicates this possibility.
Redesigning Household Surveys
The third area of interest is the periodic redesign of household surveys. Based on ACS information, one might want to reselect PSUs or to resort
Page 38
housing units within a PSU (when using systematic sampling) to incorporate more up-to-date information. This could happen either more frequently than every 10 years or when ACS data indicated that it would be useful. However, there is a cost in making such changes. There is also an unknown amount of time that is required to implement a redesign. For example, if the counties (PSUs) are changed, then new interviewers would have to be trained in those areas. So a key question is whether more frequent redesign would be cost-efficient.
Supplying Covariates for (Variance-Reducing) Regression Models
Finally, there are complications concerning the addition of questions to the ACS questionnaire to support (combination of information) modeling. First, the ACS will be broad-based with respect to subject matter; that is, it will have relatively few questions for any specific area, e.g., income or health. For some models for some areas of interest, it may not be necessary to add many questions to the ACS questionnaire. However, it is likely that additional questions could be useful for models of many subject-matter areas. It is not possible to add hundreds of questions to the ACS, so some process for selecting additional questions will be needed. An important issue is whether these additional questions will be a permanent part of the survey. Other issues include where on the questionnaire these additional questions are placed. They could be incorporated in with other questions related to that subject area, or they could be placed at the end of the ACS questionnaire. The answer will affect nonresponse and quality of response of the other questions. Another key issue is the extent to which the information collected is increased with a question that is administered on a self-response basis (except for the CATI nonresponse cases).
Discussant Cathryn Dippo focused mainly on the practical issues raised if one were to use the ACS for redesigning current household surveys between decennial censuses. The most important feature for this discussion was the development of the master address file (MAF). Before turning to this topic, Dippo discussed some background issues.
The Bureau of Labor Statistics conducts some surveys where the units of interest are persons and some where the units of interest are households. Also, for the CPS, the real interest is in estimates of change over time, not estimates of level. In the second stage of the design of many household surveys (selecting housing units within PSUs), there is some implicit stratification based on the short form. The Consumer Expenditure Survey (CEX) is one example: before the 2000 census, it made use of information from the short form on household rent (the contracted rent amount for rental housing or the corresponding value of owner-occupied housing). Some surveys use area sampling at the second stage. If this is performed using sampling
Page 39
proportional to size, these size estimates have to be useful at low levels of geographic aggregation. Other related issues are the treatment of new construction and input into surveys that use random digit dialing.
A major issue concerns the operational aspects of changing PSUs between censuses and the costs associated with this. Every change to a PSU requires firing some interviewers and hiring and training the new ones. New interviewers, on average, obtain a lower response rate than experienced interviewers. With the consumer price index (CPI), for example, there are three interacting surveys that all use the same PSUs to take advantage of the benefit of experienced interviewers.
The real potential for ACS to provide assistance to household surveys is with respect to the MAF. Currently, the census address list, not comprehensively updated between censuses, is used until it is as much as 16 years out of date. For example, the CEX will use (essentially) the 1990 census address list until about 2006, when it begins implementing a new panel. The introduction of a continuously updated MAF is therefore a real advance. However, several questions can be raised: How good is the MAF going to be between censuses for seasonal housing units? How good is it going to be for new construction?
For the CPS, another consideration is that the ingoing and outgoing rotation groups are in neighboring segments, which helps in terms of panel correlations in composite estimation. If using the ACS means using different address lists, it may be difficult to maintain these correlations. One benefit may be to use symptomatic information from the ACS at a slightly aggregate level, such as blocks, for aiding in the efficiency of the sample design. Of course, the variance of and overall benefit from use of these symptomatic variables would be important to assess for this purpose.
If the ACS is going to be used to screen for various subpopulations, for example, to expand the CPS to increase the sample for specific race groups, the currency of the information collected when used for this purpose is an important consideration.4 Another consideration, mentioned above, is that the 3-month window used in the ACS for data collection from a sample may miss some recent renters, which is an important issue for the CPI Rent Survey. In addition, any information on the ACS nonsampling error structure would be useful to have. Toward this goal, it would be useful to include the ACS with the proposed CPS-Census match, especially to understand within-
4 The use of ACS for this purpose raises the hard problem of the sampling weight those people or households should get in subsequent analyses. Also, one needs to consider in the sample design that there will be both false positives and false negatives in the responses to the screening question.
Page 40
household coverage and the characteristics of the population that are undercovered relative to the decennial census.
Possibly the next most important step was to address the relevant policy issues more seriously now that large-scale data collection is a reality, since these issues are extremely difficult. The two most difficult and important policy issues are data sharing and the use of the ACS for screening. An example is the National Health Interview Survey, which uses screening to support an area sample, which is used since the Census Bureau could not provide access to its address list for a list survey. This lack of access is a serious problem. Second, with respect to the process of adding a question or a set of questions to the ACS questionnaire and the related costs, if the decennial census is the guide, this approach will raise a substantial problem since adding questions to the long form has proved to be a flawed process. Furthermore, the costs are a worry. The decision on which questions to include is one that requires years to plan. One would need to start addressing this problem right now for the 2003 date of full ACS implementation.
FINAL POINTS
A number of opportunities for ACS and current household surveys to jointly benefit from each other were suggested. The extent to which these advantages can be obtained depends on various things, e.g., the correlations between ACS responses and responses on household surveys, and the impact of the addition of new ACS questions on the quality of the response to existing ACS questions. Therefore, the benefits cannot be determined before more is learned about the ACS. Also, the benefits of an updated MAF for redesigning household surveys is complicated by several factors; thus, the degree to which ACS can be beneficial is an empirical question and needs further work.
