The Consumer Expenditure (CE) Quarterly Interview and Diary Surveys comprise a major program of the Bureau of Labor Statistics (BLS). The CE data are used to provide the market basket budget shares for one of the Nation’s most important statistics, the Consumer Price Index. The CE surveys’ unique and valuable data on the spending patterns of American consumers are used in a multitude of ways, including a new Supplemental Poverty Measure, IRS sales tax information, and economic research.
The design of the CE surveys must be updated periodically to align their methodology with changes in society, technology, consumer products, and consumer spending methods on survey estimates. Without such updates, the CE surveys will not be able to continue to fulfill their mission of producing high-quality expenditure estimates in support of their critical uses. All household surveys today face well-known challenges that include increasingly busy respondents, confidentiality and privacy concerns, many competing surveys, controlled-access residences, and non-English speaking households. In addition, the CE surveys face a unique challenge as the phenomena the surveys seek to measure have changed over the past 30 years, and continue to do so. Although the CE has made a number of improvements to the survey design, such as transitioning to computer-assisted personal interviewing (CAPI), it has not implemented changes within a systematic, comprehensive framework to address this, and other, challenges.
The CE surveys are faced with multiple issues that directly impact the quality of the data collected. These issues, presented in order of importance below, include (a) evidence of measurement error, most importantly underreporting, in the survey data, (b) changes in the survey environment due to both new technology and consumer behaviors, and (c) the need for greater flexibility in the mode of data collection. (See “Background Material” on p. 227 for links to more information.)
Measurement Error. Underreporting in the CE is evidenced by a growing deviation from other data sources and by the results of several studies. Since 1984, the ratios of aggregate expenditure estimates from the CE compared with Personal Consumption Expenditures (PCE) data from the National Accounts have shown a declining trend for many expense categories. Internal methodological studies and the 2008 CE Program Review had similar findings, providing further evidence of a growing concern about the quality of reported data. Underreporting in the CE may result from respondent burden due to survey length and complexity, panel or questionnaire conditioning, increasing telephone administration of a survey originally designed for personal visit interviews, proxy reporting by a single household member, recall effects stemming from a 3-month reference period, and/or other causes.
Changes in the Survey Environment. To remain effective, the CE surveys must adapt to changes in purchasing behaviors. For example, respondents may purchase a variety of types of items in a single large store, such as Costco or Walmart, rather than buying a single type of item from a single store. The topic-specific design of the current survey instrument may not best aid respondent recall or reporting of this type of buying. It is equally unknown whether the current survey instrument is effective for capturing on-line purchases or automatic payments. Questionnaire design and data collection methods may have to be adapted to better account for these issues. Parallel to the changes in consumer behaviors, a transformation has been occurring in the technology available for collecting data. The availability of new technology and software, such as a Web diary, a portable digital assistant (PDA) instrument, or financial software sheets, offers new opportunities to collect data. Finally, administrative sources of expenditure data (such as transaction databases built from credit/debit and from loyalty card use) now exist that provide potential alternatives to survey data.
Flexible Mode of Data Collection. One size does not fit all, and the CE needs greater flexibility in data collection modes for two main reasons: to be responsive to the needs of respondents and to allow faster implementation
of changes to collection instruments. Regarding the first, a multimode design would allow data collection to be tailored to the needs and preferences of the respondent. For example, some respondents have very little time, others have difficulty keeping a diary, and still others do not want an interviewer in their home. Each respondent would have different ways of optimally reporting their data. The second motivation for greater flexibility is to allow for design changes to be made responsively as society or technology changes. Currently making changes to the CAPI instrument requires considerable lead time, and even minor changes often impact the whole instrument. A survey design utilizing multiple modes, modules, and/or technologies may allow for changes to be made to a single mode without disrupting others.
National Academy of Sciences
The Bureau of Labor Statistics has undertaken a multiyear process to develop and implement a redesign of the CE surveys to address the above issues. As part of this process BLS has contracted with the National Academy of Sciences Committee on National Statistics (CNSTAT) to conduct an independent review of the design options available, and to make specific recommendations for redesign. CNSTAT has organized a panel of experts (subsequently referred to as the panel) from across the country to carry out that task. (Panel membership is presented in Appendix A.) The panel is seeking proposals for design options and a discussion of the relative merits of the options, particularly from organizations with experience in designing complex data collection methods. It is in this context that the panel initiated this Request for Proposal through the National Academy of Sciences. A subcommittee of the panel will serve as the selection panel for this contract.
Description, Scope, and Primary Tasks
The Contractor will produce a research report laying out a design (including research studies to evaluate the proposed design) to collect information required by the primary users of the current CE. The design should be robust to recent and potential changes in the data collection and retail environment discussed above. The proposed design should focus on features that are proactive in collecting information such as the use of scanners, PDAs, or external data sources. A recall survey would be used only to supplement information not otherwise available. Thus, the proposed design would be fundamentally different from that of the current Quarterly CE survey. NAS anticipates that a team approach may be needed to meet the conditions of this contract, and the Contractor may consult with experts outside its own organization.
The report will (1) summarize the Contractor’s evaluation that led them to reach the final proposal, (2) detail the proposed design, and (3) provide recommendations for evaluating the proposal in a CE-specific context. Each of these elements is specified below.
1. The Contractor will develop a carefully balanced evaluation of the prospective benefits, costs and risks of their proposed methodological and technological options compared to the current CE surveys. This evaluation should be
• developed and presented within the context defined by the primary user needs identified in the background material.
• based on extensive and carefully balanced review of:
o literature and industry knowledge on household and establishment survey methodology and practice that is currently available or likely to be available in practical form in the next five years.
o data collection technologies and external sources of expenditure data currently available or likely to be available in practical form with the next five years. Examples include, but are not limited to: bar code or receipt scanners, audio diaries, PDA instruments, and retail “loyalty card” or transaction-processor databases.
• reflective of the tradeoff between cost and improvement on measurement error in terms of external benchmarks (e.g., CE to PCE comparisons) and other standard evaluations of data quality (e.g., subgroup comparisons).
2. Based on results from the evaluation described above, the report should detail a comprehensive proposal for a survey design, and/or other data acquisition process, which collects the required information. The proposal should describe all relevant elements for the development and implementation of the proposed design, including, but not limited to:
• data sources;
• sample design, including size, weighting and precision implications;
• data collection mode (s) and/or description of alternative acquisition;
• data collection procedures;
• estimates of the cost requirements for development, implementation, and ongoing data collection; and
• estimates of time requirements for development and implementation of the design, as well as time required to collect and process data to create required estimates for publication.
• Underreporting of expenditures.
• Fundamental changes in the social environment for collection of survey data.
• Fundamental changes in the retail environment (e.g., online spending, automatic payments).
• The potential availability of large amounts of expenditure data from a relatively small number of intermediaries such as credit card companies.
• Declining response rates at the unit, wave, and item levels.
The proposal may include a moderate number of methodological and technological options, (e.g., a mixture of household-based and establishment/ intermediary-based data collection) but should go beyond simply listing possible options. Instead the report should provide, to the extent possible, concrete and specific design elements supported by in-depth explanations and evidence from the Contractor’s evaluation. In other words, any proposed design should be practical and supported by evidence to demonstrate feasibility and have a rationale for expectations of improvement to the issues associated with the CE surveys.
3. In addition to proposing a new CE design, the report should include clear recommendations for evaluating the proposal in a CE-specific context, such as a pilot study. The recommendation should include
• the design of a study or studies to evaluate the feasibility and effectiveness of the proposed design,
• a careful description of the ways in which the study or studies would provide practical insights into the proposed new design, and
• estimated financial and time requirements for the recommended study or studies.
The Contractor will review background material pertinent to the potential redesign of the CE. Many of these materials can be found at the Gemini Project website (see http://data.bls.gov/cgi-bin/print.pl/cex/geminimaterials.htm). These materials include, but are not limited to, the following:
- Gemini Project Vision Document.
- BLS Handbook of Methods, Chapter 16, Consumer Expenditures and Income.
- BLS Statement of CE data priorities (available May 1, 2011).
- CPI Requirements of the CE.
- CE Data Quality Definition Report.
- Papers presented at the June 2010 CE Data Users’ Needs Forum.
- Papers presented at the Consumer Expenditure Survey Methods Workshop, held December 8-9, 2010 in Suitland, MD.
- Papers presented at the March 2010 Data Capture Technology Forum.
The following deliverables are required under this contract:
- Contractor will participate in a kickoff meeting no later than fourteen (14) days following the award of this contract.
- Contractor will produce monthly progress reports on the project.
- Contractor will produce a draft report for NAS no later than ninety (90) days after the award of this contract. Draft report may be submitted electronically.
- Contractor will revise the draft report based on NAS comments and provide 8 copies of the full report developed under this proposal and based on Description, Scope and Primary Tasks listed above. Final report is due 120 days from the contract award.
Contractor will present the basic information in the full report to the panel at its upcoming Workshop on Redesign Options for the Consumer Expenditure Surveys, scheduled for October 26–27, 2011, and participate with the panel and other members of the workshop in a discussion of design options.