Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 47
4
The Panel’s Investigation into
the Issues with the CE
In 2009, the Bureau of Labor Statistics (BLS) initiated a systematic,
comprehensive study of the challenges faced by the Consumer Expenditure
Surveys (CE) with the goal of redesigning the existing surveys to reduce
measurement error. BLS states that the mission of this venture, known as
the Gemini Project, is
to redesign the Consumer Expenditure surveys (CE) to improve data
quality through a verifiable reduction in measurement error, particularly
error caused by underreporting. The effort to reduce measurement error
will combat further declines in response rates by balancing any expected
benefits of survey design changes against any potential negative effects on
response rates. Any improvements introduced as part of the Gemini Project
should not increase budgetary burden, but instead, should remain budget
neutral. (Bureau of Labor Statistics, 2011e, p. 1)
Since the beginning of the Gemini Project, BLS has undertaken a number
of information-gathering meetings, conference sessions, forums, and work-
shops to aid in its mission. All of these have provided valuable information
for the Panel on Redesigning the BLS Consumer Expenditure Surveys in its
current task, and many of the papers presented at them are cited in this
report. These events included the National Bureau of Economic Research’s
Conference on Improving Consumption Measurement (July 2009); Survey
Redesign Panel Discussion, cosponsored by the Washington Chapter of the
American Association for Public Opinion Research (DC-AAPOR) and the
Washington Statistical Society (January 2010); Data Capture Technology
Forum (March 2010); AAPOR Panel on Respondent Record Use (May
47
OCR for page 48
48 MEASURING WHAT WE SPEND
2010); Data User Needs Forum (June 2010); and CE Methods Workshop
(December 2010). More information can be found about these events, plus
copies of papers presented at them, on the BLS website (see http://www.
bls.gov/cex/geminimaterials.htm). Additionally, BLS has conducted internal
research in support of the Gemini mission and has contracted targeted re-
search from the private sector.
The panel commends BLS on its multiyear, systematic review of the
methodology used in the CE.
Building on the work of the Gemini Project, the panel investigated the
opportunities and drawbacks related to the CE. As described in this chapter,
their additional investigation included feedback from CE data users, panel
members’ reactions when they assumed the role of survey respondents,
and a workshop to learn more about other large-scale household surveys.
Redesign options developed by two outside groups in response to a Request
for Proposal also formed an important part of the panel’s investigations,
and the chapter concludes with some of the main points and discussions
elicited by these two options.
FEEDBACK FROM DATA USERS
The panel was diligent in reaching out to data users and trying to
understand the many uses of the CE. Many of those uses are outlined in
Chapter 2, including input into calculation of the Consumer Price Index
(CPI), development of government programs, and as the basis for research
and analysis. Several panel members are themselves regular users of the CE
microdata. The panel reviewed a broad set of published research that used
the CE as a source of information. As noted above, members studied the
papers from the BLS 2010 Data User Needs Forum. They also attended
conferences held by the National Bureau of Economic Research and held a
session with microdata users at the 2011 CE Microdata Users’ Conference.
Finally, the panel spoke one-on-one with many users of the CE data.
The panel studied the complexities of the CPI program and how the
CE supports those important indices. Considerable detail on this topic is
provided in Chapter 2, in the section “CE Data Provide Critical Input for
Calculating the Consumer Price Index.” From their investigation, the panel
made the following two conclusions.
C
onclusion 4-1: The CPI is a critical program for BLS and the nation.
This program requires an extensive amount of detail on expenditures,
at both the geographic and product level, in order to create its various
indices. The CPI is the current driver for the CE program with regard
for the level of detail it collects. The CPI uses over 800 different expen-
diture items to create budget shares. The current CE supplies data for
OCR for page 49
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 49
many of these budget shares. However, even with the level of detail that
it currently collects, the CE cannot supply all of the budget shares used
by the CPI. There are other data sources from which the CPI currently
generates budget shares.
C
onclusion 4-2: The CPI does not utilize the panel nature of the current
CE. Instead the national and regional estimates employed by the CE
assume independence of households between quarters on the Interview
survey, and independence between weeks on the Diary survey.
As discussed in the Chapter 2 section “The CE Provides Data Critical
in Administering Government Programs,” the CE is used by a number of
federal agencies to administer portions of their programs. To learn more
details about this particular use of the CE, the panel held in-depth con-
versations with staff at these agencies. A summary of those conversations
appears in Appendix C. From their investigation, the panel makes the fol-
lowing conclusion.
C
onclusion 4-3: The administration of some federal programs depends
on specific details collected from the CE. There are currently no other
available sources of consistent data across years for some of these
programs.
A third large group of users of the CE data are economic researchers
and policy analysts from academic institutions, government agencies, and
private organizations. These users work with tabular estimates produced
by the BLS, and increasingly with microdata files from the CE. The panel
talked with a number of these data users, researched the types of questions
that their analyses addressed, and the characteristics of the CE that were
important for those analyses. Many examples are provided in the Chap-
ter 2 section, “CE Data: A Cornerstone for Policy Analysis and Economic
Research.”
Much of this work is geared to understanding household behavior
and how households adjust their consumption in response to changes in
circumstances. These changes may be affected by personal events such as a
change in income, marriage, loss of a job, retirement, the birth of a child, or
the onset of a disability. Government program changes (such as tax reform,
adjustments in minimum wage, or health care legislation) can also impact
household behavior.
For data to be useful in this endeavor, users say it is necessary to have
panel data with at least two observations. Many analysts indicate the strong
advantage to having a third observation. A related issue is the length of
each panel period. Data collected over a short period, such as in the cur-
OCR for page 50
50 MEASURING WHAT WE SPEND
rent two one-week Diary surveys, are able to answer questions related to
how households respond to events that happen relatively frequently, such as
receipt of monthly Social Security benefits (e.g., Stephens, 2003). However,
a wide range of questions requires examining the same household both
before and after a less frequent event such as a tax rebate, a job loss, or a
divorce. These questions are more difficult to address with data collected
over a short time period unless the sample size is rather large.
Regardless of the period over which expenditure is measured, an impor-
tant complement is relevant household information over the same interval.
In order to examine whether changes in household circumstances lead to
changes in household consumption, these circumstances must be measured
during the same period. The principal variables of interest are income,
employment, retirement, disability, and marital status.
When panel data have been lacking, researchers have been able to cre-
ate panels by using “synthetic cohorts.” The idea behind synthetic cohorts
is that in place of following the behavior of the same individuals over time,
researchers can create a panel by modeling individual household activity
based on data from similar groups of households. Using these synthetic
cohorts, researchers can examine the relationship between changes over
time. While synthetic cohort data are more difficult to work with, they
may prove useful for answering some questions. However, for a number of
policy questions, synthetic cohort data do not provide a useful tool. Thus,
the following conclusion is made regarding use of the CE for research and
analysis purposes.
C
onclusion 4-4: Economic researchers and policy analysts generally do
not use CE expenditure data at the same level of detail required by the
CPI. More aggregate measures of expenditures suffice for much of their
work. However, many do make use of two current features of the CE
microdata: an overall picture of expenditures, income, and household
demographics at the individual household level; and a panel component
with data collection at two or more points in time.
PANELISTS’ INSIGHT AS SURVEY RESPONDENTS
Panel members wanted to gain firsthand insight into the CE from the
viewpoint of a respondent, so approximately three-quarters of panel mem-
bers were interviewed by a Census field representative. Most experienced
the Interview survey, one kept the Diary, and several did both. Box 4-1
provides some reactions of panel members to this experience. During the
process, panel members asked their own questions of the field representa-
tives. Thus, the interview experience for the panel was partly trying to recall
and answer specific questions about their own expenditures, and partly
OCR for page 51
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 51
BOX 4-1
Reactions of Panel Members Following Their Interviews with
Field Representatives (FRs)
“The FR said I was the first respondent to EVER consult paper and electronic
records extensively.”
“The FR interviewed me extremely quickly—fast talker. This seems to be the solu-
tion to respondent burden: get through as fast as possible.”
“I got a strong sense of how easy it becomes to say ‘no’ to a category simply
because saying ‘yes’ so clearly leads to more trouble.”
“After the interview, the FR told me about the suspicion of government and con-
cerns about intrusiveness that the FR regularly encounters, and it is much more
intense and extreme than I had expected.”
“The diary appears to have some very significant strengths compared to quarterly
recall. I did not see the immediate problems of being unable to respond to ques-
tions, as I experienced when doing the CE quarterly interview. This is a much
easier task, even though at first blush it seems like keeping a diary for two weeks
was going to be extraordinarily difficult.”
SOURCE: Committee on National Statistics Panel on Redesigning the Consumer Expenditure
Survey (2011).
trying to understand the overall nature of the interviews as experienced
by others. The field representatives made a number of comments to panel
members about their “typical” respondents and what they considered nor-
mal respondent behavior. The panel believes that this entire process brought
realism into their discussion of the cognitive issues and potential solutions
(Committee on National Statistics Panel on Redesigning the Consumer
Expenditure Survey, 2011).
HOUSEHOLD SURVEY PRODUCERS WORKSHOP:
DESCRIPTION AND INSIGHTS
Many of the problems and issues facing the CE are also faced by other
large household survey programs, and the panel wanted to leverage the
work done on these surveys toward solutions for the CE. In this endeavor,
the panel planned and held a Household Survey Producers Workshop in
June 2011 in Washington, DC. (The agenda for the workshop appears in
Appendix E.) The panel would like to extend its appreciation to the present-
ers at this workshop for the insights they provided.
OCR for page 52
52 MEASURING WHAT WE SPEND
BOX 4-2
Sessions at the Household Survey Producers Workshop
Session 1: Alternative Ways of Measuring Consumer Expenditures—International
Experiences
Session 2: Designs That Add Flexibility in Data Collection Mode
Session 3: Designs That Effectively Mix Data from Multiple Surveys and/or
E
xternal/Administrative Data to Produce Estimates
Section 4: Designs That Effectively Mix Global and Detail Information to Reduce
Burden and Measurement Error
Session 5: Designs That Use “Event History” Methodology to Improve Recall and
Reduce Measurement Error in Recall Surveys
Session 6: Diary Surveys That Effectively Utilize Technology to Facilitate Re-
cordkeeping or Recall
NOTE: See Appendix E for the full agenda of the workshop.
The program for this workshop was built around six topics (see
Box 4-2), each of which was specific enough to inform the panel’s redesign
deliberations yet broad enough to be able to present different perspectives
of the topics. After different presentations on a topic, one member of the
panel discussed the insights that these presentations had for the CE rede-
sign. A summary of the main points raised in the six sessions follows.
Session 1: Alternative Ways of Measuring Consumer
Expenditures—International Experiences
The purpose of this session was to have representatives from other
countries talk about how they collect consumer expenditures, the issues
they face, and their approach to these issues. The panel was looking for dif-
ferences and similarities that might inform redesign options for the U.S. CE.
Dubreuil et al. (2011) discussed how Statistics Canada redesigned
its consumer expenditure survey. The new design of Canada’s Survey of
Household Spending looks similar to the current CE in the United States. It
uses a combination of a recall interview and 14-day diary for each selected
household, with varying recall periods for different expense items. The
previous Canadian design incorporated a “balance edit,” a feature that a
number of users of the CE would like to see incorporated in the CE rede-
sign. The Canadian redesign no longer includes this feature.
Horsfield (2011) discussed the UK Living Costs and Food Survey. It is
OCR for page 53
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 53
a relatively short household survey to collect regular expenditures such as
rent and mortgage payments, along with retrospective information on cer-
tain large, infrequent expenditures such as those on vehicles. The program
predominantly uses a Diary survey, as each individual aged 16 and over is
asked to keep diary records of daily expenditures for two weeks. Children
(aged 7–15) complete a simplified diary. Household members receive in-
centives for completing the diary: 10 pounds ($15.68) per adult, 5 pounds
($7.84) per child.
Borg (2011) discussed consumer expenditure surveys in Europe and the
European Union’s efforts to harmonize survey results. The EU countries all
have their own expenditure surveys carried out under the responsibility of
their national statistical offices. These surveys are generally periodic rather
than annual. The primary purpose of these surveys is to produce the budget
shares for the national consumer price indices, although there has been an
increasing use of the information at both the national and EU levels. There
remain some comparability issues among these surveys.
Session 2: Designs That Add Flexibility in Data Collection Mode
One of the primary reasons for the CE redesign is the need to update
data collection strategies to create greater flexibility in the data collection
mode. The Interview survey is conducted in person, with a fallback to tele-
phone interviewing when a personal visit is not feasible. The Diary survey is
dropped off and picked up in person, and the diary information is collected
on paper forms. This session provided examples of how other surveys are
incorporating response flexibility or newer data collection methods.
Smyth presented results from Olson, Smyth, and Wood (2011), an ex-
periment within the ongoing Nebraska Annual Social Indicators Survey that
allows the respondents to choose their mode preference. The experiment
then uses the respondent’s preferred mode of data collection and tests to see
if this treatment makes a difference in response. In this limited experiment,
they found that response rates were higher for those being surveyed in
their preferred mode. They also found that Web survey response rates were
lower than those with mail and phone contacts across all preference groups.
However, they found that results changed within a mixed mode framework.
The Business Research & Development Innovation Survey (BRDIS),
conducted by the Census Bureau for the National Science Foundation’s Na-
tional Center for Science and Engineering Statistics, is the nation’s primary
source of information on business R&D expenditures and the workforce.
Hough (2011) reported that unlike its predecessor, which was sent to a
single respondent within a company, the new BRDIS questionnaire is struc-
tured to allow and encourage different experts within a single business to
provide responses in their areas of expertise. There are both paper and elec-
OCR for page 54
54 MEASURING WHAT WE SPEND
tronic versions of questionnaires. They have also developed an online tool-
kit to assist business respondents that includes spreadsheets, fillable PDFs,
and personalized support by an account manager. Clearly, establishment
surveys are different from household surveys in many ways, but there are
similarities from which to extract ideas, such as multiple mode options and
a toolkit for respondents. Different households and members of the same
household might have a different comfort level with different collection
modes. A key point from this presentation is that “one size” does not fit all
respondents. The BRDIS recognizes that point up front and designs it into
its methodology. Another point is the toolkit to further assist respondents.
Wine and Riccobono (2011) discussed the National Postsecondary
Student Aid Study (NPSAS), a survey conducted by RTI International for
the National Center for Education Statistics that mixes multiple sources of
data and data collection modes with incentives to obtain and keep student
respondents.
NPSAS data come from multiple sources, including institutional records,
government databases, and student interviews. Detailed data on par-
ticipation in student financial aid programs are extracted from institu-
tional records. Data about family circumstances, demographics, education
and work experiences, and student expectations are collected from stu-
dents through a web-based multimode interview (self-administered and
computer-assisted telephone interviews [CATI]). (National Center for Edu-
cation Statistics, 2012)
A tailored incentive program is designed into the process to encourage early
response.
Session 3: Designs That Effectively Mix Data from Multiple Surveys
and/or External/Administrative Data to Produce Estimates
Some of the information collected on the CE may be available in ad-
ministrative records or collected on other government surveys. This session
highlighted surveys that, while collecting large quantities of information
themselves, also utilize administrative records and/or combine data from
other survey data collections to reduce the overall burden of the survey or
to improve the overall quality of data.
Machlin (2011) described how the Medical Expenditure Panel Survey
(MEPS) matches survey data to health records, combining them with in-
formation collected from household members and their medical providers.
Upon completion of the household interview and obtaining permission
from the household survey respondents, a sample of medical providers are
contacted by telephone to obtain information that household respondents
cannot accurately provide. This part of the MEPS is called the Medical
OCR for page 55
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 55
Provider Component (MPC), and information is collected on dates of visit,
diagnosis and procedure codes, charges, and payments. The Pharmacy
Component (PC), a subcomponent of the MPC, collects drug detail infor-
mation, including National Drug Code (NDC) and medicine names, as well
as date(s) prescriptions are filled, sources, and amounts of payment (Agency
for Healthcare Research and Quality, 2012).
O’Brien (2011) discussed the Residential Energy Consumption Survey
(RECS), which collects a multitude of information on houses, appliances,
and home energy usage. It collects utility records from energy suppliers in
lieu of self-reports from respondents. As part of this process, the interviewer
asks household respondents to name their energy suppliers and to produce
a bill from each supplier. The interviewer uses a portable scanner to scan in
the bills. The Energy Information Agency then contacts the energy suppliers
to obtain records for the sampled household unit for the previous year (U.S.
Department of Energy, 2011)
Schenker and Parsons (2011) discussed combining data from multiple
surveys to improve quality and reduce burden within the survey program
of the National Center for Health Statistics (NCHS). They provided four
examples:
• Combining information from the National Health Interview Survey
(NHIS) and the National Nursing Home Survey to obtain more
comprehensive estimates of the prevalence of chronic conditions
for the elderly;
• Using information from the National Health and Nutrition Exami-
nation Survey (NHANES) to improve analyses of self-reported data
on the NHIS;
• Combining information on the Behavioral Risk Factor Surveillance
System with the NHIS to enhance small-area estimation; and
• Creating links between various NCHS surveys and administrative
data sources such as air quality data available from the Environ-
mental Protection Agency, death certificate data from the National
Death Index, Medicare enrollment and claims data from the Cen-
ters for Medicare & Medicaid Services, and benefit history data
from the Social Security Administration.
Session 4: Designs That Effectively Mix Global and Detail
Information to Reduce Burden and Measurement Error
This session highlighted surveys that, while collecting large quantities
of information, do so using design strategies and questionnaire modules
that avoid asking every respondent for all details on each contact.
Aune (2011) discussed the Agricultural Resource Management Survey,
OCR for page 56
56 MEASURING WHAT WE SPEND
an expense and income survey of farming establishments conducted by the
National Agricultural Statistics Service. It is an annual survey that collects
detailed information related to the farming enterprise and, to a lesser ex-
tent, to the farm household. This survey has multiple modules or versions,
with sample units assigned to a specific version during the selection process.
Most versions are designed for personal enumeration, but one is designed
for mail/Web collection. For a given expense item (such as fuel expenses),
some versions will ask only the global expense item (total spent on fuel
of all kinds) and others will ask a detailed breakout of that expense item
(amount spent on gasoline, diesel, propane, etc.). Regardless of the version
and mix of global/detail questions, all data are combined in summary esti-
mates and contribute to the state, regional, and national estimates.
Fields (2011a) discussed the current structure of the Survey of Income
and Program Participation (SIPP) and its use of both “core” and “topical”
questionnaire items. The SIPP follows households for multiple waves. Core
questions are asked in all waves, such as the global item “total income.”
Topical questions are those that are not repeated in each wave. Topical
modules are designed to gather specific information on a wide variety of
subjects. Some topical modules cover items such as assets and liabilities,
real estate property, and selected financial assets. In some instances, the
topical questions are intermixed with core questions in the interview to
make the questionnaire flow more smoothly.
Gentleman (2011) discussed two alternatives for asking questions about
the entire family in the National Health Interview Study. The first alterna-
tive asks a global question “does anyone in the family. . . .” An alternative
questionnaire goes through the family roster and asks individual questions
for each family member. The NHIS is also used as a screening vehicle for
follow-on surveys, with many detailed questions saved for those follow-on
surveys. One result from their experiments on screening questions showed
that respondents gave fewer “yes” answers to filters as they learned that
such answers led to additional questions.
Session 5: Designs That Use “Event History” Methodology to
Improve Recall and Reduce Measurement Error in Recall Surveys
This session highlighted surveys that utilize “event history” method-
ology to improve the quality of recalled information. The Panel Study of
Income Dynamics (PSID) was the first major survey to implement “event
history” methodology to improve the ability of respondents to recall infor-
mation. Stafford (Beaule and Stafford, 2011) discussed the implementation
of this methodology in the PSID, which has been a prototype for other
surveys. They conducted a number of methodological studies as they de-
veloped this methodology.
OCR for page 57
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 57
Fields (2011b) discussed a newly redesigned SIPP that uses event his-
tory methodology, pulling from the experiences of the PSID. The SIPP staff
believe they may be able to use a one-year recall period as effectively and
accurately with this new methodology as the current design, which uses a
four-month recall. The new design is scheduled to be operational in 2014.
This presentation discussed the implementation of “event history” method-
ology and presented what has been learned so far with the pilot program.
Session 6: Diary Surveys That Effectively Utilize
Technology to Facilitate Recordkeeping or Recall
Newer technology, such as the Web, smart phones, and portable scan-
ners, has opened possibilities for diary surveys. This session highlighted
surveys that utilize this newer technology to field innovative diary-type
surveys.
The National Household Food Acquisition and Purchase Survey
(FoodAPS) is a new pilot survey sponsored by the U.S. Department of Ag-
riculture designed with an innovative approach to a food diary. Cole (2011)
discussed the survey, which collects information on food sources, choices,
quantities, prices, timing of acquisition, and nutrient characteristics for all
at-home and away-from-home foods and beverages. It also collects house-
hold information that may influence food acquisition behaviors. The pilot
uses color-coded booklets, portable scanners for receipts, regular telephone
contact to encourage diary-keeping, and incentives as part of the data col-
lection process (U.S. Department of Agriculture, 2011).
Kizakevich (2011) discussed personal diary and survey methodologies
for health and environmental data collection used by RTI International.
Among these examples were
• PFILES, a real-time exposure-related diary of product use and
dietary consumption in the context of activity, location, and the
environment. It uses Pocket PCs with headsets for use by respon-
dents, who record survey responses and even take pictures of their
environment;
• Personal Health Monitor for use by patients suffering from post-
traumatic stress disorder and mild traumatic brain injuries to help
clinicians monitor patients’ status while observing symptoms and
medication usage within the context of daily activities and environ-
mental factors; and
• BreathEasy, an Android App, which allows a daily assessment of
asthma triggers, health, and ventilation.
OCR for page 58
58 MEASURING WHAT WE SPEND
Bailey (2011) discussed Nielsen Life360 Program, which uses a “digital
ethnography” approach to measure attitudes, preferences, and behaviors of
the targeted population using mobile phone surveys, photography, Internet-
based journals, video cameras, and Web surveys. The specially equipped
smart phone prompts respondents to complete a short survey on an hourly
basis in addition to capturing an image using the built-in camera as a pic-
ture description of their surroundings and activities in real time.
Summary of the Workshop
The panel found the workshop presentations to be highly informative
and to provide important input into panel deliberations. A number of key
points emerged from the prepared remarks that discussants delivered dur-
ing the workshop, as amplified in subsequent discussion among panelists.
First, the international comparisons demonstrate that concerns about
data quality and burden that have led to the need for a redesign of the
CE are not unique to U.S. data collection efforts, although the size of and
variability among the U.S. population present particular challenges. The
alternate methods that the panel observed from other countries made clear
that a bounding interview is not a universal method, and that it is plausible
to rethink this aspect of CE administration. It was also clear to the panel
that, although the methods and approaches from other nations have many
strengths, they also have their own challenges, and simple wholesale adop-
tion of those methods is unlikely to be a panacea for improving the CE.
Second, adding new modes of data collection needs to be done thought-
fully, attending carefully to whether adding new modes or providing re-
spondents with a choice of mode increases data quality, reduces respondent
burden, or reduces nonresponse sufficiently to be worth the design, opera-
tional, and analytic costs. As the Smyth presentation in session 2 illustrated,
the scientific community is not yet at a point to fully understand why par-
ticular modes work for different respondents.
Third, while it is quite attractive to consider replacing or supplement-
ing respondent-reported data with data from other sources (administrative
records, data from other surveys) to reduce respondent burden and admin-
istrative costs, this is not as straightforward an enterprise as it might seem.
The hurdles are notable enough—from mode and questionnaire differences,
to sampling and weighting incompatibilities, privacy and confidentiality
issues, linkage difficulties, increased agency efforts, data sharing difficul-
ties, and lack of knowledge of costs—that it does not seem plausible to the
panel that alternate sources could suffice in the short term. There is also
considerable concern about whether external data would be consistently
available over time.
Fourth, the panel was impressed by efforts in other U.S. surveys to
OCR for page 59
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 59
streamline data collection and rethink what kinds of general and specific
information need to be asked of respondents. Although the immediate ap-
plications to the CE of the particular approaches described at the workshop
are not entirely clear, and although much remains to be understood about
the relationship between survey length and respondent burden, the panel’s
subsequent deliberations and proposals were influenced by such efforts.
Fifth, whether or not event history methods are the only or best way
to stimulate all respondents’ recall, the panel took note of the insight that
emerges from studies of alternative interviewing methods. A redesigned CE
needs to go as far as it can to accommodate respondents’ natural ways of
thinking about and recalling their expenditures, rather than asking respon-
dents to conceive of their expenditures from the researcher’s perspective.
More broadly, assuming that respondents can recall purchases accurately
without consulting records is problematic, and a redesigned CE needs to
promote the use of records far more than current methods do.
Finally, the panel took very serious note of the opportunities for us-
ing new technologies to facilitate more direct and in-the-moment self-
administered reporting of expenditures, as well as for passive measurement
of expenditures. It will be important for a CE redesign to make as much
use of these opportunities as feasible, and to start a new forward-thinking
mode of research and production that continually assesses the changing
technological landscape and prepares as much as possible for changes be-
fore they happen.
REDESIGN OPTIONS WORKSHOP: DESCRIPTION AND INSIGHTS
In order to elicit a broader perspective on possible solutions to the CE’s
problems, the panel sought formal input from organizations with experi-
ence in designing complex data collection methods. It is in this context that
the panel initiated a Request for Proposal (RFP) and competitively awarded
two subcontracts, one to Westat (project leader: David Cantor) and the
second to a consortium from the University of Wisconsin–Milwaukee,
University of Nebraska–Lincoln, and Abt-SRBI (Nancy Mathiowetz, proj-
ect leader; Kristen Olson; and Courtney Kennedy). The Statement of Work
(see Appendix D) required the subcontractors to produce a comprehensive
proposal for a survey design, and/or other data acquisition process, that
collects the data required for the primary uses of the current CE while ad-
dressing the following issues:
• Underreporting of expenditures
• Fundamental changes in the social environment for collection of
survey data
OCR for page 60
60 MEASURING WHAT WE SPEND
• Fundamental changes in the retail environment (e.g., online spend-
ing, automatic payments)
• The potential availability of large amounts of expenditure data
from a relatively small number of intermediaries such as credit card
companies
• Declining response rates at the unit, wave, and item level
The full reports from Mathiowetz, Olson, and Kennedy (2011b) and
Westat (2011c) are available online, and the panel summarizes them in
this chapter. The panel would like to commend both subcontractors on
the reports they submitted. Both designs were innovative and well thought
out. The time frame was very short for completing this contract, and both
groups met the challenge. Their work provided very valuable input into the
panel’s work from specific design options, use of technology, and review
of relevant literature. The panel used their research and ideas extensively.
The panel hosted a Redesign Options Workshop on October 26, 2011,
and an informal roundtable on October 27 to facilitate a public discussion
of the two proposals and their relative merits in regard to the current CE.
An agenda for the workshop is in Appendix F. Kulka (2011) discussed both
reports with a focus on the cognitive issues related to the CE, and Bowie
(2011) talked about issues relative to implementing major changes in a large
ongoing survey. Data users also addressed the proposed redesigns from the
perspective of their use of the CE data.
A number of important insights arose from the discussion of these pro-
posals. One concerned the amount of detail that is required for the CE. A
strong opinion was offered that one cannot collect that quantity of detail
without a lot of measurement error. An understanding of what constitutes
a tolerable measurement error must be clear, followed by a move back to
collecting data at a more aggregate level for the prescribed level of quality.
Another important round of discussion concerned the amount of additional
research that would be needed to be ready to field a newly redesigned CE
survey. If the redesign includes fairly major changes, as did the two propos-
als offered at the workshop, then a significant amount of targeted research
will lie ahead.
Most important, these discussions led BLS senior management to mod-
ify their original charge to the panel. In this modified charge (see Appendix
B), the panel is asked to view the Consumer Expenditure Survey (CE) Data
Requirements (Henderson et al., 2011) as the mandatory requirements for
the survey. The CPI data requirements document (Casey, 2010) was no
longer a part of the mandatory requirements that the redesign would need
to meet.
OCR for page 61
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 61
Redesign Proposal: Westat
Westat’s proposed redesign (Westat, 2011b,c) focuses on three inter-
related goals: (1) reducing respondent burden, (2) incorporating admin-
istrative and personal record information, and (3) improving self-report
methodology. It calls for greater reliance on records and less reliance on
respondent recall. Key features of this proposal and a link to the full report
are provided in Box 4-3. The Westat proposal continues from the base of
separate diary and interview surveys, but implemented differently than
in the current CE. It introduces the concept of a “data repository” and a
separate Administrative Record Survey to obtain certain records directly
from retailers, utilities, and mortgage companies. The authors discussed
their deliberations concerning access to external data:
Data obtained directly from retailers are likely to be more accurate than
respondent-provided data are likely to be. Other federal surveys, such
as the National Immunization Survey and the Residential Energy Con-
sumption Survey, have employed administrative data to supplement and
improve the quality of data reported by respondents. Conceivably, CE
respondents could provide their loyalty card numbers to interviewers, who
would then ask the retailers to provide the purchasing histories for those
loyalty cards. This method would not be perfect; a consumer may some-
times forget to give a loyalty card to the cashier or may lend the card to
friends. Moreover, retailers do not routinely release purchasing histories.
BOX 4-3
Key Features of the Westat Proposal
• eparate diary and interview surveys but implemented differently than the
S
current CE.
• ultiple diary-keepers within a household.
M
• ata repository into which respondents can upload scanned receipts and
D
records.
• ata electronically extracted from receipts and records, and a Web survey
D
electronically generated to request missing information.
• wo recall interview surveys, one year apart. Variable recall periods used.
T
• espondents contacted three months before recall interview and encouraged
R
to keep and scan receipts during three-month period.
• onsent requested to obtain expenditure records directly from retailers, utili-
C
ties, and mortgage companies. A separate administrative records survey to
obtain those records.
NOTE: Link to full report: http://www.bls.gov/cex/redwrkshp_pap_
westatrecommend.
pdf and http://www.bls.gov/cex/redwrkshp_app_westatrecommend.pdf.
OCR for page 62
62 MEASURING WHAT WE SPEND
The BLS might explore the feasibility of obtaining purchasing history data
by contacting large retailers with loyalty card programs. Expenditure data
is also potentially available from utility companies, rental agents, and lend-
ers. (Westat, 2011c, p. xiii)
For the Diary survey, each person age 14 and over in a sampled house-
hold would be asked to report expenditure data for 14 days. Having
multiple respondents minimizes concerns about proxy reporting. The re-
spondents are given a variety of reporting options. They could use the cur-
rent paper diary forms, mail in their receipts and records, or report data
electronically. All respondents are asked to save and then supply receipts. A
key component of the redesigned Diary survey is a “data repository” into
which respondents upload various types of expense records. The repository
system would extract purchase data from the uploaded records/receipts
and generate a Web survey that would ask the respondent to supply any
remaining information that the CE program needs about those purchases.
Respondents who chose to report their data electronically would be given
a portable scanner. Using specially designed software, they would e-mail
files of their scanned receipts and other records of purchases to the data
repository.
Respondents would also be asked to download data files from various
financial accounts and e-mail these files to the data repository. Respondents
could opt to report their data by mailing in their receipts and financial
statements, and staff would scan these receipts into the data repository.
The authors discussed in their report the potential of asking respondents
to supply financial records:
Consumers today commonly make purchases using modes that leave an
electronic record. When an electronic record exists, respondents poten-
tially could provide the expenditure data by retrieving information about
the purchase from a database, or by printing out a record of the purchase,
rather than by trying to remember the details of the purchase or by find-
ing a receipt. For example, soon after a consumer makes a purchase using
credit card, debit card, check, electronic fund transfer, or PayPal, a record
of the transaction appears in a file that can be downloaded from the web-
site of a financial institution. When a consumer makes an online purchase,
the vendor typically sends a confirmation email, or provides a confirmation
page, that the consumer can print out. The CE program does not currently
ask respondents to provide these electronic records of expenditures. Their
potential role in the CE data collection process deserves attention. These
records cannot provide all of the data required by the CE program, how-
ever. They potentially offer a way to help respondents remember expendi-
tures without a great deal of effort. (Westat, 2011c, p. 5)
OCR for page 63
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 63
Respondents would also be asked to provide consent to collect their
purchasing history data directly from retailers. The authors discuss this
recommendation, saying:
If CE respondents provided their loyalty card numbers, and retailers were
willing to release purchasing data, the CE program would have access to
objective information about the respondents’ expenditures. Of course, this
idea has some drawbacks. Consumers sometimes forget to provide their
loyalty card to the cashier when they make a purchase. Some consumers
may lend their loyalty cards to friends. Also, most retailers, including
Walmart, have no loyalty card programs. (Westat, 2011c, p. 6)
A field representative would monitor the respondent’s reporting activ-
ity, increasing contact and assistance to those not reporting regularly. A
telephone or personal visit to the household would be scheduled after 7
and 14 days: a telephone interview for those that have been providing the
information on a regular basis and a personal interview for households that
have not been providing the information.
For the Interview survey, Westat proposes a change from the current
data collection schedule. A new panel would enter the CE program each
quarter, so that four panels would enter each year. The first wave of data
collection for each panel would begin with a bounding interview, followed
three months later by a recall interview. The second wave of data collection
would start nine months after the recall interview. Westat (2011c, p. 53)
indicated that this change is made to reduce both cost and burden: “The
current design has a total of five in-person interviews per household, creat-
ing significant cost and respondent burden. Reducing this number to three
in-person interviews would substantially reduce this burden and may lead
to greater cooperation, fewer dropouts, and better data quality.”
At the start of the second wave, the household would receive a package
via U.S. mail reminding them to resume data collection activities, includ-
ing keeping receipts. If the household has changed, it would receive a per-
sonal visit. Data collection would end with a recall interview three months
later. Westat also proposes a change to the recall period, so that it varies
by expense item (one-, three-, or 12-month recall). The proposal places a
very strong emphasis on having households save receipts and use records.
Respondents would also be asked to provide consent for collecting their ex-
penditure history data directly from retailers, utilities, and mortgage com-
panies. Respondents would be encouraged to scan receipts and records into
the data repository as they receive them, rather than waiting for the field
representative’s return interview. As with the Diary survey, the repository
would generate a Web survey based on the information still needed about
the receipt/record. The field representative would monitor the number of
OCR for page 64
64 MEASURING WHAT WE SPEND
records/receipts coming in during the three-month period and contact by
telephone households that were not turning in receipts regularly.
The redesign also includes a separate Administrative Record Survey
that would be developed to obtain records directly from retailers, utilities,
and mortgage companies. The emphasis on obtaining data from records
rather than the respondent’s memory is intended to improve data quality
and reduce respondent burden.
Westat estimated that the proposed diary redesign would cost ap-
proximately 60 percent more than the current diary survey. This increase in
cost is primarily attributable to having multiple diary-keepers within each
household. Without a budget increase, the number of sampled households
would have to be reduced accordingly, and the precision of the estimates
would therefore also diminish. Westat estimates that the proposed inter-
view redesign would cost approximately twice that of the current interview
survey. The increase is attributable to the increased effort in contacting
more households, an effect of reducing the number of panels. The new
Administrative Record Survey contributes to cost increases reported for
both surveys. The redesigned methods for the interview survey result in
some increase in precision of the estimates, due to eliminating the within-
household correlation across panel waves within the same year. This offset
does not entirely make up for the increase in cost. The report provides a
simulation of the effect on the precision of the estimates using one-, three-
and 12-month reference periods (Westat, 2011c).
Redesign Proposal: Mathiowetz, Olson, and Kennedy
The Mathiowetz/Olson/Kennedy proposal (Mathiowetz, Olson, and
Kennedy, 2011a,b) recommends a single integrated sample design, with
two components: (1) a cross-sectional one-month diary, and (2) a panel
component for which a household would complete the one-month diary for
three different waves within the year. The proposed design makes extensive
use of tablet computers, receipt scanners, and flexible memory “triggers.”
Box 4-4 provides key elements of this proposal and a link to the full report.
The design provides for active monitoring of the diary-keeping activities of
household members, with interventions when this activity appears inad-
equate. Their design minimizes the reliance on retrospective recall, elimi-
nates the need to combine data from two distinct surveys, and provides an
important panel component within the data structure.
In discussing the advantages to their proposed design, the authors
stated
Our design addresses the issue of underreporting by minimizing reliance
on retrospective reporting, promoting “real time” recording of all ex-
OCR for page 65
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 65
BOX 4-4
Key Features of the Mathiowetz/Olson/Kennedy Proposal
• iary only
D
• ross-sectional sample households keep diary for one month. Panel com-
C
ponent keeps one-month diary three times during the year.
• ultiple diary-keepers within the household
M
• se of tablet PCs with Internet connection to permit real time uploading of
U
data
• ngoing monitoring of uploaded data with feedback to household
O
• se of memory triggers encouraged
U
NOTE: Link to full report: http://www.bls.gov/cex/redwrkshp_pap_ btsrbirecommend.
a
pdf.
penditures and payments, and emphasizing self reporting among all CU
members. The use of a web-based diary, via web-enabled tablets, provides
an efficient means by which each member of the CU can log on to his or
her own personal diary to record expenditures. The flexibility and com-
puting power of a tablet will allow CE staff to develop an instrument that
minimizes burden (e.g., pick lists; scanning of receipts and barcodes; ease
of selecting repeat purchase items) and facilitates consistency in reporting
at the level of detail necessary for the CPI. We envision a data collection
approach with the tablet that allows for the use of apps, integration with
other technology, online help for the CU members, and real time moni-
toring of diary entries by the CU. (Mathiowetz, Olson, Kennedy, 2011b,
p. 11)
Each adult (age 16 and older) member of the selected household would
be asked to keep a 30-day diary, reporting expenditures “real-time” dur-
ing that period. Younger children (aged 7–15) would be asked to keep a
“mini diary” for that same time period. During an initial personal visit to
the selected household, the field representative would collect demographic
and socioeconomic data, including asking some global questions related
to certain expenditures and annual income. The field representative would
probe about regular monthly payments for housing and utilities, and any
automatic payment schedules. The 30-day diary process would be explained
with appropriate training on use of the diary tools. The authors provide
their rationale for multiple diary-keepers within the household.
With respect to multiple reporters per CU, the limited literature suggests
that the use of multiple diaries per CU increases the reporting of expen-
diture items and CU expenditures (Grootaert 1986; Edgar et al. 2006).
OCR for page 66
66 MEASURING WHAT WE SPEND
If the source of the increasing discrepancy between CE and the Personal
Consumption Expenditure data from the National Accounts is due to mea-
surement error, then increasing self reports and minimizing recall periods
are two well established means for improving data quality (Bound, Brown
and Mathiowetz 2001). Furthermore, the use of technology, in which each
member of the CU can log in to his or her individual diary with their own
login and password, permits persons who make purchases that they would
rather not have other members know about to answer confidentially (e.g.,
teenagers not wanting their parents to know about certain purchases),
more so than if a paper diary is used (e.g., Stinson, To and Davis 2003).
(Mathiowetz, Olson, and Kennedy, 2011b, p. 11)
During the 30-day diary period, household members would be asked
to keep receipts and record expenditures on a real-time basis using one
or more of the diary tools provided, with a computer tablet with Internet
connection as the primary recording tool. The tablet would be available for
use by all household members. It would feature an instrument that mini-
mizes burden and facilitates consistency in reporting of required details.
An attached scanner and bar code reader would facilitate data capture of
products and receipts. The proposal also recommends the use of a person-
alized e-mail account to forward receipts and electronic records. The field
representative would conduct a “wrap-up” interview and data review at the
end of the diary period, with retrospective questions asked as needed to fill
gaps in the diary-keeping.
Mathiowetz/Olson/Kennedy encourage the adoption of multiple por-
table means of capturing triggers that help household members remember
a purchase so that it can be recorded later. These include the respondent’s
own smart phone to record pictures, voice recordings, and notes. A small,
simple pocket diary could also be used as a memory trigger.
A panel component of the design is recommended to better support
micro-level analysis for the entire year. It is formed by a subset of the overall
sample that is asked to complete a 30-day diary for months 1, 7, and 13.
The authors deliberated on the best length for the diary reporting interval
and the panel, stating that:
A critical design issue is the length of the panel—that is, for how many
weeks or months we ask CU respondents to serve as diarists. This is defi-
nitely an issue of cost-error tradeoffs, one that impacts the costs of data
collection, the willingness to participate, the extent to which the data are
impacted by panel conditioning/fall-off in reporting, and the need for
month-to-month and/or year-to-year comparisons among the same CUs.
No single design can optimize for all of these objectives, which is why we
are recommending both a cross sectional and a panel component to the
single integrated sample approach. (Mathiowetz, Olson, and Kennedy,
2011b, p. 15)
OCR for page 67
THE PANEL’S INVESTIGATION INTO THE ISSUES WITH THE CE 67
This proposal recommends that BLS continue to look at sources of
administrative data for benchmarking and microlevel use. Mathiowetz/
Olson/Kennedy discuss a number of existing data sources, including three
federal surveys that might be used to benchmark CE data. The authors
also discuss nonfederal sources of data but do not incorporate a specific
recommendation for their use into the current proposal. They state that
they “were initially optimistic about micro-level integration of non-federal
administrative data sources with CE data. However, the current state of
knowledge about these 16 sources and the incredible task involved in turn-
ing administrative records from private companies into survey data for all
sampled persons makes us cautious in recommending their use for purposes
other than nonresponse monitoring and benchmarks” (Mathiowetz, Olson,
and Kennedy, 2011b, p. 16).
Summary of the Two Proposals
While the panel does not recommend implementing either of these
two designs wholesale, the designs embody important insights that became
central to its deliberations, and aspects of each design are incorporated
into one or all of the panel’s three proposed designs presented in Chapter
6. Both proposals place renewed emphasis on the use of survey personnel
to provide help, consultation, and monitoring of respondents’ efforts, and
the panel’s thinking was clearly inspired by this model.
The most notable adoption from the Mathiowetz/Olson/Kennedy pro-
posal is a focus on supported self-administration and the use of a tablet
data collection interface. These concepts are a central feature in all three
of the panel’s prototypes described in Chapter 6. One prototype, Design
A, Detailed Expenditures Through Self-Administration, follows much of
the Mathiowetz/Olson/Kennedy proposal, as does the diary component of
Design C, Dividing Tasks Among Multiple Integrated Samples. The panel’s
proposed designs were inspired, in different ways, by the Westat proposal’s
strong focus on encouraging the use of records. Design C, Dividing Tasks
Among Multiple Integrated Samples follows the Westat design that encour-
ages respondents to keep receipts and record expenditures throughout the
quarter prior to a visit by the field representative. The Westat data reposi-
tory proposal was viewed as desirable in the future but less practical in the
nearer term.
