Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 201
Appendix A
Dissent and Panel Response
T
his dissent is intended to clarify some of the concerns of three of the
panel’s economist members. Enormous effort went into this panel
report and is reflected in its many good elements, which we whole-
heartedly endorse. For example, the report describes several prototypical
surveys. Exploring these survey types to their limit is the right approach to
determine the best form for a new survey, although we are concerned about
the extent to which resources will be available to BLS to pursue this course.
Despite the efforts that went into the report, some important issues did
not get the attention that we think they deserve. This result is not surpris-
ing given the short time frame and a mission for the panel that included
convening a data producers’ workshop and commissioning and examining
two prototype surveys from contractors, and a set of requirements that
is challenging indeed (Casey, 2010; Henderson et al., 2011). In addition,
some important information only became available to the panel late in the
process. The purpose of this dissent is to highlight several issues and to
stress information that the panel as a whole did not have the time to fully
discuss. In particular:
• The report emphasizes a greater role for diary-style data collection,
which we fear will lead to lower quality data. This emphasis is
supported by a statement in the report that the quality of existing
interview and diary survey data cannot be compared and other
statements that are equivocal about their relative merits. This state-
ment is inconsistent with recently estimated measures of bias and
precision, two standard indicators of data quality that generally
201
OCR for page 202
202 MEASURING WHAT WE SPEND
show the existing interview data to be of higher quality than the
existing diary data.
2010, six of the eight largest categories of expenditures avail-
In
able in both the Interview and Diary surveys show greater under-
reporting in the Diary survey (two other large categories of
expenditures are not collected in the Diary survey). The weighted
average of Diary totals to national income account totals for cat-
egories that line up with the national accounts is 17 percentage
points lower than the ratio for the Interview survey, though this
partly reflects the different coverage of the two surveys.
For the 37 categories that can be compared in the Interview and
Diary, the weighted average of the coefficient of variation is 58
percent higher in the Diary, with 33 of the 37 categories subject
to greater variation even after accounting for differences in sample
size (Bee, Meyer, and Sullivan, 2012). The higher variability of
responses from the Diary survey mean that 2.5 diary responses are
needed to equal the precision of one interview response, an issue
that is not reflected in any of the cost accounting in the report.
Response rate differences between the surveys and the frequency of
reports of no expenditures are also consistent with higher Interview
survey data quality. The recent experience of Canada is instruc-
tive: a switch from interview to diary led to 9–14 percent more
under-reporting on average (Dubreuil et al., 2011). This evidence
is mentioned in the report, but is not deemed to have important
implications for a redesign. Additional evidence of problems with
diary surveys can be found in Crossley and Winter (2012). While
these findings refer to existing surveys, they raise important con-
cerns about the collection of expenditure data that relies heavily
on diary methods.
The suggested redesigns, of course, aim to improve upon existing
methods, although there is not a great deal of evidence to support
that the effects will be dramatic. For example, monetary incentives
for respondents are emphasized, but the evidence on such incen-
tives from randomized trials suggests small effects that are even
smaller for high-income households (e.g., Clark and Mack, 2009).
The differential effects by income are particularly important since
one of the most significant problems with the CE is lower response
and under-reporting among high-income households (Sabelhaus
et al., 2012).
OCR for page 203
APPENDIX A 203
• We believe that the discussion of the prototypes in the report
should highlight the difficulty of using modeled data, data with
incongruent time periods, and short panels. The charge from the
BLS included the following requirement: “The CE considers the
ability to support micro-level analyses over a 12-month period to
be a requirement, with data collected at more than one point in
time.” The report correctly advises the BLS that setting priorities
is an important task, but we believe that the implications of the
report’s recommendations for this requirement are not adequately
addressed.
Modeling of microdata to construct data at the household level
is problematic at best, especially since one of the purposes of the
household data is to facilitate the design and estimation of behav-
ioral models. While both statistical and economic theory provide
a guide for modeling, even careful and well-intentioned methods
may yield results with dramatically misleading policy implications
such as the gross overstatement of the economic returns to obtain-
ing a Graduate Equivalence Degree (GED) when using imputed
observations in the Current Population Survey (CPS) (Bollinger
and Hirsch, 2006). Along with different reporting periods for dif-
ferent goods, the lack of an annual record at the household level
will prevent obtaining a complete picture of household spending.
panel that lasts only a few weeks or months is too short for most
A
changes in expenditures to occur. Even when expenditure changes
do occur, the relatively higher dispersion due to collecting data
over shorter intervals (e.g., two weeks) at each point in the panel
may result in the dispersion of the change in expenditure being
too unwieldy to be of use for standard panel data analysis. We
feel that the discussion of prototypes should have emphasized the
implications of the different designs for construction of datasets at
the household level since, in our view, some of the designs will not
produce usable data.
• More emphasis could have been given to the need to develop ways
to monitor the quality of the data. Given that concerns about data
quality are motivating the redesign, having a way to better monitor
and hopefully improve data quality is vital. For instance, the ability
to link to outside sources would be enhanced by a change in lan-
guage in the introductory letter to respondents getting preapproval
for such links such as is currently used in the American Commu-
nity Survey (ACS), Survey of Income and Program Participation
OCR for page 204
204 MEASURING WHAT WE SPEND
(SIPP), and the CPS. These links could be used to compare survey
responses to administrative or other data. A second approach—the
development of an intensive “gold standard” survey that could be
linked to a subset of the sample—is developed in one of the proto-
types. A third approach is using internal data check, such as com-
parisons of spending to income that would be enhanced by having
spending and income for the same time period and emphasizing
budget balance. Research done with the data is also an important
check on the extent to which the data have sensible patterns. Some
of these issues are mentioned in the report, but we believe the goals
and methods of monitoring data quality merit recommendations by
the panel.
• A redesigned CE could also be more informed by how survey
information affects the accuracy of the CPI. Information on what
categories of expenditures and expenditures by which household
members appear to be under-reported could be reflected more in
the designs. Ideally, the designs would reflect which categories of
expenditures for which extra detail in spending would be especially
helpful in reducing bias in the CPI and which other categories
have price changes sufficiently correlated across different goods
to indicate that current detail is more than is needed. If the struc-
ture of the CPI as a plutocratic price index is going to continue,
then under-reporting by high-income households needs to be better
addressed.
• Finally, the report would benefit from further exploration of eco-
nomic census data and other outside sources of data to obtain
detailed expenditures. Some of these sources employ crosschecks
that validate the data and make it more likely to be close to the
truth. These sources would be especially useful for the calculation
of CPI weights and aggregate expenditure tables.
We reiterate that these comments are not meant to criticize the work
of the panel, but to emphasize that it is unfinished. The CE survey is an
important statistical program, and all of the panel members have their own
views on what the priorities of the survey should be and how best to ad-
dress these priorities. The diversity in views on this panel has been one of
its greatest strengths.
Robert Gillingham
Bruce D. Meyer
Melvin Stephens
OCR for page 205
APPENDIX A 205
REFERENCES
Bee, A., B.D. Meyer, and J.X. Sullivan. 2012. The Validity of Consumption Data: Are the
Consumer Expenditure Interview and Diary Surveys Informative? NBER Working Paper
No. 18308. Cambridge, MA: National Bureau of Economic Research. Available: http://
www.nber.org/papers/w18308.pdf.
Bollinger, C.R., and B.T. Hirsch. 2006. Match bias from earnings imputation in the Cur-
rent Population Survey: The case of imperfect matching. Journal of Labor Economics
24(3):483–519.
Casey, W. 2010. CPI Requirements of CE. Available: http://www.bls.gov/cex/ovrvwcpi
requirement.pdf.
Clark, S.M., and S.P. Mack. 2009. SIPP 2008 Incentive Analysis. Washington, DC: U.S. Cen-
sus Bureau, Demographic Statistical Methods Division.
Crossley, T.F., and J.K. Winter. 2012. Asking Households about Expenditures: What Have
We Learned? Paper prepared for the Conference on Improving the Measurement of
Consumer Expenditures sponsored by Conference on Research in Income and Wealth
and the National Bureau of Economic Research. Available: http://www.nber.org/chapters/
c12666.pdf?new_window=1.
Dubreuil, G., J. Tremblay, J. Lynch, and M. Lemire. 2011. Redesign of the Canadian Survey
of Household Spending. Presentation at the Household Survey Producers Workshop,
June 1–2, Committee on National Statistics, National Research Council, Washington,
DC. Available: http://www.bls.gov/cex/hhsrvywrkshp_dubreuil.pdf.
Henderson, S., B. Passero, J. Rogers, J. Ryan, and A. Safir. 2011. Consumer Expenditure
Survey (CE) Data Requirements. Bureau of Labor Statistics, U.S. Department of Labor.
Available: http://www.bls.gov/cex/cedatarequirements.pdf.
Sabelhaus, J., D. Johnson, S. Ash, D. Swanson, T. Garner, J. Greenlees, and S. Henderson.
2012. Is the Consumer Expenditure Survey Representative by Income. Paper prepared for
the Conference on Improving the Measurement of Consumer Expenditures, sponsored by
the Conference on Research in Income and Wealth and the National Bureau of Economic
Research, Washington, DC, December. Available: http://www.federalreserve.gov/pubs/
feds/2012/201236/201236pap.pdf.
PANEL’S RESPONSE TO DISSENT
The uses of the Consumer Expenditure Surveys (CE) are complex and
diverse. Our panel reflected that diversity and complexity. The members
came together from different academic disciplines and collectively provided
high levels of expertise in economic analysis, survey measurement, data
collection, sample design, and technology. All of these skills were critical
to the panel’s work, and each area of expertise reflected insight, ideas, and
opinions important to designing high-quality consumer expenditure surveys
and to preparing a consensus report.
During panel discussions, three members continually emphasized the
importance of the CE for economic analysis based on longitudinal micro-
data. Their ideas are important and, indeed, are part of the whole that is
the panel’s report. These members contributed substantially to both the
substance of the report and to the balance and tone in which ideas have
been laid out. The report is better because of the juxtaposition of their in-
OCR for page 206
206 MEASURING WHAT WE SPEND
sights and ideas with those of other panel members, and we sincerely thank
them for their efforts on this report. We regret that they have now decided
to emphasize their personal perspective through a dissent.
A prominent theme of the dissent is the relative quality of the current
CE Interview survey in comparison with the Diary survey. The two surveys
were designed differently in order for each to collect certain types of ex-
penditures. The panel demonstrates in this report that both surveys suffer
from substantial data quality issues. BLS currently uses results from both
surveys, publishing estimates from one or the other of them after consider-
ing the overall quality for each individual expense item.
The majority of the panel thinks that the three dissenting members, by
implying that diary-style data collection invariably produces lower-quality
data than recall-style data collection, have reached a premature conclu-
sion. The majority concluded that it is neither possible nor necessary to
assess which of these two current surveys would be “better” for collecting
all types of expenditures. Instead, the panel kept its focus on how to use a
combination of improved modes, along with newer collection technology,
external data, and respondent incentives, to create a more effective CE
survey in the future. We believe the report has done that well. It provides
several design options, including a supported journal using a tablet com-
puter that is distinctly different from the current Diary survey. Each option
has a different mix of interview-like and supported journal components, all
of which should receive serious consideration and evaluation.
Some of the other issues in the dissent—modeling, the length of refer-
ence periods for data collection, and linking to administrative data for a
richer dataset—relate directly to the dissenters’ focus on creating the most
useable microdata for analysis. Each of these issues is discussed in the
report, both in a broad context and in relationship to creating microdata.
The report proposes a number of ideas to improve the understanding
of expenditure data and their quality, including the evaluation of intensive
methods to reconcile household income and expenditures, the use of extant
administrative data for quality assessment, research on how households
keep financial records, and the possibility to augment the sample through
more intensive interviews with a subsample of wealthier households. All are
discussed in some depth in the report.
The need for an incentive program is thoroughly developed in the
panel’s report. Several panel members have considerable personal experi-
ence in using and evaluating incentive programs. Knowledge of the exten-
sive scientific literature led to our recommendation that fairly substantial
incentives be used in order to affect response behavior. The report has also
thoroughly developed issues related to the use of outside sources of data
to obtain detailed expenditures. The panel encourages BLS to continue to
OCR for page 207
APPENDIX A 207
research and evaluate these options, but we concluded that none of them
is currently a feasible alternative given the associated risks and costs. Far
from being incomplete, we believe our report, with its broad discussions,
will be an important tool for the administrators and policy makers who are
responsible for determining the next steps for the CE.
