This section discusses the panel’s recommendations for redesigning the Residential Energy Consumption Survey (RECS). Although the RECS is methodologically a more straightforward survey than the Commercial Buildings Energy Consumption Survey (CBECS), the main goals and many of the survey’s main uses are comparable to those of the CBECS. Thus the priority areas identified by the panel are also the same as for the CBECS: (1) the timeliness and frequency of the survey and (2) data gaps. We will first discuss these two topics and the panel’s recommendations for addressing them, then provide recommendations for more incremental revisions for updating the RECS. Given that there is significant overlap between the recommendations for the two surveys, we will not go into the same level of detail here concerning the background of the survey or overall cost implications as we did in Chapter 5, where much of the information relevant to the RECS was first discussed in the context of the CBECS. However, those recommendations that are specific to the RECS and that appear for the first time in this chapter will be given a more thorough discussion.
TIMELINESS AND FREQUENCY OF THE RECS DATA
By law the RECS should be collected every three years, but a lack of funding has limited the survey to a quadrennial schedule since 1993. The delay between the time the data are collected and the time they are released to the public is a concern for the RECS, as it is for the CBECS, although
the RECS has not experienced the same extraordinary challenges in recent years as the CBECS has, and the U.S. Energy Information Administration (EIA) was able to release the data from the 2009 RECS much faster than in previous years, in part by releasing a preliminary public-use microdata file—with a limited number of variables—in October 2011.
Rotating Panel Design for the RECS
Just as with the CBECS, a rotating sample design would be a suitable option for addressing data users’ need for quicker and more frequent access to data from the RECS. Rotating design options were discussed in detail in Chapter 5. Briefly, the new approach would involve collecting the data over a four-year period instead of once every four years. This would enable annual data releases, with the new data released each year combining with data from the previous three years to add up to the same total sample size as is now provided by RECS once every four years. Ideally, a subset of the sample would be followed over time to create a longitudinal data set that would lead to an improved capability to analyze trends and changes. Specific options for a sample design are discussed in Chapter 5 in connection with the CBECS, and the approach most suitable for the RECS would have to be evaluated. An overall assessment of the costs and benefits of introducing a rotating sample design should receive priority as part of EIA’s near future plans for the RECS as well.
Recommendation RECS-1: EIA should evaluate the usefulness of implementing a rotating sample design for the RECS to improve the timeliness of the data.
Recommendation RECS-2: EIA should consider integrating a longitudinal element into the RECS sample design to obtain better estimates of change.
RECS Multimode Data Collection
As with the CBECS, the RECS should be gradually transitioned to a multimode data collection effort, in part to enable faster processing and release times. The panel acknowledges that many of the existing procedures will have to be modified and simplified in order to make it possible to have a more rapid turnaround. The current set of editing steps is one example
of a procedure that will need to be revised; this is discussed further in the next section. Despite the effort required to move to multimode collection, the panel believes the change is necessary because of the high and steadily increasing costs of data collections that are based exclusively on face-to-face interviewing and the particularly high costs associated with nonresponse follow-up in the RECS.
The transition to multimode data collection will be just as complex as with the CBECS, but with an additional complication: In the RECS, square footage is measured by field interviewers (although respondents are also asked), and EIA is especially concerned about shifting the collection of this data to respondent self-reports. The panel understands the importance of square footage data in statistical models, but it remains unconvinced that it would not be feasible to collect adequate information from the respondents. The CBECS already relies on self-reported data on square footage, and with thorough testing it should be possible to develop a series of questions that can produce reliable information on square footage through a web survey as well. As in the case of the CBECS, square footage data is one of the items that require the most editing in the RECS, so it is not clear that the presence of interviewers sufficiently mitigates the challenges related to obtaining this type of information as part of a survey. EIA will need to conduct methodological research to determine the best methods and likely trade-offs when collecting self-reported data on square footage.
In terms of operational considerations, sending an advance letter with log-in instructions to the sampled address may be a feasible first step in the data collection, especially given that EIA already sends a postcard and a letter to the address prior to the interviewer’s visit. In most cases, this letter would eventually have to be followed up with another data collection mode for nonrespondents, but the number of surveys completed over the web is expected to increase over time. The multimode approach will be discussed further in later sections of this chapter, in combination with discussions of the possibility of implementing an address-based sampling (ABS) approach for the RECS.
Recommendation RECS-3: Informed by a methodological research program, EIA should begin developing procedures for a multimode approach and should begin moving some of the RECS data collection to the web.
Revised Editing Procedures for the RECS
As with the CBECS, the RECS relies on a series of complex edits to increase the accuracy of the data collected, and these edits contribute to the length of time required to release the data. The most edited items in the RECS tend to be the information about square footage and the main source of space heating. Supplier data also require significant editing.
The most common edits built into the interview are signals that warn about inconsistencies in the reporting. Examples include the lack of a water heater, a heat pump reported as used for cooling, but not for heating, and a single-family detached house with fewer than 2 bedrooms or more than 10.
The types of edits that take place after the interview include
• Critical edits for items that cannot be wrong, missing, or machine-imputed.
• Reviewing “other, specify” responses for possible recoding.
• Reviewing selected interviewer comments for possible recoding.
• Identifying and resolving potentially out-of-scope cases.
The data collection contractor for the RECS typically performs most of the editing work. After the contractor completes the above editing steps, EIA performs additional editing, focusing on the following:
• Key items (for example, cooling not used in home, but more than zero rooms cooled).
• Problems with square footage data, such as:
o An unusually small single-family detached house (less than 800 square feet overall, less than 294 square feet per person, or less than 187 square feet per room).
o An unusually large single-family detached house (more than 6,045 square feet).
o The respondent-estimated square footage is more than twice or less than half of calculated square footage.
• Comparing household and rental agent responses.
EIA reported that it is also resource intensive to edit the supplier survey data. In 2009, in addition to errors that were in the data received from the suppliers, the data collection contractor introduced a number of additional
errors. These included (1) scanning errors; (2) keying errors, including errors interpreting the data received from the suppliers; (3) partial expenditure data; (4) a floating decimal point problem in the data entry program; and (5) dates entered incorrectly.
To address problems associated with missing data and to resolve some misreported items, EIA uses a hot deck imputation method, similar to the technique used by the CBECS. The items that most often need imputing include respondent-estimated square footage, measured square footage, the age of the home, and household income.
While the editing procedures developed by EIA undoubtedly enhance the quality of the data, revisions to the RECS editing procedures will have to sacrifice some of these rules and simplify the overall process, to facilitate faster processing of the data. As with the CBECS, emphasis should be placed on a shift to editing procedures that can be implemented in a self-administered setting, and an analysis should be conducted to evaluate whether there are editing rules that do not improve the data quality sufficiently to justify the resources invested and which could possibly be eliminated. In addition, it may be necessary to carry out a review of the processes in place to assure that the editing steps performed by the contractor are completed in a timely manner.
The possibility of releasing a subset of the data before the edits are completed for all of the variables should also be evaluated as an alternative to the above changes or as something done in combination with them. Given that many data users analyze only a subset of the data, releasing some of the variables before the full data set is edited could allow some researchers earlier access to all of the variables they need.
Recommendation RECS-4: EIA should investigate strategies for releasing the RECS data faster, for example, by revising the editing procedures altogether or by completing the editing for a subset of the variables and releasing these data prior to completing the editing for the full data set.
RECS DATA GAPS
The design of the RECS makes it most suitable for producing summary statistics for larger geographical areas (the entire country, census regions, and census divisions) and for selected states. Between 1993 and 2005, the
RECS released data for the four most-populated states; and in 2009, it released data for 16 states.
As is the case with the CBECS, the lack of granularity in the data available from the survey is one of the concerns raised most often by data uses, and, as is the case with the CBECS, which data can be released is largely dependent on the sample size. While the recent significant increase in the sample size (from 4,382 cases in 2005 to 12,083 in 2009) greatly improved EIA’s ability to provide data for subnational geographies, the need remains to have data available for each state in order to assess, for example, the effect of state-level programs on energy consumption. Insufficient sample sizes also hinder the ability of researchers to conduct multivariate analysis, making it difficult, for example, to evaluate changes resulting from the adoption of new technologies and energy efficiency measures.
Recommendation RECS-5: As part of its efforts to address the needs of data users, EIA should make it a priority to identify opportunities for increasing the sample size in order to enable the release of more of the RECS data that are currently being collected.
As discussed in Chapter 5, another way to make more of the data collected available to researchers is the use of a research data center (RDC).
Recommendation RECS-6: EIA should consider establishing a research data center or evaluate the option of using an existing RDC maintained by another organization to provide data users with secure access to RECS data that are currently not publicly released.
Like the CBECS, the RECS is in the difficult position of being a lengthy, high-burden survey whose design must take into account and prioritize a wide range of requests to collect data on additional topics and in more detail. The list below provides examples of the types of data that researchers would like to see included in the RECS. The list is by no means exhaustive, but it provides an illustration of the broad range of interests and possibilities for the survey.
• More information about building characteristics (for example, building orientation, wall insulation, length, width, height, floor-to-floor height, and location of ductwork).
• More information about roofs (for example, roof orientation, area, material composition, reflectivity, insulation).
• More information about windows (for example, number, window-to-wall ratio, orientation, overall heat transfer coefficient).
• More information about building systems characteristics (for example, location of equipment in an attic versus a basement or crawlspace, more detail on space heating equipment, and additional categories of water heating equipment).
• More detail on electric appliances and miscellaneous electric loads, such as data on saturation (devices per home) and duty cycles (or time spent in an active state), for a variety of individual technologies, including televisions, computers, set-top boxes, game consoles, monitors, audio-visual equipment, imaging equipment, and chargers for portable devices.
• Information about the efficiency ratings of end use equipment (for example, energy efficiency ratio, seasonal energy efficiency ratio, annual fuel utilization efficiency ratings).
• More detail about vehicle use and fuel use.
• Better data on multiunit buildings.
• Better measures of energy conservation behavior and indicators of the degree of success of energy efficiency programs.
• Thermostat set-point and set-back information based on direct observation of the thermostats instead of self-report.
• Energy consumption data metered in the field and connected to building system characteristics.
• Time-of-use energy consumption data.
• Monthly billing data instead of annual.
• Information about utility rate structures and incentives, along with marginal and average prices at the household level.
• Energy arrearages and bad debt related to the household’s energy suppliers.
• Better household income measures.
It is obviously impossible for the RECS to meet everyone’s needs completely, and, as is the case with the CBECS, EIA has to weigh the potential of a new measure to advance research and inform policy against the practical limitations and scope of the survey. If any new questions are added, every effort should be made to identify questions that can be dropped. Data needs that involve collecting additional technical details about buildings and
systems would probably impose an unrealistic burden on both interviewers and respondents, but they could possibly be accommodated as part of a special study involving energy auditors. Some types of information, such as rate structures, would perhaps be more efficient to collect from the energy utilities than from the households.
One topic area that the panel would like to see pursued as part of the RECS is the integration of smart meter data. Given that this topic is also relevant to the CBECS, a joint effort could be mounted to evaluate the consumption surveys’ potential use of this technology.
Recommendation RECS-7: EIA should prepare for the more widespread availability of smart meter data in the future by evaluating potential uses of such data, strategies for collecting them, and ways of addressing new confidentiality challenges.
Another topic that deserves careful consideration is the possibility of collecting household transportation data as part of the RECS. As mentioned previously, EIA’s consumption survey portfolio has not included a transportation survey since the 1990s, when the Residential Transportation Energy Consumption Survey was discontinued. In 2009, a small section of transportation-related questions was added to the RECS; the decision to do so was based in part on the argument that household vehicles are a part of household energy consumption. With the growing presence of electric vehicles, which are generally charged at home, this discussion becomes especially relevant. Given that electric vehicles could, by the next round of the RECS data collection, become an end use with a significant effect on residential energy consumption, the panel encourages EIA to be ready to collect more information about this topic.
Recommendation RECS-8: In anticipation of the spread of electric vehicles, EIA should prepare for the RECS to collect more information about these, especially more detail about the capacity to charge electric vehicles.
EIA should also develop a process for evaluating changes in end uses more generally in order to determine if the RECS needs to be updated with new questions and whether any of the existing questions can be dropped. As noted previously, miscellaneous end uses—and electronics use, in
particular—do not vary substantially by climate or geography, which implies that collecting meaningful data can be accomplished without having to increase the sample size.
Recommendation RECS-9: EIA should develop a process for the periodic review of new energy end uses or end uses that are becoming more widespread in the residential sector and which may need to be included on the RECS questionnaire. The process should also identify end uses that are becoming obsolete and that can be removed from the questionnaire.
In considering updates to the RECS questionnaire, EIA finds itself in a challenging position because of the already very high burden imposed on respondents by the RECS questionnaire. The panel acknowledges that it is easier to add new questions than to truly revise a questionnaire, but nevertheless it encourages EIA to consider alternative ways of accommodating new topics instead of increasing the burden of the RECS instrument on respondents. For example, EIA could play a leading role in the collection of data on specialized topics in the context of a variety of different arrangements that do not strictly speaking involve the RECS.
One topic that would lend itself to such a specialized study is the energy consumption of multiunit residential buildings. These types of buildings and their energy use are not covered adequately by the RECS, which gathers information only on individual apartments in multiunit buildings. A number of researchers would be interested in whole building energy consumption and additional details about these buildings, such as the number of floors above ground, the number of units, the percent of the building devoted to common space, whether there are central or in-unit laundry facilities, the configuration of utility meters, whether the tenant or owner pays for the utilities, and whether the rent paid is “affordable” or “market-rate.” Collecting this type of information would be particularly challenging in the case of buildings that are part residential and part commercial. EIA will have to decide if and how to collect the commercial component of the consumption in these types of buildings and perhaps develop a modeling approach to partition the residential and commercial components. Various organizations have expressed interest in collaborations to fill the data gaps on this topic, and EIA should consider taking on a more active role in addressing this need.
Recommendation RECS-10: EIA should consider implementing a “whole building” supplement to the RECS to address the data gap related to multiunit residential buildings.
A possible option, focused on accommodating questions that the RECS is ideally suited to collect, would be to redesign the data collection approach in a way that involves a longer and a shorter questionnaire. EIA would have to evaluate the optimal distribution of content between the two forms and the size of the sample for each, taking into consideration both costs and analysis needs. One possibility is to design a short form that collects only basic information from all buildings in the sample, and a long form that collects additional details, further customized by building type, from a subset of the sample.
Recommendation RECS-11: To accommodate data user needs for more detailed information, EIA should evaluate the possibility of administering a short-form and a long-form RECS questionnaire.
REVISIONS TO THE RECS SAMPLE DESIGN AND DATA COLLECTION PROCEDURES
In this section, we discuss additional potential changes to update the RECS, including revisions focused on increasing the efficiency of the data collection, and making the survey more useful to researchers and policy makers.
RECS Sample Design
The RECS is based on a stratified, multistage area probability sample of occupied housing units. This is a resource-intensive method, and the sampling frame requires updating every four years. As with the CBECS, EIA has tried a variety of updating methods over the years but difficulties remain.
In recent years, concerns about declining response rates of surveys in general and about the declining coverage rates of random-digit dial surveys due to the spread of cell-phone-only households in particular have led to significant interest in address-based sampling (ABS) methods. The use of the U.S. Postal Service (USPS) delivery sequence file (DSF) as a source for
an ABS frame appears to be a particularly promising approach for conducting household surveys of the general public (Link et al., 2008).1
Using the DSF to perform address-based sampling provides a very high coverage rate of residential households, and the lists can offer a variety of auxiliary information matched to the addresses, ranging from geocodes to landline telephone numbers and even Spanish-surname indicators for the head of household, all of which can help researchers enhance a survey design and develop a data collection plan (American Association for Public Opinion Research Cell Phone Task Force, 2010).
Researchers have also started evaluating the potential use of DSF and postal mail to collect data on the web (Messer and Dillman, 2011; Smyth et al., 2010). While relying solely on the web is far from realistic for a survey that aims to be representative of the U.S. population, studies have shown that web responses can be obtained by mailing initial requests via postal mail, which is then followed by another data collection mode (Messer and Dillman, 2011; Smyth et al., 2010).
EIA experimented with the use of the DSF for portions of the 2009 sample, and it should further evaluate the possibility of using ABS as the primary method for constructing the sampling frame for the RECS. This change should be specifically assessed in the context of the need to gradually transition some of the interviews to the web and the opportunities represented by a combination of these two methods. EIA should review the lessons learned from the 2007 CBECS and work with the new data collection contractor to better understand the cost implications of this approach to sampling.
Recommendation RECS-12: Research should be conducted to evaluate the possibility of replacing the RECS area probability sample with address-based sampling, using an address list developed by commercial vendors based on the U.S. Postal Service delivery sequence file.
1 The USPS licenses a variety of address products and services to companies that repackage these and then license them in turn for commercial and research applications. The services include access to a weekly snapshot of the USPS computerized delivery sequence (CDS) file, which contains all delivery point addresses serviced by the postal service, with the exception of general delivery. Although researchers do not have direct access to the USPS delivery sequence file, for the sake of simplicity we will refer to lists based on this file as DSF.
RECS Data Collection Procedures
Administrative records for buildings in the residential sector are becoming more readily available online. Although these records are kept in a large variety of formats, and it is not always possible to know how accurate they are, EIA should continue to assess the potential of such data as an alternative to an exclusively survey-based data collection that could help address various challenges related to interview costs and nonresponse. The use of alternatives to self-reported data is especially worth pursuing for square-footage data and for the development of intensity-of-use measures, in other words for measures of the energy consumed per units of interest, such as energy consumed per square foot.
Recommendation RECS-13: EIA should conduct an ongoing evaluation of administrative records as potential sources for substantive data for the RECS, especially for square footage data.
Data from Energy Suppliers
EIA should evaluate whether working more closely with energy suppliers to obtain RECS data would be feasible to increase data collection efficiency and reduce the burden on respondents in the RECS sample. Specifically, collecting energy consumption and cost data from energy suppliers prior to the housing unit interview should be explored. Establishing closer collaborations with utilities may also be necessary to begin developing procedures for obtaining smart meter data.
Recommendation RECS-14: EIA should evaluate whether working more closely with the energy suppliers could lead to procedures and to serve as a source of ideas efficiencies in the data collection process.
The chapter on CBECS describes a number of potential advantages of involving professional energy auditors in the data collection. A test to evaluate the use of auditors should also be conducted for the RECS because the relative advantages may be different in the case of a household population.
Recommendation RECS-15: EIA should test the use of professional energy auditors on a small scale to better understand the costs and benefits related to having experts collect data for a subset of the RECS sample.
Data Collection Instruments
The RECS questionnaire should be reviewed on a routine basis to assure that it is not becoming out of date. Although EIA has revised some of the RECS questions recently, it is important to develop a formalized process for a periodic review. This review should be separate in concept from a major redesign, and it should involve a variety of techniques to evaluate the questions, as appropriate.
Integrating cognitive interviews into the questionnaire-updating process may be particularly important to better understand how respondents relate to the questions and the answer options and to evaluate whether updates are needed. The cognitive interviews should also examine the strategies used by respondents to come up with answers to those questions that are suspected of being especially difficult to answer and of possibly producing less accurate data.
For example, the RECS questions about televisions in the home have recently been revised, but the question about the type of the television still lists “standard tube” as the first answer option on the show card. As tube televisions become less common, presumably fewer people will be familiar with this terminology. This, combined with the use of the word “standard” and the placement of the answer option first on the list could lead to over-reporting of these types of devices. The survey research literature indicates that respondents attribute more meaning to the range and ordering of answer options provided than researchers would prefer (Schwarz, 1996), and testing may reveal that a reordering of the answer options is needed.
Another example of a question in the RECS that may need updating is the one about wireless access in the home. This question seems to assume that a yes answer implies the presence of a wireless router. However, with the spread of smart phone devices, this question could have a different meaning for some of the respondents.
An important trend in energy consumption is the increase in the number of appliances, tools, and electronics that use energy in a typical home. The current RECS questions simply give respondents a few examples of re-
chargeable devices and ask for the number used in the home. This approach could become increasingly inadequate as the number of rechargeable devices increases, making it more difficult for respondents to perform a quick but comprehensive mental calculation. An alternative question format, such as a list of devices that prompts respondents to answer yes or no to each item and then provide the number applicable to that category, may have to be considered (along with an “other, specify” answer category).
The panel cannot recommend specific wording changes without further testing. The examples above are instead intended to illustrate the types of issues that should be considered. The panel also acknowledges that the continuity of the time series is an important consideration, but it believes that questions should still be evaluated with an eye on the future and considering which content is expected to be most relevant going forward.
As discussed, one important aspect of the questionnaire review process is identifying items that have become outdated or are no longer useful enough to warrant inclusion in the RECS. Removing some questions is imperative if new questions are to be added.
Recommendation RECS-16: EIA should invest in periodic reviews of the RECS questionnaire content and wording. This should be understood as a routine updating of the instruments, separate from the concept of a major redesign of the survey.
Greater EIA Involvement in the Data Collection Process
The panel recommends greater EIA involvement in the tasks that are performed by data collection contractors. As long as these highly technical data continue to be collected by interviewers, interviewer training deserves particular attention. EIA should also work with the data collection contractor to schedule periodic interviewer debriefings in order to obtain feedback about the fieldwork.
Recommendation RECS-17: Interviewer debriefings should become an integral part of the RECS data collection process in order to identify problems with the questionnaires and procedures and to serve as a source of ideas for increased procedures and to serve as a source of ideas efficiencies.