Click for next page ( 87


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 86
Conclusions and Recommendations The previous chapters examine several factors that may be important influences on energy demand but that have not been addressed extensively in existing formal models. Some of these, such as consumer mistrust of information, informal social influence, and the marketing of financial incentive programs, are rarely highlighted by modeling efforts and may be difficult to assimilate into existing models. Such factors, it might be said, are blind spots of existing models. Other factors, such as qualitative distinctions among types of financial incentives, are not emphasized by the theoretical frameworks on which most models are based, but are likely to emerge in modeling efforts and could be assimilated into models without very much difficulty. Still other factors, such as the dis- tinction between appliance list prices and transaction prices and the effects of price changes as distinct from price levels, are significant in economic theory and could be incorporated readily into existing models, even though they have not been in the past. Our analysis shows, in sum, that existing demand models describe the behavioral environment of energy demand only incompletely. Some of the gaps can be filled if models are built from more com- plete and detailed data, but modeling efforts are likely to overlook systematically some important features of the environment of energy use. Because some important gaps in demand analysis seem unlikely to be filled through modeling efforts, we judge it unrealistic to try to build a single comprehensive analytic framework to answer policy makers' and research- ers' many questions about energy demand. AS an alterna- tive to seeking such a framework, learning should proceed by developing a portfolio of analytic approaches--some general, others focused on particular policy questions. 86

OCR for page 86
87 With diverse methods of analysis, hypotheses will be gen- erated and tested that any single method might overlook. In addition, one method can sometimes provide a valuable check on the results of another. This chapter presents the panel's conclusions and recommendations regarding the use of formal models and problem-oriented research in energy demand analysis and regarding the needs for data to inform that analysis. We also discuss an approach to using different analytical approaches in concert in order to improve the quality of energy demand analysis. THE ROLE OF FORMAL MODELS Policy analysts sometimes express two erroneous opinions concerning formal energy models: one is that a policy question can be answered simply because it is represented in a model; the other is the obverse--that a question cannot be answered because no model exists to answer it. Both opinions equate energy policy analysis with formal modeling. They reflect an overreliance on formal models that is not justified by the validity of existing models and that is not necessary given the availability of other analytic techniques. Formal models do have considerable appeal as a means of energy policy analysis. They are broad, multipurpose tools that can address a wide range of policy questions and call attention to unanticipated effects of policies on other parts of the energy or economic system. They can give the sort of quantitative responses decision makers want to their questions, and they can often do this quickly. And when correctly formulated, models can pro- vide necessary checks of consistency with physical and economic constraints that might otherwise be overlooked in a policy analysis. Compared with methods that involve gathering new data, models can save both money and time. They can also evolve, along with the questions that face policy makers. From a policy maker's viewpoint, models are familiar tools once they have been used, so it is easy to continue to rely on them, sometimes even when they are outdated. In addition, the apparent objectivity of com- puterized analysis is impressive to some decision makers. But models have many limitations. As the previous chapters have demonstrated, there is no behavioral knowl- edge to support assumptions about the values of parameters and about the functional forms of the equations used to represent behavioral relationships. Many models do not

OCR for page 86
88 treat their internal uncertainty with sufficient explicit- ness. And important variables are often excluded from models, sometimes for lack of data and sometimes because modelers' conceptual frameworks do not include them. Another limitation of models that are used for policy analysis and forecasting is that they are usually vali- dated only by matching them to past events, often with numerous post hoc adjustments. This procedure does not justify confidence in a model's ability to predict the future: newer versions of a model may be as likely to need readjustment as the older ones. The documentation, validation, and maintenance of models have generally been given insufficient attention. A model must first be documented: complete records must be made of the model's contents, its assumptions, and the sources of the parameter estimates and the chosen func- tional forms of its equations. Only with full documenta- tion can a model's behavioral assumptions be identified for testing. Validation is also essential, and not only when a model is built. In our view, it is important to validate models by testing them regularly against empiri- cal data. This means, for instance, making before-the- fact predictions with a model and comparing them with actual outcomes or comparing a model's results with the results of problem-oriented studies. Such empirical testing may be the only way to build credibility for models, in light of the fact that many models undergo almost continuous revision in their structure, input, and output. This ongoing validation, combined with a commit- ment to updating the documentation of a model as it evolves, constitutes maintenance--a much neglected part of modeling. When an organization buys expensive capital equipment it usually commits itself to a budget for main- tenance. But energy models, which are not as reliable as most equipment, are often expected to work well without maintenance. The size and complexity of some energy models makes documentation, validation, and maintenance particularly . This shortcoming of models is due not to the nature of modeling but to the frequent practice of inferring regu- larities in human behavior from the evidence of past cor relations. Other research methods, particularly survey methods and exploratory data analysis, have the same shortcoming when their findings are used uncritically to make projections. -

OCR for page 86
89 difficult. Although larger models are being better able to include feedbacks among parts of the energy system, the burden of documentation and validation increases. some- times geometrically, with the number of relet inch ins i, ~ -~^ Are represented. Models that include large numbers of param- eters compared to the volume of data shiv -=nl ~ i n are particularly suspect. : _ _ ~ . ~ ~ ~ . . ~ _em,! ~~,~ ~ ~ 81 ~~ ~ Also suspect are large models, ~nc~ua~ng many of the system dynamics variety, whose results are sensitive to the effects of variables whose values and relationships are merely postulated. Given the state of the art, we conclude that more knowledge can be gained by improving the quality of models than by -increas- ing their size. With constant resources, modeling can be improved, on the whole, by sacrificing some comprehensive- ness in order to gain quality. This ongoing validation, combined with a commitment to updating the documentation of a model as it evolves, constitutes maintenance--a much neglected part of modeling. In addition to the above substantive problems, the process of funding for models gives cause for skepticism. When quick answers are in greater demand than documenta- tion and validation, model builders are under pressure to sacrifice quality control. ~ ~ ~ ~ . . Poorly validated models can o~ expeccea co De usea more often and better--and there- fore more expensive-- models will fail to command the support their higher quality deserves. As a result of all the above factors, when any existing energy demand model Gives an answer h~ ~ an, its? - fact ;~" ~ ~ ~ ~ ~ ^ ~~ ~ ~ ~ ^y ~,L4~= ~ EVER t gnat answer is to a large extent taken on faith. Despite these limitations, models remain popular with policy analysts--so popular that they are sometimes overused or misused. Sometimes models are used to answer factual questions that could be answered almost as easily and much more accurately by other methods. When an oil short- age threatens, for example, it makes more sense to find out how much consumers are adding to their inventories by surveying a sample of consumers than by estimating behav- ior from a model. Sometimes models are used to answer policy questions that they are not equipped to address. For instance, most models have difficulty representing efforts to improve information. To estimate the effect of energy-efficiency labels for appliances, a modeler might postulate an effect of the labels on consumers' discount rates and use a model to estimate that effect on purchase behavior. It would make more sense to conduct a field experiment that actually tested the effects of labels.

OCR for page 86
Do Models may be quick and inexpensive relative to alter- native research methods, but there is no such thing as good, cheap energy policy analysis. If policy analysts are to offer knowledge rather than mere answers, the empirical basis of their analyses must be strengthened. We believe this can be done by making some changes in the way models are developed and by drawing more on knowledge gained by other methodologies. We offer seven conclusions and recommendations about formal energy models and their use. 1. Policy makers should maintain a healthy skepticism about the outputs of formal energy demand models. We do not assert that judgment is necessarily better than exist- ing models. Rather, the point is that models, like judg- ments, should not be accepted without corroborating empirical evidence. The support of a second model is much less convincing evidence than the support of a field experiment, a good evaluation study, or even a well- conducted survey. 2. The current system dynamics models in use at the U.S. Department of Energy should not be relied upon as heavily as they are for forecasts of energy demand. Fore- casts from those models are too dependent on postulated relationships and on judgmental elements incorporated in them to make them consistent with expectations. 3. Resources allocated to modeling should be shifted to ensure adequate documentation, validation, and main- tenance. 4. Within the modeling community, more attention should be paid to building models that are better tested and maintained. These efforts are necessary to make demand models more credible. 5. For testing purposes, some versions of some models should be "frozen, archived, and then used from time to time without judgmental readjustments to make forecasts and policy analyses that are then tested against new data and against the findings of studies that use other research methods. This step should be considered an - essential part of the validation process, and is concep- tually separate from the normal process of using new data to revise and update models. With this step, the modeling

OCR for page 86
91 community can build a track record on the basis of which formal models can be judged. 6. Innovation in modeling should be directed toward decreasing the dependence of model outputs on assumptions about parameters and the functional forms of equations. 7. Sensitivity testing of models should be used to generate hypotheses for empirical research, and resources used for validating models should be devoted in part to carrying out this research. When a model's output is highly sensitive to a parameter whose value is not well established, empirical research should be done to estab- lish the value. As these conclusions and recommendations make clear, we believe much more emphasis should be given to building the empirical base for energy demand analysis than to further elaboration of formal models based on inadequate data. Other research methods are required to build this empirical base. THE ROLE OF PROBLEM-ORIENTED RESEARCH Five types of problem-oriented research are surveys, analyses of existing data, natural experiments, controlled experiments, and evaluation research. Surveys National general-purpose surveys can provide invaluable data for problem-oriented research. Because their primary role in energy demand analysis has been to gather the multivariate time-series data essential for much policy analysis, including formal demand modeling, our conclu- sions and recommendations for these surveys appear in the next section on data collection. Specialized surveys have been responsible for most of the detailed analytical work on the effects of consumer knowledge, attitudes, and beliefs on energy use (e.g., Kempton, Harris, Keith, and Weihl, 1982; Stern, Black, and Elworth, 1982b). Specialized surveys are especially useful for explaining phenomena that appear inconsistent in terms of the variables represented in models. For example, surveys can be used to understand why conserva-

OCR for page 86
92 tion programs that offer the same financial incentives vary so widely in their levels of consumer acceptance (see Chapter 3; Berry, 1982). Surveys that explore public responses to the marketing and implementation of conser- vation programs have shown that consumer protection and convenience are two nonfinancial variables that affect consumer response to financial incentives (e.g., Stern, Black, and Elworth, 1981). But as we have mentioned, surveys also have l~mita- tions. One is the unreliability of self-reports of some variables, such as attitudes. There is evidence both of reliability and unreliability in responses to energy sur- veys (e.g., Beck, Doctors, and Hammond, 1980; Geller, 1981). Two reasonable but unproven hypotheses are that self-reports of major investments are more reliable than self-reports of changes in habits and that reports of past action are more reliable than reports of future action. There is reason to question the worth of self-reports about planned energy-saving actions. Although reported intentions to act are often good predictors of behavior (Ajzen and Fishbein, 1977), the relationship depends, among other things, on the absence of constraints on action. For expensive investments in energy efficiency that involve many steps before completion, behavioral intentions would seem a questionable predictor. We have also noted that inferences drawn from even the most accurate self-reports may not be accurate because of errors in analysts' assumptions relating energy-saving actions to subsequent energy use. And when a policy innovation is being considered, people's predictions of how they will respond to hypothetical situations are not as good a source of information as actual observation. For such situations, small-scale experiments and program evaluation studies can give more useful information, even if their generalizability is unknown. The conjunction of several small-scale behavioral studies can give more con- fidence in the conclusions about a new policy than the best-designed national survey of people's intentions. Analysis of Existing Data Analysis of existing data can still improve understanding of energy demand. Many interesting data sets collected by government agencies go unanalyzed, in whole or in part. For example, the data from the Residential Energy Con- sumption Survey (RECS) have been only partly analyzed.

OCR for page 86
93 Inadequate information about individual respondents is cited by some researchers as one reason for the lack of detailed analysis, but lack of funding for data analysis is a more serious deterrent. More analysis could also be done on utility companies' data on residential and com- mercial energy use to give a more solid empirical basis to studies of energy demand. Limited funding for research and the narrow focus given to research questions have limited what could be learned; noncomparability of data across utility companies has made analysis difficult; and access to the data has been a major problem. Concerns about customer privacy and about possible use of data in adversary proceedings make many utility companies unwill- ing to give researchers access to their files. Regulatory agencies have sometimes forced the release of data when they believe release will serve a public purpose, but as a rule, utility data are not readily available to researchers. Existing data can be studied in various ways to learn about energy demand and to generate hypotheses. Several techniques of exploratory data analysis (Breiman, Fried- man, Olshen, and Stone, 1984; Donoho, Huber, and Thoma, 1981; Fisherkeller, Friedman, and Tukey, 1974; Friedman and Tukey, 1974; Huber, 1981) can be used to examine data sets for regularities and to generate hypotheses to be tested on future data or with additional research. These methods rely heavily on informal graphic techniques and Reemphasize formal statistical models or tests of hypotheses. 8. We recommend that some of the resources devoted to energy demand analysis be redirected toward exploratory analysis of existing data. Disaggregated data should be systematically collected on energy use in the commercial and industrial sectors of the economy and on energy prices and equipment stocks. Specialized surveys relating measured energy use and observed investments in energy efficiency to demographic, institutional, and attitudinal factors are also much needed. Natural Experiments Natural experiments can produce a wealth of data that should be analyzed systematically. Ongoing data collec-

OCR for page 86
94 tion efforts would make this more possible. Analysis of the records of utility companies would allow additional studies. Natural experiments would teach more if a capability were developed to move researchers quickly into the field to study natural experiments in energy demand. Surveys of consumer response to changing utility rates or the recent decrease in inflation rates would have been a way to learn from one class of natural experiments (see Chapter 2); studies of initial responses to the threat of an oil supply cutoff could be a vehicle for learning from another class of natural experiments. 9. We recommend that some of the resources available for energy demand analysis be made available on short notice for field studies of natural experiments that occur when there are rapid changes in the energy environment. Controlled Experiments Controlled experiments are an especially valuable tool for assessing the effects of interventions that are non- financial in character and for which existing models are particularly inadequate. For example, psychologists have conducted many field experiments on the effects of energy- use feedback (reviewed by Geller, Winett, and Everett, 1982) and smaller numbers of field experiments to assess the effect of nonfinancial factors such as personal com- mitment (e.g., Pallak, Cook, and Sullivan, 1980), self- monitoring of energy use (e.g., Becker, 1978), and the presentation of energy conservation as a way to save money versus as a way to avoid losing money (Yates, 1982). Financial incentive programs are also appropriate subjects for field experiments (see Chapter 3), both because they have important nonfinancial features and because consumer responses to the incentives themselves are not well understood. Experimental techniques offer great benefits for policy analysis of conservation programs: controlled field experimentation should be the method of choice for evalu- ating promising innovations in the implementation of such programs. Conservation programs are complex and contain important elements of promotion and implementation that cannot easily be expressed or analyzed in models. For example, results from Residential Conservation Service (RCS) programs have varied greatly across the utilities that run them, leading to controversy about whether the

OCR for page 86
95 national RCS program is worthwhile, and conflicting judg- ments have been offered to policy makers on the basis of very weak evidence. But many of the likely sources of variation could easily be the subject of experimentation at low cost. A utility company could randomly assign some of its customers to receive telephone marketing efforts or to be contacted as a follow-up to energy audits, or to receive lower-cost audit procedures as controlled alter- natives to the procedures the utilities now use. Despite the fact that strong inferences could be drawn from such experiments, conservation programs are almost universally designed and implemented without the controls necessary for identifying low-cost means to improve their chances of success. 10. We recommend that controlled field experimentation be used whenever possible to evaluate promising innova- tions in Policy affecting energy demand. As we have mentioned, laboratory experiments also are appropriate analytic tools in some circumstances. They are particularly useful in efforts to design energy information so that consumers will notice and understand it (see Chapter 4). It is often feasible to experiment in a laboratory setting with alternative choices about what information to include, what metrics to use to sum- marize information, and how to design appliance labels, automobile fuel economy guides, utility bill inserts, and so forth. The laboratory approach is much cheaper than field experimentation and can be used to screen out alternatives that would almost certainly fail in field trials. Evaluation Research Evaluation research can, at least in principle, allow analysts to learn from what may be the greatest untapped source on information about energy demand--the thousands of energy programs and policies that have been tried dur- ing the last decade. The knowledge that could be gained has great practical value because the success or failure of a conservation program is probably due to more than the sum of the specified features it offers; thus, it is not enough to build a program from single features that have proved effective--even in well-controlled experiments. The experience gained in past programs and policies, if

OCR for page 86
96 it can be interpreted, can help identify and possibly harness forces that may be more important than many of those usually considered in formal energy analyses. The best example is the fact that consumer use of incentives for conservation can vary by two orders of magnitude among programs offering the same financial incentive (see Chap- ter 3). This finding presents a riddle for analysts if they define the programs simply in terms of the financial value of the incentives they offer. The riddle can prob- ably be solved only by carefully examining the ways the different programs are implemented. Such process evalua- tions, which emphasize qualitative research methods based on close observation and interviews of program staff and clients, can offer the needed insight. Outcome evalua- tions, which can use many of the research methods dis- cussed in this section, can offer quantitative estimates of program effects. Careful comparisons of outcome studies can also provide estimates of how much difference process factors make. Although much can be learned from thorough process and outcome evaluation of the experiences of energy programs, we wish to reemphasize that the most reliable information comes from explicitly treating programs and policies as experiments from their beginning. Such an approach requires the creation of a suitable comparison group, randomly assigned if possible, and careful measurement of effects in all groups (fuller accounts of issues in eval- uation research design can be found in texts such as Cook and Campbell, 1979). Experimental research methods do not imply, we repeat, rigid constriction of a program's oper- ation for the sake of some notion of scientific rigor. When controlled experiments are not feasible, some quasi- experimental research designs retain many of the advan- tages of controlled experiments. Whatever the type of research design, however, more can be learned from the experience of a program if an evaluation plan is developed as a program is developed; an evaluation plan tacked on after a program has been-operated inevitably produces weaker research because of the inability to measure pre- program conditions and because important questions must be answered from memory or by reference to incomplete archives rather than by observation. 11. Resources devoted to energy demand analysis should be shifted to favor collection and analysis of empirical

OCR for page 86
97 data over further elaboration of models that are poorly supported empirically. 12. Additional efforts should be made to identify and quantify important variables that are now omitted from formal energy models. The most obvious example is the set of marketing and implementation variables that appear to dwarf the effect of financial incentives in energy con- servation incentive programs. Evaluation research appears to be the best method for identifying the relevant vari- ables; evaluation research or field experimentation might be useful for estimating their size. 13. Additional analytic effort should be made to incorporate key nonfinancial variables into the process - of demand analysis. The effects of marketing and manage- ment in energy information programs or of consumer mis- trust may be interpreted as changes in discount rate, changes in lag coefficient, or in other ways. It will prove valuable, however, not only to quantify the impor- tant nonfinancial factors in energy demand but to improve their conceptualization. 14. The federal government should establish a fund . . . for basic research on decision making relevant to energy - efficiency, with grant awards recommended by an outside peer review Panel. Such research should include studies of nonfinancial influences on energy demand and studies with only indirect implications for existing government- supported energy programs. THE ROLE OF DATA COLLECTION In efforts to model energy demand, data on energy use and on factors that influence it have too often been imputed rather than measured. Energy use is often calculated from data on production, stocks, and imports and then allocated to end uses, sectors of the economy, and geographic regions. Data on energy use by energy-efficient technol- ogy are often estimated from engineering models rather than measured in actual operation. And the nature of consumers' and manufacturers' decisions, program imple- mentation, and other social processes is most often assumed (or ignored). Insufficient knowledge exists to justify relying on imputations or presumptions rather than measured data. Prudence dictates building some national

OCR for page 86
98 estimates from disaggregate measurements and surveys, more direct methods that can act as a check on procedures of imputation. It makes sense for such measurement efforts to emphasize major energy uses (e.g., gasoline for auto- mobiles); politically sensitive uses (e.g., home heating, which especially concerns low-income consumers and their advocates); uses for which major fuel switching is pos- sible (e.g., industrial process heat); and uses about which little is known, such as energy use in commercial and public buildings. The best current example of national data collection on energy demand is the Residential Energy Consumption Survey (RECS) of the Energy Information Administration (EIA), a detailed longitudinal survey of a rotating panel of households that has been a particularly important source of knowledge for demand analysts. Careful thought has gone into the construction of the RECS questionnaires, which have served as a model for some other surveys and could be used more in research by state and local govern- ments and by utility companies. For several reasons, however, national surveys have not achieved their potential. For example, the initial plan for EIA to survey energy use in nonresidential sectors of the economy has not been followed. Understanding energy demand in the industrial and commercial sectors--the bulk of national energy demand--is obviously critical for national demand analysis, yet the EIA survey of industrial energy use was abruptly discontinued in 1981, and a planned new survey has not yet appeared. The survey of nonresidential buildings has been a sporadic effort and deserves more support. And EIA'S data on transportation are restricted to the residential sector. These weak- nesses in EIA'S surveys should be corrected. 15. Serious and continuing support should be given to EIA surveys that address all major energy-using sectors of the economy, that use a panel design, and that are conducted by experienced and competent data collection . ~ organlzatlons. 16. The industrial energy-use survey of the Energy Information Administration should be reinstated to gain . . essential data on a major segment of national energy demand. Technical problems have made it difficult for some researchers seeking to use the RECS public data base.

OCR for page 86
99 Data tapes are not available for up to 2 years after the data are collected.2 More important, details at the individual level, which analysts often need for micro- analysis, are not available from RECS because of concerns about privacy, disclosure, and informed consent. In par- ticular, these concerns have resulted in limiting infor- mation available about the specific location of respon- dents' homes. Without this information, however, researchers cannot take advantage of information available from other sources on such factors as prevailing wind speed and direction, differences in utility rate struc- tures, the exposure of households to local or state con- servation programs, or local consumer price indices. Information at the level of three digits of a zip code would allow analysts to assess the effects of local vari- ables more adequately than they now can. The privacy problem might be solved by requesting respondents to release more detailed information to investigators, by relying on smaller surveys in which participants volunteer to release the information needed to answer particular questions, or by allowing the data collection organization to merge a researcher's data set with the RECS data for a subscription fee. ETA has occasionally merged data sets or done additional data analyses on the request of and with funding from other federal agencies.3 17. The Department of Energy should, wherever feasible, cooperate with other federal agencies and the private sector in data collection. RECS has also failed to include enough detail to be useful to certain specialized groups of researchers. For example, it has not assessed the importance of energy efficiency and other factors in appliance purchases. It has also done little to assess motivational and social- psychological factors in energy demand. Of course, there are limits to how much a survey can include, and some 2 The delay is due at least in part to the operational difficulty of collecting and checking data from disparate sources. For example, RECS must collect data from house- holds and subsequently from energy suppliers. It can take six months or more simply to collect energy use data from fuel oil dealers. 3 Information from L. Carlson, Energy Information Administration.

OCR for page 86
100 potentially important questions will always be left out We do not offer proposals to restructure the RECS survey, but we do believe it should be improved. And all of EIA'S surveys should be designed to obtain the best and most useful data for research and policy analysis. 18. A formal advisory board of energy demand researchers should advise the Energy Information Adminis- tration on the contents of its surveys. 19. ContinuiEv is a blah or for iEv in data collection Surveys of energy consumers should repeat items over time and use a panel or rotating panel design. We also wish to emphasize the occasional need to gather representative national data on energy issues on short notice or at relatively little expense. For example, EIA conducted a survey in fall 1979 of the oil-heated house- holds in its national sample to see if people were having trouble obtaining heating oil in the wake of the oil shortage of that year (Energy Information Administration, 1979). The existence of a well-chosen representative sample for which baseline data were available made it possible to conduct a survey on short notice from which meaningful conclusions could be drawn, and we believe such samples should be maintained. 20. A large national panel for which past data exist, such as the RECS respondents, should be made available for subsampling so quick telephone surveys can be used to help answer immediate policy questions. Such a subsample might be made available to independent researchers who could insert questions on a subscription basis.4 4We have not addressed legal questions that may arise from selling subscription access to respondents to a federally sponsored survey, particularly to profit-making organizations. The point is not that the RECS survey should necessarily be the vehicle for collecting the data, but that some preexisting national survey would be valuable as background for more focused survey efforts by public or private organizations. In the residential sector, RECS is the best such survey in existence.

OCR for page 86
101 US ING VARIOUS RESEARCH METHODS IN CONCERT Energy policy analysis and related data collection tend to be closely responsive to policy questions: current policy issues drive the development of models, and the requirements of models determine data collection efforts. Immediate policy questions and formal modeling tend to dominate the research enterprise to the neglect of other methods and of more basic research. The demand for answers to today's questions today has diverted resources from the more basic task: building a knowledge base for answering policy questions more accurately. Instead, emphasis has been given to elaborating formal models even when their assumptions are poorly tested, the necessary data are lacking, and important variables are not included in them.5 We have noted the important place of formal models in energy analysis, and we believe that because of their great value for forecasting and for identifying effects of policies on disparate parts of the energy system, improving their behavioral foundation is a high priority. Models can also be useful for more narrowly focused policy analyses, though they have been overused in relation to other methods. In this context, models are most appro- priate for anticipating effects of interventions that are quantitative and that operate by processes that are well understood or that have been successfully modeled in past similar situations. In the more typical case, however, when the path of implementation is less straightforward (e.g., energy conservation tax credits, regulations, informational efforts), existing models are less useful. They have even less value for analyzing policies that are qualitative in nature, or that obviously involve institu- tional, organizational, or psychological elements (e.g., residential conservation programs). For such analyses, sModels are most often constructed by engineers, operations researchers, and economists, with little consultation with researchers in other disciplines. This lack of breadth is one reason there has been so little effort to model such variables as incomplete information, communication processes, marketing of programs, and decision under uncertainty. Data on these variables are hard to get, but the effort has not seriously been made and the variables tend, as a result, to drop out of consideration in policy analyses based on formal models.

OCR for page 86
102 problem-oriented studies are more likely to offer useful information to policy makers. In short, although good energy models are desired and needed, existing formal models are not yet up to the tasks for which they are used. Better analysis requires a serious research and data collection effort driven not only by immediate policy concerns but by a desire to improve understanding of energy use and general theories of consumer behavior. Such an effort implies changes in the use of formal models and other research methods. Formal models need input from other research methods, which are especially useful for supplying empirical tests of modeling assump- tions and predictions. Researchers should use the various research methods in a complementary fashion, using each to answer the kinds of questions for which it is best suited and, when more than one method is appropriate, using each as a check on the others. AS a general strat- egy, we advocate a combination of research methods as the best way to advance understanding of energy demand. 6 Problem-oriented research can combine with models in various ways. The results of some problem-oriented studies raise questions about which variables are most important to consider in formal demand analysis. For example, the data from evaluation studies on the wide disparity of response to a constant financial incentive suggests that something about the implementation of incen- tive programs (not now represented in models) may be more important than the monetary value of the incentive (a 6 Our discussion of the character of multimethod research on energy demand is not meant to minimize the real insti- tutional barriers to making this a normal part of policy analysis. The people who construct formal models and those who use other research methods often come from dif- ferent disciplinary backgrounds, belong to different pro- fessional associations, and communicate little with each other. And in policy-making organizations, there is often a similar split between units that do modeling and units that do other research, for example, program evaluation. There are some signs of improved communication, including some interdisciplinary conferences on energy demand issues and the existence of the present study--but the problems institutionalizing a multimethod approach still loom large.

OCR for page 86
103 major focus of modeling efforts). Problem-oriented studies are the only available way to estimate the effects of variables that are not now represented in models. Evaluation research on the implementation of conservation programs is one example; another is research on improving the quality of information available to energy users with feedback (Chapter 4). Problem-oriented research methods, in concert, can provide empirical help in estimating the parameters of models. For example, to estimate the effect of appliance labels that offer information on energy efficiency, small- scale laboratory experiments might first be used-to deter- mine what information is effective on labels and what presentation formats people consider useful. Field experiments in which labels are used in some locations and not in others would be the best way to get a realistic estimate of how much difference the best available labels make. Surveys of appliance purchasers can produce empiri- cally based estimates and act as a check on the findings from the smaller experiments. The results of these problem-oriented research efforts can inform policy about appliance labeling more usefully than can predictions from a model. They may also prove useful to modelers by pro- viding parameter estimates that would not otherwise be available in any empirically supported form. It might be possible, for example, to interpret information on the effect of labels as a change in a discount rate or a lag coefficient. (In a discrete choice model equation such as the one in Appendix A, labels might change the coeffi- cient of response to energy efficiency.) Sometimes research on qualitative factors such as pro- gram implementation or interpersonal communication cannot be used to estimate the parameters of variables in models because the variables are too hard to define and measure precisely. In those instances, however, problem-oriented studies can estimate the range of uncertainty for those parameters and can offer explanations for the variation. Problem-oriented studies and analyses of existing data can also be used to test models. A model that can predict the results of evaluation research, field experiments, or analysis of RECS data is more likely to be correct than one that cannot. Given the uncertainties in models, it would be wise practice to compare the output and assump- tions of models with empirical findings as a way of test- ing and refining models. Thus, a multimethod approach would have several impli- cations for energy modeling. It would lead to many small

OCR for page 86
104 changes in models--in parameter values and possibly in the ways variables such as price are represented--and it would probably change the variables included in models. And when qualitative factors, such as trust in information, prove important, it might lead to important innovations in the ways models are structured. In all these ways, energy modeling would be improved. A multimethod approach to research would also affect the conduct of problem-oriented research. Models would help set priorities for other research by identifying unanticipated effects of policies that call for more specific attention and, when a model's output depends critically on the value of a particular parameter and the estimate of that parameter is uncertain, by calling for research or data collection, using other methods, to estimate that value. The most important change that might arise from a multimethod approach, we hope, would be a shift of emphasis in the way energy demand analysis is conducted. Consumers of energy demand analysis might become less inclined to see in models the distillation of all knowl- edge about energy demand and more willing to see models realistically, as part of an ongoing process of analysis that relies on many techniques to build understanding.