APPENDIX B
FIRST ADVISORY REPORT
DEVELOPMENT OF THE NATIONAL ENERGY MODELING SYSTEM
COMMITTEE ON THE NATIONAL ENERGY MODELING SYSTEM
Energy Engineering Board
Commission on Engineering and Technical Systems
in cooperation with the
Committee on National Statistics
Commission on Behavioral and Social Sciences and Education
January 30, 1991
NATIONAL RESEARCH COUNCIL
WASHINGTON, D.C. 1991
COMMITTEE ON NATIONAL ENERGY MODELING SYSTEM.
PETER T. JOHNSON, (Chairman). Former Administrator, Bonneville Power Administration, McCall, Idaho
DENNIS J. AIGNER, Dean, Graduate School of Management, University of California, Irvine
DOUGLAS R. BOHI, Director, Energy and Natural Resources Division, Resources for the Future, Washington, D.C.
JAMES H. CALDWELL, Jr., Washington, DC 20016
ESTELLE B. DAGUM, Director, Time Series Research and Analysis Division, STATISTICS CANADA, Ottawa, Ontario, Canada
DANIEL A. DREYFUS, Vice President, Strategic Planning and Analysis, Gas Research Institute, Washington, D.C.
EDWARD L. FLOM, Manager, Industry Analysis & Forecasts, Amoco Corporation, Chicago, IL
DAVID B. GOLDSTEIN, Senior Staff Scientist, Natural Resources Defense Council, San Francisco, CA
LOUIS GORDON, Professor, Department of Mathematics, University of Southern California, Los Angeles, CA
VELLO A. KUUSKRAA, Chairman, ICF Resources Inc., Fairfax, VA
JAMES W. LITCHFIELD, Director of Power Planning, Northwest Power Planning Council, Portland, Oregon 97204
STEPHEN C. PECK, Director, Environment Division, Electric Power Research Institute, Palo Alto, CA
MARC H. ROSS, Professor, Department of Physics, University of Michigan, Ann Arbor, MI
EDWARD S. RUBIN, Professor, Department of Engineering & Public Policy, Carnegie-Mellon University, Pittsburgh, Pennsylvania
JAMES L. SWEENEY, Professor, Department of Engineering-Economic Systems, Terman Engineering Center, Stanford University, Stanford, California
DAVID O. WOOD, Director, Center for Energy Policy Research, Massachusetts Institute of Technology, Sloan School of Management, Cambridge, MA
Staff
MAHADEVAN (DEV) MANI, Study Director, and Acting Director, Energy Engineering Board
ARCHIE L. WOOD, Acting Executive Director, Commission on Engineering and Technical Systems
JUDY AMRI, Administrative Associate
PHILOMINA MAMMEN, Study Assistant
I. INTRODUCTION
The National Research Council's Committee on the National Energy Modeling System was established at the request of the Department of Energy (DOE) primarily to provide guidance to DOE and the Energy Information Administration (EIA) on the long-term development of a modeling system to support national energy analysis and strategic planning. This report is an interim advisory report that has been prepared by the Committee at the request of the Secretary of Energy.
The Committee held its first meeting in Washington, D.C, on July 31 and August 1, 1990, and has since met again on September 20 and 21, 1990 and November 8 and 9, 1990. At the first two meetings, the Committee was briefed on a preliminary system of existing models configured and applied by DOE and EIA to the analysis underway of National Energy Strategy (NES). That system comprised the 2-tier set of models shown in Figure 1.
The Committee has conducted its work to-date using information provided through presentations from DOE and EIA (See Appendix 1) and from reviews of selected documents (See Bibliography). The Chairman and committee members also met with the Secretary of Energy and his staff on November 6 and 9, 1990 to exchange views on the use of existing models and data in on-going energy strategy analysis at the department and on needs for future capabilities of a National Energy Modeling System (NEMS). Largely on the strength of these presentations and exchanges, the committee has made observations on DOE's use of existing models that is the subject of this advisory report.
The report addresses, in broad terms and at a high level of aggregation, the efficacy of existing models configured by DOE to support the NES activities and the adequacy of the underlying data, assumptions and methodology. It also outlines the Committee's goal in focusing on the long-term development of modeling capabilities at DOE. The committee expects to complete its work and publish a final report in the Fall of 1991.
II. FINDINGS
A. Background
The analytical staff of the Department of Energy (including EIA) has been called upon to support the effort leading to the formulation of a National Energy Strategy (NES). A part of that effort involved the generation of one or more “Reference Case” projections of the U.S. energy situation to the year 2030 along with a set of “excursions” (scenarios) off the reference cases.
The existing modeling capability of the department was used in this effort. The predominant group of models used resided in the Energy Information Administration (EIA) and the DOE Office of Policy, Planning and Analysis (OPPA), although some use
was made of other models, such as the ARGUS (Argonne Utility Simulation) model of the Argonne National Laboratory.
Ad hoc interagency staff organizations, particularly the NES Modeling Subgroups, were created to assist in developing assumptions and off-line parameters for the modeling effort (See Figure 1). The cooperation involving the DOE policy and program offices, the EIA, the national laboratories and ad hoc inter-agency staff organizations in developing assumptions and conducting off-line estimations for the NES exercise is to be commended.
In the Committee's view, the foregoing process was constructive. The working group format served to pull together and synthesize information. The interactions and the pooling of expertise were commendable uses of personnel resources.
B.General Observations
There is no comprehensive model or group of integrated models existing within DOE, or probably within the federal government, that has the analytical capability to respond to the long time horizon needs imposed by the NES effort. In the Committee's view, the approach taken by DOE in using available models as appropriate, along with off-line supplemental analysis as necessary, was a rational response to the department's need for expedient support of the NES process. The rough integration of the modeled and off-line intermediate analyses that DOE accomplished through the calibration of the FOSSIL2 model has been a useful way to maintain consistent accounting and reporting of results. (See Note in Figure 1 on the calibration of FOSSIL2.)
The aggregate structure of the models used, however, has significant limitations relative to the analytical results reported by DOE and presented to the Committee. Thus, it is important for the decisionmakers who will be using the results of the NES analyses to appreciate the limited power of the existing set of models used for evaluating policy choices. It would be misleading to assign too much quantitative precision to the results of the model runs or to presume that the models incorporate a great deal of relevant detailed information that can enhance judgments about the future impact of policy choices, particularly impacts beyond a decade or two. In the presentations that were made to the committee, little reference was made per se to the validation of the models used in the current national energy strategy analysis exercise. Policymakers should appreciate the important role of the a priori assumptions and simplifications, and the off-line contributions made by the NES Modeling Subgroups in shaping the excursions or scenarios, which to a great extent dictated the results of model runs. In instances where such working groups play such an important role, DOE ought to consider enhancing the working group format with greater outside participation.
As the Committee understands it, the NES analysis and the development of a National Energy Modeling System were initiated by the DOE Office of Policy, Planning and Analysis, which was also responsible for the choice of excursions or scenarios to be analyzed and for the selection and application of the FOSSIL2 integrating model. EIA has provided extensive analytical support in these activities, but not in its customary role
of generating, independently, energy information and forecasts such as the Annual Energy Outlook.
C. Specific Comments
1. Long-Run Modeling and Forecasting
None of the current models used in the NES analysis was developed to generate 40 year projections. It would be misleading to presume that the models incorporate a great deal of information relevant to the long-term implications of the excursions or scenarios analyzed.
The models are deterministic, and are based either on historical data and relationships that may or may not have relevance to situations anticipated in the future, or on external judgments about the course of technological and institutional changes. The variances can be high in the relevant parameters as well as in the structure of relationships within the models. It may well be that uncertainties are so great for forecasts over a 30 to 40 year period that deterministic forecasts cease to be useful over such time frames.
There are issues still to be resolved concerning the internal consistency of the models within the two-tier hierarchy of which they are part. (See The FOSSIL2 Model, below.)
2. The FOSSIL2 Model
In the current NES exercise, the FOSSIL2 model is the integrating vehicle used by DOE to yield long-term forecasts. While the calibration and application of FOSSIL2 may be a useful way to (a) achieve rough integration of the analysis performed with other models, and (b) maintain consistent accounting and reporting of results, the aggregate modeling structure (as depicted in Figure 1) has major limitations relative to the analytical results being obtained from the models.
The FOSSIL2 model itself, according to DOE, is not well understood in the department except by a few people in the Office of Policy, Planning and Analysis. And, the staff at EIA indicated that they too did not have in-depth knowledge of the characteristics and workings of the FOSSIL2 model.
The compatibility of the FOSSIL2 model with the other models is also not well understood by DOE. The influence of off-line inputs to the FOSSIL2 model does not seem to have been adequately studied. For example, the influence of the coal-dominated ARGUS model in determining future fuel choices for electric power production in the FOSSIL2 model could be important and perhaps overly restrictive.
Many of the excursions (from the reference cases) analyzed by DOE with the FOSSIL2 model involved substantial changes in the use of energy technologies, and assumptions on technology availabilities, efficiencies and costs. While the FOSSIL2
model does solve for an energy market equilibrium (both prices and quantities), the excursion analyses entailed no reexamination by DOE of the macroeconomic consequences of the technology changes and other changes assumed. Under these circumstances, the Committee questions the outcomes of the excursions (especially for instance those involving high levels of energy conservation) as being potentially inconsistent with reference case economic assumptions. (See also Adequacy of Data, below.)
The Committee sought but did not obtain clear justification from DOE for the selective use of the FOSSIL2 model for virtually all time horizons considered in the NES analysis i.e, to the exclusion of other available integrating models such as the Intermediate Future Forecasting System (IFFS) used by EIA to generate mid-term forecasts.
3. Adequacy of Data
Much of the energy data pertaining to the NES modeling and analysis are on the supply side of the energy markets. To-date, much less effort has been expended on obtaining demand side data. As a consequence, analyses of the demand side are weak.
Parameter values used in the models are based on very limited data and aggregated information about energy consumption and demand side management. For example, there is little information (at DOE and elsewhere) of the potentials for energy efficiency improvements in the industrial sector. The feasibility of new or alternative energy supply and some end-use technologies appear to rely on estimates made by DOE that are based on limited experience and have a documented history of over-optimism. For instance, specifications made by DOE of technology-derived gains associated with clean coal technologies and methanol are important influences in the analysis, but appear to be speculative.
Energy efficiency improvements were invoked by DOE in some cases and ignored in others. Many efficiency measures seem to have been inadequately studied preparatory to the analysis, and may lead to undue reliance being placed on modeling results.
4. Dealing with Uncertainty.
In the analyses to date, it is the Committee's view that little attention has been given to dealing with uncertainties regarding assumptions, input data and results, and to risks posed by an uncertain future. More direct treatment of uncertainties will strengthen the NES analyses.
Ignoring uncertainties could lead to undue reliance on “model outputs” and a tendency to overlook the speculative nature of the projections, particularly over long-term time horizons. In context it is worth emphasizing that, for the models considered in the NES analysis, the Committee is not aware of any basis to assign a greater level of certainty to the differences (“deltas”) in outcomes among various excursions than to the outcomes of the individual excursions themselves.
III. Concluding Remarks
Despite the general limitations of forecasting models and aside from the specific limitations of the models used by DOE in the NES analysis, the Committee believes that models can play a crucial role in enabling informed judgments and decisions to be made in matters of national energy policy. Thus, the committee considers it vital that DOE continue to develop and sustain capabilities for analyzing national energy issues using resources from within the department and from appropriate organizations in both the public and private sectors.
But the models used for energy forecasting and policy analysis have inherent limitations that must be clearly recognized. Indeed, if the future were not malleable there would be little interest and less controversy over the validity of forecasting models in general. In terms of the particular models used in the NES analysis, DOE should take care not to overemphasize the models' capabilities in light of the types of deficiencies observed by the committee. Perhaps, in documenting its current analytical efforts with regard to national energy strategy and communicating findings to the public, DOE should emphasize that the analysis of policy options and combinations thereof were defined and developed drawing on many resources, only one of which was DOE's modeling capabilities.
The decision taken by the Department of Energy to develop and maintain an effective National Energy Modeling System (NEMS) is an important step. The Committee is pleased to participate in that process and to assist DOE in achieving that objective.
The Committee views its charge as follows:
The Committee will develop a statement of the desirable analytical, modeling, and supporting database capabilities necessary to support the evaluation of U.S. energy policy options. The committee will examine the correspondence between the current and desirable capabilities and will recommend priorities for development of the National Energy Modeling System (NEMS). Special attention will be given to the regional and international context for U.S. energy policy development, and to the interaction between energy and environmental policy formulation and analysis.
The more specific objectives of the committee include:
-
Recommend a process and priorities for on-going modifications to the existing EIA/DOE modeling capabilities to better serve the purposes of planning, forecasting and policy formulation as exemplified by the ongoing NES process, considering data needs, model enhancements and integration. However, detailed technical assessments will not be undertaken of specific models currently in use at DOE and EIA.
-
Examine whether improved information, both new data series and focused case studies on how energy systems function, would be critical to achieving the recommended modifications and improvements.
-
Discuss the appropriate role for models in the policy formulation process, the inherent limitations of models, and the specific practical limitations and tradeoffs of data collection and integration in developing more effective modeling capabilities.
-
Examine broadly the capabilities of relevant public, academic, industrial, regional, state and international interests and make general recommendations of how the activities of these parties can complement and enhance the capabilities of the DOE and EIA.
The execution of this charge will require significant effort by the Committee to better understand the prospective applications for a National Energy Modeling System within DOE and possibly by other users; to evaluate the existing modeling resources that are available for use by DOE either internal to the department or elsewhere; and finally, to propose an approach which will enhance the modeling support for future policy analysis and strategic planning given available resources. This effort will require further collection of information from DOE and other sources, and time for evaluation and deliberation by the Committee.
BIBLIOGRAPHY
Energy Information Administration (EIA). July 2, 1990. Requirements Analysis For a National Energy Modeling System. Paper prepared by an Energy Information Administration Working Group. Washington, D.C. U.S. Department of Energy (USDOE).
Energy Information Administration. September 19, 1990. A Comparison of Requirements with Current Capabilities and Issues in the Design of a New System. Prepared by an EIA Working Group. Washington, D.C. USDOE.
The AES Corporation, Arlington, VA. July 6, 1990. An Overview of the FOSSIL2 Model: A Dynamic Long-term Policy Simulation Model of U.S. Energy Supply and Demand . Prepared for the United States Department of Energy, Office of Policy and Evaluation. DOE Contract No. DE-ACO1-89PE79041. Washington, D.C., USDOE.
Energy Information Administration. October 1990. Improving Technology: Modeling Energy Futures for the National Energy Strategy (Draft). EIA. Washington, D.C. U.S. DOE.
U.S. Department of Energy. April 1990. Interim Report. National Energy Strategy. A Compilation of Public Comments. DOE/S-0066P. Washington, D.C.: U.S. DOE.
U.S. Department of Energy. July 26, 1989. Statement of Admiral James D. Watkins, Secretary of Energy, before Committee on Energy and Natural Resources, United States Senate, Washington, D.C.: U.S. DOE
Energy Information Administration (EIA). 1989. Directory of Energy Data Collection Forms: Forms in Use as of October 1989. DOE/EIA-0249(89). Washington, D.C. U.S. DOE.
Energy Information Administration (EIA). 1990. Annual Energy Outlook with Long-Term Projections. DOE/EIA-0383(90). Washington, D.C.: U.S. Government Printing Office.
APPENDIX
DOE and EIA PRESENTATIONS TO THE COMMITTEE ON THE NATIONAL ENERGY MODELING SYSTEM (NEMS)
July 31 & August 1, 1990
“The National Energy Strategy and the National Energy Modeling System”
Linda G. Stuntz, Deputy Under Secretary, Policy, Planning and Analysis, U.S.
Department of Energy (DOE)
“The Development and Operation of the NEMS: An EIA Perspective”
Calvin A. Kent, Administrator, Energy Information Administration (EIA)
“Reference Case for the National Energy Strategy”
Eric Petersen, DOE Office of Policy, Planning and Analysis
“Current Configuration and Applications of the NEMS”
W. Calvin Kilgore, Director, EIA Office of Energy Markets and End Use
“Use of Energy Models and Data Systems at EIA”
Lawrence A. Pettis, Deputy Administrator, EIA
“Requirements Analysis for the NEMS”
C. William Skinner, Technical Assistant to the Administrator, EIA
September 20 & 21, 1990
“Update on NES and Key Policy Issues”
Linda G. Stuntz, Deputy Under Secretary, Policy, Planning and Analysis, DOE
“Context for the Analysis of Key Policy Issues”
--Descriptions of Options
--Definition of the “Reference Case”
--Conceptual basis for integrated analysis
--Choice of issues to “demonstrate” characteristics & capabilities of models applied to the current NES effort
Eric Petersen, DOE Office of Policy, Planning and Analysis
“Sectoral Energy Demand: PC-AEO Models”
John D. Pearson, Director, Energy Analysis & Forecasting Division, EIA Office of Energy Markets and End Use
“Energy Supply-Coal & Electricity: NCM & ARGUS”
Mary J. Hutzler, Director, Electric Power Division, EIA Office of Coal, Nuclear, Electric and Alternate Fuels
“Energy Supply-Oil & Gas” GAMS & PROLOG”
Susan Shaw, Analysis and Forecasting Branch, Reserves and Natural Gas Division, EIA Office of Oil and Gas.
“Integration Model: FOSSIL2”
Roger Nail, AES Corporation, Arlington, Virginia
“Oil Market Simulation Model”
Erik Kreil, International/Contingency Information Division,
EIA Office of Energy Markets and End Use
“DRI Macroeconomic Model”
Ronald Earley, Economics and Statistics Division, EIA Office of Energy Markets and End Use
“Wrap-up on Current NES Analysis Effort”
Robert C. Marlay, Acting Director, DOE Office of Program Review and Analysis
“Update on NEMS Development/Look-Ahead”
Calvin A. Kent, Administrator, EIA