Assessing the Use of Agent-Based Models for Tobacco Regulation (2015)

Appendix C

Assessing Agent-Based Models for Regulatory Applications: Lessons from Energy Analysis

Alan H. Sanstad¹

ABSTRACT

Agent-based modeling (ABM) has been proposed as a promising method for analyzing behavior patterns related to smoking in connection with regulation of tobacco products. This possibility raises questions about model validation and evaluation, uncertainty quantification, and how the models should be judged with respect to suitability for policy applications. These issues have long been present in social science–based and policy-focused computational modeling; computational energy modeling is an important example. This paper reviews energy modeling from methodological and epistemological perspectives to draw lessons for ABM regarding model validity, the treatment of uncertainty, and criteria for decision makers to apply when considering agent-based models for use in regulation.

INTRODUCTION

Computational modeling of social and economic systems and behavior developed in the 1950s and 1960s and became well established in the 1970s. In this category of modeling, so-called agent-based modeling (ABM) has emerged as an active research field.² Recently, ABM has been proposed as a promising method for analyzing smoking-related behavior patterns, for use in regulating tobacco products.

__________________

¹Berkeley, California. alansanstad@gmail.com.

²Throughout this paper, ABM will be used as an abbreviation of both “agent-based modeling” and “agent-based model(s).”

Such applications of ABM raise basic questions that have a long history in social science–based and policy-focused computational modeling, including

• What can be learned from computational models about behavior or other phenomena of interest?
• How can the uncertainty associated with their structure and inputs and their quantitative outputs be analyzed?
• How should their validity and or utility—that is, their usefulness—be evaluated in a regulatory context?

Regarding ABM specifically, here as in other contexts, some claims have been made about its capabilities and putatively unique advantages over other modeling approaches, including its ability to represent social and behavioral phenomena with a high degree of detail, and the model fidelity that this is said to provide. How should these claims be assessed? More generally, does ABM have characteristics that pose validation or evaluation questions that are different from ones that are relevant for other types of computational social science models? Or does the nature of ABM somehow obviate such considerations?

Such questions are fundamentally epistemological: They pertain to defining and characterizing what knowledge can be generated by computational models and to understanding how their outputs should be applied to decision making. Notwithstanding both their importance and the attention devoted to them in different social science and policy modeling domains over many years, those questions continue to be challenging and generally unresolved.

The issues are well illustrated in the field of computational energy modeling. Since the 1970s, this type of modeling has steadily expanded in scope and in importance for energy regulation and policy making. In recent years, in addition to energy analysis specifically, it has increasingly been applied to the problem of mitigating emissions of carbon dioxide (CO₂) from energy production and consumption. In terms of the level and range of activity, prevalence, and influence, it is perhaps the primary example of computational policy modeling.³ Epistemological issues were recognized early in energy modeling’s more-than-four-decade history and continue to be important. The premise of the present paper is that, notwithstanding technical and methodological differences, energy modeling can provide valuable insights and lessons for ABM with respect to model validity and evaluation, uncertainty quantification, and policy applications.

__________________

³Energy modeling is also an academic field; however, the preponderance of such work deals at least implicitly with policy and regulatory applications rather than constituting “basic” research.

The paper is organized as follows. Immediately below, key terminology is explained. A brief overview of energy modeling is presented next. The paper then turns to a discussion of methodological and epistemological issues, particularly model calibration and its relation to uncertainty, and the value of increasing model complexity. Examples of agent-based modeling of energy systems are then presented, and two particular studies of this type discussed in detail. The concepts of fundamental model uncertainty and robustness and their potential relevance to ABM are then briefly reviewed. Lessons from energy modeling that are applicable to ABM are followed by recommendations for assessing potential regulatory applications of ABM and concluding remarks.

Terminology

In the context of computational modeling, such terms as validity and validation have not only different technical meanings but different connotations among and in some cases within disciplines. Particularly in the social sciences, the terms may be interpreted as representing concepts and methods that are more appropriate to the physical sciences. In energy modeling specifically, validity and validation as such are not only not generally discussed or practiced, respectively, but in some quarters are viewed as fundamentally inapplicable. There are, however, no standard or generally accepted alternatives either conceptually or in nomenclature. Thus, in this paper, the terms validity and validation will be used, as well as evaluation, and quality. But the reader should understand that these terms are highly approximate and simply provide a shorthand for discussing the assessment of computational models and their usefulness in applications.

Energy models here refers to computational models, based on economic and optimization principles, of energy systems, entire economies with particular detail on energy sectors, or specific energy-using sectors, particularly residential and commercial. Technically, the system and economic models are generally (although not exclusively) of the equilibrium type (represented by systems of nonlinear algebraic equations), the mathematical programming type, or the optimal control type.⁴ “Partial equilibrium” models in this case represent energy demand and supply sectors and their market interactions. “Computable general equilibrium” (CGE) models also have similar components of energy demand and supply sectors and their market interactions, but these components are embedded in a full representation of a complete economy, which among other features includes direct or indirect interactions between the energy sector and all other parts of the

_____________________

⁴In economics, these types are not mutually exclusive, but the distinctions among them are useful in understanding the contemporary landscape of energy modeling.

overall economy. Many models of the electric power system are of the linear programming type.

In this context, computation means finding solutions to the optimization problem or the system of equations represented by the model.

An important development in the last several decades has been the advent of “integrated assessment” models for the economic and policy analysis of global climate change. Those are energy models coupled to reduced-form representations of the climate system, and in some cases, other physical and ecologic systems.⁵ In what follows, for brevity’s sake the term energy model will generally be used rather than energy and/or integrated assessment model; this will not distort the discussion or the conclusions, but the reader should be aware of the distinction.

THE CURRENT LANDSCAPE OF ENERGY MODELING

This section discusses and gives examples of non-agent-based energy models and their policy applications.

Energy models are not just widely applied to but have become the predominant analytical methodology for energy policy and regulatory analysis in the United States. Although in this, as in other computational modeling applications, the models are often referred to as “tools,” such a characterization understates their role and influence. In fact, to a great degree they define the universe of policy discourse and determine what questions can be asked, what form answers take, and what constitute useful data by virtue of providing model inputs.

Energy models are used by agencies and other policy and regulatory bodies at all levels of government. At the federal level, the Energy Information Administration (EIA) maintains and applies the National Energy Modeling System (NEMS), which projects the evolution of the U.S. energy system over several decades (EIA, 2009; NRC, 1992).⁶ The primary use of NEMS is in the production of the Annual Energy Outlook (AEO), which documents the details of each year’s updated projection, including details on individual fuels (electricity, natural gas, and so on), energy production, and energy consumption in different sectors. (An example is presented below.) It is also used to analyze the potential effects of proposed policies

_____________________

⁵The best-known example is Nordhaus’s DICE (Dynamic Integrated Climate Economy) model, which is based on optimal control principles (Nordhaus, 2008).

⁶EIA is the federal agency that has principal responsibility for analyzing energy issues. Its activities include the collection and dissemination of energy statistics and computational energy-policy modeling, both short-term and long-term and on national, international, and regional scales. Although in the U.S. Department of Energy, EIA reports to Congress, for which it conducts analyses of energy topics on request.

on the energy system, including effects on supply, demand, prices, and costs.⁷

In applications of this type, NEMS and other models are used in a standard analytical structure: the model is solved for a “reference” or “baseline” or “business-as-usual” projection without the policy in question, and then re-solved with the proposed policy introduced into the model (for example, by introducing an emissions tax or a policy-induced technological improvement). The results are then compared to identify the policy’s effects as represented by the model. Both types of projection are called scenarios. An example is shown in Figure C-1, which displays the output of a NEMS–AEO reference case, specifically, how much electricity is generated from different sources, out to the year 2040. (The vertical axis unit is trillions of kilowatt hours.) The “History” period is based on empirical observations and “Projections” on model output.

A recent example of the use of NEMS for policy analysis is a study of the proposed Clean Energy Standard Act of 2012, which aimed to reduce greenhouse gas (GHG) emissions from electric power generation (EIA, 2012).⁸

The U.S. Environmental Protection Agency (EPA) uses several types of policy models for energy regulatory analysis. It uses two CGE models for analyzing environmental policy problems related to energy: the Applied Dynamic Analysis of the Global Economy (ADAGE) model (Ross, 2005), and the Intertemporal General Equilibrium (IGEM) model (Goettle et al., 2007). Whereas NEMS was developed and is maintained and operated by EIA itself, these two are proprietary models developed outside EPA but used by the agency under long-term contracts. An example of their use was an analysis of the proposed legislation HR 2454 of 2009, colloquially known as the Waxman–Markey Act, which called for the establishment of a GHG emissions “cap and trade” regime and other emissions-reducing measures to reduce U.S. emissions to more than 80 percent below their 2005 level by 2050 (EPA, 2009).⁹ EPA also uses a commercial proprietary linear programming model of the electricity system (EPA, 2010) to analyze pollution emissions; a current application is the development of regulations to reduce CO₂ emissions from power generation.

At the state level, some energy regulatory entities use modeling for such purposes as utility regulation, energy-efficiency policy analysis, and,

__________________

⁷NEMS is a partial-equilibrium model and represents the interconnected system of markets that links energy supplies and demands driven by economic, demographic, and other factors over about a 3-decade horizon with annual steps.

⁸The study was requested by Senator Jeff Bingaman (D-NM), chairman of the Senate Committee on Energy and Natural Resources at the time.

⁹The study was requested in 2009 by Henry Waxman (D-CA), chairman of House Energy and Commerce Committee, and Edward Markey (D-MA), chairman of the Energy and Environment Subcommittee at the time.

FIGURE C-1 NEMS projection of energy use for electric power generation, in trillions of kilowatt hours by year.
SOURCE: EIA, 2014.

in recent years, GHG emissions abatement (e.g., Goldstein et al., 2008). Modeling also plays a crucial role in operations management of and planning for the electric power transmission system. Outside the government, political advocacy groups have increasingly turned to modeling to develop and lobby for (or against) specific energy-related and environment-related proposals for legislation. For example, using a version of the linear programming model mentioned above, the Natural Resources Defense Council developed a policy architecture for electricity CO₂ emissions abatement (Lashof et al., 2013).

Methodological Aspects

Although, as noted above, energy models are for the most part built on a small number of underlying principles, there is a great deal of variety among models across the entire field; to some extent, this reflects the many ways that these principles can be implemented in practice. However,

a small number of models can be considered predominant and most influential in the United States. Broadly speaking, they share several critical features. First, they are deterministic: There is no explicit representation of uncertainty in the model on the part of either the model builder or the agents—consumers and firms.¹⁰ Second, they are high-dimensional; that is, they contain a great deal of detail regarding elements that, depending on the model, may include production, consumption, technologies, economic sectors, markets, policies, and so forth.

With few exceptions, energy models designed for and applied to long-run—multidecade—analysis are parameterized by calibration.¹¹ The meaning of “calibration” varies among fields of computational modeling. In energy modeling, the essential aspect is that values of model parameters are set without the direct use of statistical methods or techniques relating the model to empirical data. Some models contain simplified engineering descriptions of individual technologies or technology types, and parameter values are obtained from, for example, engineering studies or surveys. In those and in models that have a more aggregate structure, there are two primary calibration techniques. First, most model parameters are taken from other sources. A key example is “substitution elasticities,” which characterize how consumers and firms make trade-offs in their choices among goods and services. These elasticities have often been adopted from empirical studies that did use statistical methods but typically applied to econometric (statistical) models that have considerably different structure from and simpler structure than numerical energy models.¹² In the common circumstance in which a range of estimates appears in the literature, the mean estimate is often used.

Second, parameter values can be set by “tuning.” The most important examples in energy modeling are parameters determining the magnitude of aggregate improvements in energy productivity, which are directly analogous to the labor-productivity parameters that are common in macroeconomic models. The magnitudes of the values of these energy-productivity parameters are commonly determined by exogenously setting them informally (i.e., without using a statistical method) to reproduce approximately the historical observed trends in the ratio of economic output to energy

_____________________

¹⁰This is an oversimplification in that what could be called experimental uncertainty quantification has been conducted with several of the models in question. However, their basic and typically used mode is deterministic.

¹¹The qualification “long-run” is used to distinguish them from modeling conducted in the electric utility industry for real-time or 1-day-ahead forecasting and optimization, which increasingly incorporates explicitly statistical and stochastic components and techniques.

¹²A rare exception is the IGEM model mentioned above, in which case many of the parameter values are obtained econometrically directly, that is, by an integrated estimation of the equilibrium model itself with suitable empirical data.

input, whether at the level of an individual sector or of the entire economy (the ratio of gross domestic product to economywide energy consumption).

CURRENT STATE OF THEORY AND PRACTICE

Energy models are subject to widely varying degrees and types of assessment. Those used by some federal agencies, for example, are subject to review requirements. However, there are no specific, generally applied theoretical, empirical, or computational methods or procedures for defining or determining energy model validity, verisimilitude, or quality. This in part reflects a sustained inattention to epistemological issues after a period in the 1970s and 1980s during which they received considerable attention.¹³ However, it also has to do with the special challenges of addressing such issues for models designed for and applied to projections of an energy system or economy decades into the future. Even if such a model is in some fashion empirically grounded in current or historical data, it is not clear whether or how such grounding provides evidence or assurance of its suitability for such long-run analysis. Indeed, it is difficult even to define suitability or validity in this circumstance. Moreover, there is prima facie a level of uncertainty associated with multidecade projections that is categorically different from that involved in, for example, short-run economic forecasting—up to several calendar quarters.

In practice, broadly speaking, model quality is claimed on such grounds as internal consistency, the plausibility of assumptions and results, and the usefulness of generating “insights,” whereas uncertainty is addressed by appealing to the logic of scenarios. The following subsections discuss those topics in more detail.

Calibration and Knightian Uncertainty

Returning to the energy-modeling overview and scenario example (from the NEMS model) of the previous section: How should such projections be interpreted? According to EIA (2014, p. iii),

__________________

¹³This history is sketched in the annex to this appendix.

This statement is a candid acknowledgment of fundamental modeling limitations and a straightforward and reasonable disclaimer. However, it also expresses a perspective on uncertainty and scenario—and implicitly model—validity that, although written regarding NEMS and the AEO, captures much of current standard energy and integrated assessment modeling epistemology more generally. This is reflected in methodological views expressed by other modeling groups. Clarke et al. (2007), for example, state, “Model-based scenario analysis is designed to provide quantitative estimates of multiple outcomes and to assure consistency among them that is difficult to achieve without a formal structure” (p. 43). They also suggest that “[a]lthough the future is uncertain and the scenarios are strongly dependent on many underlying assumptions, this research provides useful insights for those engaged in climate-related decision making” (p. 5). Similar views are expressed by other important stakeholders, such as the Intergovernmental Panel on Climate Change (2000, p. 3):

Although not so intended, statements of this type are in a way implicit explanations of the general absence of formal, quantitative uncertainty analysis and validation in energy modeling. An important reason is the calibrationist methodology within which the models are developed and applied. Dawkins et al. (2001) discuss calibration in considerable detail; although their focus is macroeconomic modeling and applied general equilibrium modeling especially in international trade applications, their observations apply to energy modeling as well (including both computable general equilibrium and other types):

__________________

¹⁴The alternative cases are a small number of scenarios that assume different economic growth rates, world oil prices, or technologic progress.

That models are deterministic and their outputs conditional does not, of course, imply that they do not contain and reflect uncertainty. However, in the calibrationist approach to energy modeling and the models’ use to generate scenarios, such uncertainty is of a type often referred to as Knightian; it cannot be described or quantified by assigning probabilities.¹⁶ In energy modeling, in principle, the scenario approach is a reasonable way of addressing Knightian uncertainty. But the common practice of computing small numbers of reference cases or scenarios and policy scenarios that are incremental to the reference cases in effect suppresses a great deal of uncertainty.

The reason is that it is rarely the case that the particular set of input parameter values chosen for the underlying model is uniquely justified.¹⁷ The evidence often suggests no more than that the appropriate magnitudes of specific parameters, such as substitution elasticities governing economic choices among different commodities, probably fall within a particular range. However, it is customary for modelers to choose, for example, the midpoints of the ranges for the values of the parameters. In deterministic models, this is not justified as, say, the mean of a uniform distribution; rather, it is simply a heuristic. But in such a case, the available information implies that every point in the entire interval is equally plausible for use in the model. For a given model, there may be multiple such “equiplausible” intervals for various parameters. Under those circumstances, the complete warranted input set constitutes all possible selections of specific parameter values from this set of intervals. (For example, if there are two uncertain parameters and two such intervals, the entire input set is a rectangle.) Thus, all the simulations that would in principle be generated by running the model for all of these selections of parameter values are equally “valid” or plausible. Even when particular scenarios are defined by, for example, specific assumptions regarding policy—such as the magnitude of a CO₂ emission tax—this implies that taking account of all the relevant information requires that the potential effects of the policy should be computed by running the model across the entire input set defined above.

That, with a small number of exceptions, this is not done in standard energy modeling practice is what was referred to above as a suppression

_____________________

¹⁵These observations are about “applied general equilibrium” economic modeling, which is the application of microeconomic principles to the representation of an entire economy. Hence the use of the term micro-.

¹⁶This distinction was introduced by the economist Frank Knight (1921).

¹⁷This is pointed out, for example, by Clarke et al. (2007) in their discussion of reference-case scenarios in global integrated assessment modeling.

of uncertainty, in this case Knightian. In effect, the conventional method of choosing a single set of representative inputs means that scenarios are, abstractly, single points in a very high-dimensional space of equally credible points—that is, the hypothetical space of all model simulations, indexed by parameter value choices.¹⁸

In calibration-based modeling, parameter sensitivity analysis is sometimes applied to address model or scenario validity putatively, whether implicitly or explicitly. Specifically, a finding of low sensitivity of model output to small changes in the value of one parameter or a small number of parameters is claimed as evidence that the base output is in some sense valid. With calibration, however, that logic is flawed. Conceptually, sensitivity analysis is “intrinsic” in the sense that, essentially by definition, it provides information about the model itself, not about the relationship between the model and the system or phenomenon that it represents. When a model can actually be demonstrated to be invalid—to be in some way an inaccurate representation—a finding of low sensitivity merely indicates that it is, in a manner of speaking, robustly inaccurate around the base parameter values.

However, a similar point holds in contexts like the one discussed here, in which validity itself is difficult to define or measure but a model has recognized credibility, usefulness, or acceptable quality. A given calibrated model with chosen parameter values may provide useful information to a decision maker. In this case, a finding of low sensitivity to the values is itself useful insofar as it suggests that the model may be robustly informative around the values. Nevertheless, that cannot be claimed to demonstrate any form of validity in the sense of empirical fidelity, nor would a finding of high sensitivity necessarily demonstrate invalidity.

Even in calibrated models, there may be cases in which uncertainty regarding a parameter value is either absent, on the one hand, or can be represented by a probability distribution (that is, is non-Knightian), on the other. In some instances, the available evidence may support the specific parameter values that are chosen as opposed to suggesting only a range of equally plausible values. If the exact values themselves are known with certainty, sensitivity analysis provides no information, because nearby values cannot occur. It is in the more common circumstance, in which the values are known to have a most likely but not certain value, that sensitivity analysis may be most informative. In such cases, however, the implication is that some sort of probability distribution can be assigned to the parameter (as opposed to its value being characterized by Knightian

_____________________

¹⁸A noteworthy exception is the work of Lempert and colleagues (e.g., Lempert, Popper, and Bankes, 2003; Lempert, Bryant, and Bankes, 2008), which will be discussed later in the paper.

uncertainty). Thus, although still informative, sensitivity analysis is an element of uncertainty quantification.¹⁹

Complexity and Validity

Although, as noted above, there is an absence of concepts and methods for assessing energy models objectively, contemporary energy-modeling discourse and practice reveal a widely held belief that increasing levels of model detail and complexity yield greater validity or verisimilitude and improved usefulness for policy applications. Calibration is both a cause and a consequence of the trend toward ever more complex models. On the one hand, once a calibration rather than estimation philosophy is adopted, models are to a great degree freed from constraints of the data requirements that arise when statistical techniques are used. On the other hand, that freedom encourages the addition of detail, in that model components need not be directly tied to data for purposes of parameterization. In turn, the possibility of validation in any traditional sense becomes increasingly remote.²⁰

At the same time, however, without actual metrics for gauging the various dimensions of model quality, there are no formal grounds for asserting that increasing detail and complexity yield greater validity. But greater detail inarguably results in a larger and more complex system of relationships between model inputs and outputs. The EIA phrase quoted above is useful in understanding the implications: “[Model] projections are not statements of what will happen, but [rather] of what might happen” (emphasis added). Putting more detail in a model increases the number of possibilities of what might happen, the complexity of any single “instance,” or both. In this respect, increased detail increases implicit uncertainty in calibrated models.

Beyond energy applications specifically, the belief that greater detail itself necessarily increases the validity of a calibrated computational model has little or no formal theoretical or empirical justification. There is an instructive contrast with how this and related issues are addressed in statistics and information theory. First, in the latter the degree of validity—accuracy or inaccuracy—of a model can be precisely defined in terms of statistical bias, and the extent to which additional model detail or complexity re-

_____________________

¹⁹This assumes that the parameter itself has previously been statistically estimated, not the model containing the parameter, so that the model itself is still of the calibration type.

²⁰The steady increase in model complexity also reflects the economic consequences of radical improvements in computational hardware and software over the last several decades. Computation itself has become extremely inexpensive while software advances have made model construction much easier. At the same time, observation and measurement—empirical analysis, including that which supports model construction—have remained expensive. Thus, in effect, computation has been steadily substituted for measurement.

duces bias—increases accuracy—can be quantified. Second, the uncertainty associated with a model’s representation of the underlying phenomenon, system, or relationship that it describes can be defined and quantified in terms of variance. Moreover, the relationship between these two quantities of accuracy and uncertainty can be exactly characterized; the result is known as the bias–variance trade-off: An increase in model dimensionality that reduces bias is directly associated with an increase in variance (Burnham and Anderson, 2002). Those concepts have no well-established analogues in calibration-based computational policy modeling in general (including energy modeling).²¹

EXAMPLES OF AGENT-BASED ENERGY MODELING

Energy has not been a major focus of ABM, but there is a sufficient body of work on which to base discussion of several of the key issues discussed so far in this paper.

Electricity-Market Modeling

Most ABM in energy has focused on the electric power system, in particular wholesale electricity markets. An excellent critical review of this work is provided by Weidlich and Veit (2008). As they describe, these markets have characteristics that make them challenging to analyze with conventional economic optimization and equilibrium modeling. They deviate from the conditions of the “perfect competition” assumption that underlies most such modeling, and the participants in the markets engage in “strategic behavior”; that is, in making decisions they take into account the potential decisions, and reactions, of other participants. ABM methods have been applied to study, for example, the market consequences of variation in agents’ behavioral rules. Another ABM application is comprehensive modeling of overall electric power systems that incorporates high levels of detail about technology and other elements (Barton et al., 2000; Koritarov, 2004). More recently, models for large-scale energy and GHG policy analysis have been created that contain the same general components as the energy models discussed in this paper but are based on ABM principles; Gerst et al. (2013), for example, represents the global economy and energy system and is used to analyze such topics as international negotiations on emissions-abatement agreements.

Weidlich and Veit reported sparse attention to validation and verifica-

_____________________

²¹Although framed in different terms, an insightful perspective on several of the issues discussed in this section was provided by the Congressional Research Service in a report on energy modeling; see annex.

tion issues in agent-based energy modeling as of the time of their review. Regarding the electricity system models specifically, they observe that “the scientific usefulness and academic contribution of large-scale [agent-based] models that integrate an enormous amount of details has not yet been proven” (2008, pp. 1750–1753). It is in addition worth noting that, to the present day, these large-scale models have not gained acceptance by regulators or policy makers.

Models of Individual Behavior

In addition to those market and system models, Weidlich and Veit (2008) note an example of agent-based energy modeling studies focused on consumer or household behavior. Ehlen et al. (2007) studied “dynamic” or “real-time” electricity pricing, which refers to retail (consumer) prices, also called tariffs, that vary over the course of one day (24 hours) to reflect the marginal cost of power generation at different times. Most residential electricity customers in the United States have tariffs (stipulating prices as a function of the level of consumption and its timing) that are uniform or “flat,” not reflecting the time of use (although in some parts of the country the tariffs incorporate a “block rate” price that varies with the level of consumption). However, the marginal generation cost is typically higher during daylight hours, when, for example, air-conditioning demand is highest (in some regions and climates). The purpose of dynamic pricing is to align marginal prices with marginal costs to increase the economic efficiency of electricity markets and the operation of the electric power system.

Economists have long advocated dynamic electricity pricing, and it is the subject of a theoretical and empirical literature on the behavior of consumers who face dynamic electricity prices and on how consumer preferences regarding energy consumption interact with their technological and economic environment to determine their patterns of electricity use. However, it has been difficult to implement in practice, largely because of consumer reluctance to adopt it voluntarily (that is, to use electricity under a dynamic tariff) or to accept its imposition. In recent years, it has gained increasing attention as a means of supporting the development of the “smart grid,” an electricity system that uses advanced information technology to improve operations and that incorporates advanced energy technology. For this among other reasons, understanding consumer adoption or non-adoption of dynamic electricity tariffs is an important and policy-relevant goal. The following paragraphs discuss in turn Ehlen et al. (2007) and a more recent study of the same topic.

The Effects of Adoption of Dynamic Pricing in an Electricity Market

Ehlen and colleagues use ABM to study the effects of real-time pricing on residential electricity consumption and in turn on the operation of the electricity market. Their framework comprises an ABM of residential, commercial, and industrial electricity users and a simulation model of the power-generation sector. A “demand aggregator” buys power from the generator and sells it to users, and the analysis focuses on the effects of households’ choice of pricing contract on the functioning of the electricity market.

Households’ hourly demand during the course of a day is divided into three categories: “optional,” which is of “relatively low consequence” and can be interrupted without households’ needing to “make it up,” such as power for lighting and television; “moveable” use, which is of “medium consequence” and can be shifted to other times, such as dish-washing or clothes-washing; and “immoveable” use, which is highly time-sensitive, such as food refrigeration or air conditioning during hot weather. Households have a desired mix of use represented by the percentage of their preferred use assigned to each category, which is fixed (numerically) a priori. Households face either uniform or real-time pricing contracts and have a fixed budget for electricity.

Households allocate their use throughout the day with the goal of consuming their desired mix among the categories. Under uniform pricing, that is straightforward. In contrast, at the beginning of a day, a household under dynamic pricing calculates the cost of its desired use and timing for that day under the hourly prices. If the cost exceeds the budget constraint, the household reallocates its use according to a “greedy scheduling algorithm,” a search-based optimization procedure. First, the movable use that is of highest cost is shifted to other, lower-cost hours; if the budget constraint is still exceeded, the next-highest-cost movable use is shifted, and so on. If this procedure does not result in a planned pattern of use that satisfies the budget constraint, optional use is curtailed.

Those calculations are embedded in a higher-level process of households’ choosing, or not, to adopt real-time pricing contracts and, once they are adopted, whether to stay with the new contract or revert to a uniform tariff. The model’s representation of these phenomena is justified as follows:

The details of how these assumptions are implemented are not clear from the paper’s description. It is stated both that each household is randomly assigned a probability of experimenting with real-time pricing if it has not already adopted one, and that “social networks are modeled explicitly” and affect the decision to adopt (or to stay with the new contract if the shift is made). The greater the number of existing adopters, the more likely that a household with a uniform contract will switch; conversely, the greater the number who revert to uniform, the more likely that a dynamic-tariff household will also revert.

The model simulations explore the relationships among tariff types, tariff switching, and consumers’ electricity use and the implications for the functioning of the modeled electricity market, including the profitability of demand aggregation. To briefly assess the model: First, whether the adoption decision is a “complex process” and whether or how the four factors noted above affect it are empirical questions (as is whether factors other than, or in addition to, those four are relevant). However, the authors do not cite or draw on the literature on dynamic-pricing adoption by households. Economists would readily agree that the first two factors are important. But Ehlen et al. provide no rationale for the particular structure—that is, mathematical—with which costs and the economic decision rules are incorporated into the model. That “willingness to experiment” is a factor that could be considered vacuously true, inasmuch as an unwilling household, by definition, will not adopt voluntarily. In any case, however, the claims that it is “likely a function of the current level of adoption” and that adoption is in part a “function . . . of social/cultural interactions with other households” are speculative; no evidence is provided to motivate or support them.

Second, the model’s representation of electricity demand and consumer choices as conditional on the tariff type is problematic. Actual households’ responses to dynamic pricing—including what end-use energy services are “optional,” “moveable,” and “immoveable”—are functions of occupants’ preferences, equipment, and costs. The quantitative measurement of consumer responses to dynamic pricing, including the effects of such a factor measured through detailed econometric (statistical) analysis (see, e.g., Borenstein, 2013, and the references therein), is part of the extant dynamic-pricing literature that is ignored by Ehlen et al. No source or explanation is given for the values of several basic parameters related to consumers’ decisions: their monthly budgets, the cost of switching, and the “contract transaction cost.”

Finally, the authors assert that their framework for consumer behavior has “sufficient fidelity to characterize households” (2007, p. 4) that have different characteristics and constraints of several types. But there is no discussion or demonstration of how or why the model’s structure results in fidelity. Moreover, although the simulations assume 10,000 residential households, the key calibration—that of the allocation of electricity consumption among different end-use types—is carried out with aggregate data on residential energy use in California, that is, statewide averages of consumption among all households. Thus, what is being analyzed is not a large number of truly heterogeneous agents, which is implicitly part of the basis of the claim of “fidelity,” but rather a large number of representative agents.

Analyzing Adoption of Dynamic Pricing

A second example is another ABM-based study, by Kowalska-Pyzalska et al. (2014), of dynamic electricity pricing, in this case households’ decisions of whether to switch to a dynamic tariff or pricing structure. They motivate their work on the basis of the importance of better understanding of the determinants of consumers’ decisions of whether to adopt dynamic pricing. They argue that, given both the high cost of empirical research on consumer adoption and the special capabilities of ABM for studying it, ABM is not merely an acceptable method but a preferred method for analyzing the adoption problem.

The paper describes the rationale for, structure of, and computational experiments with an ABM of customer adoption of dynamic pricing. The model is based on the idea that decisions are a function of the joint effects of influences on consumers’ attitudes and of the degree of indifference with which they regard the desirability of switching. The model is characterized as “hypothetical yet plausible” (p. 172). The paper contains an extensive background review of sociological and other research on the effects of consumer attitudes and social influences on behavior (including energy- and environment-related behavior); electricity use and ways of altering it; and consumer responses to dynamic electricity pricing.

The model itself takes the form of a “social system” represented by a square grid in which each cell corresponds to an agent, called a spin-son (or “spinson”) because of its basis in “spin models” in statistical physics. Each spinson has a value of either −1 (a preference for a flat tariff) or +1 (a preference for the dynamic). Initially, each spinson is assumed to have a flat tariff, but it may switch, and the dynamics of the switching behavior in the population is the focus of the simulation.

Notwithstanding the long background discussion mentioned above, the authors acknowledge that there is not “one established theory on how exactly a decision is made” (p. 168). Instead, referring informally to research

on the relationship of attitudes to behavior and the importance of attitude “stability,” they assert that “consumers have to be really convinced before making a decision” (p. 167). Spinsons’ attitudes to the dynamic tariff are either “positive” (favorable) or “negative” (unfavorable). Starting in state −1 (flat tariff), a positive opinion for a sufficient length of time will result in a switch; starting in state +1 (dynamic tariff), a negative opinion for a sufficient length of time will result in a switch back to the flat tariff.

The authors posit that a spinson’s opinion at any time is a function of three factors: The level of “indifference,” the degree of “conformity,” and “product features.” A parameter p ∈ [0,1] represents the level of indifference. A brief description seems to say that conformity is represented by defining a topology in which a spinson has “neighbors” and its opinion is affected by theirs. A parameter h ∈ [0,1] represents the degree of “influence.” A parameter t > 0 represents the time a spinson takes to make a decision. The numerical values assigned to p and to h are the same for all spinsons and represent “average [societal] values.”

Monte Carlo simulations are run for three values of t and four values of both p and h. The computational analysis entails the emergence of “clusters” of spinsons in the topology generated by the model as the number of simulations increases and the relationships between indifference and decisions.

The authors make several recommendations on the basis of their results, including the need to “communicate . . . the potential benefits of adoption,” “provide clear and full information to customers,” and “provid[e] enough incentives that will overcome the cost and discomfort when switching to a new tariff” (p. 173).

Because this work both is presented as an alternative to empirical research and claims to draw on and be supported by previous work in a number of disciplines, assessing it should begin with a close examination of the relationship between ABM and analysis and of the research cited to support it.

First, the basic rationale for dynamic pricing is that it will improve the economic efficiency of the electric power system by aligning retail electricity prices with marginal costs of generation, which vary over a 24-hour period. That is in contrast with “green” electricity pricing, which refers to tariffs that electricity consumers voluntarily pay to be supplied by renewable or low-carbon generation sources. A secondary rationale is the potential usefulness of dynamic pricing in facilitating the operation of the power system with increased renewable generation, but this is a complex technical topic that consumers might not easily grasp or relate to their own electricity use. Indeed, in one of the paper’s few cited studies on consumer views on dynamic (as opposed to green) pricing, only half the respondents indicated that they were motivated by environmental issues (Dütschke and Paetz,

2013). In another of the cited studies, only 10 percent of respondents were reported as saying that “environmental benefits played a key role” in their adoption decision (Paetz et al., 2012, p. 33). Yet another cited study reported that “environmental reasons were not motivating” customers of an American utility who were surveyed on their views on adopting dynamic pricing (Star et al., 2010, p. 265). Thus, not only is the work cited by the authors on green-pricing adoption of questionable relevance, but the cited work on dynamic pricing is contrary to the suggestion that “green” values or priorities among consumers are relevant to dynamic-pricing adoption.

Second, much of the other work that the authors cite focuses on the adoption of energy-efficient household equipment and on reducing energy use itself. However, dynamic pricing is not itself an “energy-saving” mechanism: Higher prices during hours in which electricity demand is relatively high are intended to reduce demand during these hours, but this may occur through consumers’ shifting consumption to off-peak periods rather than reducing their total electricity consumption during a 1-day cycle.²² That contrasts, for example, with a carbon-emission tax, which—if passed along to electricity consumers in the form of higher retail prices—would be expected to reduce overall energy consumption. Here again, one of the cited studies (Paetz et al., 2012) found that consumers emphasized monetary benefits as a motivation to adopt, with the understanding that the benefits would be achieved at least initially through changing behavior—that is, shifting their consumption—not necessarily through reducing consumption by purchasing energy-efficient technology or otherwise.

Similarly, as evidence of the importance of social influence on energy behavior, the authors cite recent work documenting that providing a household with comparative information on energy use—relative to neighbors, the community, and so on—can be effective in reducing use (Allcott, 2011; Ayres et al., 2013). But the likely normative underpinning of that effect—that energy saving is a worthy social and environmental goal—is not obviously present in the case of dynamic pricing, in which individual or household costs and benefits are primary factors. Moreover, Kowalska-Pyzalska et al. (2014, p. 169) themselves acknowledge that

__________________

²²The extent to which dynamic pricing would lower overall energy use depends on, for example, the local climate and the presence or absence of air conditioning in dwellings.

Turning to the modeling results, the researchers (Kowalska-Pyzalska et al., 2014, pp. 172–173) characterize their analysis as providing

One might first point out that, reflecting the discussion above, this opinion–participation discrepancy has been reported in the green-pricing, not dynamic-pricing, literature (e.g., Ozaki, 2011). Beyond that, the statement is at best difficult to interpret. That consumer opinions are “unstable” and that this affects adoption is mentioned previously in the paper as a finding of previous research (although neither of the papers cited mentions “opinion instability”). Here, however, the authors seem to be discussing their computational results. Moreover, the claimed causal link to “indifference” also seems to be intended as either a description or an explanation of the model results, inasmuch as “indifference” is one of the input factors in the model. Despite the connotation that the modeling has yielded an empirically meaningful finding, on the contrary “indifference” is here only a parameter label, not a defined or measured psychological state of some sort.

Finally, beyond the examples noted above, little of the work cited by the authors to support their modeling and analysis deals with dynamic-pricing adoption specifically (and none of it contributes quantitatively—for example, to parameterizing the ABM).²³ However, although the literature on this topic that is not cited by the authors is relatively small, it is not nonexistent. Moreover, it contains findings that are highly relevant to the ABM analysis in their study. For example, Baladi et al. (1998) found that volunteers for dynamic pricing were distinguished from nonvolunteers by their understanding of their own electricity use patterns and their belief in their ability to respond effectively to the new rates. Similarly, Ericson (2011) found that adoption was more likely among consumers who have home energy-management systems that would help them to adapt effectively to the dynamic rates. Lineweber (2011) reported survey results in which consumers rated “having more control over electricity use” and “reducing bills by avoiding peak use” (p. 95) as the most important motivations for adopting dynamic pricing. A recent consumer behavior study reported findings that included the importance of “opt-out” versus “opt-in” program designs and that the adopters’ primary motivation is financial,

_____________________

²³Among nearly 80 papers in the reference list, this writer counted three on dynamic-pricing adoption specifically (not including the authors’ cited previous work).

that is, reducing their own electricity expenditures (Todd et al., 2013). In part on the basis of such work, the types of actions and policies that the agent-based modelers recommend to increase adoption have in fact long been recognized and promoted, including customer-knowledge hurdles, the importance of effective information provision, and the need for more effective marketing and recruitment approaches (e.g., Barbose et al., 2004). That is, their modeling provides no new or useful practical information on encouraging adoption of dynamic pricing.

Discussion

The two studies just reviewed illustrate the epistemological issues discussed previously but also provide cautionary tales for ABM and its potential use in policy making. The extensive detail in the Ehlen et al. (2007) model is claimed to yield “high fidelity,” but no argument or evidence is provided to support the claim. Moreover, data used to populate and parameterize parts of the model are average values and are far from the level of resolution that would support the model’s level of mathematical detail. Those are examples of the “complexity and validity” problem. (Also, as noted, sources of several key parameter values are not given at all.)

By using only several or single values of parameters that are identified as having values that could lie anywhere within specified ranges, the Kowalska-Pyzalska model embodies unaddressed Knightian uncertainty. But other aspects of the model and the study are more troublesome. The researchers explicitly claim that their modeling can substitute for empirical research. However, notwithstanding extensive citations of other work that is claimed to support the model’s assumptions and structure, close examination reveals that little of that work is directly relevant to or provides evidence for the actual ABM. At the same time, existing work on dynamic pricing that bears directly on the model and the analysis is not cited, and some of it contains evidence counter to the assumptions of the model. Findings and policy recommendations that are generated by the ABM and presented implicitly by the modelers as “new” are on the contrary well known and discussed in the literature. All in all, this study not only provides stark evidence against the use of ABM in lieu of empirical research but illustrates the risks of using ABM for policy analysis.

FUNDAMENTAL MODEL UNCERTAINTY AND ROBUSTNESS

The introduction to this Appendix noted nuances associated with the term validation in computational energy modeling. An emergent view regarding modeling for regulatory purposes in general (not just energy-focused) is that traditional validation—in particular, as practiced in the

physical sciences—is impossible for models of the complex natural systems that are typically the objects of regulation. In this view, “evaluation” is the appropriate alternative (NRC, 2007, p. 3):

This description can be interpreted as an attempt to articulate a decision-theoretic approach to model evaluation in this domain, which has the potential to address the fundamental issues described above. It is a promising direction but has not been systematically operationalized in regulatory modeling (Sanstad, forthcoming). At the most basic level, for this concept of evaluation to be meaningful, confidence, understanding, strength, and limitation must be defined—a difficult task that for the most part remains to be carried out.²⁴

Nevertheless, a decision-oriented approach to modeling is a promising direction. A general framing is that a decision maker either has a model of a system that she or he does not fully trust or is faced with multiple models and does not know which is the correct or otherwise most credible or appropriate one. But the decision maker seeks to use the models, for example, to design policies that affect the system. Macroeconomists have developed and applied several frameworks for analyzing this general problem of “fundamental model uncertainty” (Brock et al., 2003, 2007; Hansen and Sargent, 2008). They differ technically, but they share a focus on “robustness analysis”: the study of how decision makers can make choices that will yield outcomes that are acceptable even if not necessarily optimal, given that the model used to predict the outcomes cannot be verified as being “true.” Several researchers have adapted those ideas to energy modeling (e.g., Cai and Sanstad, 2014; Loulou and Kanudia, 1999).

A complementary approach to robustness analysis—in effect addressing the Knightian uncertainty problem discussed earlier in this paper—has been developed and implemented by Lempert and colleagues (e.g., Lempert, Popper, and Bankes, 2003; Lempert, Bryant, and Bankes, 2008). (See also Dalal et al., 2013.) In this approach, the space of model solutions generated by using an entire set of equally plausible inputs is computationally generated and explored by using machine learning, visualization, and other

_____________________

²⁴It is important to point out that in contrast with the situation in energy modeling and social science–based and policy modeling more generally, validation, verification, and uncertainty quantification in computational modeling in the physical and engineering sciences have become active and productive topics of basic and applied research. Oberkampf and Roy (2010) is a recent and authoritative source on the subject.

techniques. The output space is analyzed to find “regions” of similar or equivalent policy-relevant outcomes with respect to criteria that reflect a decision maker’s preferences and that can be used to identify decisions (represented by input parameter values) that yield outcomes that are robust to the underlying (Knightian) uncertainty.

The idea of fundamental model uncertainty is in principle useful for assessing ABMs. As discussed above, there is a generic problem in ABM of specifying “plausible” agent decision rules when (possibly many) others might be equally justified. A model-uncertainty perspective could facilitate the explicit analysis of this problem and provide firmer grounding for, and increased credibility of, ABMs. This point is taken up in the next section.

LESSONS FOR AGENT-BASED MODELING AND RECOMMENDATIONS FOR REGULATORY APPLICATIONS

In its present stage of development, ABM is a heterogeneous field in many respects, including the level of attention paid among its sub-disciplines to empirical foundations, validation, and uncertainty quantification. As discussed by, for example, Fagiolo et al. (2007) and Windrum et al. (2007), work on such issues is active in some quarters of the ABM community. Bianchi et al. (2007, 2008) describe a rigorous validation and empirically grounded calibration analysis of the agent-based Complex Adaptive Trivial System economic model of financial and capital productivity dynamics in a population of firms. The wider use in ABM of the types of methods that they describe could facilitate rigorous quantification and analysis of, for example, the “bias–variance” trade-off mentioned above and the relationship between complexity and uncertainty in ABMs. However, as the dynamic-pricing examples discussed above demonstrate, concern with such issues is not universal among ABM modelers. (Those examples also demonstrate that ABMs cannot be excused from scrutiny only because they contain a high level of detail or generate “interesting” results.²⁵)

Lessons

Three key lessons can be drawn from the discussion of energy modeling in this paper. We first note that much of ABM is clearly “calibrationist”: it focuses on the computational implications of sets of assumptions rather than on the validity of the assumptions themselves. Unlike the case of standard economic modeling, however, there is for the most part no general, underlying theoretical framework to guide the specification of

_____________________

²⁵See also McNamara et al. (2011) for a critical review of social and behavior modeling, including ABM, specifically for defense applications.

the assumptions. So rather than being—in the phrase of Dawkins et al., 2001—“numerical implementations of theoretical structures” (p. 3672), ABMs in many cases are numerical implementations of particular assumptions, which are justified by being plausible or interesting. Thus, in contrast with calibrated economic models, ABMs with some exceptions cannot draw on a large body of knowledge about and insight into a theoretical core of ideas regarding the behavior of agents and the aggregate consequences of their interactions. Those considerations raise the question of whether the computational findings of any given ABM are just idiosyncratic.

Second, although, as discussed, energy modeling is mostly organized around temporal scenario analysis whereas ABM is generally atemporal, there is a basic formal equivalence between the two methods: the logic and structure of energy-scenario modeling are quite similar to those of ABM simulation. The ABM household-electricity examples illustrate that point: The simulations are essentially scenarios of agents’ collective behavior. Just as energy modelers disclaim that scenarios are actually predictions, Kowalska-Pyzalska (2014), for example, characterized their model results as “hypothetical but plausible.” This again reflects the calibrationist philosophy of focusing on the consequences of underlying assumptions rather than on the assumptions themselves. Thus, like energy modeling, ABM involves a high degree of Knightian uncertainty by virtue of typically not exploring entire plausible input parameter spaces. The characteristic computational intensity of ABM reflects the computational demands of simulating the behavior of agent populations given specific values of key input parameters.

Third, the identification of a high level of model detail or “resolution” with validity or verisimilitude is a hallmark of ABM. However, as in energy modeling, in the absence of explicit criteria for validity, there are no particular theoretical or empirical grounds for this belief that high level of detail entails high level of validity; this is illustrated by the Ehlen et al. (2007) model. Such criteria are generically lacking in ABM. As a consequence, ABM is particularly vulnerable to the illusion-of-precision problem, that is, incorporating great detail that may be only for detail’s sake.

Recommendations

As in energy modeling, there is no well-developed, systematic, general method for validating or evaluating ABMs, nor do there appear in general to be topic-specific techniques that might be used to assess ABM for regulatory applications, including applications for tobacco. But several basic questions should be addressed in considering such applications.

First, in a given model, where do the rules governing agents’ actions

come from? What other equally plausible rules might be used? Why are the ones chosen better justified or more informative than others?

Second, what are the sources of numerical parameters and other model inputs? It has been claimed that an advantage of ABM is that it enables meaningful simulations to be conducted with limited or even no data. But any computational model requires the use of actual numerical values for inputs, which should be well documented and fully justified. The electricity (dynamic-pricing) examples discussed in this paper illustrate the problem.

In this regard, as noted previously, sensitivity analysis can provide useful information about the behavior of a model and can help in interpreting its quantitative output. But it is not a model validation technique and is not a means of justifying particular values for model inputs. A finding that model outputs are relatively insensitive to particular input choices, for example, is instructive but should not be taken as evidence about model validity one way or the other.

The two previous points are illustrated in an ABM of addictive behaviors developed by Moore et al. (in press, Table 1). A “discussion of choice for value” is presented for eight model parameters. For all but one, the justification is based partially or wholly on results of sensitivity analysis.

These issues related to ABM parameter choice and sensitivity analysis and the problem of Knightian uncertainty in ABM indicate the potential usefulness in ABM of the scenario discovery and robustness approach of Lempert and colleagues (e.g., Lempert, Popper, and Bankes, 2003; Lempert, Bryant, and Bankes, 2008). In the Kowalska-Pyzalska and Moore et al. models, each key parameter is associated with a plausible range, but the simulations are based primarily on the use of a single default value or a small number of default values chosen from those ranges. The Lempert et al. methodology of full analysis of model simulations across such ranges would potentially be effective in addressing the parameter choice, sensitivity, and Knightian-uncertainty problems in ABM.

The next question is, what is the intended use of the model output? If it is to provide qualitative insight, careful consideration should be given to whether the insight is into the empirical phenomenon of interest rather than only into the workings of the model itself, including its particular structure and assumptions. That is especially important if the model will be used in establishing a quantitative regulatory rule or criterion. In this case, however, an explicit accounting should be made of whether and how the quantitative validity of the model has been demonstrated sufficiently to support this application.

Finally, what are the potential consequences for the regulatory process and its outcomes if the model is wrong? Notwithstanding the manifold difficulties involved in model assessment, validation, or evaluation, this

question can and should be addressed by regulators by drawing on their domain-specific knowledge—and common sense.

An example of what can happen if a computational model is wrong is provided by a type of energy modeling not discussed in this paper: real-time and 1-day-ahead electricity modeling that is used in power-system management. In such operational (as opposed to long-range, policy-focused) applications, modeling errors can result in power blackouts.²⁶ In tobacco regulation, for example, what might be the public health consequences of a modeling error?

CONCLUSION

ABM is an expanding field of social science research and is increasingly considered for use in applied policy analysis and regulation. Thus, it is important for agent-based modelers and their potential constituents to address epistemological issues, such as defining and evaluating model validity, quantifying model uncertainty, and understanding how ABMs should be assessed for use in decision making.

Such issues have a long history in computational energy modeling. This paper is based on the premise that, notwithstanding the differences between the two fields of ABM and computational energy modeling, energy modeling can provide valuable insight and guidance for ABM. It reviewed energy modeling methods, applications, and epistemology, particularly the issues of calibration and its relationship to Knightian uncertainty, the relationship between complexity and model validity, and questions that arise when computational modeling is used for public decision making. It also analyzed several examples of agent-based energy modeling. This review and analysis were then used to draw lessons on validation and uncertainty quantification in ABM and as a basis for recommending guidelines for assessing ABM for regulatory use. It is hoped that the discussion in this paper will contribute to advancing rigorous and policy-relevant assessment of ABM.

__________________

²⁶Here error is used not in a statistical sense but rather to refer to, for example, model misspecification.

Appendix C Annex

A BRIEF HISTORY OF ENERGY-MODEL ASSESSMENT

Through the 1960s, much of “energy modeling” was econometric or statistical regression analysis and short-term forecasting, often applied to energy demand—for example, by electric utility planners. Computational energy modeling became predominant during the 1970s, coinciding with the emergence of energy issues as major public-policy priorities. Environmental effects of energy production and use had been brought to public attention by energy industry experts, scientists, and activists, and the first “oil crisis” sparked by conflict in the Middle East had resulted in extreme concern regarding fuel supplies. “Energy independence” attained high policy and political priority, and one of the first major examples of computational energy modeling during that era was a linear program created for Project Independence, which was initiated by President Nixon in 1973 (Hogan, 1975).

Coinciding with both the continuing attention to energy issues and the policy and regulatory philosophy and priorities of the Carter administration, energy modeling quickly broadened and expanded in number of models, practitioners, and specific applications. Technical work was accompanied by serious, sustained attention to methodological issues, particularly validation, evaluation, and uncertainty quantification. The following are examples. A 1978 bibliography on validation in social science–based and policy modeling, including energy modeling, contained more than 700 entries (Gruhl and Gruhl, 1978). The Energy Modeling Forum (EMF) was created at Stanford University in 1976 “to provide a structural framework within which energy experts, analysts, and policymakers could meet to improve their understanding of critical energy problems” (EMF, 1988, p. i).²⁷ Concurrently, a model-analysis project was established at the Massachusetts Institute of Technology to conduct rigorous third-party validation of energy-model codes (MIT, 1978). Summary reports of studies organized by the National Bureau of Standards around 1980 reveal a diverse group of scholars and practitioners grappling with fundamental and challenging issues in energy-model evaluation, validation, and uncertainty quantification (e.g., NBS, 1980).

By the end of the 1980s, however, academic and government interest in and resources devoted to energy-model evaluation and related topics had attenuated, following the end of that era of “energy crises” and a politi-

_____________________

²⁷The EMF pioneered systematic, quantitative model comparison and continues as the world’s leading center for such activities, playing an important and influential role in energy modeling and policy.

cal shift away from energy regulation and policy at the national level. But energy modeling itself continued in the government, national laboratories, and universities, and in the succeeding decades it steadily expanded and increased in importance for policy applications. As noted in this paper, it has come to dominate quantitative energy-policy analysis, including the subject of carbon-emission abatement.

A 2008 study by the Congressional Research Service (CRS) is a notable exception to the contemporary paucity of attention to energy model validation. It was prepared as background on energy modeling for U.S. Senate hearings on prospective GHG-reduction legislation (Parker and Yacobucci, 2008). At the Senate’s request, six energy-modeling studies of potential long-run—to the year 2050—costs and benefits of the proposed policies were conducted. CRS observed that

CRS does allow that “despite models’ inability to predict the future, cases examined here do provide insights on the costs and benefits [of the proposed legislation]” (p. i). The report also quoted a 1990 CRS study of the potential effects of the prospective “cap and trade” program of sulfur dioxide emission from electric power plants:

That is a rare but frank and accurate assessment of the limitations of energy modeling for long-run policy analysis, its primary application. It is especially noteworthy that these conclusions were reached by analysts who were reporting to Congress, a primary constituency for these models’ outputs.²⁸

__________________

²⁸In keeping with the federal budgeting process, Congress has been a primary funder of energy modeling, through executive agency budgets and research grants to universities and national laboratories.

REFERENCES

Allcott, H. 2011. Social norms and energy conservation. Journal of Public Economics 95(9–10):1082–1095.

Ayres, I., S. Raseman, and A. Shih. 2013. Evidence from two large field experiments that peer comparison feedback can reduce residential energy usage. Journal of Law, Economics, and Organization 29(5):992–1022.

Baladi, S. M., J. A. Herriges, and T. J. Sweeney. 1998. Residential response to voluntary time-of-use electricity rates. Resource and Energy Economics 20(3):225–244.

Barbose, G., C. Goldman, and B. Neenan. 2004. A survey of utility experience with real time pricing. Berkeley, CA: Lawrence Berkeley National Laboratory.

Barton, D. C., E. D. Eidson, D. A. Schoenwald, K. L. Stamber, and R. K. Reinert. 2000. Aspen-EE: An agent-based model of infrastructure interdependency. Albuquerque, NM: Sandia National Laboratories.

Bianchi, C., P. Cirillo, M. Gallegati, and P. A. Vagliasindi. 2007. Validating and calibrating agent-based models: A case study. Computational Economics 30(3):245–264.

———. 2008. Validation in agent-based models: An investigation on the CATS model. Journal of Economic Behavior & Organization 67(3–4):947–964.

Borenstein, S. 2013. Effective and equitable adoption of opt-in residential dynamic electricity pricing. Review of Industrial Organization 42(2):127–160.

Brock, W. A., S. N. Durlauf, and K. D. West. 2003. Policy evaluation in uncertain economic environments. Brookings Papers on Economic Activity 1:235–322.

———. 2007. Model uncertainty and policy evaluation: Some theory and empirics. Journal of Econometrics 136(2):629–664.

Burnham, K. P., and D. R. Anderson. 2002. Model selection and multimodel inference: A practical information-theoretic approach. 2nd ed. New York: Springer-Verlag.

Cai, Y., and A. Sanstad. 2014. Model uncertainty and energy technology policy: The example of induced technical change. RDCEP Working Paper No. 14-01.

Clarke, L., J. Edmonds, H. Jacoby, H. Pitcher, J. Reilly, and R. Richels. 2007. Scenarios of greenhouse gas emissions and atmospheric concentrations: Sub-report 2.1a of synthesis and assessment product. Washington, DC: Climate Change Science Program and the Subcommittee on Global Change Research. Department of Energy, Office of Biological & Environmental Research.

Dalal, S., B. Han, R. Lempert, A. Jaycocks, and A. Hackbarth. 2013. Improving scenario discovery using orthogonal rotations. Environmental Modelling & Software 48:49–64.

Dawkins, C., T. N. Srinivasan, and J. Whalley. 2001. Calibration. In Handbook of econometrics. Vol. 5, edited by J. J. Heckman and E. Leamer. Amsterdam, Netherlands: Elsevier. Pp. 3653–3703.

Dütschke, E., and A. G. Paetz. 2013. Dynamic electricity pricing—which programs do consumers prefer? Energy Policy 59:226–234.

Ehlen, M. A., A. J. Scholand, and K. L. Stamber. 2007. The effects of residential real-time pricing contracts on Transco loads, pricing, and profitability: Simulations using the N-ABLE™ agent-based model. Energy Economics 29(2):211–227.

EIA (Energy Information Administration). 2009. The national energy modeling system: An overview. Washington, DC: Energy Information Administration, Office of Integrated Analysis and Forecasting, U.S. Department of Energy.

———. 2012. Analysis of the Clean Energy Standard Act of 2012. Washington, DC: Energy Information Administration, U.S. Department of Energy.

———. 2014. Annual energy outlook 2014 with projections to 2040. Washington, DC: Energy Information Administration, U.S. Department of Energy.

EMF (Energy Modeling Forum). 1988. North American natural gas markets: EMF report 9. Palo Alto, CA: Energy Modeling Forum, Stanford University.

EPA (U.S. Environmental Protection Agency). 2009. EPA analysis of the American Clean Energy and Security Act of 2009: H.R. 2454 in the 111th Congress. Washington, DC: U.S. Environmental Protection Agency, Office of Atmospheric Programs.

———. 2010. Documentation for EPA base case v.4.10 using the integrated planning model. Washington, DC: U.S. Environmental Protection Agency, Clean Air Markets Division.

Ericson, T. 2011. Households’ self-selection of dynamic electricity tariffs. Applied Energy 88(7):2541–2547.

Fagiolo, G., A. Moneta, and P. Windrum. 2007. A critical guide to empirical validation of agent-based models in economics: Methodologies, procedures, and open problems. Computational Economics 30(3):195–226.

Gerst, M. D., P. Wang, A. Roventini, G. Fagiolo, G. Dosi, R. B. Howarth, and M. E. Borsuk. 2013. Agent-based modeling of climate policy: An introduction to the engage multi-level model framework. Environmental Modelling & Software 44:62–75.

Goettle, R. J., M. S. Ho, D. W. Jorgenson, D. T. Slesnick, and P. J. Wilcoxen. 2007. IGEM, an Inter-temporal General Equilibrium Model of the U.S. economy with emphasis on growth, energy and the environment. Prepared for the U.S. Environmental Protection Agency, Office of Atmospheric Programs, Climate Change Division.

Goldstein, G. A., L. A. Goudarzi, P. Delaquil, and E. Wright. 2008. NE-12 MARKAL final report structure, data and calibration: Final deliverable to Northeast States Center for a Clean Air Future under IRG contract. New York: International Resources Group.

Gruhl, J., and N. Gruhl. 1978. Methods and examples of model validation: An annotated bibliography. Cambridge, MA: MIT Energy Laboratory Working Paper MIT-EL 78–022 WP.

Hansen, L. P. and T. J. Sargent. 2008. Robustness. Princeton, NJ: Princeton University Press.

Hogan, W. W. 1975. Energy policy models for Project Independence. Computers & Operations Research 2(3–4):251–271.

Intergovernmental Panel on Climate Change. 2000. Special report: Emissions scenarios—Summary for policymakers. Cambridge, UK: Cambridge University Press.

Knight, F. 1921. Risk, uncertainty, and profit. Chicago, IL: University of Chicago Press.

Koritarov, V. S. 2004. Real-world market representation with agents: Modeling the electricity market as a complex adaptive system with an agent-based approach. IEEE Power & Energy Magazine 2(4):39–46.

Kowalska-Pyzalska, A., K. Maciejowska, K. Suszczyński, K. Sznajd-Weron, and R. Weron. 2014. Turning green: Agent-based modeling of the adoption of dynamic electricity tariffs. Energy Policy 72:164–174.

Lashof, D. A., S. Yeh, D. Doniger, S. Carter, and L. Johnson. 2013. Closing the power plant carbon pollution loophole: Smart ways the Clean Air Act can clean up America’s biggest climate polluters. New York: National Resources Defense Council.

Lempert, R. J., S. W. Popper, and S. C. Bankes. 2003. Shaping the next one hundred years: New methods for quantitative, long-term policy analysis. Santa Monica, CA: RAND Corporation.

Lempert, R. J., B. P. Bryant, and S. C. Bankes. 2008. Comparing algorithms for scenario discovery. Santa Monica, CA: RAND Corporation.

Lineweber, D. C. 2011. Understanding residential customer support for—and opposition to—smart grid investments. The Electricity Journal 24(8):92–100.

Loulou, R., and A. Kanudia. 1999. Minimax Regret strategies for greenhouse gas abatement: Methodology and application. Operations Research Letters 25(5):219–230.

McNamara, L. A., T. G. Trucano, and C. Gieseler. 2011. Challenges in computational social modeling and simulation for national security decision making. Fort Belvoir, VA: Defense Threat Reduction Agency, Office of Strategic Research and Dialogues.

MIT (Massachusetts Institute of Technology). 1978. Independent assessment of energy policy models: Two case studies. Cambridge, MA: MIT Energy Laboratory, MIT Model Assessment Group.

Moore, T. W., P. D. Finley, B. J. Apelberg, B. Ambrose, N. S. Brodsky, T. J. Brown, C. Husten, R.J. Glass. In press. An opinion-driven behavioral dynamics model for addictive behaviors. European Physical Journal B.

NBS (National Bureau of Standards). 1980. Validation and assessment issues of energy models: Proceedings of a workshop held at the National Bureau of Standards, Gaithersburg, MD. Washington, DC: National Bureau of Standards, National Engineering Laboratory, U.S. Department of Commerce.

Nordhaus, W. 2008. A question of balance: Weighing the options on global warming policies. New Haven, CT: Yale University Press.

NRC (National Research Council). 1992. The national energy modeling system. Washington, DC: National Academy Press.

———. 2007. Models in environmental regulatory decision making. Washington, DC: The National Academies Press.

Oberkampf, W. L. R. and C. J. Roy. 2010. Verification and validation in scientific computing. New York: Cambridge University Press.

Ozaki, R. 2011. Adopting sustainable innovation: What makes consumers sign up to green electricity. Business Strategy and the Environment 20(1):1–17.

Paetz, A.-G., E. Dütschke, and W. Fichtner. 2012. Smart homes as a means to sustainable energy consumption: A study of consumer perceptions. Journal of Consumer Policy 35(1):23–41.

Parker, L., and B. Yacobucci. 2008. Climate change: Costs and benefits of S. 2191. CRS report for Congress RL34489. Washington, DC: Congressional Research Service.

Ross, M. T. 2005. Documentation of the Applied Dynamic Analysis of the Global Economy (ADAGE) model: Working paper 05-01. Research Triangle Park, NC: RTI International.

Sanstad, A. Forthcoming. Abating carbon dioxide emissions from electric power generation: Model uncertainty and regulatory epistemology. Journal of Legal Studies.

Star, A., M. Isaacson, D. Haeg, L. Kotewa, and C. Energy. 2010. The dynamic pricing mousetrap: Why isn’t the world beating down our door? In ACEEE summer study on energy efficiency in buildings. Pp. 257–268.

Todd, A., P. Cappers, and C. A. Goldman. 2013. Residential customer enrollment in time-based rate and enabling technology programs: Smart grid investment grant consumer behavior study analysis. Berkeley, CA: Lawrence Berkeley National Laboratory.

Weidlich, A., and D. Veit. 2008. A critical survey of agent-based wholesale electricity market models. Energy Economics 30(4):1728–1759.

Windrum, P., G. Fagiolo, and A. Moneta. 2007. Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation 10(2):8.

This page intentionally left blank.