Undiscovered Oil and Gas Resources: An Evaluation of the Department of the Interior's 1989 Assessment Procedures (1991)

In preparing this report, the committee undertook to provide the USGS and the MMS with concrete suggestions for improving their future estimates of undiscovered oil and gas volumes. Such estimates may be more important now than at any time since the energy crises of the 1970s. With the recent upheaval in the Persian Gulf, domestic pressure to decrease the nation's reliance on imported oil and to produce more oil from U.S. sources will probably escalate. Reliable resource assessments are critical for gauging the extent to which the United States can depend on undiscovered domestic petroleum for its future energy supply.

Estimating how much oil and gas remains to be discovered is necessarily an inexact process. Without actually drilling, one cannot know precisely what volume of petroleum a prospective reservoir contains. Nevertheless, new assessment methods developed since the early 1970s have the ability to increase the reliability of resource assessments if properly employed. These new methods, however, have also increased the complexity of assessment procedures. Today, conducting a successful assessment requires expertise not only in petroleum geology, but also in statistics. Because of the complex, interdisciplinary nature of resource assessments, outside review of assessment procedures is critical to ensuring that evaluators have correctly applied the most up-to-date geological and statistical appraisal methods.

This chapter summarizes the committee's most important recommendations related to the 1989 assessment. This summary does not cover every finding and recommendation from Chapter 3. Instead, it focuses on the committee's key conclusions—those most important to the assessment's overall quality. For this summary, the committee has divided its recommendations into five areas:

assessment boundaries, USGS and MMS management, geological approaches in data base use and play analysis, statistical methods, and assessment results.

The committee found that the assessment's limitation to “conventional” resources was unnecessary in some cases. While the types of oil the assessors classified as unconventional generally contribute little to the nation's energy supply, some types of natural gas labelled unconventional represent a significant fraction of domestic production. For example, natural gas produced from low-permeability sandstones, fractured shale, and coal beds amounts to about 1.5 to 2 trillion cubic feet per year. The failure to include estimates of unconventional natural gas in addition to the estimates of conventional natural gas obscures a significant portion of the potential domestic energy supply.

The committee also found that the assessment's limitation to recoverable resources made the estimates unnecessarily sensitive to future changes in recovery technology. There is a continual flux of resources from the “unrecoverable” category to the “recoverable” category, as advances in reservoir characterization, drilling, and completion increase the amount of in-place petroleum that can be produced. Thus, a more thorough assessment would include estimates of in-place resources in each play, which could be used to judge the potential for new recovery technology to increase petroleum production.

Recommendation: The DOI should include estimates of natural gas from unconventional sources (separate from the estimates of conventional natural gas) in future resource assessments.

Recommendation: The DOI should include estimates of in-place resources in future assessments. Assessors should estimate each play's in-place resource volume first, and then calculate the recoverable resource volume by applying a recovery factor to the in-place value.

To be done well, national-scale resource assessments demand a major commitment of funding and professional expertise. Credible resource assess-

ments also require continuity of effort. Continued funding and effort are necessary to improve the coverage and quality of data bases and to translate the newest theoretical research on assessment methods into well-tested modifications of actual assessment procedures. Sporadic application of intensive effort just prior to a national resource assessment is unlikely to result in an outcome that achieves a uniform standard of excellence in all dimensions by which such assessments are judged. The soundness of assessment procedures, the caliber of their execution, and the credibility of their results are thus linked to the fashion in which the program for accomplishing the assessment is managed.

The committee found that, as of the time of the 1989 assessment, the USGS management had provided insufficient manpower, funds, and incentives to carry out a national oil and gas assessment at a uniform level of excellence in all dimensions. The USGS, unlike the MMS, does not maintain a permanent resource assessment group. The absence of a permanent assessment group created the following problems:

• a lack of staff experience in resource assessments and insufficient knowledge of regional geology in some provinces;

• a distribution of personnel among the provinces that did not correlate with the provinces' potential for containing undiscovered oil and gas or with the amount of data available for analysis, with a concentration of attention on the Rocky Mountains, California, and Alaska and a comparatively low level of effort in the Gulf of Mexico, Midcontinent, and Illinois Basin areas;

• the lack of a clear explanation of how the 1989 assessment methodology compared to methodologies used in previous assessments;

• an absence of a clearly defined assessment procedure that was unambiguously understood by all members of the assessment team;

• insufficient statistical support for data analysis; and

• a lack of provision for improving assessment methodologies or testing the sensitivity of the methodologies to critical input parameters.

The MMS avoided some of these problems by maintaining a permanent resource assessment group. A permanent assessment group is a necessity for the MMS, since it is required to report undiscovered oil and gas estimates for the Outer Continental Shelf to Congress every two years. Nevertheless, the committee is concerned that funding cutbacks caused by moratoria on lease sales may cause the MMS to reduce personnel levels. As documented by the problems

the USGS encountered because of its lack of a permanent resource assessment team, cutbacks at the MMS could have a negative impact on future assessments.

Recommendation: The USGS should establish a group of specialists within its offices to design and implement on an ongoing basis a program for improving oil and gas assessment methodologies. In particular, this permanent assessment group should emphasize data validation, training of geologists in assessment methods, and more aggressive use of modern statistical methods in the assessment process.

Recommendation: Managers at the USGS and the MMS should establish complete, continued cooperation between the assessment groups at the two agencies.

The committee examined two aspects of the assessment's geological framework: (1) the adequacy of the geological data bases available to the assessors and (2) the formulation and analysis of plays.

Geological data bases provide the foundations for resource assessments. Without good data, the most advanced statistical methods are powerless. The committee found weaknesses in the data bases used by both the USGS and the MMS.

After reviewing the USGS's data, the committee found that the USGS lacked adequate seismic data for the lower 48 states and also for Alaska's North Slope. In addition, the committee found that the information compiled by USGS assessors into basin reports varied in scope and quality, and that the preparation of this information did not occur for all basins as an established, early part of the assessment process.

From a review of the MMS's data, the committee concluded that while the MMS's seismic data base was relatively complete, there were gaps in the MMS's geochemical data. The committee also concluded that the MMS did not make

adequate use of potential data from key industry exploratory wells in Alaskan offshore basins and in the Atlantic's Baltimore Canyon.

Recommendation: Both the USGS and the MMS should conduct a data audit. The audit would have three purposes: (1) to evaluate the accuracy and completeness of the data; (2) to identify areas where the data base requires improvements; and (3) to provide explicit measures of the data quality to assessment users.

Recommendation: Based on the results of the data evaluation, both the USGS and the MMS should attempt to expand the available data for areas where their present data bases are incomplete. Better use of existing data bases should precede the creation of extensive new data bases. The agencies should seek data from outside sources like state geological surveys, state regulatory agencies, other federal agencies, and the private sector. For example, it is possible that the USGS could obtain some degree of access to proprietary seismic lines from industry in key areas.

Recommendation: To expand its geochemical data base, the MMS should ensure that full geochemical evaluations are conducted for wells drilled in offshore areas where existing data are inadequate.

Fundamental to modern oil and gas assessment methods is the proper grouping of prospects into plays. When plays are defined improperly —when diverse depositional systems are combined in one play—the result is play mixing. Play mixing can result in inaccurate statistical characterization of the play's prospects. Inaccurate statistics, in turn, may lead to an inaccurate assessment of the play's potential to contain undiscovered resources. The most likely result of play mixing is the underestimation of undiscovered resource volumes.

Also important in play analysis is the consideration of conceptual plays: those that do not contain discoveries or reserves, but that geological analysis indicates may exist. Analyzing conceptual plays is especially important for assessing natural gas, because natural gas exploration is less mature than oil exploration. Like play mixing, the failure to consider conceptual plays results in an underestimation of resource volumes.

The committee found that the USGS created some excessively large and lithologically diverse plays, leaving open the potential for play mixing. Large plays were most notable in the Gulf Coast and mixed plays were most notable in the Permian Basin, both important provinces. The committee also found that the USGS did not provide adequately for conceptual plays. The evidence of play mixing and incomplete consideration of conceptual plays indicates that the 1989 resource estimates may have been too low.

The USGS and the MMS used different procedures for delineating plays. The MMS's plays consisted either of a summation of prospects or a set of broad stratigraphic subdivisions, as in the Pacific Coast region. Thus, the committee concluded that the MMS's play delineation procedure created even more potential for play mixing and the consequent underestimation of resource volumes than the USGS's procedure. Part of the reason for the MMS's imprecision in play delineation was a lack of exploration and thus a shortage of drilling data everywhere except in the Gulf of Mexico. However, even in the Gulf, plays included a wide range of geologic settings, lithologies, stratigraphies, depositional systems, and structural characteristics, making the mixing of plays unavoidable.

As with the USGS, the committee found that the manner in which the MMS defined conceptual plays did not result in the proper identification and categorization of such plays and thus missed their potential contribution to the overall resource endowment. In Alaskan waters, the limited number of conceptual plays was inconsistent with the region's early development stage—a stage at which conceptual plays should be at a maximum.

Recommendation: The USGS should analyze play content to determine the impact of play formulation—especially formulation of plays with diverse depositional systems—on the resource volumes reported in the 1989 assessment. In future assessments, the USGS should avoid creating excessively large or geologically diverse plays.

Recommendation: The MMS should define plays more carefully to avoid the mixing of diverse geological and reservoir engineering characteristics in one play. Plays should not be simply summations of prospects. The MMS should recognize that the availability of an extensive seismic data base could lead toward plays excessively dependent on recognition of structural traps. Once it has formulated plays appropriately, the MMS should institute testing to ensure that play mixing does not significantly alter resource estimates.

Recommendation: Both the USGS and the MMS should incorporate more conceptual plays in future assessments. Analyzing conceptual plays may require that the agencies develop assessment techniques different from those used for known plays.

Statistical methods, modulated by expert judgement, play a central role in translating geological, geochemical, and geophysical data into projections of undiscovered oil and gas resources. A scientifically credible assessment program must be flexible enough to adapt statistical methods to changes in the quantity and scope of available data. Consequently, a credible assessment program should vigorously embrace two data analytic activities: data validation and model validation. Equally important, a successful assessment program must train participating geologists and geophysicists in statistical methods.

The USGS's attempt to incorporate play analysis into the 1989 assessment, despite the initial difficulties it experienced in implementing this concept, is commendable. However, the program of statistical work leading up to the assessment had four significant weaknesses:

1. The USGS's method for extrapolating discovery data to determine the number and size distributions of undiscovered fields in a play relied principally on subjective judgment. Extrapolations were based on observations about the way the shape of the undiscovered field-size distribution changes with time—observations obtained by fitting sizes of discovered fields to truncated shifted Pareto distributions (see Chapter 2, under Assessment Methods, for an explanation of this procedure). Where a moderate to large discovery history is available, more up-to-date, objective discovery-process models could have provided projections of undiscovered oil and gas in plays unencumbered by judgmental uncertainties that accompany subjective extrapolations. Objective discovery-process models could also have provided a benchmark for the use of expert opinion.

2. With the exception of reserve growth studies, objective statistical methods were not applied as aids in analyzing and interpreting data and plays. For example, the committee uncovered little evidence that the USGS used currently available statistical methods for partitioning data into descriptively homogeneous subsets (here, subsets composed of distinct individual play data).

3. No formal statistical analysis of data was employed to support or reject fundamental assumptions about probabilistic independence of play attributes like “adequate timing,” “existence of migration paths,” and “existence of reservoir.” The USGS reported no tests to determine how sensitive its undiscovered oil and gas projections are to this independence assumption.

4. The USGS assumed incorrectly that the number and size distribution of fields remaining undiscovered are statistically independent. Models of the petroleum discovery history show that the sizes of undiscovered fields in a play are somewhat related to the number of undiscovered fields in the play. As the number of undiscovered fields decreases (i.e., as more fields are discovered), the average size of remaining fields decreases, because large fields tend to be discovered first.

These four problems with the USGS's statistical work signal a major gap: the absence of a well-structured portfolio of statistical activities designed to support oil and gas resource assessments.

Though the MMS has a permanent staff devoted to resource assessments, the committee pinpointed weaknesses in its statistical methods, too. Like the USGS, the MMS assumed variables were probabilistically independent more often than was justified. In delineating plays in the Atlantic offshore, for example, the MMS assumed that structure, size, porosity, and depth are independent. In calculating sizes of prospective fields, it assumed area, thickness, and recovery are independent parameters. And in simulating prospect drilling, it assumed that individual prospect outcomes are mutually independent conditional on the presence of at least one field. Without conducting statistical tests to verify these assumptions of independence, it is impossible to conclude definitively whether the assumptions are valid.

In addition to overlooking these possible probabilistic dependencies, the MMS may have unintentionally imposed economic constraints on calculations of technically recoverable resources. For example, in Alaska the MMS excluded from its technically recoverable resource estimate prospects that were smaller than one-half a leasing block. This exclusion contains an implicit economic assumption: that prospects smaller than one-half a leasing block are too small to yield profits. When explicit economic screens are applied to technically recoverable resource volumes that were calculated with implicit economic assumptions, the result may be unintended double discounting. Such double discounting may have extended to the MMS's calculation of area risk levels (i.e., the chance that a given area contains petroleum). The committee judged that

area-level risks in many regions in the Atlantic and Alaska were too high, possibly because of unintended but implied economic constraints. In these areas, risk was as high as 98 percent prior to any economic screening (i.e, the MMS assumed there was only a 2 percent chance that the area contained petroleum).

In addition to the limitations of the USGS's and MMS's statistical methods, the committee found problems with the way expert opinion was guided and articulated. In both agencies, the interpretation of important probability terms varied among geologists. For example, different geologists defined terms such as “minimum” and “maximum” possible values and “.05 fractiles” and “.95 fractiles” differently. Also, both agencies lacked standards to guide assessors in assigning play risk (the chance that a play contains petroleum). The lack of such standards leaves open the question of whether the risk numbers were systematically too high or too low. Finally, in both agencies the diversity of opinion among geologists within groups assessing play and province resources was masked by presenting only numbers that represented a consensus opinion.

Recommendation: The USGS should consider the use of objective discovery-process models wherever possible, taking into account the necessity to incorporate within these models the effect of new ideas and technologies on the discovery process.

Recommendation: To avoid unintended double discounting, the MMS should develop methods for separating technically recoverable resource calculations from those that determine the volume of economically recoverable resources.

Recommendation: Both the USGS and the MMS should conduct statistical studies of risk factors, field-size distributions, prospect drilling outcomes, and other play attributes to determine whether assumptions of probabilistic independence are justifiable.

Recommendation: Both the USGS and the MMS should consider ways to carry diversity of opinion through to the final resource estimates. For example, the agencies could report outlier opinions that, while masked by the aggregated results, would lead to significantly different resource estimates.

Recommendation: Both the USGS and the MMS should develop explicit standards for estimating the risk that a play contains petroleum. These standards

should be tied to qualitative descriptions of source, migration, reservoir, and timing. The agencies should train their assessment staffs in the use of these standards so that different assessors use a common frame of reference when assigning risk numbers. The standards should be designed so that they can be checked against the available geologic data and replicated by other assessors.

Recommendation: Both the USGS and the MMS should periodically train their oil and gas geologists in subjective probability assessment. The training should be more extensive than a “short course” just prior to the next national assessment and should be focused on real case histories.

The results of MMS and USGS resource assessments have tended to be inadequately reported by the media and poorly understood and misinterpreted by key users, including members of Congress. Although some misunderstanding may be inevitable given the desire of the media and some users to deal with simple numbers, some measure of responsibility for the overall level of misunderstanding falls on the assessment team and report writers.

In particular, the committee concluded that the assessment report did not state clearly to the reader the degree of uncertainty in the various steps of interpretation, extrapolation, and conclusion from a limited data base of resource characteristics. It is critical that those preparing the resource estimates clearly communicate the uncertainties inherent in their calculations. Given the magnitude of these uncertainties, the committee is concerned that the method of reporting resource estimates focuses too much on point estimates of undiscovered oil and gas, such as the mean. Emphasizing to users the whole range of potential resource values—not just a point estimate —is especially important in Alaska because of the limited data available there and because Alaska contains such a potentially large share of the undiscovered resource base (26.9 percent of technically recoverable undiscovered oil and 22.7 percent of economically recoverable undiscovered oil, calculated with mean values).

Recommendation: In reporting future assessments, both the MMS and the USGS should place more emphasis on the range of uncertainty in their resource estimates. For example, the agencies could create more graphic displays to demonstrate visually the ranges of uncertainty.

Recommendation: The USGS and the MMS should take special care to insure that assessment users understand the relative role that undiscovered resources play in the resource base. The agencies should consider explaining and emphasizing the relative share of the undiscovered resource base as a source of reserve additions.

Incorporating the committee's recommendations in future resource assessments is likely to result in significant changes in the estimates of undiscovered oil and gas. The committee believes the overall impact will be to increase both the point estimates and the breadth of the range of estimated resource volumes. Problems with geological and statistical methods (such as mixing diverse petroleum reservoirs in plays and overlooking possible probabilistic dependencies between variables) and with the assessment boundaries (such as the exclusion of important natural gas sources) point to the conclusion that the assessment understated undiscovered resource volumes.

Implementing the recommendations requires allocation of resources over an extended time period. Especially at the USGS, the size and variable quality of data bases and limits placed on manpower dedicated to resource assessment have constrained the ability to attack resource assessment methodological problems with the vigor they deserve. Nonetheless, unless concerted effort is made to update assessment methodologies, the next national assessment may raise the same questions as the most recent assessment. By carefully considering the recommendations this report offers, the DOI can improve the credibility of future assessments.