Undiscovered Oil and Gas Resources: An Evaluation of the Department of the Interior's 1989 Assessment Procedures (1991)

Periodically, geologists at the U.S. Department of the Interior (DOI) convene to assess how much undiscovered crude oil and natural gas remains in the United States. Recent events in the Persian Gulf have underscored the need for such assessments. These events have once again forced policymakers to consider seriously how domestic production can be boosted to decrease reliance on petroleum from the unstable Middle East. The DOI's resource assessments provide policymakers with a key tool for evaluating ways to increase domestic production: a projection—based on scientific methods—of the quantity of oil and gas that could be developed from domestic sources.

Methods for assessing undiscovered petroleum are relative newcomers to the field of science. The U.S. government, for example, has been compiling petroleum resource estimates only since the early 1970s (though petroleum companies compiled their own estimates before that). Consequently, petroleum assessment methods are still evolving rapidly. Petroleum assessments currently in use employ a complex interplay of geology, statistics, and economics. New research in these disciplines continually offers ways to improve the reliability of the assessments. An assessment method judged reliable for an appraisal one year may be outdated by the time another appraisal is conducted a few years later.

Because petroleum resource assessments are so important to national energy policy and because assessment methods are advancing so rapidly, it is important that the DOI's assessments be evaluated periodically. This report presents a review, conducted by the National Research Council's Committee on Undiscovered Oil and Gas Resources, of the DOI's most recent resource

assessment, which was published in 1989. The following summary encapsulates the committee's review; each section of the summary corresponds to a chapter in the report.

When the DOI released its resource estimates, some members of the natural gas industry questioned the results. Although the DOI's estimates of undiscovered oil and gas volumes were in the same range as estimates prepared by some industry analysts, they were substantially lower than other industry analysts had expected. These analysts pointed out that the 1989 estimates were lower than estimates from the two prior DOI resource assessments, published in 1975 and 1981. The volume of petroleum discovered in the years between the assessments was not enough to explain the decline in the estimated resource volumes.

In part to respond to the natural gas industry's questions, the Secretary of the Interior asked the Committee on Undiscovered Oil and Gas Resources to “review the assumptions and procedures employed in the [assessment].” Determining the degree to which the estimates are too low or too high, or whether the numbers are correct, was beyond the committee's charge, because such a determination would require almost as much effort as the assessment itself. Instead, the committee focused on evaluating the procedures used to produce the estimates.

Much of the committee's review centered on the way in which the DOI implemented a resource assessment procedure called “play analysis.” The 1989 assessment was the first comprehensive national assessment in which the DOI applied play analysis, which requires a much higher level of geologic information than methods used in previous assessments. In play analysis, petroleum fields are grouped into subsets called “plays”—families of petroleum reservoirs that have geologically similar opportunities for petroleum accumulation. By combining geological judgment and statistical analysis of the history of discovered fields in a play, analysts can determine the likelihood that the play contains additional, undiscovered fields. Thus, play analysis provides a framework within which predictions about undiscovered resources are closely tied to knowledge about discovered resources.

It is essential to recognize that estimates of undiscovered oil and gas resources are just that: estimates. They are an attempt to measure uncertain quantities that cannot be accurately known until the resource has been essentially depleted. Resource estimates should be viewed as assessed at a point in time based on whatever data, information, and methodology were available at that time. They are subject to continuing revision as undiscovered resources are converted to reserves and as improvements in data and assessment methods occur.

Some of the uncertainty associated with resource estimates is inherent in play analysis. Though methods based on play analysis are widely regarded as the best way to assess petroleum resources, play analysis necessarily incorporates the subjective judgments of the resource analysts. There are no hard and fast rules, for example, for deciding which petroleum reservoirs are similar enough to include in a particular play. Furthermore, for each play, assessors must make an “educated guess” about the probabilities of occurrence of various geologic factors necessary for petroleum accumulation (a source rock, a reservoir rock, a trap, and so forth).

In addition to uncertainties inherent in assessment methods, another source of uncertainty in resource assessments is limitations in data bases. Assessors rely on two types of data bases: one containing geologic information, the other containing production and drilling information. The detailed knowledge of regional geology required for grouping reservoirs into plays comes from geologic data bases. Knowledge about the discovery history of known oil reservoirs—necessary, for the statistical analyses that project the number and size of undiscovered reservoirs—comes from production and drilling data bases. Resource assessments can be only as good as the basic data that support them, so it is essential that the data bases be as complete and accurate as possible. Unfortunately, this is not always the case; for some areas, the necessary data do not exist.

Changing technology and economics are another cause of uncertainty in resource assessments. Resource estimates never include every potential source of petroleum in their final projections. Typically, they contain technical and/or economic overlays that screen out petroleum that either is not extractable with current technology or will cost more to produce than it is worth. The 1989 DOI resource assessment, for example, includes two sets of resource estimates: one that covers all resources for which recovery technology is available (termed

“technically recoverable” or simply “recoverable” resources), and another that includes only the recoverable resources that will yield a profit when produced (termed “economically recoverable ” resources). Yet, recovery technology is constantly improving, and petroleum that may not be extractable one year may be developed with new production methods a few years later. Likewise, changing petroleum prices and production costs may move a reservoir that is judged uneconomic in one assessment into the category of profitable resources in a future assessment.

In summary, some of the major causes of uncertainty in resource assessments are the subjective judgments that are necessarily a part of assessment methods, the data bases employed in the assessments, the continual evolution of extraction technology, and the fluctuation of production costs and petroleum prices.

After a detailed examination of the DOI's data bases, geological methods, and statistical methods, the committee judged that there may have been a systematic bias toward overly conservative estimates. Eliminating the probable sources of this bias will improve the accuracy and credibility of future assessments.

Two agencies within the DOI oversaw different portions of the assessment: the U.S. Geological Survey (USGS) assessed oil and gas onshore and in state waters, while the Minerals Management Service (MMS) assessed resources for the Outer Continental Shelf (or OCS, the part of the U.S. offshore that falls under federal jurisdiction). Though many of the committee's findings apply to both agencies, some relate specifically to the USGS or the MMS, because the agencies used somewhat different assessment procedures.

The committee found four key characteristics of the assessment procedure that may have biased the assessment toward overly conservative resource estimates. These characteristics were improper play definition, inadequate consideration of conceptual plays, insufficient consideration of probabilistic dependencies between variables, and the unintended imposition of economic

constraints on technically recoverable resource computations.

Improper play definition: The proper grouping of like reservoirs into plays is the foundation for play analysis. Mixing dissimilar reservoirs in a play may lead to biased statistical characterization of the maturity of the play. Biased statistics, in turn, may lead to a biased assessment of the potential volume of undiscovered resources in the play. The committee found that in some cases, both the USGS and the MMS created excessively large plays that contained reservoirs with diverse systems for petroleum deposition. For example, in the Pacific Coast region, the MMS divided prospective petroleum fields into plays according to stratigraphy, not by the more rigorous consideration of all the factors necessary for petroleum deposition that true play analysis requires. The USGS, in areas such as the Permian Basin, combined into plays prospective petroleum fields containing mixtures of carbonate and sandstone lithologies, suggesting that diverse depositional systems were mixed in the plays. To determine how the grouping of unlike reservoirs within a play might affect the overall resource assessment, the committee examined ten USGS Permian Basin plays in detail (see Appendix C). The analysis suggests that mixing unlike depositional systems in a play may create the false impression that the play is mature in its discovery history: i.e., that a large number of its petroleum reservoirs have already been discovered. Thus, the mixing of diverse geological systems in plays may have caused the resource estimates to be low.

Insufficient consideration of conceptual plays: Conceptual plays are plays that do not contain discovered petroleum reservoirs but that geological analysis indicates may exist. Conceptual plays may be especially important for assessing natural gas supplies, because natural gas exploration is less mature than oil exploration, so many potential natural gas producing areas lack discovered reservoirs. Though the agencies incorporated conceptual plays in some areas, the committee found that conceptual plays were not given adequate weight. For example, the MMS, in evaluating northern California' s Eel River Basin, analyzed only four conceptual prospects, compared with 92 “identified” prospects. Because the information used to evaluate conceptual plays is sketchier than the data for plays with known reservoirs, conceptual plays require a separate, much more subjective evaluation procedure. Nevertheless, overlooking the importance of conceptual plays discounts a significant fraction of the nation's undiscovered resources and may have caused the estimates to be overly conservative.

Inadequate consideration of probabilistic dependence: When two unknown events are unrelated (i.e., probabilistically independent), the probability that both events will occur can be determined by multiplying the individual probabilities

of occurrence of each event. However, when two unknown quantities are related (i.e., dependent), computing the probability that both will occur is more complex. In several cases, both the USGS and the MMS treated variables as independent when they may have been dependent. For example, USGS analysts assumed that the geologic events that lead to petroleum accumulation—source, timing, migration, and reservoir formation—are probabilistically independent. They multiplied the probabilities of occurrence of each of these events together to yield the marginal probability that a play contains petroleum. In most plays, however, the events necessary for petroleum accumulation are related geologically and therefore may be probabilistically dependent. No careful statistical studies of this hypothesis have been conducted, but treating such factors as independent may yield a lower probability that the play contains petroleum than if the factors were treated as dependent. Thus, the assumption that these uncertain quantities are independent when they may be dependent could have caused the estimates to be low.

Imposition of economic constraints on recoverable resource calculations: Recoverable resource estimates should be independent of economic constraints, because such estimates provide a prediction of how much petroleum could be discovered if economics were not a factor. However, the committee found that the MMS may have imposed economic constraints on its recoverable resource calculations. In Alaska, for example, MMS assessors excluded prospects that were smaller than one-half a leasing block from their technically recoverable resource estimates. This exclusion contains an implicit economic assumption: that prospects smaller than one-half a leasing block are too small to yield profits. This implicit economic assumption results in lower technically recoverable resource estimates. Also, when explicit economic screens are applied to technically recoverable resource volumes that were calculated with implicit economic assumptions, the result may be unintended double discounting and a reduction of the economically recoverable resource estimates.

In addition to finding that these four methodological limitations may have biased the estimates toward low values, the committee found that by confining the assessment to “conventional” resources, the DOI overlooked a significant portion of the potential domestic energy supply. The assessment covered only potential petroleum reservoirs that could be tapped with conventional technology. While unconventional oil deposits (like those from tar sand and oil shale) have contributed little to domestic production, unconventional natural gas deposits from low-permeability sandstones, fractured shale, and coal beds are making an important contribution to current production, thanks to new

technology. Because unconventional natural gas is potentially so important to future production, furnishing estimates of unconventional gas would have painted a more realistic—and optimistic—picture of the nation's undiscovered petroleum supply.

In its review, the committee looked beyond factors that could have affected the magnitude of the estimates. The committee evaluated the training of staff, the integration of the diverse opinions of geologists into the final assessment results, the use of subjective judgment, the completeness of data bases, and the consideration of in-place resources.

Staff training in statistical methods could improve understanding of key probability concepts. According to a senior USGS manager whom the committee interviewed, for the 1989 assessment staff geologists ' knowledge of statistics was, in many cases, limited to what they had learned in a “short course” just before the assessment. At the USGS, a permanent group of employees devoted to conducting resource assessments could help implement a statistical training program. (The MMS, unlike the USGS, maintains a permanent resource assessment staff, because it is required by Congress to perform resource assessments biennially.)

Greater consideration of the differing opinions of geologists could reflect more accurately the uncertainty inherent assessments. At both agencies, several experts' evaluations of the aggregate resources in each play were consolidated into one consensus result for the 1989 assessment. Consensus was used to obtain both the probability that each play contains petroleum and the distributions showing the range of possible resource volumes. It is inevitable that different individuals with comparable knowledge will assign different probabilities to the likelihood that petroleum is present in the same situation. Consolidating these divergent opinions into one consensus result may have artificially constricted the range of the resource estimates.

Limiting the reliance on subjective judgment could reduce some of the uncertainty in future assessments. USGS assessors used subjective judgment to extrapolate the number and size of undiscovered fields from available data about discovered fields. For analyzing plays where a moderate to large number of fields have already been discovered, there are objective models (called “discovery-process” models) available that could provide projections of

undiscovered oil and gas unencumbered by judgmental uncertainties.

Expanding geologic data bases could facilitate the correct application of play analysis in some areas. Without adequate geologic data, defining plays correctly and identifying conceptual plays is difficult. The committee found that the USGS assessment was limited in particular by a lack of seismic data for the lower 48 states and also for portions of Alaska's North Slope. While the MMS's seismic data base was more complete, there were important gaps in the MMS's geochemical data. The committee found that data available from state geological surveys, state regulatory agencies, and private-sector sources (including industry) could have helped fill some of the gaps in the agencies' data bases.

Finally, expanding the assessment boundaries beyond the limits of recoverable resources could decrease assessment sensitivity to technological advances. The 1989 assessment included no estimates of in-place resources —the total volume of petroleum trapped within each play, without consideration of whether or not the petroleum is extractable. Improvements in reservoir characterization, drilling, and completion continually increase the percentage of a reservoir's total petroleum supply that is recovered. Consequently, estimates that are based on recovered resources, instead of in-place resources, become outdated as technology progresses.

Estimating how much oil and gas remains to be discovered is necessarily an inexact process. Without actually drilling, one cannot know precisely what volume of petroleum a prospective reservoir contains. Nevertheless, new assessment methods developed since the early 1970s have the ability to increase the reliability of resource assessments if properly employed. In evaluating the USGS's and MMS's assessment procedures, the committee undertook to provide the agencies with recommendations that, if implemented, will increase the level of confidence in their future assessments. The committee divided its recommendations into five categories. The first category, Assessment Boundaries, addresses how the agencies can ensure that every potential petroleum source is included in future assessments. The second category, USGS and MMS Management, suggests ways the agencies' managers can increase confidence in the assessment process. The third category, Geological Approaches in Data Base Use and Play Analysis, suggests how the agencies' geologists can upgrade their data bases and play analysis techniques. The fourth category, Statistical

Methods, recommends ways to ensure the proper application of statistics. The final category, Assessment Results, recommends ways to report future assessments so that users understand better the uncertainty inherent in the results. Below is a list of specific recommendations in each of these categories.

• The DOI should include estimates of natural gas from unconventional sources (separate from the estimates of conventional gas) in future resource assessments.

• The DOI should include estimates of in-place resources in future assessments. Assessors should estimate a play's in-place resource volume first. and then calculate the play's recoverable resource volume by applying a recovery factor to the in-place value.

• The USGS should establish a group of specialists to design and oversee on an ongoing basis a program for improving and implementing oil and gas assessment methodologies. This permanent assessment group should emphasize data validation, training of geologists in assessment methods, and more aggressive use of modern statistical methods.

• Managers at the USGS and the MMS should establish complete, continued cooperation between the assessment groups at the two agencies.

• Both the USGS and the MMS should conduct an audit of their drilling, geological, and geophysical data bases. The audit would have three purposes: (1) to evaluate the data's accuracy and completeness; (2) to identify areas where the data base requires improvements; and (3) to provide explicit measures of the data's quality to assessment users.

• Based on the results of the data evaluation, both the USGS and the MMS should attempt to expand and refine the available data for areas where their present data bases are incomplete. Better use of existing data bases should

precede the creation of extensive new data bases. The agencies should seek data from outside sources like state geological surveys, state regulatory agencies, other federal agencies, and the private sector. For example, it is possible that the USGS could obtain access to proprietary seismic lines from industry in key areas.

• To expand its geochemical data base, the MMS should ensure that full geochemical evaluations are conducted for wells drilled in offshore areas where existing data are inadequate.

• The USGS should analyze play content to determine the impact of play formulation—especially formulation of plays with diverse depositional systems—on the resource volumes reported in the 1989 assessment. In future assessments, the USGS should avoid creating excessively large or geologically diverse plays.

• The MMS should define plays more carefully to avoid the mixing of diverse geological and reservoir engineering characteristics in one play. The MMS should recognize that the availability of an extensive seismic data base could lead toward plays excessively dependent on recognition of structural traps. Once it has formulated plays appropriately, the MMS should institute testing to ensure that play mixing does not significantly alter resource estimates.

• Both the USGS and the MMS should incorporate more conceptual plays in future assessments. Analyzing conceptual plays may require that the agencies develop assessment techniques different from those used for known plays.

• The USGS should use objective models based on the discovery process in place of subjective extrapolations in areas with sufficient discovery data.

• To avoid unintended double discounting, the MMS should develop methods for separating technically recoverable resource calculations from those that determine the volume of economically recoverable resources.

• Both the USGS and the MMS should conduct statistical studies of risk factors, field-size distributions, prospect drilling outcomes, and other play attributes to determine whether assumptions of probabilistic independence are justifiable.

• Both the USGS and the MMS should consider ways to carry diversity of opinion through to the final resource estimates. For example, the agencies could report outlier opinions that, while masked by the aggregated results, would lead to significantly different resource estimates.

• Both the USGS and the MMS should develop explicit standards for estimating the risk that a play contains petroleum. The agencies should train their assessment staffs in the use of these standards so that different assessors use a common frame of reference when assigning risk numbers. The standards should be designed so that they can be checked against the available geologic data and replicated by other assessors.

• Both the USGS and the MMS should periodically train their oil and gas geologists in subjective probability assessment. The training should be more extensive than a “short course” just prior to the next national assessment and should be focused on real case histories.

• In reporting future assessments, both the MMS and the USGS should place more emphasis on the range of uncertainty in their resource estimates. For example, the agencies could create more graphic displays to demonstrate visually the ranges of uncertainty.

• The USGS and the MMS should take special care to insure that assessment users understand the relative role that undiscovered resources play in the resource base. The agencies should explain and emphasize the undiscovered resource base's relative share as a source of reserve additions.

Incorporating the committee's recommendations in future resource assessments is likely to result in significant changes in the estimates of undiscovered oil and gas. The committee believes the overall impact will be to increase the estimated resource volumes. The DOI's procedures for the 1989 assessment appear to have been overly conservative, and therefore the assessment may have understated undiscovered resource volumes.

Implementing these recommendations requires allocation of resources over an extended time period. The lack of a permanent resource assessment group at the USGS has constrained the ability to address resource assessment methodological problems with the vigor they deserve. Unless concerted effort is made to update assessment methodologies, the next national assessment may raise the same questions as the most recent assessment.