1993 Data Forum: A Review of an Implementation Plan for U.S. Global Change Date and Information Systems (1994)

Several individuals were asked to prepare written critiques of the Plan in advance of the forum. Those asked to contribute were chosen to provide a wide range of disciplinary perspectives. The critiques appear in the order presented, with minor editing for continuity of style. The last three critiques, written by John Pastor, Nicholas Chrisman, and Allen Hammond, were distributed at the forum, but were not presented orally.

Director, Byrd Polar Research Center

The Ohio State University

General Comments:

The Implementation Plan is an excellent follow-on document to the bold concepts articulated in the report “The U.S. Global Change Data and Information Management Program Plan”. Much in the spirit of the predecessor document, this report is a statement of guiding principals and overarchiving policies for determining who is likely to use the Global Change Data and Information System, what information and data will likely get into the system, and how users will retrieve material from the system. It is important to document these management principals at the beginning of the project and here the report succeeds very well. But the report does not (nor probably should it) go on to describe the implementation of an actual system. (Indeed the only vague reference to system development is NASA's role in providing information services designed to facilitate interoperability.) Still, someone eventually will be tasked to build or modify a system and so the link to those scientists and engineers should be identified. That said, it probably is worth considering renaming this report something such as “The U.S. Global Change Data and Information Implementation Program Plan”. Regardless of the title, the statement of the Plan's goal is splendid.

Specific Comments:

• Most of the right concepts are highlighted in the Plan, for example: user involvement at all levels; development of an open, distributed system; interoperability across agencies; long-term data archiving and data stewardship. My observations of the development of EOSDIS are that it is extremely difficult to actually implement these concepts. The Plan does not identify new ideas that might overcome these difficulties and I cannot offer any new suggestions.

• The Plan seems to view researchers and policy makers as passive users. Though several examples suggest otherwise (e.g. the example of lowland flooding caused by rising sea level), there is no concrete statement suggesting that data reprocessing, multi-set data assimilation and data fusion functionalities will be resident in the system.

• Based on the executive summary, I tried to illustrate the ‘Tower of Babel' structure the implementers are facing (Figure 1). I think we all recognized how diverse the various agency approaches to data management were but the task required to present the user with a homogenizing interface without recreating each of the heterogenous systems is formidable.

• My interpretation of the management structure based on reading the text varies somewhat from Figure 1 in the report. Based on my reading, I interpret the management structure as illustrated in my Figure 2.

• I attempted to look at the functional relationships between the various management groups as illustrated in Figure 3. Lacking from the Plan are data fusion capabilities and links to private industry (denoted by the gray circles). Regarding the later for example, access to oil company data on the effect of the Alaskan Pipeline on permafrost and habitat could ultimately be useful to future policy makers.

• In reviewing the section on Agency Implementation Overviews, I found information on activities within agencies but very little information about activities between agencies. This seemed surprising because cross agency activity is at the heart of this Plan.

• The Plan recognizes that all the agencies will not come on line in GCDIS simultaneously. Nevertheless it would be useful to see a rough time line delineating what the expectations are for each agency especially regarding the degree of interoperability expected (e.g., not many groups will be able to rapidly implement browse functions I suspect).

Outstanding Questions:

• Who are the users? One could interpret the report as implying that everyone will be a user.

• Who will be the SYSTEM implementers?

• GCDIS is a bold concept. Do users expectations already exceed probable system attributes?

Institute of Atmospheric Physics

University of Arizona

The plan is very complete and comprehensive as an abstract description of everything GCDIS should or might do. However, from my perspective as a practicing scientist involved mostly in the development and usage of climate models, many aspects of GCDIS that are important to me are treated very lightly, with the indication that they will be planned further. Another disappointment is the long time frame required to get GCDIS fully operational a much longer time than it took to get a man on the moon. My impatience is in part a result of having been a participant for at least 20 years in discussions on the need for improved data for climate modeling. As I will likely retire in another 10, I don't want to have to wait another 20 years for this. Fortunately, there are a lot of quite valuable data activities now supporting global change research. A plan such as this can be constructed “top-down” from an overall abstract definition of what is needed or “bottom-up” by analyzing what now exists and how to best build and improve upon that. I think that the best plan would be put together from both directions, and indeed the present plan does this. However, I find the bottom-up descriptions to be largely bureaucratic and having, at best, large gaps in the recognition of how the global change community is now supplied with the data it uses. I do not see any evidence in the plan that its authors have systematically gathered such information.

Many of the most successful current activities for providing the climate community, and especially the climate modeling community, with essentially needed data have been organized through WCRP. The near absence of mention of WCRP and any of its programs in the chapter on “International Links and Coordination” is a clear sign to me that this, among other chapters, must have been written by a bureaucrat with little linkage to current research activities. (James Baker, the new director of NOAA, has served for many years on the Joint Scientific Committee of WCRP and could provide a significant amount of input on this.) Data programs of WCRP that currently have a large impact on climate model-

ing include the International Satellite Cloud Climatology Project, the Surface Radiation Budget (SRB) Project and the Global Precipitation Climatology Project (GPCP).

As a climate modeler, I have long had conversations with the various communities in need of support for the acquisition of output from climate modeling activities. Therefore, I have been especially curious as to the possible evolution of that ingredient of Global Change Data systems. This present report recognizes this as an issue, but does not show what progress has been made in addressing it. The information I do find in the plan is:

• CEES will have, or has, a working group on integrated modeling that will supply the requirements

• By 1995 somebody will carry out a model input and data review to consider needed input data and issues related to model output data.

In the write-up on individual agencies, I find no mention of modeling data anywhere. NOAA, NSF, DOE, and NASA all have very large investments in modeling programs, but I am not aware of plans anywhere of how the output from their models should be maintained and provided as Global Change Data. This is definitely an issue that can only be successfully addressed from a bottom-up approach. One important issue is that modeling groups in the United States are jointly using ≻ $20M per year, of computational resources, to carry out climate simulations using many different individual models. These groups currently do not have any resources to put the data they generate into forms useful to the outside communities that would like access to this information. Furthermore, they tend to think of their output as proprietary and so may be somewhat hostile to the idea of releasing it. Meanwhile, the community of potential users needs, or may need, many kinds of output potentially available from the models but which currently are not saved at all by the modelers. (Saving would require identification as to what should be saved, writing the necessary software to save it, and allowing for this additional data in the modeling archives.) Also, a clearer recognition is needed from the top-down viewpoint that this is currently a serious data availability prob-

lem that should be solved. Its solution would presumably involve these ingredients:

• informing the modelers as to what is needed, providing them with the additional resources they need to be able to provide that, and gaining their cooperation

• getting the potential modeling data user community to be more familiar with what the capabilities of the models are, and to clearly express what data they need.

School of Forestry and Environmental Studies

Economics Department

Yale University

Strengths and weaknesses of the Plan with regard to policy analysis.

Observations

• Current Plan stresses sharing primary observations made of earth natural systems.

• Plan provides no structure for sharing results of results of research.

• Plan provides no protocol for how data should be used.

Strengths

• Natural scientists needing access to observations should be able to find data previously hard to get.

• Whether specific observations are being made can be quickly determined.

• The consistency of observation methodology can be compared across funding sources and types of investigators.

Weaknesses

• The Plan may facilitate improper use of observations by people unfamiliar with the limitations of the data.

• The absence of any coordinating structure, which interprets and integrates data across disciplines and locations makes the data set unwieldy and impractical for policy decisions. The data set will consequently be irrelevant for policy analysis.

• The Plan will not identify which pieces of data are of greatest need for making decision. The Plan can therefore not identify data collection priorities.

• No data about either the costs or the damages from global warming are mentioned in the Plan. Clearly, the Plan has no intention of addressing what society should do about controlling greenhouse gases or how society should adapt to changing climates.

Suggestions

1. If the dataset is merely a collection of primary earth observations, a protocol for using each component of the data should be established prior to data release.

2. If the dataset is to be useful for making decisions about global warming, the data must be organized using an integrated assessment framework. The framework will explain how the results of different efforts must intertwine. Only with such a structure, can the data (results) be organized into a meaningful whole for policy analysis.

3. Data collection should include information about the costs of reducing greenhouse gases.

4. Data collection should include projecting future man-made emissions of greenhouse gases and how these projections are tied to different economic activities.

5. Data collection should include measurements of the damages from climate change. Researchers should be looking from natural experiments, which resemble climate change scenarios to begin to understand how economic activity and human behavior will react to climate change.

Institute for Social Research

The University of Michigan

General Comments:

• Broad Plan is excellent.

  1. Provisions for access to data are laudatory.

  2. Building on existing facilities is wise.

  3. Attention to connections among systems is essential.

• Commitment to include data relevant to human dimensions of Global Environmental Change is wonderful.

• No agency of the U.S. national government is responsible for human dimensions data in the same way that NASA and NOAA are responsible for natural science data.

• The agencies Implementation Plans relating to human dimensions data generally lack specificity.

Human Dimensions Data are different:

• Little has been done to identify data that are essential to understanding the human dimensions of global environmental change.

• Much relevant data is collected and maintained by foreign governments and international governmental organizations.

• The U.S. Department of Defense and Intelligence Agencies collect much relevant data.

• Human dimensions research requires considerable non-quantifiable data.

The role of CIESIN

• Because social science data are different, an even more decentralized and non-national system will be essential for human dimensions research.

• CIESIN is the Data and Information System for the HDP of the International Social Science Council.

• CIESIN can be a hub for the multiplicity of national and international agencies that are involved.

• CIESIN's role in the Implementation Plan needs to be strengthened.

Director, Eastern Region

National Weather Service, NOAA

• The Plan is well written; unexpectedly covers many details one would not have thought would be included in such a high-level document.

• The overall Plan is ambitious, to say the least. Thus, one is immediately given to skepticism in the review. However, experience has taught that only the boldest initiatives gain the attention deserved.

• RESOURCES, RESOURCES, RESOURCES. . .

  1. Reliance on individual agencies' plans and agendas will not make the whole. The Implementation Plan can and should serve as a keystone and driver of agency initiatives for their individual data and information management programs.

  2. Agency Plans often overlook or fall short of anticipating the human and time resources needed for documentation during implementation; and for recurring costs needed for general maintenance, system upgrades, revisions and distribution of system documentation.

• Documentation is the key to success. The definition of the term must include publication of general information and advertisements widely distributed throughout the country. Too often, only a small or elite group of individuals are aware of, much less knowledgeable, about data and information availability.

• The Plan has not yet grasped the entire scope of data needs to support the development and promulgation of public policy. The full spectrum of environmental information will reach back in time, perhaps including synthetic data streams (e.g., use of tree rings and carbon dating techniques for paleo-runoff regimes) and extend forward for decadal forecast services. Along this spectrum of information, users should be able to pull out slices or sections of data, from five minute high resolution event based, to long duration, annual or interannual time series.

• The archival problems associated with development and sustenance of this full spectrum of environmental information are not understood; the archival system requirements not defined; the retrieval through networked systems as yet infeasible. Agencies are just now taking the first steps to ensure the preservation of the marvelous data being collected by new techniques and technologies.

• Program oversight through the proposed committee structure seems cumbersome. It is doubtful such structure will provide the support for users envisioned in the examples given in the draft Plan. And it is equally doubtful that the committee will have the clout necessary to push agencies to implement, update, document, (and all the things necessary to sustain and improve) their information processes and systems. Casting the oversight role into one more bent on sustaining the mission motivating and supporting budget initiatives might be more productive.

• The Plan should consider the need for quid pro quos and a sliding scale of fees for users.

• It will be necessary to ensure that processed, refined, and derived information sets resulting from the user of the databases be returned to or made available to the next generation of users. Thus the relationship between the public/academic nature of the system(s) and the private sector needs to be better understood and defined.

• International connections, documentation, input to the Plan, use of systems, and payments are more complicated than portrayed. Some of the greatest benefits may be gained in developing areas, thus increasing the need for technical cooperation through bi-lateral and cooperative programs. These countries are the least able to afford user fees, which are becoming ubiquitous in the international community.

Manager, Integrated Earth Studies

Battelle - Pacific Northwest Laboratories

Randomly Aimed Bullets

Sources of Data

• Consider including data from Regional Climate Centers

• Consider inclusion of phenology datasets

Data Quality

• Should the extent of data documentation and quality assurance depend on “the importance of any particular global change dataset”? Who knows now what will be important later?

• International global datasets are problematical (for all the reasons stated in 4.3.4) and are also extremely variable in quality. They are critically important, however, and should be patched up as much as possible.

• “Quality Tags”. Is there ever justification for using data of questionable quality? (4.7.2)

User Issues

• Pointing out strengths and weaknesses of data (4.5.2) is extremely important, especially for the generation of analysts now rising whose members have never, themselves, made a measurement.

• Educating the user community (especially the analyst) on data quality, alternative sources and approaches is extremely important, but since useful analyses draw upon many kinds and sources of data, who will do the educating?

Organizational Issues

• Are we convinced that systemizing access to all the desired information in the absence of or in anticipation of future needs will be more effective (efficient? economic?) than letting the users define their needs and go after the data themselves? [This does not agree with the need for good, high quality, long-term, reliable and accessible data.]

• Does any one group or entity have the ‘teeth' to make GCDIS go, considering the many agencies and complex bureaucratic structures involved? Reassurance, please!!

Woods Hole Oceanographic Institute

Marine Policy Center, Woods Hole

Comments:

• Change the name. How do we know “global change” will be as popular in 10 years as it is now?

• There should be different levels of accessibility. The demand for different datasets will change, in some cases systematically. Provide for flexibility in changing levels.

• Keep track of users of different datasets.

• Think about what not to include. Don't duplicate existing databases (e.g., U.S. Census Bureau). If economic and other social scientific data are included, then economists and other social scientists (not natural scientists with social science hobbies) should be involved.

• Learn from successes (and failures) of existing system. For example, in my opinion, NCDC is very good. So is Census. Getting data from France and Japan is hard.

• Spell out role of scientists in management. How can a broad range of scientific approaches and views be guaranteed? Don't tell me what data I need.

• Plan for maximum flexibility. Learning by doing.

• Satellite data may be over-emphasized in the Plan. Including output from multiple runs of large-scale models may pose special problems.

• Define “parameter” (I think you mean “variable”).

• I am concerned about how far this Plan goes in trying to impose organization to “global change” research. Too much anticipated research (i.e., addition of value). Too sensitive to individuals involved in planning. I do not like the questions format described on p. 20 (i.e., survey important questions; decide what data are needed to answer them).

Chairman, Department of Earth & Planetary Sciences

Harvard University

Strengths and weaknesses with regard to atmospheric chemistry

Strengths

• Degree of interagency cooperation

• Promise of wide and easy access to “global change” literature grey and otherwise.

• Emphasis on documentation and metadata

• Access where needed to more specialized databases

Weaknesses

• Still preliminary and difficult to judge how useful it will be until samples of final product are available

• Lack of information on cost of access

• Lack of information on funds available to users to allow access

• Difficult to grasp the real function of the system due, in part, to the complex, legalistic, bureaucratic style of the document

• Critical assessments will require critical analysis--with exception of large visible issues, unclear by whom and how this will be developed.

Suggestions

• It would be useful to see a more specific, detailed account of how the system would operate for one or a few specific issues that might be considered mature, for example, stratospheric ozone or regional air quality.

• Important to define more carefully how model data should be included. In particular, I believe that at least some summary information

from general circulation models used in major assessments would be valuable. But the question is what information and from which models.

National Center for Atmospheric Research

Strengths and weaknesses from a terrestrial ecology perspective

Strengths

• extensive interagency cooperation

• Plan is ambitious in scope, interface with libraries and non-scientists is a particularly important aspect

• NASA's experience with large databases makes them particular appropriate implementation coordinators

• timetables for implementation are detailed (but a little incomprehensible)

• emphasis on documentation is important to ensure that the user understands the data and its limitations

The development and assembly of these global databases is a rate limiting step in advancing the science of global change. The quality of science done on a global or regional scale will be determined by the quality of these databases.

As a young scientist, I have a particularly keen interest in the development of this database because it will determine what I can do in the coming years.

Weaknesses

• What are the agency incentives to ensure that database development continues in the current stringent budget environment? or Where is the big stick?

• The current document lacks specificity about what data will be incorporated and how disparate databases will be combined.

• A potential hole in the data set is information about plant species ranges. Rate of movement of species will determine how quickly vegetation boundaries change. Involvement of the Missouri Botanical

Garden, the Smithsonian and/or the Sustainable Biosphere Initiative might be an appropriate remedy.

• How will model output be handled? Commonly used models and algorithms should be included as part of the database. Extensive metadata will be needed to describe and reference modes.

• How will regional and local datasets be included? For example, assembly of ozone, carbon monoxide.

• Where will be the databases reside?

Data Needs

1. Species range maps

2. Land Use History

3. Natural Disturbance Cycles

4. Climatic Data

5. Soils information: texture, organic matter content, depth to water table, depth to permafrost, topography (slope and aspect)

6. %N of vegetation

7. Biomass allocation-will determine C storage and turnover times

Comments

• Ecological data does not provide spatially explicit coverage of a given area. It is local, idiosyncratic, and may not be spatially referenced. In most cases, a model will need to be used to provide the needed spatial coverage.

• Terrestrial ecologists are not used to working cooperatively on big science programs. Database development has been done on more of an individual basis. In many cases, ownership of a large database has determined career success. Many of the currently used large public domain databases date back to the International Biome Program of the 1970s.

• Pilot studies funded by some sort of granting program would be extremely useful tests of concept as well as providing insight into how to combine different kinds of data. For example, assembly diverse data-

bases like climate data, Normalized Difference Vegetation Index (NDVI), soil carbon, vegetation biomass, land use and human population.

Natural Resources Research Institute

University of Minnesota

Strengths and weaknesses with regard to terrestrial ecology

Strengths

• Comprehensive Plan involving numerous federal agencies

• Distribution of data responsibilities among agencies well-thought out (I would like to see the Soil Conservation Service more involved in the soils databases, however).

• Timetables for implementation detailed

• Emphasis on documentation and metadata

Weaknesses

• This reads like a plan to make a plan. It is imperative that the next version deal with very specific issues, such as what follows.

• The model database structure is taken from geophysical sciences complete, spatially explicit, global coverage of a few variables. In contrast, ecological data is very different local, not spatially explicit, idiosyncratic, often concerned with variables other than carbon (species lists, diversity, etc). The Plan needs to give more thought to difficulties of merging these two very different datasets.

• Money: as the Plan reads, money will be reallocated within each agency but there is no agency with overall jurisdiction or budgetary authority. There is a danger that database development and implementation will proceed at different rates because of differences in bureaucratic commitment.

• Where will the databases reside? There must be one or, at most, a few designated centers for archiving, retrieving, and updating the databases with sufficient funds and personnel committed for many years to do this. Either the Carbon Dioxide Information Analysis Center

(CDIAC) in Oak Ridge or NOAA Geophysical Database Center in Boulder are the best choices.

• Geographic Information System structure or program not specified

• University involvement: needs a granting program

• Digitized species range maps also need to be included: how ranges will shift in response to global change is an important process controlling terrestrial carbon flux. Perhaps this could be done by including organizations like the Smithsonian and/or Missouri Botanical Gardens.

Improving the Plan from interdisciplinary perspective

• See comment about geophysical vs. ecological data above.

• Begin pilot studies trying to assemble diverse databases, such as climate data, NDVI, soil carbon, vegetation biomass, land use, and human population. Such pilot studies will identify the problems involved quicker than any further discussion.

Department of Geography

University of Washington

While the Plan references the Federal Geographic Data Committee (FGDC), that group plans for a National Spatial Data Infrastructure. This Plan should be carefully articulated with the Global Change DIS, since so much of the global change data has a spatial reference.

The Mapping Science Committee of the NRC has produced a series of reports on spatial data needs and policies (mostly dealing with the national Mapping Program USGS, but creating the basic agenda for FGDC).

High level principles:

• GCDIS as a system should indeed strive towards user satisfaction, but data quality and content cannot be entirely driven by user satisfaction. The fundamental basis for data quality has to be measured against ground conditions, not what the user thinks they want to know.

• The organizing principles of the Spatial Data Transfer Standard (SDTS) Federal Information Processing Standard 173 apply strongly to GCDIS,

  1. A Data Quality Report must be prepared by the producer

  2. The user must evaluate the report for fitness for their particular use

  3. Metadata contents specified in SDTS and FGDC draft standards.

Spatial character of GCDIS metadata:

• Access to director, guide, inventory and browse functions space/time references (continuous, not confined by keywords)

• Change studies often set grand goals of “setting down a baseline”

1. such baselines rarely get used, since next inventory is done differently.

2. lots of failures dating back through this century.

• Change studies usually done in rearview mirror (no access to past environment) prevalent use of snapshot data misses access to processes

• Mapping is neither an art nor a science; it is a regulated utility.

  1. Society gets all the mapping it is willing to pay for

  2. U.S. data policies diametrically opposed to policies for mapping in most countries

  3. Scientific investigators may need to learn from disciplines that have been data providers for centuries

World Resources Institute

Washington, DC

GCDIS is at present largely a program to coordinate data programs. That in itself is a valuable and worthwhile goal. But the stated commitment to a program that can provide “summarized, evaluated information” or policy-relevant information to a wide range of users is more rhetorical than substantive. The problem is structural, since the basic assumption of the Implementation Plan is that agencies will “coordinate” their activities but that there will not be mandated inter- or supra-agency activities. Yet the analysis, assessment, visualization, and active communication activities needed to transform a data program into an information program are not included in the implementation program no agency or other entity is tasked with such activities. And to produce policy-relevant information and develop the specialized information products needed for active communication to policymakers and the public or even to assemble the complex sets of physical, biological, and socioeconomic data required for policy-relevant assessments requires an institutional base (or bases) and financial resources not at present identified.

It is appropriate that GCDIS serve the needs of the research community. The scale of the Global Change Research Program is such, however, that the U.S. government and the world are not going to build a separate monitoring and assessment system to serve policy, resources management, and general public information needs. Thus continuing pressure to transform GCDIS (and for that matter EOSDIS) so that it also serves those needs to provide or enable the provision of policy-relevant information can be anticipated. How might these needs be met?

It is useful to consider the model of the National Hurricane Center an issue-oriented assessment activity that provide active, policy-relevant information products over a variety of media under deadlines, on the basis of which local and national authorities issue evacuation orders, small craft advisories, etc. Hurricanes are more dramatic and more immediately life-threatening than global change, but the model may still be useful. Consider that CDIAC already provides assessments, specialized informa-

tion products, and specialized datasets on the basis of which assessments can be made for one aspect of global change. CIESIN may represent an emerging capacity to locate, assemble, and assess socio-economic data and help combine them with physical or biological data. Other issue-oriented centers are likely to be needed as well, and the implementation report should state clearly that it cannot fulfill its goals without the creation of such additional capacity, even if it is presently outside the scope or resources of GCDIS to create them. Note that both examples considered here are outside the line-agency framework: one is a national laboratory, the other a consortium of universities. Both of these models should be explored further as sites for institutionalizing synthesis, assessment, and information communication activities.

Finally, there is a truth-in-packaging issue here. If GCDIS cannot implement what is required to become a policy-relevant information system, then it should say so clearly in the report. That may have the salutary effect of increasing support for the allocation of resources (within or without GCDIS) needed to do the job. The alternative, to parade the rhetoric while continuing to build a coordinated data program accessible primarily only to scientists, risks a severe future loss of credibility.