Environmental Data Management at NOAA: Archiving, Stewardship, and Access (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction NOAA is a mission agency with legal, administrative, and mission mandates that require it to archive and provide access to a wide variety of environmental data, including model output. The data managed by NOAA’s National Environmental Satellite, Data, and Information Service (NESDIS) stretch from the surface of the sun to the core of the Earth and provide a broad range of benefits to society. Customers of this investment include other offices within NOAA, other federal agencies, state and local governments, industry and business interests, scientists, educators, and the public, including the international community. The needs of these users are diverse, making it difficult to assess the value of any particular environmental data stream, much less the sum of all data managed by NOAA. However, the substantial resources typically required to collect or generate environmental data imply that they have significant social, economic, and/or scientific value. As described in Chapter 2, the volume and variety of data collected by NOAA and its partners have increased dramatically over the past decade, as has the demand for access to these data by an increasingly diverse range of users. These trends are expected to continue over the next few decades as new satellite systems, modeling capabilities, and other data-generating activities come on line. It is essential to provide effective   Throughout this report, the term “environmental data” is used broadly to indicate all types of Earth System observations (including physical samples as well as in situ and re- motely sensed data), model output, and synthesized products derived from these data.  

10 ENVIRONMENTAL DATA MANAGEMENT AT NOAA stewardship for these data sets and to expand and improve access to them in order to fully realize their social, economic, and environmental benefits. Unfortunately, the resources available for data management activities at NOAA have remained relatively flat in recent years. In addition, NOAA has yet to implement a coordinated, enterprise-wide plan that explicitly spells out the data management responsibilities of various program ele- ments, or how these activities will mesh with data management activities at other agencies and international groups to ensure the long-term preser- vation and enhancement of all important environmental data sets, as well as access to these data by a broad range of users. NOAA deserves praise for the steps that it has already taken to evalu- ate and improve its data management activities, such as the report Inte- grating the Pieces: Assessment of NOAA’s Environmental Data and Information Management (NOAA, 2006a), the formation of the Data Archiving and Access Requirements (DAAR) Working Group, and the conceptual plan for the Global Earth Observation Integrated Data Environment (GEO- IDE) (NOAA, 2006b), all of which are discussed in Chapter 2. These preliminary steps will need to be connected and extended by an effective, integrated, enterprise-wide data management plan, and accompanied by broad support and sustained funding, in order to meet the formidable challenge posed by anticipated increases in data archive volumes and data access demands and to ensure that NOAA’s environmental data con- tinue to be available to meet societal needs. This report provides the foun- dation for such a plan by describing the principles that underlie effective environmental data management and providing guidelines and examples that explain and illustrate how these principles could and should be implemented within an integrated data management framework. It should be emphasized from the outset that the sheer volume and diversity of NOAA’s environmental data holdings, coupled with the dif- ficulty in predicting the future value of these data, demands that a struc- tured but flexible process be used when making data management deci- sions. For example, as discussed in later chapters, the decision to upgrade legacy systems to meet the demands of increasing data volume and com- plexity should be made on a case-by-case basis, with input from the broad user community. NOAA’s end-to-end data management process, which includes acquiring, processing, storing, maintaining, updating, and providing access to data, represents an interconnected series of activities involving a large number of different people and systems and serving a broad range of stakeholders. Hence, in addition to providing principles related to the types of scientific data that should be archived and how to provide the best possible access to these data, this report addresses some of the key organizational, management, and planning activities, including inter- and intra-agency relationships, that are critical for making effective 

INTRODUCTION 11 data management decisions. A key theme that emerges from these consid- erations is that NOAA needs to establish, codify, and maintain a formal, flexible, ongoing process for data management decision making. This report is organized as follows. Chapter 2 provides an overview of current data management activities at NOAA and highlights some key findings and recommendations from previous reports that discuss archiving and access of environmental data at NOAA and other federal agencies. Chapter 3 presents five overarching principles that underlie effective environmental data management. Chapters 4, 5, and 6 define and describe in greater detail the three fundamental functions of a data man- agement system: data stewardship; the decision-making process for data archiving; and data discovery, access, and integration. Each of these three chapters is organized around a single major principle, supplemented with more detailed guidelines and examples that describe and illustrate how these principles could be applied to current and future environmental data sets at NOAA. The final chapter provides one final principle and outlines how the nine principles and guidelines offered in this report, when considered in the context of NOAA’s existing data management activities and accompanied by sufficient ongoing support, could yield a reliable and cost-effective “system-of-systems” for data management that meets or exceeds anticipated user requirements.