A Short History of EOSDIS, 1986 to 1995
Two revolutionary developments, one in science and one in technology, shaped early planning of EOS and its information system in the early 1980s. Both were related to rapidly increasing computational capabilities. Earth scientists developing computer models of the atmosphere and ocean realized that exchanges of energy, momentum, and mass with other parts of the Earth must be modeled and taken into account to develop an accurate simulation. They began to speak of the Earth system, realizing that the atmosphere, ocean, and biosphere were intimately connected. Simultaneously, empirical and theoretical study of the consequences of nonlinearity showed that interactions between and within the subsystems must be modeled over a wide range of spatial and temporal scales.
The same electronic components that made computers possible were being used to create sensors and spacecraft to obtain observations from space of the Earth and the heavens. The first scientific spacecraft produced new perspectives of weather systems and near-Earth space physics; those that followed returned revolutionary data about the energy balance of the Earth and about the patterns on its surface and the processes that shaped them. Together, the Earth and space sciences were building and launching satellites for making observations from and in space that produced unprecedented flows of data. The amount of data flowing from space was truly astounding in comparison with traditional sources of surface-based observations. Management of data, in addition to observing capability, became a critical issue. And scientists were beginning to realize that the key data management challenges were administrative, programmatic, and political, not technological. A recent comprehensive history of the growing tensions and their consequences in the Earth sciences demonstrates the complexities of the
issues involved.1 Here we focus more directly on the antecedents and history of the data and information system associated with the EOS.
In response to a “perception that data problems were pervasive throughout the space sciences,” the Space Studies Board of the National Research Council formed the Committee on Data Management and Computation (CODMAC). In its first report this committee observed that the “majority of the current data problems are not due to technological barriers. ”2 It cited problems arising from lack of scientific involvement in data system planning; inadequate funding; inadequate scientific oversight of data operations; and a long list of problems in data processing, distribution, retrieval, and archiving. After considering a number of case studies, CODMAC proposed a number of principles to guide data management, including active scientific involvement throughout data system planning and operations and a deliberate focus on users ' needs. The committee also recommended that data analysis funds should be adequate and protected against reprogramming owing to delays and cost overruns.
The growing realization that human activities might be inducing global-scale change and the tremendous scientific opportunities evident in the accelerating capabilities for observation from space both stimulated new and adventuresome thinking about Earth observations in the early 1980s. In response to these new ideas, NASA appointed the Science and Mission Requirements Group for EOS and began to plan a global space-based observing system that would create a revolution in Earth science and more comprehensive understanding of the planet and its subsystems. Among its recommendations, the Requirements Group urged that observations of the Earth be continued and expanded; that a data system providing ready and integrated access to past, present, and future data be developed; and that research in understanding the data be supported.3
Recognizing the importance of the associated data system, the scientists and program managers involved with the fledgling EOS assembled the Data Panel to develop a rationale and recommendations for planning, implementing, and operating the EOS Data and Information System (EOSDIS). The report of the Data Panel amplified the CODMAC themes and provided a detailed examination of issues that had to be resolved for EOS and EOSDIS to be successful:4
Involving scientists directly and intimately in the planning and oversight of operations of EOSDIS.
Creating a distributed system to stimulate creativity, enable prototypes, and facilitate evolution.
Enabling scientists to interact with a wide range of datasets that are and will be widely dispersed.
Creating the flexibility to adapt readily to rapid advances in electronic communications, networks, and computing capabilities.
Ensuring that archiving approaches and facilities are both responsive and reliable.
The functional system architecture developed by the EOS Data Panel envisioned that scientific users of the system would acquire data, process it, and add value to it and then return their results to the data system for use by other scientists. EOSDIS was to become an environment for stimulating scientific progress and interaction, not just a system for converting space observations transmitted to the ground as electronic bits from the instruments into datasets with variables in scientific units.
These ideas generated both optimism and further reflection in the scientific community. Some members of the community foresaw new opportunities, with EOSDIS being “responsive to the needs of science and scientists, rather than being a data archive in the form of the write-only memory . . . all too frequently encountered in the Earth sciences. [Instead], a successful EOS Data and Information System must do much more: it must be designed and implemented so that it will engender powerful new modes of research, foster synergistic interactions between observation and simulation with models, and promote thought about the Earth and its processes at higher levels of abstraction.”5 The EOSDIS, in this view, would integrate advances in workstations, local- and wide-area networks, and graphical and visualization techniques to create a new environment for scientific research. “The scientist 's workstation [thus] has the potential to become a window on the world.”
But establishing responsive governance of the system was critical to realizing the hopes germinated by the EOS Data Panel. EOS and EOSDIS were being created to enhance scientific understanding of the Earth and so it was argued that: “the scientific community studying the Earth must be deeply involved in the creation and management of these systems. Only the scientists at the frontiers of Earth System research can ensure that these systems remain responsive to the needs and opportunities of science.”6
As it turned out, NASA did not take the advice of the NRC and its own advisory groups—the NASA project management structure did not permit it, and the rationale for the structure was not then questioned. Design of the EOSDIS task was made part of the EOS project. A project team at the Goddard Space Flight Center was created and tasked with the engineering design of the system. At the same time, NASA solicited proposals from the scientific community for instruments and interdisciplinary studies. Virtually all of the scientists who might have been willing to review NASA's early progress on EOSDIS were candidates to be science investigators and thus were excluded from EOS and EOSDIS activities during the competitive process. But at the end of the process, a linked group of scientific advisory committees was created, with the charter to advise (and not to decide or command) emphasized in their titles.
During this period, the design of EOSDIS had proceeded in parallel with design of the spacecraft, both efforts using the standard federal protocol for the design of hardware systems developed by the U.S. Department of Defense. For EOSDIS, formal requirements specified data flow rates, simultaneous and inter-
active data processing, browse and search capabilities, reprocessing demands, and archiving arrangements. Eventually, a system concept involving a highly centralized monolithic data system emerged. The community was deeply upset, the advisers protested vigorously, redesign ensued, and a system with geographically distributed components emerged that linked a number of copies of the original NASA system proposal.
The original design, so different from what the scientific community had expected, opened a crevasse between the scientific community and the NASA system designers. It was never to be satisfactorily bridged. The kind of system engineering protocol for hardware system development based on a top-down specification of requirements is antithetical to the upward flow of new capabilities envisioned to come from a logically distributed information system that stimulated creativity, using standards to link the diverse components.
Eventually, a procurement process led to award of a large contract to a private firm to complete the design and implement the system. After the first design was presented, both the NRC and NASA advisory groups advised further redesign, Congress forced EOS budget cuts, and finally an NRC group urged a return to the concept of a highly distributed system, intimately involving a range of scientists and users, that would be created and managed as a federation.7
This most recent NRC recommendation about EOSDIS thus echoes the Data Panel, which more than a decade ago concluded with the statement:
There are two fundamental principles that should be followed throughout the EOS data and information system evolutionary process. They are: (i) involve the scientific community at the outset and through all subsequent activities, since the data will be acquired, transmitted, processed, and delivered for scientific research purposes; and (ii) provide the researcher with an oversight and review responsibility, since the most successful examples of data management rely on the active involvement of scientists.8
Unfortunately, the NASA project system could not accommodate this recommendation, but the recommendation has proved prophetic. While the Data Panel's members did not foresee explicitly the wondrous capabilities of the Internet and the World Wide Web that today can support a highly distributed EOSDIS, the panel members did understand that electronic computational and communications capabilities were rapidly creating an entirely new environment in which the vision of EOS and EOSDIS could be realized.
Some might say that the scientific community piled so many conflicting expectations on EOSDIS that it could not succeed, that the system could not be sustained with the ebb and flow of support from different parts of the community, and that it was unrealistically expensive from the very beginning. But all of these issues are independent of science or technology; they are related to governance and political sophistication. While the Data Panel and every other scientific
advisory group realized that direct scientific involvement and governance were the key issues, they did not shout loudly enough, they did not provide a compelling alternative, and they did not convince those who made the decisions. Again and again, the project management juggernaut swept over their concerns and rolled onto the next redesign.