Click for next page ( 94


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) Space Studies Board Annual Report—1994 Appendix: Reports of the Panel to Review EOSDIS Plans On November 22, 1991, NRC Chair Frank Press received a letter from NASA Administrator Richard H. Truly requesting an assessment of the Earth Observing System (EOS) Data and Information System (EOSDIS) in the context of a recently completed restructuring of the flight elements of the program. In response, the NRC Commission on Physical Sciences, Mathematics, and Applications assembled a study group, the Panel to Review EOSDIS Plans, chaired by Mr. Charles A. Zraket. The activities of this panel, which was a collaboration of the Space Studies Board, the Computer Science and Telecommunications Board, and the Board on Earth Sciences and Resources, were managed by the Commission office. Because the EOSDIS was the subject of a government procurement at the time that the study was initiated, the panel first addressed those issues that REPORT MENU could be analyzed without the detailed system design information being NOTICE developed by the industrial proposal efforts under way. Preliminary conclusions FROM THE CHAIR on these issues were presented to NASA in an interim report delivered on April 9, CHAPTER 1 1992, under letters from Dr. Press and Mr. Zraket. These letters and the interim CHAPTER 2 report are reproduced in Appendix A.1. CHAPTER 3 CHAPTER 4 CHAPTER 5 Based on agency responses to this initial assessment, panel Chair Zraket APPENDIX A.1 prepared and submitted a second (letter) report on September 28, 1992; this APPENDIX A.2 letter report is reprinted here as Appendix A.2. By mutual agreement with the APPENDIX A.3 agency, the panel then suspended its activities pending completion of the EOSDIS procurement. After the selection of the vendor for EOSDIS, the panel resumed its work to assess the design presented in the winning proposal. The results of this second phase were documented in a third report, delivered to NASA on January 11, 1994. This third report is reproduced, with a cover letter from NRC Chair Bruce M. Alberts, in Appendix A.3. file:///C|/SSB_old_web/an94ap.htm (1 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) A.1 Interim Report of the Panel to Review EOSDIS Plans On April 9, 1992, the Panel to Review EOSDIS Plans completed the first of three reports and submitted it to NASA Administrator Daniel S. Goldin. Two cover letters accompanied the report. The first cover letter was from NRC Chair Frank Press. Enclosed is an interim report by the National Research Council on NASA's plans for EOSDIS as well as a transmittal letter from the Chair of the Panel that prepared this report. As you know, EOSDIS is a very complex program, and the demands on the Panel that prepared this interim report were extraordinary—in understanding the program, in coping with a demanding schedule, and in reaching judgements. At the same time, my colleagues and I appreciate the importance of EOSDIS. To quote from the attached report: "If EOSDIS fails, so will EOS, and so may the U.S. Global Change Research Program." It was against such an understanding that the National Research Council accepted this task, believing that we are obliged to assist the government, even when the time is short, the amount of information to be marshalled great, and the imperative to provide judgements urgent. I believe the Panel that prepared this report has done an exceptional job, ably assisted by the people of NASA. At the same time, the judgements as well as the limits of this interim report should be clear. While the Panel supports the schedule for procuring a contractor for the EOSDIS Core System, it finds major shortcomings in the actual plans for EOSDIS, and provides substantial recommendations for implementing the program that the Panel believes will help ensure its success. Therefore, this report cannot be construed as an endorsement of NASA's current plans for EOSDIS, but rather a substantial critique of flaws, which, if addressed, will in the Panel's judgement help ensure a strong and responsive program over the long term. The Panel believes that the terms of the contract as stated in the Request for Proposal are sufficiently flexible to accommodate its recommendations. The limits of the report should also be plain. It is an interim report, provided in response to requests from NASA and other interested parties for an early alert as to the Panel's views of EOSDIS plans. The Panel's final report this August will offer detailed analyses for these interim judgements, and will also respond directly to the specific issues as posed in the Terms of Reference for this task. I look forward to your comments on this interim report. And the Panel looks forward to a discussion with NASA officials involved in EOSDIS planning on this report and any further issues to be considered in preparing the final report. We are arranging for your colleagues at NASA with responsibility for the EOSDIS file:///C|/SSB_old_web/an94ap.htm (2 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) Project to be briefed by the Panel next week, and intend to release it publicly on April 17th. Signed by Frank Press Chair, National Research Council The second cover letter to Administrator Goldin for the April 9, 1992, interim report was from panel Chair Charles A. Zraket. I am pleased to submit the interim report of the National Research Council's Panel to Review Earth Observing System Data and Information System (EOSDIS) Plans. This contains the panel's preliminary observations and recommendations on the current plans for EOSDIS, based on the information provided. The panel looks forward to an early opportunity to discuss these recommendations with NASA and other interested parties, as well as to issuing its final report in August 1992. On behalf of the panel, I wish to thank all of those at NASA who responded quickly and professionally to our very substantial requests for information and to our many and often difficult questions. We could not have done our work without their full and ready cooperation. I also wish to express our gratitude for the splendid cooperation from the staff of the National Research Council that enabled the panel's work on this interim report to be completed in less than two months. Signed by Charles A. Zraket Chair, Panel to Review EOSDIS Plans Panel to Review EOSDIS Plans Interim Report This interim report identifies several issues regarding NASA's plans for developing the Earth Observing System Data and Information System (EOSDIS) and offers a number of recommendations that NASA should consider as it proceeds with procuring a contractor to build the system. This report does not respond in detail to the items in the terms of reference—that will be the subject of the panel's final report. Given the short time available for the panel's initial assessment, it has not been able to pursue the issues it identified to the depth it would like. The panel hopes, nevertheless, that NASA will find its interim conclusions and recommendations useful in the negotiations that will take place file:///C|/SSB_old_web/an94ap.htm (3 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) with the selected contractor to define the ongoing work plans for the EOSDIS Project. The appendices of this report include NASA's letter of request for this study, the terms of reference for the task, a list of the members of the panel and brief biographies, the work done and the meetings held to enable the panel to write this interim report, a brief description of EOSDIS for readers not familiar with the Project, and a brief description of the U.S. Global Change Research Program and its objectives. [These items are not provided in this annual report.] The panel was selected to have the competencies demanded by its charge—in understanding the needs of those who will use EOSDIS (including both EOS and non-EOS investigators), in the computer science and technology underlying EOSDIS, in the creation and implementation of large data systems, and in the recent history of large space-based data systems. The fact that the procurement for the EOSDIS Core System was concurrent with the panel's work required extreme care to avoid either the reality or perception of conflict of interest. Thus, in addition to following the National Research Council's standard procedures for dealing with bias and conflict of interest, the panel—and those who provided it information and briefings—took pains to consider only publicly available information. The panel, to the best of its knowledge, has not been provided with nor has it considered any proprietary information related to the procurement. OBJECTIVES AND MAJOR FINDINGS In combination with other programs of the U.S. Global Change Research Program, the Earth Observing System (EOS) is intended to reduce the current uncertainties about global climate change. Its Data and Information System (EOSDIS) is essential to the success of EOS. If EOSDIS fails, so will the Earth Observing System and so may the U.S. Global Change Research Program. The panel has been told repeatedly by responsible government officials that EOS is critical to the larger, global change program—one involving many agencies of government, and other national and international participants—and that EOSDIS offers a unique opportunity to begin building a national, and eventually, international, information system for global change research. To achieve these aspirations, EOSDIS will have to evolve to meet the changing needs of global change research over the next two decades and beyond. The panel believes that the recommendations offered in this report are necessary to ensure that growth and evolution. Specifically, the panel offers its judgments in terms of the following objectives it believes essential to the success of EOSDIS: EOSDIS must facilitate the integration of data related to the aims of the U.S. Global Change Research Program. Without this integration, the file:///C|/SSB_old_web/an94ap.htm (4 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) multidisciplinary and interdisciplinary research objectives of the U.S. Global Change Research Program will not be achieved. The EOSDIS program must be structured and managed to facilitate interactions with the other agencies involved in the U.S. Global Change Research Program so that existing data and future data collected by NASA and by other national and international organizations—using research and operational satellites as well as in situ sources—are available to all global change research scientists. EOSDIS must serve a large and broad set of users to facilitate the aims of the U.S. Global Change Research Program in supporting a community concerned with understanding the earth as a system. To serve that larger community, EOSDIS must provide its information in a manner that is simple, transparent, and inexpensive; it also must assure availability of its data to both the earth science community and the larger scientific community. EOSDIS must ensure that service to current users—including those involved with Version 0—will not be interrupted as the development of the system proceeds, and that Version 1 and subsequent versions will be implemented as soon as possible to meet the needs of the users, both in the EOS program and in the larger U.S. Global Change Research Program. EOSDIS, as it evolves, must maintain the flexibility to build rapidly on relevant advances in computer science and technology, including those in databases, scalable mass storage, software engineering, and networks. Doing so means that EOSDIS should not only take advantage of new developments, but also should become a force for change in the underlying science and technology where its own needs will promote state-of-the-art developments. Flexibility also requires organizational and management structures and processes that can respond to evolving requirements and implement the means for meeting them. EOSDIS needs substantive user participation in the design and development of the system, including involvement in the decisions on data acquisition and archiving, standard or ad hoc product generation, and interfaces that directly affect science users. The structure of the EOSDIS management organization and the attention it gives to the project should reflect the importance of the program in terms of its role as one of the major and most costly programs NASA has ever undertaken as well as its central role in the U.S. Global Change Research Program. The EOS program was recently restructured from a mission consisting of two large, orbiting platforms containing a total of 30 instruments to a series of six smaller spacecraft containing a total of 20 instruments. The amount of data expected to be collected from EOS, however, has decreased only slightly: from 330 gigabytes/day to 240 gigabytes/day. The estimate for the total amount of processed data (from the EOS spacecraft and the other missions and instruments that will be flown) that will be managed by EOSDIS changed from file:///C|/SSB_old_web/an94ap.htm (5 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) 1300 gigabytes/day to about 1100 gigabytes/day, a reduction of only 15 percent. Furthermore, the capabilities of the EOSDIS System are tied to the existence of the seven Distributed Active Archive Centers (DAACs) and the data they contain, rather than to the flight rates. Although the panel will certainly examine this issue further for the final report, it appears that the recent restructuring of the EOS flight program has had little effect on the requirements for EOSDIS and thus does not affect the preliminary conclusions of this interim report. In general, the panel does not see any serious risk to the EOSDIS program due to unavailable or inadequate technology. The panel believes that the prototyping plans of the EOSDIS Project Office, to be implemented after the contractor is selected, should be accelerated in order to assure that Version 1 is completed in accord with design objectives. There are risks, however, in two aspects of the planning for EOSDIS. One area of risk derives from the scale and pace of changes in computer and data management technology that can be expected over the long-term life of the program, and from the great diversity of users who must interface with EOSDIS. NASA needs to focus immediate attention on planning how EOSDIS will evolve to continue to be a useful system as the scientific needs and the technology change over time. Another area of risk concerns the management structure of EOSDIS. EOSDIS is an exceptionally large and complicated project that will cost several billion dollars, involve thousands of people, and continue for many years. The management will involve a complex mix of government, contractors, and a scientific community that is diverse and spread around the world. Each has an important role to play, and each will interact in a variety of ways with the other elements. In its recommendations in this interim report the panel has attempted to provide a number of mechanisms and approaches that it believes will help define these roles and interactions. NASA, of course, must have the ultimate responsibility for implementing EOSDIS. To do so effectively, however, NASA should first ensure proper internal management attention and also should use its own personnel in earth science and computer science, who can contribute significantly to the successful design of the system. Secondly, NASA needs to bring the scientific user community into the project as a partner, rather than regarding users simply as customers. Finally, NASA must accept the leadership role necessary to provide the essential unity among the user community (including other federal agencies and international participants), DAAC elements (management and scientific), and contractors. The complexity of this project demands that a structure be developed to ensure that all interests are properly integrated into the design of EOSDIS. The panel believes that NASA can proceed prudently with the procurement process for EOSDIS, provided the agency builds in the flexibility to make the adjustments necessary to ensure the success of the project. The conclusions and recommendations offered in this interim report can help NASA to incorporate that flexibility into work plans during the contract negotiations that will file:///C|/SSB_old_web/an94ap.htm (6 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) soon take place. This flexibility can be accommodated within the scope of the current procurement as long as it is planned ahead of final contact negotiations and the contract terms are compatible with this approach. The panel believes that its recommendations should not materially affect the EOSDIS schedule and that they can be implemented in work plans resulting from the pending contract negotiations. It is important to all users that EOSDIS implementation proceed as closely as possible to the planned schedule. The panel has divided its assessment into three parts: user interactions, EOSDIS architecture, and EOSDIS management. The recommendations for each area offer actions that NASA should consider in order to meet the objectives of the program described above without halting the current procurement. The panel also recognizes that requirements may change over time and that NASA may have to adjust its work plans over the life of the project. In order to be of service to NASA during this important stage of negotiating with the selected contractor, the panel believes that it is necessary to provide this advice now, in this interim report. The final report will expand on the issues discussed in this interim report and will respond in detail to the terms of reference. CONCLUSIONS AND RECOMMENDATIONS The following are the panel's judgments concerning the user interaction, architecture, and management issues that it believes must be addressed if EOSDIS is to meet the objectives integral to its success. In each instance, the panel points to strengths and weaknesses in the program, and offers recommendations. User Interactions Strengths NASA has stated its intention to incorporate user feedback throughout EOSDIS development and evolution. The panel applauds this approach. The ability of EOSDIS to serve the broad spectrum of users will be the final measure of EOSDIS success. In this context, it should be acknowledged that NASA has led other agencies in developing the Global Change Master Directory, which will be a comprehensive description of all global change data sets. The panel also commends NASA for its plan to share software code and toolkits with users who wish to import them for their own systems. Panel Concerns file:///C|/SSB_old_web/an94ap.htm (7 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) In its review, the panel has identified several areas in which an augmentation or strengthening of critical user interactions could substantially improve the likelihood for success of the EOSDIS program. Areas of concern are NASA's Science Data Plan, links with other agencies, use of Pathfinder data sets, treatment of operational and historical data, long-term archiving, involvement of nontraditional communities, and the ability to provide customized data sets. Science Data Plan. Version 0 science data requirements are being compiled into a Science Data Plan by the EOSDIS Project through regular interactions with the user community. The intent is to solicit regular review of these requirements from the science community to make certain that evolving needs are adequately reflected in the EOSDIS Project planning. Care must be taken to ensure that the Science Data Plan continues to emphasize the links between global change research objectives and the acquisition of individual data sets. A clearer picture of base-level requirements can be achieved by a continuing assessment of science objectives, existing holdings that might meet the objectives, and requirements for future data streams. The panel recommends that the Science Data Plan identify the links between global change research objectives and existing and planned data sets. Interagency Links. The research priorities of the U.S. Global Change Research Program cut across the missions of individual federal agencies. The distribution of current holdings as well as data to be acquired underscores the need for interagency interoperability and cooperation. NASA has been an active participant in interagency efforts for the U.S. Global Change Research Program through a variety of working groups, and is currently a full partner in developing a tri-agency (NASA, NOAA, USGS) data and information implementation plan, of which EOSDIS is a critical component. The panel endorses the efforts of these agencies to work cooperatively. The Global Change Master Directory is an excellent first step in helping users to identify relevant data sets for global change research. A similar effort is needed in achieving interoperability for access to the data. Success will require both technical developments and leadership in order to integrate and provide broad access to disparate data types currently distributed throughout the agencies. The panel believes that NASA is the logical agency to initiate this step in the context of EOSDIS. Moreover, EOSDIS will be much more effective in broadening its user base if it serves as the vehicle for integrating data. The panel recommends that NASA expand its efforts to increase interagency links by assuming an active leadership role among the agencies in achieving interoperability not only at the level of the Global Change Master Directory, but also at the level of providing access to the actual data. file:///C|/SSB_old_web/an94ap.htm (8 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) Pathfinder Data Sets. Prototyping has been a routine component of EOSDIS planning and Version 0 implementation by the Project Office. NASA has been successful in establishing prototype earth science data systems that are currently acquiring, processing, distributing, and archiving pre-EOS data. Lessons from such prototyping activities can identify problems associated with the manipulation and distribution of extremely large data sets. Pathfinder data sets provide an early means to evaluate the handling of large data sets, the development of products, and the distribution of data and products. NASA and NOAA are cooperating in a Pathfinder data program for selected satellite data. This program will be extremely valuable to the U.S. Global Change Research Program and to the prototyping of various functions of the overall data and information system. The panel recommends that NASA develop ways to integrate the efforts of existing data centers and centers of data supported by NSF, DOE, and USGS with the NOAA/NASA Pathfinder activities. Further, the Pathfinder data program now under way should be accelerated. Operational and Historical Data. Data from past and currently operating satellites already are being provided to several DAACs. NASA has shown considerable foresight in recognizing the importance of data streams from NASA, NOAA, DOD, and foreign satellites in establishing long-term data sets for global change research. Although the EOSDIS Request for Proposal addresses data management of NASA's EOS platform instruments as well as NASA's commitment to maintaining data sets acquired by pre-EOS sensors, the panel wishes to emphasize the need for the accessibility of non-EOS instrument data streams to EOSDIS users. The panel believes that the full benefit of EOSDIS to the U.S. Global Change Research Program will not be realized until an effort similar to that for EOS data is undertaken to manage the immense collection of historical data related to global change research already collected through operational observing systems. This collection includes the routine data from the space- based and surface-based observing systems of NOAA and DOD, as well as the routine and special data collected by USGS, USDA, EPA, DOE, NSF, and the Census Bureau. Integration, interpretation, and synthesis of such data, as part of a modern data and information system for long-term operational measurement, are critical to the goals of the U.S. Global Change Research Program and the interpretation of EOS measurements. The panel recommends several ways to address the issue of integrating the operational and research data from other agencies into EOSDIS: a. NASA should articulate a plan for incorporating operational and non- EOS instrument data streams into EOSDIS. Where EOS and non-EOS file:///C|/SSB_old_web/an94ap.htm (9 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) instruments have similar functions, NASA should develop a strategy to enhance the use of both data streams. This strategy should also include consideration of cross-calibration between basic radiometric data and higher-level products of an EOS instrument with a non-EOS instrument. b. To test the interoperability of EOSDIS and to integrate the critical long- term operational data that now exist at Affiliated Data Centers into a global change data and information system, NASA should perform a full-function test of the EOSDIS architecture and software on some of the Affiliated Data Centers, in particular, centers with holdings (such as long-term satellite or in situ data records) critical to the U.S. Global Change Research Program and to the synthesis and interpretation of data from EOS instruments. c. NASA should articulate its policy on how Affiliated Data Centers will move up through the different levels of interoperability that are specified for linkage with EOSDIS. Long-Term Archiving. Long-term archiving of EOS data is an issue that has not been addressed. Long-term commitment to maintaining data collected as part of EOSDIS is a critical component of the U.S. Global Change Research Program. NASA, in its response to questions from the panel, correctly pointed out that the issue of maintaining long-term archives is one that must be addressed by all participating federal agencies. Without a concrete plan and agency coordination for establishing permanent data archives, however, the overall objectives of EOS, and, therefore, of the U.S. Global Change Research Program, are jeopardized. As in the case of increasing interagency links, the panel believes that NASA can provide the leadership in addressing this need. The panel recommends that NASA develop an adequate plan and technology for long-term data archiving in conjunction with the other federal agencies participating in the U.S. Global Change Research Program. Involvement of Nontraditional Communities. NASA has identified ways for broadening the user community and providing information about EOSDIS to those unfamiliar with the system through professional journals and newsletters. Such publications may be adequate for reaching users in certain disciplines but may be ineffective for those in other fields, particularly in the nonphysical sciences. For example, one of the science priorities identified in the U.S. Global Change Research Program is to assess the human dimensions of global change. A detailed plan for involving potential user communities beyond the traditional disciplines associated with the earth and environmental sciences has not been clearly delineated for the panel. Many approaches could be taken to encourage users from nontraditional communities (e.g., legal, educational, political, and social). A useful approach could include the distribution of sample products that would allow users to become familiar with the various types of data sets available and to judge file:///C|/SSB_old_web/an94ap.htm (10 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) whether those data would be helpful to their research. The panel recommends that NASA take an active role in facilitating access to EOSDIS by other, nontraditional disciplines through a program that includes representatives from those disciplines in NASA's user advisory groups and develops products useful to them. Customized Data Sets. NASA clearly recognizes the importance of involving the user community in the development of EOSDIS. An approach to encourage active user participation is to provide customized data integration and synthesis of various products. The availability of software tools that conform to standards in an open architecture environment would facilitate participation by active users. For example, these tools might enable a user to assemble a customized set of specific time- and/or space-averaged data that could not otherwise be assembled without the user having to develop new software. The panel recommends that NASA encourage broad user participation by providing greater opportunities to create customized data sets. EOSDIS Architecture Strengths The panel in its several lengthy discussions with EOSDIS technical staff was impressed by the staff's competence and motivation. The staff has devised a process for designing the EOSDIS Core System that would rely on open systems, including multiple levels of interoperability for both users and the DAACs as well as the ability to handle evolving international standards. These two approaches—use of an open system and adoption of standards even though they will change over the lifetime of EOSDIS—will strengthen the program. The Project plans to deliver EOSDIS in incremental stages (via Versions 1 to 6 and Data Product Levels 0 to 6) that are expected to provide the flexibility necessary to meet user needs, to respond to budget uncertainties over the next decade, and to adjust to EOS flight schedules. Panel Concerns Design Control. Any large software system requires design criteria that are set by project management and articulated clearly and precisely throughout the project hierarchy. This is particularly true for EOSDIS because of four reasons: (1) the unprecedented size of the system's storage and processing capacity; (2) the extraordinary heterogeneity of both user computation systems and user file:///C|/SSB_old_web/an94ap.htm (11 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) requirements; (3) the large variation in scale of both the mass stores and the granules of data to be simultaneously managed; and (4) the high degree of evolution expected in the system. The combination of these factors will make the design, implementation, and evolutionary control of the system a substantial architectural challenge. Although NASA has assured the panel that EOSDIS will serve the needs of global change researchers, the EOSDIS Core System Statement of Work and the Functional and Performance Requirements documents of the Request for Proposal seem to be based on the management of data holdings resident with or owned by NASA or the DAACs and the created data products related to those holdings. It is entirely likely that data and/or data archives that are not within the exclusive purview of NASA or the DAACs will need to be made accessible to users through EOSDIS, without changing ownership of the data or the autonomy of the data repository. In anticipation of the need for accessibility, EOSDIS software should be built in the form of modular components with open, configuration-controlled interfaces so that other national and international agencies will be able to link with the system and provide products and services to the broader global change research community. The panel believes that responsibility for the design criteria and for their enforcement to guide the system architecture must reside with the government. The government must assure that the contractor's detailed architecture and implementation decisions follow the directions given by the government system architects. The panel recommends that NASA produce a clear, concise statement of the design criteria for EOSDIS that focuses on facilitating global change research and that NASA communicate these criteria throughout the Project hierarchy. The panel recommends that NASA strengthen its internal system architecture team by acquiring additional experienced people and that it give them the responsibility, authority, and budget to ensure that the design criteria are met as the system design and implementation proceed. A technical project of the magnitude and complexity of EOSDIS should have the very best system architecture team possible. NASA should make every effort to acquire such talent. Logically Distributed System. The research that will be possible through the resources provided by EOSDIS is difficult to characterize at present. Some research will focus on narrow disciplinary questions, while other work will be interdisciplinary. Since we cannot, indeed should not, attempt to specify the future directions that earth science research will take, EOSDIS must be flexible enough to respond to a wide variety of approaches. Furthermore, EOSDIS will be only a part, albeit a major one, of the efforts directed at managing data and information for global change research. file:///C|/SSB_old_web/an94ap.htm (12 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) The EOSDIS development plan provides for centralized control over the specification and implementation of the system. Each DAAC will implement an Information Management System that will be centrally developed by a single contractor. Although a centralized system is desirable for the management, operation, and control of the satellite and its instruments, the data will be distributed and dispersed among geographically separate and discipline-specific DAACs. Achieving the proper balance between the common elements that should be developed centrally and those that should be developed in a distributed fashion is critical to the success of the overall U.S. Global Change Research Program. At present, it appears as though the EOSDIS development plan is too heavily oriented toward a centralized approach. The panel recommends that the EOSDIS Project adapt its development plan to ensure a more logically distributed system, including: a. Designing EOSDIS so that all users (EOS and non-EOS investigators, DAACs, other data centers) can easily build selectively on top of EOSDIS components. EOSDIS should not constrain local implementation of diverse functions by users and DAACs. The development plan should reflect a philosophy that it is "easy to interact with EOSDIS" with minimum loss of autonomy. EOSDIS must be able to tolerate different versions of functionality and partial sharing of the components and toolkits it exports. b. Identifying those areas of interdisciplinary research that will require special interfaces among discipline-specific products and formats. The Project should specify the interfaces, build prototypes, and run simulations to exercise them, permitting users to evaluate them prior to developing final specifications and proceeding to full implementation. A contractor team that resides at each DAAC and works closely with the DAAC as well as the contractor's "central core" team should facilitate the development of these prototypes. This type of distributed development can be accomplished within the scope of the current procurement as long as it is planned ahead of final contract negotiation, and contract terms are compatible with this approach. Incremental Prototyping. The current EOSDIS development plan closely ties the availability of the distributed archive and product generation functions to the EOS flight schedule. There is much work that should be done, however, prior to the first scheduled launch of EOS instruments in 1998 to strengthen prototyping efforts already under way. For example, there are both existing archives and data expected from pre-EOS satellites that will be invaluable to the U.S. Global Change Research Program. Although the EOSDIS Project team has initiated the early prototyping effort for Version 0, more can and should be done to benefit current global change research and to enhance user feedback for final system design. The panel recommends that EOSDIS Project management extend file:///C|/SSB_old_web/an94ap.htm (13 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) its incremental development plan so that all user interfaces, all toolkits, and the end-to-end network system are: a. Specified in detail early in the development of Version 1 and prototyped or simulated sufficiently, and b. Evaluated in depth by users and DAACs prior to full implementation in Version 1. This will require a system network simulation and sufficient testing tools for users to assess and validate the specified functionality. Usability Evaluation. Prudent practice in the design of complex data management systems ordinarily includes a means of measuring the usability of the data. To the extent possible, such measures should be quantitative. Early evaluation exercises should be designed to measure ease of use, quality of interface specifications, and convenience of interoperability of heterogeneous system components. These exercises should ensure that individual users and data archivers can acquire piecemeal both functional capabilities and data sets. It is also prudent practice to involve independent judgment by having this evaluation performed by a group other than those responsible for developing the system. The panel recommends a usability evaluation program starting as soon as possible that involves: Selecting key functions, interfaces, and system behavior attributes for evaluation; Defining a set of metrics and expected values of those metrics for each parameter to be evaluated; Creating prototypes, simulations, and test suites to stress aspects of usability; Using the evaluations to guide final specification of system components; and Implementing this program so that most of the evaluation and validation is done by groups other than the prime contractor. EOSDIS Management Strengths NASA is to be commended for developing the plans for EOS as its file:///C|/SSB_old_web/an94ap.htm (14 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) flagship for U.S. participation in global climate change research. NASA and the EOS Project are further to be commended for their dedication to producing an adequate data system for EOS and for its user community. The unprecedented level of funding allocated for EOSDIS and the high level of planned contingency funding are evidence of the commitment NASA has made to this important national research effort. The panel is impressed with the degree of dedication and commitment of the EOSDIS Project team. The team is working diligently and competently toward both prototyping key system and subsystem capabilities and planning for the procurement of the full EOSDIS system. Panel Concerns Visibility and Management Attention. Although EOSDIS appears to receive substantial attention from management at NASA Headquarters, in the panel's view, EOSDIS lacks the attention of senior management at the Goddard Space Flight Center. The EOS Project is the largest single development effort the Goddard Center has undertaken. Even without the flight hardware components, EOSDIS by itself probably satisfies that description. EOSDIS is an extremely complex interdisciplinary science project and must integrate the most advanced data and system technologies. EOSDIS also contains both the flight operations segment and the ground data system. The fact that schedules overlap and that the prime contractor probably will use different groups of personnel to implement these two very different elements will amplify the government's oversight and management challenge. Yet the panel has heard substantial evidence that from the management standpoint, EOS and EOSDIS are treated like an ordinary project within the Goddard Center. For example, the Project Manager for EOSDIS is two management levels down within the Flight Operations Directorate, which is only one of ten directorates at the Goddard Center. In addition, the Project Office is quite small for the task at hand, with plans for only 45 government employees when fully staffed. This small core of dedicated staff provides inadequate programmatic and managerial depth and expertise in the development of large, distributed data systems and in computer science and technology. Given the preeminent position of EOS and EOSDIS in the U.S. Global Change Research Program, the panel believes that it is essential to increase the level of management visibility of the Project and the size and skills of the Project staff. In addition to learning from other government agencies that have had experience in the development and operation of large distributed data handling systems, NASA could, as needed, add to the Project experienced systems development personnel from other parts of the government. The panel suggests that greater flexibility in defining success criteria and in using the process for setting award fees for direct feedback from the Project Manager to senior-level contractor management would help to assure that the contractor will do an outstanding job on EOSDIS. The panel commends NASA for including users in its performance board for contract evaluation and urges the active participation of users in setting award fees. file:///C|/SSB_old_web/an94ap.htm (15 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) The panel recommends that the EOSDIS Project Manager have higher management visibility within Goddard Space Flight Center. The staff authorizations and skills should be sized to the scope and complexity of the Project. Further, the Project could augment its staff with experienced personnel from other parts of the government in addition to NASA. The panel recommends that the EOSDIS Project use the award fee process to best advantage through greater differentiation of success and failure criteria for evaluating contractor performance and by involving users in determining award fees. Scientific Involvement at Goddard Space Flight Center. The Goddard Center's in-house earth scientists have a very limited role in the management and operations aspects of the EOSDIS Project. Although NASA has established a variety of science advisory and data working groups, such groups cannot replace the continuing and even daily involvement of the external scientific community and the Goddard Center staff to ensure that the eventual system is responsive to user needs. Likewise, the nation's computer science community currently has very limited involvement in the Project, despite the fact that EOSDIS, to be successful, must implement the latest advances in scientific data management technology and, in some cases, stimulate the development of new technologies. The development of EOSDIS would benefit from substantive use of expertise in systems design and exploitation of information processing technology. Because underlying technologies, such as storage density, processor speeds, and transmission rates, are doubling roughly every three years, EOSDIS must be able to exploit rapidly expanding capabilities during its lifetime of a generation or more. EOSDIS will also stretch the limits of what can be done by a mammoth database management system shared by a very diverse and demanding user community. Certainly, many of the underlying technologies such as storage will evolve on their own. Other technologies, however, will have to be encouraged, such as large-scale data management, visualization, and integration of heterogeneous information. Possible ways to stimulate technology include establishing an intramural computer science research capability comparable to those in other sciences, supporting and using the external computer science community, and using DAACs to establish formal and informal links with the computer science research community in their neighboring universities. The panel recommends that NASA involve Goddard Space Flight Center earth scientists to a greater degree in the management and operations of EOSDIS and also involve computer scientists both inside and outside of NASA to explore research and technology in those areas where EOSDIS will stress the state of the art in science and technology and where EOSDIS will evolve most rapidly. file:///C|/SSB_old_web/an94ap.htm (16 of 18) [6/18/2004 10:43:31 AM]

OCR for page 93
Space Studies Board Annual Report-1994 (Appendix A.1) DAAC Involvement. The DAACs are not well integrated into the EOSDIS management structure, particularly during the development phase. The DAAC managers do not have well-defined authority or accountability in building EOSDIS. DAACs should be involved early, in contrast to the current plan, in which their primary role appears to be to operate the hardware and software at their sites after delivery, and to deliver data products to users. There should be mechanisms for feedback on scientific utility and operational effectiveness from the individual DAACs and associated archive centers to the central Project since the DAACs will be the primary sites for user interaction. There should be a coherent overall development, management, and science advisory structure that includes the DAACs. The panel understands that DAAC managers and scientists are involved in advisory roles. Advisory roles, however, are not sufficient for developing capabilities for and at the DAACs. Overall, the centralized management of the design and implementation of EOSDIS functions at each DAAC is not conducive to active user involvement and responsiveness to changing technology. What is needed is a structure that strengthens the local role of each DAAC beyond the present DAAC advisory group and thus enhances the responsiveness of each DAAC in meeting the needs of its user community, gives the DAAC some control over its destiny, and yet ensures that an interoperable system is developed to meet the requirements of EOSDIS. The panel recommends that NASA create, at each DAAC, a Development Team of full-time staff and active science users to address DAAC and user concerns. These teams should evaluate EOSDIS planning and implementation, including architecture, DAAC interface definitions, and other deliverables essential to ensuring that the DAACs will be responsive to user needs and that the EOSDIS system will be interoperable. In accomplishing these tasks, the teams should monitor the contractor's activities on behalf of user communities and prepare test data sets to verify system interfaces. Each DAAC Development Team should validate that DAAC's operational capability to use the evolving EOSDIS system as each of the program releases is implemented. Finally, NASA should provide the DAACs with modest funding to respond to specific user needs so that the DAACs will be able to parallel the evolution of the user community's ability to manipulate, integrate, and model data. file:///C|/SSB_old_web/an94ap.htm (17 of 18) [6/18/2004 10:43:31 AM]