Assessment of the Usefulness and Availability of NASA's Earth and Space Science Mission Data (2002)

Chapters 1 and 2 of this report discuss the importance of NASA data in advancing our knowledge of the world around us and describe the structure of the active archives that distribute the data. This chapter looks at the question of who uses NASA data and assesses user satisfaction with the systems that are in place. The assessment is based on input from relevant NRC standing committees, the chairs of three NASA advisory committees that have addressed data usage issues, interviews with colleagues, briefings from NASA data system and education program representatives, and data collected by this task group on NASA’s major data facilities and services (see Appendix C for the questionnaire used).

The mission of NASA’s Earth Science and Space Science Enterprises is to conduct science and communicate the resulting knowledge to the public.¹ Scientists, who design the experiments, analyze the returned data, and publish results in scientific journals, are the key link in the knowledge-generation chain, and meeting their needs is given the highest priority in both enterprises. Scientific users can be divided into two categories: discipline scientists who have an in-depth understanding of a particular instrument or observatory and multidisciplinary scientists who need to access and integrate data from a variety of sources. The latter category is particularly prevalent in the earth sciences and is becoming more common in some of the space science subdisciplines. It is important for these users to have easy access to the data, adequate and standardized metadata and data formats, and tools to organize the data.

Other major user groups include NASA engineers and managers, who use data from previously flown spacecraft to plan future missions (see Appendix B); the general public, which is particularly interested in images, movies, and popular science features; and the education community. Although some of these users need to combine data from a variety of sources, most require small data sets, packaged for their particular application.

In addition to the user groups mentioned, the data systems of the Earth Science Enterprise (ESE) serve commercial companies, which use remotely sensed data to create value-added products targeted to specific customer groups; federal, state, and local government agencies, which use NASA data for operational purposes, such as predicting weather patterns or making land use plans; and policy makers, who need to make decisions on subjects such as managing the

Earth’s resources. These groups typically require custom data sets and comprehensive user services. Some of these users are served by the Distributed Active Archive Centers (DAACs), but increasing numbers are served by short-term, focused programs such as Earth Science Information Partners (ESIPs) and Regional Earth Science Applications Centers (RESACs).

Tables 2.2 and 2.5 in Chapter 2 indicate that both the earth and space science active archives serve a large user community. The DAACs supplied data to more than 104,000 unique users in FY 2000,² greatly exceeding even NASA’s original expectations. The number of space science users is more difficult to determine, because some of the active archives only count Web site hits, which are considerably higher than the number of data requests. Nevertheless, it is clear that there are many tens of thousands of space science users. Using the membership of the American Geophysical Union as a proxy for the size of the earth and space science community (38,000 members from 115 countries),³ it is apparent that the data facilities and services serve more than just NASA investigators.

As noted above, the user community—especially of the DAACs—is quite diverse and thus is challenging to characterize in detail. All DAACs track electronic address extensions (e.g., .com, .edu, and so on) as a general measure of who accesses a site or obtains data. Figure 3.1 shows that scientists and government agencies—the highest-priority users—make up only a small fraction of total DAAC users. This observation highlights the importance of paying significant attention to the needs of the nonscientific community. Although the DAACs are aware of the broad characteristics of their user communities, a recent National Research Council (NRC) report found that few DAACs have a detailed understanding of their user profiles.⁴ The task group’s survey (see Appendix C) found the same still to be true.

The task group was unable to obtain user statistics on most of the space science active archives. However, based on the response of these archives to the task group’s survey, they appear to use similarly inadequate metrics to characterize their user communities. Some centers (e.g., Solar Data Analysis Center [SDAC]) do not even keep track of IP addresses and thus do not know the size or composition of their user community. All of the centers should know this basic information. A better understanding of their user profile would help them know whether it is scientifically necessary to expand the user base or provide new specialized services. The task group recognizes, however, that obtaining this information would increase operational costs and would require a re-evaluation of priorities by the supporting NASA office.

In the earth sciences, some specialized products and services are provided by ESIPs. The ESIPs were created in 1998 in part to develop value-added products from EOS and related data and to provide data services that are not being provided by the DAACs.⁵ Like the DAACs, the ESIPs track users by electronic address extensions, which provides only limited information about users. Using this categorization, the breakdown of users in the first quarter of 2001 was as follows: 33 percent education, 17 percent government, 14 percent commercial, 0.2 percent

military, and 35 percent other.⁶ Since their establishment, the ESIPs have created more than 200 new information products and served more than 16,000 users each quarter.

In response to the task group’s survey, only three centers identified potential new user groups: Goddard Space Flight Center (GSFC) DAAC (students and disaster-warning organizations); Socioeconomic and Applications Center (SEDAC) DAAC (journalists, librarians, and policy advisors); and SDAC (heliospheric scientists). The other centers appear to seek only marginal changes in their user base, either by developing and distributing specialized materials and sampler data sets (e.g., High Energy Astrophysics Science Archive Research Center [HEASARC], Planetary Data System [PDS], and Langley Space Flight Center DAAC), or by “advertising” their holdings and services at scientific conferences and through newsletters (e.g., NASA/Infrared Processing and Analysis Center Extragalactic Database, National Space Science Data Center, and National Snow and Ice Data Center DAAC). A number of centers (e.g., Infrared Science Archive [IRSA] and EROS Data Center DAAC) rely on users to find them by word of mouth, citations in journal articles, or Web search engines. New NASA policies in some space science disciplines (e.g., astrophysics) have also broadened the user base over the past several years. These policies include minimal proprietary periods for data, guest investigator programs on nearly all satellites that allow scientists to design their own observing programs, the coupling of support for analyzing data with the award of observing time, online access to data, and support for archival research. Finally, advances in data storage and distribution technologies, coupled with the growth of the World Wide Web, have broadened the user base by increasing the amount of online data and by making data easier for everyone to find.

All earth science data held at the DAACs (including, eventually, some scientific products created by the ESIPs) are available through the EOSDIS Data Gateway, and a considerable amount of space science data is available electronically through the individual active archives or mission Web sites.⁷ Most of the remaining space science data is available on media or in various forms from the PIs. Such data, particularly in solar and space physics, are commonly less accessible than are data held in active archives, because distribution and user services are not explicitly supported, and rewards to scientists come from publishing papers, not from depositing organized data sets into national data centers.⁸ Consequently, many PI data sets are not fully exploited, and much new information remains to be uncovered. Moreover, PI data sets are very much at risk when mission resources end.

A task group review of Web sites suggests that all of the active archives are usable by scientists. However, the data are not all easily accessible. Determining which data are appropriate for a specific need can be difficult. The Web pages offer detailed instructions, but it takes time and practice to learn how to navigate them to obtain the right data and tools. Those scientists who are or expect to be frequent users will invest the time in learning the system, but casual users may not. Similarly, descriptions of the data and tools may be insufficient for users who are not closely involved with a particular project. Indeed, insufficient documentation of satellite data has been a widespread and persistent problem.⁹ In general, obtaining data does not seem to be a problem, although users may have to retrieve larger volumes of data than are needed or convert the data into more useful formats. These observations echo those of a recent NRC report, which indicates that a majority of science users found access to be “somewhat easy” to “very easy.”¹⁰ Enhancements in bandwidth, better documentation, and increases in online data sets held in publicly accessible data facilities should improve access to scientific data.

In addition to providing access to scientific data, most centers provide easily found links to education and outreach materials that are readily understood by nonscientists (see Table 3.1). For example, HEASARC provides information about black holes and supernovas in nontechnical terms, and the Physical Oceanography DAAC (PO.DAAC) provides a tutorial and time-series data on El Niño and La Niña events. Given that a mission of all the active archives is to serve educators and the general public, it would be valuable if sites that are designed for professionals, such as the Multi-mission Archive at Space Telescope (MAST), the Alaska SAR Facility (ASF) DAAC, and IRSA would also provide prominent links to nonspecialist information.

As noted above, the active archives primarily serve the scientific community, and their success depends largely on how useful scientists find the data. The active archives measure user satisfaction through customer feedback via Web sites, user services, and comment cards; citations of data in journals; and user surveys (see Table 3.2). They also infer user satisfaction from informal feedback at conferences and from increases in the number of users and repeat customers. In addition to these metrics, the DAACs have begun to track (1) kudos and complaints and (2) errors, in response to recommendations from a 1998 NRC report.¹¹ The task group notes that these are reasonable measures of satisfaction but that most active archives adopt only some of these measures. For example, only about half of the centers survey their users, and even fewer (Astronomical Data Center [ADC], ASF, PDS, PO.DAAC) were able to provide the results of a recent survey to the task group. The most comprehensive of these surveys was conducted by the ASF DAAC, which queries users about user expectations, data quality, receipt of correct data in a timely manner, quality of user services, and usefulness of interfaces and software. The PDS survey focused on characterizing the user community, data set content and availability, and data analysis languages tools. In both cases, the majority of users describe

themselves as satisfied with the center. These results agree with a 1998 NRC survey of DAAC users, which found that the majority of respondents are satisfied with the DAACs and judge their performance to be above average.¹².

User surveys are valuable not only because they gauge user satisfaction, but because they identify products, tools, and services needed for improving the usefulness of the data. ASF users identified a long list of tools for making the data more useful, and cited improvements in documentation, interfaces, and technical and scientific support for making the data easier to find and use. PDS users would like to see a better Web interface, finer-grained searching, and the ability to choose nonstandard data volumes. Such information would be beneficial for all of the centers and their users, and the task group strongly encourages all centers to collect this information through regular, comprehensive user surveys.

An indirect measure of user satisfaction is growth in the number of users, although this measure may not be appropriate to centers that serve small, highly specialized user communities. Most of the centers report significant growth in the number of users from FY 1995 to FY 2000. The number of users more than doubled at ADC, HEASARC, and the DAACs and increased by an order of magnitude at MAST and the NASA/Infrared Processing and Analysis Center Extragalactic Database. The number of PDS users stayed constant. Changes in the size of the user community could not be determined for IRSA, which was created in 1999, and the National Space Science Data Center (NSSDC) and SDAC, which do not track numbers of unique users. The numbers referred to here are not directly comparable, because the space centers count users differently from each other and from the DAACs. Nevertheless, it is clear that the user community is growing, suggesting that users are increasingly finding the holdings useful to their work.

User working groups, NASA advisory committees, and senior reviews provide a means of gauging the satisfaction of scientific users, as well as identifying improvements needed by the scientific community. Science advisory committees make specific suggestions for obtaining new data sets, setting priorities for data processing, monitoring data and image quality, developing new tools, and improving user services in the individual active archives. Because advisory committees are dominated by working scientists, they provide an effective mechanism for improving the usefulness of the active archives to the scientific community. Each center has an advisory committee (usually designated user working groups in the earth sciences), and the most effective ones operate independently of the center rather than under it, providing ongoing, critical reviews of archive operations.¹³

The space science senior reviews also provide an important assessment tool for comparing the usefulness of the active archives within a discipline program. The 2000 astrophysics senior review urged strong support for the active archives and commented that the reviewers “saw no other way of assuring that the enormous data troves now being gathered by increasingly sophisticated astronomical missions in space be made rapidly accessible for scientific analysis.”¹⁴ The 2001 senior review of solar physics found that the SDAC is serving its users well but that many of the NSSDC holdings are still poorly documented,¹⁵ thus limiting their usefulness. However, the report notes that the quality of the data and services provided by the NSSDC has greatly improved in recent years and that it remains a cost-effective data resource for the Sun-Earth Connection community. Both the astrophysics and solar physics senior reviews mentioned the importance of finding ways to combine resources from different space missions and different active archives. In order to facilitate interoperability of the active archives, both reviews recommended streamlining data analysis tools, minimizing duplication of effort, and, in the case of astrophysics, providing transparent access to all of NASA’s databases currently being handled by the different centers.

Finally, most of the active archives track journal citations as a measure of scientific usefulness (see Table 3.2), either by consulting scientists directly or by using literature search or abstract services such as the Astrophysics Data System. The number of citations they report varies, ranging from tens to hundreds of citations per year. However, these numbers can be difficult to interpret, because publication counts depend on several factors, including (1) the usefulness of data in scientific research; (2) the aggressiveness of the center in obtaining the information, since many authors do not cite data sets; and (3) the age of the data, with recently collected data being cited more frequently than older data.

Although many of these measures of satisfaction are difficult to interpret in isolation, taken together they suggest that the active archives are serving their science users well. Such a result agrees with the positive feedback that the task group received from NRC standing committees and with the task group’s own experience with the data facilities.

As noted above, nonscientists require information rather than data. The active archives and the education and applications programs provide a wide variety of value-added data products tailored to specific applications. In addition, several offer user support services and search tools geared toward less-sophisticated users. For example, both the SEDAC and GSFC DAAC convert data into geographic-information-system-compatible formats, and the GSFC DAAC offers subsetting and data-mining capabilities so that users can obtain small, manageable chunks of data.

NASA’s Strategic Plan contains a mandate “to involve the education community in our endeavors to inspire America’s students, create learning opportunities, enlighten inquisitive minds,” and to “communicate widely the content, relevancy, and excitement of NASA’s mission and discoveries to inspire and to increase understanding and the broad application of science and technology.”¹⁶ The education community, both formal and informal, is served by the active archives, flight projects, ESIPs, and education and outreach programs within the Space Science and Earth Science Enterprises.

The key to preparing useful science education products is packaging. In order to serve the education community, NASA has made—and must continue to make—a substantial investment in packaging specific data products to meet the complex needs of a very diverse community. Educators are normally not trained—nor do they have the time—to make use of research data. Both the research data and the software needed to access them vary with mission architecture and science goals, and effective use of mission-specific tools requires training and practice. Common, intuitive, affordable, and easily used visualization tools that work with a wide variety of data sets are needed to serve the education community.

The national science education standards developed by the National Research Council specify age-appropriate content goals for the teaching of science in grades K-12.¹⁷ These standards emphasize the teaching of science through inquiry-based methods. Engaging students in the active process of inquiry can help them develop a deeper understanding of both scientific concepts and of how we know what we know about science. Both earth and space science data constitute a rich resource for inquiry-based curricula. An increasing number of schools is able to connect to the Internet, which makes data, images, and tools widely available. Using these resources, students have discovered a supernova, dozens of novae in the Andromeda galaxy, and a new Kuiper Belt object through examination of ground-based data. In the earth sciences, students are using NASA data and images to monitor environmental change at local (e.g., Boreal Forest Watch) to global scales (e.g., Global Learning and Observations to Benefit the Environment Program [GLOBE]).¹⁸ Virtual research expeditions in which students use satellite observations and climate models to study the Earth as a system of interacting components are offered by the Planet Earth Sciences ESIP.¹⁹ Many other types of earth and space data, properly packaged, have at least comparable potential for bringing the thrill of discovery into the classroom.

Earth and space science data play a key role in programs and exhibits at museums, science centers, and planetariums. The audience is large; for example, approximately 28 million visits are made to the planetariums in the United States each year. Astronomical and earth science data offer a rich variety of images that both illustrate important scientific advances and are aesthetically pleasing. Astronomy has long had an obvious appeal, in large part because of the kinds of questions it addresses: Where did we come from? What will be our ultimate fate? Are we alone in the universe? Earth sciences have a great deal of practical importance: characterizing

the past history of the Earth, mapping current resources, and making predictions about the impact of current decisions on the environment. Space-based data can be used to enrich inquiry-based exploration of the world in which we live in informal settings; to develop interactive “kiosk” applications that enable unguided exploration of space data sets; and to create kits useful in hands-on demonstrations, image-rich presentations, and easily replicated exhibits/For example, the “Dynamic Earth” exhibit at the Discovery Science Center in Santa Ana, California, features topics such as plate tectonics, the sun’s influence on convection within the Earth’s atmosphere and oceans, and the impact of humans on the Earth’s atmosphere.²⁰ The exhibit is supplemented with a formal education component, supplying corresponding materials to students and cultivating teachers. Such museums need not be physical. A digital museum created by the Museums Teaching Planet Earth ESIP in association with the Houston Museum of Natural History provides interactive displays of earth and space science data.²¹

Numerous data products have been created by the active archives and flight projects to serve the education community. However, making interesting data products does not guarantee that anyone will use them. Both the Office of Space Science (OSS) and the ESE are developing metrics to evaluate the success of their education programs, but it will take at least a decade to obtain results. Moreover, both enterprises have commissioned external reviews of parts of their programs. For example, researchers at Cornell University are reviewing the ESE’s informal education program, and University of Arizona researchers have assessed the usefulness of equipment grants for working with remote-sensing data for teaching and training. A trio of reports from Lesley University evaluates the infrastructure and activities of the OSS education and public outreach activities. Among the findings is that even if teachers have access to the Internet, they have difficulty finding information they can understand; most of the material on the Web appears to be aimed at those who are already familiar with space science.²² This kind of external evaluation should be an ongoing part of the education and outreach activities of the active archives and flight projects to ensure the best use of limited resources.

Since the Web has become an essential source of information for the education community and general public, the first place many people turn when they wish to explore a topic is no longer to the library but rather to online sources of information. Many of the NASA centers, including the Space Telescope Science Institute and Jet Propulsion Laboratory, have developed very effective sites that have won national awards.²³ In addition, the OSS Education and Public

Outreach Program plans to implement a coordinated electronic dissemination system that ensures that all NASA educational activities and data products are available through appropriate networking technologies (i.e., the Internet and satellite or cable television). It is the nature of Web-based interactions that people expect timely information with frequent updates and changes to the material presented. Responding to this expectation requires an ongoing investment, with resources already in place when especially newsworthy events are likely to occur.

Discoveries in astronomy and solar system exploration are well covered by the media, and the earth sciences have a similar but underexploited potential for attracting the general public. Some lessons in how to work with the media can be derived from the successful experience of the Space Telescope Science Institute, which has made the Hubble Space Telescope (HST) a household name. Key elements of their success are (1) making a long-term investment in establishing good relations with the media by providing easy access to scientists who can explain the data obtained from NASA missions in understandable terms; (2) developing confidence on the part of the media that the science highlighted in press releases, for example, is accurate; (3) using sophisticated visualization techniques to present data in ways that are suitable for video as well as print and Internet publications; and (4) becoming recognized as a resource for background information and commentary on results in astrophysics, whether or not they come from HST.

In addition to reporting discoveries, the news media itself can develop value-added products and disseminate them to the general public. Meteorologists in the broadcasting industry already provide such services using NOAA data, and the StormCenter ESIP is adding high-resolution NASA imagery to network and local broadcasts.²⁴ A goal of this ESIP is to educate fellow media professionals about the usefulness of certain NASA data.

As with the education community, commercial users and decision makers require valueadded products tailored to their specific needs. None of the major data facilities, focuses on serving these users, so NASA has initiated a number of short-term projects aimed at creating specialized products for a wide range of applications. Chief among them are the ESIPs, RESACs, Infomarts, A Remote Sensing Product Development Partnership for Agriculture (A20/20), and Food and Fiber Applications of Remote Sensing (FFARS) (see Box 3.1 and Table3.3). The overall program responsibility for all but the ESIPs rests with the Applications Directorate of the Earth Science Enterprise.

NASA is primarily a research and development agency. Consequently, all of these applications programs are expected to become self-sustaining after their initial three to five years of funding.²⁵ How successful they will be in transitioning to the marketplace remains to be seen. It is likely that many products and services will be discontinued because they are not commercially viable. Those that are maintained may become less accessible, since commercial entities must often control access to make a profit.²⁶

Products created by these programs are currently accessible via the Web site of the facility. No comprehensive list of products or metadata exists, although the Applications Directorate has begun to catalog the metadata. To facilitate the process, future solicitations will require that data products comply with Federal Geographic Data Committee (FGDC) metadata standards and register in the FGDC clearinghouse.²⁷ These steps should help future users find information from the applications programs.

By making value-added products and tools more usable for specialized user groups, the ESIPs, RESACs, Infomarts, and similar programs increase the value of NASA’s existing holdings. How much value they will add is unknown, because all of these programs are still underway and none has been formally evaluated. However, based on the large number of users reported by the ESIPs (see Box 3.1), the value could be significant.

The task group concludes that NASA has done a good to excellent job in making data available to the research communities that it serves. User committees, other advisory and oversight committees, and a variety of formal and informal contacts with the researchers provide NASA with the information it needs to monitor the effectiveness of its programs to distribute scientific data and to make any needed midcourse corrections. NASA has also made a major commitment to enhancing science literacy and understanding at all levels of the educational system. External evaluation has been initiated by both the Office of Space Science and the Earth Science Enterprise to evaluate the effectiveness of this investment. In general, the space science community sees researchers and educators, including the media, as the primary “customers” of NASA data. Earth science data have potentially a much broader customer base, including commercial users, policy makers, and others. The DAACs do not appear to have either the mandate or the resources to provide extensive custom data sets or user services to nonscientists. Rather, such services are mainly provided by other ESE programs. While evaluation is needed to determine whether this distributed approach is the most effective strategy, it is clear that specific investment in meeting the unique requirements of each of a diverse set of end users will be required in order to maximize the usefulness of NASA data.

Recommendation. NASA planning and project funding should continue to include provisions for the timely generation and synthesis of data into information and the dissemination of this information to the diverse communities of users. This plan should take into account the needs—and the contribution to information generation—of end users, including other federal and state agencies, educational organizations, and commercial enterprises. The plan should include provisions for ongoing assessment of the effectiveness of data transfer and its educational value.