4
Beyond Federal Partnerships: Engagement with the Broader Community of Users

This chapter examines the extent to which the National Aeronautics and Space Administration’s (NASA’s) Applied Sciences Program (ASP) engages the broader community of users in researching, developing, and validating the scientific basis and applications of NASA products. “Broader community” means organizations that are not federal agencies, but are currently benefiting from collaborations with NASA or have the potential to do so. Examples of members of the broader community are academia and academic research institutions; state and local governments; tribal nations; and the private sector, including manufacturing, processing, and service entities; nongovernmental nonprofit organizations; and international organizations.

These communities have many diverse interests. Academia generally is interested in advancing knowledge through creative research and education programs; state and local governments want products that enhance their decision-making capabilities that allow them to improve their services, including emergency response and resource management. The private sector wants data to improve their products and services, and hence their competitive position. Nonprofit organizations provide services for a specific issue, and the tribal nations seek information and products for the general betterment of their people. These diverse interests complicate the assessment of how much society is benefiting from ASP’s work. The committee chose to examine ASP’s process of engagement, the products involved, and its current practices in engaging the broader community.

Before addressing the process and practices involved in ASP’s engagement of the broader community, it is important to recall what the ASP mission entails and how it is supposed to benefit society. NASA has



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 77
Assessment of the NASA Applied Sciences Program 4 Beyond Federal Partnerships: Engagement with the Broader Community of Users This chapter examines the extent to which the National Aeronautics and Space Administration’s (NASA’s) Applied Sciences Program (ASP) engages the broader community of users in researching, developing, and validating the scientific basis and applications of NASA products. “Broader community” means organizations that are not federal agencies, but are currently benefiting from collaborations with NASA or have the potential to do so. Examples of members of the broader community are academia and academic research institutions; state and local governments; tribal nations; and the private sector, including manufacturing, processing, and service entities; nongovernmental nonprofit organizations; and international organizations. These communities have many diverse interests. Academia generally is interested in advancing knowledge through creative research and education programs; state and local governments want products that enhance their decision-making capabilities that allow them to improve their services, including emergency response and resource management. The private sector wants data to improve their products and services, and hence their competitive position. Nonprofit organizations provide services for a specific issue, and the tribal nations seek information and products for the general betterment of their people. These diverse interests complicate the assessment of how much society is benefiting from ASP’s work. The committee chose to examine ASP’s process of engagement, the products involved, and its current practices in engaging the broader community. Before addressing the process and practices involved in ASP’s engagement of the broader community, it is important to recall what the ASP mission entails and how it is supposed to benefit society. NASA has

OCR for page 77
Assessment of the NASA Applied Sciences Program a research program that puts instruments into space to produce observations of Earth processes such as climate change, agriculture and food supply, air pollution, water resources, land dynamics, natural disasters, aviation safety, and weather forecasting. The remotely sensed observations are used for analysis of resource changes and to develop models for predicting future conditions, such as crop production, El Niño forecasts, and propagation of invasive species. The analyses and models are then used as supporting evidence for decision making on such matters as the pricing of agricultural products, disaster management, water management, and resource conservation. A part of the process is benchmarking the value of the results or models generated and making them a part of decision making. Through application, decision-support systems (DSS) emerge that can be provided to and applied by organizations. The desired result is to benefit society through better management of resources and disasters, weather forecasting, food supply, and transportation safety, among others. One question this committee has been asked to address is how well the broader community is being engaged in this important endeavor and whether the experience of this community is being incorporated into the feedback process to decision making about the future direction of the ASP. THE PROCESS OF ENGAGEMENT The process of engagement by ASP is the pursuit of partnerships with organizations for the development of DSS that benefit society. NASA relies heavily on federal agencies as its partners to develop decision tools for implementation (Chapter 3). NASA’s partnering primarily with federal agencies inevitably leads to such agencies having a major influence on the development of decision-support tools possibly at the expense of other sources, especially the private sector. NASA works with the partners through ASP to validate and incorporate Earth science data into tools to enhance established relationships that the partner agencies have with other organizations, which mostly by serendipity, include some members of the broader community. The engagement process has many facets, among which are internal NASA operations that can affect implementation of ASP activities. The ASP operates through the NASA field centers, Earth science laboratories, and the Distributed Active Archive Centers. These organizations identify Earth science results, design products, and provide information to partners.

OCR for page 77
Assessment of the NASA Applied Sciences Program ASSESSMENT OF ENGAGEMENT ACTIVITIES NASA puts into place agreements and initiatives to collaborate with other organizations in the application of NASA technologies (Table 3.2). Examples are the large number of agreements with interagency and nonfederal organizations and U.S. and international committees relating to NASA Earth science. Even though the NASA agreements include nonfederal organizations, an examination of the participants in the ASP reveals essentially no direct engagement of the broader community. Rather, the nonfederal participants are such national and regional organizations as the Association of American State Geologists, the International City/County Management Association, the National Association for Search and Rescue, and the Western Governors’ Association. There is limited evidence that these national organizations have strong ties with many key members of the broader community, especially the private sector. Academic institutions have been able to partner with local constituents in certain of the Applications of National Priority, such as agricultural efficiency, ecological forecasting related to fire dynamics, water quality monitoring, and carbon management; several institutions located near NASA centers appear to have garnered a disproportionate amount of attention and resources. The international committees in which NASA is involved have a similar government and institutional framework with no direct evidence of any strong links with the broader community. Should ASP consider the establishment of direct links with such users as the private sector and other nongovernment organizations as too much of a burden for the program, alternative solutions should be considered, some of which are discussed later. The extent to which ASP has focused its partnering efforts on federal agencies and encouraged them to take on the task of engaging the broader community can be seen in the list of ASP-funded projects in FY 2006 (Table 4.1). As noted in Chapter 3, over half of the funded projects (75 of 141) were with federal agencies. An additional 48 funded projects were based at academic institutions, which have the goal of developing application products targeted to 11 of the 12 Applications of National Priority (energy management did not have an academic partner). A limited number (8) of nonprofit organizations were funded and only three for-profit enterprises were funded. State and local partners were also poorly represented, with just seven receiving funding. The data in Table 4.1 show a lack of ASP direct partnerships with such broader community entities as nonprofit, private, state, and local organizations. The data in Table 4.1 do not show unfunded activities associated with use of NASA products, as the unfunded usage was not possible to track given the data available to the committee.

OCR for page 77
Assessment of the NASA Applied Sciences Program Table 4.1 ASP-Funded Projects Application Areas Federal Academic Nonprofit Private State Local Agricultural Efficiency 3 5 1   1   Air Quality 9 4   2     Aviation 6 4     1   Carbon Management 5 5         Coastal Management 5 2     1   Disaster Management 3 4 1     3 Ecological Forecasting 4 3 4       Energy Management 5           Homeland Security 3 3         Invasive Species 3 5 1       Public Health 4 3         Water Management 5 3   1     Cross Cutting 20 7 1   1   Total 75 48 8 3 4 3 SOURCE: ASP (2006). ASP’s focus on federal agency partners is also reflected in usage statistics for the various websites and gateways serving as programmatic reference sites. For example, from January through August 2006, 48.1 percent of the hits on NASA’s Application Implementation Working Group (AIWG) website were from users with the .gov domain extension (Table 4.2), 4.3 percent were from U.S. academic users (.edu), 4.8 percent commercial users (.com), and only 3.0 percent from nonprofit organizations (.org). The distribution of hits was similar in 2005 and 2004.

OCR for page 77
Assessment of the NASA Applied Sciences Program Table 4.2 Domain Names of Visitors to the AIWG Website, By Year     2006a 2005 2004b Domain/Country Domain Extensions Number of Hits Percent of Hits Number of Hits Percent of Hits Number of Hits Percent of Hits U.S. Government .gov 27,130 48.1 50,242 52.9 9,570 42.4 Unknown .ip 15,734 27.9 22,742 24.0 6,089 27.0 Commercial .com 2,707 4.8 7,662 8.1 1,532 6.8 Network .net 5,812 10.3 6,349 6.7 2188 9.7 U.S. Educational .edu 2,400 4.3 4,630 4.9 2309 10.2 NonProfit Organizations .org 1,718 3.0 564 0.6 365 1.6 USA Military .mil 450 0.8 486 0.5 135 0.6 Germany .de 174 0.3         China .cn 96 0.2         United States .us 203 0.4         Italy .it     521 0.5     Canada .ca     141 0.1     Australia .au         31 0.1 Unknown       87 0.1 197 0.9 Other       1,506 1.6     a Statistics from January through August of 2006 b Statistics from September through December of 2004 SOURCE: http://aiwg.gsfc.nasa.gov/stats/. Usage data from NASA’s Earth Science Gateway shows a similar bias towards government-based domains of users during FY06 (Table 4.3): Table 4.3 Domains of Users of NASA’s Earth Science Gateway in Fiscal Year 2006 Domain Percentage .gov 70 .com 6 .edu 2 .net 8 .ca 3 .uk 1 Unknown 7 Other 3 SOURCE: http://aiwg.gsfc.nasa.gov/stats/.

OCR for page 77
Assessment of the NASA Applied Sciences Program The decision by NASA leadership in 2001 to focus on partnering with federal agencies to facilitate and enhance the development of DSS has had the predictable effect of limiting the involvement of other members of the broader community of users in developing models and applications. While it appears to have been NASA’s belief that it was the role of the partner federal agencies to advance the development of DSS among the broader community, the lack of a formal structure that delineates the roles of NASA and its partners leaves success largely to chance. Examples of successful development and employment of DSS by nonfederal entities certainly exist, and the projects competitively funded by the ASP include such grantees. However, the generally ad hoc nature of the relationship between the ASP and potential nonfederal users of DSS suggests that maximum benefit is not being realized. An example of NASA’s failure to engage the broader community is in the area of high-resolution satellite imaging systems, which is discussed more in Chapter 5. In particular, significant requirements exist for radiometrically calibrated, frequent, synoptic products at spatial resolutions that range from 0.25 to 100 meters, with a spectral resolution sufficient to accommodate between 6 and 20 visible spectral bands (color), and temporal resolution ranging from twice daily to two-week revisit cycles. Although spatial resolution and temporal frequency are generally tradeoffs, commercial providers do not meet the requirement for science-quality, high-spatial, spectral, and temporal resolution data. These requirements are shared by a large number of entities in the broader community, including those with national security, emergency management, resource management, enforcement, and research responsibilities, and include tribal, state, and local urban planners (Jensen and Cowen, 1999), terrestrial resource managers and researchers focused on land processes (NSGIC, 2006), coastal managers and scientists (Carder, et al., 1993), and a wide range of users of satellite imagery at the global, continental, or coastal zone scale (NRC, 2002b). This class of data has the largest group of users, and yet they have the greatest lack of response from NASA in addressing their need for high-resolution data. NASA’s addressing these requirements would satisfy a large cross-section of the broader community and would also address critical requirements of several federal operational (FEMA, DHS, NOAA, EPA, DOI, DOE) and research agencies (NSF, USGS, NASA) (Chapter 3), likely facilitating some additional interest from the community to engage NASA through ASP, and vice versa. One overarching issue in attempting to assess ASP’s engagement with the broader community (and for its engagement with federal agencies [Chapter 3]) is the lack of a prescribed feedback system from users. ASP’s Integrated System Solutions Architecture (Figure 2.2) refers to a model of moving data inputs to outputs to outcomes and finally to

OCR for page 77
Assessment of the NASA Applied Sciences Program impacts (NASA, 2004). This system envisions data accessed from satellite, airborne, and in situ platforms being incorporated into Earth science models that, in turn, produce predictions that become incorporated into decision-support tools. Ultimately, these decision-support tools should inform policy and management decisions. ASP sees this overall process as moving research to applications, with various partners playing a fundamental role in developing and applying DSS. With the partners having the primary burden of making the transitions from research missions and observations to applications, it appears that ASP has become almost disengaged from the broader community. In addition, the committee found a lack of supporting evidence that the federal partners are successfully engaging the end users. As a result the process is largely a unidirectional one as opposed to being applications driven. From the view of the end users, there are several limitations of this one-way flow model. The lack of feedback loops from partners (federal agencies, state agencies, nongovernment organizations) who develop and employ DSS to the developers and maintainers of sensory platforms (and, to some extent, model developers) leads to three missed connections: Users of NASA’s products are frustrated with a number of technical issues associated with aging NASA platforms and sensory arrays. The committee could not identify a mechanism for end users to communicate effectively with NASA through ASP about these shortcomings. While NASA may be prohibited from getting directly involved with DSS, this does not preclude an agency or ASP from building conduits for timely feedback from partners. The issue of missing operational support, and whose responsibility it is to provide such support, arose frequently in the briefings the committee received from users and potential users of NASA products. Frustrations also arise because NASA is a research and development organization and the continuity of data from a NASA sensor is not assured in the long term even if the data have useful applications. Interestingly, the success and longevity of NASA missions such as Landsat and NOAA’s Advanced Very High Resolution Radiometer (AVHRR) have created a user community that is dependent on older-generation sensors. And while many of these users have made a transition to MODIS products in place of AVHRR, it is unclear what will happen in three to five years when this platform retires and National Polar-orbiting Operational Environmental Satellite System does not possess the same capability.

OCR for page 77
Assessment of the NASA Applied Sciences Program A process for selection of sensors for operational transition is also lacking. Such a process seems to be nonexistent or infrequent at the design stage of new sensors and platforms (or at least when decisions are made as to which sensors will be placed on a new platform). Evidence of this lack of communication is the number of orphaned sensors that do not appear to be used by any partners. The lack of a feedback loop suggests that the role of the broader community has not been made sufficiently transparent in ASP to support a comprehensive assessment of benefits to society. Users of NASA products may not know that NASA (or ASP) was even involved in the production of the information they rely upon in their decision making, compounding the difficulty of engaging the broader user community. It is a problem of too many “unconnected dots” in the path from research to operations. Among the reasons given by ASP for not having more direct involvement with the broader community is the view shared by some government entities such as the Office of Management and Budget that NASA is a research agency and applications development is not a primary part of the NASA mission. These same entities point out that many federal agencies have science application responsibilities and that engaging NASA in the applications and implementation business would be duplicating such responsibilities. Partnering primarily with federal agencies to assess the potential for ASP products to improve the partner’s decision-support tools is apparently the adopted strategy. Meanwhile, the partner agency is tasked with engaging the broader community in the application of ASP products. This strategy constrains the transfer of information and technical knowledge between the applications and the user community, because a partner agency may not have sufficient knowledge of the NASA product or may not know how these products could be modified to meet user needs. Benchmark reports are supposed to help, but they often come late and sometimes not at all. In spite of ASP’s disproportionate engagement of federal agency partners and users, examples of research transfer to application products can be found in a range of ASP-funded projects encompassing a range of user groups (Box 4.1). Other NASA programs have successfully engaged the broader community in applications development without ASP involvement (Box 4.2); while this type of success is commendable and should be encouraged, it may also show the potential for enhanced communication and engagement within NASA, and specifically between ASP and other NASA units or programs where research with “applications potential” is being conducted. ASP’s role in bridging the transfer of such research and data to external partners and users could be more effectively incorporated through

OCR for page 77
Assessment of the NASA Applied Sciences Program solid internal communications at NASA, and could potentially increase the rate and number of successful transfers to the private sector. BOX 4.1 Examples of Transfer of Research to Applications in the Broader User Community The Institute for Technology Development—a nonprofit organization established to facilitate hyperspectral application development for industrial and public service usage in areas of earth resources, forensics, and biomedical applications of remote sensing—has developed partnerships with the National Corn Growers Association, the National Cotton Council, the United Soybean Board, and the National Association of Wheat Growers. These partnerships define the transfer of NASA remotely sensed information to the targeted commodity groups. The Yellowstone Ecological Research Center, another nonprofit organization, has been funded to develop remotely sensed data products to assist in regional conservation efforts, river management, and land use management in the Great Yellowstone region. A program that has received periodic attention from NASA Earth Science and Space Grant has been the establishment of the Geospatial Extension Specialist (GES) Program as part of Cooperative Extension in the Land Grant University system (http://www.geospatialextension.org/). It began in 2000 with 3 state pilot projects (Arizona, Mississippi, Utah) and has grown since to 14 states without continuing NASA support and only sporadic attention (some with Space Grant, some that have partnered with USDA). In the initial model, NASA offered 3 years of support with the understanding that, at the end of that period, a tenure-track extension position would be created to sustain the program (different models have been used in different states). Recently, some interest in the GES Program has resurfaced in ASP, and discussions between NASA, NOAA, USDA and USGS have occurred over the past year to look into the program and potentially move it forward again in a more substantive way (see also Box 5.3). Similar examples can be found in the air quality and weather forecasting applications. Sonoma Technology Inc. and Baron Advanced Meteorological Systems (BAMS) have been active in using NASA’s satellite data on optical density to provide air quality products to state and local governments and to industrial clients. BAMS has also developed weather forecasting products that are being used by local government for disaster planning associated with storms along the coastline of North Carolina. BOX 4.2 Technology Transfer to the Private Sector Without ASP Involvement NASA’s Goddard Space Flight Center and Jet Propulsion Laboratory (JPL) have developed prototypes of LIght Detection and Ranging (lidar) and InterFerometric Synthetic Aperture Radar (IFSAR) technology with the goal of technology transfer to the private sector. EarthData Inc. uses the IFSAR technology (renamed GEOSAR) developed in conjunction with JPL (see http://www.earthdata.com/.). NASA’s Solid Earth Program in the Earth Science Division was involved in the IFSAR and lidar work and has successfully engaged the private sector on technology development and transfer of differential GPS and the International GPS Service. SOURCE: John LaBrecque, personal communication (2006).

OCR for page 77
Assessment of the NASA Applied Sciences Program There are opportunities for ASP to engage the broader community. Many of the federal agencies have strong ties with academia, the private sector, and the tribal nations. Some academic institutions prefer joint NASA and other federal agency cooperative institute personnel to be located on campus for effective engagement. One clear example of a successful engagement activity is the Environmental Protection Agency (EPA) and its relationships with state and regional agencies on air quality, water management, and public health. Other agencies or entities in the agencies that have strong ties with members of the broader community include the National Weather Service (NWS) and the Departments of Agriculture, Transportation, and Energy. There are however, sometimes barriers to such agencies’ engaging the broader community. For example, EPA is a regulatory agency, which by law must keep an appropriate distance from those they regulate. The difficulty is that many regulated organizations are important members of the broader community. In addition, agencies that are not in the regulatory business have their own interests to protect and it is not always best from their perspective to reach out more than they are specifically required to do. They, too, are competing for funds, research, and recognition, and agencies tend to do what is specifically required. The result is that it is almost impossible to get a good indication of the benefits to society of NASA products, because those in the best position to measure the benefits are at the end of the applications pipeline and are not involved in a major way. Little direct evidence from the NASA partners documenting their direct engagement of the broader community suggests that what is left to assess is how well ASP packages NASA products for their partners to apply, implement, and distribute to society. Of course, this is more of an activity of the application teams (made up of NASA and partner personnel), which further complicates any assessment of ASP’s explicit performance. However, at a qualitative level the achievements of ASP’s program elements and focus areas cut across an impressive list of applications for the betterment of society. There is no doubt that considerable analysis and innovation have gone into the selection of projects. A review of NASA documents reveals many references to ASP projects that benefit society. For example, consider the DSS for the program element, Agricultural Efficiencies. NASA indicates that data emerging from this program element serve not only the federal government but also a large community of other users, such as commodity trading companies, farmers, relief agencies, and anyone with an interest in global crop production. However, searches for supporting evidence of the engagement process with such members of the broader community as farmers, trading companies, relief agencies, and the many private sector organizations providing products and services to the agricultural community do not

OCR for page 77
Assessment of the NASA Applied Sciences Program produce documentation that can be tied back to direct input and communication with NASA or the ASP (e.g., Russo, 2006). Nevertheless, evaluation of the many precursor activities, such as the nature of the products and the tools of their implementation, can yield some proxy assessment of the degree to which NASA’s and ASP’s engagement in the process influence the projects’ successes in the broader community. METRICS, DOCUMENTATION, REQUIREMENTS, AND ACCOUNTABILITY Applied science performance metrics associated with ASP do not seem to exist. The goals and objectives of the program are qualitative— with descriptors such as “improving our understanding” and “expand and accelerate societal benefits.” Metrics based on principles that provide both quantitative and qualitative measures can have many benefits, including improved program management, greater accountability to Congress and the public, and a roadmap for interacting with stakeholders. Guidance on a metrics model for ASP to consider is provided in NRC (2005b), a report that examined metrics for the Climate Change Science Program (Chapter 5). ASP performance evaluations also lack assessments by the beneficiaries of the applied science products, something that could be overcome with a well-designed metrics program that included user feedback and a formal peer-review process. As best can be determined by the committee, ASP managers do the actual performance evaluations. Little evidence was found of any significant engagement of the broader community in this process. On the contrary, the discussions of detailed performance data usually begin with reference to working with federal agency partners, with no mention of any member of the broader community. An example is the following quotation from the 2004 ASP performance and accountability report (http://aiwg.gsfc.nasa.gov/esappdocs/PAR_ESA_FY04.pdf; p. 1): By working with Federal agency partners, NASA improves essential public services like tracking hurricanes, assessing crop health and productivity, evaluating forest fire risks, ensuring aviation safety, improving energy forecasts, and determining the potential for the climate-driven spread of infectious disease. NASA’s Earth observing systems and Earth science models advance researchers’ ability to understand and protect Earth, its resources, and its diverse and precious life.

OCR for page 77
Assessment of the NASA Applied Sciences Program These are important activities with far-reaching potential; however, the performance and accountability reports provide little documentation on which to assess the extent of engagement of the broader community. On occasion there is reference in NASA’s reports to involving and benefiting the private sector, but little actual performance information is presented. NASA takes the position that their partners, the federal agencies and national organizations employ their directives and networks to engage such members of the broader community as local governments (state, county, city) and the private sector. As previously noted (Chapter 3), examples of partner agencies that have a legacy of strong ties with members of the broader community are EPA, USDA, and NOAA, the latter especially through the NWS. EPA has relationships with state and regional agencies on air quality, water management, and public health. NASA indicates that EPA works with such organizations on tools to support their activities. What is missing is direct evidence from organizations like the state and regional agencies, and the private organizations supporting them, about the benefits they derive from NASA products. There are no formal or quantitative procedures, standards, or methods for measuring federal agency use of NASA data, much less their use by members of the broader community. ASP has in the past taken the position that the partners and others must take the initiative to make outcomes to improve decision support. As ASP writes, “The outcomes and impacts largely lie with our partner organizations.” ASP indicates that they are constrained on such activities by “personnel commitments” and time management. Progress and activity reports are provided by funding recipients at weekly, monthly, annual intervals and constitute ASP’s only documentation of its products. Some broader community exposure is provided through published journal articles and special conferences and presentations where benchmarks, benchmark reports, and other ASP products are referenced and are part of the source material. The benchmark reports are considered the best source of information on the quality of the decision-support capabilities. A major activity of the benchmarking is that of the ASP applications teams’ development of proper documentation of procedures and guidelines to describe the steps to access and assimilate the Earth science observations and products. Assessment of socioeconomic benefits is generally left to the partners to conduct informally. In the absence of direct feedback from users and beneficiaries (society in general) of ASP products, the benchmark reports are probably the best metric for assessing programmatic benefits to society. This is because the benchmark reports are intended to measure the performance of

OCR for page 77
Assessment of the NASA Applied Sciences Program a product “according to specified standards and reference points.” Based on the 2005 ASP Annual Report, three benchmark reports were published among the 12 Applications of National Priority (NASA, 2005). Two of the reports were by the Air Quality application team and one by the Water Management application team. An additional nine reports were published in 2006 (in Agricultural Efficiency [1], Aviation [3], Disaster Management [4], and Invasive Species [1]). The committee did not see an integrated and comprehensive review of the disposition of the benchmark reports in terms of their effectiveness in engaging the broader community and obtaining feedback from the beneficiaries of NASA products (Chapter 2). A question as to the validity of the benchmarking process also arises, given that little to no input from the end users of the NASA products is incorporated in them. In many cases the end users will be those in the private sector who are often in the best position to apply the test of practicality to technological products. The private sector and local governments have extensive and direct involvement with the true beneficiaries of the NASA products, namely the public at large. It is clear that interactions with the broader community are left up to the partner agencies and national organizations. This arrangement often provides little impetus for the partners to engage members of the broader community since partners generally have their own research and operations priorities. Since government agencies, including NASA, respond best to specific requirements, it would seem that having the broader community receive benefit from partnering with NASA through ASP will require the establishment of new program requirements and perhaps legislative action. The issue of the benefits to society of government research activities extends beyond NASA, and legislation that addresses this issue for NASA should be directed at government research in general. The performance and accountability evaluation process of ASP appears to be more promotional than a clinical and critical process of quantifying performance on the basis of feedback from the user groups. The committee could find little to no evidence of input from the broader community on the evaluation of the societal benefits of ASP’s work, thereby limiting the credibility of any such assessment. The most meaningful measure of performance in the name of benefiting society comes from the beneficiaries of the services and products provided. INVOLVEMENT OF THE BROADER COMMUNITY IN APPLICATIONS OF NATIONAL PRIORITY The Applied Sciences Program defined 12 Applications of National Priority to focus the partnerships implemented primarily with federal

OCR for page 77
Assessment of the NASA Applied Sciences Program agencies (Table 2.1). The initial community workshops that NASA conducted in the late 1990s to discuss the priority areas (see also Chapter 2) were explicitly intended to gather general requirements from state, local, and tribal governments. The private sector and academia were involved as resources, to help explain what was available to these governments. Since then, the influence of these nonfederal groups has been largely absent and ASP’s strategic planning process does not match outcomes and impacts with specific community goals making efficient prioritization of its activities and resources difficult. The partnerships, have been focused, controlled, and largely kept within the federal government. ASP has identified several societal benefits across the 12 application areas: data and decision-support tools that supported the September 11 recovery operations, Hurricane Andrew, the Montana wildfires, location of lost aircraft, and flood recovery activities along the Mississippi River. They are proud that NASA data (from the Tropical Rainfall Measuring Mission) have shortened warning time for tropical depressions and hurricanes and thus put the process of emergency evacuations on a much more scientific basis. NASA indicates that ASP funding to enhance DSS has resulted in hundreds of millions of dollars saved as a result of much improved El Niño forecasts. An impact of great national significance is the improvement in weather forecasting, in general, as a result of implementing NASA products. The questions are whether the present paradigm of the ASP has an adequate cost-benefit ratio, and whether opportunities for improvement can be identified. Of the 14 federal agencies involved in ASP partnerships, USDA, USGS, EPA, and NOAA dominate. Drawing in part from the FY 2005 annual report (NASA, 2005) as well as from the NASA Applications Implementation Working Group website (http://aiwg.gsfc.nasa.gov/) and testimony presented to the committee, the most active programs in terms of publications and conferences appear to be Agricultural Efficiency, Air Quality, Ecological Forecasting, and Public Health. In terms of available benchmark reports, Agricultural Efficiency, Air Quality, and Water Management have the lead. Lagging in benchmarking activity are the application areas of Disaster Management, Public Health, and Water Management. The Agricultural Efficiency Program element is one of the strongest application programs, with Air Quality, Disaster Management, Public Health, and Water Management having considerable potential. These elements are also active at the international level. The trade-offs ASP might have to consider as a result of an international versus domestic emphasis are difficult to assess. The program elements Energy Management, Carbon Management, and Homeland Security do not appear as strong in their activities as the other programs at this time.

OCR for page 77
Assessment of the NASA Applied Sciences Program Disaster Management is an ideal program element for involving many sectors of the broader community, such as local government organizations (the first responders), the private sector (the owner of most physical assets), the service industry (operators of our infrastructure), and the engineering and construction industries (the designers, constructors, and recovery organizations). The committee could not find any direct involvement of first responders (local organizations) and infrastructure providers. Incorporating some of these broader community user groups might allow ASP to realize the potential of this application area more quickly and directly. Ecological Forecasting is another program element where the committee found little evidence for an integrated approach to account for community and public interests. Realistic ecological forecasting is best done by spanning political jurisdictions and defining geographical boundaries based on scientific knowledge. Policy decisions must balance community development programs with impact of activities from industries that may include transportation, resource extraction and use, and manufacturing on land, in the ocean, or the atmosphere. In addition to industry and community input, scientific data like that provided by NASA plays a balancing role to weigh various policy options offers another opportunity for ASP to establish direct communication between NASA and these broader community sectors. Establishing the Ecological Forecasting Program element in place of the previous Community Growth Program element is an example of an action that could distance NASA from the broader community. Ecological forecasting without the engagement of local organizations further increases the danger of setting restrictive boundaries on the activity. Another example of strong interest by the broader community is in Water Management. This is one of the most important issues facing local governments in the western states and in Florida, among others, but the committee again found little evidence of ASP seeking involvement in regional water management or hydrological planning programs. Most of the ASP program elements involve activities associated with climate and weather. Considering that the NWS has a long tradition of interacting with the private sector, the opportunity exists to establish a framework for partner agencies to engage the broader community, especially the private sector. USDA has a similar long-term relationship with nonfederal agencies and could contribute to such a framework. EPA and DOE also have a legacy of extensive interaction with nonfederal organizations. The opportunity exists to develop a model for the ASP to extend NASA products to nonfederal entities in the private sector. Overall, the committee did not identify much documentation of direct interaction between NASA and the broader community among the

OCR for page 77
Assessment of the NASA Applied Sciences Program approximately 147 projects that ASP currently has underway. NASA is relying on its federal partners to engage these groups and be the conduit for societal benefits. However, the lack of information about relationships between such federal agency partners and the broader community makes it difficult to carry out a complete benchmarking process to assess societal benefits; this is a missed opportunity to address ASP’s mandate to enhance the benefits to society of NASA’s science and technology investments. BENEFITS OF ENGAGING THE BROADER COMMUNITY Many benefits could be generated for NASA and society through more direct engagement of the broader community with the ASP. These benefits are noted without full knowledge of the current relationships between the NASA partners and members of the broader community; such relationships are not part of the ASP documentation in any definitive form provided to the committee. The benefits to NASA include attempting to increase the applications of their data and research to the benefit of a much larger segment of society (as exemplified by NASA’s Solid Earth Program involvement in the development of IFSAR [Box 4.2]). Such an approach could lead to a greatly expanded resource base to facilitate improvements in sensor technologies and innovative applications. Establishing feedback mechanisms from the broader community to NASA and ASP is critically important to assure higher payoffs for ASP activities. Feedback of this nature would yield broad-based information for guiding engineering and technology development efforts, among others. A DSS is an information system that accumulates input from a variety of sources. The ASP focus on the federal government severely limits the diffusion of NASA products and technologies. Nonfederal entities also need assistance to develop fully benchmarked and developed DSS, while others, especially the competitive private sector, may only want the raw data for applying their own algorithms and models to develop innovative new product lines, a possible major contributor to the process by which NASA products can benefit greater segments of society through applications. Involving a broader community and deployment of NASA products among nonfederal users could also contribute to better integration of the nation’s capabilities, capacity, and infrastructure. Feedback information on data needs and data sharing could contribute extensively not only to guiding technology development but to the integration process as well.

OCR for page 77
Assessment of the NASA Applied Sciences Program ENGAGING THE BROADER COMMUNITY An important question is “what is the best path forward for engaging the broader community?” Two factors compromised the committee’s assessment of ASP’s engagement of the broader community: (1) NASA does not have the lead in the process; (NASA asks partner federal agencies to assume that role), and (2) the partner agency process for engaging members of the broader community is not clearly defined in the ASP documentation. Much more engagement may exist than is apparent in the documentation provided to the committee. Nevertheless, several factors can greatly improve the involvement of non-federal organizations in ASP. One such factor would be to improve the reporting requirements of the partners on user interfaces and the disposition of information. Another factor would be to make greater use of external review groups that can better identify the full range of users and their present and emerging needs. Every effort will need to be made to choose partners and establish requirements of the partners that clearly integrate the resources of the nation. The integration process needs to be guided by the scope of ASP to avoid engaging community members not in a position to make a significant contribution. Absent from the ASP engagement process (though not absent from applications development in NASA in general) is the private sector which provides leadership in the development of innovative products that benefit society, and therefore needs to be a major force in testing the practicality of NASA's products. The private sector, for example, is extremely interested in disaster management and application of remotely sensed data to these problems, as they own most of the assets under threat. This area could be fruitful ground for ASP to establish new partnerships between NASA and the private sector. Besides the greater direct involvement of the private sector, all ASP partners will need to document and highlight plans of engaging nonfederal organizations in the implementation of NASA products to the benefit of all. SUMMARY ASP relies primarily on federal agency partners for implementing practical applications of its products and expects these partners to engage the broader community. Socioeconomic benefits of ASP products are primarily assessed by the federal partner agencies. The process by which partner agencies engage the broader community lacks transparency and documentation.

OCR for page 77
Assessment of the NASA Applied Sciences Program Quantitative metrics that connect the research to applications transition of ASP products are lacking. Comprehensive assessments of benefits to society of ASP products that include feedback from beneficiaries are not performed and documentation of implementation processes and practices of applying ASP products compromise third-party evaluations of benefits. Key documentation of DSS in the form of benchmark reports, while effective in providing guidance on the application of DSS, lacks critical input from the end users and especially local governments and the private sector. The benchmark reports do provide a critically important database across many application areas for guiding future applications and the direction of the ASP. Inefficient transition from research to operations of NASA products impairs the consideration of the applications community to commit to use of science products because of a lack of assurance of product continuity. The performance and accountability evaluation process of ASP does not appear to be a clinical and critical process of quantifying performance on the basis of feedback from the user groups. The committee could find little to no evidence of input from the broader community on the evaluation of the societal benefits of ASP's work, thereby limiting the credibility of any such assessment.