5


Lessons Learned

As a whole, the MAR led to greater integration of science into weather service activities and improved outreach and coordination with state and local government, emergency management, and local communities. Technological improvements provided forecasters with a wealth of new data and observations, allowing them to provide more accurate and timely forecast and warning services to the nation. This chapter examines whether the execution objectives of the MAR were met, and whether the promised benefits were achieved. It presents the committee’s key findings about the MAR as a whole and an assessment of the lessons learned from the committee’s analysis of the execution and impact of the MAR. The committee recognizes that many of the lessons presented in this report would apply to any large, complex project. However, this does not make the lessons any less useful. The fact that they are common makes it even more important that they be considered in future planning.

The stated objective of the MAR in the Strategic Plan (NWS, 1989) was

to modernize the NWS through the deployment of proven observational, information processing and communications technologies, and to establish an associated cost effective operational structure. The modernization and associated restructuring of NWS shall assure that the major advances which have been made in our ability to observe and understand the atmosphere are applied to the practical problems of providing weather and hydrologic services to the Nation.

It is clear that the NWS succeeded in the deployment of observational, information processing, and communications technologies that have improved weather and hydrologic services. The MAR significantly increased the amount of data and information available to field forecasters. The forecast and warning products produced by the post-MAR NWS are greater in both quantity and quality. The cost-effectiveness of the operational structure is more difficult to assess, because of the challenges involved in assessing the value of decreased loss of life and property as a result of improved forecasts and warnings. Understanding of the economic impacts of weather events still needs improvement and the benefit of weather forecasts and warnings cannot be measured only in economic terms. However, recent work has estimated the annualized value of public weather forecasts and warnings to be about $31.5 billion, compared to an annual cost of $5.1 billion to produce the information (Lazo et al., 2009). Variations in weather have been shown to cause variations of $485 billion in U.S. annual economic output (Lazo et al., 2011). Because weather services clearly have great value, it is hard to argue that an increase in both the quantity and quality of forecasts, and a decrease in the total number of staff, has not yielded a more cost effective operational structure.

The Strategic Plan (NWS, 1989) also set forth the specific benefits the NWS hoped to achieve with the MAR

More uniform weather services across the Nation. This was achieved, as summarized in Findings 3-3a, 4-2a, and 4-3a.

Improved forecasts. This was achieved for local area forecasts issued by Weather Forecast Offices and regional forecasts issued by centers such as the Storm



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 73
5 Lessons Learned A s a whole, the MAR led to greater integra- weather and hydrologic services. The MAR signifi- tion of science into weather service activities cantly increased the amount of data and information and improved outreach and coordination with available to field forecasters. The forecast and warning state and local government, emergency management, products produced by the post-MAR NWS are greater and local communities. Technological improvements in both quantity and quality. The cost-effectiveness provided forecasters with a wealth of new data and of the operational structure is more difficult to assess, observations, allowing them to provide more accurate because of the challenges involved in assessing the and timely forecast and warning services to the nation. value of decreased loss of life and property as a result This chapter examines whether the execution objec- of improved forecasts and warnings. Understanding tives of the MAR were met, and whether the promised of the economic impacts of weather events still needs benefits were achieved. It presents the committee’s key improvement and the benefit of weather forecasts and findings about the MAR as a whole and an assessment warnings cannot be measured only in economic terms. of the lessons learned from the committee’s analysis of However, recent work has estimated the annualized the execution and impact of the MAR. The commit- value of public weather forecasts and warnings to be tee recognizes that many of the lessons presented in about $31.5 billion, compared to an annual cost of $5.1 this report would apply to any large, complex project. billion to produce the information (Lazo et al., 2009). However, this does not make the lessons any less use- Variations in weather have been shown to cause varia- ful. The fact that they are common makes it even more tions of $485 billion in U.S. annual economic output important that they be considered in future planning. (Lazo et al., 2011). Because weather services clearly The stated objective of the MAR in the Strategic have great value, it is hard to argue that an increase Plan (NWS, 1989) was in both the quantity and quality of forecasts, and a decrease in the total number of staff, has not yielded a to modernize the NWS through the deployment of proven more cost effective operational structure. observational, information processing and communications The Strategic Plan (NWS, 1989) also set forth the technologies, and to establish an associated cost effective specific benefits the NWS hoped to achieve with the operational structure. The modernization and associated restructuring of NWS shall assure that the major advances MAR which have been made in our ability to observe and under- stand the atmosphere are applied to the practical problems • More uniform weather services across the Nation. of providing weather and hydrologic services to the Nation. This was achieved, as summarized in Findings 3-3a, 4-2a, and 4-3a. It is clear that the NWS succeeded in the deploy- • Improved forecasts. This was achieved for local ment of observational, information processing, and area forecasts issued by Weather Forecast Offices and communications technologies that have improved regional forecasts issued by centers such as the Storm 73

OCR for page 73
74 THE NWS MODERNIZATION AND RESTRUCTURING: A RETROSPECTIVE ASSESSMENT Prediction Center, as well as the guidance products 4-3a and 4-3c), but whether maximum benefit was produced by the National Centers (Findings 3-3a, achieved cannot be determined. 4-2b, and 4-3a). Global model forecasts (e.g., the • User acceptance and support of NWS modernization Global Forecast System) improved, but their skill still and associated restructuring service improvement objec- lags behind some of the other leading global models tives. There was some initial resistance from employ- (Finding 4-4). ees (Finding 3-3b), as well as the general public and • More reliable detection and prediction of severe Congress in some regions, but this goal was eventually weather and flooding. This was achieved, as summarized achieved. in Findings 3-3a and 4-2b. • Strengthening cooperation with the mass media, • More cost effective NWS. As noted above, the universities, the research community and the private challenges involved in assessing the value of decreased hydrometeorological sector to collectively fulfill the Nation’s loss of life and property as a result of improved forecasts w eather information needs from provision of severe and warnings make it difficult to quantitatively assess weather warnings and general forecasts for the public as whether a more cost effective NWS was achieved. a whole, which is a Government responsibility; to provi- However, estimates of the value of weather informa- sion of detailed and customer specific weather information, tion seem to support the notion that the post-MAR which is a private sector responsibility. This was achieved, NWS is indeed cost effective. The MAR significantly although improvement in the relationship between the increased the quantity and quality of NWS products NWS and the private sector took longer. Collabora- while decreasing the total number of staff. tions with academia and government laboratories are • Higher productivity for NWS employees. This beneficial, with some exceptions where the colocation promised benefit is also difficult to assess quanti - is not optimal (Findings 3-5, 4-3c, and 4-5). tatively. With a greater number of higher quality • Achievement of productivity gains through auto- products produced by a smaller workforce with more mation and replacement of obsolete technological systems. t echnical capabilities, and with a greater amount O bservations were automated and obsolete techno- of higher quality data and information available to logical systems were replaced (Findings 3-2 and 4-2a), them, productivity of NWS employees has certainly leading to more products and new capabilities. increased (Finding 4-3a). • Operation of the optimum NWS warning and forecast system consistent with service requirements, user The initial National Implementation Plan (NWS, acceptability, and affordability. It is not possible to assess 1990) expanded and clarified the list of promised whether the post-MAR NWS operates optimally. benefits: Operations certainly improved dramatically, and this goal, with some exceptions (e.g., the tornado and flash • Operational realization of a predictive warning flood warning False Alarm Ratios remains high), was program focusing on mesoscale meteorology and hydrology. largely met. This was achieved, as summarized in Findings 3-3a and Key Finding 1 4-2. The National Weather Service (NWS) had been • Advancement of the science of meteorology and unable to keep up with the pace of technological hydrology. This was achieved although there were some advances and had nearly become obsolete by the issues with the application of science and technology 1980s. Therefore the NWS was not utilizing the to operational hydrology (Findings 3-4 and 4-7a). full potential available to provide the best possible W hile numerical weather prediction improved steadily, meteorological services to the nation. The $4.5 bil- there are still some cases where capabilities could be lion national investment in the Modernization and improved (Finding 4-4). Associated Restructuring (MAR) was both needed • Development of NWS human resources to achieve and generally well spent. Overall, the MAR was maximum benefit from recent scientific and technical successful in achieving major improvements for the advances. The scientific and technical capabilities of the weather enterprise. workforce increased as a result of the MAR (Findings

OCR for page 73
75 LESSONS LEARNED The MAR was large and complex in terms of both evant information; and interactions with staff at sev- breadth and magnitude. The NWS was reengineered in eral WFOs, the committee identified six lessons that a revolutionary manner. Many critical technologies were resulted from the MAR, that will be helpful to the replaced, the field office structure was changed, and the NWS as it plans future improvements. workforce was retrained. This was necessary at the time Lesson 1 because the NWS had fallen behind the pace of tech- If a science-based agency like the National Weather nological growth. Given that the pace is increasing, it is Service, which provides critical services to the nation, critical for the NWS to develop the capability to keep waits until it is close to becoming obsolete, it will up with technological change in a more evolutionary require a complex and very expensive program to manner, so that a transition the size and complexity of modernize. the MAR will not be necessary in the future. Risk reduction activities were an important part of the MAR, and one of the key facets of the MAR Implementation of the MAR occurred during a was the prototyping of new operational concepts. This period of rapid technological change, and involved a pre-operational prototype paradigm has been advanced number of major systems deployed across a geographi- following the MAR and embraced by the NWS, which cally diverse nation, as well as involving several federal now has a number of testbeds that support risk reduction agencies and the direct participation of three NOAA and the transition of research-to-operations. The MAR line offices (NWS, the National Environmental Satel- created a framework that allows the NWS to be more lite, Data, and Information Service [NESDIS], and the capable of evolution, and decreases the need for revolu- Office of Oceanic and Atmospheric Research [OAR]). tion. One example of this evolutionary framework is the Any such undertaking requires rigorous management NEXRAD Product Improvement Program and the fur- to be successfully executed. In addition to the planned ther development of AWIPS. The further development system improvements that were the objective of the and improvement of guidance products and their appli- MAR, execution of the project itself left a legacy of cations, as well as the increasing use of research results institutional and cultural changes at NWS, largely for from the social sciences, are also examples of continued the betterment of the organization. improvement facilitated by the framework put in place by Lesson 2 – Management and Planning the MAR. However, issues remain. Some of the current- The budget, schedule, and technological issues generation testbeds are too narrow in scope to adequately encountered during execution of the Moderniza- prototype new operational concepts, while others fail to tion and Associated Restructuring of the National engage key stakeholder groups such as emergency man- Weather Service (NWS) reflected traditional chal- agement. Finally, apparent issues with the deployment lenges of large projects: inexperience of the gov - of AWIPS-II and upgrading the NEXRAD system to ernment project-level leadership, shifting budget dual-polarization radars indicate that lingering process constraints, ambitious technology leaps, multiparty issues, particularly with large procurements, may hinder stakeholder pressures, cultural inertia, contractor the evolution of the NWS. shortcomings, and oversight burdens. Each repre- Key Finding 2 sents important lessons for the NWS with regard to A framework was created and left in place following future projects of a similar nature: the Modernization and Associated Restructuring that allows and encourages the technology and to some - Expertise in system design, procurement, and extent the workforce composition and culture of the deployment is essential to successful implementation National Weather Service to continue to evolve. of any complex technical upgrade. - Dedicated leaders are crucial for resolving road- blocks and ensuring ultimate project success. Based on input from a range of stakeholders and - Clearly defined system-level requirements, and participants in the MAR; a review of the literature, competent management of those requirements, are oversight reports, NWS documents, and other rel-

OCR for page 73
76 THE NWS MODERNIZATION AND RESTRUCTURING: A RETROSPECTIVE ASSESSMENT essential to any contractual acquisition of a major - establish the capacity for continual upgrades of system. complex systems, particularly those involving digi- - Statistical performance indicators are a major ele- tal technology (e.g., high performance computing, ment for gaining and maintaining support for imple- communications); menting changes. - continually assess and apply the lessons of past sys- - It is necessary to establish comprehensive perfor- tems, whether successful or unsuccessful. mance metrics at the beginning of a process, evaluate them throughout the process, and reevaluate them The MAR brought significant changes to the after the process is complete. NWS workforce. It closed offices and moved others. Great amounts of training were necessary to famil- The MAR included the development, procure- iarize staff with the new technologies. Professional ment, and deployment of technologies in five major meteorology training was provided for technicians who areas: surface observations, the radar network, satellites, wanted to qualify for a position in the new workforce computing upgrades, and a forecaster interface to inte- structure. While many of these changes were viewed grate the data and information available by the other negatively by some NWS employees during the MAR elements of the modernization. It was among the larg- period (NRC, 1994a), hindsight has shown that they est and most complex procurements ever undertaken in have greatly improved the capability of the NWS to the Department of Commerce. While the technologies provide weather services to the nation, and are now involved in the MAR all had scheduling or budget viewed favorably by the staff. issues, they contributed to the capability of the NWS to Lesson 4 – Restructuring of Forecast Offices and provide improved weather services to the nation. This Staff is particularly true for the forecasting and detection of severe weather such as tornadoes and flash floods. The Modernization and Associated Restructuring Lesson 3 – Modernization of Technology (MAR) of the National Weather Service (NWS) The time scale for implementing major change in faced initial resistance from NWS employees and, government systems is very long compared to the to some extent, the general public. This resistance time scale for major technological change. The pace could have been lessened by, very early in the plan- of technological progress complicates the planning, ning stages: procurement, and deployment of large, complex sys- tems. While technology is changing so rapidly, in every - Engaging those whose career and livelihood were to aspect of the project where it is feasible, it is crucial to be affected in planning the changes - Better engaging a diffuse public, and to some extent - establish clear metrics for evaluating improvement Congress, regarding the benefits of improved weather in forecasts and warnings at the beginning of a major forecasts and warnings as opposed to the perceived technological upgrade; cost of losing a forecast office in their community - use rapid prototyping and system demonstrations. A n example includes the Program for Regional The restructuring dictated a degree of standard- O bserving and Forecasting Service (PROFS) and ization between forecast offices, however it has their Denver AWIPS Risk Reduction and Require- become apparent that this needs to be effectively ments Evaluation (DAR3E) effort, which proved balanced with the flexibility needed to allow for cus- critical to the success of the Modernization and tomization at individual offices to respond to local Associated Restructuring; requirements. - evaluate such prototype systems under a variety of actual operational situations with multiple classes of The MAR increased the overall education level of users and stakeholders in order to refine the system the workforce and set in place the need for routine design; training to keep the staff on pace with technological

OCR for page 73
77 LESSONS LEARNED and meteorological advancements in the community. NWS better understand user needs and secure ‘buy- S taff development through in-person, hands-on in’ to new initiatives. training in a centralized classroom or laboratory of the type that occurred during the MAR has great Throughout the execution of the MAR, the NWS value. Where relevant, online courses or self-directed received a large amount of oversight and technical study can be a useful supplement, but can sacrifice advice both from within and outside the government. quality of learning and the connections made with In many cases, the reviews drew attention to important colleagues that are essential to the overall operations issues, issues whose resolution was important to the of the NWS. success of the MAR. Successful reviews not only help management understand and react to technical, sched- While the MAR was a reengineering of the NWS, ule, and budget issues, but help build communities of its execution depended on the involvement of many knowledgeable support. partners. Development and deployment of all the obser- Lesson 6 – Oversight and Advice vational systems of the MAR involved other NOAA line The Modernization and Associated Restructuring offices (e.g., NESDIS, OAR) as well as other federal of the National Weather Service (NWS) showed agencies. The NWS worked with the private sector that candid yet non-adversarial advice from out- through contracted work, and the research community side experts and other interested parties was useful played a large role in the development and demon- in the design and deployment of a large complex stration of MAR technologies. In general, the MAR system. Because NWS management was receptive strengthened the relationships between the NWS and to such oversight and advice, the outside input was other members of the weather enterprise, although in effective. the case of the private sector, it took some time after the MAR to develop these strengthened relationships. The MAR was a large, complex process that lasted Lesson 5 – Partnerships a decade, and cost an estimated $4.5 billion. Despite The execution of the Modernization and Associated issues, some more significant than others, in the end Restructuring required working with many partners, the MAR was a success. New technologies deployed which provided cost-sharing and improved under- during the MAR now provide forecasters with more standing of user needs. However, the relationships observations of higher quality. NWS forecast and warn- with the partners were not always as well conceived ing products were dramatically improved, in both qual- or managed as would have been desirable. This could ity and quantity. NWS now has stronger relationships have been avoided by involving all known stakehold- with many of its partners in the weather enterprise. ers (e.g., other agencies, academia and the research Changes in the distribution of field offices have allowed community, the private sector, media, and emergency stronger connections with local communities. Weather management) from the outset. The National Weather services have great value to the nation, and the MAR Service (NWS) operational staff is also a stakeholder, was clearly well worth the investment. In Phase II of its and need to be involved early in the design and pro- study, the committee will build on the lessons presented curement process to ensure system functionality and in this report to develop actionable recommendations practicality. Engagement with stakeholders from for the NWS to best plan, deploy, and oversee future both inside and outside the NWS would help the improvements.

OCR for page 73