As a whole, the MAR led to greater integration of science into weather service activities and improved outreach and coordination with state and local government, emergency management, and local communities. Technological improvements provided forecasters with a wealth of new data and observations, allowing them to provide more accurate and timely forecast and warning services to the nation. This chapter examines whether the execution objectives of the MAR were met, and whether the promised benefits were achieved. It presents the committee’s key findings about the MAR as a whole and an assessment of the lessons learned from the committee’s analysis of the execution and impact of the MAR. The committee recognizes that many of the lessons presented in this report would apply to any large, complex project. However, this does not make the lessons any less useful. The fact that they are common makes it even more important that they be considered in future planning.
The stated objective of the MAR in the Strategic Plan (NWS, 1989) was
to modernize the NWS through the deployment of proven observational, information processing and communications technologies, and to establish an associated cost effective operational structure. The modernization and associated restructuring of NWS shall assure that the major advances which have been made in our ability to observe and understand the atmosphere are applied to the practical problems of providing weather and hydrologic services to the Nation.
It is clear that the NWS succeeded in the deployment of observational, information processing, and communications technologies that have improved weather and hydrologic services. The MAR significantly increased the amount of data and information available to field forecasters. The forecast and warning products produced by the post-MAR NWS are greater in both quantity and quality. The cost-effectiveness of the operational structure is more difficult to assess, because of the challenges involved in assessing the value of decreased loss of life and property as a result of improved forecasts and warnings. Understanding of the economic impacts of weather events still needs improvement and the benefit of weather forecasts and warnings cannot be measured only in economic terms. However, recent work has estimated the annualized value of public weather forecasts and warnings to be about $31.5 billion, compared to an annual cost of $5.1 billion to produce the information (Lazo et al., 2009). Variations in weather have been shown to cause variations of $485 billion in U.S. annual economic output (Lazo et al., 2011). Because weather services clearly have great value, it is hard to argue that an increase in both the quantity and quality of forecasts, and a decrease in the total number of staff, has not yielded a more cost effective operational structure.
The Strategic Plan (NWS, 1989) also set forth the specific benefits the NWS hoped to achieve with the MAR
• More uniform weather services across the Nation. This was achieved, as summarized in Findings 3-3a, 4-2a, and 4-3a.
• Improved forecasts. This was achieved for local area forecasts issued by Weather Forecast Offices and regional forecasts issued by centers such as the Storm
Prediction Center, as well as the guidance products produced by the National Centers (Findings 3-3a, 4-2b, and 4-3a). Global model forecasts (e.g., the Global Forecast System) improved, but their skill still lags behind some of the other leading global models (Finding 4-4).
• More reliable detection and prediction of severe weather and flooding. This was achieved, as summarized in Findings 3-3a and 4-2b.
• More cost effective NWS. As noted above, the challenges involved in assessing the value of decreased loss of life and property as a result of improved forecasts and warnings make it difficult to quantitatively assess whether a more cost effective NWS was achieved. However, estimates of the value of weather information seem to support the notion that the post-MAR NWS is indeed cost effective. The MAR significantly increased the quantity and quality of NWS products while decreasing the total number of staff.
• Higher productivity for NWS employees. This promised benefit is also difficult to assess quantitatively. With a greater number of higher quality products produced by a smaller workforce with more technical capabilities, and with a greater amount of higher quality data and information available to them, productivity of NWS employees has certainly increased (Finding 4-3a).
The initial National Implementation Plan (NWS, 1990) expanded and clarified the list of promised benefits:
• Operational realization of a predictive warning program focusing on mesoscale meteorology and hydrology. This was achieved, as summarized in Findings 3-3a and 4-2.
• Advancement of the science of meteorology and hydrology. This was achieved although there were some issues with the application of science and technology to operational hydrology (Findings 3-4 and 4-7a). While numerical weather prediction improved steadily, there are still some cases where capabilities could be improved (Finding 4-4).
• Development of NWS human resources to achieve maximum benefit from recent scientific and technical advances. The scientific and technical capabilities of the workforce increased as a result of the MAR (Findings 4-3a and 4-3c), but whether maximum benefit was achieved cannot be determined.
• User acceptance and support of NWS modernization and associated restructuring service improvement objectives. There was some initial resistance from employees (Finding 3-3b), as well as the general public and Congress in some regions, but this goal was eventually achieved.
• Strengthening cooperation with the mass media, universities, the research community and the private hydrometeorological sector to collectively fulfill the Nation’s weather information needs from provision of severe weather warnings and general forecasts for the public as a whole, which is a Government responsibility; to provision of detailed and customer specific weather information, which is a private sector responsibility. This was achieved, although improvement in the relationship between the NWS and the private sector took longer. Collaborations with academia and government laboratories are beneficial, with some exceptions where the colocation is not optimal (Findings 3-5, 4-3c, and 4-5).
• Achievement of productivity gains through automation and replacement of obsolete technological systems. Observations were automated and obsolete technological systems were replaced (Findings 3-2 and 4-2a), leading to more products and new capabilities.
• Operation of the optimum NWS warning and forecast system consistent with service requirements, user acceptability, and affordability. It is not possible to assess whether the post-MAR NWS operates optimally. Operations certainly improved dramatically, and this goal, with some exceptions (e.g., the tornado and flash flood warning False Alarm Ratios remains high), was largely met.
Key Finding 1
The National Weather Service (NWS) had been unable to keep up with the pace of technological advances and had nearly become obsolete by the 1980s. Therefore the NWS was not utilizing the full potential available to provide the best possible meteorological services to the nation. The $4.5 billion national investment in the Modernization and Associated Restructuring (MAR) was both needed and generally well spent. Overall, the MAR was successful in achieving major improvements for the weather enterprise.
The MAR was large and complex in terms of both breadth and magnitude. The NWS was reengineered in a revolutionary manner. Many critical technologies were replaced, the field office structure was changed, and the workforce was retrained. This was necessary at the time because the NWS had fallen behind the pace of technological growth. Given that the pace is increasing, it is critical for the NWS to develop the capability to keep up with technological change in a more evolutionary manner, so that a transition the size and complexity of the MAR will not be necessary in the future.
Risk reduction activities were an important part of the MAR, and one of the key facets of the MAR was the prototyping of new operational concepts. This pre-operational prototype paradigm has been advanced following the MAR and embraced by the NWS, which now has a number of testbeds that support risk reduction and the transition of research-to-operations. The MAR created a framework that allows the NWS to be more capable of evolution, and decreases the need for revolution. One example of this evolutionary framework is the NEXRAD Product Improvement Program and the further development of AWIPS. The further development and improvement of guidance products and their applications, as well as the increasing use of research results from the social sciences, are also examples of continued improvement facilitated by the framework put in place by the MAR. However, issues remain. Some of the current-generation testbeds are too narrow in scope to adequately prototype new operational concepts, while others fail to engage key stakeholder groups such as emergency management. Finally, apparent issues with the deployment of AWIPS-II and upgrading the NEXRAD system to dual-polarization radars indicate that lingering process issues, particularly with large procurements, may hinder the evolution of the NWS.
Key Finding 2
A framework was created and left in place following the Modernization and Associated Restructuring that allows and encourages the technology and to some extent the workforce composition and culture of the National Weather Service to continue to evolve.
Based on input from a range of stakeholders and participants in the MAR; a review of the literature, oversight reports, NWS documents, and other relevant information; and interactions with staff at several WFOs, the committee identified six lessons that resulted from the MAR, that will be helpful to the NWS as it plans future improvements.
Lesson 1
If a science-based agency like the National Weather Service, which provides critical services to the nation, waits until it is close to becoming obsolete, it will require a complex and very expensive program to modernize.
Implementation of the MAR occurred during a period of rapid technological change, and involved a number of major systems deployed across a geographically diverse nation, as well as involving several federal agencies and the direct participation of three NOAA line offices (NWS, the National Environmental Satellite, Data, and Information Service [NESDIS], and the Office of Oceanic and Atmospheric Research [OAR]). Any such undertaking requires rigorous management to be successfully executed. In addition to the planned system improvements that were the objective of the MAR, execution of the project itself left a legacy of institutional and cultural changes at NWS, largely for the betterment of the organization.
Lesson 2 – Management and Planning
The budget, schedule, and technological issues encountered during execution of the Modernization and Associated Restructuring of the National Weather Service (NWS) reflected traditional challenges of large projects: inexperience of the government project-level leadership, shifting budget constraints, ambitious technology leaps, multiparty stakeholder pressures, cultural inertia, contractor shortcomings, and oversight burdens. Each represents important lessons for the NWS with regard to future projects of a similar nature:
- Expertise in system design, procurement, and deployment is essential to successful implementation of any complex technical upgrade.
- Dedicated leaders are crucial for resolving road-blocks and ensuring ultimate project success.
- Clearly defined system-level requirements, and competent management of those requirements, are
essential to any contractual acquisition of a major system.
- Statistical performance indicators are a major element for gaining and maintaining support for implementing changes.
- It is necessary to establish comprehensive performance metrics at the beginning of a process, evaluate them throughout the process, and reevaluate them after the process is complete.
The MAR included the development, procurement, and deployment of technologies in five major areas: surface observations, the radar network, satellites, computing upgrades, and a forecaster interface to integrate the data and information available by the other elements of the modernization. It was among the largest and most complex procurements ever undertaken in the Department of Commerce. While the technologies involved in the MAR all had scheduling or budget issues, they contributed to the capability of the NWS to provide improved weather services to the nation. This is particularly true for the forecasting and detection of severe weather such as tornadoes and flash floods.
Lesson 3 – Modernization of Technology
The time scale for implementing major change in government systems is very long compared to the time scale for major technological change. The pace of technological progress complicates the planning, procurement, and deployment of large, complex systems. While technology is changing so rapidly, in every aspect of the project where it is feasible, it is crucial to
- establish clear metrics for evaluating improvement in forecasts and warnings at the beginning of a major technological upgrade;
- use rapid prototyping and system demonstrations. An example includes the Program for Regional Observing and Forecasting Service (PROFS) and their Denver AWIPS Risk Reduction and Requirements Evaluation (DAR3E) effort, which proved critical to the success of the Modernization and Associated Restructuring;
- evaluate such prototype systems under a variety of actual operational situations with multiple classes of users and stakeholders in order to refine the system design;
- establish the capacity for continual upgrades of complex systems, particularly those involving digital technology (e.g., high performance computing, communications);
- continually assess and apply the lessons of past systems, whether successful or unsuccessful.
The MAR brought significant changes to the NWS workforce. It closed offices and moved others. Great amounts of training were necessary to familiarize staff with the new technologies. Professional meteorology training was provided for technicians who wanted to qualify for a position in the new workforce structure. While many of these changes were viewed negatively by some NWS employees during the MAR period (NRC, 1994a), hindsight has shown that they have greatly improved the capability of the NWS to provide weather services to the nation, and are now viewed favorably by the staff.
Lesson 4 – Restructuring of Forecast Offices and Staff
The Modernization and Associated Restructuring (MAR) of the National Weather Service (NWS) faced initial resistance from NWS employees and, to some extent, the general public. This resistance could have been lessened by, very early in the planning stages:
- Engaging those whose career and livelihood were to be affected in planning the changes
- Better engaging a diffuse public, and to some extent Congress, regarding the benefits of improved weather forecasts and warnings as opposed to the perceived cost of losing a forecast office in their community
The restructuring dictated a degree of standardization between forecast offices, however it has become apparent that this needs to be effectively balanced with the flexibility needed to allow for customization at individual offices to respond to local requirements.
The MAR increased the overall education level of the workforce and set in place the need for routine training to keep the staff on pace with technological
and meteorological advancements in the community. Staff development through in-person, hands-on training in a centralized classroom or laboratory of the type that occurred during the MAR has great value. Where relevant, online courses or self-directed study can be a useful supplement, but can sacrifice quality of learning and the connections made with colleagues that are essential to the overall operations of the NWS.
While the MAR was a reengineering of the NWS, its execution depended on the involvement of many partners. Development and deployment of all the observational systems of the MAR involved other NOAA line offices (e.g., NESDIS, OAR) as well as other federal agencies. The NWS worked with the private sector through contracted work, and the research community played a large role in the development and demonstration of MAR technologies. In general, the MAR strengthened the relationships between the NWS and other members of the weather enterprise, although in the case of the private sector, it took some time after the MAR to develop these strengthened relationships.
Lesson 5 – Partnerships
The execution of the Modernization and Associated Restructuring required working with many partners, which provided cost-sharing and improved understanding of user needs. However, the relationships with the partners were not always as well conceived or managed as would have been desirable. This could have been avoided by involving all known stakeholders (e.g., other agencies, academia and the research community, the private sector, media, and emergency management) from the outset. The National Weather Service (NWS) operational staff is also a stakeholder, and need to be involved early in the design and procurement process to ensure system functionality and practicality. Engagement with stakeholders from both inside and outside the NWS would help the NWS better understand user needs and secure ‘buy-in’ to new initiatives.
Throughout the execution of the MAR, the NWS received a large amount of oversight and technical advice both from within and outside the government. In many cases, the reviews drew attention to important issues, issues whose resolution was important to the success of the MAR. Successful reviews not only help management understand and react to technical, schedule, and budget issues, but help build communities of knowledgeable support.
Lesson 6 – Oversight and Advice
The Modernization and Associated Restructuring of the National Weather Service (NWS) showed that candid yet non-adversarial advice from outside experts and other interested parties was useful in the design and deployment of a large complex system. Because NWS management was receptive to such oversight and advice, the outside input was effective.
The MAR was a large, complex process that lasted a decade, and cost an estimated $4.5 billion. Despite issues, some more significant than others, in the end the MAR was a success. New technologies deployed during the MAR now provide forecasters with more observations of higher quality. NWS forecast and warning products were dramatically improved, in both quality and quantity. NWS now has stronger relationships with many of its partners in the weather enterprise. Changes in the distribution of field offices have allowed stronger connections with local communities. Weather services have great value to the nation, and the MAR was clearly well worth the investment. In Phase II of its study, the committee will build on the lessons presented in this report to develop actionable recommendations for the NWS to best plan, deploy, and oversee future improvements.
