An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build (1997)

The last major technical system required to complete the decade-long modernization of the National Weather Service (NWS) is the Advanced Weather Interactive Processing System (AWIPS). When commissioned for operation, AWIPS will be a distributed data processing system used at NWS field offices, regional offices, and headquarters to integrate information received from all other observational and analytical elements of the modernized system. AWIPS will support the work of hydrologists and meteorologists by delivering state-of-the-art forecasts and warnings. AWIPS will also provide a nationwide communications network, with the NWS offices as its nodes.

In 1995, NWS adopted an incremental approach to developing and deploying AWIPS, an approach in which increasingly capable software “builds ” are deployed to increasing numbers of field offices. With each build, the system grows in terms of both operable functions and the number of nodes in the operating network. Each deployment of a major new build requires a period of field testing by real users (forecasters) engaged in real operations. The results are fed back to AWIPS program managers and developers to guide subsequent incremental development and deployment (IDD). The first operational test and evaluation (OT &E) occurred in the fall of 1996 when AWIPS Build 1 was installed at 12 sites, 9 of which are NWS weather forecast offices or river forecast centers. In this report, the National Weather Service Modernization Committee (NWSM Committee) presents a retrospective evaluation of the first OT&E as an essential component in bringing AWIPS to maturity and a prospective commentary on key issues raised by the OT&E.

The results of the OT&E were gathered from on-site teams; formal questionnaires prepared by AWIPS users; teleconferences among field staff, program managers, and developers; and formal reports of the network control facility (NCF), which acts as the network manager for the entire AWIPS. These mutually reinforcing modes for OT&E feedback contributed greatly to its success. The following major system improvements, made in response to OT&E results, highlight the value and effectiveness of the OT&E process:

• Intermittent problems with reception of the satellite broadcast signal that conveys satellite data, weather model data, and other information from the NCF to field sites were diagnosed and now appear to have been corrected.

• The workstation processes for retrieving weather model data for display were confirmed to be too slow for use in operations. NWS managers relied on these and other OT&E results in deciding to replace the user interface portion of the Build 1 AWIPS with an alternative that had been developed in parallel with the main design. The alternative interface will be incorporated in AWIPS Build 3.

• Problems with NCF performance will be addressed partly by the previously planned addition of automated trouble detection and recovery processes in later builds and partly by changes in staffing.

• The procedures and automated tools used during the OT&E to catalog and track reported problems have evolved into an effective change management process and configuration management tool for the AWIPS program.

Based on these observations, the NWSM Committee concludes that the OT&E process is an effective, valuable, and necessary element in the incremental development and deployment of a system capable of fulfilling the objectives of the NWS modernization program.

Recommendation. The AWIPS program should continue, and perhaps expand, a deployment strategy of maximizing the diversity of weather office forecasting operations to test the performance ranges and to identify site-specific and systemic improvements as early as possible.

Recommendation. The AWIPS program should, as already planned, include a formal OT &E when each major build is deployed to ensure that operational performance continues to improve.

Looking beyond the Build 1 OT&E, the NWSM Committee suggests improvements in two areas as the NWS proceeds with work on AWIPS. These two areas are systems engineering and operational risk management.

An AWIPS systems engineering team was named in May 1997, but a committee-like team may not be able to perform all of the functions normally undertaken by a project systems engineer and a team of subsystem engineers who have been assigned specific responsibilities and authority. As the NWSM Committee understands the current AWIPS organization, no single individual has the assigned responsibility and the delegated authority to act as a system-wide, comprehensive systems engineer. Nor does the AWIPS program have a hierarchical systems engineering team composed of subsystems engineers. Instead, members of the systems engineering team appear to have been assigned to represent organizational entities. The AWIPS contractor's systems engineer is not a regular member of the AWIPS systems engineering team.

Among the outstanding issues that call for a strong systems engineering approach are (1) problems related to locally developed code, which is application software written by NWS field office staff for local use but attached to the AWIPS software system, and (2) the difficult task of transferring development and maintenance responsibilities for major software modules from the development group that initially writes the software to another group (i.e., the transfer of ownership of AWIPS software). The second problem is most obvious in the transfer of ownership of the new workstation interface and data retrieval modules from the Forecast Systems Laboratory, which developed them, to the NWS Office of Systems Development.

Recommendation. The NWS should establish the position of AWIPS project systems engineer with system-wide, comprehensive responsibility and delegated authority for performing the functions detailed in this report.

Recommendation. The NWS should formally establish an AWIPS systems engineering team led by the AWIPS project systems engineer and composed of systems engineers for each subsystem.

Recommendation. The AWIPS contractor's systems engineer should be formally and directly included on the AWIPS engineering team, and the contractor's systems engineering processes should be included in the overall AWIPS systems engineering functions.

Recommendation. The AWIPS systems engineer should ensure that processes are in place to (1) accommodate locally developed code, including programming support, testing support, and regression testing to ensure that commissioned functions continue to perform as expected; (2) provide sufficient resource margins to accommodate a reasonably expansive scenario of local code development; and (3) manage the system configuration, including locally developed code.

Recommendation. Under the leadership of the AWIPS project systems engineer, AWIPS management should establish a process for transferring responsibility for, and the detailed knowledge of, software elements from the developing organization to the deploying organization and subsequently to the operating organization. If the complexity of the system, its components, or its interfaces precludes a complete transfer, the AWIPS project systems engineer and AWIPS managers should, through formal agreements, ensure that support will continue to be available.

The AWIPS program has already adopted a number of risk management practices that have been valuable in reducing development risks. As work progresses for Build 4, risk management for operational performance of the full, multisite AWIPS network is becoming the focus of risk management. The NWSM Committee has specific suggestions to offer on several aspects of continuing risk management.

First, even with the current contingency plans for the AWIPS satellite broadcast network, the committee is concerned that the master ground station in this network is a potential single point of failure for the entire AWIPS. Second, although plans have already been made for improving the performance of the NCF (by addressing problems with its troubleshooting and recovery functions for active AWIPS nodes), the committee encourages the NWS to demonstrate that these corrections will provide a reasonable margin of safety for the fully implemented AWIPS. Third, the automated processes and operator procedures for a neighboring site backing up a “down” field site and for recovering operational capability at the down site should be carefully reviewed and tested for the possibility of cascading failures, which could bring down many sites. Fourth, the mechanisms in place for emergency replacement of critical hardware components should be tested periodically. Fifth, physical threats to external communications systems that support AWIPS or unauthorized access to AWIPS can jeopardize operational performance and should be included in the risk management plan.

Recommendation. A realistic operational test of the contingency plan for failure of the master ground station should be planned and conducted well before AWIPS

is commissioned. The AWIPS risk management program should include (1) an exploration of scenarios under which the alternative uplink is unavailable and (2) an evaluation of remedial actions.

Recommendation. NCF performance should be watched closely to ensure that necessary improvements are forthcoming. This monitoring should be a top priority in the Build 3 time frame.

Recommendation. If improvements in the Build 3 time frame do not bring NCF performance up to operational standards, the AWIPS program should begin a risk reduction program to find a systemic solution to NCF performance problems. The NWS should consider reevaluating the design assumptions for monitoring and problem solving and should explore a wider range of solutions. At a minimum, NWS should reexamine the feasibility of the fundamental design concept for the NCF in light of experience since the Build 1 deployment.

Recommendation. To assess NCF performance and evaluate the NCF design concept, particularly as the number of active nodes in the AWIPS network increases, current or alternative NCF operations to perform designated emergency recovery functions should be tested under realistic conditions.

Recommendation. The AWIPS team should develop a plan to test the backup and recovery scenarios for AWIPS sites under field conditions. Documented procedures should be used to ensure that the system will perform as designed. A comprehensive analysis of failure modes for AWIPS as a system should be performed to identify all potential failure modes and develop preventive measures and recovery procedures to protect the system.

Recommendation. The site backup and recovery testing planned in the Build 4 time frame should include a thorough evaluation of the potential for the cascading failure of nodes. As many conditions under which such failures might occur as can be identified should be included in tests of the system's ability to detect and limit cascading failures.

Recommendation. Some form of periodic “drill” to test vendors' capability to replace system-critical hardware within the contractually agreed upon time should be included in the AWIPS risk management plan.

Recommendation. Detailed contingency plans for countering external threats to the integrity of the AWIPS system should be an integral part of the AWIPS risk management plan.