Toward a New National Weather Service: Preliminary Assessment of the Operational Test and Evaluation Process for the Advanced Weather Information Process Systems (1996)

Toward a New National Weather Service

Preliminary Assessment of the Operational Test and Evaluation Process for the Advanced Weather Interactive Processing System

Since 1991, the National Weather Service Modernization Committee of the National Research Council (NRC) has been continuously involved in reviewing the plans for and development of the Advanced Weather Interactive Processing System (AWIPS). In 1995, the National Oceanic and Atmospheric Administration (NOAA) asked the NRC to “review plans and progress on the development and implementation of the Advanced Weather Interactive Processing System.”¹ This letter report specifically addresses the subtask of “evaluat(ing) the adequacy of the conduct of the AWIPS operational test and evaluation [OT&E] and the results or actions taken by NOAA and NWS [National Weather Service] in response to the tests and evaluations. ” A more comprehensive and detailed study by the committee is under way, which will include a full description of the AWIPS OT&E process, the results of NOAA and NWS responses to OT&E, and extensive background information and justification for the committee's findings and recommendations. Although the observations, findings, and recommendations presented here are preliminary, the committee welcomes this opportunity to advise the NWS early in the AWIPS deployment process.

AWIPS consists of 130 geographically dispersed sites connected by a network. The sites include weather forecast offices (WFOs), river forecast centers (RFCs), and National Centers for Environmental Prediction (NCEP). The network includes a satellite broadcast network (SBN) for disseminating satellite images and the output (gridded data products) from weather prediction models produced by the NCEP, a wide area telecommunications network that connects the sites, and a network control facility (NCF). The NCF monitors the health and status of all AWIPS sites and can transfer software upgrades to the sites.

According to the original AWIPS contract, development of the system, culminating in the delivery of full AWIPS capability to all sites, was scheduled to take five years. Several independent review bodies recommended that NWS adopt an evolutionary process, which requires incremental deployment of components of the system. Operational feedback could then be used to guide the development of the final system. ²; Incremental deployment is crucial in this

case because the NWS is evolving new operational practices and procedures in parallel with AWIPS deployment.

The incremental deployment of AWIPS began with an operational prototype known as Pathfinder. For the past year, Pathfinder has been undergoing evaluation at the WFO in Pittsburgh, Pennsylvania, the co-located WFO/RFC in Taunton, Massachusetts, and the NCF. In addition to demonstrating the early functional capability of AWIPS in an operational setting, Pathfinder has demonstrated the functionality of the AWIPS SBN and the interaction between the NCF and operational sites. Pathfinder has also demonstrated the NCF's ability to detect faults in the system and to transfer new software to field sites with a minimum of disruption.

For the next several years, incremental development and deployment will continue with a series of software upgrades (called “builds”) that will increase the functional capacity of AWIPS. The process for verifying the functionality and performance of each build is well defined. First, a build undergoes a period of testing at the factory of the contractor to verify that the functionality and performance of the software meet specified requirements for that build. This testing is followed by OT&E at field sites to confirm that the functionality and performance are retained in the deployed system, to establish that NWS personnel at the sites can use the system, and to identify operational problems or deficiencies that must be addressed in subsequent builds or software corrections.

The initial installation of AWIPS includes the initial software release (Build 1) and the hardware. Once the hardware and Build 1 software have been deployed, OT&E will be conducted to obtain feedback from forecasters on the functionality, performance, and utility of the system. The OT&E plan calls for Build 1 to be deployed and evaluated at nine field sites. The results will be evaluated as part of the AWIPS production decision to authorize the full-scale deployment of AWIPS at all 130 sites.

Prior to observing the OT&E of Build 1, the committee observed the performance of the WFO advanced meteorological applications software (WFO-Advanced), which was developed by the Forecast Systems Laboratory of the NWS as part of the risk reduction for AWIPS. WFO-Advanced software runs on the AWIPS hardware and ingests the same data sets as AWIPS. The Forecast Systems Laboratory has stated that WFO-Advanced is functionally equivalent to the AWIPS Build 4.³; Although the detailed functional specifications for Build 4 were being revisited at the time of this report, the AWIPS program office verified the Forecast Systems Laboratory characterization of the functionality of WFO-Advanced.

The committee assessed the process being used by NWS for OT&E, including installation, training, preparation for OT&E, conduct of OT&E, and follow-up, within the broader context of the operational deployment of AWIPS. The committee reviewed the OT&E plan to ensure that the OT&E process was well documented and that evaluation criteria were clearly defined. Members of the committee observed the preparations for OT&E and its conduct at four of the nine field sites involved in the evaluation. The four sites were the RFC and WFO at Pleasant Hill,