An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build (1997)

The National Research Council established the National Weather Service Modernization Committee (NWSM Committee) in 1990 to review and evaluate the modernization program of the National Weather Service (NWS) under Public Law 100-685, Title IV (1988). Since the NWSM Committee's inception, reports have been published under the series title Toward a New National Weather Service. Starting with the first report (NRC, 1991), the committee has commented on the status of the Advanced Weather Interactive Processing System (AWIPS), emphasizing the key role of this computer-based communications and data processing system in integrating information from all other elements of the modernized system and delivering properly interpreted forecasts, warnings, and other products to the public. A subgroup of the full NWSM Committee, identified as the AWIPS panel, has been assigned to keep abreast of the details of AWIPS development and report on them to the full committee.

In keeping with the rules of the National Research Council, the work of the AWIPS panel and of the full NWSM Committee has been restricted to making technical and scientific assessments within the scope of the committee's statement of task. Members of the committee are unpaid volunteers selected and approved by the National Research Council, not by the NWS or any other government agency. The conclusions and recommendations of the committee are presented to NWS only through written reports that have undergone a rigorous external peer review supervised by the National Research Council.

This is the committee's second report focusing entirely on AWIPS, although recommendations urging attention to delays in this vital component of the

modernized weather service have been included in four other committee reports (NRC, 1991, 1992, 1994, 1996a). As AWIPS development and deployment proceeds over the next several years, further reports will be forthcoming.

The original plan for AWIPS was based on sequential, temporally distinct phases of design, development, and deployment. In light of the problems with AWIPS development—problems frequently encountered in the development of complex, integrated systems—and as recommended by an independent review requested by the National Oceanic and Atmospheric Administration (NOAA) (Kottler, 1994), NOAA and the NWS adopted a more evolutionary approach to AWIPS development, in which increasingly complex “builds ”¹of the system will be deployed incrementally (see Appendix B). Each deployment is followed by field testing of the functions in that build. Test results and user reactions under operational conditions are then used to guide succeeding stages of system development. The formal, systematic testing and gathering of information on the performance of a newly deployed build is called operational test and evaluation (OT&E). In an incremental approach to system development, an OT&E period should follow the deployment of each major build.

In February 1996, the NWSM Committee agreed to participate in the planning and execution of the OT&E for the first deployed build of AWIPS (Build 1) by providing an independent, outside review (Appendix A). Although the committee's AWIPS panel continued to act as the principal “eyes and ears” of the committee, other committee members also participated in meetings with NWS staff and on-site visits. Prior to the actual OT&E, committee members reviewed several rounds of draft OT&E planning documents, discussed the documents and the planning approach with NWS staff to clarify what was being done and why, and visited the system test facilities at both NWS headquarters and the prime contractor's facilities. During the early weeks of OT&E, teams of committee members visited four of the nine field sites at which AWIPS Build 1 had been installed. During these visits, the committee members observed the OT&E firsthand, watched NWS field staff as they worked with the new system, and discussed users' reactions to AWIPS Build 1 and the OT&E process.

In September 1996, the NWSM Committee issued a preliminary assessment of the Build 1 OT&E and an evaluation of the AWIPS incremental deployment approach that could be used as input to a key deployment decision for the AWIPS program by the secretary of commerce (NRC, 1996b). Since that preliminary assessment was issued, the full NWSM Committee has been briefed at its four

regular meetings by NOAA and NWS staff and by the AWIPS panel on the continuation of the OT&E. These briefings included reports on efforts by NOAA/NWS and the AWIPS contractor to incorporate lessons learned from the OT&E into program decisions, system redesigns, and the ongoing development of AWIPS Build 2 (primarily communications and infrastructure software upgrades) and Build 3 (integration of an improved workstation user interface). AWIPS panel members and committee staff met with NWS and NOAA staff and the contractor on four additional occasions and were present at the formal design review for AWIPS Build 3. Panel members and staff have also monitored the periodic (initially weekly) teleconferences among staff from the AWIPS Build 1 field sites, NWS and NOAA staff responsible for the program, and the software developers.

The information obtained through these sources is the basis for the committee's evaluation of the Build 1 OT&E and of responses to the OT&E by NOAA/NWS. Chapter 2 is the committee's general evaluation of the OT&E processes implemented for Build 1 and the overall response to OT &E results. Chapter 3 and Chapter 4 focus on specific issues in AWIPS development the committee believes are important enough to the long term success of AWIPS—and therefore to the success of the NWS modernization—to warrant more detailed comments.