An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build (1997)

The OT&E process implemented by the AWIPS development team is an effective, valuable, and necessary element in the incremental development and deployment (IDD) of a system capable of fulfilling the objectives of the NWS modernization program. The remainder of this chapter explains the basis for this conclusion by first reviewing the basic structure of the Build 1 OT&E and then presenting the significant results for the AWIPS IDD and the long-term success of AWIPS in the modernized National Weather Service.

The OT&E for AWIPS has two complementary parts: a system evaluation and a services evaluation. The system evaluation focuses on system performance and its impact on forecasting operations. The services evaluation assesses the broader impact of the new capabilities on forecasts and other products and services. The system evaluation was designed not only to test the system against its contractual performance requirements in operational conditions but also—as an appropriate and necessary part of an IDD process—to evaluate whether the system's performance is satisfactory for users. The evaluation included questionnaires on site installation, checkout, and training, as well as the forms used to gather data on functional tests of system performance and user reactions during the period of intensive OT&E surveying for each site. The Build 1 services evaluation was based primarily on a questionnaire that asked AWIPS users in field offices to compare AWIPS performance with current systems. In the OT&E for later AWIPS builds, the services evaluation will also include substantial input from users of field offices' services and products.

Each part of the Build 1 OT&E had its own national evaluation team, its own questionnaires, and its own on-site teams, which were responsible for involving field office staff in the tests and evaluations and ensuring that the questionnaires were completed and forwarded to headquarters for the national evaluation. In addition to structured-response questions, all of the OT&E questionnaires included room for, and encouraged, free-form comments from users.

The OT&E period at each site started after installation and checkout of the AWIPS Build 1 system and a week of on-site training for the local staff. The on-site evaluation teams were most active during the first week or so after this training week. Installation at the nine field offices was staggered over six weeks, so the intensive week of OT &E activity overlapped at two sites, at most, at any given time.

The OT&E for AWIPS is intended to evaluate the effectiveness and suitability of each incremental build to support field services for warnings and forecasts. The objectives of the OT&E process specified in the OT&E plan are to:

• evaluate the effectiveness and suitability of AWIPS to the NWS mission and operations

• identify necessary modifications or improvements in AWIPS or in NWS operational procedures

• provide information on organizational and personnel requirements for logistics, maintenance, and real-time support

• verify the adequacy of manuals, documentation, and training

• verify that the system and all interfaces function as required in an operational environment

Based on the review of the OT&E plans and procedures, observations of OT&E implementation, participation in regular teleconferences, and a review of OT&E systems and services evaluation results, the NWSM Committee concluded that the OT&E process has accomplished its objectives. Although some major problems were found with Build 1 during the OT &E, these problems were uncovered precisely because it was tested under operational conditions. In addition to identifying necessary improvements, the Build 1 OT&E led to substantial improvements in the practices and procedures for making AWIPS operational.

One of the primary reasons for adopting the IDD approach (Kottler, 1994) was to allow for frequent evaluations of system performance and to provide early feedback. The NWSM Committee considers the OT &E process to be crucial to the success of AWIPS operation and strongly supports the NWS plan to conduct an OT&E at the time each major system build is deployed (NWS, 1996).

As a follow-up to the review of the OT&E process, the committee also

reviewed the actions taken by NWS in response to problems identified during the OT&E. The objectives of this review were to determine whether the problems were resolved in a timely manner and whether the fixes actually improved operational performance. Based on the committee's review of the work-off rates of discrepancy reports (DRs), the installations of intermediate builds, and modifications to processes and procedures based on the results of the OT&E, the committee believes that NWS actions in response to problems were appropriate and did improve the operational performance of the AWIPS.

This section summarizes some of the major product and process improvements that resulted from the Build 1 OT&E.

Early in the OT&E, satellite signal interference was encountered at some of the field sites. The problem was severe at Dodge City, less severe at Wichita and Topeka, and did not occur at the other sites. An initial analysis suggested that the cause was interference from local radio frequencies. However, as the OT&E continued to collect data and identify possible causes, engineering analysis indicated that there were systemic weaknesses in the design for signal processing and coding and in the ground-site antenna covers intended to protect the antenna dish from precipitation. A number of participants in AWIPS, including AWIPS OT&E staff and program managers in NWS, the AWIPS Acquisition Office in NOAA, the network control facility (NCF), the prime contractor, and several subcontractors, cooperated in working out the analysis and ensuring that corrective actions were taken to solve the problem completely. The solutions—broadcasting the signal via a different transponder with a better noise environment and changing the design for melting snow off the antenna—have now been tested in the operational environment and have greatly improved signal reception. In addition, demodulators to improve error correction algorithms are in production and are scheduled to be installed at field sites after the Build 3 OT&E.

Several major performance issues associated with the Build 1 workstation user interface first identified during the development and test of Build 1 were subsequently confirmed by the OT&E as significant operational problems. For example, the performance of AWIPS Build 1 when retrieving model data was unacceptable in operation, particularly when multiple data sets were requested. The Build 1 AWIPS system was designed to be completely flexible on the number and types of parameters that could be requested by a forecaster. This design feature turned out to have serious performance problems—such as long delays—

when multiple data sets involving derived parameters were requested for assimilation into a single loop display.

To resolve this problem and others related to the response times of workstation operations and the layout of the graphical interface, NWS decided to incorporate the superior user interface features of the AWIPS prototype known as “WFO-Advanced” into the formal AWIPS delivery baseline. ¹ WFO-Advanced was developed by the NOAA Forecast Systems Laboratory (FSL) as a risk reduction prototype with the principal objective of ensuring the technical feasibility of writing software to encode the algorithms for meteorological applications envisioned for AWIPS. This prototype evolved from more than a decade of exploratory development and testing by FSL and thus had already incorporated significant user feedback on the format and content of display interfaces.

Some NWS staff at various Build 1 field sites, as well as AWIPS program staff, were already familiar with WFO-Advanced, and some had been able to use the prototype system at FSL or other locations. Even before the Build 1 OT&E, AWIPS program staff had known about features in WFO-Advanced that seemed superior to corresponding features in the AWIPS user interface. The responses of users during OT&E provided overwhelming confirmation that certain aspects of the AWIPS Build 1 interface were not well suited to users' needs and preferences (not “user friendly”). Also, the data storage and retrieval techniques used in WFO-Advanced were judged to be better than the AWIPS Build 1 design for some functions, particularly for the rapid retrieval and display of model data distributed over the satellite broadcast network (SBN).

In the NWSM Committee's interim report (NRC, 1996), the committee made the following recommendation:

To meet the full operational requirements of AWIPS, the NWS should take advantage of alternative solutions such as the functional capabilities demonstrated in WFO-Advanced.

In truth, the committee's recommendation was just one of many suggestions and recommendations from various sources. The comments and evaluations from users during OT&E provided hard data supporting the change. By mid-November 1996, the decision had been made to incorporate the WFO-Advanced user interface with its superior information handling capabilities and more robust data reception and storage capabilities into the AWIPS system.

The technical approach adopted by the AWIPS Program Office in NWS ² and approved by the AWIPS Acquisition Office in NOAA required FSL to prepare a

“deliverable” version of its workstation software, referred to now as “Forecast Office-Advanced” or FOA. The Office of Systems Development in NWS has been assigned the technical lead in integrating the FOA code into the AWIPS site software, with FSL and the prime contractor playing supporting but critical roles. ³ The first integrated version will be deployed in AWIPS Build 3. Thus, the Build 1 OT&E provided the basis for a major change in the AWIPS design.

The NCF, which is located near NWS headquarters in Silver Spring, Maryland, began performing its duties as AWIPS network administrator when the first AWIPS site became active. NCF's duties include remote monitoring via the network of the AWIPS servers and workstations at each field site; taking calls from field sites when problems are encountered; diagnosing and solving problems; acting via the network or through directions to field office staff to correct problems found by either route; and maintaining records of problems (by writing up trouble tickets). The role of the NCF as trouble-finder and trouble-fixer for all nodes in the AWIPS network is a key element in the NWS plan to maintain high levels of performance and reliability without burdening the field office staff with system administration and maintenance duties.

During and after the formal OT&E, field office staff reported problems with NCF's performance of its designated troubleshooting and recovery functions for the active AWIPS nodes. The NCF failed to meet its design concept and, more important, the operational needs of the field staff at AWIPS nodes.

According to the current design concept, AWIPS users report problems to the NCF (over the wide area network or by telephone), and the NCF fixes the problem. Users do not have to rely on a system manager or an on-site information systems specialist to diagnose the problem and perform the appropriate recovery procedure. For many problems with workstations or servers, the NCF staff can perform the recovery procedure remotely. In fact, the network infrastructure has enough built in automated monitoring and control so that it is technically feasible for NCF operators to identify and resolve many problems before the site staff is even fully aware of them. A crucial argument in favor of this design concept is that it will substantially reduce system life cycle costs, particularly with regard to constraints on site staff under the NWS Modernization program, compared to systems in which individual sites must handle most of their own monitoring and recovery.

The performance of the NCF improved as factors contributing to the inadequate initial rate of response were identified and corrective actions were taken. For example, the NCF software in place for the OT&E required a much higher

level of operator attention to site activity and more manual entry of commands for recovery sequences than was planned for a fully functioning AWIPS. NCF staff have also had experience with the AWIPS monitoring and control functions and with their own administrative responsibilities and action sequences. The latest in a series of build-by-build increments to automate NCF monitoring and control functions, including firmware called MC/ServiceGuard, was incorporated into Build 2, which was released in May 1997.⁴

The procedures for reporting problems also revealed staffing difficulties at the NCF, including high turnover, staff who were not knowledgeable enough to address routine problems efficiently, and insufficient staff on all shifts to handle problems even at the current AWIPS and Pathfinder sites. Although the contractor has taken steps to correct these deficiencies and improve NCF staff performance, much remains to be done before NCF performance can be considered adequate. (See Chapter 4 for the committee's assessment of NCF's current status.)

The FOA integration into AWIPS is the most significant product improvement to result from the Build 1 OT&E, but the periodic teleconference is the most significant process improvement. During the approximately six weeks of intensive OT&E, daily teleconferences were held for all parties involved at that point in the AWIPS OT&E. The participants included the AWIPS program manager, the evaluation team leaders (often with other team members present), one or more key technical managers for the prime contractor, someone from the NCF, and one or more staff members at each of the nine field site where AWIPS Build 1 was operational. These teleconferences were candid reports of what was happening— good and bad—at each field site, with no-holds-barred discussions of what needed to be done, what priority it should have, and who should do it. The teleconferences continued after the period of intensive OT&E but became less frequent, changing from daily to weekly during the latter part of the formal OT&E and then becoming less frequent by the spring of 1997 (six months after the beginning of Build 1 OT&E).

The significance of the teleconferences is in the way they are used. They have been open forums with broad participation (especially important is the participation by field office staff), serious attention from decision makers, and follow-through on the issues raised. Although teleconferences are held

less frequently between formal OT&Es, the “open forum” culture has been maintained reinforcing the continuing, constructive involvement of busy field staff members as “operational testers” and ensuring that managers and technical leads receive steady feedback, both positive and negative, on results of IDD.

Teleconferences since the Build 1 OT&E formally ended in October 1996 have been used to plan and review releases of software between builds, as well as the Build 2 release in May 1997. The teleconference culture has also been used to good effect by the geographically dispersed participants engaged in developing and testing the FOA-integrated AWIPS (Build 3). An assumption of the IDD approach to system implementation is that tough problems that require close cooperation among participants are bound to arise throughout the process. The robust systems engineering functions described in Chapter 3 are crucial to solving these problems within temporal and financial constraints. The teleconference culture that emerged during the Build 1 OT&E should play an important role by bringing problems and issues to the early attention of systems engineers and by providing a sounding board for proposed solutions.

As used in this report, a “change management process” is primarily an administrative process based on human actions and interactions, with bookkeeping and testing implemented through software. A change management process includes recording conditions that may require changes to a managed system, deciding if changes are needed and analyzing options, overseeing the execution of changes, and testing changes for resolving the initial problem. Change management processes usually make use of various automated systems, such as databases, electronic mail systems, project management and support tools, and an automated management tool called a “software configuration management tool.” Although the automated tool is often thought of as a complete configuration management system, in fact the rules established for its use are also part of the system. These rules include, for example, giving one person (the configuration manager) supervisory control of the automated tool and establishing the rules software developers are supposed to follow when checking out modules and when checking them back in after changes have been made and the unit tested.

Prior to the Build 1 deployment, the AWIPS prime contractor's change management process included a software configuration management tool, a configuration management database for hardware configurations, documentation, and specifications. NWS units that were developing AWIPS software modules had different procedures and tools for configuration management. The procedures met the immediate needs of each developer-group as long as they worked independently, within the traditional model of software system implementation. However, in the spring of 1996, the AWIPS panel was concerned that the program as a whole did not have a change management process that encompassed all partici-

pants, including access to a shared configuration management tool that could stand up to the rigors and complexity of an IDD process. The panel addressed a number of questions to the program staff to clarify the situation. The program staff subsequently briefed the panel and the NWSM Committee indicating that progress had been made toward implementing a better system. Nevertheless, some of the glitches that occurred during the deployment of Build 1, particularly at the early sites, resulted from weaknesses in the change management process. ⁵

An effective change management process covering all current AWIPS IDD participants has now emerged in response to the OT&E discrepancy reporting process. The preparations for the formal OT&E included a system for deriving a formal discrepancy report (DR) from trouble reports filled out by users at field sites, from NCF trouble tickets, and from other formal and informal user responses during the OT&E. Problems reported might involve AWIPS hardware, software, telecommunications, or documentation. The software DRs were entered and maintained in a software configuration management database using the same management tool (package) that the contractor had been using, but broadened in administrative scope to include configuration management of all the AWIPS codes, including programs and modules that were the responsibility of NWS or NOAA entities. After determining whether a software DR represented a deficiency in meeting an AWIPS specification or would require an enhancement, a review board assigned the DR a priority and incorporated the planned fix into the schedule of AWIPS interim and major builds. The software DR was then distributed to the entity responsible for developing that software—either the contractor or one of the NWS units preparing government-furnished software.

The difficult part of creating an effective change management system, in the view of the NWSM Committee members, was to make the system comprehensive. To be effective, a change management system must capture as input all of the modes of indicating an operational problem. It must trace each problem to a (suspected) root cause in hardware or software (documentation deficiencies are easier to analyze), assign a priority, designate someone to fix the problem, and ensure that the problem is fixed. Throughout the problem report life cycle, the system must ensure that all parties follow rules for repairing each and every deficiency and ensure that modifications do not create new problems. As new participants are brought into the circle of AWIPS developers (such as FSL when the decision was made to incorporate FOA into AWIPS Build 3), the change management process must be modified to include them.

Based on observations to date, the NWSM Committee concludes that the change management process implemented during the Build 1 OT& E has been a

practical success in meeting all of the operational objectives described above. Maintaining the effectiveness of this system as AWIPS deployment expands, in terms of both capabilities and the number of field sites, will be critical to the success of AWIPS. A highly effective change management process is essential to keep the systems engineering components of the AWIPS team (see Chapter 3) abreast of emerging problems so they can assess them accurately and implement tested solutions. A particular concern of the committee, discussed in Chapter 3, is how the change management process will be expanded to cover “locally developed” software applications.

To appreciate the significance of the Build 1 OT&E, one must understand how many organizational entities are now “players” in the overall process of AWIPS development and the complex relations that have evolved among them. One prime contractor is still responsible for developing important subsystems of the software, integrating all the software and hardware into a baseline system, physically installing the system, operating the NCF, and training users (NWS staff at field offices, headquarters, and other sites). The prime contractor oversees and has contractual responsibility for the work of several subcontractors for the SBN and the production, installation, and maintenance of system hardware.

Application software is also being developed at the NWS Office of Hydrology and the Technique Development Laboratory. With the decision to integrate FOA into AWIPS, FSL became another major “development player,” and the NWS Office of Systems Development took on additional responsibilities as the integrator for all government-furnished software. The Modernization Systems Management Office (formerly the AWIPS Program Office), which is administratively under the NWS, and the AWIPS Acquisition Office, administratively within NOAA, continue to provide joint overall program management for the entire development team, including contractors and “in-house” members.

With the Build 1 OT&E, staff at field offices where AWIPS is installed also became important players in the ongoing IDD process. Two field offices, where the earlier Pathfinder prototype system had been installed in 1995, were already participants, to a limited extent. However, regular interaction between AWIPS users and other members of the team increased dramatically with the Build 1 OT&E and its periodic teleconferences. Status reports back to field offices through the AWIPS change management process, along with a reasonably good record for following through on user-reported problems, have kept system users involved in the IDD in ways that have contributed to the overall success of AWIPS.

The flow of information and products through this multicentered organizational network is not hierarchical although there are, necessarily, hierarchical trees for administrative, contractual, and decision-making purposes. But information, products, and cooperative efforts cut across the up-and-down hierarchies. The

network will continue to change as players come and go and, more importantly, as roles change. For example, the roles of the various developers will change as AWIPS matures. And field office staff (at least some of them) will begin to act as highly decentralized developers, as well as system users and testers. Specific issues related to the role of field office staff in local development are addressed in Chapter 3.

Based on the review of the OT&E plans and procedures, observation of how OT&E was conducted, participation in regular teleconferences, and a review of results of OT&E systems and services evaluations, the NWSM Committee has drawn the following conclusions regarding the OT& E process as implemented for the ongoing IDD of AWIPS.

Conclusion. The OT&E process implemented by the AWIPS development team is an effective, valuable, and necessary element in the IDD of a system capable of fulfilling the objectives of the NWS modernization program.

Conclusion. Continued success in the IDD process requires that each major AWIPS build be deployed at operational sites and evaluated in operational conditions.

Conclusion. The deployment of AWIPS at an increasing number of field offices with each new build is especially important for testing the system 's functionality and multisite interactions and revealing problems early so they can be corrected in subsequent builds.

Conclusion. Further deployment and OT&E of AWIPS at this stage of development is appropriate and essential to the ultimate success of the NWS modernization.

Recommendation. The AWIPS program should continue, and perhaps expand, a deployment strategy of maximizing the diversity of weather office forecasting operations to test the performance ranges and to identify site-specific and systemic improvements as early as possible.

Recommendation. The AWIPS program should, as already planned, include a formal OT &E when each major build is deployed to ensure that operational performance continues to improve.