Review of the Draft Plan for the Modernization and Associated Restructuring Demonstration (1999)

The public law that authorized the National Weather Service (NWS) to modernize its technical systems and restructure its field office organization set the requirements for what has become known as the modernization and associated restructuring demonstration (MARD). The law requires testing of the integrated performance, over a sustained period, of the modernized components and an operational demonstration that no degradation in service will result from the modernization and associated restructuring. In this report, the National Research Council's National Weather Service Modernization Committee reviews the September 1998 draft plan for MARD. The committee drew three conclusions from its review of the plan and offers five recommendations to the NWS.

If MARD achieves the goals and objectives stated in the MARD Plan, it will meet the requirement in the public law for a multistation operational demonstration that tests the performance of the modernization in an integrated manner for a sustained period.

The MARD evaluation plan is intended to assess a wide range of local office capabilities, operations, and services. Before this plan is implemented, it will require refinement to link proposed evaluation variables to the evaluation and analysis objectives, provide directions for documenting actual site conditions during MARD, and minimize the burden on staff and operations.

Controlled tests of multistation capabilities, such as interoffice backup, that progressively stress the system up to and including total failure of a field office are needed to ensure system robustness. Tests of this kind for routine coordination and for interoffice backup during failure, covering both system capabilities and operational procedures, can be incorporated into the ongoing process of operational program evaluation, beginning with the Build 4.2 operational test and evaluation (OT&E) for the Advanced Weather Interactive Processing System (AWIPS).

The National Weather Service should test failure modes at progressive levels of failure, including the complete failure of a field office (for both weather forecast offices and river forecast centers). These tests should be conducted during the Build 4.2 operational test and evaluation and demonstrated during MARD. The tests should focus on the effectiveness of capabilities and procedures for inter-office backup and intra-office recovery and document site configurations.

The National Weather Service should continue to make periodic assessments of technical, operational, and service capabilities as part of the ongoing operational program evaluation by expanding the operational test and evaluation process being used for AWIPS builds to include the integrated suite of technical systems. The assessments should include service programs such as coastal marine, fire, and mountain weather forecasts and warnings, which are not likely to be fully tested in the MARD area.

To ensure adequate evaluation of a stable system configuration, MARD should be continued for a reasonable period of time (several months at least) after AWIPS is commissioned at the MARD sites. During this period, the MARD sites should use the modernized systems without significant reliance on legacy systems.

The National Weather Service should use outside experts in program evaluation and survey design:

• to review the MARD evaluation plan, survey questionnaires, and other data forms
• to assist in developing a plan for analyzing the evaluation

• to assist in the collection of evaluation data in local offices

Surveys used in the service assessments of future operational program evaluations should also be reviewed by outside experts.

The National Weather Service should use an independent evaluator or evaluation board to assist in assessing MARD results.