2
Review of the MARD Plan

General Perspective on MARD

The NWS modernization has occurred in increments, proceeding more gradually than anticipated. Efforts to develop and implement the technical systems for the modernization have been delayed well beyond the original assumptions of the NWS planners. The NWS is currently targeting the summer of 1999 for operational deployment of the final suite of modernized systems and the completion of organizational changes that will approach end-state staffing.

AWIPS, which is the last of the new systems, is expected to be deployed by the summer of 1999 in a version that can be commissioned as an operational system. MARD is scheduled to begin shortly after this version of AWIPS, called "Build 4.2," is deployed and will continue for one year. The MARD Plan states that installation of AWIPS Build 4.2 at the MARD field sites is an assumption for the initiation of MARD. However, the operational test and evaluation (OT&E) of Build 4.2 will continue for several months after Build 4.2 is installed. The MARD Plan projects that the commissioning of AWIPS as the official system for use in field office operations will not occur until the early months of 2000, or about two-thirds of the way through the MARD year.

The timing of commissioning with respect to Build 4.2 deployment and the subsequent OT&E seems reasonable. The OT&E process can be expected to lead to numerous small changes in AWIPS, and perhaps a few major changes, during and soon after the rigorous testing period, which will continue for several months after deployment. These changes should be installed and confirmed prior to AWIPS commissioning. This timing is also fortunate for evaluating AWIPS for any software problems related to the change of year to 2000 ("the Y2K problem").

MARD will be in progress throughout the OT&E period, so changes resulting from OT&E will be made during MARD. The committee assumes that the system after AWIPS commissioning will remain unchanged until the year-long MARD is completed. If the schedule holds, there should be several months of MARD with the stable (after OT&E changes) AWIPS system. These timing considerations are important factors in the committee's assessment of the extent to which the MARD Plan can meet its objectives.

Even after MARD and the commissioning of AWIPS, improvements to AWIPS are planned to enhance its capabilities for supporting interoffice coordination and backup, improving system reliability, and delivering products and services. The committee recently reviewed the OT&E process that the NWS has implemented as part of the incremental deployment of AWIPS (NRC, 1997). If the NWS preserves and enhances the strengths of this process and addresses the weaknesses previously noted, OT&E will play a valuable role in continuing to improve the entire suite of technical systems and the operational procedures for employing them. The committee assumes that the AWIPS OT&E process will become an integral part of a continuing operational program evaluation.

MARD and the Public Law

The NWSMC agrees with the implication in the MARD Plan that, if MARD achieves its stated goals and objectives, MARD will satisfy the requirement in Public Law 102-567, Section 703(a)(4) for "a multistation operational demonstration which tests the performance of the modernization in an integrated manner for a sustained period."

MARD Goals and Objectives

The committee believes that the planned MARD will address all three MARD goals to a limited extent prior to AWIPS commissioning and more fully afterward. But the results will be conditioned by the state of the technical systems and operations in effect at the MARD sites. The adequacy of the one-tier concept, including the end-state



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 7
2 Review of the MARD Plan General Perspective on MARD The NWS modernization has occurred in increments, proceeding more gradually than anticipated. Efforts to develop and implement the technical systems for the modernization have been delayed well beyond the original assumptions of the NWS planners. The NWS is currently targeting the summer of 1999 for operational deployment of the final suite of modernized systems and the completion of organizational changes that will approach end-state staffing. AWIPS, which is the last of the new systems, is expected to be deployed by the summer of 1999 in a version that can be commissioned as an operational system. MARD is scheduled to begin shortly after this version of AWIPS, called "Build 4.2," is deployed and will continue for one year. The MARD Plan states that installation of AWIPS Build 4.2 at the MARD field sites is an assumption for the initiation of MARD. However, the operational test and evaluation (OT&E) of Build 4.2 will continue for several months after Build 4.2 is installed. The MARD Plan projects that the commissioning of AWIPS as the official system for use in field office operations will not occur until the early months of 2000, or about two-thirds of the way through the MARD year. The timing of commissioning with respect to Build 4.2 deployment and the subsequent OT&E seems reasonable. The OT&E process can be expected to lead to numerous small changes in AWIPS, and perhaps a few major changes, during and soon after the rigorous testing period, which will continue for several months after deployment. These changes should be installed and confirmed prior to AWIPS commissioning. This timing is also fortunate for evaluating AWIPS for any software problems related to the change of year to 2000 ("the Y2K problem"). MARD will be in progress throughout the OT&E period, so changes resulting from OT&E will be made during MARD. The committee assumes that the system after AWIPS commissioning will remain unchanged until the year-long MARD is completed. If the schedule holds, there should be several months of MARD with the stable (after OT&E changes) AWIPS system. These timing considerations are important factors in the committee's assessment of the extent to which the MARD Plan can meet its objectives. Even after MARD and the commissioning of AWIPS, improvements to AWIPS are planned to enhance its capabilities for supporting interoffice coordination and backup, improving system reliability, and delivering products and services. The committee recently reviewed the OT&E process that the NWS has implemented as part of the incremental deployment of AWIPS (NRC, 1997). If the NWS preserves and enhances the strengths of this process and addresses the weaknesses previously noted, OT&E will play a valuable role in continuing to improve the entire suite of technical systems and the operational procedures for employing them. The committee assumes that the AWIPS OT&E process will become an integral part of a continuing operational program evaluation. MARD and the Public Law The NWSMC agrees with the implication in the MARD Plan that, if MARD achieves its stated goals and objectives, MARD will satisfy the requirement in Public Law 102-567, Section 703(a)(4) for "a multistation operational demonstration which tests the performance of the modernization in an integrated manner for a sustained period." MARD Goals and Objectives The committee believes that the planned MARD will address all three MARD goals to a limited extent prior to AWIPS commissioning and more fully afterward. But the results will be conditioned by the state of the technical systems and operations in effect at the MARD sites. The adequacy of the one-tier concept, including the end-state

OCR for page 7
staffing model, cannot be fully tested until the transition has been completed from a combination of modernized and legacy systems to modernized systems alone. Similarly, the goal of refining new operational procedures and resolving implementation issues cannot be fully met before the transition has been completed. The goal of demonstrating (or documenting) that there has been no degradation in the quality of warnings and forecasts cannot be fully met until AWIPS is in use for disseminating these products, but the AWIPS dissemination functions will not be used until AWIPS is commissioned. The committee believes that AWIPS commissioning at the MARD sites will be necessary before the three MARD goals can be fully met. A reasonable period of operation relying on modernized systems, including AWIPS, for office operations will be essential for a successful demonstration. Design and Metrics of the MARD Evaluation Capturing Site-Specific Variability The MARD plan does not have an explicit provision for planned experiments, or controlled tests, of systems and procedures. Without planned tests, the MARD results will be limited to whatever system stresses or unusual conditions (severe weather events, power outages, etc.) happen to occur at participating offices during the demonstration period. The response to these conditions will be limited by the systems and procedures in use at the time. Therefore, it is essential that each participating office document the systems and technologies in use at that site, especially the legacy systems and experimental systems that are used concurrently with the modernized systems (even if the former are used only to supplement, confirm, or back up the modernized systems). It is also imperative that each MARD site document (1) the specific staffing configuration at the site, (2) the specific procedures for coordinating operations with other offices during routine and stressed operations, and (3) the internal (intra-office) and interoffice backup systems and procedures actually in place and (as need arises) actually used. The committee anticipates that there will be a great deal of variability among MARD sites in these and other respects, which should be captured in the MARD documentation and considered in the evaluation of results. Evaluating Operational Effectiveness and Quality of Service The metrics that will be used to evaluate operational effectiveness and quality of service rely heavily on subjective assessments by office staff and the immediate recipients of an office's products and services (i.e., its primary customers). These subjective assessments will be supplemented by statistics on NWS warning and forecast skills. Drafts of questionnaires and other evaluation forms for surveying staff and customers are included in the MARD Plan as appendices. In addition, the plan mentions other potential data sources (for example, forecast data from adjacent sites and the Product Availability Monitoring System) that could be used as supplements to subjective surveys. The MARD Plan states that the NWS will obtain assistance in refining the MARD design and in collecting and analyzing the data. The committee strongly encourages the NWS to use outside resources (e.g., a support contractor) to assist in these areas. Surveys The surveys appear to be adaptations of the surveys used in the AWIPS OT&E service evaluation. The adapted surveys focus on staff assessments of internal office operations; customer assessments of NWS and RFC products; and managerial assessments of the support from national and regional NWS offices and, when backup services are required, from other field offices. The MARD surveys would benefit from a close review by experts in survey design and program evaluation to assess their relevance to MARD objectives and analytic goals; parallelism between surveys; and the structure and clarity of survey questions, instructions, and formats for data collection, statistical analysis, and interviews. Analysis Plan The MARD Plan includes many variables, questionnaires, and respondents, spread across different time frames. But it does not explain how data will be analyzed and used to address the MARD goals. Streamlining the questionnaires and data collection processes will minimize the burden on respondents and administrative personnel. Specifying the methodology and objectives of analyzing data will help the NWS recognize which variables and data are essential and which are just "nice to have." Logistics of Data Collection The plan presumes that the numerous surveys will be collected by WFO and RFC staff. The committee strongly advises the NWS to consider the use of outside support services for this task to ensure that data are collected independently and systematically and that the staff is not unduly burdened. Data Utilization The plan does not specify how the MARD results will be incorporated into refinements of procedures and the resolution of implementation issues. The NWS should establish specific mechanisms for incorporating the MARD results into organizational practices.

OCR for page 7
Ongoing Operational Test and Evaluation Process An expanded OT&E process would provide a continuing assessment and improvement process for all of the modernized systems, not just for AWIPS. Periodic assessments after future AWIPS builds or other significant changes to one or more of the technical systems should be standard practice for the NWS. The end of Section 1 in the MARD Plan refers to "future operations program evaluation," which the committee interprets as a reference to the AWIPS OT&E process expanded to cover all of the modernized systems and operations. The OT&Es should include designed experiments to supplement observations from naturally occurring events. The sites selected for the MARD do not seem to provide for demonstrations of coastal marine or mountain weather forecasts and warnings. As a practical matter, these sites have a relatively low level of significant fire incidents to support. Thus, the network of sites in the MARD area will not provide a fair evaluation of marine, fire weather, or mountain weather operations, interoffice operability between these programs and other programs, or interoffice backup responsibilities for these programs. Successive OT&Es should therefore focus on different regions of the country (as is already being done for the AWIPS OT&Es) to assess operations for different weather regimes and seasons and different responsibilities. Expertise from outside the NWS should be brought in to supplement in-house capabilities. Outside expertise can be useful for solving technical and substantive problems, as well as for assuring managers in NWS, NOAA, the U.S. Department of Commerce, and Congress that the evaluations are independent and credible. The expanded OT&E process ( or operations program evaluation) should include the following elements: the development of plans and rehearsal of procedures for interoffice backup (see discussion below) assessments of staffing configurations (including shift structures, staffing combinations, and variations in the discretion allowed local staff in modifying procedures) under different types of weather conditions examinations of methods of network coordination under different conditions (including routine interactions among the WFOs, RFCs, and NCEPs and interactions during severe weather or periods of system stress) service assessments that include the systematic collection and evaluation of customer inputs, including customer satisfaction with dissemination and receipt of products and services from backup WFOs and RFCs, and customer comments on new ways to provide or improve products and services (local offices should identify current and potential customers from whom feedback should be obtained) the systematic collection of staff perceptions of operational and technical issues Finally, NWS should establish processes for incorporating OT&E results into operating procedures at field offices and national centers. Interoffice Backup and Operational Risk Management Field offices have long had procedures for backing up a failure of a key system with an alternative or work-around within the office. They have also established procedures for another field office (or offices) to take over the essential duties of an office incapacitated by a long-term power outage, loss of communications lines, or a similar event that would prevent an office from handling its full complement of routine responsibilities. The modernization provides many approaches for continuing, replacing, and extending these service backup capabilities, both within an office (intra-office backup) and between offices (interoffice backup). In the past, service backup procedures were developed by individual offices. With the completion of the NWS modernization, it is technically feasible to achieve much greater standardization and depth in backup and recovery procedures, particularly in the speed and effectiveness of interoffice backup. However, the MARD Plan does not specify whether procedures have been developed, tested, or rehearsed to ensure that the operational system of multiple field offices, central facilities (such as the Network Control Facility), and communications links (such as the AWIPS wide-area network) can use these new technical capabilities for effective interoffice backup. Since the OT&E for Build 1 of AWIPS, NWS has systematically tested the intra-office backup capabilities of the new technical systems, particularly of the communications links provided by AWIPS. NWSMC members have observed these tests and followed the analysis and responses to their results. As AWIPS has progressed to its current operational version (Build 4.1), the standard capabilities and procedures (i.e., those common to all offices) for actions such as failover to a backup server have been tested and practiced under near-operational conditions.3 In areas where alternative radar 3   "Failover" is the transfer of operations to a reserve or backup component of a system when a component ceases to function adequately or fails completely. Ideally, the transfer should occur with minimal loss of operational capability and no loss of data or processes. For good reason, the OT&E test teams have not conducted failover and other capability-degrading tests during times of significant weather in a field office's area of responsibility. In other respects, the realism of the test conditions has increased as the AWIPS system has matured. For example, in recent OT&Es the timing of deliberate "failures" (controlled outages) of a server or other system component was unknown to the staff at the field office or the Network Control Facility, which must identify and respond to failure conditions. The test teams have also conducted operational "stress tests" by bringing down multiple components and assessing the degree of degradation in operational capability, short of closing down an office completely.

OCR for page 7
coverage is technically possible (a function of the phenomenon to be observed and the distance and terrain between the NEXRAD installations and the location of the phenomenon), field offices have demonstrated the ability to acquire radar products from the "backup" NEXRAD. Individual offices have also worked out "internal" procedures for using various legacy or experimental components as backups and work-arounds to improve their intra-office backup capability. According to the latest draft statement of AWIPS Build 4.2 capabilities, this version should provide a great deal of the planned technical capability for enhanced interoffice backup (Box 2-1). The OT&E for Build 4.2 seems to be a logical alternative to MARD for controlled, but progressively strenuous, testing of intra-office backup capabilities and operational effectiveness, similar to what earlier OT&Es provided for testing intra-office capabilities. As further capabilities and incremental improvements become operational after Build 4.2, the OT&E process can be used to continue to validate, rehearse, and stress-test the expanding operational procedures and technical system capabilities for interoffice backup. For example, in the midst of certain severe weather events of limited areal extent the NWS could go into a simulated backup mode. This would require that the backup WFO not have any significant amount of severe weather (convective or winter storm-related) in progress within its primary area of responsibility. In this mode the actual public products would be issued by the responsible office (as normal), while the backup office simultaneously went through a mock exercise issuing forecasts and warnings that would not be publicly distributed. This type of exercise would yield valuable experience "under fire" for the backup office without increasing risk to the public. The NWSMC has previously discussed some of the technical capabilities of AWIPS that require testing and validation to ensure that site backup and recovery procedures operate as expected (NRC, 1997). The following suggestions for ensuring that backup is operationally effective, as well as technically demonstrated, should be added to those recommendations: The NWS should develop (or review and update) a list of situations in which an office might need partial or BOX 2-1 AWIPS Build 4.2 Capabilities That Support Interoffice Backup Backup Provide ability for the site that provides backup to access WSR-88D for the area being backed up (if available) via radar dial-up. Provide the ability for the site that provides backup to acquire local data for the area being backed up, via dial-in to or dial-out from its LDAD [local data acquisition and dissemination]. Provide the ability for the site that provides backup to receive and process ASOS, microART, and ROSA [dial-in reporting facility for Cooperative Observers] for locations in the backup area. Provide the ability for the site that provides backup to acquire routinely all SBN [Satellite Broadcast Network] data needed for the backup area. Provide the ability for the site that provides backup to display all local data and SBN data acquired for the site being backed up. The data will be displayable on the AWIPS local scale (state scale) with appropriate map backgrounds. Provide the ability for the site that provides backup to create all assigned text products for the are being backed up. This includes warnings, watches, and statements generated by Warngen [the AWIPS warning generation utility], and other products created manually on the text workstation. All products will have appropriate header information automatically prepended. Normally product Ids [identification numbers] and headers will be the same as those generated at the site being backed up, except that the "issued by" line will the office preparing the product. Message Handling and Communication Provide capability to request/reply for either NCF [Network Control Facility] or nearby WFO and RFC. During transition from AFOS [the legacy communications system] to AWIPS, provide ability to send products to AFOS and corresponding products to AWIPS WAN [wide area network] as appropriate. Automated test message to test load on system. Send subset of radar products via the WAN to limited number of nearby sites. Source: NWS, 1999

OCR for page 7
full backup from another office or offices, up to and including the complete failure of a field office (either a WFO or an RFC). The NWS should prioritize the list, based on likelihood, importance, similarity, and other pertinent factors. For similar situations, the NWS should choose one or more planning scenarios, based on their likelihood and importance (and perhaps additional factors). For each scenario, the NWS should think through (1) the criteria for deciding what action is necessary; (2) the steps to perform that action; (3) the equipment and other resources necessary for taking each step; and (4) the preparations that should be made in advance, such as acquiring the telephone numbers of local data acquisition and dissemination users and determining alternative communication pathways. The NWS should develop plans and procedures for carrying out the action for each scenario and commit them to writing. Depending on the problem and the action, these written plans and procedures can be used as "templates" to be filled in with details for individual offices. The generic plans and procedures should be useable by all offices with minimal modifications. The NWS should provide the template plans and procedures, along with the categorization of potential circumstances, to each field office. Each office should be required to study the plans, make adjustments for local conditions, and develop written office-specific versions. If the written version is stored as an electronic file, each office (and its backup[s]) must also have hard copies available in case of power outage or other mishap that would interfere with electronic access. The NWS should train office personnel by scheduling near-real operational exercises and simulations when local weather activity is minimal. The guiding principle for the amount of detail in the procedures and the number of rehearsals should be to preclude encountering an emergency for the first time when lives may depend on proper actions being taken. In terms of preparedness, much of the benefit would be derived from the planning and preparation of nearly realistic exercises and simulations.