Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 19
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build 3 Systems Engineering The AWIPS project has responded to increasing organizational and system complexity by appointing a systems engineering team. The NWSM Committee recognizes the value of this team as a forum for resolving systems issues but believes that further improvements are vital to ensure that AWIPS can be successfully implemented and maintained. In this chapter, the committee clarifies a concept of systems engineering for evolutionary systems development and recommends specific actions. Systems engineering facilitates the coordinated design of systems that are made up of many elements and subsystems of elements so the system as a whole can meet the requirements and constraints imposed on it. The definitions of “systems engineering” and “systems engineer” vary by organization and area of application. In this report, these terms, the underlying principles, and the implementing organizations are characterized in the context of the development of complex information systems. The responsibilities of the systems engineer include compositing requirements, developing a system architecture, allocating requirements to subsystems and elements, identifying and managing performance and resource margins, and tracking the development process to evaluate results. Throughout the system life cycle, the systems engineer anticipates, recognizes, and solves problems to ensure that the system continues to fulfill requirements within project resources. System optimization may (and often does) entail the suboptimal design of some subsystems and elements. The systems engineer must have authority over the system/subsystem architecture, the allocation of requirements among subsystems, and the specification of interfaces between and among subsystems. The designer-architect functions of the systems engineer are performed
OCR for page 20
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build TABLE 3-1 Generic Systems Engineering Functions and Their Roles during Incremental Development and Deployment Generic Systems Engineering Functiona Applicability to Interbuild Phase of Incremental Development and Deploymentb Compile, composite, prioritize, and verify all customer requirements. Reconfirm priority and specifics of requirements in light of user response to interim builds (e.g., respond to user input from OT& E and other field testing activity). Define the system/subsystem architecture; derive requirements applicable to each subsystem; specify all internal (subsystem-subsystem) and external (system-environment) interfaces. Ensure either (1) that the implementation remains consistent with the overall architecture, subsystem requirements, and interface specifications OR (2) that changes proposed to the architecture, subsystem requirements, or interface specifications in light of ongoing redesign decisions are applied throughout the system and that all implications are understood and accepted. Determine the impact of all requirements on system, subsystems, and elements. Monitor and allocate requirements, resources, and tolerances (margins) across subsystems and elements, with increasing precision as implementation proceeds; ensure that impacts of redesign decisions on these allocations are understood and planned for. Derive an understanding of development risks, expendable resources, and costs. Allocate and manage resource margins (contingency resources) as needed; continue to update and refine risk analyses for performance, schedule, and cost (implementation costs and life cycle costs); monitor and manage resource margins on the basis of latest test data; undertake risk reduction activities as indicated by results of risk analyses; maintain and update a life cycle risk management plan. Conduct trade-off studies for optimizing the overall architecture to provide required robustness while minimizing risk, cost, and resource requirements consistent with system requirements. Repeat and/or broaden trade-off studies when dictated by most recent assumptions. Provide a model and mechanisms to test and ensure that subsystems meet the derived requirements and that the system as a whole meets the customer's requirements. Continue testing incremental implementations (builds) to ensure that subsystems still meet their requirements; revise model and test mechanisms as required by redesign decisions, changes in system or subsystem requirements, or risk analyses that impact model assumptions; perform baseline regression testing.
OCR for page 21
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build Monitor the progress of development; make systems design changes as appropriate to mitigate problems. At each build deployment with an OT&E, monitor results for indications of problems that may initiate any of the systems engineering activities specified above; make design changes to future builds on the basis of results. a The NWSM Committee wishes to acknowledge Frank Schutz (retired Deputy Director for Space and Earth Science Programs) and Arthur Zygielbaum (Senior Member of the Technical Staff and AWIPS Panel Advisor) of the Jet Propulsion Laboratory, National Aeronautics and Space Administration, for their assistance in formulating these generic systems engineering functions. b The IDD functions were formulated by the NWSM Committee for relevance to the multibuild development and deployment process being used for AWIPS. throughout the life cycle of the system, from initial specification through sustaining engineering. Because questions about the details of implementation and the allocation of resources and margins will continue to arise, initial systems engineering decisions must be revisited, refined, and expanded as experience is gained, uncertainties are resolved, and changes in design or initial requirements come up for consideration. This is especially important for an IDD process. In Table 3-1, principal systems engineering functions are listed (on the left) as they are commonly expressed. The NWSM Committee characterized these functions (on the right) as they apply to the “interbuild” phase of an IDD—the current state of AWIPS. The committee wishes to focus on the continuing need for systems engineering as characterized by the functions in the right-hand column. SYSTEMS ENGINEERING FOR AWIPS AWIPS is a complex system, and the AWIPS systems engineer must contend with many distributed elements, multiple sites, and intricate interconnections through multiple networks: the SBN (satellite broadcast network), the global network, local connections to radars and other data sources (for observations), links to the NOAA National Centers for Environmental Prediction (for forecast model data), and the National Environmental Satellite, Data, and Information Service (for weather satellite data). In the development of a complex system, systems engineering is required at several levels. Engineering functions for each subsystem are equivalent to the functions performed for the entire system. The project systems engineer, who reports to the highest level of project management, typically heads a team that includes the subsystems engineers. The team helps the project systems engineer
OCR for page 22
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build make design trade-offs, design the development and testing process, and track progress. A major concern of the NWSM Committee is that the hierarchy of system and subsystem engineering responsibilities for AWIPS has not been established with sufficient rigor to ensure that the functions listed in Table 3-1 can be performed when needed. CURRENT SYSTEMS ENGINEERING STRUCTURE Figure 3-1 summarizes the current systems engineering structure for AWIPS, along with organizational reporting and relevant functional flows. 1 The key organizational components of systems engineering for AWIPS are the systems engineering team, which is shown in the center of the diagram, and the systems engineering component of the contractor hierarchy, which is indicated by the flow of information from subcontractors through the prime contractor's director of engineering to the AWIPS acquisition manager. This complex organization may limit the NWS 's ability to perform essential systems engineering functions. The committee's concern is not about which structure is developed but about how well the functions in Table 3-1 can be implemented now and throughout the IDD. Systems Engineering Team as a Committee The members of the systems engineering team represent several governmental organizations participating in the AWIPS IDD. This team is “under the direction of” the AWIPS technical manager, although its recommendations also go to the AWIPS program manager and acquisition manager. It is not clear to the NWSM Committee who, if anyone, has the responsibility and authority typically vested in the project systems engineer. It is also unclear if systems engineering responsibility and authority devolves to individuals on the team. The present structure is consistent with a committee approach, in which no one on the team is specifically accountable because no one has specific responsibility or authority to carry out responsibilities. Good systems engineering requires the exercise of authority and the acceptance of responsibility within a technical hierarchy. The project systems engineer must report to the highest level of project management and must have clear, simple lines of technical authority and reporting that encompass the entire project. Systems engineering must be the full-time concern of this individual. The administrative hierarchy of the systems engineering team must be transparent in the sense that team members represent subsystems rather than organizational units. 1 Formation of the AWIPS systems engineering team was announced in a memorandum from the assistant administrator of weather services, NOAA, on May 9, 1997 (NWS, 1997). The operational role of the team was described as follows: “This team will function under the direction of the AWIPS Technical Manager and will be responsible for the resolution of system-engineering issues.”
OCR for page 23
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build FIGURE 3-1 AWIPS developers and systems engineering organization. aISL/ADDL integrates and tests software from NWS and FSL developers and provides software to AWIPS contractor for integration and testing. bTDL, OH, and FSL give developed application software to OSD (ISL/ADDL) for integration and testing. cISL/ADDL and TDL formally report to OSD. OH and FSL report to NWS and NOAA offices at a level higher than those shown here. dSystems Engineering Team is responsible for system design and evolution; and recommends baseline changes to accommodate new functionality and performance enhancements. AWIPS contractor support is provided as required. eAWIPS contractor reports to AAO and is responsible for design of network segment, operation, and management of NCF, integration and test of government furnished software, software builds and upgrades, field installation and testing of delivered systems, and maintenance and logistics services to field systems. fAAO manages AWIPS contract and acquisition budget and acts as government testing and quality assurance (QA) team. gPrime contractor formally reports to acquisition manager but takes technical direction from AWIPS technical manager. hSubcontractors provide hardware; design, install, and maintain hardware systems; and provide other technical support to prime contractor. iNCEP is the National Centers for Environmental Prediction (part of NOAA). Engineering Responsibility for Software Subsystems Systems engineering is necessary at all levels of a complex information systems development project. In modern practice, a subsystems engineer is responsible for all aspects of that subsystem, and the project systems engineer is responsibile for the entire system. He or she ensures that systems engineering functions are conveyed from “top to bottom” of the entire system. For AWIPS
OCR for page 24
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build Build 3, the prime contractor appears to retain the responsibility for integrating the hardware and software subsystems into the total AWIPS system and for integrating the software subsystem in its entirety. The NWS Office of Systems Development (OSD) is responsible for integrating all government-developed software, as well as for developing some code to integrate FOA into the AWIPS networking and communications “infrastructure” software. The hydrology and hydrometeorology applications being developed by the Office of Hydrology can be treated as a subsystem, but it may not be feasible—or effective—to define other subsystems to coincide with organizational units. Subsystem engineering responsibility and authority should coincide with functional subsystems. Integrating the Prime Contractor's Systems Engineering Component NWSM Committee members and advisors who have observed AWIPS development operations over the past several years report that systems engineering capability on the part of the prime contractor has improved greatly. Signs of improvement include the addition of an experienced senior systems-level engineer who has demonstrated the ability to understand and resolve system design issues and implement configuration control. As Figure 3-1 shows, the prime contractor's systems engineer (director of engineering) is in a position to oversee systems engineering internal to the prime contractor and by the AWIPS subcontractors. The figure also illustrates the committee's concern about the integration of system-wide and team-wide systems engineering capability. The prime contractor's systems engineer does not have a formal position on the AWIPS system engineering team, which is supposed to recommend system decisions to high-level managers. Systems engineering issues will probably not fall neatly into issues that can be addressed by a government-developer-only team and issues that can be addressed by the contractors' systems engineers. Given the extent of integration of government-developed software and contractor-developed software and hardware subsystems, it seems appropriate and efficient that the contractor's systems engineer be included on the AWIPS systems engineering team. Conclusions and Recommendations on the Current Structure Conclusion. As the NWSM Committee understands the current AWIPS organization, no single individual is assigned the responsibility and given the authority to act as a system-wide, comprehensive systems engineer. Recommendation. The NWS should establish the position of AWIPS project systems engineer with system-wide, comprehensive responsibility and delegated authority for performing the functions detailed in this report.
OCR for page 25
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build Conclusion. The AWIPS program does not have a hierarchical systems engineering team composed of subsystems engineers. Current members of the team appear to represent organizational entities. Recommendation. The NWS should formally establish an AWIPS systems engineering team led by the AWIPS project systems engineer and composed of systems engineers for each subsystem. Conclusion. The AWIPS contractor's systems engineer is not a member of the AWIPS systems engineering team. Recommendation. The AWIPS contractor's systems engineer should be formally and directly included on the AWIPS engineering team, and the contractor's systems engineering processes should be included in the overall AWIPS systems engineering functions. LOCALLY DEVELOPED CODE The AWIPS requirements were designed to include the capability of NWS field staff to write their own application software to run on AWIPS workstations. This capability continues the historical role of NWS staff as developers of (1) experimental approaches to various hydrologic and meteorological analytic and forecasting tasks, some of which may ultimately be adopted across the NWS, and (2) tools to meet specific needs of a local forecast area that may not have wide applicability. From the perspective of systems engineering, allowing for locally developed code and supporting local developers who create and maintain it raises issues that must be addressed on a continuing basis. First, local developers must have an environment in which they can efficiently create AWIPS-based applications without interfering with the commissioned system, either during the development and testing phase or while the application is being used. To support local developers, the applications programming interfaces (APIs) should provide access to commissioned AWIPS routines to avoid having to duplicate functionality and to minimize development and operations risk. The AWIPS design approach to providing both the APIs and the directions for using them is called “rules and tools.” Although the general approach as described to AWIPS panel members seems reasonable, the design seems only to have been developed to a relatively early, conceptual level. The NWSM Committee is concerned that the APIs are not adequate to protect commissioned AWIPS functions from interference by locally developed code attached to the system. Second, resources (for example, storage space and processing cycles) must be allocated for locally developed code. Allocation of resources, definitions of interface requirements and standards, and the other systems engineering activities related to optimizing the functioning of the entire system must include
OCR for page 26
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build margins that will support locally developed code. The NWSM Committee has not seen evidence that such margins are available and appropriately managed. Third, locally developed code presents special requirements for configuration management. Each site that supports local development must have a development environment separate from the AWIPS operating environment, just as the AWIPS software developers shown in Figure 3-1 must work in a development environment distinct from the operational AWIPS accessed by users. Operational libraries of local code must be kept separate from developmental versions. Perhaps the most difficult requirement is to establish an effective system for preserving libraries of locally developed code when the common, system-wide AWIPS is upgraded and for ensuring adequate regression testing of the operational local code with the upgraded system. In discussions between NWSM Committee members and staff and AWIPS managers, these three issues were acknowledged as important for providing adequate local development capability in a mature AWIPS. These and other issues must be addressed as part of AWIPS systems engineering. Conclusion. The safe use of locally developed code requires that (1) commissioned functions not be adversely affected, (2) sufficient performance, storage, and other resource margins be available to accommodate both local code and commissioned code, and (3) the system configuration, including all locally developed code, be formally maintained through an appropriate change management process. Recommendation. The AWIPS systems engineer should ensure that processes are in place to (1) accommodate locally developed code, including programming support, testing support, and regression testing to ensure that commissioned functions continue to perform as expected; (2) provide sufficient resource margins to accommodate a reasonably expansive scenario of local code development; and (3) manage the system configuration, including locally developed code. TRANSFERRING OWNERSHIP OF SOFTWARE DEVELOPED BY THE FORECAST SYSTEMS LABORATORY The integration of FOA into AWIPS Build 3 presents AWIPS program managers with a problem that will recur throughout IDD and, to a lesser extent perhaps, throughout the operating life of AWIPS. Although the FSL developed the FOA software, the responsibility for developmental modifications after Build 3 is supposed to pass to OSD (NWS Office of Systems Development). This will free FSL to continue developing application modules to be incorporated in later AWIPS builds. Eventually, when a mature AWIPS is operating as an NWS commissioned system (some time after deployment of Build 4), continuing maintenance would pass to the Office of Systems Operation. These transfers of ownership would free
OCR for page 27
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build FSL programming staff from their responsibilities for continuing upgrades and maintenance, as well as for troubleshooting, for the components of AWIPS software derived from FOA and other FSL-developed modules. FSL could then return to its primary mission as a laboratory for developing new forecasting techniques and tools, and AWIPS would be brought within the standard organizational structure for maintaining commissioned NWS systems. A proven mechanism for transferring ownership of FSL-developed code to OSD (or to another AWIPS player) must be established as soon as possible for the FOA code and for additional AWIPS software modules. Transferring ownership of complex software requires that “deep” or “tacit” knowledge of the code be transferred to the new owners. In the AWIPS program, this knowledge is transferred through face-to-face meetings, teleconferences, and written documentation for programmers. Although progress has been made in the transfer of knowledge, the NWSM Committee is still concerned that the transfer does not appear to be as far along as it should be for FSL to be freed of troubleshooting and continuing developmental responsibilities for the FOA-derived code after Build 3 is deployed. The committee suggests that one or both of the following courses of action would avoid systemic problems as AWIPS matures: Designate a development-to-operations transfer team, including a team leader (who has appropriate authority and accountability for meeting objectives) and other personnel (those who know, those who need to know, and those who can help in the transfer), and establish a schedule for effecting the transfer. Create a list of the documentation required, including the person(s) responsible for creating each document and those responsible for understanding and updating the documents as the software is modified during AWIPS IDD and beyond. Accept as a reality that FSL will always have to play a supporting role for code it develops. Identify one or more persons in FSL who will be long-term “consultants” on FOA and other FSL-originated code that is incorporated in AWIPS. Although the issue of transferring ownership is at present most visible with respect to FSL and the FOA code, it also applies to other developers of AWIPS software modules, including developers of local applications. AWIPS is expected to continue to evolve even beyond the series of builds now planned. Unless the systems engineering component of the AWIPS team can implement practices for capturing the developers' deep knowledge of their codes during development, increasing amounts of AWIPS software will, in time, end up as “orphan code” that cannot be improved or updated. The only option for improvement will then be to abandon the orphan code and redevelop the functionality—that is, write, test, and integrate new code for the system that does what the old code did as well as meets new requirements.
OCR for page 28
An Assessment of the Advanced Weather Interactive Processing System: Operational Test and Evaluation of the First System Build Conclusion. Although code responsibility is scheduled to be transferred from development organizations to OSD or the Office of Systems Operations, the NWSM Committee believes that an effective transfer process must be defined, established, and managed. Recommendation. Under the leadership of the AWIPS project systems engineer, AWIPS management should establish a process for transferring responsibility for, and the detailed knowledge of, software elements from the developing organization to the deploying organization and subsequently to the operating organization. If the complexity of the system, its components, or its interfaces precludes a complete transfer, the AWIPS project systems engineer and AWIPS managers should, through formal agreements, ensure that support will continue to be available.
Representative terms from entire chapter: