APPENDIX D

Overview of
ASET IV&V Methodology

Briefing Document Given to the Committee
By Intermetrics, Inc.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes APPENDIX D Overview of ASET IV&V Methodology Briefing Document Given to the Committee By Intermetrics, Inc.

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes This page in the original is blank.

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes APPENDIX D OVERVIEW OF ASET IV&V METHODOLOGY1 INTRODUCTION This paper presents a general description of the technical analysis process used by Intermetrics in performing independent verification and validation (IV&V) of Shuttle flight software under the NSTS Avionics System Engineering Task (ASET) contract. Attachments provide further details on key elements of this methodology. BACKGROUND The Intermetrics ASET IV&V effort has, as its principal objective, the identification of potential safety-of-flight issues from within the ongoing flow of Shuttle flight-software changes. Intermetrics is charged with applying a multi-disciplinary, systems perspective to find safety problems that might otherwise go unrecognized. This perspective complements the expertise of the various Shuttle engineering subgroups which concentrate on their particular subsystems or engineering disciplines. The primary focus of ASET IV&V is on two Shuttle problem reporting and change instruments--Space Shuttle Orbiter Avionics Software Discrepancy Reports (DRs) and Shuttle Software Change Requests (CRs). While these instruments are directed at software, the IV&V analysis of them takes into account the software 's effects on, and interrelationships with, other elements of the avionics system with which the software interacts. This includes the on-board guidance, navigation, and control (GN&C) systems in general, as well as with crew and ground procedures. The principal value added by the ASET IV&V effort is independent technical findings deriving from in-depth understanding of the nature and ramifications of these problems and changes. The principal technical interface of ASET IV&V is with the Shuttle Avionics Software Control Board (SASCB), which reviews and approves or disapproves all flight-software DRs and CRs. There are typically numerous DRs and CRs considered for each new software build, or Operational Increment (OI), for multiple shuttle flights, and a lesser number that apply to individual flights. The ASET IV&V provides written briefings to the SASCB in the form of Software IV&V Reports (SIRs), and the IV&V personnel routinely attend Board meetings to provide supporting information. These briefings describe the problem or proposed change from a systems standpoint, and present a risk assessment to aid the Board in making its approval decision. 1   Briefing document given to the Committee by Intermetrics, Inc. A few format changes have been made. Attachments are not included in this Appendix.

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes The ASET IV&V analysts also routinely interact with the general Shuttle flight software and engineering communities. This includes participating in technical reviews and special task force groups working software/avionics problems. In some cases these groups address issues raised by Intermetrics. When warranted, the ASET IV&V analysts will write DRs on safety issues they have found. For changes approved by the SASCB that carry significant risk, followup analyses are performed to evaluate the correctness of the implementation and the adequacy of testing. Updated SIRs are submitted to document these follow-up analyses. STANDARDIZED METHODOLOGY Central to the process summarized above is a standardized approach to safety analysis adopted by the ASET IV&V organization. This approach has been devised and refined over the four-year duration of the ASET contract. The framework for the standardized analysis is the Analysis Checklist, Attachment 1.2 The checklist, in turn, contains a key element--Risk Assessment--that is defined in attachment 2. Both are described in the context of a multi-level IV&V concept. LEVELS OF IV&V ANALYSIS The ASET IV&V process entails three levels of analysis that correspond to the scope parameters described earlier in this chapter--limited, focused, and comprehensive. These are cumulative in the order presented, that is, focused goes beyond limited, and comprehensive goes beyond focused. For those CRs and DRs that are within scope (as defined below), a risk assessment is performed to determine which level of effort will be applied to a given CR or DR. Due to the volume of changes and the resource limitations of the ASET contract, it is not possible to perform a complete, comprehensive IV&V on every Shuttle flight-software CR and DR. And, for the same reason, certain categories of problems or changes are ruled out of scope, such as those dealing exclusively with Vehicle Utility (VU) software, System Management/Payload (SM/PL) software, and software development tools. For those CRs and DRs that are within scope, such as the ascent GN&C, entry GN&C, on-orbit GN&C, sequencing, data processing system, and main engine controller, established criteria are applied in selecting the level of analysis to be performed. The criteria and the nature of the analysis are defined below for each of the three levels. LIMITED ANALYSIS A Limited analysis consists of determining answers to five basic questions. Listed under the section heading that appear on the SIR, these are as follows: 2   Attachments are not included in this Appendix.

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes Problem/Change Description What is the true nature of the problem being described by a DR or the change being proposed by a CR? System Impact Analysis What is the effect of the problem or the change on the overall Shuttle system? Requirements Analysis For a DR, what requirements/constraints are being violated? For a CR, are the prescribed requirements changes appropriate, correct, and complete? Risk Assessment For a CR, and for a DR resulting in changes, what are the implementation and safety risks associated with implementing the change versus not implementing it? For a DR for which no change is proposed, what is the risk of not finding the problem? Disposition Analysis Is the proposed disposition appropriate? A Limited analysis is performed on every CR and DR that is within the ASET IV&V scope. From this it is determined if further analysis, in the form of a Focused or Comprehensive analysis, needs to be performed. Limited analysis is deemed sufficient if the CR or DR is low in risk, needs very little or no testing, and requires no code change. Examples of items that fall into this category are DRs that are closed with a program note or waiver. Such DRs may eventually require a Focused or Comprehensive analysis on a later OI when a software change is implemented. A key portion of this first stage of analysis is risk assessment, as it both aids the SASCB in its approval decision and serves as a basis for determining what further analysis is required. Risk assessment consists of evaluating two types of risk--safety risk and implementation risk. Safety risk is the risk that the system will be less safe with a change than without. Implementation risk is the risk that the change will not be done correctly due to its complexity or other factors. Assessment categorizes both kinds of risk as to whether they are low, medium, or high. FOCUSED ANALYSIS A Focused analysis consists of Limited analysis plus determination of answers to the following additional questions:

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes Code Analysis Have the code changes been correctly implemented, and do they create any new problems or risks? Level 6/7 Test/Verification Analysis Has development testing, Levels 6 and 7 (the first two levels of official qualification test) demonstrated the correctness and safety of the changes? Documentation Assessment Have all affected documents been changed and are those changes correct and complete as prescribed? Safety Assessment What safety-of-flight issues were revealed by the analysis and what other ones (already known to the program) exist? A Focused analysis is performed on all CRs of moderate or greater risk and on DRs that require code changes. Focused analysis is generally deemed sufficient for changes that are adequately tested during software development (Levels 6 and 7), that have easily understood requirements, and that do not significantly impact Shuttle hardware of operational procedures. During the Focused analysis the earlier decision on level of analysis is reevaluated. It may be decided at this point to change the ultimate analysis from Focused to Comprehensive or vice versa. COMPREHENSIVE ANALYSIS A Comprehensive analysis consists of Focused analysis plus answering the following additional questions: Analysis of Other Systems Implementations Have other changes besides code (hardware, I-loads, crew procedures, etc.) been correctly implemented, and do they create any new problems or risks? Complete Test/Verification Analysis Have official tests (Levels 6, 7, 8 and SAIL) collectively demonstrated the correctness and safety of the changes?

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes All high risk and selected medium risk changes receive a Comprehensive analysis. These generally include ones for which adequate analysis requires a look at system-level testing (Level 8 and SAIL), that have very complex requirements, or that have significant impact on other systems besides software or on operational procedures. Also included are any late-breaking changes to flight software introduced as patches after Final Load. KEY FEATURES OF METHODOLOGY The ASET IV&V methodology includes three major features to enhance efficiency and ensure the quality of the analysis product: written analysis guidelines computer-based analysis tools peer reviews The analysis guidelines are published in an Intermetrics internal document, the General Analysis Guide, which includes, among other things: a checklist of analysis tasks; guidelines for doing risk assessment; instructions for preparing SIRs; and lists and descriptions of analysis resources. This guide promotes uniformity and thoroughness in the work of multiple analysts. The computer-based analysis tools were developed specifically for the ASET IV&V effort and operate on copies of the actual Shuttle flight software downloaded from NASA to local computer systems. Included are parameter tracing, flowcharting, structured display and printout generation, and other tools. Also, a relational data base is used to track the status of all CRs and DRs subject to analysis. The mechanism of peer review is used for all analyses, regardless of level to ensure the quality of the analysis product. When a SIR has been drafted, a group is assembled consisting of the designated analyst and any supporting analysts that contributed to the SIR, plus an appropriate number of other analysts (peers) from the ASET IV&V group. The draft SIR is evaluated in a supportive atmosphere, using the analysis checklist as a framework. If significant rework is needed a follow-up peer review may also be held. Such peer reviews are conducted when the first stage, Limited analysis is completed prior to SASCB review, and again when the Focused or Comprehensive level analysis has been performed. These peer reviews have been found to contribute significantly both to the motivation of the analyst and to the quality and uniformity of the analysis product.

OCR for page 131
An Assessment of Space Shuttle Flight Software Development Processes This page in the original is blank.