APPENDIX D
OVERVIEW OF ASET IV&V METHODOLOGY1
INTRODUCTION
This paper presents a general description of the technical analysis process used by Intermetrics in performing independent verification and validation (IV&V) of Shuttle flight software under the NSTS Avionics System Engineering Task (ASET) contract. Attachments provide further details on key elements of this methodology.
BACKGROUND
The Intermetrics ASET IV&V effort has, as its principal objective, the identification of potential safety-of-flight issues from within the ongoing flow of Shuttle flight-software changes. Intermetrics is charged with applying a multi-disciplinary, systems perspective to find safety problems that might otherwise go unrecognized. This perspective complements the expertise of the various Shuttle engineering subgroups which concentrate on their particular subsystems or engineering disciplines.
The primary focus of ASET IV&V is on two Shuttle problem reporting and change instruments--Space Shuttle Orbiter Avionics Software Discrepancy Reports (DRs) and Shuttle Software Change Requests (CRs). While these instruments are directed at software, the IV&V analysis of them takes into account the software 's effects on, and interrelationships with, other elements of the avionics system with which the software interacts. This includes the on-board guidance, navigation, and control (GN&C) systems in general, as well as with crew and ground procedures. The principal value added by the ASET IV&V effort is independent technical findings deriving from in-depth understanding of the nature and ramifications of these problems and changes.
The principal technical interface of ASET IV&V is with the Shuttle Avionics Software Control Board (SASCB), which reviews and approves or disapproves all flight-software DRs and CRs. There are typically numerous DRs and CRs considered for each new software build, or Operational Increment (OI), for multiple shuttle flights, and a lesser number that apply to individual flights. The ASET IV&V provides written briefings to the SASCB in the form of Software IV&V Reports (SIRs), and the IV&V personnel routinely attend Board meetings to provide supporting information. These briefings describe the problem or proposed change from a systems standpoint, and present a risk assessment to aid the Board in making its approval decision.
1 |
Briefing document given to the Committee by Intermetrics, Inc. A few format changes have been made. Attachments are not included in this Appendix. |
The ASET IV&V analysts also routinely interact with the general Shuttle flight software and engineering communities. This includes participating in technical reviews and special task force groups working software/avionics problems. In some cases these groups address issues raised by Intermetrics. When warranted, the ASET IV&V analysts will write DRs on safety issues they have found. For changes approved by the SASCB that carry significant risk, followup analyses are performed to evaluate the correctness of the implementation and the adequacy of testing. Updated SIRs are submitted to document these follow-up analyses.
STANDARDIZED METHODOLOGY
Central to the process summarized above is a standardized approach to safety analysis adopted by the ASET IV&V organization. This approach has been devised and refined over the four-year duration of the ASET contract. The framework for the standardized analysis is the Analysis Checklist, Attachment 1.2 The checklist, in turn, contains a key element--Risk Assessment--that is defined in attachment 2. Both are described in the context of a multi-level IV&V concept.
LEVELS OF IV&V ANALYSIS
The ASET IV&V process entails three levels of analysis that correspond to the scope parameters described earlier in this chapter--limited, focused, and comprehensive. These are cumulative in the order presented, that is, focused goes beyond limited, and comprehensive goes beyond focused. For those CRs and DRs that are within scope (as defined below), a risk assessment is performed to determine which level of effort will be applied to a given CR or DR.
Due to the volume of changes and the resource limitations of the ASET contract, it is not possible to perform a complete, comprehensive IV&V on every Shuttle flight-software CR and DR. And, for the same reason, certain categories of problems or changes are ruled out of scope, such as those dealing exclusively with Vehicle Utility (VU) software, System Management/Payload (SM/PL) software, and software development tools. For those CRs and DRs that are within scope, such as the ascent GN&C, entry GN&C, on-orbit GN&C, sequencing, data processing system, and main engine controller, established criteria are applied in selecting the level of analysis to be performed. The criteria and the nature of the analysis are defined below for each of the three levels.
LIMITED ANALYSIS
A Limited analysis consists of determining answers to five basic questions. Listed under the section heading that appear on the SIR, these are as follows:
2 |
Attachments are not included in this Appendix. |
-
Problem/Change Description
What is the true nature of the problem being described by a DR or the change being proposed by a CR?
-
System Impact Analysis
What is the effect of the problem or the change on the overall Shuttle system?
-
Requirements Analysis
For a DR, what requirements/constraints are being violated? For a CR, are the prescribed requirements changes appropriate, correct, and complete?
-
Risk Assessment
For a CR, and for a DR resulting in changes, what are the implementation and safety risks associated with implementing the change versus not implementing it? For a DR for which no change is proposed, what is the risk of not finding the problem?
-
Disposition Analysis
Is the proposed disposition appropriate?
A Limited analysis is performed on every CR and DR that is within the ASET IV&V scope. From this it is determined if further analysis, in the form of a Focused or Comprehensive analysis, needs to be performed. Limited analysis is deemed sufficient if the CR or DR is low in risk, needs very little or no testing, and requires no code change. Examples of items that fall into this category are DRs that are closed with a program note or waiver. Such DRs may eventually require a Focused or Comprehensive analysis on a later OI when a software change is implemented.
A key portion of this first stage of analysis is risk assessment, as it both aids the SASCB in its approval decision and serves as a basis for determining what further analysis is required. Risk assessment consists of evaluating two types of risk--safety risk and implementation risk. Safety risk is the risk that the system will be less safe with a change than without. Implementation risk is the risk that the change will not be done correctly due to its complexity or other factors. Assessment categorizes both kinds of risk as to whether they are low, medium, or high.
FOCUSED ANALYSIS
A Focused analysis consists of Limited analysis plus determination of answers to the following additional questions:
-
Code Analysis
Have the code changes been correctly implemented, and do they create any new problems or risks?
-
Level 6/7 Test/Verification Analysis
Has development testing, Levels 6 and 7 (the first two levels of official qualification test) demonstrated the correctness and safety of the changes?
-
Documentation Assessment
Have all affected documents been changed and are those changes correct and complete as prescribed?
-
Safety Assessment
What safety-of-flight issues were revealed by the analysis and what other ones (already known to the program) exist?
A Focused analysis is performed on all CRs of moderate or greater risk and on DRs that require code changes. Focused analysis is generally deemed sufficient for changes that are adequately tested during software development (Levels 6 and 7), that have easily understood requirements, and that do not significantly impact Shuttle hardware of operational procedures.
During the Focused analysis the earlier decision on level of analysis is reevaluated. It may be decided at this point to change the ultimate analysis from Focused to Comprehensive or vice versa.
COMPREHENSIVE ANALYSIS
A Comprehensive analysis consists of Focused analysis plus answering the following additional questions:
-
Analysis of Other Systems Implementations
Have other changes besides code (hardware, I-loads, crew procedures, etc.) been correctly implemented, and do they create any new problems or risks?
-
Complete Test/Verification Analysis
Have official tests (Levels 6, 7, 8 and SAIL) collectively demonstrated the correctness and safety of the changes?
All high risk and selected medium risk changes receive a Comprehensive analysis. These generally include ones for which adequate analysis requires a look at system-level testing (Level 8 and SAIL), that have very complex requirements, or that have significant impact on other systems besides software or on operational procedures. Also included are any late-breaking changes to flight software introduced as patches after Final Load.
KEY FEATURES OF METHODOLOGY
The ASET IV&V methodology includes three major features to enhance efficiency and ensure the quality of the analysis product:
-
written analysis guidelines
-
computer-based analysis tools
-
peer reviews
The analysis guidelines are published in an Intermetrics internal document, the General Analysis Guide, which includes, among other things:
-
a checklist of analysis tasks;
-
guidelines for doing risk assessment;
-
instructions for preparing SIRs; and
-
lists and descriptions of analysis resources.
This guide promotes uniformity and thoroughness in the work of multiple analysts.
The computer-based analysis tools were developed specifically for the ASET IV&V effort and operate on copies of the actual Shuttle flight software downloaded from NASA to local computer systems. Included are parameter tracing, flowcharting, structured display and printout generation, and other tools. Also, a relational data base is used to track the status of all CRs and DRs subject to analysis.
The mechanism of peer review is used for all analyses, regardless of level to ensure the quality of the analysis product. When a SIR has been drafted, a group is assembled consisting of the designated analyst and any supporting analysts that contributed to the SIR, plus an appropriate number of other analysts (peers) from the ASET IV&V group. The draft SIR is evaluated in a supportive atmosphere, using the analysis checklist as a framework. If significant rework is needed a follow-up peer review may also be held. Such peer reviews are conducted when the first stage, Limited analysis is completed prior to SASCB review, and again when the Focused or Comprehensive level analysis has been performed. These peer reviews have been found to contribute significantly both to the motivation of the analyst and to the quality and uniformity of the analysis product.