**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

**Suggested Citation:**"APPENDIX E: An Improved Critical Item Risk Assessment Procedure for the National Space Transportation System." National Research Council. 1988.

*Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management*. Washington, DC: The National Academies Press. doi: 10.17226/10616.

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

APPENDIX E AN IMPROVED CRITICAL ITEM RISK ASSESSMENT PROCEDURE FOR THE NATIONAL SPACE TRANSPORTATION SYSTEM (With an Example of Application to the 51-L Field Joints) 1. INTRODUCTION ] On May 28, 1987, a NASA representative made a presentation to the Committee on Shuttle Criti- cality Review ant! Hazard Analysis Auclit entitled, `'Critical Items List (CIL) Prioritization." The methoc! discusser! was subsequently issued in moclifiec! form as NSTS Instruction 2249 1, Reference F34. This Instruction for the preparation of Critical Item Risk Assessments (CTRA) provides a methoc! for prior- itizing the failure modes in the CIL. It contains many excellent ideas and is a significant step forward. However, the Committee has some con- cerns and some related suggestions on how to simplify ant! clarify the method. This Appendix also contains in Section 5 an example of the application of trend analysis and Probabilistic Risk Assessment (PRA) to the pre- Challenger O-rings. This application, include(l here only as an example of some applicable analysis techniques, makes heavy use of moclern statistical science ant] Bayesian ideas. 2. CONCERNS WITH THE CURRENT METHOD The Committee's concerns with the CIRA method, as currently formulated, can be summarized as follows: I. In Table ~ of Reference t31 (shown here in Attachment I) the column labeled "SEVER- ITY" DEFINITIONS really contains worst- case ciamage states. 2. In Table I, the columns labeler! SUCCESS PATHS and STATUS CODE FOR REDUN- DANCY/BACKUP are really descriptions of system or subsystem architectures. They affect risk by affecting the probabilities in the last Failure Mode 1 2 Severity Definition (A) Loss of Life (A) Loss of Life Success Paths o o Redundancy/ Backup (a) None (a) None 125 two columns. However, the relevant informa- tion is in the probabilities themselves not in the architecture. Any guidelines written on how to assess the probabilities, either empir- ically or subjectively, should contain much discussion on how success paths, redundancy structure, ant] periodic checking strategy af- fects the probabilities in columns 4 and 5. 3. The probabilities in the last two columns of Table ~ are qualitative ant] open to interpre- tation as to what the terms "Very Likely," "Likely," "Unlikely," ant! "Very Unlikely," mean. The two columns, which have the same qualitative scale, appear to have different quantitative scales associates] with them. In column 4, '~Very Unlikely" appears to mean something like c ~ 0 - 6 and "Very I=ikely" means something like 10- i. In column 5, the scale depends on whether or not there is reclundancy . If there is no reclundancy, then "Very Unlikely" means something like 10-2 and "Very Likely" means something like greater than .95. But if there is redundancy, then "Very Unlikely" may mean ]0-6. With the qualitative definitions of probability, it is quite possible that two engineers working on two failure modes with the same severities and probabilities would assign them to dif- ferent probability categories and therefore produce inconsistent priorities. It is very im- portant that the probabilities have opera- tional clefinitions. Terms like "Unlikely" are not operational definitions. 4. There is no way to produce a unique priority. Suppose there are two failure modes, and Table 1 is filled out as follows: Design Conf idence Likelihood of Worst Case (iv) Unlikely (ii) Likely

Which one should have the highest priority? Suppose that the last two columr~s were replaced by the following structure: Failure Mode 2 Probability of Failure Likely= 01 Unlikely = 00001 Probability of Worst Case Given Failure . Unlikely= 01 ! ikely = 5 Probability of Worst Case 0001 000005 Now it is clear that failure mode ~ presents a higher risk. 3. PROPOSED IMPROVEMENTS As an improvement to Reference f3I, the Com- mittee proposes the procedure described in Tahie E-1 below: All failure modes with the same Worst Damage State Given Lack of Redundancy or Reclunclancy Failure wouic] be ranked by column Z. The probabilities shown in Tahie E-l are for illustration only anc! do not reflect any specific example. In actual application, it wouIc! he highly desirable for the analyst to include confidence limits (or the equivalent) for each of the probabilities listed in the tahies produced through the CTRA. The Committee recommends strongly that such probabilities be documented by a rationale. Many of the facts mentioned in the current CIL "Rationale for Retention" would be cites! in the probability rationale but in the quantitative manner illus- trated by the example in Section 5. In addition, facts that imply higher probabilities wouIc! also be analyzecl. For example, the long-run frequency of catastrophic failure for solid rocket motors of a TABLE E-1 Improved Risk Assessment Procedure Failure Mode . 2 3 u Criticality OR . OR V Probability of Primary Failure During Mission - 00 00 w Probability of Redundancy Failure Given Primary Failure 001 999 1- 0505 01 1 126 mature design is I/50; ant! therefore I/25 for two solic! rocket motors. A dis-aggregation of this frequency by ~ aiTure mode wouic! be a useful baseline for an analysis. How are our clesign and failure modes different from history? For example, the field joint is similar to Titan ITI, but also different. The redundant O-ring points to a smaller probability, but the insulation geometry points to a higher probability. In Table E-l, failure mode 3 has the most risk, even though it is only a Criticality JR item. For this case, the computation of column W uses the following estimates: (i) There is one success path remaining after the primary failure. (ii) The availability of the backup is not readily detectable ant] is checked every thirc! flight; and the estimated availability is .99. (iii) The probability of a secondary failure is .05. The formula for column W is W= PrlBackup Available) x PrlSeconclary Failure) + PrlBackup not Available) = (.99) (.05) + (.01) = .0595 For failure mode I, there is no backup; but, it is a relatively rare (probability = .001~ failure mode and infrequently (probability = .01) causes the worst damage state. Failure mode 2 is much less risky. The compu- tation of column W uses the following estimates: (i) There is one success path remaining after the first failure. x Worst Damage State Given Lack of Redundancy or Redundancy Failure (A)Loss of Life and/or Vehicle (A) Loss of Life n~l/~r V~hi~.l~ (A) Loss of Life and/or Vehicle 1 y Probability of Worst Damage State, Given Lack of Redundancy or Redundancy Failure 01 .1 Z = (V)(W)(Y) .- Probability of Worst Damage State Event . 00001 0000001 999 000595

(ii) The backup is readily detectable and fixed when failed and the availability of the backup is .999. (iii) Given the backup, the probability of sec- ondary failure is .001the same as the primary. Use of equation (~) in this case yields W= (.9991~.001) + (.001) = .001999 4. RELATIONSHIP BETWEEN IMPROVED PROCEDURE AND TABLE E-1 There is a strong relationship between the im- provements described in Section 3 and NASA's Table ~ (Attachment 1 here). From the "SEVER- ITY" DEFINITIONS in column ~ of Table I, we can deduce the following Worst Damage States: A. Loss of Life and/or Vehicle B. Mission is Aborted C. Degraded Operational Capability or Early Mission Termination or Damage to a Vehicle System D. Loss of Some Operational Capability of Ve- hicle, but Full Mission Duration. E. No Operational Effect The probability scales couic! be set up as categories with the definitions given in Table E-2. The Committee urges the use of quantitative definitions of probability. Even though for some failure modes the probabilities will be assessed subjectively, it is very important that the analyst have art operational clefinition. To reiterate, terms like "Unlikely" are not operational definitions. In TABLE E-2 Probability Scales For Improved Risk Assessment Procedure aciclition, use of a quantitative probability scale will augment the pure engineering judgment ap- proach. The factors in Reference F3], Section 3.4, are very relevant to assessing the Probability of Primary Failure During Mission in Table E-1. Other factors include: Product design certification test results Manufacturing process qualification test re- sults · Engineering analytical models · Related industry data · Etc. The number of SUCCESS PATHS ant] the REDUNDANCY/BACKUP scenarios given in NASA's Table 1 (Attachment 1 to this appendix) are very relevant to assessing the Probability of Redundancy Failure Given Primary Failure in Table E-1. The factors relevant to assessing the Probability of Worst Damage State Event in Table E-1 are very similar to those listed in Reference f3], Section 3.5. As part of the exercise of assessing this probability, one could list all the events subsequent to redun- dancy failure that do not lead to the worst damage state. 5. APPI ICATION TO THE D-RINGS Only as an example to illustrate the foregoing proposal, consicler the field joint O-rings prior to the Challenger flight 51-L at a joint temperature of 31°F, which was predicted for the Challenger flight. It is based only on a limited knowledge of the subject derived from References A] and t2], Center Point of Ranges of Probability Values 1 1 Description Very Likely Likely Possible Unlikely Very Unlikely Probability of Primary Failure During Mission 10 1 10 2 10 3 10 5 10 7 Probability of Redundancy Failure Given Primary Failure 10 1 10 2 10 3 10 5 10 7 127 Probability of Worst Damage State Given Lack of Redundancy or Redundancy Failure 0 5 0-~ -2 0-3

and thus must be viewed ONLY AS AN ILLUS- TRATION OF A PROCESS. To keep things simple only one failure scenario is considerecI. In the language of Table E-1 we have: TABLE E-3 Application of Table E-1 to the SAM Field Joint Language of Table E-1 Primary failure during mission Redundancy failure given primary failure Worst damage state Application to Field Joint Erosion and blowhy of the primary O-ring Failure of the secondary O-ring given erosion and blowhy of the primary O-ring Loss of life and vehicle The reason for considering this scenario is that ciata are readily available. Also in Reference FI] p. 135 it is stated that bypass erosion or blowby was considered much more serious than just im- . . plngement erosion. The data set used in this analysis (see Attachment 2) is taken from pages 129-131 of Reference FI]. The subset of these data user! here involves only the actual flights and only the field ant! nozzle joints. A useful organization of this subset is shown in Attachment 3. In the columns labeled erosion blowby and erosion or blowby the blanks mean that the event (lid not occur. In the column where labelect blowhy given erosion the blank means there was no erosion and the zero means that there was erosion but no blowby. Most of the data are for the primary O-rings; but the data with an asterisk are for the secondary O-rings. 5.1 Primary Failure For primary O-ring failures we consider the scenario of erosion ant] blowby. The primary failure probability is: PrlPrimary Failure) = Pr{Primary Erosion) Pr mary Primary x Pr{Blowhy Erosions. (2) The vertical bar in the probability expression (2) reacls conditional on. So for example Pr{Blowhy ~ Erosion) would read probability of the event Blowhy conditional on the event Erosion occurring. For two events A ant! B a funciamental law of prob- ability is PrlA and B) = PrlA) x Pr{B ~ A) . 5.1.1 Primary Erosion A plot of the incidents of field joint primary O- rings with erosion is shown in Attachment 4. For example flight 51-C in January 1985 had two field] joints with primary O-ring erosion; this mis- sion experienced a joint temperature of 53° F and a leak check pressure of 200 psi. The fitter! curves are derived from a statistical model which allows for possible joint temperature ant! leak check pres- sure effects. Flight 5 l-C experienced both erosion and blowhy of the field joint. At a subsequent Flight Readiness Review where 51-C was discusser! there was a conclucling statement I=o`v temperature enhancer! probability of blow-by (Reference tI] p. 1471. On page H-73 of Reference f21 it is statec! that Frequency of O-ring damage has increased since the incorporation of . . . higher stabilization pres- sures in leak test procedures ... . So it is of interest to statistically mocle! the effect of temperature ant! leak check pressure on O-ring anomalies. pit, s') = Probability of erosion per field joint primary O-ring, t= Joint temperature s = Leak check pressure. The assumptions for this statistical model are: I. The model for pit, s) is: {t /J(~'s)} This is caller! a Logistic Regression moclel. The variables a,F,~ are unknown parameters to be estimated from the data. Different values of these parameters represent different relationships be- tween erosion probability and (temperature, pressure). For example, if ~ < 0, then probability (lecreases with temperature; but if ,(3 ~ O. then probability increases with temperature. We will let the data determine which of these is most likely. 2. Given pit, s'), the field joints are statistically independent. 128

Let aft, s) = Number of field joint primary O-rings with erosion for a launch with joint temperature t ant] leak check pressure s. Uncler these assumptions, the probability distri- bution of aft, s) given pit, s) is binomial with parameters n = 6 (i.e., 6 fielc! joints) and p = pit, s). So for k = 0, I, . . ., or 6, Pr {X(t,s) = k j ptt,sy} = {6) Spit s)]k;] pit S)46-k Let the subscript i represent the ith launch in Attachment 3. So i = I, 2, . . ., 23. Let xi = Number of field joint primary O-rings with erosion ti= loins temperature si = Leak check pressure Pi = P(ti, si) Also let t X = (x I, X,, . . ., X, 3) = (t,, t,, . . ., ten) S = (S 1, S., . . ., S. 3). The likelihood function, A, given the data x, is defined as the probability of observing x conclitional on t, s, and (a,Q,~). The variables t and s are regarded as known variables (in standard] regression analysis they are callec! inclepenclent variables); ant! (`x,Q,~) are the unknown parameters. The likeli- hood function is regarcled as a function of (`x,h,~) and is Li a ~ A) = ~ ( 6 ) pxi ( } _ p )6 -xi Recall that Pi is a function of (`x,,B,~y). The maximum likelihood estimates of the (a, [3,~y) are those values that maximize the likelihood! function. In effect, they are the values of (`x,h,~) that make the observer] value of x the most probable under our model. There is a close relationship between maximum likelihood] estimation ant! least squares. The least squares estimates of (a,,(3,A) are those values that . . . mlnlmlze where 6pi is the expected value of xi under our moclel. If the xi's had a Gaussian (normal) distri- bution with common variance, then the maximum likelihood estimates and the least squares estimates would be the same. This is because the Gaussian probability density would then be monotonically related to the sum of squares above. However, the probability densities of the xi's in our problem are binomial ant] not Gaussian. Ant! it is a well established fact in statistical science that maximum likelihooc! estimation is usually more efficient (closer to the truth) than least squares; so we use maximum likelihoocI. The results of a maximum likelihood analysis of these ciata under the above mociel yields the values in Table E-4. TABLE E-4 Maximum Likelihood Analysis of the SRM Field Joint Primary O-Rino Erosion Data Parameter cY Maximum Likelihood Estimate 78 -.17 0024 _- 90% Confidence Interval [ -.1, 15 7] [-.28 - 06] [ - .01 2, 01 6] The 90% Confidence Interval reveals the fact that from our data we cannot learn the "true" value of ((x,,B,A) with great precision. For example, a Bayes interpretation of the interval [-.28, -.06] for the temperature effect, A, is that given our data, there is a .9 probability that the "true" value of lies in the interval L-.28, -.064. Note that this interval does not include the value ~ = 0 (i.e., no effect). This means that the temperature effect is "statistically significant;" or that there is only a very small probability that the true value of ,l3 is greater than or equal to zero. Also note that there is no statistically significant pressure effect on field joint erosion. That is because most of the variation is explained by temperature variation. This is curious, because in Reference L1], blow-holes caused by high pressure were cited as a cause of erosion. Plugging the maximum likelihood estimates into equation (3) yields A (Xi- 6pi)2, i= 1 In[1 p(ti200) ] = 7.8 - ( 17)t + ( 0024)(200) = 8.3 - (.17)t 129

~1 ~ 1 nls 1rnplles elf ~ (. 17)tl p(t,200) = 1 + elf ~-('7)tl (4) The curve for 200 psi (plotted in Attachments 4 and S) is (6)p(t,200), because there are 6 field 1olnts. The predicted probability per joint of primary O-ring erosion at 31° ~ joint temperature and 200 psi leak check pressure is p(31,200~=.95 [probability of - Primary Erosion The 90 percent confidence interval for the "prob- ability of primary O-ring erosion" is shown in Attachment S and is t.S, 1.04. This shows that the extrapolation to 3-~° E; introduces considerable uncertainty in the estimate. The propagation of this uncertainty to the final result will be discussed in Section 5.5. S.1.7 Pri~7zar~' Blc''~by Gin Priorly Erosions The frequencies per primary O-ring of blowby given erosion were extracted from Attachment 3 and are given in Table E-S. An analysis of the blowby given erosion data shows no statistically significant effects of joint type, joint temperature, or leak check pressure. So we use the estimate p ~ Primary Blowby ~ Primary Erosion r ~ for Field Joint I for Field Joint J t primary Blowby Primary Erosion] = Pr] for Field or for Field or t Nozzle Joint Nozzle loins J = .292 TABLE E-5 Frequency per Primary O-Ring of Blowhy Given Erosion 1 1-- Joint . Field Plugging (5) and (6) into (2) yields Pr{Primary Failure} = (.95) (.292) = .277 It is revealing to look at the frequency of primary O-ring blowby, given no erosion, in Table E-6. TABLE E-6 Frequency per Primary O-Ring of Blowhy Given No Erosion 1 1 Joint - Field Nozzle Frequency Joint per O-Ring 2 Field 7= 286 Nozzle ~ 7= 294 I, Field plus ! 7 - .292 I Nozzle 1 24 Frequency per O-Ring 2= 50 1= 20 5 Field plus 7 = 286 Comparison with Table E-S shows that there is a strong statistical dependence between primary O-ring erosion and blowby particularly for the field joint. For the field joint, blowby was rare (frequency = .015) when there was no erosion, but not rare (frequency = .286) when there was erosion. no PrlBlowby ~ Erosion) >> PrlBlowby ~ No Erosionl, which implies strong statistical dependence. If blowhy and erosion were statistically indepenclent, then these two conditional probabilities wouic! be the same. The strong statistical dependence shown above suggests that erosion might be a causal factor for blowhy. This idea is born out by field data ant! various experiments. Experiments (reference L21, p. H-82) showed that an O-ring will fad! to seal with an erosion depth of 0.15 inches. In flights 51-C (6) an`] 5~-B, there was both erosion and blowby of the field primary O-ring, and a heat effect or erosion of the secondary O-ring. In both cases, the erosion of the primary O-ring was among the worst ero- sions experienced (reference F2], p. H-7l, H-72) as measured by cross-sectioned depths of 0.038 and 0.171 inches, cross-sectionecl perimeters of 130° and 360°, and a top view of affected lengths of 58.75 anc! 12 inches. This implies that blowby can be caused by excessive erosion. So our model that the higher the probability of primary O-ring ero- sion, the higher the probability of primary O-ring blowby, is plausible. 130

5.2 Probability of Secondary Failure Next we consicler the Probability of Redundancy Failure Given Primary Failure in Table E-~. This would be failure of the secondary O-ring. Our n~ode! of secondary failure is secondary erosion and failure given primary erosion and blowhy. Therefore Pr ~ Secondary ~ Primary Erosion Failure I end Blowby J _ p ~ Secondary ~ Primary Erosion r ~ Erosion I and Blowhy x Pr{SeCondary | Secondaryl Failure ~ Erosion i. (7) A statistical analysis of secondary erosion given primary erosion and blowby shows no statistically significant effects of joint type joint temperature or leak check pressure. So we use the estimate from Table E-7 below: ~ . Primary Erosion and] Pr] Secondary Erosion Blowhy ~ 2 t for Field joint for Field Joint Secondary Erosion Primary Erosion and] = Pr] for Field or Blowhy for Field t Nozzle loins or Nozzle Joint J Joint Field (8) TABLE E-7 Frequency per SRM Joint of Secondary O-Ring Erosion Given Erosion and Blowhy of the Primary O-Ring in 23 Flights Prior to Challenger 51-L 1 Secondary Erosion Given Primary Erosion and Blowbv ..._ ~ Nozzle i Field plus 2 = 286 1 of Secondary O-Ring Time After ignition Ignition Transient: O to 170 ms 1 70 to 330 ms 330 to 600 ms The estimation of Pr | Secondary | Secondary | Steady State: Failure | Erosion J 60 ms to 2 min 131 in equation (7) presents some difficulties because there were no secondary failures before 51-~. So we shall express the solutions parametrically in terms of the parameter A4 = PrISecondary Failure~Secondary Erosion) (9) The state of knowledge curve (described in Appen- dix D) for A4 could be determined on the basis of engineering information. Examples of relevant en- gineering information which was available before 51-L are: loins rotation created doubt about the ability of the secondary O-ring to seal. In fact the O-ring failure mode was considered Critical- ity 1, not Criticality 1R. So, officially, the FMEA did not recognize the secondary 0- rings as providing redundancy. However, ac- cording to Reference f 1 l, p. 126, NASA management and Thiokof still considered the joint to be a redundant seal because there were flights where the primary O-ring failed and the secondary O-ring sealed in accord- ance with its design intent. In July 1985, a ThiokoT engineer, in light of the 51-B nozzle joint secondary O-ring ero- sion, expressed his concern that if the same scenario should occur in a field joint (and he believed it could), then it would be a "jump ball" as to the success or failure of the joint because the secondary O-ring could not re- spond to the clevis opening rate and might not be capable of pressurization (i.e., in the 51-E design, which has been changed in the redesigned joint). (See Reference F1], p. 139.) 3. The qualitative assessment (Reference t2], p. H-84, Chart 166) of the probability that the field joint secondary O-ring will fait given erosion penetration of the primary O-ring seal is listed in Table E-8. TABLE E-8 Qualitative Probability of SRM Secondarv O-Rino Failure Given Erosion Penetration Qualitative Probability of Secondary O-Ring Failure low medium high high

There were only two incidents of secondary O-ring erosion in a field] joint. So there was no solid statistical evidence that the secondary O-ring would work given primary O-ring failure; i.e. nothing like ~ 000 successes with- out a failure. Also as seen in Table E-8 the probability of secondary O-ring failure cle- F'ends on time after ignition. r. The night before the Challenger launch a chart provicled to NASA by a Thioko! engi- neer about the possible temperature effect on the O-rings (Reference tI] p. 89 Chart 2-2) included concerns that: (i) lower temperature of the O-rings would result in a change in their sealing timing function which would result in higher O-ring pressure actuation time; (ii) if the actuation time increases threshold of secondary seal pressurization capability is approached; (iii) if threshold is reached then secondary seal may not be capable of being pressurized. Plugging (8) and (9) into (7) yields Pr ( Secondary Failure J = (.286 ~ As Probability of ) ~ ~ 0) Secondary Failure 5.3 Probability of Worst Damage State Given Redundancy Failure If the Felt! joint seal were to fad! there is some possibility that the crew anal vehicle wouIcl survive. For example the seal might fad! right before the solid rocket motors completed their burn. How- ever the chances are very high that such a failure shouic! it occur would be earlier in the flight. This suggests a value approaching ~ for the probability of Toss of life and vehicle given total seal failure. Thus the closest probability value of ~ from Table E-2 column Probability of Worst Damage State, is selectee! in this example. 5.4 Probability of Worst Damage State Event Using the estimates derived above the value for column Z in Table E-l is Z= (.277~.286jA4 l'Probabilityper~oint: ~ of Worst Damage J A4 = = (.0792)A4 . 5.5 Probability of At Least One Field Joint Failure The estimated probability in Section 5.4 is for only one field joint. The estimated probability of held joint failure for the mission is Pry Mission Field ~ joint Failure J = ~ _ prtNO Field = ~t!(.0792) A446 (Probability of Failure) (12) It is clear from the statistical analyses that there is uncertainty in the estimates of the probabilities used. For example the 90 percent confidence in- tervals in Table E-4 show that the parameter estimates are uncertain. Also the .286 estimate in equation (8) was based on two failures out of seven and is therefore uncertain. The uncertainty associated with equation (12) is quantified in At- tachment 6. The two almost linear curves form a 90 percent confidence interval for the "probability of mission fielcl joint failure," conditional on the value of As. So if the value of A4 is .25, for example, then the conclitional 90 percent confidence interval is t0.010,.1184. A subject matter expert could analyze the rele- vant engineering information and assess a state of knowledge curve for 4. If this curve were centered on A4 = .25 with a considerable variance, then the unconditional 90 percent confidence interval for the "probability of mission field joint failure," would be much wider than the F.010, .118] interval cited above. The 90 percent confidence intervals in Attach- ment 6 were derived by a Bayesian analysis (see Appendix D for more discussion). For the ST-L environment (e.g., 31° F), we define the following Tong run "true" frequency probabilities: H 132 = Probability of mission field joint failure per mission; and for a given field joint, ~ = Probability of failure A I = Probability of primary O-ring erosion A, = Probability of primary O-ring blowby given primary O-ring erosion Probability of secondary O-ring erosion given primary O-ring erosion and blowhy Probability of secondary O-ring failure given secondary O-ring erosion.

Our mocie! is that ~ = ~ - ~ ~ - l)6 4 d)= 11 A. i = I Let /i = A,A,A~ then D= 1 - [1AA4]6 /1 ~ (14) . (15) (16) In the Bayesian analysis we assume that, condi- tional on our data, Al, A,, and As are statistically independent. This is reasonable because the Ai's are successive conditional frequencies. The state of knowledge curves for the inclividual Ai's were derived from Bayesian analyses assuming "flat" a priori state of knowledge curves. This means that we die] not use much information external to the ciata in Attachment 3. For example, we macie no attempt to use the engineering models described in, e.g., Reference F2], p. H-60. This may have been possible by modeling the uncertainties in the variables of the engineering models. This idea was curves for the Ai's through equation (151. This was done by a discrete probability approximation tech- nique. The implied 90 percent confidence interval for ~ is t.007, .0824. The upper and lower curves in Attachment 6 are clerived from equation (16) and are 6~(A4) = 1f 1 - (.082) A4] 6 6~(A4 ) = 1f 1 - (.007) A4] 6 REFERENCES (17) t1] Report of the Presiclential Commission on the Space Shuttle Challenger Accident, Volume 1, June 6, 1986, Washington, D.C. t2l Report of the Presiclential Commission on the Space Shuttle Challenger Accident, Volume 2, June 6, 1986, Washington, D.C. suggested by Feynman (Reference t2l, Appendix F). The uncertainties in the engineering mociels are a. l . . . f3l National Space Transportation System, "In- a possible explanation as to why the models old . . r ,~ . c,- . A. . not predict very well. Finally, the state of knowledge curve for A was clerive(l by propagating the state of knowledge 6 struct~ons tor Preparation ot critical item Mask Assessment (CIRA)," NSTS 2249 1, June 19, 1 987.

ATTACHMENT 1 NASA's Proposed CIRA Technique. 0 uJ u <: ~ ~ ~ `,, ,,, ,x, ~ ·r~ `1` O Off LL ~ L1~ ~ Us U ~ ~ <~ m ~ ~ us Up O _' ~ U ~ > ~ U ~ ~ ~ Lo ~ ~ U ~ ~ ~ U Ud ~ ~ ~ . , ~ ~ . LL .2 := := .> I ~ he Z ~ Z I ~ Z ~ I 3 ~ Y X ~ Z ~ ~ m us Z _ Y O Z > ~ ~ > car 1 - car ~ Z up ~ ~ up up uJ ~ ~ ~ ~ C) Z ~ ~ ,x, ,~, LL O LL Z Z Z Z Z Z O O O O 0- O ~ U U U U' ~J L1J ~ L1J UJ L~ Z Z ~ ~ C) ~ ~ ~ O 1 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ L1J ~ o ~ ~ ~ ~ :) v~ ~) I ,= ,~ _ u~ ~ L., ~ , L~ ~ O L~ O ~ J ~ . ~ ' . LLI . = _ U | O ~ ~, m ~ Z ~ ~ Z ~ - z ,~ Z ,~ C ~ c, ~ - U O 1: L~ O Z ~ ^ ~ I UJ ~ 0 z ~ ~ L~ ,= z ~ ^ ~ ~ ~ ~ ~ ~ Z U C ~ ~ ~ Z ~ m ~ ~ ~ ~ ~ ~ = ~ ~ ~ ~ ~ O z ~ C ~ O ~ _ ~ ~ ~ O O ~ O ~ `~ ~ ~ O O ~ u~ ~ ~ c O ~ C) ~ uJ 2 I _ ~ C Z ~ 0 m ~ m ~ u ~ ~ m ~ m Lu 1l u m ~ ~ ~ c u ~ 1l m c~ , , . . Q ~ ~ V~ ~ ~ m ~ z u~ I ~ ~ _ c~ V ~ ~ V L~ ~( ~= 3 ~ ~ O U O 3 0 ~ ~ ' ~ · r . z I <( ~ O ~ O < 0 z O 5 ~ O v~ v~ I v~ ~ O ~ <~ ~ ~ ~' ~ <~ ~ ~ ~ ,= ~ U ~ O ~ ~ u ~ U ~I L~ ~ O Lu C~ ~C > Z ~ ~ 4< LL ~ Z O L~ t~~ ~ t1~ Ud C~ O ~ ~ 11 -m ~ ~ ~ _ ~ ~ Z ~ ll ~ ~ ~ ~ ~ O c~ ~ ·= O O , m 134

ATTACHMENT 2 0-Ring Anomall" Compared with Jolnt Temperatur" and Leak Check Pressurc Flight (Solid Prossurc Jolnt or Rockot JolnV (In pal) Tomp. Motor Date Booster) O-Rlaa Fleld Nozzlc Eroslon Blowbv °E DM-1 07/18m DM-2 01/18n8 DM-3 10/19n8 DM-4 OV17/79 QM-1 07/13/79 - NA NA - - 84 NA NA - - 49 NA NA - - 61 NA NA - - 40 NA NA - - 83 OM-2 09/27179 - - NA NA - - 67 QM-3 OV13/80 - - NA NA - - 45 STS-1 04/1V81 - - 50 50 - - 66 STS-2 11/1V81 (Right) AM FloldlP~ma~ 50 50 X - 70 STS-3 03m/82 - - 50 50 - - 69 STS-4 DM-5 STS-5 QM-4 STS-6 STS-7 06/18/83 STS-8 08/30/83 STS-9 11/28/83 STS 41 -B OV03/84 06t27/82 unknown: hardware lost at "a 10t21/82 11/11/82 03/21/83 04/04/83 (RIS]ht) (Len) (Right) (Len) STS 41 -C 04J06/84 (Right) (LoR) (Right) (Right) STS 41 -D 08t30/84 STS 41 -G 10/05/84 DM-6 10125/84 STS 51-A 11/08/84 STS 51 -C 01/24/85 - Nozzle/Prlmary Nozzle/Prlmary Nozzle/Prlmary - (LeR) (Right) - (Rlght) (Right) (Right) (LeR) (Leh) Nozzle/Prlmary Forward Fleld/ Prlmary Nozzle/Prlma~y Att FleldfPrlmary IgnNorfPrlmary FonNard Fleld/Prlmary NozziolPrlmary l~nNerfPrlma~y Inncr GaskcV Prlmary Center Fleldt Prlmary Ccntcr Fleldt Secondaly Nozzle/Prlmary Forward Fleld/ Prlmary Nozzis/Prlmary Dash (-) denotes no anomaly; NA denotes not appilcable. See cnd of attachment for footnotes. 50 50 NA NA 50 50 NA NA 50 50 50 W 50 50 100 50 100(2) 100 200 100 NA NA X (1) (1) _ _ X 200 100 X 200 100 X 200 100 (3) NA NA - 200 200 NA 200 NA 200 200 100 X 100 X NA 100 NA 100 100 200 100 200 100 200 200 135 100 100 - 80 58 68 60 67 67 72 73 70 57 - - _ _ 57 63 63 X 63 70 X 70 - X 70 78 X X X X (4) _ X X X X 52 67 53 53 53 53 53

ATTACHMENT 2 (continued) Flight (Solid Pressure Jolnt or Rocket JolnV (In p~l) Temp. Motor Dato Booster) O~R~ Fleld Nozzle Eroslon Blowbv °£ STS 51-D 04/1 V85 (Right) Nozzle/Prln~ary 200 200 X - 67 (RIght) l~niter/Prlmary NA NA - X 67 (Loffl Nozzh/Prlmary 200 200 X - 67 (Lett) l~niterIPrlmary NA NA - X 67 STS 51-B 04/29/85 (RIght) Nozzh/Prlmary 200 100 X - 75 (L.n) Nozzle/Prlmary 200 100 X X 75 (Len) Nozzl~Socondary 200 100 X - 75 DM-7 05/09/85 Nozzle/Prlmary NA NA X - 61 STS 51-G 06/17/85 (RIght) Nozzle/Prlmary 200 200 X (5) X 70 (Left) Nozzle/Prlmary 200 200 X X 70 (Lett) Igniter/Prlmary NA NA - X 70 STS 51-F 07/29/85 (RIght) Nozzle/Prlmary 200 200 (6) - 81 STS 51-1 08127/85 (Leff) Nozzle/Prlmary 200 200 X (7) - 76 STS 51-J 10/03/85 - 200 200 - - 79 STS 61 -A 10/30/85 (RIght) Nozzle/Prlmary 200 200 X - 75 (Len) ARFlel~P~ma~ =0 ~ - X 75 (Lett) Center Fleld/ Prlmary 200 200 - X 75 STS 61-B 11126/85 (Right) No~l~Prlma~ 2 - 200 X _ 76 (Len) No~l~Prlma~ 200 200 X X 76 STS 61-C 01/1V86 (Right) Nozzlc/Prlmary 200 200 X - 58 (Len) AM FleldJPrlmary 200 200 X _ 58 (LMt) NozzlelPrlmary 200 200 _ X 58 STS 51-L 01/28/86 200 200 31 (1) On STS-6, both nozzl" had a hot gas path detected In thc putty with an Indlcatlon of heat on thc primary O-rlng. (2) On STS-9, onc ot thc right Solid Rocket Boostcr fleld ~olnts was pressurized at 200 psl after a destack. (3) On STS 41-C, btt aft flold had a hot gas path detected In thc putty wIth an Indlcatlon ot heat on thc primary O-rlng. (4) On a center fleld Jolnt of STS 51-C, soot was blown by thc prlmaty and there was a heat effect on the secondary. (5) On STS 51-G, right nozzle had aroslon In ~o places on thc primary O-rlng. (6) On STS 51-F, right nozzle had hot gas path detected In putty with an Indlcatlon of heat on thc primary O-rlng. (7) On STS 51-l, Ictt nozzle had croslon In two plac" on thc primary O-rlng. 136

rat oS88 8888 8 ~ ~ ~ ~ ~ _ _ _ o o o o o o o o o o 8 o o o 8 o 8 g 8 8 8 g o 8 ~ ~ ~ ~ In ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ 3~$~` ~~G ~~` ~~-~ ~0~ 81 ~ ~ ~ ~ ~ 0 = ~ _ o 4. o ._ ._ _ C o ._ o ~ 1- _ _ Io, o . . g In ~ o o o . . o . I. 8 i, - ~ 6'sm~ o ~ o _ ._ . o 8 o o ~ _ 1 o ._ 8 ~ ~ ~ CY ret rut rat ~ ~ ~ ~ ~ ~ ~ ~ us ~ us us us Ut us ~0 %0 0 0 co a~ oo oo a) ca 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 t~J ~J N ~ ~ 0 0 0 ~1 ~ O 1^ 0 ~ ~J ~ ~ ~ ~ ~ O ~ ~ 0 ~ ~ ~ t~J ~ O ~ ~ I%J O O ~ O O ~J ~ ~J _ - 1 ~J O - 1 ~ ~ ~J C) O ~ O ~ O O O ~ O O O ~ ~ O O O O O O ~ ~ ~ O O 4~ CJI - _ ~ ~ ~ ~ o O O 0 ~i 137

ATTACHMENT 4 Occurrence of Field Joint Primary O-rings with Erosion. 3.0 2.5 _ In z 2.0 LL At id 1.5 a a: m 1.0 0.5 0~0 . . -\~e $ . \ \ \ ~ ~ \ \ ~ ~ _ \ ~ \ ~ ~ . v. 50 55 ATTACHMENT 5 Maximum Likelihood Estimate and 90% Confidence Interval for the Number of Field Joint Primary O-rings with Erosion at 200 psi. ., 5 4 Q 3 3 at . 1 o ·. ·. 30 40 50 Pressure Data 50 100 ~ 200 # \~#` # # . . \ .. N." me. ·. a' ,~ 60 65 70 75 80 85 TEMPERATURE ATTACHMENT 6 90 Percent Confidence Interval for the "Probability of Mission Field Joint Failure," as a Function of A4. . 6 _............ it\ : - \ J _ \ .. 2 ~ . i. . \ .... .25 .20 - - o o. In ~ .10 o - Q o .15 ~ / 90% Confidence Interval .05 70 Temperature 80 138 ~4 Probability of Secondary O-ring Failure Given Secondary O-ring Erosion