Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 39
Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile 4 Comparison and Contrast of the Use of QMU Task 3: Evaluate how the use of that [the QMU] methodology compares and contrasts between the national security laboratories. Earlier reports1 raised concerns that differences in the implementation of QMU among the three national security laboratories (both among the labs and among various groups within a single lab) could cause confusion and limit the efficacy of QMU as a framework for assessing and communicating the reliability of nuclear weapons. Finding 3-1. Differences in the implementation of QMU methodologies among the national security laboratories can be beneficial for promoting a healthy evolution of best practices. Sandia National Laboratories’ requirements are different from those of the design labs, because many of their components can be extensively tested under many of the required conditions (unlike the design laboratories’ nuclear explosive packages).2 1 U.S. Government Accountability Office, NNSA Needs to Refine and More Effectively Manage Its New Approach for Assessing and Certifying Nuclear Weapons, GAO-06-261 (2006); U.S. Department of Defense, Report on the Friendly Reviews of QMU at the NNSA Laboratories, Defense Program Science Council, (March 2004). 2 A slight revision in Finding 3-1 has been made to correct wording that mistakenly implied all of Sandia’s components could be extensively tested. Other changes were made in the second paragraph of text following this finding.
OCR for page 40
Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile Methods for implementing QMU continue to evolve, as they should, and the laboratories should explore different approaches as a means to determine the best approach for a given warhead. For example, LANL has focused on estimating uncertainties by a sensitivity analysis that examines the variations in simulated primary yield resulting from variations in input parameters (e.g., pit mass) for a given weapon. LLNL, on the other hand, has attempted to develop a comprehensive model that explains the test results for a variety of different primaries and thus addresses modeling uncertainties as well as parameter uncertainties. Some differences in approach arise naturally from the different missions of the laboratories. For example, much of SNL’s work is different from that of the design labs, because it involves warhead components other than the nuclear explosive package (e.g., the firing set and the neutron generator). In principle, SNL can test its systems under many of the relevant conditions; for these conditions SNL is not forced to rely on simulation codes to generate estimates of thresholds, margins, and uncertainties. For practical reasons, SNL cannot test statistically significant numbers of some of its components and therefore still uses computational modeling; however, the models can be challenged in many cases by full system tests. LANL and LLNL, on the other hand, cannot perform full system tests and must instead rely heavily on simulation codes in their assessments of margins and uncertainties. SNL also cannot test its components in “hostile” environments, in which the warhead is subjected to a nearby nuclear explosion, and thus much of its hostile-environment work shares many of the challenges faced by the design laboratories. Recommendation 3-1. The national security laboratories should continue to explore different approaches—for example, using different codes—for QMU analysis for estimating uncertainties. These different methods should be evaluated in well-defined intra- and interlaboratory exercises. Differences in methodology are potentially positive, leading to healthy competition and the evolution of best practices. To determine a best practice, however, would require an ability to assess various competing practices. The committee has not seen an assessment of competing uncertainty quantification methodologies at any of the laboratories, nor has it even seen an organized attempt to compare them. Finding 3-2. Differences in definitions among the national security labs of some QMU terms cause confusion and limit transparency. Table 4-1 shows that, in some cases, earlier concerns about inconsistencies continue to be valid. (More information on this topic is included
OCR for page 41
Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile TABLE 4-1 Comparison of QMU Usage at the Nuclear Weapons Laboratories for the 2007 Annual Assessment Item Sandia Usage Los Alamos Usage Livermore Usage Minimum primary yield, Yp,min N/A Ytot > 80% Valid model Ytot > 90% Valid model Magnitude of U One sigma Two sigma One sigma Determination of U Statistical observation Various calculations and/or expert judgment Various calculations and/or expert judgment Figure of merit K-factora M/U M/U aK = M/U with U measured at one sigma. in Table B-2 in the classified Annex.) For example, in their presentations made to the committee, it became clear that researchers at LANL and LLNL were using different definitions of uncertainty (one sigma vs. two sigma) and different definitions of the important threshold Yp,min (minimum primary yield). The committee learned that the design labs have established a joint group on QMU and are working to establish common definitions and agreement on common metrics and failure modes, but clearly this is a work in progress. It is particularly important that common definitions be developed for parameters used in external communications. Recommendation 3-2. To enhance transparency in communicating the results arising from the QMU analyses, national security laboratories should agree on a common set of definitions—such as the sigma level designating the magnitude of uncertainty—and terminology. Inconsistencies in the definition of uncertainty and common terms such as Yp,min (minimum primary yield) are unnecessary and confusing and should be eliminated. Finding 3-3. The consistency and transparency of the application of QMU are being inhibited by the lack of consistency within each lab and the lack of documentation. In their presentations, NNSA and each of the laboratories told the committee that draft QMU guidance documents were being prepared by each organization. At this writing, however, none of these documents had been completed. The committee believes that documentation of the QMU approaches used by each lab (even in draft form), as well as overarching
OCR for page 42
Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile policy guidance on the implementation of QMU from NNSA, will be essential for improving consistency and transparency in the implementation of QMU and for facilitating peer review. Accordingly, documentation must be given high priority. As noted above, the use of different approaches to QMU can be a strength as long as methods are documented to make them more transparent and to assist researchers in communicating effectively with one another, with management, and with outside audiences. Recommendation 3-3. It is urgent that NNSA and the national security labs complete the development and issuance of QMU guidance documents in time for the current assessment cycle. This process should be used to drive consensus among lab scientists. The documents should be updated as the methodology matures. Finding 3-4. The QMU framework has yet to be clearly defined by the national security laboratories collectively or individually. This framework must identify a more comprehensive set of performance gates and describe how QMU is used to analyze each. A possible outcome of this process is that QMU is not appropriate for a particular performance gate. QMU is often conflated with the whole set of tasks and tools that must be carried forward for stockpile stewardship and design of the RRW. These tasks and tools exist independently of the way that QMU is defined. The tools used in the QMU process are, for the most part, already in widespread use; this is not the issue. Rather, the issue is that the overarching QMU process needs to take into account various divergent views on the essence of the process. An incomplete QMU methodology could also result in a situation in which the blind application of QMU increases the likelihood of missing an alternative failure mechanism or of hiding it altogether. If this happened, efforts to increase a margin and improve the apparent confidence factor of a nuclear explosive package determined from the application of QMU could activate an alternative failure mechanism. For example, design changes that enhance yield margin could introduce one-point-safety concerns. Recommendation 3-4. The national security labs should carry out interlaboratory comparisons of different methods for finding and characterizing the most important uncertainties and for propagating these uncertainties through computer simulations of weapons performance.