THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
Division on Engineering and Physical Sciences
500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 2605 Fax: 202 334 2318 E-mail: firstname.lastname@example.org
July 13, 2009
Dr. Dimitri Kusnezov,
Office of Research and Development for National Security Science & Technology Defense Programs (NA-121)
National Nuclear Security Administration
1000 lndependence Ave., S.W. Washington, D.C.20585
Dear Dr. Kusnezov,
In 2006, the U.S. Congress and the National Nuclear Security Administration of the Department of Energy asked the National Academy of Sciences to carry out an evaluation of the quantification of margins and uncertainties framework used by the national security laboratories in support of their nuclear weapons stockpile stewardship activities. The request was in two parts. The first part appeared in the National Defense Authorization Act of FY2007, P.L. 109-364, Sec. 3. That portion of the request was independently endorsed by NNSA, which added another task to those listed in the Act. On November 5, 2008, we delivered to you a report, Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile, which addressed those tasks specially identified in the legislation.
During the final stages of preparing that report, you spoke to the study committee in more detail about the second portion of the request. At that time, you expressed interest in having the committee provide a high-level overview of the way the archival underground nuclear test data was being used by the labs in its application of QMU. Specifically you asked the committee to,
Assess how archived data are used in the evaluation of margins and uncertainties. This includes use for baselining codes, informing annual assessment, assessing significant finding investigations (SFls), etc. Are the design labs fully exploiting the data for QMU? Are they missing opportunities?
The study committee began this last phase with a two and one-half day visit to Los Alamos National Laboratory on August 13-15, 2008. During the course of that meeting, we met with several LANL staff members to discuss their applications of the archival data and the laboratory's efforts to make the data more easily usable. A copy of the agenda for that meeting is attached. On November 3, 2008, three
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Division on Engineering and Physical Sciences 500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 2605 Fax: 202 334 2318 E-mail: email@example.com July 13, 2009 Dr. Dimitri Kusnezov, Director Office of Research and Development for National Security Science & Technology Defense Programs (NA-121) National Nuclear Security Administration 1000 lndependence Ave., S.W. Washington, D.C.20585 Dear Dr. Kusnezov, In 2006, the U.S. Congress and the National Nuclear Security Administration of the Department of Energy asked the National Academy of Sciences to carry out an evaluation of the quantification of margins and uncertainties framework used by the national security laboratories in support of their nuclear weapons stockpile stewardship activities. The request was in two parts. The first part appeared in the National Defense Authorization Act of FY2007, P.L. 109-364, Sec. 3. That portion of the request was independently endorsed by NNSA, which added another task to those listed in the Act. On November 5, 2008, we delivered to you a report, Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile, which addressed those tasks specially identified in the legislation. During the final stages of preparing that report, you spoke to the study committee in more detail about the second portion of the request. At that time, you expressed interest in having the committee provide a high-level overview of the way the archival underground nuclear test data was being used by the labs in its application of QMU. Specifically you asked the committee to, Assess how archived data are used in the evaluation of margins and uncertainties. This includes use for baselining codes, informing annual assessment, assessing significant finding investigations (SFls), etc. Are the design labs fully exploiting the data for QMU? Are they missing opportunities? The study committee began this last phase with a two and one-half day visit to Los Alamos National Laboratory on August 13-15, 2008. During the course of that meeting, we met with several LANL staff members to discuss their applications of the archival data and the laboratory's efforts to make the data more easily usable. A copy of the agenda for that meeting is attached. On November 3, 2008, three
OCR for page 1
members of the study committee made a visit to Lawrence Livermore National Laboratory to discuss these same issues with LLNL staff. A copy of that agenda is also attached. This letter report presents the results of the study committee's analysis based on those two meetings. The report begins with a short background section followed by the findings, recommendations, and analysis. A short description of the key diagnostic methods is given in an Appendix. We hope you find this report useful, and we are pleased to have been of service to you on this important topic. Respectively subm itted, John Ahearne, Chair Committee on the Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile
OCR for page 1
Background The archival data base is derived from the data collected from over 1000 nuclear tests including over 800 underground tests carried out from 1958 to 1992. These tests have provided the basis for determining the performance and margins of nuclear weapons now making up the stockpile. Over the years—but particularly since the end of testing—lab scientists and engineers have been engaged in major efforts to re-analyze archival data to better understand the science of nuclear explosions. Re-analysis using modern tools and techniques is one key objective of working with the archival data. This includes a "forward analysis" whose output is the quantity actually measured by the test diagnostics. Re-analysis requires accounting for the effects of all of the test equipment-cables, sensors, oscilloscopes, etc. ln particular, the re-analysis attempts to determine the important sources of uncertainty from the data and to reduce those uncertainties to the extent possible. The increased understanding arising from the re-analysis is directed at developing better simulation models for use in quantifying margins and uncertainties of weapon behavior. An important challenge is to balance the work on this re-analysis with the work on new data being obtained from a variety of experiments. Several different kinds of diagnostics were used to monitor the nuclear weapon test behavior. These included methods for measuring gamma-ray flux, neutron flux, and X-ray flux (the set of 'prompt' diagnostics), as well as the production of radionuclides (radiochemistry). Elaborate experimental arrangements were set up for each test to make these measurements, and large quantities of data were produced. These data were recorded primarily in the form of oscilloscope traces, film, chemical samples, and written records which now make up the archival data base. Re-analysis requires that these data be in a useful form; that is, digitized, searchable, and readily accessible. Archiving these data by scanning the original records, converting to a digital format, and cataloging the digital records has been an ongoing and critical activity of the national security labs over the last few decades. This letter report presents an evaluation of the efforts by LANL and LLNL to make use of the archival data in applying the QMU framework to the assessment and certification of the weapons stockpile. The objective is to provide a broad overview of these efforts and not a detailed analysis. ln addition, the report presents an assessment of key issues related to the archiving process itself. Task Statement Evaluate how archived data are used in the evaluation of margins and uncertainties. This includes use for baselining codes, informing annual assessment, assessing SFIs, etc. Are the labs fully exploiting the archival data for QMU? Are the Labs missing opportunities in their use of archival data? Findings and Recommendations In brief, QMU has three main functions: 1) uncertainty quantification (UQ); 2) providing a basis for avoiding performance cliffs; and 3) the management functions of 3
OCR for page 1
prioritizing programs and transparently communicating assessments of reliability and confidence. The program of data archiving and data mining is heavily, and perhaps understandably, weighted toward UQ. This involves reanalysis of old UGT data with modern advanced simulation and computing (ASC) codes and machines, supplemented in many cases with expert judgment to reduce errors (as, for example, with digitization of old data films and scope traces) and to eliminate certain anomalies (as, for example, in identifying incorrect or misplaced entries in tabular data). Ultimately, the purpose of data archiving and mining is to produce a living and dynamic record of the labs' past activities, especially in UGTs, that must, as time goes by, partially substitute for the corporate wisdom of the designers and technicians who produced these data when these people are no longer available for mentoring. lt is important not only to reduce uncertainties in the conventional sense of tightening error bars, but, perhaps more importantly, to also inform overall uncertainties in the minds of the new generation of designers in what they and their predecessors really know about nuclear weapons. In other words, this would provide a documented record of the limits of knowledge. There are two kinds of uncertainties in any major science-based enterprise: the first is the traditional uncertainty associated with a specific measurement or modeling of a physical process and the second is the more general uncertainty arising from incomplete or (partly) erroneous archived data that are not knowledgeably used. lt is this second kind of uncertainty that we address here. ln particular, we are concerned with the distinction between information—organized data however well-archived—and knowledge stemming from expert use of the data. Using information without knowledge will always create more uncertainties than those arising from the raw information. The key is peer-reviewed, contextual understanding. The labs have embraced the QMU process and are using this to frame their understanding of all aspects of nuclear weapons and the nuclear weapons program. The labs are exploiting archived data to the best of their understanding with the effort available. They are evaluating the usefulness and relevance of the data, quantifying the "quality" and "uncertainty" in the data, and using modern analysis to derive as much information as possible from the data. Therefore, one can conclude that they are exploiting the data in a QMU context. 1 lt should also be noted that the archived data analysis process is helping with the knowledge transfer between designers with test experiences and our 21st Century designers. Finding l-a: The design labs rely heavily on archived data from nuclear tests in their assessment of margins and uncertainties. Their assessment makes appropriate, judicious use of these data in combination with simulations, data from non-nuclear experiments (past and ongoing), and modern analysis 1 Any attempt or effort to document the contextual use of data must also be peer-reviewed in order to ensure that “expert judgment” does not introduce unwarranted bias resulting from “implicit mental models” that are not made “explicit.” 4
OCR for page 1
techniques. 2 Finding 1-b: one component of uncertainty arises from physics that is modeled poorly or not at all. The only way to assess this component is to compare against data. Thus, comparison against archival data is an essential part of the labs' efforts to quantify uncertainty. Finding 1-c: The labs rely on archival nuclear-test data in their everyday work on stockpile stewardship. These data are the foundations for: Setting up and applying "baseline" computational models of stockpile system Resolution of SFls Certification of altered designs in LEPs Annual assessments of stockpile weapons Finding 1-d: The younger staff members at the labs are excited about working with the archival data. The labs are making extensive use of archived data in improving their understanding of the performance of nuclear weapons. This includes reevaluation of margins and uncertainties and baselining of nuclear explosion simulation codes. There is a good deal of re-analysis of experimental data from nuclear tests, including prompt diagnostics and radiochemistry. The primary emphasis to date has been on re- evaluation of tests that are relevant to weapons that are currently in the stockpile. 3 The care and attention to detail is impressive. There are many excellent examples of re- analysis of the archival data from both labs. All this re-analysis has led to new baseline computational models for the stockpile weapons along with improved metrics or identifying performance margins and uncertainties. There is considerable collaboration between LANL and LLNL on developing re-analysis tools as well on other aspects of using the archival UGT data. The process of re-analysis has led to discovery of a number of errors in the reported experimental data, in data analysis, and in computational modeling. The result is better understanding across the board. ln addition, systematic differences between LANL and LLNL have recently shown up in some areas-in particular inference of yield from radiochemistry data. The current difference is in the conversion of the measured fission products observable to a number of fissions. The two labs are currently working 2 This use of data is discussed at some length in this committee’s earlier report “Evaluation of Quantification of Margins and Uncertainties: Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile.” 3 Less attention has been given to the re-analysis of data relating to testing of non-standard designs, It is possible, however, that more thorough examination of data from non-standard designs could give a fuller understanding of the science of nuclear explosives (which could be highly relevant to above-ground experiments being fielded now such as inertial confinement fusion). In addition, such examination could help better anticipate and understand foreign developments. 5
OCR for page 1
together to resolve this issue. The archival data (that is the collection of different types of measurements) are being used to check the code predictions in a careful and systematic manner (driven in part by the goal to better quantify the uncertainties). The QMU framework has provided a certain amount of formality and focus regarding these comparisons. Two points are particularly noteworthy: a.) The re-analysis of the old data is being done carefully and a variety of modern approaches are being used to quantify the precision of the actual measurements (e.g., the current recorded on the oscilloscope). The code outputs are being propagated forward by combining these outputs with quantitative descriptions of the measurement systems to calculate the actual quantity that was measured (e.g., the current on the oscilloscope). This approach makes for an 'apples-to-apples' comparison. More importantly, it is requiring the "new" people doing this work to develop an improved understanding of the measurement capabilities used to obtain the data and to critically analyze what was actually done in the UGT experiments. lt is clear that the scientists are employing rigor and looking at the data in depth. b.) lt was very pleasing to see that the younger scientists are approaching re- analysis of the archival data with enthusiasm and are looking at the old measurements carefully to gain an in-depth understanding of the quality of the data (measurement precision, intended purpose). The labs are exploiting the archival data well in answering questions related to stockpile assessment and certification issues. Given the resources available, a balance has to be struck between digitally archiving past information and current work. lt appears that the labs are striking a good balance. They clearly stated their prioritization for using the archival data: Priority 1: Directed Stockpile Work activities Priority 2: Verification and validation activities and science campaigns. Several staff members at the labs stated that because of resource constraints, the science campaigns have been given a lower priority. The number of staff in radiochemistry has dramatically declined since testing ended. There remains a small core that focus on re-analyzing the data and developing new ways to examine the data. It was clear to the panel that all the scientists who gave presentations recognize the value of the archived data and are utilizing them well. As noted by LLNL, “Many Weapons Program personnel now expect information to be available online, are willing to put some effort into that process, and some have even stepped up to champion ongoing funding." Finding 2-a: Given finite resources, NNSA and the labs must balance investments in archived data with investment in other essential areas. No area, including archival data, receives enough resources to fully tap its 6
OCR for page 1
potential. The balance chosen by the labs appears reasonable. Finding 2-b: There has been enormous progress in making data accessible to users, but there are concerns about shortcomings that somewhat hinder this effort. The labs have been carrying out a major effort over the last several years to archive the UGT data in an easily accessible way. At LLNL, digital scanning of the data is nearly complete for all critical data for stockpile weapons. About three years will be needed to complete scanning of all records except those that cannot be readily scanned, such as large engineering drawings. At LANL, substantial progress has been made on digitization and organization of the data, but much more is needed. When new UGT documents are found they are scanned and cataloged. Elaborate systems of electronic content management are in operation with robust and stable browse and search capabilities. The staff members at the two labs now expect the data to be on-line and argue that they cannot do their work otherwise. Nevertheless, there are concerns and shortcomings about the current archives. Data availability and format can be a hit-or-miss proposition on occasion. ln some cases, the data are missing, in the wrong format, in remote locations, or not as seamless as desired. For example, original notebooks of the designers are sometimes missing, although this is not common. Also, there are cases in which the collections have stopped because of lack of resources. Some data are in individuals' safes and have not been put into the formal archives. For earlier tests, the data are less complete, and as the designers familiar with those tests retire, the ability to fill in the gaps decreases significantly. Some of those data are important because they are more likely to include failures or anomalies that are needed for QMU applications. Those data are also fundamental to understanding nuclear designs that might be pursued by other nations or terrorist groups. ln general, while substantial progress is being made on digitizing and organizing the archival data, much more would be desirable. To some extent, these concerns could be alleviated by more resources. Current and expected limitations on resources for the weapons program, however, require that difficult choices have to be made. Nevertheless, it is the committee's judgment that the balance currently being struck by the labs—given the current budget realities—appears to be about right. 4 Recommendation 2 5 : The current funding balance between the larger weapons 4 Another issue that has been raised is the balance struck by the Labs between analyses of older tests with less sophisticated diagnostics that nevertheless may have focused on resolving specific physics issues and the analyses of more recent tests of devices similar to those in the stockpile that featured more sophisticated diagnostics. The committee did not receive presentations on this balance, but it appears to be an issue that should be explored in the future. 5 Recommendations are numbered to correspond to findings with the same number. 7
OCR for page 1
program and data archiving appear appropriate to the committee and, therefore, the committee recommends maintaining that balance. Nevertheless, to the extent possible, there are some specific areas of possible improvement with data archiving, storage, and access that should be addressed. Finding 3: No matter how correction and annotation of the archives are ultimately handled, it is imperative that the Labs ensure that previously recognized archive errors are not propagated, only to be re-discovered at a Iater date or, worse yet, missed altogether in later re-analysis. The ongoing work with the archival data has repeatedly shown that there exist occasional inconsistencies or errors in these records. The fact that these often subtle mistakes are being caught speaks highly of the ongoing work. ln some cases, discovery of such errors has led to clarification of long-standing discrepancies in model comparisons with UGT data. Errors introduced by the digitization process itself should also be considered. This can be a source of noise in its own right. It is extremely important that the labs construct a systematic means by which such problems are tracked and the archives accordingly annotated. A formal change process is not now in place; currently erratum documents are inserted with the original data. There exist, however, software tools that allow such annotations of document archives ensuring that any future extraction of data, or construction of derivative data sets or database, is always updated with the latest annotated version of whatever resides in the parent document data archive. Furthermore, these corrections need to be propagated both forward and backward to past and current users of the data. Such software tools were developed originally specifically for version control of software but have since seen much broader application. Recommendation 3: Formal change control should be instituted. Finding 4: LANL and LLNL have made significant progress on data exchange and establishing coordinated archiving procedures. Less progress is apparent across the entire weapons complex. lmproved data exchange and coordination could help reduce duplication of effort. The committee acknowledges that there are several factors that complicate this coordination. Among them are differing security procedures and need-to-know philosophies within the various complex entities; increased vulnerability as a result of more extensive connectivity; and greater resource requirements to support requests from different parts of the complex. Nevertheless, between LANL and LLNL at least, the championing of greater cooperation for data exchange has resulted in a substantial volume of data and analyses transferred between labs and a dramatic drop in response time for fulfilling requests between the labs. In addition to the archives at LANL and LLNL, Y-12 has the Stockpile Knowledge Repository (SKR) that contains as-built information for a significant number of our nuclear tests and much of our production builds. It is currently in use at both LANL and 8
OCR for page 1
LLNL for supplying information to set up baseline calculations. While each lab has funded the operation in the past, apparently neither lab is providing funding now. Recommendation 4-a: lnter-lab data exchange should continue to be strongly encouraged and improved across the weapons complex. Recommendation 4-b: There should be greater cross-complex coordination for archiving. Even without cross-complex coordination, however, NNSA should ensure that data archiving programs are being maintained as a priority across the complex, including Yl2, Pantex, etc. Finding 5: A systematic plan for remote, secure data backup does not appear to be evident at the labs. Because of the high value of the digital archived data and the large investment in ongoing re-analysis of these data (and therefore the high value of the resulting derivative data), it is essential that there be secure backups of both; preferably at a geographic location well separated from individual labs. (For example, one might consider SNL as the backup site for LANL and vice versa). The Labs might be able to benefit from established data management and curation practices used in other parts of the scientific community. Disaster recovery (off-site) is under development at LLNL, but the archived data are not yet all backed up in case the entire lab went down. However, we did not discern in our discussions any indication of a systematic plan for such backups: our understanding is that there are a variety of backup schemes, some relying on a lab-wide backup system that is regarded with some caution among several of the lab staff we interviewed. Recommendation 5: Electronic back-up at a different geographic location should be a high priority. We recommend that the Labs formulate and implement an integrated backup plan for all of the digitized data archives, plus all derivative data sets, based on one or more off-site backup locations. Finding 6: The committee finds the work of J. Mercer-Smith (at LANL) to systematically organize data from numerous underground tests to be noteworthy. Mercer-Smith has taken on the task of producing what amounts to a catalog raisoner of several hundred UGTs that fall into an especially important class for understanding weapons physics. A major goal of the database is to identify “near neighbor” data for new designers. 6 This task would be completely impractical without the scanning and digitization of the several hundred records, 6 At issue is whether a “near neighbor” is “near” because the tested device is similar to that of a stockpile weapon or “near” because it reveals a sensitivity to a particular phenomenon that, if a stockpile weapon were sensitive to that phenomenon, could reveal a problem. 9
OCR for page 1
along with the prospect of modern re-analysis of the data with ASC assets. The resulting compilation, informed with Mercer-Smith's expert judgment will be a useful, perhaps indispensable, tool for the younger generation of designer. This activity is to be applauded, but younger scientists at the labs (as well as those in the Inertial Confinement Fusion program) need to be involved in its implementation so that modern software and cultural approaches are implemented and so that its relevance and cross-fertilization with the National Ignition Facility are not lost. Recommendation 6-a: Methods of searching the archived data to identify data that is relevant to a specific design should be developed and extended to other areas. The labs should encourage senior designers to work on compilations in the same spirit, in other complementary areas of weapons knowledge. Recommendation 6-b: Such compilation work should extend across lab boundaries, ideally involving collaborations among senior designers at different labs. Finding 7: Publication in appropriate journals of the knowledge gained by the current generation of designers in reanalysis of the archival data is not taking place to the extent that is desirable. lssues of data archiving go well beyond the present-and most valuable-efforts to make it possible to retain knowledge from the past. Even a perfect digitization of all UGT data amounts to the organization of information, and not necessarily the creation or preservation of knowledge. lf the next generation of designers goes on as they do now, not setting down in a suitable scientific paper the knowledge they have gained from a scientific project, but being content just to generate information, their contributions are seriously incomplete and the technical validity and integrity of the “stewardship” could be seriously compromised. The goals of QMU are best met within the traditional scientific process in which results are not just catalogued in a database, but exposed to serious peer review by the scientific community. This sort of knowledge will last. Unfortunately, there is little reward for scientists to write up their results in this manner. The Labs might benefit from exchanges with other disciplines (aerospace, seismic research, etc.) that are trying to interpret data with a complex technical content across generations of researchers. Recommendation 7: We strongly recommend that lab management encourage and reward appropriate publication of results in such peer-reviewed journals as Defense Research Review, and that-somewhat in the spirit of academic researchers in the university-management set up a scientific staff evaluation process that has the necessary rewards for publication. Unless this peer-review publication system happens, uncertainties will continue to propagate in what the new generations produce. 10