National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories

EXECUTIVE SUMMARY

In recent years, there has been concern in Congress about security andς operations management at Los Alamos National Laboratory (LANL) and Lawrence Livermore National Laboratory (LLNL). As a result of this concern and a general preference for holding regular competitions for ongoing federal contracts, Congress directed the Department of Energy (DOE) to hold open competitions in 2004-2005 for the management and operations (M&O) contracts for both LANL and LLNL, as well as for other DOE national laboratories. (Since the 1940s, the University of California has been the M&O contractor for LANL and, since its inception in 1952, for LLNL.)

The execution and accomplishments of the scientific programs1 did not appear to be a factor in the decision to hold a competition for the M&O contracts at LANL and LLNL. In view of that, the National Nuclear Security Administration (NNSA), a semi-autonomous part of the DOE that is responsible for both laboratories, wanted to ensure that the contract competitions preserve the high-quality science and engineering currently being performed there, and so it asked the National Research Council (NRC) to do the following:

1  

Throughout this report, “scientific programs” should be interpreted broadly as not only science and engineering R&D, but also any work at the laboratories that requires a combination of technical skill and creativity.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

[I]dentify key management principles for ensuring high scientific quality in world-class weapons and energy R&D and recommend how best the NNSA can create meaningful qualification and selection discriminators to help ensure world-class scientific quality is maintained in programs and activities at LANL and LLNL. The [NRC] will conduct its study with careful attention to the missions of LANL and LLNL and the needs of NNSA, the current situations at LANL and LLNL and operating requirements imposed upon them, the trends in the management of scientific activities at other relevant federal R&D organizations, and the future availability of scientific manpower required at LANL, LLNL, and similar laboratories.

The NNSA informed the NRC that it had assigned to others the responsibility for ensuring that security and operations management would be properly addressed by the competitions. To address the study’s charge, the NRC constituted the Committee on Criteria for the Management of Los Alamos and Lawrence Livermore National Laboratories, a group of 15 scientists and engineers with experience in science and technology (S&T) management in a variety of institutions who met during January-April 2004 in Washington, D.C., and at each of the two laboratories, conducted a set of site visits to other DOE laboratories, and gathered additional information. The committee developed the following primary findings and recommendations for the NNSA:

  • Run the competitions for both LANL and LLNL simultaneously. Because of their unique role in nuclear weapons research and development (R&D) and stewardship, LANL and LLNL serve as peer communities for one another. The interplay between these two laboratories is very important, and there is a strong sentiment at the laboratories that their coordination and constructive competition are facilitated by their being managed by the same contractor. The NNSA should hold the M&O competitions for both laboratories simultaneously, allowing offerors to bid on an individual laboratory or on both, and should then evaluate the resulting proposals to ascertain which hold the best promise for ensuring that the laboratories’ programs remain coordinated and that the laboratories continue to serve as peer reviewers of one another’s work. The NNSA should also evaluate its own capabilities for playing that coordinating role in the event that different M&O contractors are selected for LLNL and LANL.

  • Use a single proposal evaluation board assisted by a panel of experts in science and technology management. The committee recommends that the NNSA constitute a single very knowledgeable evaluation board to evaluate the proposals it will receive for the management and operation of LANL and LLNL, to better en-

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

able comparison of management ideas from across the entire set of proposals. The committee also recommends that the evaluation board have available, as an added resource, a panel of experts in S&T management to help assess offerors’ responses in areas where additional S&T experience is required and to help formulate questions for the evaluation board to pose to the offerors.

  • Preserve and improve what’s working well. Congressional concerns about security and project management at LLNL and LANL—which are beyond the scope of this study—were not accompanied by criticism of the S&T programs at the two laboratories. Therefore, attempts to address the former should be balanced and integrated with actions to preserve and improve S&T quality. The NNSA also should conduct the competitions in a way that minimizes uncertainty for the laboratories’ staffs, including working toward contractor arrangements that are readily extendible beyond the 5 years stipulated in federal procurement regulations and immediately communicating any ground rules for the competitions that bear on continuation of employment and benefits.

  • Seek contractors that can retain the laboratories’ world-class scientific staffs. The NNSA’s requests for proposals (RFPs) and proposal evaluations should ensure that the laboratories continue to attract, retain, and develop the world-class S&T staffs that are essential to accomplishing the laboratories’ missions. The RFPs and subsequent proposal evaluations should address the aspects under direct control of the contractor (e.g., laboratory facilities and site conditions) as well as the contractor’s contributions to the community that enable the attraction and retention of highly qualified science and engineering staff. Because of LANL’s isolated location, the community aspects should be weighted more heavily in proposals for LANL management.

  • Seek contractors with excellent strategic management capabilities. The RFPs and proposal evaluations should lead to M&O contractor(s) with experience in running a large S&T organization and the ability to create and follow a long-term S&T agenda, and to do so with impeccable integrity and a commitment to national service.

  • Seek contractors that will appoint excellent key personnel to manage the laboratories. The RFPs and resulting proposal evaluations should lead to key management teams for LANL and LLNL, including governing or advisory boards and visiting committees, that are of the highest quality. The credentials of the team members should be comparable to those of individuals in similar positions at other national laboratories.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
  • Seek contractors that are expert at best practices. Effective and efficient business practices facilitate and support the delivery of high-quality S&T work. The RFPs and resulting proposal evaluations should be tailored to find contractors with demonstrated skills in process, financial, and human and physical resource management. The contractors should be able to plan, construct, and operate major facilities and execute effective but efficient security and safety procedures and training programs.

  • Seek contractors that are skilled at stimulating and supporting innovation. The contractors should be effective stewards of their Laboratory-Directed Research and Development (LDRD) funds, as these funds provide one of the principal means for the laboratories to remain on the cutting edge of science and engineering and to recruit and retain outstanding young scientists, and the RFPs and resulting proposal evaluations should identify offerors with the best capabilities in this area. The contractors should also creatively manage interactions with the outside technical community, including foreign nationals, and should provide productive and stimulating processes for managing intellectual property, facilitating technology transfers, and rewarding exceptional contributions by the S&T staff.

  • Seek contractors that can provide intellectual leadership. It is essential to the missions of the laboratories and to the continuance of mission-oriented S&T work of the highest quality that the NNSA’s RFPs and resulting proposal evaluations identify contractors that can lead the laboratories to continued technical excellence.

This report provides background for these recommendations and includes additional recommendations to the NNSA about how it can evaluate the proposals that result from these competitions. It makes some suggestions for incentives to include in the contract to increase the likelihood of receiving proposals from high-quality R&D organizations. The report also discusses more generally some of the principles of good R&D management, including the need to strike a balance between managerial control and the flexibility that encourages scientific and technical creativity.

INTRODUCTION

The U.S. nuclear weapons laboratories—Los Alamos National Laboratory (LANL), Lawrence Livermore National Laboratory (LLNL), and Sandia National Laboratories—are rightly considered to be national treasures. They have excelled for over half a century at creating and sustaining the nation’s nuclear deterrent capabilities. In recent years, their mis-

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

sion has turned to stockpile stewardship, encompassing a range of challenges related to ensuring the reliability, safety, and efficacy of the existing stockpile of nuclear weapons, without full-scale testing. Additionally, the laboratories’ scientific and engineering prowess has been of great value to conventional defense, energy research and development (R&D), and other technical enterprises as wide-ranging as high-performance computing, transportation modeling, and genomics. The high science and engineering quality of these laboratories was well known to the authors of this report through their long experience in the science and technology (S&T) enterprise, and that impression is supported by many metrics of R&D quality.2

The success of the weapons laboratories in meeting their weapons missions and in contributing to R&D well beyond that mission is evidence of their scientific depth and breadth, a depth and breadth that allows the laboratories to recognize and apply research developments from a wide range of fields that might be of importance to their missions a decade or more in the future. The committee believes that this broad and sustained base of scientific, engineering, and technological expertise is necessary in order for the nuclear weapons complex to be prepared for contingencies beyond those addressed by the near-term program.

In recent years, there has been concern in Congress about security and operations management at LANL and LLNL. As a result of this concern and a general preference for holding regular competitions for ongoing federal contracts—and with the concurrence of an external blue-ribbon commission3 appointed by the Department of Energy (DOE)—Congress directed DOE to hold open competitions in 2004-2005 for the management and operations (M&O) contracts for both LANL and LLNL, as well as for other DOE national laboratories.4 (Since the 1940s, the University of California has been the M&O contractor for LANL and, since its inception in 1952, for LLNL.) It is important to note that the execution and accomplishments of the scientific programs5 did not appear to be a factor

2  

For example, LLNL staff authored 1140 peer-reviewed publications in 2003, and LANL produced nearly 1700, while each laboratory received more than 70 patents in that same year. In addition, laboratory staff members have achieved numerous professional recognitions.

3  

U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003.

4  

Energy and Water Development Appropriations Act for Fiscal Year 2004, PL 108-137, General Provisions, Department of Energy, Section 301.

5  

Throughout this report, “scientific programs” should be interpreted broadly as not only science and engineering R&D, but also any work at the laboratories that requires a combination of technical skill and creativity.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

in the recent decision to hold a competition for the M&O contracts at LANL and LLNL. In fact, these scientific programs are widely regarded in the science and engineering community as being world-class. In view of that, the National Nuclear Security Administration (NNSA), a semi-autonomous part of the DOE that is responsible for both laboratories, wanted to ensure that the contract competitions preserve the high-quality science and engineering currently being performed there, and so it asked the National Research Council (NRC) to do the following:

[I]dentify key management principles for ensuring high scientific quality in world-class weapons and energy R&D and recommend how best the NNSA can create meaningful qualification and selection discriminators to help ensure world-class scientific quality is maintained in programs and activities at LANL and LLNL. The [NRC] will conduct its study with careful attention to the missions of LANL and LLNL and the needs of NNSA, the current situations at LANL and LLNL and operating requirements imposed upon them, the trends in the management of scientific activities at other relevant federal R&D organizations, and the future availability of scientific manpower required at LANL, LLNL, and similar laboratories.

The NNSA informed the NRC that it had assigned to others the responsibility for ensuring that security and operations management would be properly addressed by the competitions. To address the study’s charge, the NRC constituted the Committee on Criteria for the Management of Los Alamos and Lawrence Livermore National Laboratories,6 a group of 15 scientists and engineers with experience in S&T management in a variety of institutions who met during January-April 2004 in Washington, D.C., and at each of the two laboratories, conducted a set of site visits to other DOE laboratories, and gathered additional information.7

THE SPECIAL CHALLENGES OF MANAGEMENT AT LANL AND LLNL

The successful accomplishment of the LANL and LLNL missions requires that DOE, the contractor(s), and the laboratories strike the right balance between the needs to (a) meet mission milestones in the near term, (b) maintain the staff and the creative scientific culture necessary to meet the unforeseen challenges of the future, and (c) be able to recognize and pursue novel or breakthrough advances. There is also a widespread perception that these laboratories need to improve their operations management and security—issues that are important but are outside the scope of

6  

See Appendix 1.

7  

See Appendix 2.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

this study—and the committee believes that these goals can and should be pursued in an integrated fashion with the goal of preserving, and even improving, the S&T. Because this study was not asked to address the security and operations management goals, this report discusses them only where they are connected to scientific quality.

The NNSA, in crafting requests for proposals (RFPs) for the management of LLNL and LANL, will have to consider the key aspects of operations management and security and balance those aspects with the considerations about S&T management raised in this report. An overemphasis on detailed program planning and management, or on process definition and management, is often counterproductive to stimulating the best scientific work. Trust, and an expectation that the staff will perform at a high level of quality, create the incentives for excellent accomplishments in science and engineering. Similarly, systems for security inevitably entail risks, and pro forma efforts to eliminate, rather than manage, those risks inherently detract from the quality and productivity of the S&T workforce. It is essential that the M&O contractors for LANL and LLNL be adept at managing within the natural ambiguity of R&D—the inherent uncertainty about how programmatic thrusts will actually play out—and that they have the capacity to respond to unanticipated S&T opportunities.

Fortunately, the conditions that attract and retain an excellent S&T staff are the same conditions that facilitate their best progress toward mission goals: both rely critically on an environment that encourages self-motivated work targeted toward the mission and that provides connections to a wide range of external scientists and engineers. Thus, high scientific quality at LANL and LLNL is dependent on each laboratory’s continuing to attract and retain an excellent scientific staff, and each laboratory’s success at attracting and retaining such a staff requires a laboratory environment that enables the staff to excel.

The contractor also must serve as a buffer between the laboratories’ staffs and pressures to micromanage the laboratories’ operations. Many experts who have examined the working of the national laboratories—most notably the 1994-1995 Galvin Commission,8 and more recently the DOE’s Blue Ribbon Commission on the Use of Competitive Procedures for Department of Energy Labs9—have decried the degree of micromanagement to which these laboratories are subjected and have called for a

8  

Task Force on Alternative Futures for the Department of Energy Laboratories, Robert Galvin, chair.

9  

U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003, p. 13.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

reestablishment of the spirit of trust that formerly made the government-owned/contractor-operated (GOCO) model of laboratory management successful. The M&O contractors for LANL and LLNL can help address this problem by working to restore and improve the level of trust among the DOE, Congress, and the laboratories’ staffs. Among other things, each laboratory’s management and staff should be given some flexibility and latitude in the expenditure of R&D funding. Especially important are the Laboratory-Directed Research and Development (LDRD) funds, which should continue to be deployed largely at the discretion of the LANL and LLNL directors. While the DOE sets the overall missions and oversees the laboratories’ programs, the laboratories’ directors and staffs should be given leading roles in defining and executing the programs of their laboratories because they are in the best position to develop the very-long-term view that is necessary given the magnitude of the challenges faced by these institutions.

ENABLING EFFECTIVE COMPETITIONS FOR LANL AND LLNL

Retaining a Healthy LANL-LLNL Interplay

During the course of its study, the committee became acutely aware of the vital interplay between LANL and LLNL. The need for two centers of competence in nuclear design and management was recognized by the government more than 50 years ago. Nuclear stewardship and the scientific aspects of national security have been and remain too important to leave to any single group, no matter how competent. Absent the normal scrutiny of outside peers, such a deliberate overlapping of missions is desirable, and it has been very successful over the decades in enabling the two laboratories to provide a healthy check of one another’s work. The spirit of competition among equals that now exists encourages the science and engineering staffs of both laboratories to aim for high achievements and ensures that the best S&T efforts are brought to bear on the main missions.

There is a strong sentiment at LLNL and LANL that coordination and constructive competition are facilitated by the laboratories being managed by the same contractor. 10 The committee notes that the following issues could arise if LANL and LLNL were managed by different contractors:

10  

See, e.g., Harold M. Agnew et al., personal communication from 10 former LLNL and LANL directors to Secretary of Energy Spencer Abraham, December 8, 2003. The letter notes the peer review the laboratories provide one another, the need to cooperate and share information, and the

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
  • Coordination between the laboratories might deteriorate;

  • Communication channels would have to be more explicitly identified and monitored;

  • Exchange of scientific personnel and transfer of programs between the laboratories would likely be inhibited;

  • The NNSA would have to provide a non-trivial coordination function, which would complicate the governance of these weapons laboratories and change the nature of the NNSA’s involvement, while risking duplicative oversight; and

  • Different contractors, particularly different kinds of contractors (e.g., one for-profit and one non-profit), might face structural incongruities with respect to regulations or guidelines for such issues as conflict of interest, personnel management, or intellectual property, giving rise to reduced scientific interchange between the laboratories.

The committee cannot assess the magnitude of these risks, and it suspects that proposals for the two M&O contracts could identify promising mechanisms for continuing and perhaps improving the effective interplay between LANL and LLNL. In order for the NNSA to judge most effectively the merits and risks of various combinations of M&O contractors, however, the committee concluded that the NNSA must see those proposals at the same time. Therefore, to provide for that option, the committee makes the following recommendation:

Recommendation 1. The NNSA should run simultaneous competitions, with the same proposal evaluation board, for the management of LANL and LLNL, in which offerors could make proposals for the management of either or both laboratories, and the RFPs should ask all offerors to discuss how they would effect coordination and constructive competition between LANL and LLNL. The prospects for continuing that constructive competition could then be given appropriate consideration in the NNSA’s decision about which contractor(s) to select. The NNSA should also evaluate its own capabilities for playing the coordinating role if LANL and LLNL were managed by different contractors and should factor that evaluation into its contractor selection decisions.

With this recommendation, the committee disagrees with the Blue Ribbon Commission on the Use of Competitive Procedures for the De

   

value of staff interchanges, and concludes: “In our experience all of these are difficult under a single contractor and would be greatly exacerbated with separate contractors.” The letter also asserts that there would be “great value” in competing the contracts simultaneously.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

partment of Energy Labs, which specifically recommended competing these two contracts at different times because each competition “must welcome all interested and qualified bidders to participate.” 11 The committee does not believe that simultaneous (but separate) competitions would hinder such participation.

While the committee does not refute the Blue Ribbon Commission’s conclusion that “there [is] little need to specifically require that the same M&O contractor manage the two laboratories,” the possible problems that could arise should LLNL and LANL end up with different contractors are significant enough that it is prudent to run simultaneous competitions in order to evaluate how well those risks can be managed. The Blue Ribbon Commission’s argument that Sandia National Laboratories (SNL) successfully collaborates with LANL and LLNL in spite of having a different M&O contractor is not completely convincing, because the mission overlaps between SNL and LANL and between SNL and LLNL are not nearly as striking as the overlap between LLNL and LANL programs.

Contract Incentives

The committee believes that a critical element in ensuring that high-quality science and technology are maintained at LANL and LLNL as their management contracts are competed is that the NNSA receive proposals from the most highly qualified organizations. Organizations are motivated to manage a national laboratory for a variety of relatively intangible reasons, such as a sense of national service, prestige, and access to talented individuals and relevant technology; the management fee is probably not the primary incentive nor, the committee believes, should it be.

There is a concern among those familiar with DOE national laboratories that the responsibility of managing a weapons laboratory is of only marginal appeal to potential contractors because of the many possible liabilities and risks and the micromanagement in evidence at many laboratories. Therefore, to ensure that qualified potential contractors will expend the substantial effort involved in developing a proposal, the NNSA should seek and articulate non-financial incentives to be associated with these contracts. The following are probably the most important of these incentives:

  • The contracts could clarify, and expand if possible, indemnification against the consequences of laboratory operations (in the ab-

11  

U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for the Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003, p. 23.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

sence of wrongdoing by the contractor) to the full extent allowed by law and federal regulations. Qualified contractors simply will not put their entire assets and/or reputation at risk for the sake of the small financial reward and the prestige value of managing a national laboratory. The committee does not believe that DOE should require contractors to accept lesser fees in exchange for increased indemnification.

  • The contracts could facilitate the contractors’ use of their best practices (for business practice, risk management, human resources, and so on) as the law permits, and not automatically constrain the contractors to follow DOE practices.

  • Each contract should be formulated in accordance with the principle of performance-based results rather than prescriptive requirements for how the work should be done. Such an approach allows for maximizing the capabilities of the S&T staff and directs the focus of DOE/NNSA oversight toward the significant issues requiring such attention.

  • The contracts should make use of the DOE’s procurement provisions12 that allow it to approve 10-year M&O contracts, with 5-year option terms, in order to create an incentive for long-term commitments appropriate to the laboratories’ missions and to make the up-front investment in proposal preparation less of a disincentive to some potential offerors. The committee’s concern about the use of 5-year contracts for the management of LANL and LLNL echoes that expressed in the recent draft report from the Blue Ribbon Commission on the Use of Competitive Procedures for the Department of Energy Labs, which suggested a framework for retaining an incumbent contractor for as long as 20 years, as long as its performance meets or exceeds DOE expectations.13 A 5-year contract with a simple mechanism for a 5-year extension, or the use of a rolling 5-year contract in the absence of underperformance by the contractor, would seem to be one such desirable mechanism. This approach would also be desirable because it would avoid the disruptions noted below in the section titled “Preserve and Improve What’s Working Well.” As an example of this kind of practice, NASA’s current contract with the California Institute of Technology for the management of the Jet Propulsion Laboratory allows

12  

Acquisition Letter 1996-09, Attachment B.

13  

U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for the Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003, p. 20.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

the contract duration to be extended or reduced according to performance. The committee observes that the United Kingdom moved, in 2003, to a 25-year management contract for its nuclear weapons laboratories. Currently, federal contracting regulations in the United States permit 5-year contracts with extensions of up to 5 years, for a total of 10 years.14 Contracts of longer duration would probably require a change in these regulations.

  • To the extent permitted by law, each contract should have streamlined constraints for managing conflicts of interest to minimize in-hibition of exchanges of personnel and information between the laboratory and its M&O contractor. Also, the contractor should be allowed to participate in the development of inventions and make use of intellectual property originating at the laboratory.

The NNSA should take care that its incentives appeal to both for-profit and non-profit offerors and keep in mind that the reward structure and performance incentives in the contracts will play a prominent role in how the laboratories perform and develop. Obviously, it is very important that the contracts’ performance incentives reward the desired behaviors, and the NNSA proposal evaluation board will have to be mindful of how the performance awards (if any) built into the laboratory M&O contracts will drive management priorities.

One of the greatest disincentives for highly qualified offerors to compete for the M&O contracts is the potential for significant post-award surprises associated with pre-existing environmental, infrastructure, or human resource issues. In at least the case of the latest competition for management of the Oak Ridge National Laboratory, the constraints of the competition led to significant unanticipated challenges for the new management team, which in turn limited its ability to target other important issues and to minimize surprises to the staff. The committee urges the NNSA to arrange for all qualified offerors to be given adequate opportunity to both visit LLNL and LANL and to interact with a broad cross section of laboratory staff. Recognizing that such access will necessarily be somewhat disruptive to ongoing work at each laboratory, the NNSA might consider an extended “bidders’ conference” to be held on-site over a multiday period. Such a conference could be organized with a mix of plenary sessions and parallel tracks dealing with specific issues. This format would limit the disruption of ongoing technical work to a specified period and at the same time ensure that all qualified offerors obtain exten-

14  

48CFR970.1706-1.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

sive access to information that will be necessary to develop a truly informed proposal. This step would therefore maximize the chances of attracting proposals from the best prospective offerors and remove one factor that can contribute to disruptive management transitions.

Evaluating Proposals from M&O Partnerships

It is likely that the NNSA will receive some proposals from various partnering arrangements of different organizations. Such partnerships increase the breadth of the offeror, but they open the possibility for communication gaps within a partnership. In evaluating proposals from partnerships, the NNSA may want to ask whether the strengths of the parent organizations have been combined logically and whether the partnership has identified a workable mechanism for applying the various capabilities and corporate experiences against the varied tasks and responsibilities of the M&O contractor.

Special Considerations for Los Alamos National Laboratory

For a laboratory in an isolated location like LANL, staff attraction and retention and, consequently, scientific excellence at the laboratory, are strongly influenced by the quality of the community schools, opportunities for spouses, employee family assistance programs such as child care, and so on. M&O proposals should include a description of plans for contributing to the Los Alamos community’s attractiveness to current and potential LANL staff. Not only is the town of Los Alamos highly dependent on LANL, but LANL is also by far the largest employer in northern New Mexico, so it has important and highly visible interactions with regional political bodies and segments of the regional economy. Offerors should be asked to demonstrate their understanding of LANL’s connections to the community and region and to describe how they propose to nurture these relationships, consistent with federal regulations governing regional partnerships.15

Other special features of LANL include its distribution of facilities across a large geographic area, a large number of aging facilities, and substantial environmental issues arising from past weapons-related activities. Some of the facilities challenges—such as managing the aging LANSCE accelerator complex, developing a replacement for the Chemistry and Metallurgy Research Building (which is necessary for pit produc-

15  

48CFR970.2673.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

tion and actinide research but does not meet seismic standards), and handling or transferring the functions of TA-18, a training facility of critical importance that requires expensive safeguards—lie at the heart of the laboratory mission and constitute significant pre-existing conditions that will require substantial planning and funding to remedy. Efforts to remedy these facilities challenges have begun, but they will not be resolved by the time the next M&O contract is awarded. The environmental impacts of LANL’s dispersed and incompletely documented legacy wastes are likely to be exacerbated by the effects of the large-scale fires of 2000. These circumstances underscore that stewardship of the local environment constitutes a set of challenges specific to LANL and its future.

Special Considerations for Lawrence Livermore National Laboratory

Lawrence Livermore National Laboratory lies at the edge of the technical megalopolis of the Silicon Valley and is close to other major intellectual centers that include Stanford University, the Universities of California at Berkeley, Davis, San Francisco, and Merced, several campuses of the California State University system, and private colleges and universities. This situation gives LLNL obvious advantages in ease of interaction with others in the research community, and collaboration with University of California faculty members has apparently been very readily available and advantageous for LLNL, because it is almost equivalent to a campus of the university. LLNL’s location also helps it attract and retain scientific and technical staff, though the many R&D opportunities in the region also make that staff more mobile. The Livermore location also means that laboratory employees are affected by the high real estate costs of the San Francisco Bay area, and many employees now live in the Central Valley. Offerors should be asked to describe their capabilities and intentions for dealing with both the strengths and challenges of the community and region.

Just as the mission and activities of LANL are strongly affected by the special needs of nuclear stewardship and its large user facilities, so also are the mission and activities of LLNL affected by the special features of NIF, the National Ignition Facility. About one-fifth of LLNL’s annual expenditures are devoted to NIF, and the requirements of this special project must be reflected in LLNL’s RFP. Offerors should be asked to describe how they will maintain a balanced program of activities at LLNL in view of the large scale of NIF as a relatively new part of the LLNL program.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

Proposal Evaluation Board

Evaluating how well offerors will be able to address the special nature of the LANL-LLNL interplay and the special considerations for each of the laboratories will require a very skilled proposal evaluation board, because the offerors’ capabilities in these areas may be difficult to judge. Similarly, offerors who propose an M&O partnership will have to be assessed against their abilities to manage and integrate the strengths of the component parts of their teams—and that is also difficult to judge. Finally, the section below titled “Recommendations for the RFPs and for Evaluation of the Resulting Proposals” recommends mostly non-quantitative measures for evaluating the S&T aspects of the proposals. For these aspects, non-quantitative measures are more important than any quantitative metrics. For all of these reasons, the committee found that the LANL and LLNL competitions will require a proposal evaluation board with strong expertise in R&D management, or access to that expertise. The latter can be accomplished through establishment of a panel of experts in S&T management, implemented with care to avoid conflict-of-interest issues under the Procurement Integrity Act. An example of the successful use of such an adjunct panel in a DOE procurement was in the source selection for a $200 million DOE contract to extract medical isotopes from the U-233 accumulated at Oak Ridge National Laboratory.

Recommendation 2. The NNSA should constitute a single very knowledgeable proposal evaluation board to evaluate the proposals it will receive for the M&O of LANL and LLNL. (As noted in Recommendation 1, the committee consensus is that the two contracts should be competed simultaneously, and the use of a single evaluation board would allow that one group to compare management ideas set forth in the entire set of proposals.) For that reason, the committee also recommends that the proposal evaluation board have available, as an added resource, a panel of experts in S&T management to help assess offerors’ responses in areas where additional S&T experience is required and to help formulate questions for the proposal evaluation board to pose to the offerors.

Transition Planning and Execution

The instability and uncertainty associated with a competition will, at the least, affect staff productivity, and it will cause many people to consider retiring or leaving. Such departures could have a significant short-term effect on the quality of science and technology at the two laboratories, and the premature retirement of those with nuclear test experience, in particular, could have long-term ramifications. The committee suggests

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

that the NNSA accompany the release of its RFPs with a question-and-answer process directed to the staffs of LLNL and LANL, who will surely have questions and concerns about the ground rules and process for the competitions and who may not know whom to ask during the period of uncertainty. An appealing process that was set up during the 1999-2000 management transition at Oak Ridge National Laboratory allowed for questions to be submitted either online or through a voice mailbox, after which a knowledgeable individual called the questioner to ensure that the question was truly understood before an answer was formulated, and then questions and answers were quickly placed on a Web site that all employees could view. While such a system entails costs and time, it is invaluable for maintaining morale and for minimizing misinformation, and for LANL and LLNL it would reduce the far greater costs of employee frustration and anxiety, which lead to staff departures.

If one or both laboratory competitions lead to a change in contractor, then the following steps would be helpful to minimize uncertainty and disruption:

  • A transition plan should be developed that represents the merger of both the outgoing and the incoming contractors’ transition plans, and this plan should be shared with the laboratory staff. The 1993 transition plan for Sandia National Laboratories appears to be a good model.

  • The NNSA should establish a question-and-answer system similar to the one suggested above to accompany the release of RFPs, but involving knowledgeable individuals from the NNSA and both the outgoing and the incoming contractors so as to be certain that the answers provided are accepted by all parties.

  • The contractor should make no sudden or hasty changes, waiting instead until it thoroughly understands the strengths and weaknesses of current laboratory management procedures.

RECOMMENDATIONS FOR THE RFPs AND FOR EVALUATION OF THE RESULTING PROPOSALS

An accomplished and effective R&D laboratory needs a compelling and well-defined mission, excellent staff, adequate tools and facilities, efficient business and project management, and strong leadership. The quality of the staff is the most important asset, and it can largely compensate for aging facilities, bad management, fuzzy focus, and other deficiencies, but staff quality is also the most difficult asset to rebuild if it declines. Nevertheless, any of those deficiencies can undermine the performance of the laboratory’s mission even with excellent staff. One

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

could argue that the quality of the staff is even more critical for nuclear weapons laboratories in this era of no nuclear weapons testing.

In what follows, the committee makes six major recommendations to the NNSA (Recommendations 3 through 8). The first recommends an overriding principle for the competitions, and the other five offer specific guidance on broad categories of information that the NNSA should solicit through its RFPs; the NNSA might wish to adapt Recommendations 4 through 8 for inclusion in the RFPs. The rationale for each major recommendation, which includes findings from the committee’s study, precedes the recommendation. Following each of Recommendations 4 through 8 are subsidiary recommendations (e.g., Recommendations 4a-4d) that advise the NNSA about how to evaluate the resulting proposals. In cases where the offeror is a new legal entity created for the current competition(s) and therefore does not have its own history, the language in these recommendations about past accomplishments and capabilities of the offerors should be interpreted as referring to accomplishments and capabilities of their parent organizations.

Preserve and Improve What’s Working Well

As argued above, it is critical not to undermine the ability of LANL and LLNL to accomplish their missions, and that ability is heavily dependent on the quality of their S&T. Because the incidents that contributed to the decision to hold open competitions for the LLNL and LANL M&O contracts did not suggest that the laboratories’ S&T programs were lacking, the competitions for these contracts should be structured so as to cause little or no deleterious effect on the execution of the S&T programs. In the experience of the committee members, a laboratory’s efforts to improve security and operations management should be integrated with efforts to continuously improve the S&T, so that management’s emphases are balanced and focused on accomplishing the laboratory’s mission.

The NNSA is already very aware that the laboratory staffs are affected negatively by uncertainty: the uncertainty of their continued employment, benefit plans, laboratory funding, scientific directions, managers, and so on. Therefore, the next M&O contractor(s) should make special efforts to identify quickly and accurately all processes that will and will not be changed, so as to minimize the disruption caused by the competitions. It seems likely that the essential features of the scientific program should not change, because they are not “broken,” but each contractor’s intentions vis-à-vis those programs should be clearly stated to each laboratory’s staff as soon as possible after the management competition. The contractor should do careful homework before trying to effect changes that affect the S&T workforce. That is, the contractor should avoid insti-

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

tuting procedural changes until it has a clear understanding of any real problems that might exist.

In this light, the committee suggests that the NNSA consider adopting the approach followed by the RFP for Sandia National Laboratories in 1992, which also had a solid technical program that needed to be preserved rather than “fixed.” In that case, DOE’s goal was apparently to provide continuity in transition, and that approach seems to have worked well.16 The following section of that RFP no doubt contributed to this end by reducing uncertainty among the scientific staff:

The Offeror shall commit to: (a) offer employment, except for Laboratory Key Personnel, to all other Regular (As defined in Appendix A) Laboratory personnel currently employed and at their existing salary rates; (b) take over existing pension plans established specifically for the current contract and establish or take over savings plans at the same employer and employee contribution rates allowable under the AT&T savings plans; (c) credit employees’ length of service with the incumbent contractor toward any length of service requirements of the replacement contractor for fringe benefits such as severance pay and enrollment in group insurance plans; and, (d) recognize the currently certified collective bargaining agents and their existing bargaining agreements.

The committee is very concerned that the proposed 5-year duration of the LLNL and LANL M&O contracts will lead to additional periods of uncertainty, with their attendant disruptions to laboratory staffs and program execution, for many months every 5 years, and for a longer period whenever a new management team comes in. Such disruptions, when rare, are part of every work environment, but their impact every 5 years could be significant.

Therefore, the committee makes the following overarching recommendation:

Recommendation 3. The important efforts to improve the management of security and operations at the laboratories should be balanced and integrated with efforts to preserve and improve the ability of the laboratories to perform the essential S&T components of their missions. The committee also recommends that the management competitions, and the transition (if there is one) to any new contractor(s), be carried out with minimal disruption to the S&T program. As noted in the section above titled “Contract Incentives,” the committee recommends that the NNSA consider arrangements that reduce the risk of disruptive competitions every 5 years, both to provide an incentive to prospective offerors and to minimize del-

16  

For example, the number of peer-reviewed publications trended upward during the years surrounding the management transition, while the average citations per paper stayed fairly constant, according to data from the ISI Science Citation Index.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

eterious effects on the laboratories’ productivity associated with competitions.

Workforce

The primary determinant of how well the laboratories can address their missions, given the degree to which science, engineering, and technology pervade the programs, is the quality of the S&T staffs. In addition, the long-term nature of the work of the weapons laboratories is benefited by having a stable staff and effective continuity planning, so that corporate knowledge (which, at least in the case of past weapons testing, remains valid on a timescale of decades) is retained.

Through a wide range of discussions with staff members at various levels of LANL and LLNL, the committee found that high-quality scientists and engineers are attracted to the laboratories by the importance and demanding nature of their missions, by the unique facilities, and by the chance to work with excellent colleagues. The existence of funding for fundamental and innovative R&D (the LDRD fund) is a critical motivator, because it assures the S&T staff that an option exists to pursue important long-range questions that are not otherwise addressed by the laboratories’ programs. Many staff members also reported that they were attracted by the university-like environment at the laboratories, which is characterized, for example, by easy and open communication on unclassified S&T issues, by the large number of postdoctoral appointments and visiting students from the United States and a wide range of foreign countries, the presence of other visiting researchers, laboratory-sponsored seminars and conferences, ample opportunity for publishing and collaboration, reliance on peer review by a mix of internal and external experts, and so on. The committee members agreed, based on their experience in a wide range of R&D settings, that such an environment is the most conducive to the best scientific inquiry because it encourages intellectual integrity and scientific objectivity. Therefore, the laboratories’ contractor(s) should have the ability to continue LANL’s and LLNL’s access and interaction with the broad, worldwide science and engineering community and to foster the principles of intellectual integrity and scientific objectivity. These aspects are critical to the success of the weapons laboratories, and they are very important in the attraction and retention of high-quality science and engineering staffs. The contractors’ familiarity with managing a high-quality S&T workforce, and its style and high-level planning, can enable or hinder the ability of laboratory management to successfully manage the S&T workforce.

Staff attraction and retention are also driven by the quality of the work environment, including both tangible factors (e.g., facilities, equipment,

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

support services, child care, benefits, and other personnel policies) and intangibles (e.g., degree of management support, freedom from retaliation for raising problems, an effective grievance process, pride in the organization). As mentioned above, staff attraction and retention also can be strongly influenced by the quality of the community schools, opportunities for spouses, opportunities for future employment, and so on, and the successful M&O contractor must be mindful of these factors. Staff retention also depends on the availability of viable career paths, so that staff members (including those doing highly classified or specialized weapons work) do not feel boxed into their areas should their interests, or programmatic priorities, evolve.

Therefore, the committee makes the following recommendation to the NNSA:

Recommendation 4. The RFPs for the LANL and LLNL competitions should ask offerors to demonstrate their capabilities for attracting, retaining, and developing the excellence of a broad-based, world-class science, engineering, and technical staff and their plans for enabling the laboratory management in this critical area. These plans and capabilities should cover aspects under the direct control of the contractor (e.g., laboratory facilities and site conditions) as well as the contractor’s contributions to those aspects in the communities that enable the attraction and retention of highly qualified science and engineering staff. Because of LANL’s isolated location, the community aspects should be weighted more heavily in proposals for LANL management.

The committee makes four subsidiary recommendations to the NNSA to guide its evaluation of the proposals for the LANL and LLNL M&O contracts as they apply to the laboratories’ S&T workforces:

Recommendation 4a. The NNSA should evaluate the offerors’ plans, track records, and reputation for attracting and retaining a stable and world-class S&T workforce. Dimensions of this evaluation should include the offerors’ high-level plans for the management of the LDRD program; its success and plans for operating postdoctoral, student, continuing education, and training programs; any history with facilitating Intergovernmental Personnel Act (IPA) transfers; and its ability to provide facilities and technical support for an advanced S&T workforce. The evaluation should also include the offerors’ plans for the support of viable career paths for members of the S&T workforce who cross between classified and unclassified activities and its track record and plans for involving S&T expertise in its top management and boards.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

Recommendation 4b. The NNSA should evaluate offerors’ plans for employee benefits to determine if they are competitive with the current benefits in the state of California’s defined-benefit retirement plan, which is widely believed to be generous. If retirement benefits for existing employees are not kept at this level, the loss of many employees eligible for retirement seems highly likely, leading to (among other things) a serious diminution in the laboratories’ corporate experience in the design, construction, and testing of nuclear weapons. If a change in benefits for new employees is considered, the NNSA should be mindful of the effects such a change would have on recruiting and employee morale.

Recommendation 4c. The NNSA should evaluate the offerors’ plans and track records for maintaining other aspects of the work environment that contribute to its attracting and retaining an excellent S&T workforce—such factors as the quality of the buildings and equipment, existence and quality of recreational facilities, child care, safety, and so on—even though they might not be directly necessary for the conduct of S&T work (as is the case with those factors listed or implied in Recommendation 4a).

Recommendation 4d. For each laboratory, the NNSA should evaluate the offerors’ plans and track records for contributing to the quality of life of staff outside of work, such as measures to compensate for the difficulties associated with LANL’s remote location, programs to assist relocating spouses in finding employment, and efforts to enhance the quality of local schools. Proposals for the management of LANL should also include the offerors’ plans for fostering the economic development of northern New Mexico because of the effect that such development has on the ability of LANL to attract and retain excellent staff.

Strategic Management: Vision and Integrity

While the overall missions of LLNL and LANL are set by Congress and the DOE, the M&O contractor(s) must ensure that broad goals are translated into actionable scientific programs that address both near-term needs and long-term goals. The contractors must also ensure that the laboratories sustain and expand the very-long-term knowledge base needed for the nation’s nuclear deterrence capability. The stockpile stewardship program requires a substantial investment in S&T to develop the means both to test the stockpile for reliability, safety, and effectiveness and to maintain those qualities over a period of decades in the absence of full-scale testing of weapons. To carry out this mission, the laboratories must have stable and effective leadership as evidenced by expert strategic, staffing, program, and transition planning. Long-term program needs must not be sacrificed for short-term performance, and so management must resist

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

the common temptation in times of budget stress to cut back on very basic and speculative research efforts without obvious near-term application.

The M&O contractor(s) must be capable of playing a strong leadership role so that the DOE, Congress, and the laboratories’ customers are confident that the laboratories are carrying out their mission without compromising national security and are applying sound financial and project management approaches. Furthermore, the contractor(s) must shield the staffs from any deleterious effects that might arise from the complex interaction among these parties. The contractor(s) must also be seen as having total integrity and enough independence so that the laboratory directors can credibly carry out their statutory obligation to recommend annually to the secretary of energy the certification (or non-certification) of each nuclear weapons system based solely on their laboratory’s technical assessment.

This capability to provide objective and authoritative advice to the secretary of energy is a vital part of the technical mission of the weapons laboratories, and its successful accomplishment relies on a marriage of technical competence with very high levels of real and perceived integrity. The contractor(s) cannot be influenced, or be perceived to be influenced, by political pressure or by the goals of other parts of their organization or parent organizations. A spirit of intellectual independence within the laboratories is also essential because of the synergistic relationship between LANL and LLNL, wherein each has a “devil’s advocate” role to play to challenge and stimulate the other’s work.

A section from the 1992 RFP for the management of Sandia National Laboratories, on “Contribution of Parent Organization(s),” is an excellent statement of the responsibility of the M&O contractor. After revision to reflect the S&T foundations of the LANL and LLNL missions, it might be suitable for wholesale incorporation into the RFPs for those laboratories:

The proposal will be evaluated with respect to the Parent Organization(s)’ (1) strength of ethics, integrity, and accountability, particularly as these qualities relate to stewardship of the nation’s nuclear weapons, (2) strength of technological, engineering, and managerial leadership and reputation, both nationally and internationally, and (3) the value of associated benefits to be derived through the association of the Laboratory with the Parent Organization(s).

The proposal will be evaluated with respect to the Parent Organization(s)’ depth of commitment, strength of organizational structure and nature of responsibilities, and how these are expected to benefit the operation of the Laboratory. Emphasis will be placed on the value added by the Parent Organization(s) and any other major suborganization as well as the rationale for its involvement and the anticipated benefits to accrue to the Laboratory. The roles and projected benefits of any boards and committees proposed to support operation of the Laboratory will be evaluated, including those providing executive management

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

and guidance. The proposal will also be evaluated with respect to the significant contributions and the stature of the Parent Organization(s)’ major contributing personnel (i.e., the personnel in those Parent Organization(s) as defined by the Offeror and not to be confused with Laboratory Key Personnel which are identified by DOE in Section L, Attachment No. 1).

In addition, the value added and appropriateness of the level of the Parent Organization(s)’ proposed oversight efforts of the Laboratory’s activities will be evaluated, including Programmatic Activities, Facilities Management, Site Services and Business Management. The effectiveness of how oversight is currently being performed at the Parent Organization(s)’ current operations will also be evaluated.

The proposal will be evaluated with respect to the effectiveness of the Parent Organization(s)’ strategic planning methods and processes and the degree to which those capabilities are expected to benefit the Laboratory.

The proposal will be evaluated with respect to the strength of the quality management philosophy, effectiveness of initiatives and accomplishments throughout the Parent Organization(s), and the extent to which that quality culture can be expected to assist the Laboratory in its quest for excellence and continuous improvement of its performance.

Therefore, the committee makes the following recommendation to the NNSA:

Recommendation 5. In order to assess the offerors’ competencies in strategic management, the RFPs for the LLNL and LANL competitions should ask offerors to describe their plans for, and demonstrate their capabilities in, the following:

  • Understanding deeply the missions of the laboratory and the NNSA;

  • Managing large, complex S&T organizations whose work ranges from basic, long-term research to near-term technology, project and program management, manufacturing, and the planning, construction, and operation of major user facilities;

  • Developing strategic, staffing, and program plans for the laboratory;

  • Addressing the challenges of work with many safety hazards and associated regulations;

  • Interacting with the DOE and other sponsors of the laboratory’s work;

  • Facilitating technology transfers to other laboratories and the private sector;

  • Managing relationships and interactions with other DOE weapons laboratories;

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
  • Insulating the laboratory from pressures within the contractor’s institution that might be deleterious to the laboratory’s mission;

  • Enhancing the laboratory’s relationship with Congress and the DOE;

  • Managing conflicts of interest in personnel, funding, and project management matters within the framework of federal regulations;17 and

  • Establishing a solid management transition plan for the start of the new contract.

The RFPs should also ask offerors to describe their following attributes:

  • Commitment of top management to governance of the laboratory;

  • Experience within the top executive level with the norms and practices of science;

  • Commitment to national security or other public service;

  • Commitment to intellectual honesty and ethics; and

  • Commitment to developing a culture of excellence in S&T project management.

The competition for the management of LLNL should also ask offerors to describe their capabilities and plans for the completion and operation of NIF, while the competition for the management of LANL should ask offerors to address how they will manage LANL’s major user facility, LANSCE, and manage or restore aging facilities elsewhere on the site.

To help the NNSA judge the responses, the committee makes the following subsidiary recommendations to the NNSA to guide its evaluation of the proposals for the LANL and LLNL M&O contracts as they apply to the leadership of the laboratories:

Recommendation 5a. The NNSA should evaluate the offerors’ capabilities in managing a large, complex S&T organization whose activities range from basic, long-term research to near-term technology, manufacturing, and the operation of user facilities. This evaluation should examine the likely effectiveness of such structural features as the proposed advisory and oversight committees, peer review processes, and the reporting structure. The NNSA should also expect evidence that the top manage-

17  

48CFR970.0905.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

ment in the winning offerors’ organizations is committed to the management contract.

Recommendation 5b. The NNSA should evaluate the offerors’ past accomplishments that required long-term and independent vision, their prior commitment to national service—particularly in the area of national security—and their accomplishments in managing initiatives that push the limits of what is technically feasible (as does the NIF). The NNSA should also evaluate the offerors’ skill and experience in strategic planning, succession planning, transition planning (if needed), and project planning, and its principles for deciding when to invoke formal project management procedures.

Recommendation 5c. The NNSA should evaluate the familiarity of the offerors’ key personnel with the norms and practices of science and engineering, such as openness, good peer review practices, aggressive questioning of fundamental assumptions, accommodation of divergent but technically credible views, and so on. The NNSA should also assess the reputation and recognition of each offeror or its parent organization(s) in the scientific community.

Recommendation 5d. The NNSA should evaluate the offerors’ capabilities in organizing and managing cooperative research endeavors with other organizations, including universities, government laboratories, or corporate entities, as evidence that they can continue the laboratory’s success in assembling a variety of teams to address national defense problems. The NNSA should also evaluate the offerors’ accomplishments in technology transfer.

Recommendation 5e. The NNSA should evaluate the capabilities and reputation of the offerors or their parent organizations in working effectively with the DOE, Congress, other weapons laboratories, and other federal sponsors of the laboratory’s work, including demonstration of ability to manage conflicts of interest where appropriate.

Recommendation 5f. In proposals for the management of LLNL, the NNSA should evaluate the offerors’ capabilities and plans for the completion and operation of NIF and their past experience at operating major user facilities of a similar scale. In proposals for the management of LANL, the NNSA should evaluate the offerors’ capabilities and plans for the restoration and operation of the LANSCE facility and their past experience at operating major user facilities of a similar scale. In proposals for either laboratory, offerors should demonstrate their capabilities in constructing and maintaining world-class technical facilities, in managing and restoring aging facilities, and in managing very-large-scale computing facilities.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

Key Personnel

The top laboratory personnel (the laboratory director and the people who report directly to that level) are responsible for developing and enabling much of the vision and leadership described in the previous sections of this report. In large measure, they set the tone for the quality of the S&T staff and a laboratory’s ability to attract and retain talented individuals. Therefore, the committee makes the following recommendation to the NNSA:

Recommendation 6. The RFPs for the LANL and LLNL competitions should ask offerors to articulate the qualifications, roles, and responsibilities of the key management personnel they intend to bring to the laboratory or retain, relative to the mission of the laboratory. In addition to the key laboratory management personnel, the offeror should describe the structure of the laboratory’s governing board and of the highest-level visiting committee. The active and continuing involvement of the parent organization’s top management must be explicitly promised and the mode of delivering on this promise articulated.

Recommendation 6a. The NNSA should evaluate the offerors’ proposed key personnel according to their experience with managing a broad-based, complex S&T laboratory. In particular, it seemed essential to the committee that the individuals named as the proposed laboratory director and technical associate directors should have a strong understanding of S&T.

Tactical Management: Best Practices

The M&O contractors for LLNL and LANL should be expert in best business practices for project management, integrated safety and security management, human resource management, infrastructure management, and other administrative areas. Excellence in these practices can contribute directly to the scientific quality of a laboratory and the fulfillment of its mission if they are appropriately integrated into the S&T management. For instance, an expert contractor could work with the DOE to implement improvements in project planning, including aspects such as integration of pre-project planning with long-term mission statements; maintenance and improvement of DOE’s project management procedures and practices; and selection of effective and experienced project managers for first-of-a-kind projects. The contractor must also be capable of effective stewardship of the laboratory’s physical infrastructure, because that is an

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

important factor (for reasons of scientific capability, safety, and the appeal of the work environment) in attracting and retaining excellent staff.

The technical work at the laboratories varies from large focused engineering projects to fundamental scientific research in areas of science important to the long-term missions of the laboratories. This range of efforts involves many different approaches to setting priorities, fixing schedules, budgeting resources, and assessing progress, yet it is important that all of this work be of the highest quality. The overall goal of the contractor, with respect to the S&T programs, should be to set high expectations and to establish oversight and support functions that enable each S&T employee to make maximal use of his or her own specialized skills toward achieving the laboratory’s mission. At the same time, the contractor should have experience at managing an organization with many safety hazards and corresponding regulations. Ways to achieve good tactical management include establishing and defining the ground rules and boundaries for individuals and the organization (with broad scopes, clear rules of engagement, clear objectives, and avoidance of unnecessary change) and providing the right support of all kinds, ranging from specialized equipment, expertise in S&T project management, access to information and an appropriate mix of colleagues, adequate levels of skilled supporting personnel, and expertly conceived and executed processes for human resource management, safety, and security. This skill in tactical (day-to-day) management complements, yet differs from, the strategic management skill called for by Recommendation 5. While most tactical management will be in the hands of each laboratory’s managers, the contractor’s excellence in these areas will give it the ability to know when improvements are needed and to assist in their development or adoption.

Therefore, the committee makes the following recommendation to the NNSA:

Recommendation 7. The RFPs for the LLNL and LANL competitions should ask offerors to describe their accomplishments in the execution of processes (“best business practices”) for the day-to-day management of staff, projects, specialized technical facilities, safety, and security. Offerors should also describe any specific plans appropriate to the management of the laboratory in question. The offeror should demonstrate how security will be integrated with the scientific and engineering enterprise to ensure a culture of security awareness and compliance without stifling scientific creativity and performance.

The committee makes the following more specific recommendations to the NNSA to guide its evaluation of the proposals for each laboratory’s M&O contracts as they apply to best business practices:

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

Recommendation 7a. The NNSA should evaluate the offerors’ past successes in managing a large and complex portfolio of varied technical projects and facilities, along with the associated scientists, engineers, and technicians. The evaluation should examine such metrics as milestone completion rates, customer feedback, safety record, rates of staff attrition, staff commendations or disciplinary actions associated with projects, and so on, looking in particular at whether the offerors have achieved an appropriate balance between controls and flexibility. The offerors should also demonstrate how they have achieved the flexibility to change management procedures and policies as changing situations dictate. The NNSA should require evidence that each offeror brings skills and professionalism to these tasks but also recognizes that best business practices are the means to an end, not the end in itself.

Recommendation 7b. The NNSA should evaluate the offerors’ past successes in managing a highly classified technical facility, looking in particular at whether the offerors have achieved an appropriate balance between security and the openness that is necessary for excellent S&T accomplishment. The NNSA should also evaluate the offerors’ competence in managing safety in an environment with diverse hazards.

Recommendation 7c. The NNSA should evaluate the offerors’ past successes in human resource management in support of science and engineering excellence, including recruiting, benefits management, management of support facilities (e.g., a child-care facility), career flexibility, working space, and so on. In particular, the NNSA should examine whether the offerors have achieved effective management without excessive bureaucracy, and whether they have maintained flexibility in their human resource practices. As noted in Recommendation 7a, the NNSA should look for evidence that the offeror brings skills and professionalism to these tasks but also recognizes that the measure of success of such management practices is how well they support and enable the accomplishment of the laboratory’s mission.

Intellectual Leadership

The fundamental responsibility of the scientists and engineers at each laboratory is to perform high-quality R&D that contributes to the mission of the laboratory. The M&O contractors for LLNL and LANL should be accomplished in processes that foster innovation—including stewardship of internal R&D, interactions with the outside technical community, management of intellectual property, development of strategies for technology transfer, and targeted review practices for programs and personnel. As part of this goal, the contractor must maintain an atmosphere of intellectual freedom and integrity, and it must command the trust of the staff

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

so that they feel free to express opinions on relevant matters without fear of reprisal.

As stated in Recommendation 4 in the context of sustaining a world-class S&T workforce, Laboratory-Directed Research and Development funding is a critical tool for the success of the weapons laboratories. LANL and LLNL use LDRD funding to explore new opportunities that might translate into important technologies in the future, and for developing additional fundamental knowledge that underpins their missions. The LDRD program is the only source of funding with the degree of flexibility needed to address long-term, high-risk research not included in the rest of the laboratories’ programs, and an adequate pool of LDRD funding promotes a healthy competition among laboratory staff to identify important research opportunities that should be explored now so as to provide technological options for the future. LDRD funding is also a critical incentive for staff, because it offers them the possibility of following their most promising ideas to fruition, even if there is a high risk of failure. It is essential that basic research funding (including LDRD funding) not be used to support underfunded efforts elsewhere in the laboratory. The LDRD program should, however, be subject to portfolio management; indeed, it is the responsibility of the laboratory director to provide intellectual leadership in investing this “seed corn.”

In order for an M&O contractor to command the respect of LANL and LLNL staff and mesh with the laboratories’ cultures, the contractor must have its own science and engineering strength in areas that overlap the laboratory’s expertise. The caliber of the contractor’s existing staff is important in stimulating laboratory staff, expanding the laboratory’s knowledge base, and providing contacts for recruiting and for review committees. That strength also increases the likelihood that the contractor will be able to manage a high-quality R&D laboratory. For these reasons, the M&O contractor should have technical credibility in most, if not all, of the disciplines at the heart of the laboratory’s work: physics, chemistry, computational science, and materials science. This requirement is a natural complement to the clause in Recommendation 5 that calls for the offerors to have experience at the executive level with the norms and practices of science.

A special aspect of the “norms and practices of science” is that the contractor be committed to, and have skills and contacts for executing, peer review of programs. Peer review, involving outsiders wherever possible, is the best time-tested method for accountability and the assessment of quality in science and engineering research. Such a system is valuable because peers are the only people who can judge whether a novel method or R&D direction is well addressed, and because the system encourages

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

scientists to aim for very high standards because they are being reviewed by the people they most admire.

Therefore, the committee makes the following recommendation to the NNSA:

Recommendation 8. The RFPs for M&O of LLNL and LANL should ask offerors to describe their plans, and demonstrate their ability (track record and reputation), to contribute intellectually to the future success of the laboratory.

The committee makes the following recommendations to the NNSA to guide its evaluation of the proposals for the laboratory M&O contracts as they apply to the ability to foster innovation:

Recommendation 8a. The NNSA should evaluate the offerors’ proposed structuring of the LDRD portfolio—e.g., what fraction would be closely related to program activities—and how they will review LDRD proposals and monitor LDRD work. More broadly, the NNSA should evaluate the offerors’ success in stimulating innovation and developing the resulting ideas, and also examine the offerors’ past history of investing in, and allocating, internal R&D.

Recommendation 8b. The NNSA should evaluate the offerors’ past history of investment in internal staff training, student or postdoctoral programs, reliance on peer review, and other contributions to the intellectual stimulation of their organizations, and the incentives and rewards directed toward their technical staffs. The NNSA should assess the processes used and the skill and integrity with which the offerors have conducted peer reviews and the caliber of the reviewers involved.

Recommendation 8c. The NNSA should assess the value of the scientific and technical attributes brought by the offerors according to the following measures:

  • Scientific and technical reputation of the offerors’ staff, as indicated by peer judgment, size of the R&D operations, number and impact (citations) of publications, and ranking in number of patents and patent citations/referrals;

  • Any specialized facilities operated by the offerors that would be of value to the laboratories’ missions; and

  • The offerors’ success in involving highly qualified scientists and engineers from the broad technical community in research collaborations, review panels, short visits, and special assignments.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

Overall Proposal Evaluation

In conducting its study to identify key management principles for ensuring high scientific quality in world-class weapons and energy R&D, the committee quickly realized that most of management’s functions in complex scientific laboratories such as LANL and LLNL can have key effects on the scientific quality. Thus, Recommendations 2 through 8 above are all essential to the scientific quality at both laboratories, and they also cover a large fraction of the scope of the RFPs. Similarly, the 17 measures (Recommendations 4a-d, 5a-f, 6a, 7a-c, and 8a-c) by which the NNSA should evaluate the quality of the proposals against the goal of maintaining high S&T quality also cover a large fraction of the general duties of management at the laboratories. The committee suggests that the RFP language in response to Recommendations 2 through 8 be collected in one section of each RFP, so that the strength of the resulting proposals relative to S&T concerns will be readily apparent. The committee feels strongly that, given the central role of S&T throughout most functions of LLNL and LANL, the NNSA source selection process should allot two-thirds of its total points for proposal responses that relate to the S&T matters in Recommendations 4a-d, 5a-f, 6a, 7a-c, and 8a-c, assuming the proposals evince solid strength in management of security, finances, and safety, which are essential underpinnings of the S&T work. This emphasis would be consistent with the sentiment of the Blue Ribbon Commission, with weightings during the 1999 competition for management of Oak Ridge National Laboratory, with performance metrics in NASA’s contract with the Jet Propulsion Laboratory, and with weightings used in the award for the United Kingdom’s nuclear weapons laboratory.

CONCLUSION

Although its recommendations apply specifically just to the RFPs for M&O of LANL and LLNL, the committee is emphatic in its view that the RFPs, the review process, the resulting contracts, and the scoring and reward systems in the contracts should all be carefully and clearly aligned with the missions of the two laboratories.

The committee feels strongly that the continuing scientific excellence and productivity of Los Alamos and Lawrence Livermore National Laboratories are critical to the nation’s nuclear deterrence capability, and it believes that the recommendations in this report will both protect the scientific quality of the laboratories and allow for strengthening of their management practices. The committee urges the NNSA and the offering organizations to examine with care the recommendations in this report and act in accordance with the spirit behind them.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×

This page intentionally left blank.

Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 1
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 2
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 3
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 4
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 5
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 6
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 7
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 8
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 9
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 10
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 11
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 12
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 13
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 14
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 15
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 16
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 17
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 18
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 19
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 20
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 21
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 22
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 23
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 24
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 25
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 26
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 27
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 28
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 29
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 30
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 31
Suggested Citation:"Contents of Report." National Research Council. 2004. Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/11009.
×
Page 32
Next: Appendix 1: Biographical Sketches of the Committee Members »
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In recent years, there has been concern about security and operations management at Los Alamos and Lawrence Livermore National Laboratories (LANL and LLNL). As a result, Congress directed the Department of Energy (DOE) to hold open competitions for the management and operations (M&O) contracts for both LANL and LLNL. The quality of the scientific programs, however, did not appear to be a factor in that action, and the DOE's National Nuclear Security Administration (NNSA) wanted to ensure that the contract competitions preserve the high-quality science and engineering currently being performed at the labs. It asked the NRC to recommend how best the NNSA can create meaningful qualification and selection discriminators to help ensure world-class scientific quality is maintained in programs and activities at LANL and LLNL. This report presents those recommendations along with other important factors that should be considered in developing the request for proposals for the upcoming contract competition.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!