Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories EXECUTIVE SUMMARY In recent years, there has been concern in Congress about security andς operations management at Los Alamos National Laboratory (LANL) and Lawrence Livermore National Laboratory (LLNL). As a result of this concern and a general preference for holding regular competitions for ongoing federal contracts, Congress directed the Department of Energy (DOE) to hold open competitions in 2004-2005 for the management and operations (M&O) contracts for both LANL and LLNL, as well as for other DOE national laboratories. (Since the 1940s, the University of California has been the M&O contractor for LANL and, since its inception in 1952, for LLNL.) The execution and accomplishments of the scientific programs1 did not appear to be a factor in the decision to hold a competition for the M&O contracts at LANL and LLNL. In view of that, the National Nuclear Security Administration (NNSA), a semi-autonomous part of the DOE that is responsible for both laboratories, wanted to ensure that the contract competitions preserve the high-quality science and engineering currently being performed there, and so it asked the National Research Council (NRC) to do the following: 1 Throughout this report, “scientific programs” should be interpreted broadly as not only science and engineering R&D, but also any work at the laboratories that requires a combination of technical skill and creativity.
OCR for page 2
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories [I]dentify key management principles for ensuring high scientific quality in world-class weapons and energy R&D and recommend how best the NNSA can create meaningful qualification and selection discriminators to help ensure world-class scientific quality is maintained in programs and activities at LANL and LLNL. The [NRC] will conduct its study with careful attention to the missions of LANL and LLNL and the needs of NNSA, the current situations at LANL and LLNL and operating requirements imposed upon them, the trends in the management of scientific activities at other relevant federal R&D organizations, and the future availability of scientific manpower required at LANL, LLNL, and similar laboratories. The NNSA informed the NRC that it had assigned to others the responsibility for ensuring that security and operations management would be properly addressed by the competitions. To address the study’s charge, the NRC constituted the Committee on Criteria for the Management of Los Alamos and Lawrence Livermore National Laboratories, a group of 15 scientists and engineers with experience in science and technology (S&T) management in a variety of institutions who met during January-April 2004 in Washington, D.C., and at each of the two laboratories, conducted a set of site visits to other DOE laboratories, and gathered additional information. The committee developed the following primary findings and recommendations for the NNSA: Run the competitions for both LANL and LLNL simultaneously. Because of their unique role in nuclear weapons research and development (R&D) and stewardship, LANL and LLNL serve as peer communities for one another. The interplay between these two laboratories is very important, and there is a strong sentiment at the laboratories that their coordination and constructive competition are facilitated by their being managed by the same contractor. The NNSA should hold the M&O competitions for both laboratories simultaneously, allowing offerors to bid on an individual laboratory or on both, and should then evaluate the resulting proposals to ascertain which hold the best promise for ensuring that the laboratories’ programs remain coordinated and that the laboratories continue to serve as peer reviewers of one another’s work. The NNSA should also evaluate its own capabilities for playing that coordinating role in the event that different M&O contractors are selected for LLNL and LANL. Use a single proposal evaluation board assisted by a panel of experts in science and technology management. The committee recommends that the NNSA constitute a single very knowledgeable evaluation board to evaluate the proposals it will receive for the management and operation of LANL and LLNL, to better en-
OCR for page 3
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories able comparison of management ideas from across the entire set of proposals. The committee also recommends that the evaluation board have available, as an added resource, a panel of experts in S&T management to help assess offerors’ responses in areas where additional S&T experience is required and to help formulate questions for the evaluation board to pose to the offerors. Preserve and improve what’s working well. Congressional concerns about security and project management at LLNL and LANL—which are beyond the scope of this study—were not accompanied by criticism of the S&T programs at the two laboratories. Therefore, attempts to address the former should be balanced and integrated with actions to preserve and improve S&T quality. The NNSA also should conduct the competitions in a way that minimizes uncertainty for the laboratories’ staffs, including working toward contractor arrangements that are readily extendible beyond the 5 years stipulated in federal procurement regulations and immediately communicating any ground rules for the competitions that bear on continuation of employment and benefits. Seek contractors that can retain the laboratories’ world-class scientific staffs. The NNSA’s requests for proposals (RFPs) and proposal evaluations should ensure that the laboratories continue to attract, retain, and develop the world-class S&T staffs that are essential to accomplishing the laboratories’ missions. The RFPs and subsequent proposal evaluations should address the aspects under direct control of the contractor (e.g., laboratory facilities and site conditions) as well as the contractor’s contributions to the community that enable the attraction and retention of highly qualified science and engineering staff. Because of LANL’s isolated location, the community aspects should be weighted more heavily in proposals for LANL management. Seek contractors with excellent strategic management capabilities. The RFPs and proposal evaluations should lead to M&O contractor(s) with experience in running a large S&T organization and the ability to create and follow a long-term S&T agenda, and to do so with impeccable integrity and a commitment to national service. Seek contractors that will appoint excellent key personnel to manage the laboratories. The RFPs and resulting proposal evaluations should lead to key management teams for LANL and LLNL, including governing or advisory boards and visiting committees, that are of the highest quality. The credentials of the team members should be comparable to those of individuals in similar positions at other national laboratories.
OCR for page 4
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Seek contractors that are expert at best practices. Effective and efficient business practices facilitate and support the delivery of high-quality S&T work. The RFPs and resulting proposal evaluations should be tailored to find contractors with demonstrated skills in process, financial, and human and physical resource management. The contractors should be able to plan, construct, and operate major facilities and execute effective but efficient security and safety procedures and training programs. Seek contractors that are skilled at stimulating and supporting innovation. The contractors should be effective stewards of their Laboratory-Directed Research and Development (LDRD) funds, as these funds provide one of the principal means for the laboratories to remain on the cutting edge of science and engineering and to recruit and retain outstanding young scientists, and the RFPs and resulting proposal evaluations should identify offerors with the best capabilities in this area. The contractors should also creatively manage interactions with the outside technical community, including foreign nationals, and should provide productive and stimulating processes for managing intellectual property, facilitating technology transfers, and rewarding exceptional contributions by the S&T staff. Seek contractors that can provide intellectual leadership. It is essential to the missions of the laboratories and to the continuance of mission-oriented S&T work of the highest quality that the NNSA’s RFPs and resulting proposal evaluations identify contractors that can lead the laboratories to continued technical excellence. This report provides background for these recommendations and includes additional recommendations to the NNSA about how it can evaluate the proposals that result from these competitions. It makes some suggestions for incentives to include in the contract to increase the likelihood of receiving proposals from high-quality R&D organizations. The report also discusses more generally some of the principles of good R&D management, including the need to strike a balance between managerial control and the flexibility that encourages scientific and technical creativity. INTRODUCTION The U.S. nuclear weapons laboratories—Los Alamos National Laboratory (LANL), Lawrence Livermore National Laboratory (LLNL), and Sandia National Laboratories—are rightly considered to be national treasures. They have excelled for over half a century at creating and sustaining the nation’s nuclear deterrent capabilities. In recent years, their mis-
OCR for page 5
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories sion has turned to stockpile stewardship, encompassing a range of challenges related to ensuring the reliability, safety, and efficacy of the existing stockpile of nuclear weapons, without full-scale testing. Additionally, the laboratories’ scientific and engineering prowess has been of great value to conventional defense, energy research and development (R&D), and other technical enterprises as wide-ranging as high-performance computing, transportation modeling, and genomics. The high science and engineering quality of these laboratories was well known to the authors of this report through their long experience in the science and technology (S&T) enterprise, and that impression is supported by many metrics of R&D quality.2 The success of the weapons laboratories in meeting their weapons missions and in contributing to R&D well beyond that mission is evidence of their scientific depth and breadth, a depth and breadth that allows the laboratories to recognize and apply research developments from a wide range of fields that might be of importance to their missions a decade or more in the future. The committee believes that this broad and sustained base of scientific, engineering, and technological expertise is necessary in order for the nuclear weapons complex to be prepared for contingencies beyond those addressed by the near-term program. In recent years, there has been concern in Congress about security and operations management at LANL and LLNL. As a result of this concern and a general preference for holding regular competitions for ongoing federal contracts—and with the concurrence of an external blue-ribbon commission3 appointed by the Department of Energy (DOE)—Congress directed DOE to hold open competitions in 2004-2005 for the management and operations (M&O) contracts for both LANL and LLNL, as well as for other DOE national laboratories.4 (Since the 1940s, the University of California has been the M&O contractor for LANL and, since its inception in 1952, for LLNL.) It is important to note that the execution and accomplishments of the scientific programs5 did not appear to be a factor 2 For example, LLNL staff authored 1140 peer-reviewed publications in 2003, and LANL produced nearly 1700, while each laboratory received more than 70 patents in that same year. In addition, laboratory staff members have achieved numerous professional recognitions. 3 U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003. 4 Energy and Water Development Appropriations Act for Fiscal Year 2004, PL 108-137, General Provisions, Department of Energy, Section 301. 5 Throughout this report, “scientific programs” should be interpreted broadly as not only science and engineering R&D, but also any work at the laboratories that requires a combination of technical skill and creativity.
OCR for page 6
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories in the recent decision to hold a competition for the M&O contracts at LANL and LLNL. In fact, these scientific programs are widely regarded in the science and engineering community as being world-class. In view of that, the National Nuclear Security Administration (NNSA), a semi-autonomous part of the DOE that is responsible for both laboratories, wanted to ensure that the contract competitions preserve the high-quality science and engineering currently being performed there, and so it asked the National Research Council (NRC) to do the following: [I]dentify key management principles for ensuring high scientific quality in world-class weapons and energy R&D and recommend how best the NNSA can create meaningful qualification and selection discriminators to help ensure world-class scientific quality is maintained in programs and activities at LANL and LLNL. The [NRC] will conduct its study with careful attention to the missions of LANL and LLNL and the needs of NNSA, the current situations at LANL and LLNL and operating requirements imposed upon them, the trends in the management of scientific activities at other relevant federal R&D organizations, and the future availability of scientific manpower required at LANL, LLNL, and similar laboratories. The NNSA informed the NRC that it had assigned to others the responsibility for ensuring that security and operations management would be properly addressed by the competitions. To address the study’s charge, the NRC constituted the Committee on Criteria for the Management of Los Alamos and Lawrence Livermore National Laboratories,6 a group of 15 scientists and engineers with experience in S&T management in a variety of institutions who met during January-April 2004 in Washington, D.C., and at each of the two laboratories, conducted a set of site visits to other DOE laboratories, and gathered additional information.7 THE SPECIAL CHALLENGES OF MANAGEMENT AT LANL AND LLNL The successful accomplishment of the LANL and LLNL missions requires that DOE, the contractor(s), and the laboratories strike the right balance between the needs to (a) meet mission milestones in the near term, (b) maintain the staff and the creative scientific culture necessary to meet the unforeseen challenges of the future, and (c) be able to recognize and pursue novel or breakthrough advances. There is also a widespread perception that these laboratories need to improve their operations management and security—issues that are important but are outside the scope of 6 See Appendix 1. 7 See Appendix 2.
OCR for page 7
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories this study—and the committee believes that these goals can and should be pursued in an integrated fashion with the goal of preserving, and even improving, the S&T. Because this study was not asked to address the security and operations management goals, this report discusses them only where they are connected to scientific quality. The NNSA, in crafting requests for proposals (RFPs) for the management of LLNL and LANL, will have to consider the key aspects of operations management and security and balance those aspects with the considerations about S&T management raised in this report. An overemphasis on detailed program planning and management, or on process definition and management, is often counterproductive to stimulating the best scientific work. Trust, and an expectation that the staff will perform at a high level of quality, create the incentives for excellent accomplishments in science and engineering. Similarly, systems for security inevitably entail risks, and pro forma efforts to eliminate, rather than manage, those risks inherently detract from the quality and productivity of the S&T workforce. It is essential that the M&O contractors for LANL and LLNL be adept at managing within the natural ambiguity of R&D—the inherent uncertainty about how programmatic thrusts will actually play out—and that they have the capacity to respond to unanticipated S&T opportunities. Fortunately, the conditions that attract and retain an excellent S&T staff are the same conditions that facilitate their best progress toward mission goals: both rely critically on an environment that encourages self-motivated work targeted toward the mission and that provides connections to a wide range of external scientists and engineers. Thus, high scientific quality at LANL and LLNL is dependent on each laboratory’s continuing to attract and retain an excellent scientific staff, and each laboratory’s success at attracting and retaining such a staff requires a laboratory environment that enables the staff to excel. The contractor also must serve as a buffer between the laboratories’ staffs and pressures to micromanage the laboratories’ operations. Many experts who have examined the working of the national laboratories—most notably the 1994-1995 Galvin Commission,8 and more recently the DOE’s Blue Ribbon Commission on the Use of Competitive Procedures for Department of Energy Labs9—have decried the degree of micromanagement to which these laboratories are subjected and have called for a 8 Task Force on Alternative Futures for the Department of Energy Laboratories, Robert Galvin, chair. 9 U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003, p. 13.
OCR for page 8
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories reestablishment of the spirit of trust that formerly made the government-owned/contractor-operated (GOCO) model of laboratory management successful. The M&O contractors for LANL and LLNL can help address this problem by working to restore and improve the level of trust among the DOE, Congress, and the laboratories’ staffs. Among other things, each laboratory’s management and staff should be given some flexibility and latitude in the expenditure of R&D funding. Especially important are the Laboratory-Directed Research and Development (LDRD) funds, which should continue to be deployed largely at the discretion of the LANL and LLNL directors. While the DOE sets the overall missions and oversees the laboratories’ programs, the laboratories’ directors and staffs should be given leading roles in defining and executing the programs of their laboratories because they are in the best position to develop the very-long-term view that is necessary given the magnitude of the challenges faced by these institutions. ENABLING EFFECTIVE COMPETITIONS FOR LANL AND LLNL Retaining a Healthy LANL-LLNL Interplay During the course of its study, the committee became acutely aware of the vital interplay between LANL and LLNL. The need for two centers of competence in nuclear design and management was recognized by the government more than 50 years ago. Nuclear stewardship and the scientific aspects of national security have been and remain too important to leave to any single group, no matter how competent. Absent the normal scrutiny of outside peers, such a deliberate overlapping of missions is desirable, and it has been very successful over the decades in enabling the two laboratories to provide a healthy check of one another’s work. The spirit of competition among equals that now exists encourages the science and engineering staffs of both laboratories to aim for high achievements and ensures that the best S&T efforts are brought to bear on the main missions. There is a strong sentiment at LLNL and LANL that coordination and constructive competition are facilitated by the laboratories being managed by the same contractor. 10 The committee notes that the following issues could arise if LANL and LLNL were managed by different contractors: 10 See, e.g., Harold M. Agnew et al., personal communication from 10 former LLNL and LANL directors to Secretary of Energy Spencer Abraham, December 8, 2003. The letter notes the peer review the laboratories provide one another, the need to cooperate and share information, and the
OCR for page 9
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Coordination between the laboratories might deteriorate; Communication channels would have to be more explicitly identified and monitored; Exchange of scientific personnel and transfer of programs between the laboratories would likely be inhibited; The NNSA would have to provide a non-trivial coordination function, which would complicate the governance of these weapons laboratories and change the nature of the NNSA’s involvement, while risking duplicative oversight; and Different contractors, particularly different kinds of contractors (e.g., one for-profit and one non-profit), might face structural incongruities with respect to regulations or guidelines for such issues as conflict of interest, personnel management, or intellectual property, giving rise to reduced scientific interchange between the laboratories. The committee cannot assess the magnitude of these risks, and it suspects that proposals for the two M&O contracts could identify promising mechanisms for continuing and perhaps improving the effective interplay between LANL and LLNL. In order for the NNSA to judge most effectively the merits and risks of various combinations of M&O contractors, however, the committee concluded that the NNSA must see those proposals at the same time. Therefore, to provide for that option, the committee makes the following recommendation: Recommendation 1. The NNSA should run simultaneous competitions, with the same proposal evaluation board, for the management of LANL and LLNL, in which offerors could make proposals for the management of either or both laboratories, and the RFPs should ask all offerors to discuss how they would effect coordination and constructive competition between LANL and LLNL. The prospects for continuing that constructive competition could then be given appropriate consideration in the NNSA’s decision about which contractor(s) to select. The NNSA should also evaluate its own capabilities for playing the coordinating role if LANL and LLNL were managed by different contractors and should factor that evaluation into its contractor selection decisions. With this recommendation, the committee disagrees with the Blue Ribbon Commission on the Use of Competitive Procedures for the De value of staff interchanges, and concludes: “In our experience all of these are difficult under a single contractor and would be greatly exacerbated with separate contractors.” The letter also asserts that there would be “great value” in competing the contracts simultaneously.
OCR for page 10
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories partment of Energy Labs, which specifically recommended competing these two contracts at different times because each competition “must welcome all interested and qualified bidders to participate.” 11 The committee does not believe that simultaneous (but separate) competitions would hinder such participation. While the committee does not refute the Blue Ribbon Commission’s conclusion that “there [is] little need to specifically require that the same M&O contractor manage the two laboratories,” the possible problems that could arise should LLNL and LANL end up with different contractors are significant enough that it is prudent to run simultaneous competitions in order to evaluate how well those risks can be managed. The Blue Ribbon Commission’s argument that Sandia National Laboratories (SNL) successfully collaborates with LANL and LLNL in spite of having a different M&O contractor is not completely convincing, because the mission overlaps between SNL and LANL and between SNL and LLNL are not nearly as striking as the overlap between LLNL and LANL programs. Contract Incentives The committee believes that a critical element in ensuring that high-quality science and technology are maintained at LANL and LLNL as their management contracts are competed is that the NNSA receive proposals from the most highly qualified organizations. Organizations are motivated to manage a national laboratory for a variety of relatively intangible reasons, such as a sense of national service, prestige, and access to talented individuals and relevant technology; the management fee is probably not the primary incentive nor, the committee believes, should it be. There is a concern among those familiar with DOE national laboratories that the responsibility of managing a weapons laboratory is of only marginal appeal to potential contractors because of the many possible liabilities and risks and the micromanagement in evidence at many laboratories. Therefore, to ensure that qualified potential contractors will expend the substantial effort involved in developing a proposal, the NNSA should seek and articulate non-financial incentives to be associated with these contracts. The following are probably the most important of these incentives: The contracts could clarify, and expand if possible, indemnification against the consequences of laboratory operations (in the ab- 11 U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for the Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003, p. 23.
OCR for page 11
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories sence of wrongdoing by the contractor) to the full extent allowed by law and federal regulations. Qualified contractors simply will not put their entire assets and/or reputation at risk for the sake of the small financial reward and the prestige value of managing a national laboratory. The committee does not believe that DOE should require contractors to accept lesser fees in exchange for increased indemnification. The contracts could facilitate the contractors’ use of their best practices (for business practice, risk management, human resources, and so on) as the law permits, and not automatically constrain the contractors to follow DOE practices. Each contract should be formulated in accordance with the principle of performance-based results rather than prescriptive requirements for how the work should be done. Such an approach allows for maximizing the capabilities of the S&T staff and directs the focus of DOE/NNSA oversight toward the significant issues requiring such attention. The contracts should make use of the DOE’s procurement provisions12 that allow it to approve 10-year M&O contracts, with 5-year option terms, in order to create an incentive for long-term commitments appropriate to the laboratories’ missions and to make the up-front investment in proposal preparation less of a disincentive to some potential offerors. The committee’s concern about the use of 5-year contracts for the management of LANL and LLNL echoes that expressed in the recent draft report from the Blue Ribbon Commission on the Use of Competitive Procedures for the Department of Energy Labs, which suggested a framework for retaining an incumbent contractor for as long as 20 years, as long as its performance meets or exceeds DOE expectations.13 A 5-year contract with a simple mechanism for a 5-year extension, or the use of a rolling 5-year contract in the absence of underperformance by the contractor, would seem to be one such desirable mechanism. This approach would also be desirable because it would avoid the disruptions noted below in the section titled “Preserve and Improve What’s Working Well.” As an example of this kind of practice, NASA’s current contract with the California Institute of Technology for the management of the Jet Propulsion Laboratory allows 12 Acquisition Letter 1996-09, Attachment B. 13 U.S. Department of Energy, Blue Ribbon Commission on the Use of Competitive Procedures for the Department of Energy Labs, “Competing the Management and Operations Contracts for DOE’s National Laboratories,” public draft, November 24, 2003, p. 20.
OCR for page 22
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories the common temptation in times of budget stress to cut back on very basic and speculative research efforts without obvious near-term application. The M&O contractor(s) must be capable of playing a strong leadership role so that the DOE, Congress, and the laboratories’ customers are confident that the laboratories are carrying out their mission without compromising national security and are applying sound financial and project management approaches. Furthermore, the contractor(s) must shield the staffs from any deleterious effects that might arise from the complex interaction among these parties. The contractor(s) must also be seen as having total integrity and enough independence so that the laboratory directors can credibly carry out their statutory obligation to recommend annually to the secretary of energy the certification (or non-certification) of each nuclear weapons system based solely on their laboratory’s technical assessment. This capability to provide objective and authoritative advice to the secretary of energy is a vital part of the technical mission of the weapons laboratories, and its successful accomplishment relies on a marriage of technical competence with very high levels of real and perceived integrity. The contractor(s) cannot be influenced, or be perceived to be influenced, by political pressure or by the goals of other parts of their organization or parent organizations. A spirit of intellectual independence within the laboratories is also essential because of the synergistic relationship between LANL and LLNL, wherein each has a “devil’s advocate” role to play to challenge and stimulate the other’s work. A section from the 1992 RFP for the management of Sandia National Laboratories, on “Contribution of Parent Organization(s),” is an excellent statement of the responsibility of the M&O contractor. After revision to reflect the S&T foundations of the LANL and LLNL missions, it might be suitable for wholesale incorporation into the RFPs for those laboratories: The proposal will be evaluated with respect to the Parent Organization(s)’ (1) strength of ethics, integrity, and accountability, particularly as these qualities relate to stewardship of the nation’s nuclear weapons, (2) strength of technological, engineering, and managerial leadership and reputation, both nationally and internationally, and (3) the value of associated benefits to be derived through the association of the Laboratory with the Parent Organization(s). The proposal will be evaluated with respect to the Parent Organization(s)’ depth of commitment, strength of organizational structure and nature of responsibilities, and how these are expected to benefit the operation of the Laboratory. Emphasis will be placed on the value added by the Parent Organization(s) and any other major suborganization as well as the rationale for its involvement and the anticipated benefits to accrue to the Laboratory. The roles and projected benefits of any boards and committees proposed to support operation of the Laboratory will be evaluated, including those providing executive management
OCR for page 23
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories and guidance. The proposal will also be evaluated with respect to the significant contributions and the stature of the Parent Organization(s)’ major contributing personnel (i.e., the personnel in those Parent Organization(s) as defined by the Offeror and not to be confused with Laboratory Key Personnel which are identified by DOE in Section L, Attachment No. 1). In addition, the value added and appropriateness of the level of the Parent Organization(s)’ proposed oversight efforts of the Laboratory’s activities will be evaluated, including Programmatic Activities, Facilities Management, Site Services and Business Management. The effectiveness of how oversight is currently being performed at the Parent Organization(s)’ current operations will also be evaluated. The proposal will be evaluated with respect to the effectiveness of the Parent Organization(s)’ strategic planning methods and processes and the degree to which those capabilities are expected to benefit the Laboratory. The proposal will be evaluated with respect to the strength of the quality management philosophy, effectiveness of initiatives and accomplishments throughout the Parent Organization(s), and the extent to which that quality culture can be expected to assist the Laboratory in its quest for excellence and continuous improvement of its performance. Therefore, the committee makes the following recommendation to the NNSA: Recommendation 5. In order to assess the offerors’ competencies in strategic management, the RFPs for the LLNL and LANL competitions should ask offerors to describe their plans for, and demonstrate their capabilities in, the following: Understanding deeply the missions of the laboratory and the NNSA; Managing large, complex S&T organizations whose work ranges from basic, long-term research to near-term technology, project and program management, manufacturing, and the planning, construction, and operation of major user facilities; Developing strategic, staffing, and program plans for the laboratory; Addressing the challenges of work with many safety hazards and associated regulations; Interacting with the DOE and other sponsors of the laboratory’s work; Facilitating technology transfers to other laboratories and the private sector; Managing relationships and interactions with other DOE weapons laboratories;
OCR for page 24
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Insulating the laboratory from pressures within the contractor’s institution that might be deleterious to the laboratory’s mission; Enhancing the laboratory’s relationship with Congress and the DOE; Managing conflicts of interest in personnel, funding, and project management matters within the framework of federal regulations;17 and Establishing a solid management transition plan for the start of the new contract. The RFPs should also ask offerors to describe their following attributes: Commitment of top management to governance of the laboratory; Experience within the top executive level with the norms and practices of science; Commitment to national security or other public service; Commitment to intellectual honesty and ethics; and Commitment to developing a culture of excellence in S&T project management. The competition for the management of LLNL should also ask offerors to describe their capabilities and plans for the completion and operation of NIF, while the competition for the management of LANL should ask offerors to address how they will manage LANL’s major user facility, LANSCE, and manage or restore aging facilities elsewhere on the site. To help the NNSA judge the responses, the committee makes the following subsidiary recommendations to the NNSA to guide its evaluation of the proposals for the LANL and LLNL M&O contracts as they apply to the leadership of the laboratories: Recommendation 5a. The NNSA should evaluate the offerors’ capabilities in managing a large, complex S&T organization whose activities range from basic, long-term research to near-term technology, manufacturing, and the operation of user facilities. This evaluation should examine the likely effectiveness of such structural features as the proposed advisory and oversight committees, peer review processes, and the reporting structure. The NNSA should also expect evidence that the top manage- 17 48CFR970.0905.
OCR for page 25
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories ment in the winning offerors’ organizations is committed to the management contract. Recommendation 5b. The NNSA should evaluate the offerors’ past accomplishments that required long-term and independent vision, their prior commitment to national service—particularly in the area of national security—and their accomplishments in managing initiatives that push the limits of what is technically feasible (as does the NIF). The NNSA should also evaluate the offerors’ skill and experience in strategic planning, succession planning, transition planning (if needed), and project planning, and its principles for deciding when to invoke formal project management procedures. Recommendation 5c. The NNSA should evaluate the familiarity of the offerors’ key personnel with the norms and practices of science and engineering, such as openness, good peer review practices, aggressive questioning of fundamental assumptions, accommodation of divergent but technically credible views, and so on. The NNSA should also assess the reputation and recognition of each offeror or its parent organization(s) in the scientific community. Recommendation 5d. The NNSA should evaluate the offerors’ capabilities in organizing and managing cooperative research endeavors with other organizations, including universities, government laboratories, or corporate entities, as evidence that they can continue the laboratory’s success in assembling a variety of teams to address national defense problems. The NNSA should also evaluate the offerors’ accomplishments in technology transfer. Recommendation 5e. The NNSA should evaluate the capabilities and reputation of the offerors or their parent organizations in working effectively with the DOE, Congress, other weapons laboratories, and other federal sponsors of the laboratory’s work, including demonstration of ability to manage conflicts of interest where appropriate. Recommendation 5f. In proposals for the management of LLNL, the NNSA should evaluate the offerors’ capabilities and plans for the completion and operation of NIF and their past experience at operating major user facilities of a similar scale. In proposals for the management of LANL, the NNSA should evaluate the offerors’ capabilities and plans for the restoration and operation of the LANSCE facility and their past experience at operating major user facilities of a similar scale. In proposals for either laboratory, offerors should demonstrate their capabilities in constructing and maintaining world-class technical facilities, in managing and restoring aging facilities, and in managing very-large-scale computing facilities.
OCR for page 26
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Key Personnel The top laboratory personnel (the laboratory director and the people who report directly to that level) are responsible for developing and enabling much of the vision and leadership described in the previous sections of this report. In large measure, they set the tone for the quality of the S&T staff and a laboratory’s ability to attract and retain talented individuals. Therefore, the committee makes the following recommendation to the NNSA: Recommendation 6. The RFPs for the LANL and LLNL competitions should ask offerors to articulate the qualifications, roles, and responsibilities of the key management personnel they intend to bring to the laboratory or retain, relative to the mission of the laboratory. In addition to the key laboratory management personnel, the offeror should describe the structure of the laboratory’s governing board and of the highest-level visiting committee. The active and continuing involvement of the parent organization’s top management must be explicitly promised and the mode of delivering on this promise articulated. Recommendation 6a. The NNSA should evaluate the offerors’ proposed key personnel according to their experience with managing a broad-based, complex S&T laboratory. In particular, it seemed essential to the committee that the individuals named as the proposed laboratory director and technical associate directors should have a strong understanding of S&T. Tactical Management: Best Practices The M&O contractors for LLNL and LANL should be expert in best business practices for project management, integrated safety and security management, human resource management, infrastructure management, and other administrative areas. Excellence in these practices can contribute directly to the scientific quality of a laboratory and the fulfillment of its mission if they are appropriately integrated into the S&T management. For instance, an expert contractor could work with the DOE to implement improvements in project planning, including aspects such as integration of pre-project planning with long-term mission statements; maintenance and improvement of DOE’s project management procedures and practices; and selection of effective and experienced project managers for first-of-a-kind projects. The contractor must also be capable of effective stewardship of the laboratory’s physical infrastructure, because that is an
OCR for page 27
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories important factor (for reasons of scientific capability, safety, and the appeal of the work environment) in attracting and retaining excellent staff. The technical work at the laboratories varies from large focused engineering projects to fundamental scientific research in areas of science important to the long-term missions of the laboratories. This range of efforts involves many different approaches to setting priorities, fixing schedules, budgeting resources, and assessing progress, yet it is important that all of this work be of the highest quality. The overall goal of the contractor, with respect to the S&T programs, should be to set high expectations and to establish oversight and support functions that enable each S&T employee to make maximal use of his or her own specialized skills toward achieving the laboratory’s mission. At the same time, the contractor should have experience at managing an organization with many safety hazards and corresponding regulations. Ways to achieve good tactical management include establishing and defining the ground rules and boundaries for individuals and the organization (with broad scopes, clear rules of engagement, clear objectives, and avoidance of unnecessary change) and providing the right support of all kinds, ranging from specialized equipment, expertise in S&T project management, access to information and an appropriate mix of colleagues, adequate levels of skilled supporting personnel, and expertly conceived and executed processes for human resource management, safety, and security. This skill in tactical (day-to-day) management complements, yet differs from, the strategic management skill called for by Recommendation 5. While most tactical management will be in the hands of each laboratory’s managers, the contractor’s excellence in these areas will give it the ability to know when improvements are needed and to assist in their development or adoption. Therefore, the committee makes the following recommendation to the NNSA: Recommendation 7. The RFPs for the LLNL and LANL competitions should ask offerors to describe their accomplishments in the execution of processes (“best business practices”) for the day-to-day management of staff, projects, specialized technical facilities, safety, and security. Offerors should also describe any specific plans appropriate to the management of the laboratory in question. The offeror should demonstrate how security will be integrated with the scientific and engineering enterprise to ensure a culture of security awareness and compliance without stifling scientific creativity and performance. The committee makes the following more specific recommendations to the NNSA to guide its evaluation of the proposals for each laboratory’s M&O contracts as they apply to best business practices:
OCR for page 28
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Recommendation 7a. The NNSA should evaluate the offerors’ past successes in managing a large and complex portfolio of varied technical projects and facilities, along with the associated scientists, engineers, and technicians. The evaluation should examine such metrics as milestone completion rates, customer feedback, safety record, rates of staff attrition, staff commendations or disciplinary actions associated with projects, and so on, looking in particular at whether the offerors have achieved an appropriate balance between controls and flexibility. The offerors should also demonstrate how they have achieved the flexibility to change management procedures and policies as changing situations dictate. The NNSA should require evidence that each offeror brings skills and professionalism to these tasks but also recognizes that best business practices are the means to an end, not the end in itself. Recommendation 7b. The NNSA should evaluate the offerors’ past successes in managing a highly classified technical facility, looking in particular at whether the offerors have achieved an appropriate balance between security and the openness that is necessary for excellent S&T accomplishment. The NNSA should also evaluate the offerors’ competence in managing safety in an environment with diverse hazards. Recommendation 7c. The NNSA should evaluate the offerors’ past successes in human resource management in support of science and engineering excellence, including recruiting, benefits management, management of support facilities (e.g., a child-care facility), career flexibility, working space, and so on. In particular, the NNSA should examine whether the offerors have achieved effective management without excessive bureaucracy, and whether they have maintained flexibility in their human resource practices. As noted in Recommendation 7a, the NNSA should look for evidence that the offeror brings skills and professionalism to these tasks but also recognizes that the measure of success of such management practices is how well they support and enable the accomplishment of the laboratory’s mission. Intellectual Leadership The fundamental responsibility of the scientists and engineers at each laboratory is to perform high-quality R&D that contributes to the mission of the laboratory. The M&O contractors for LLNL and LANL should be accomplished in processes that foster innovation—including stewardship of internal R&D, interactions with the outside technical community, management of intellectual property, development of strategies for technology transfer, and targeted review practices for programs and personnel. As part of this goal, the contractor must maintain an atmosphere of intellectual freedom and integrity, and it must command the trust of the staff
OCR for page 29
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories so that they feel free to express opinions on relevant matters without fear of reprisal. As stated in Recommendation 4 in the context of sustaining a world-class S&T workforce, Laboratory-Directed Research and Development funding is a critical tool for the success of the weapons laboratories. LANL and LLNL use LDRD funding to explore new opportunities that might translate into important technologies in the future, and for developing additional fundamental knowledge that underpins their missions. The LDRD program is the only source of funding with the degree of flexibility needed to address long-term, high-risk research not included in the rest of the laboratories’ programs, and an adequate pool of LDRD funding promotes a healthy competition among laboratory staff to identify important research opportunities that should be explored now so as to provide technological options for the future. LDRD funding is also a critical incentive for staff, because it offers them the possibility of following their most promising ideas to fruition, even if there is a high risk of failure. It is essential that basic research funding (including LDRD funding) not be used to support underfunded efforts elsewhere in the laboratory. The LDRD program should, however, be subject to portfolio management; indeed, it is the responsibility of the laboratory director to provide intellectual leadership in investing this “seed corn.” In order for an M&O contractor to command the respect of LANL and LLNL staff and mesh with the laboratories’ cultures, the contractor must have its own science and engineering strength in areas that overlap the laboratory’s expertise. The caliber of the contractor’s existing staff is important in stimulating laboratory staff, expanding the laboratory’s knowledge base, and providing contacts for recruiting and for review committees. That strength also increases the likelihood that the contractor will be able to manage a high-quality R&D laboratory. For these reasons, the M&O contractor should have technical credibility in most, if not all, of the disciplines at the heart of the laboratory’s work: physics, chemistry, computational science, and materials science. This requirement is a natural complement to the clause in Recommendation 5 that calls for the offerors to have experience at the executive level with the norms and practices of science. A special aspect of the “norms and practices of science” is that the contractor be committed to, and have skills and contacts for executing, peer review of programs. Peer review, involving outsiders wherever possible, is the best time-tested method for accountability and the assessment of quality in science and engineering research. Such a system is valuable because peers are the only people who can judge whether a novel method or R&D direction is well addressed, and because the system encourages
OCR for page 30
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories scientists to aim for very high standards because they are being reviewed by the people they most admire. Therefore, the committee makes the following recommendation to the NNSA: Recommendation 8. The RFPs for M&O of LLNL and LANL should ask offerors to describe their plans, and demonstrate their ability (track record and reputation), to contribute intellectually to the future success of the laboratory. The committee makes the following recommendations to the NNSA to guide its evaluation of the proposals for the laboratory M&O contracts as they apply to the ability to foster innovation: Recommendation 8a. The NNSA should evaluate the offerors’ proposed structuring of the LDRD portfolio—e.g., what fraction would be closely related to program activities—and how they will review LDRD proposals and monitor LDRD work. More broadly, the NNSA should evaluate the offerors’ success in stimulating innovation and developing the resulting ideas, and also examine the offerors’ past history of investing in, and allocating, internal R&D. Recommendation 8b. The NNSA should evaluate the offerors’ past history of investment in internal staff training, student or postdoctoral programs, reliance on peer review, and other contributions to the intellectual stimulation of their organizations, and the incentives and rewards directed toward their technical staffs. The NNSA should assess the processes used and the skill and integrity with which the offerors have conducted peer reviews and the caliber of the reviewers involved. Recommendation 8c. The NNSA should assess the value of the scientific and technical attributes brought by the offerors according to the following measures: Scientific and technical reputation of the offerors’ staff, as indicated by peer judgment, size of the R&D operations, number and impact (citations) of publications, and ranking in number of patents and patent citations/referrals; Any specialized facilities operated by the offerors that would be of value to the laboratories’ missions; and The offerors’ success in involving highly qualified scientists and engineers from the broad technical community in research collaborations, review panels, short visits, and special assignments.
OCR for page 31
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories Overall Proposal Evaluation In conducting its study to identify key management principles for ensuring high scientific quality in world-class weapons and energy R&D, the committee quickly realized that most of management’s functions in complex scientific laboratories such as LANL and LLNL can have key effects on the scientific quality. Thus, Recommendations 2 through 8 above are all essential to the scientific quality at both laboratories, and they also cover a large fraction of the scope of the RFPs. Similarly, the 17 measures (Recommendations 4a-d, 5a-f, 6a, 7a-c, and 8a-c) by which the NNSA should evaluate the quality of the proposals against the goal of maintaining high S&T quality also cover a large fraction of the general duties of management at the laboratories. The committee suggests that the RFP language in response to Recommendations 2 through 8 be collected in one section of each RFP, so that the strength of the resulting proposals relative to S&T concerns will be readily apparent. The committee feels strongly that, given the central role of S&T throughout most functions of LLNL and LANL, the NNSA source selection process should allot two-thirds of its total points for proposal responses that relate to the S&T matters in Recommendations 4a-d, 5a-f, 6a, 7a-c, and 8a-c, assuming the proposals evince solid strength in management of security, finances, and safety, which are essential underpinnings of the S&T work. This emphasis would be consistent with the sentiment of the Blue Ribbon Commission, with weightings during the 1999 competition for management of Oak Ridge National Laboratory, with performance metrics in NASA’s contract with the Jet Propulsion Laboratory, and with weightings used in the award for the United Kingdom’s nuclear weapons laboratory. CONCLUSION Although its recommendations apply specifically just to the RFPs for M&O of LANL and LLNL, the committee is emphatic in its view that the RFPs, the review process, the resulting contracts, and the scoring and reward systems in the contracts should all be carefully and clearly aligned with the missions of the two laboratories. The committee feels strongly that the continuing scientific excellence and productivity of Los Alamos and Lawrence Livermore National Laboratories are critical to the nation’s nuclear deterrence capability, and it believes that the recommendations in this report will both protect the scientific quality of the laboratories and allow for strengthening of their management practices. The committee urges the NNSA and the offering organizations to examine with care the recommendations in this report and act in accordance with the spirit behind them.
OCR for page 32
Maintaining High Scientific Quality at Los Alamos and Lawrence Livermore National Laboratories This page intentionally left blank.
Representative terms from entire chapter: