E

What the Laboratories’ Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering

For each laboratory management corporation, the contract is the primary document governing its relationship with the government. Each contract includes a statement of work and an annual Performance Evaluation Plan (PEP), which is included in the contract as a contract modification. Other modifications are also included as agreed between the government and the contractor. Other important documents that affect the conduct of science and engineering are the Contract Management Plan, the Parent Organization Oversight Plan, and the annual Performance Evaluation Review (PER), which is based on the PEP.

Although they have important specific differences, the contracts governing SNL, LANL, and LLNL are generally similar in form and content. In general, each specifies: (1) what the laboratory is and does; (2) what the management corporation is committed to do; (3) what work is to be done at the laboratory (the “statement of work,” or SOW); and (4) how that work is to be reviewed, evaluated, and rewarded. The contracts also specify certain rights and responsibilities of specific government offices to task and/or oversee the work. The SOWs, while lengthy, are basically general. The contracts do not assign specific program tasks. The following—from the LLNL contract—is representative:1

Inasmuch as the assigned missions of the Lawrence Livermore National Laboratory (Laboratory) are dynamic, this Statement of Work (SOW) is not intended to be all-inclusive or restrictive, but is intended to provide a broad framework and general scope of the work to be performed at the Laboratory. This SOW does not represent a commitment to, or imply funding for, specific projects or programs. The National Nuclear Security Administration (NNSA) and Department of Energy (DOE) work requirements are developed through strategic planning and program plans.

While this quoted paragraph refers primarily to programmatic work, the contracts make clear and specific that the laboratory management is committed to other duties, including operational and administrative tasks. Indeed, these management tasks constitute a large part of the work against which the management corporations are evaluated.

The performance evaluation process determines how successful the management has been for the period being evaluated (usually a year), and the contract specifies how success is rewarded. The rewards consist of an award fee (incentive fee) and an award term (i.e., an extension of the temporal term of the contract without need for competition). Independent of the review, the management corporation also receives an annual fixed fee (specified in the contract) and payment for allowable expenses associated with running the management corporation (much of which is associated with paying officials of the various boards and advisory groups).

The incentive fees and fixed fees vary across the laboratories and by year. Table E.1 displays approximate values for comparison.

_______________________________

1 See Modification No. M003 Supplemental Agreement to Contract No. DE-AC52-07NA27344, titled “Part III, Section J, Appendix B-Statement of Work” from the LLNL contract for the Statement of Work in full, available at http://www.llnsllc.com/contract/docs/AppendB_mod53_012209.pdf.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 57
E What the Laboratories’ Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering For each laboratory management corporation, the contract is the primary document governing its relationship with the government. Each contract includes a statement of work and an annual Performance Evaluation Plan (PEP), which is included in the contract as a contract modification. Other modifications are also included as agreed between the government and the contractor. Other important documents that affect the conduct of science and engineering are the Contract Management Plan, the Parent Organization Oversight Plan, and the annual Performance Evaluation Review (PER), which is based on the PEP. Although they have important specific differences, the contracts governing SNL, LANL, and LLNL are generally similar in form and content. In general, each specifies: (1) what the laboratory is and does; (2) what the management corporation is committed to do; (3) what work is to be done at the laboratory (the “statement of work,” or SOW); and (4) how that work is to be reviewed, evaluated, and rewarded. The contracts also specify certain rights and responsibilities of specific government offices to task and/or oversee the work. The SOWs, while lengthy, are basically general. The contracts do not assign specific program tasks. The following—from the LLNL contract—is representative: 1 Inasmuch as the assigned missions of the Lawrence Livermore National Laboratory (Laboratory) are dynamic, this Statement of Work (SOW) is not intended to be all-inclusive or restrictive, but is intended to provide a broad framework and general scope of the work to be performed at the Laboratory. This SOW does not represent a commitment to, or imply funding for, specific projects or programs. The National Nuclear Security Administration (NNSA) and Department of Energy (DOE) work requirements are developed through strategic planning and program plans. While this quoted paragraph refers primarily to programmatic work, the contracts make clear and specific that the laboratory management is committed to other duties, including operational and administrative tasks. Indeed, these management tasks constitute a large part of the work against which the management corporations are evaluated. The performance evaluation process determines how successful the management has been for the period being evaluated (usually a year), and the contract specifies how success is rewarded. The rewards consist of an award fee (incentive fee) and an award term (i.e., an extension of the temporal term of the contract without need for competition). Independent of the review, the management corporation also receives an annual fixed fee (specified in the contract) and payment for allowable expenses associated with running the management corporation (much of which is associated with paying officials of the various boards and advisory groups). The incentive fees and fixed fees vary across the laboratories and by year. Table E.1 displays approximate values for comparison. 1 See Modification No. M003 Supplemental Agreement to Contract No. DE-AC52-07NA27344, titled “Part III, Section J, Appendix B-Statement of Work” from the LLNL contract for the Statement of Work in full, available at http://www.llnsllc.com/contract/docs/AppendB_mod53_012209.pdf. 57

OCR for page 57
TABLE E.1 Approximate Annual Fee Structures (in millions of dollars) Laboratory Fixed Fee Maximum Incentive Fee SNL 16 9-10 LANL 22 52 LLNL 12.5 29.5 Most ( ≥90 percent) of the incentive fee is awarded when a laboratory successfully executes tasks specified in its annual PEP so that the amount of incentive fee that is really “in play” in any evaluation process is in reality a few million dollars (out of a total annual laboratory budget in excess of $1 billion). All three contracts state that science and engineering are major parts of what the laboratories do. The LLNL contract is perhaps most explicit in this regard, the SNL contract least so. The following excerpt from the LLNL contract also appears verbatim in the LANL contract (but with a different identifying number): H-36 INTELLECTUAL AND SCIENTIFIC FREEDOM 2 (a) The Parties recognize the importance of fostering an atmosphere at the Laboratory conducive to scientific inquiry and the development of new knowledge and creative and innovative ideas related to important national interests. (b) The Parties further recognize that the free exchange of ideas among scientists and engineers at the Laboratory and colleagues at universities, colleges, and other laboratories or scientific facilities is vital to the success of the scientific, engineering, and technical work performed by Laboratory personnel. (c) In order to further the goals of the Laboratory and the national interest, it is agreed by the Parties that the scientific and engineering personnel at the Laboratory shall be accorded the rights of publication or other dissemination of research, and participation in open debate and in scientific, educational, or professional meetings or conferences, subject to the limitations included in technology transfer agreements and such other limitations as may be required by the terms of this Contract. Nothing in this clause is intended to alter the obligations of the Parties to protect classified or unclassified controlled nuclear information as provided by law. (d) Nothing in the Section I clause entitled “DEAR 952.204-75 Public Affairs” is intended to limit the rights of the Contractor or its employees to publicize and to accurately state the results of its scientific research. The LLNL SOW states (emphasis added): 2.0 Laboratory Mission and Scope of Work. The Contractor shall manage, operate, protect, sustain and enhance the Laboratory’s ability to function as a NNSA Multi-Program Laboratory, while assuring accomplishment of the Laboratory’s primary mission - strengthening the United States’ security through development and application of world-class science and technology to enhance the nation’s defense and to reduce the global threat from terrorism and weapons of mass destruction. To some degree, these, and similar statements are “boilerplate” in that they have no specific obligations attached to them. However, they are indicators of what the orientation of the laboratory is expected to be. Indeed, Section H-13 of the LLNL contract says more pragmatically: 2 See Contract No. DE-AC52-07NA27344 Modification No. 241 for Sections B-H of the LLNL contract, available at http://www.llnsllc.com/contract/docs/Part_I_Section_B-H-LLNL_992011mod241.pdf. 58

OCR for page 57
This Contract is a management and operating contract, which holds the Contractor accountable for performance. This Contract uses clearly defined standards of performance consisting of performance objectives and performance incentives Identical language is found in H-12 of the LANL contract and H-10 of the SNL contract. Section 2.0 of the LLNL SOW lists 17 major points that the scope of work is to include. Five of these 17 deal explicitly with science and technology, as indicated by bold underline below: • Conducting major NNSA research and development programs including using an earned- value management system; • Fostering an environment of scientific skepticism and peer review of research programs; • Assuring the safety, security, reliability, and performance of the national nuclear weapons stockpile pursuant to national security policy and presidential and congressional directives; • Demonstrating design and development capabilities to support a Reliable Replacement Warhead strategy, and stockpile and complex transformation; • Providing scientific, engineering, and computational capabilities that support assessment, dismantlement, manufacturing, and refurbishment of the enduring stockpile at a number of sites; • Operating major facilities including the National Ignition Facility and the Device Assembly Facility that support broad national interests and users. • Ensuring the secure handling and safe disposition of plutonium, highly enriched uranium, and tritium; • Helping to deter, detect, and respond to the proliferation of weapons of mass destruction; • Conducting fundamental science research, nuclear energy development, and nuclear waste management technology in support of other DOE programs; • Contributing to civilian and industrial needs and non-NNSA defense activities through a work for others program by using the scientific and technical expertise that derives from carrying out the Laboratory mission; • Providing access to the capabilities of the laboratory to further Department of Homeland Security mission objectives; • Advancing of science, mathematics, and engineering education; • Advancing science through technological innovation, public and private sector collaboration, and technology transfer to enhance U.S. economic competitiveness and national security; • Managing and operating the Laboratory facilities and infrastructure in an efficient, cost effective, and innovative manner; • Remediating and restoring the Lawrence Livermore National Laboratory sites; • Managing waste minimization, treatment, storage, and disposal of all wastes; and • Assisting the nuclear weapons complex in waste stabilization, storage and disposition technologies. The lists for the other two laboratories are similar, but with different specific elements. These lists appear to be something more than general principles and something less than specific contractual obligations. Regarding expectations of performance, all three contracts say in Section H-2: H-2 PERFORMANCE DIRECTION (a) The Contractor is responsible for the management and operation of the site in accordance with the Terms and Conditions of the Contract, duly issued Work Authorizations (WAs), and written direction and guidance provided by the Contracting Officer and the Contracting Officer’s Representative (COR). NNSA is responsible for establishing the work to be accomplished, the applicable requirements to be met, and overseeing the performance of work of the Contractor. The Contractor will use its expertise and ingenuity in Contract performance and in making choices among acceptable alternatives to most effectively, efficiently and safely accomplish the work called for by this Contract 59

OCR for page 57
NNSA is responsible for telling the contractor what to do, and the contractor is responsible for figuring out how to do it. EVALUATION OF THE QUALITY OF SCIENCE AND ENGINEERING The contracts make clear that the primary responsibility for monitoring contract performance rests with the site offices: Los Alamos Site Office (LASO); Livermore Site Office (LSO); and Sandia Site Office (SSO). From Section G of each of the contracts: The NNSA Manager, Livermore Site Office (LSO), is the Contractor’s primary point of contact for all technical and administrative matters, except as identified in (b) below, regarding this Contract. The LSO Administrative Contracting Officers are the Contractor’s primary point of contact for all contractual matters for this Contract. The NNSA Manager, Los Alamos Site Office is the Contractor’s primary point of contact for all technical and administrative matters, except as identified in (b) below, regarding performance of this contact. The LASO Administrative Contracting Officer is the Contractor’s primary point of contact for all contractual matters. The NNSA Manager, Sandia Site Office (SSO), is the Contracting Officer responsible for this Contract. The SSO is the Contractor’s focal point of contact for all matters, except as identified in (b) below, regarding this Contract. Clause (b) cited in each of these states that “The Patent Counsel, Office of Chief Counsel, NNSA Service Center, is the Contractor’s focal point for items concerning patent, intellectual property, technology transfer, copyright, open source, licenses and technical data issues.” However, all three contracts make clear that there are major authorities that are reserved to the contracting officer, and not the site manager: (b) Clarifying the Contract Relationship NNSA will establish the work to be accomplished by the Contractor, set applicable requirements to be met by the Contractor and provide performance direction to the Contractor regarding what NNSA wants in each of its programs. NNSA will issue performance direction to the Contractor only through a warranted Contracting Officer or a designated Contracting Officer’s Representative. All other Federal staff and oversight components are therefore precluded from tasking contractor personnel. H-2 PERFORMANCE DIRECTION (a) The Contractor is responsible for the management and operation of the site in accordance with the Terms and Conditions of the Contract, duly issued Work Authorizations (WAs), and written direction and guidance provided by the Contracting Officer and the Contracting Officer’s Representative (COR). NNSA is responsible for establishing the work to be accomplished, the applicable requirements to be met, and overseeing the performance of work of the Contractor. The Contractor will use its expertise and ingenuity in Contract performance and in making choices among acceptable alternatives to most effectively, efficiently and safely accomplish the work called for by this Contract (b) Only the Contracting Officer may issue, modify, and priority rank WAs. (c) (1) The Contracting Officer and the NNSA Administrator will appoint, in writing, specific NNSA employees as CORs with the authority to issue Performance Direction to the Contractor. CORs are authorized to act within the limits of their delegation letter . . . COR functions include technical monitoring, inspection, and other functions of a technical nature not involving a change in the scope, cost, or terms and conditions of the Contract. The COR is authorized to review and approve technical reports, drawings, specifications, and technical information delivered by the Contractor. (2) The Contractor must comply with written Performance Directions that are signed by the COR 60

OCR for page 57
The following from the LLNL contract is representative (emphasis added): (b) Performance Appraisal Process. (1) Performance Evaluation Plan. (i) A Performance Evaluation Plan shall be developed and finalized by the Contracting Officer, with Contractor input . . . The NNSA Livermore Site Office Manager reserves the unilateral right to make the final decision on all performance objectives and performance incentives (including the associated measures and targets) used to evaluate Contractor performance. The NNSA Administrator reserves the unilateral right to make the final decision on all award term incentives (including the associated measures and targets) used to evaluate Contractor performance. (ii) Only the Contracting Officer may revise the Performance Evaluation Plan, consistent with the Contract’s Statement of Work, during the appraisal period of performance . . . (2) Contractor Self-Assessment. The Contractor shall prepare an annual self-assessment of its performance against each of the performance objectives and incentives contained in the Performance Evaluation Plan . . . The Contracting Officer will identify the structure and medium to be used by the Contractor in delivering its annual self-assessment. Matters are complicated by the fact that the three site offices are organized differently. Recent Contract Management Plans (CMP) for the three laboratories 3,4,5 make it clear that in the case of each location, the contract manager has most of the responsibility. The CMPs for LANL and LLNL are very similar. Note, however, that in section 5.3 (which describes the Site Manager), there is a significant difference. The LANL contract states: “The Site Office Manager is a senior NNSA manager that provides an on-site, day-to-day presence at the laboratory. The LASO Site Manager is responsible for effective contract administration at LANL to ensure the successful implementation of NNSA programs. The Site Manager relies on the Site CO . . . to administer contracts based upon demonstrated individual qualifications and Site Office needs, as well as to handle most day to-day administrative contract duties. The LASO manager currently does not possess CO authority. As such, he must rely on his COs and jointly issue direction to LASO which is or may be considered outside the current scope of the Prime Contract.” The LLNL contract states: “The Site Office Manager is a senior NNSA manager that provides an on-site, day-to-day presence at the laboratory. The LSO Site Manager is responsible for effective contract administration at LLNL to ensure the successful implementation of NNSA programs. The Site Manager is also an Administrative Contracting Officer with authority to administer contracts based upon demonstrated individual qualifications and Site Office needs. Although the LSO Site Manager is a warranted CO, she relies largely on the Site CO . . . to handle most day-to-day administrative contract duties. The SNL contract states: (section 5.1.1) The SSO Site Manager is appointed as a Contracting Officer. . . . The SSO Manager is a senior NNSA manager that provides an on-site, day-to-day presence at SNL. The SSO Manager is responsible for effective contract management and oversight at SNL to ensure the successful implementation of NNSA programs. The SSO Manager is also a CO with authority to administer the Sandia Contract based upon demonstrated individual qualifications and SSO needs. Although the SSO Manager is a warranted CO, she relies largely on the SSO CO to handle most day-to-day 3 Contract Management Plan for Los Alamos National Laboratory, Contract No. DE-AC52-06NA25396, Los Alamos CMP-9-3-08, 2008. 4 Livermore Site Office, Contract Management Plan for Lawrence Livermore National Laboratory, Contract No. DE-AC52-07NA27344, LSO_CMP_6-10-088, 2008. 5 Contract Management Plan for Sandia Corporation, 2007. 61

OCR for page 57
administrative contract duties. “5.5 Contracting Officer (CO): The CO has sole authority to enter into, administer, or terminate Federal contracts. The CO, through properly written modifications to the contract, is the only person authorized to make changes to cost, scope, and schedule. The CO must ensure that all requirements of law, executive orders, regulations, and all other applicable procedures, including clearances and approvals, have been met. The CO is also responsible for ensuring performance of all necessary actions for effective contracting, ensuring compliance with the terms of the contract, and safeguarding the interests of the United States in its contractual relationships. The FAR allows the CO wide latitude to exercise business judgment. This duty includes the balanced objective of safeguarding the interests of the United States in its contractual relationships and ensuring that contractors receive impartial, fair, and equitable treatment.” In a formal sense, the site manager is responsible for the evaluation process, but most of the authority rests with the CO. The evaluation begins with the preparation of the PEP, which is agreed (and signed) among the laboratory director, the site manager, and the CO. The laboratory then conducts a self evaluation, which the CO (with the help of others at the site office) uses in the preparation of a PER (performance evaluation report). Final sign-off on the results of the evaluation rests with the Principal Deputy (NNSA) Administrator. Typically, none of these individuals is a scientist or engineer. These three contract management plans say little or nothing about scientific quality, and nothing about the process for evaluating scientific quality. On the other hand, there is a lot of emphasis on deliverables, both substantive (i.e., related to the research and program work) and procedural/managerial/legal. PERFORMANCE EVALUATION PLAN NRC staff reviewed PEPs for LLNL 6 and for LANL. 7 For each laboratory, the annual PEP is the official basis for the evaluation of performance under the contract, which in turn is the basis for the award (incentive) fee and the award term (i.e., award of continuation of the contract). The PEP is a modification to the contract; it becomes Part III, Section J, Appendix F [of the M&O contract]. Each PEP is the product of the specific site office, and they are significantly different in form. LLNL PEP This plan lists eleven Strategic Performance Objectives: 1. Complete essential activities for core weapons program requirements. 2. Strengthen the foundation of deterrence through stockpile science, technology, and engineering. 3. Propose and implement strategies for sustaining a strong deterrent at low numbers compatible with START, NPR and CTBT goals. 4. Execute Inertial Confinement Fusion Ignition and High Yield Campaign in support of stockpile stewardship. 5. Support nonproliferation and threat reduction. 6. Provide science, technology, and engineering excellence. 7. Support current and evolving mission performance by providing effective and efficient facilities and infrastructure. 8. Maintain safe and environmentally sound operations in an efficient and effective manner in support of mission objectives. 6 LANL Performance Evaluation Plan for FY2011. 7 LLNL Performance Evaluation Plan, dated April 28, 2010, is for the evaluation of performance for FY2010 (beginning October 1, 2009). 62

OCR for page 57
9. Maintain secure operations in an efficient and effective manner in support of mission objectives. 10. Manage business operations in an effective and efficient manner while safeguarding public assets and supporting mission objectives. 11. Governance assures performance and creates long-term sustainable value for the institution. The PEP organizes the Strategic Performance Objectives as six program objectives (1-6), three operations objectives (7-9), and two institutional management objectives (10 and 11). Objective 6 deals explicitly with excellence in science and engineering and is the only objective to do so. Attachment 1 breaks the 11 objectives into more detail. The part of that table dealing with Objective 6 is shown in Table E.2. In calculating the performance objective award fee, all six of the program objectives are considered together. Roughly 35 percent of this part of the fee depends on the six program objectives; 45 percent depend on the three operations objectives, and 21 percent depend on the two institutional management objectives. Although it is not explicitly stated, it appears that Objective 6 (excellence in science and engineering) accounts for perhaps 6 percent of the incentive fee. The award term is tied to five objectives, none of which is science and technology per se. These are: stockpile stewardship mission; site transformation activities; sustainable management; safety management system; and contractor assurance system. The first of these five contains explicit science- based milestones and objectives, but these objectives are not “excellence in science.” These five “ATI”s (award term incentives) are generally combinations of near-term (i.e., current year) and near-to-midterm (i.e., 1-3) objectives. TABLE E.2 LLNL PEP Strategic Performance Objective 6 SOURCE: LANL Performance Evaluation Plan, dated April 28, 2010, for the evaluation of performance for FY2010 (beginning October 1, 2009), Attachment 1. 63

OCR for page 57
LANL PEP The plan lists and explains 19 performance based incentives (PBIs). PBI 18 is concerned with the award term incentives while the others are related to award (incentive) fee. PBI 12 is focused on science and engineering quality: PBI No. 12 Objective: Excellence in Science, Technology, and Engineering Objective Statement: Science, technology, and engineering underpin and enable Los Alamos to provide knowledge and technologies to execute its national security missions. State of the art equipment and facilities enable science to push the frontiers of knowledge; however capabilities rely on the appropriate mix of people and resources. Leadership, competence and insight drive science and position the institution for success. This is a termed a “stretch” PBI with a maximum value of $5,500,000 (or 10.7 percent of the maximum available). It is evaluated subjectively according to the guidance shown in Table E.3. PBI 18 lists five performance measures for the award term. One of these five, Measure 18.3 Demonstrate Leadership in Pu Science, is explicitly related to science. The detailed description of measure 18.3 is shown below in Figure B. TABLE E.3 Performance Measures for LANL PBI No. 12 continues 64

OCR for page 57
TABLE E.3, continued continues 65

OCR for page 57
TABLE E.3, continued continues 66

OCR for page 57
TABLE E.3, continued continues 67

OCR for page 57
TABLE E.3, continued 68

OCR for page 57
FIGURE E.1 Evaluation measures for LANL 2010 PBI No. 18. 69