Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 67
Appendix 5: What the Laboratories’ Management and
Operations (M&O) Contracts Say About the Conduct and
Evaluation of Science and Engineering
For each laboratory management corporation, the contract is the primary document
governing its relationship with the government. Each contract includes a statement of work
and an annual Performance Evaluation Plan (PEP), which is included in the contract as a
contract modification. Other modifications are also included as agreed between the
government and the contractor. Other important documents that affect the conduct of science
and engineering are the Contract Management Plan, the Parent Organization Oversight Plan,
and the annual Performance Evaluation Review (PER), which is based on the PEP.
Although they have important specific differences, the contracts governing SNL,
LANL, and LLNL are generally similar in form and content. In general, each specifies: (1)
what the laboratory is and does; (2) what the management corporation is committed to do; (3)
what work is to be done at the laboratory (the “statement of work”, or SOW); and (4) how
that work is to be reviewed, evaluated, and rewarded. The contracts also specify certain
rights and responsibilities of specific government offices to task and/or oversee the work.
The SOWs, while lengthy, are basically general. The contracts do not assign specific
program tasks. The following—from the LLNL contract—is representative:49
Inasmuch as the assigned missions of the Lawrence Livermore National
Laboratory (Laboratory) are dynamic, this Statement of Work (SOW) is not
intended to be all-inclusive or restrictive, but is intended to provide a broad
framework and general scope of the work to be performed at the Laboratory.
This SOW does not represent a commitment to, or imply funding for, specific
projects or programs. The National Nuclear Security Administration (NNSA)
and Department of Energy (DOE) work requirements are developed through
strategic planning and program plans.
While this quoted paragraph refers primarily to programmatic work, the contracts make clear
and specific that the laboratory management is committed to other duties, including
operational and administrative tasks. Indeed, these management tasks constitute a large part
of the work against which the management corporations are evaluated.
The performance evaluation process determines how successful the management has
been for the period being evaluated (usually a year), and the contract specifies how success is
rewarded. The rewards consist of an award fee (incentive fee) and an award term (i.e., an
extension of the temporal term of the contract without need for competition). Independent of
the review, the management corporation also receives an annual fixed fee (specified in the
contract) and payment for allowable expenses associated with running the management
49
See Modification No.: M003 Supplemental Agreement to Contract No.: DE-AC52-07NA27344 titled “Part
III, Section J, Appendix B-Statement of Work” from the LLNL contract for the Statement of Work in full
http://www.llnsllc.com/contract/docs/AppendB_mod53_012209.pdf
67
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 68
corporation (much of which is associated with paying officials of the various boards and
advisory groups).
The incentive fees and fixed fees vary across the laboratories and by year. Figure 9
displays approximate values for comparison.
Figure 9: Approximate annual fee structures (in millions of dollars)
LABORATORY FIXED FEE MAXIMUM INCENTIVE
FEE
SNL 16 9-10
LANL 22 52
LLNL 12.5 29.5
Most ( ≥90%) of the incentive fee is awarded when a laboratory successfully executes
tasks specified in its annual PEP so that the amount of incentive fee that is really “in play” in
any evaluation process is in reality a few million dollars (out of a total annual laboratory
budget in excess of $1billion).
All three contracts state that science and engineering are major parts of what the
laboratories do. The LLNL contract is perhaps most explicit in this regard, the Sandia
contract least so. The following excerpt from the LLNL contract also appears verbatim in the
LANL contract (but with a different identifying number):
H-36 INTELLECTUAL AND SCIENTIFIC FREEDOM50
(a) The Parties recognize the importance of fostering an atmosphere at the
Laboratory conducive to scientific inquiry and the development of new
knowledge and creative and innovative ideas related to important national
interests.
(b) The Parties further recognize that the free exchange of ideas among
scientists and engineers at the Laboratory and colleagues at universities,
colleges, and other laboratories or scientific facilities is vital to the success of
the scientific, engineering, and technical work performed by Laboratory
personnel.
(c) In order to further the goals of the Laboratory and the national interest, it is
agreed by the Parties that the scientific and engineering personnel at the
Laboratory shall be accorded the rights of publication or other dissemination of
research, and participation in open debate and in scientific, educational, or
professional meetings or conferences, subject to the limitations included in
technology transfer agreements and such other limitations as may be required
by the terms of this Contract. Nothing in this clause is intended to alter the
obligations of the Parties to protect classified or unclassified controlled nuclear
information as provided by law.
50
See Contract No. DE-AC52-07NA27344 Modfication No.:241 for Sections B-H of the LLNL contract at
http://www.llnsllc.com/contract/docs/Part_I_Section_B-H-LLNL_992011mod241.pdf.
68
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 69
(d) Nothing in the Section I clause entitled "DEAR 952.204-75 Public Affairs"
is intended to limit the rights of the Contractor or its employees to publicize and
to accurately state the results of its scientific research.
The LLNL SOW states (emphasis added):
2.0 Laboratory Mission and Scope of Work.
The Contractor shall manage, operate, protect, sustain and enhance the
Laboratory's ability to function as a NNSA Multi-Program Laboratory, while
assuring accomplishment of the Laboratory’s primary mission -
strengthening the United States’ security through development and
application of world-class science and technology to enhance the nation’s
defense and to reduce the global threat from terrorism and weapons of
mass destruction.
To some degree, these, and similar statements are “boilerplate” in that they have no specific
obligations attached to them. However, they are indicators of what the orientation of the
laboratory is expected to be. Indeed, section H-13 of the LLNL contract says more
pragmatically:
This Contract is a management and operating contract, which holds the
Contractor accountable for performance. This Contract uses clearly defined
standards of performance consisting of performance objectives and
performance incentives
Identical language is found in H-12 of the LANL contract and H-10 of the SNL contract.
Section 2.0 of the LLNL SOW lists 17 major points that the scope of work is to include.
Five of these 17 deal explicitly with science and technology, as indicated by bold underline
below:
Conducting major NNSA research and development programs including using an
earned-value management system;
Fostering an environment of scientific skepticism and peer review of research
programs;
Assuring the safety, security, reliability, and performance of the national nuclear
weapons stockpile pursuant to national security policy and Presidential and
Congressional directives;
Demonstrating design and development capabilities to support a Reliable
Replacement Warhead strategy, and stockpile and complex transformation;
Providing scientific, engineering, and computational capabilities that support
assessment, dismantlement, manufacturing, and refurbishment of the
enduring stockpile at a number of sites;
Operating major facilities including the National Ignition Facility and the Device
Assembly Facility that support broad national interests and users.
69
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 70
Ensuring the secure handling and safe disposition of plutonium, highly enriched
uranium, and tritium;
Helping to deter, detect, and respond to the proliferation of weapons of mass
destruction;
Conducting fundamental science research, nuclear energy development, and
nuclear waste management technology in support of other DOE programs;
Contributing to civilian and industrial needs and non-NNSA defense activities
through a work for others program by using the scientific and technical expertise
that derives from carrying out the Laboratory mission;
Providing access to the capabilities of the laboratory to further Department of
Homeland Security mission objectives;
Advancing of science, mathematics, and engineering education;
Advancing science through technological innovation, public and private
sector collaboration, and technology transfer to enhance U.S. economic
competitiveness and national security;
Managing and operating the Laboratory facilities and infrastructure in an
efficient, cost effective, and innovative manner;
Remediating and restoring the Lawrence Livermore National Laboratory sites;
Managing waste minimization, treatment, storage, and disposal of all wastes; and
Assisting the nuclear weapons complex in waste stabilization, storage and
disposition technologies.
The lists for the other two labs are similar, but with different specific elements. These lists
appear to be something more than general principles and something less than specific
contractual obligations.
Regarding expectations of performance, all three contracts say in section H-2:
H-2 PERFORMANCE DIRECTION
(a) The Contractor is responsible for the management and operation of the
site in accordance with the Terms and Conditions of the Contract, duly
issued Work Authorizations (WAs), and written direction and guidance
provided by the Contracting Officer and the Contracting Officer’s
Representative (COR). NNSA is responsible for establishing the work to be
accomplished, the applicable requirements to be met, and overseeing the
performance of work of the Contractor. The Contractor will use its
expertise and ingenuity in Contract performance and in making choices
among acceptable alternatives to most effectively, efficiently and safely
accomplish the work called for by this Contract
NNSA is responsible for telling the contractor what to do, and the contractor is responsible
for figuring out how to do it.
70
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 71
Evaluation of the Quality of Science and Engineering
The contracts make clear that the primary responsibility for monitoring contract
performance rests with the site offices: Los Alamos Site Office (LASO); Livermore Site
Office (LSO); and Sandia Site Office (SSO).
From Section G of each of the contracts:
The NNSA Manager, Livermore Site Office (LSO), is the Contractor’s primary
point of contact for all technical and administrative matters, except as identified
in (b) below, regarding this Contract. The LSO Administrative Contracting
Officers are the Contractor’s primary point of contact for all contractual matters
for this Contract.
The NNSA Manager, Los Alamos Site Office is the Contractor's primary point
of contact for all technical and administrative matters, except as identified in (b)
below, regarding performance of this contact. The LASO Administrative
Contracting Officer is the Contractor's primary point of contact for all
contractual matters.
The NNSA Manager, Sandia Site Office (SSO), is the Contracting Officer
responsible for this Contract. The SSO is the Contractor’s focal point of
contact for all matters, except as identified in (b) below, regarding this
Contract.
Clause (b) cited in each of these states that “The Patent Counsel, Office of Chief Counsel,
NNSA Service Center, is the Contractor’s focal point for items concerning patent, intellectual
property, technology transfer, copyright, open source, licenses and technical data issues.”
However, all three contracts make clear that there are major authorities that are
reserved to the contracting officer, and not the site manager:
(b) Clarifying the Contract Relationship
NNSA will establish the work to be accomplished by the Contractor, set
applicable requirements to be met by the Contractor and provide performance
direction to the Contractor regarding what NNSA wants in each of its programs.
NNSA will issue performance direction to the Contractor only through a
warranted Contracting Officer or a designated Contracting Officer’s
Representative. All other Federal staff and oversight components are therefore
precluded from tasking contractor personnel.
H-2 PERFORMANCE DIRECTION
(a) The Contractor is responsible for the management and operation of the site
in accordance with the Terms and Conditions of the Contract, duly issued Work
Authorizations (WAs), and written direction and guidance provided by the
71
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 72
Contracting Officer and the Contracting Officer’s Representative (COR).
NNSA is responsible for establishing the work to be accomplished, the
applicable requirements to be met, and overseeing the performance of work of
the Contractor. The Contractor will use its expertise and ingenuity in Contract
performance and in making choices among acceptable alternatives to most
effectively, efficiently and safely accomplish the work called for by this
Contract
(b) Only the Contracting Officer may issue, modify, and priority rank WAs.
(c) (1) The Contracting Officer and the NNSA Administrator will appoint, in
writing, specific NNSA employees as CORs with the authority to issue
Performance Direction to the Contractor. CORs are authorized to act within the
limits of their delegation letter . . . COR functions include technical monitoring,
inspection, and other functions of a technical nature not involving a change in
the scope, cost, or terms and conditions of the Contract. The COR is authorized
to review and approve technical reports, drawings, specifications, and technical
information delivered by the Contractor.
(2) The Contractor must comply with written Performance Directions that are
signed by the COR
The following from the LLNL contract is representative (emphasis added):
(b) Performance Appraisal Process.
(1) Performance Evaluation Plan.
(i) A Performance Evaluation Plan shall be developed and finalized by the
Contracting Officer, with Contractor input . . . The NNSA Livermore Site
Office Manager reserves the unilateral right to make the final decision on
all performance objectives and performance incentives (including the
associated measures and targets) used to evaluate Contractor performance.
The NNSA Administrator reserves the unilateral right to make the final
decision on all award term incentives (including the associated measures
and targets) used to evaluate Contractor performance.
(ii) Only the Contracting Officer may revise the Performance Evaluation
Plan, consistent with the Contract’s Statement of Work, during the
appraisal period of performance . . .
(2) Contractor Self-Assessment. The Contractor shall prepare an annual self-
assessment of its performance against each of the performance objectives and
incentives contained in the Performance Evaluation Plan . . . The Contracting
Officer will identify the structure and medium to be used by the Contractor in
delivering its annual self-assessment.
Matters are complicated by the fact that the three site offices are organized differently.
72
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 73
Recent Contract Management Plans (CMP) for the three laboratories51 52 53 make it
clear that in the case of each location, the contract manager has most of the responsibility.
The CMPs for LANL and LLNL are very similar. Note, however, that in section 5.3 (which
describes the Site Manager), there is a significant difference.
The LANL contract states: “The Site Office Manager is a senior NNSA manager that
provides an on-site, day-to-day presence at the laboratory. The LASO Site Manager is
responsible for effective contract administration at LANL to ensure the successful
implementation of NNSA programs. The Site Manager relies on the Site CO . . . to
administer contracts based upon demonstrated individual qualifications and Site Office
needs, as well as to handle most day to-day administrative contract duties.
The LASO manager currently does not possess CO authority. As such, he
must rely on his COs and jointly issue direction to LASO which is or may
be considered outside the current scope of the Prime Contract.”
The LLNL contract states: “The Site Office Manager is a senior NNSA manager that
provides an on-site, day-to-day presence at the laboratory. The LSO Site Manager is
responsible for effective contract administration at LLNL to ensure the successful
implementation of NNSA programs. The Site Manager is also an Administrative Contracting
Officer with authority to administer contracts based upon demonstrated individual
qualifications and Site Office needs.
Although the LSO Site Manager is a warranted CO, she relies largely on the Site CO . . .
to handle most day-to-day administrative contract duties.
The SNL contract states: (section 5.1.1) “The SSO Site Manager is appointed as a
Contracting Officer. . . . The SSO Manager is a senior NNSA manager that provides an
on-site, day-to-day presence at SNL. The SSO Manager is responsible for effective contract
management and oversight at SNL to ensure the successful implementation of NNSA
programs. The SSO Manager is also a CO with authority to administer the Sandia Contract
51 CONTRACT MANAGEMENT PLAN
For
LOS ALAMOS NATIONAL
LABORATORY
CONTRACT NO. DE-AC52-06NA25396
Los Alamos CMP-9-3-08
2008
52
LIVERMORE SITE OFFICE
CONTRACT MANAGEMENT PLAN
For
LAWRENCE LIVERMORE NATIONAL
LABORATORY
CONTRACT NO. DE-AC52-07NA27344
LSO_CMP_6-10-088
2008
53
CONTRACT MANAGEMENT PLAN FOR SANDIA CORPORATION, 2007
73
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 74
based upon demonstrated individual qualifications and SSO needs. Although the SSO
Manager is a warranted CO, she relies largely on the SSO CO to handle most day-to-day
administrative contract duties. “5.5 Contracting Officer (CO): The CO has sole authority to
enter into, administer, or terminate Federal contracts. The CO, through properly written
modifications to
the contract, is the only person authorized to make changes to cost, scope, and schedule. The
CO must ensure that all requirements of law, executive orders, regulations, and all other
applicable procedures, including clearances and approvals, have been met. The CO is also
responsible for ensuring performance of all necessary actions for effective contracting,
ensuring compliance with the terms of the contract, and safeguarding the interests of the
United States in its contractual relationships. The FAR allows the CO wide latitude to
exercise business judgment. This duty includes the balanced objective of safeguarding the
interests of the United States in its contractual relationships and ensuring that contractors
receive impartial, fair, and equitable treatment.”
In a formal sense, the site manager is responsible for the evaluation process, but most
of the authority rests with the CO. The evaluation begins with the preparation of the PEP,
which is agreed (and signed) among the laboratory director, the site manager, and the CO.
The laboratory then conducts a self evaluation, which the CO (with the help of others at the
site office) uses in the preparation of a PER (performance evaluation report). Final sign-off
on the results of the evaluation rests with the Principal Deputy (NNSA) Administrator.
Typically, none of these individuals is a scientist or engineer.
These three contract management plans say little or nothing about scientific quality,
and nothing about the process for evaluating scientific quality. On the other hand, there is a
lot of emphasis on deliverables, both substantive (i.e., related to the research and program
work) and procedural/managerial/legal.
Performance Evaluation Plan
NRC staff reviewed PEPs for LLNL54 and for LANL.55 For each lab, the annual PEP
is the official basis for the evaluation of performance under the contract, which in turn is the
basis for the award (incentive) fee and the award term (i.e., award of continuation of the
contract). The PEP is a modification to the contract; it becomes Part III, Section J, Appendix
F. Each PEP is the product of the specific site office, and they are significantly different in
form.
LLNL PEP
This plan lists eleven Strategic Performance Objectives:
1. Complete essential activities for core weapons program requirements.
54
For FY2011
55 This plan, dated April 28, 2010, is for the evaluation of performance for FY2010
(beginning October 1, 2009).
74
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 75
2. Strengthen the foundation of deterrence through stockpile science, technology,
and engineering.
3. Propose and implement strategies for sustaining a strong deterrent at low
numbers compatible with START, NPR and CTBT goals.
4. Execute Inertial Confinement Fusion Ignition and High Yield Campaign in
support of stockpile stewardship.
5. Support nonproliferation and threat reduction.
6. Provide science, technology, and engineering excellence.
7. Support current and evolving mission performance by providing effective and
efficient facilities and infrastructure.
8. Maintain safe and environmentally sound operations in an efficient and effective
manner in support of mission objectives.
9. Maintain secure operations in an efficient and effective manner in support of
mission objectives.
10. Manage business operations in an effective and efficient manner while
safeguarding public assets and supporting mission objectives.
11. Governance assures performance and creates long-term sustainable value for
the institution.
The PEP organizes the Strategic Performance Objectives as six program objectives (#s1-6),
three operations objectives (7-9), and two institutional management objectives (10 and 11).
Objective #6 deals explicitly with excellence in science and engineering and is the only
objective to do so.
75
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 76
Attachment 1 breaks the 11 objectives into more detail. The part of that table dealing with
objective 6 is shown below.
In calculating the performance objective award fee, all six of the program objectives are
considered together. Roughly 35% of this part of the fee depends on the six program
objectives; 45% depend on the three operations objectives, and 21% depend on the two
institutional management objectives. Although it is not explicitly stated, it appears that
object #6 (excellence in science and engineering) accounts for perhaps 6% of the incentive
fee.
The award term is tied to five objectives, none of which is science and technology per
se. These are: stockpile stewardship mission; site transformation activities; sustainable
management; safety management system; and contractor assurance system. The first of these
five contains explicit science-based milestones and objectives, but these objectives are not
“excellence in science”. These five “ATI”s (award term incentives) are generally
combinations of near-term (i.e., current year) and near-to-mid term (i.e. 1-3) objectives.
LANL PEP
The plan lists and explains 19 performance based incentives (PBIs). PBI #18 is concerned
with the award term incentives while the others are related to award (incentive) fee. PBI 12
is focused on science and engineering quality:
PBI No. 12 Objective: Excellence in Science, Technology, and Engineering
76
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 77
Objective Statement: Science, technology, and engineering underpin and
enable Los Alamos to provide knowledge and technologies to execute its
national security missions. State of the art equipment and facilities enable
science to push the frontiers of knowledge; however capabilities rely on the
appropriate mix of people and resources. Leadership, competence and insight
drive science and position the institution for success.
This is a termed a “stretch” PBI with a maximum value of $5,500,000 (or 10.7% of the
maximum available). It is evaluated subjectively according to the guidance shown in Figure
A, below.
PBI No. 18 lists five performance measures for the award term. One of these five,
Measure 18.3 Demonstrate Leadership in Pu Science, is explicitly related to science. The
detailed description of measure 18.3 is shown below in Figure B.
77
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 78
Figure A: Performance Measures for LANL PBI No. 12
78
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 79
79
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 80
80
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 81
81
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 82
Figure B: Evaluation Measures for LANL 2010 PBI
82
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
OCR for page 83
#18
83
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
