C

Response to Past Studies

OVERIEW

This appendix summarizes external reviews the Department of Energy’s (DOE’s) Office of Environmental Management (EM) over the past 20 years. It first lays out the National Research Council¹ (NRC) series on “Improving Project Management at DOE,” which began in the late 1990s and led to a number of findings and recommendations. It will summarize an external review of EM that occurred in 2011 at the request of the Secretary, and will provide an organized review of U.S. Government Accountability Office (GAO) studies that have looked specifically at EM and EM projects over that 20-year time frame.

All of these reviews and studies will be synthesized into a timeline progression of findings and recommendations unique to EM, which will help lay out issues that seem to have been resolved as well as those that seem to be recurring over the period of assessment. Conclusions will be developed providing the basis for further inquiry.

PAST EXTERNAL REVIEWS OF DOE AND EM

The purpose of this appendix is to give some context to past studies examining project and program management practices and success within DOE’s EM organization. The review will briefly cover the past 20 years starting with two NRC study teams beginning in 1998, which looked at the entirety of project and

___________________

¹ Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council are used in an historical context identifying programs prior to that date.

program management across the DOE capital investment program, including EM. It will summarize an internal independent study commissioned by Secretary Steven Chu in 2011 that looked specifically at EM. It will also summarize GAO reviews of EM’s project management and program management practices that have occurred since 1999, with emphasis on a number of recently published documents.

NRC Phase I Study

In 1997, the House Appropriations Subcommittee on Energy and Water was concerned about project performance within DOE and directed DOE to engage a respected entity to review project performance. As a result, the NRC became engaged and released a 1998 report titled Assessing the Need for Independent Project Reviews in the Department of Energy (Phase I), which was authored by Lloyd Duscha (member, National Academy of Engineering [NAE]). A study committee was formed, chaired by Dr. Ken Reinschmidt (NAE), the Committee to Assess the Policies and Practices of the Department of Energy to Design, Manage, and Procure Environmental Restoration, Waste Management, and Other Construction Projects. Its charge was to look at policies and practices across all of DOE, including the Office of Science (SC), National Nuclear Security Agency (NNSA), and EM.

Meanwhile, there was essentially an “impound” on spending for DOE project site work pending congressional review of multiple independent project review reports. The House Appropriations Subcommittee on Energy and Water remained engaged, with briefings by the NRC on its deliberations during this period. The committee published its report, Improving Project Management in the Department of Energy, in June 1999. In this report, the committee outlined that DOE fell far short of best practices in a number of areas, including:

• Organization-wide project management policy
• Clear definitions of responsibility and accountability
• Control of changes in the scope, cost, and definition of projects
• State-of-the-art project management systems
• Identification, dissemination, and implementation of lessons learned
• Preproject and preconstruction planning
• Scope definition at the project baseline stage
• Assessing and managing project risk
• Setting contingency allowances based on risk
• Cost estimation and scheduling
• Objective performance-based incentives
• Performance measurement and progress reports
• DOE-wide financial reporting systems
• Cost and performance databases and information systems

• Selection, training, and qualification of project managers
• Project management core competency and organization

Before dissolving, the committee also prepared a primer, Characteristics of Successful Megaprojects.

NRC Phase II Study

The FY 1999 Appropriations Act zeroed out the budget for the DOE office that had been handling project and facilities management. DOE moved quickly toward development and publication of Order 413.3 as its project life-cycle management model. Congress also directed additional review to be done by the NRC. This second NRC committee of experts was formed to do a series of reviews and began its work just as Order 413.3 was being finalized by a newly-formed oversight office (Office of Engineering and Construction Management (OECM) and three new Project Management Support Offices (PMSOs)). This committee was again chaired by Dr. Ken Reinschmidt (NAE), the Committee for Oversight and Assessment of Department of Energy Project Management. Meeting from 2000 to 2003, it provided the Deputy Secretary a singular opportunity for independent views of project management practices within DOE to enable best possible decisions. Its scope was to review across all DOE:

• Policies, Procedures, Documentation, and Reporting; including review of Order 413.3, its implementation and OECM
• Project Planning and Controls
• Skills, Selection, and Training of Personnel
• Project Reviews
• Acquisition and Contracting
• Risk Management
• Organizational Structure, Responsibility, and Accountability

The committee was prolific in its work and evaluations, publishing the following documents:

• Interim Letter Report to Secretary Richardson, January 2001
• Progress in Improving Project Management in the Department of Energy: 2001 Assessment
• Interim Letter Report to Secretary Abraham, May 2002
• Progress in Improving Project Management in the Department of Energy: 2002 Assessment
• The Owners Role in Project Management and Preproject Planning, 2002
• Progress in Improving Project Management in the Department of Energy: 2003 Assessment, 2004

• The Owner’s Role in Project Risk Management (in DOE), 2005

Over the course of its 3-year evaluation, it met 14 times, visited DOE headquarters and 10 DOE field sites, met with over 200 DOE and contractor personnel (some multiple times), and reviewed project information from dozens of projects. Its recommendations to DOE, as summarized in the 2003 assessment, were many:

• Develop policies and procedures to define the DOE method of managing projects;
• Create a project management culture across the agency that supports the consistent implementation of policies and procedures;
• Provide leadership that ensures disciplined planning and execution of projects as well as support for continuous process improvement;
• Provide a project management champion at the highest level of the department to ensure that a focus on the importance of project management is established and maintained;
• Develop competence in fulfilling the owner’s role in front-end project planning, risk management, and project execution;
• Apply rigorous project reporting and controls that include earned value systems, link day-to-day management data to periodic reporting, forecast time and cost to complete, and maintain historical data with which to benchmark project performance;
• Document processes and performance to support benchmarking and trend analysis;
• Invest in human capital by providing training and career development to ensure an adequate supply of qualified, skilled project directors;
• Continue, refine, and document a program of external and internal project reviews; and
• Employ innovative approaches to capital acquisition and the use of performance-based contracting.

As stated in the 2004 report, “Most of these (suggested) changes relate to inadequate planning, inadequate risk management, and inadequate monitoring and follow-up” (NRC, 2004, p. 70). Perhaps prescient, it wrote, “The concern of the committee is not so much that Order O[rder] 413.3, Manual M 413.3-1, other documents, and the Project Management Career Develop Program (PMCDP) will be rescinded, but rather that they will be circumvented.” (NRC, 2004, p. 3). It is clear that the impact of these two committees fundamentally changed the way DOE has approached projects in the past two decades, with many of its recommendations acted on within larger DOE complex.

Perhaps a poignant reminder of the optimism of the early 2000s was in this finding in the 2002 Assessment (NRC, 2002, p. 55).

• Is there consistent, credible, independent review of all aspects of project performance to support EM’s federal oversight responsibility?
• Would replicating SC’s organizational structure, with its program offices and well-defined line management responsibilities and accountabilities, in the EM organization, positively impact EM’s project performance?

The findings of this review were distributed across EM via a transmittal memorandum in September 2011 by Daniel Poneman (Poneman, 2011, p. i). The findings of the study were grouped in the following areas:

• Accountability. Roles and responsibilities for program and project management are not well understood throughout the EM headquarters and field organizations.
• Decision-making. A clear devolution of authority to responsible program line managers is needed to avoid confusion about roles and responsibilities, delays, and rework at the field level.
• Culture. Extraordinary efforts are made to report projects as “Green” (on schedule and on budget) regardless of actual project performance indicators. There is pressure “to color a project Green” with a quick fix. In this mode, efforts to “get to Green” can be detrimental to successful project delivery and has had negative consequences for the EM culture.
• Stability. Excessive turnover in EM headquarters, and federal and contractor field positions for project management has led to loss of technical capability, program and project leadership skills, and critical project knowledge.
• Structure. Delivery of projects is hampered by the lack of appropriately structured and empowered program offices at EM headquarters. These offices, staffed with program managers with line management authority, should provide strategic policy and guidance, resource allocations, project advocacy and oversight, and enable effective sharing of lessons learned across projects and sites.
• Peer Review. Independent, external peer review occurs too late in the project development cycle. Reviews have not encompassed all aspects of project performance (technical, schedule, cost, and management). Peer reviewers do not represent the mix of experience and expertise needed to adequately assess all aspects of the project.
• Alignment. Lack of well-defined project management roles and responsibilities and lines of communication contribute to a proliferation of EM headquarters data calls and information requests.

As the committee succinctly noted: “Appropriately constituted program offices with empowered program managers; strong line management with well understood roles and responsibilities; effective peer reviews; stability in

organizational structure and personnel; and a culture of open information sharing could address many of EM’s program and project performance issues” (Poneman, 2011, p. 9). In its report, the review committee “….. encourages top management at DOE and EM to adopt a new mode of leadership realizing that real change is imperative” (Poneman, 2011, p. 12). In its recommendations, the committee urged EM to:

• Transform the culture of EM to one of more open sharing, collaborative problem-solving, and transparency so that open and honest results are communicated and acted upon, resulting in continuous improvements being made to EM. This culture should be proactive in its approach to managing programs and projects, rather than reactive.
• Ensure that accountability, responsibility, and authorities for programs and projects are formally documented, effectively communicated, and executed at the right level throughout the EM organization.
• Promote EM-wide (headquarters and field) stability by making every effort to retain experienced, competent, and well-trained staff. Staff turnover at EM Headquarters and the field is problematic and does not allow for stable, consistent, effective decision-making or ownership of the project baselines.
• Ensure that peer reviews are performed for capital asset projects regularly throughout the projects’ life-cycle (Critical Decisions 0 through 4). These reviews (tailored to project size/complexity) should be comprehensive, independent project assessments similar to the SC peer review model in scope, including technical, cost, schedule, management; environment, safety and health; and risk evaluations.
• Establish EM headquarters program offices (geographic, site, or product line) staffed by program managers with clearly defined roles and strong accountability, authority, and responsibility in their area. These offices would provide strategic policy development and guidance, resource allocations, and project advocacy and oversight. This would also enable effective sharing of lessons learned across projects and sites.
• Use technical/management independent advisory committees for advice on program and project performance at the program, site, or large project levels. These advisory committees would be in addition to the Environmental Management Advisory Board (EMAB) and Site-Specific Advisory Boards (SSABs), and would give strategic guidance and assistance.
• Reduce and/or eliminate ad hoc or limited-value information requests to only those that are necessary. These EM headquarters information requests should be aligned to the clarified responsibilities, authorities, and accountabilities of the EM headquarters organization requesting the information.
• Establish a stronger partnership relationship with the executing contractors. This would recognize EM’s role as a “demanding customer” that

rewards strong performance with contract incentives, but also that real project success requires mutual respect and trust and open and honest communications.

GAO Reviews

Since 1999 when the first NRC report on DOE project management was published, the GAO has published 56 reports focused on DOE. Of these 56 reports, 36 are directed specifically at EM (or in some cases EM and NNSA). Some of the reports are focused strictly on technical issues surrounding cleanups. Others are focused on management and project management activities as identified by project reviews or review of a series of projects within a portfolio or at a site. It is worth noting that 11 of these reports have been published since 2018. The listing of these reports is provided at the end of this appendix in reverse chronological order.

As an example, in December 2019, DOE reported that it faced an estimated $494 billion in future environmental cleanup costs, with the total liability roughly tripling during the previous 20 years (GAO, 2019). GAO was asked to examine EM’s operations activities and made several recommendations including that EM should establish cleanup work classification requirements and revise its cleanup policy to follow program and project management leading practices (GAO-19-223). It pointed out that 77 percent of EM’s FY 2019 budget was classified as operations activities and not subject to outside review, while only 18 percent contained requirements for oversight as capital asset projects.

It further pointed out that EM’s cleanup policy does not follow any of 9 (GAO) selected program management leading practices or 9 of 12 selected project management leading practices. For example, EM’s 2017 cleanup policy does not follow the program management leading practice of conducting risk management throughout the life of a program or the project management leading practice of requiring independent reviews of operations activities.

GAO found that EM’s environmental liability does not include the costs of all future cleanup responsibilities. As an example, this liability did not include the cost for completing a known, large waste treatment facility, called the Waste Treatment and Immobilization Plant, at the Hanford site. It pointed out that between “30 to 60 percent” of EM’s cleanup budget goes toward recurring activities necessary to maintain the sites such as physical security and infrastructure maintenance, rather than toward reducing EM’s environmental liability. It also found that EM officials have not analyzed the root causes of the cost growth of environmental liability between 2011 and 2018, despite it increasing $214 billion (GAO-19-460T). GAO noted that EM does not have a program-wide cleanup strategy and relies primarily on individual sites to locally negotiate cleanup activities and establish priorities, which does not always balance overall risks and costs. For example, it pointed out that the Hanford and Savannah River sites plan

to treat similar radioactive tank waste differently, with Hanford’s efforts possibly costing tens of billions more than Savannah River’s. In addition, EM manages most of its cleanup work as operations activities, under less stringent requirements than other environmental remediation projects. Operations activities are not subject to independent oversight outside EM, and therefore DOE cannot hold EM accountable for its performance.

As another example, DOE stated in reports to Congress as early as 2007 that it intended to manage demolition of its three former gaseous diffusion plants (GDPs) in an integrated manner. A Decontamination and Decommissioning (D&D) Fund was established by law to pay for the cleanup costs of the GDP sites, allowing EM to coordinate and make trade-offs in its use of resources among the three GDPs. However, EM has managed the cleanup of the three GDPs as three individual sites. In addition, GAO found that EM was not following relevant leading practices that GAO recommends for managing the cleanup as a program (having a program management plan; a reliable integrated master schedule; and a reliable, integrated, comprehensive life-cycle cost estimate) (GAO 20-63). EM has reported spending a total of about $15.5 billion on GDP cleanup as of fiscal year 2018 with cost estimates of remaining work exceeding the currently allocated $25 billion. It is worth noting that at the present time, EM is asking for relief from following DOE Order 413-3B on the GDP D&D projects.

In February 2019, GAO made two recommendations to EM to review and revise its cleanup policy to include project and program management leading practices related to scope, cost, schedule performance, and independent reviews. DOE agreed with GAO recommendations and responded that it intends to replace its current cleanup policy with two separate project and program management policies that will incorporate leading practices related to scope, cost, schedule, and independent reviews, as appropriate. EM expects to issue the new project management policy in spring 2020 and the new program management policy in fall 2020 (GAO-20-285PR).

In summary, there are a large number of reports that GAO has published over the past 20 years that have been critical of the way that EM is managing its program and portfolio. A number of recent reports, some identified above, have continued to focus on needed improvements that were identified initially in the late 1990s.

Summary

Over the past 20 years a number of studies have analyzed the project and program management practices in EM. It should be noted that some progress has been made. The PMCDP program, for instance, helps with relevant training and career progression, as well as qualifying those who are in project management roles. Order 413-3B has set a process in place to assure that some level of consistency and quality control is followed for capital asset projects. Table C.1

lays out recommendations from the final National Academies 2003 Assessment report and indicates whether these issues were recurring in the 2011 Secretarial review and other recent GAO reports.

For example, the first item in Table C.1, “the need for program management and strategic management leadership at headquarters,” elicited the following statements in different reports. “Senior managers should continue to emphasize the importance of improving the project management processes and procedures to assure long-term improvement throughout the organization” (NRC, 2004, p. 96). “In this context, the committee provides its recommendation in the form of a prospectus—a set of aspirations that senior DOE and EM leadership should be focused on to develop a realistic change initiative to make EM a stable, and consistently high-performing organization” (Poneman, 2011, p. 11). EM’s cleanup policy does not follow program and project management leading practices (GAO 19-460T, p. 12). In another report it stated “EM has not followed (i.e., has not met, has minimally met, or has partially met) best practices to ensure that these systems are (1) comprehensive, (2) provide reliable data, and (3) are used by EM leadership for decision-making—which are the three characteristics of a reliable earned value management (EVM) system. Moreover, EM has allowed the contractors to categorize a large portion of their work in a way that limits the usefulness of the EVM data …. EM has not ensured that EVM Systems are comprehensive, provide reliable data, or are used by leadership for decision-making” (GAO 19-223, p. 36).

As noted in Table C.1, most of the issues identified in 2003 continue to be issues with project and program management in 2020. Please note that this assessment is not exhaustive, nor representative of all recommendations given by the NRC or GAO. It also does not review responses by EM to these past studies but instead relies on the progression of similar findings over two decades.

REFERENCES

REPORTS BY THE U.S. GOVERNMENT ACCOUNTABILITY OFFICE RELATED TO THE OFFICE OF ENVIRONMENTAL MANAGEMENT