Project Management: DOE and NNSA Should Improve Their Lessons-Learned Process for Capital Asset Projects

The MOX project, located at DOE's Savannah River Site in South Carolina and overseen by NNSA, experienced significant cost increases and schedule delays following the start of construction in 2007. After spending nearly $6 billion, NNSA terminated the project in October 2018. While DOE and NNSA have made some recent progress, they have historically struggled to complete, within their original cost and schedule estimates, other major construction projects intended to help maintain the nuclear security complex.

GAO was asked to review issues related to oversight of the MOX project. This report examines (1) when NNSA's project management oversight processes recognized cost and schedule problems at the MOX project and the actions the agency took to address them and (2) the extent to which DOE requires that project management lessons learned from MOX and other projects be documented and shared. GAO reviewed agency documents, visited the MOX project, and interviewed DOE and NNSA officials and representatives of the MOX contractor.

The Department of Energy's (DOE) National Nuclear Security Administration (NNSA) has strengthened its oversight of the Mixed Oxide Fuel Fabrication Facility (MOX) project since 2011 and, as a result, began recognizing cost and schedule problems. The project, begun in 1997, was intended to dispose of large quantities of weapons-grade plutonium no longer required for national security. Prior to 2011, NNSA's project staff failed to recognize signs that the project would not be completed on time or within its approved cost. An independently conducted analysis, prepared in 2014 in response to a GAO recommendation, determined that NNSA staff did not recognize early problems because they were inexperienced in project management. To strengthen oversight, NNSA in late 2010 and 2011 began actions, such as conducting additional reviews and transferring oversight of the project to a newly established office specializing in project management. NNSA continued to identify the contractor's performance problems, such as the lack of credible, reliable cost and schedule data. These continued problems contributed to NNSA's decision to terminate the project.

DOE requires that project staff document and share project management lessons learned on capital asset projects like the MOX project, but not all lessons are to be documented consistently or shared in a timely manner. GAO found that DOE's and NNSA's offices document project management lessons learned differently and that not all of the documented lessons learned are readily accessible to other staff. Additionally, GAO found that DOE does not require that project staff share lessons learned for capital asset projects until the start of construction, which can occur many years after the start of the project. Under key practices, such lessons should be stored in a logical, organized manner, be easily retrievable, and be submitted in a timely manner (see fig.). By developing requirements that clearly define how and where project management lessons learned should be documented and requiring that the lessons be shared in a timely manner, DOE could improve its lessons-learned process and help improve the success of future capital asset projects. Also, for capital asset projects, DOE does not require the evaluation of the results of all corrective actions to respond to lessons learned to ensure that problems are resolved, consistent with key practices. By developing requirements to evaluate the effectiveness of corrective actions, DOE could better verify whether the actions had the intended outcome.

GAO is making three recommendations, including that DOE and NNSA develop requirements for defining how and where project management lessons learned for capital asset projects should be documented and shared routinely and in a timely manner, and for evaluating the effectiveness of corrective actions taken in response to lessons learned. DOE agreed with GAO's recommendations.

The Department of Energy (DOE) and the National Nuclear Security  Administration (NNSA), a separately organized agency within DOE, are  spending billions of dollars on numerous construction projects that,  among other things, are used to help maintain the nuclear weapons  stockpile, process nuclear waste into forms suitable for long-term storage  or permanent disposal, and conduct research and development in the  areas of high-energy physics and nuclear physics. DOE and NNSA use  the capital asset acquisition process to undertake many of these  projects. The cost of these projects can vary greatly, with projects  estimated to cost less than $750 million designated as nonmajor, and  those estimated to cost greater than or equal to $750 million designated  as major. Some of the more complex, first-of-a-kind, major nuclear  construction projects can cost several billions of dollars.

For many years, NNSA has encountered numerous cost, schedule, and  technical challenges that resulted in a reassessment of alternatives and,  in some cases, the cancellation of entire projects or portions thereof.  NNSA terminated one such project—the Mixed Oxide Fuel Fabrication  Facility (MOX) project—after experiencing numerous cost, schedule, and  technical challenges. DOE initiated the MOX project in 1997 at its  Savannah River Site in South Carolina as a key component of NNSA’s  strategy to dispose of large quantities of weapons-grade plutonium  deemed no longer required for national security. Construction began in  2007 after DOE formally approved the project’s cost estimate of $4.8  billion and estimated completion date of September 2016. Since  construction began in 2007, DOE’s cost estimate for the MOX project  increased significantly to about $17.2 billion, and the estimated  completion date for the project was extended to as late as 2048, a  potential delay of nearly 32 years.

In February 2016, DOE proposed terminating the MOX project in favor of  an alternative, referred to as “dilute and dispose,” which was expected to  be less expensive than the MOX project. Congress continued funding  the MOX project. For example, the National Defense Authorization Act for  Fiscal Year 2018 required DOE to proceed with MOX construction unless,  among other things, the Secretary certified that the life-cycle cost of the  dilute and dispose option would be less than half the life-cycle cost of the  MOX project. In May 2018, the Secretary of Energy submitted the  certification and reported that the life-cycle cost estimate was $19.9 billion  for the dilute and dispose option compared with $49.4 billion for the MOX  project. In October 2018, NNSA terminated the MOX project. At the time  of its termination, NNSA had spent nearly $6 billion on the MOX project.

Significant cost increases and schedule delays are not unique to the MOX  project, as DOE and NNSA have historically struggled to complete  projects within their initial cost and schedule estimates. Since 1990, we  have designated DOE’s management of major contracts and projects,  including those executed by NNSA, as an area at high risk for fraud,  waste, abuse, and mismanagement. In response, DOE and NNSA have  undertaken a number of efforts to improve their management of contracts  and projects, and in February 2013, we narrowed the focus of the high- risk designation to major projects within DOE’s Office of Environmental  Management and NNSA to acknowledge progress made in managing  nonmajor projects. However, not all of the actions taken by DOE and  NNSA have resulted in improved project management. As we found in  April 2015, NNSA has had a long history of identifying corrective actions  and declaring them successfully resolved, only to identify additional  actions to address the same problems that had reemerged. The repeat  nature of some of these problems and their considerable effect on the  cost and schedule of DOE and NNSA projects has raised concerns that  lessons from past experiences on projects are not being identified and  applied to current and future projects.

Given NNSA’s shift in focus from the MOX project to the alternative dilute  and dispose approach, you requested that we review issues related to  DOE’s and NNSA’s oversight of the MOX project. This report examines  (1) when NNSA’s project management oversight processes recognized  cost and schedule problems at the MOX project and the actions the  agency took to address them and (2) the extent to which DOE requires  that project management lessons learned from MOX and other projects  be documented and shared.

To examine when NNSA’s project management oversight processes  recognized cost and schedule problems at the MOX project and to  examine what actions the agency took to address them, we reviewed  DOE and NNSA documents outlining the agencies’ overall direction for,  and oversight of, the MOX project. In addition, we reviewed DOE’s,  NNSA’s, and MOX Services, LLC’s (MOX Services) documents, as well  as independent reviews and assessments, concerning the performance  and status of the MOX project, including a May 2014 report prepared for  DOE that identified and analyzed the root causes behind the cost  increases for the MOX project through 2012. Additionally, we  interviewed officials from DOE and NNSA to discuss how and when they  identified the MOX project’s cost and schedule problems. We conducted  a site visit at the Savannah River Site to tour the MOX project before the  project was terminated and interviewed officials from NNSA’s MOX  Project Management Office and representatives from MOX Services. We  also monitored the status of the MOX project.

To examine the extent to which DOE requires that project management  lessons learned from MOX and other projects be documented and  shared, we reviewed DOE’s Order 413.3B, which outlines the primary  set of project management requirements governing DOE’s and NNSA’s  capital asset projects that have a total project cost of greater than $50  million. We also reviewed documents from NNSA and DOE’s Offices of  Environmental Management and Science. In addition, we collected  examples of capital asset project-management lessons learned from DOE  and NNSA, including those from the MOX project, from a variety of  sources. Further, we reviewed reports by us, the U.S. Army’s Center  for Army Lessons Learned, and the Project Management Institute that  identify and discuss key practices for lessons learned. We then compared  the project management lessons-learned requirements in DOE Order  413.3B against these key practices. We also discussed project  management lessons-learned requirements and processes with officials  from DOE’s Offices of Environmental Management, Project Management,  and Science and NNSA’s Office of Acquisition and Project Management.  See appendix I for additional information on our objectives, scope, and  methodology.

We conducted this performance audit from May 2017 to December 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

This section discusses (1) the history and status of the MOX project; (2)  the roles of DOE, NNSA, and the contractor in managing and overseeing  the MOX project; (3) project management lessons learned; and (4) DOE’s  and NNSA’s recurring project management problems.

DOE began the MOX project over 20 years ago, in 1997, as part of a  strategy to manage the disposition of large quantities of surplus,  weapons-grade plutonium no longer needed for defense purposes. This  strategy, now undertaken through NNSA’s Plutonium Disposition  program, originally planned to dispose of the plutonium through a dual  approach—(1) conversion into mixed-oxide fuel and (2) immobilization in  glass or ceramic material—but NNSA later cancelled the immobilization  approach in favor of the approach for only mixed-oxide fuel. In 1999,  DOE awarded a contract to design, construct, and operate a MOX facility  to the contractor consortium of Duke, Cogema, Stone & Webster, LLC— now called MOX Services, LLC (MOX Services).

In February 2002, NNSA reported to Congress that the construction of the  MOX project would begin in fiscal year 2004, with operations set to begin  in fiscal year 2007, and cost nearly $1 billion to design and construct.  However, as figure 1 shows, construction of the MOX project did not  begin until 2007 after DOE formally approved the project’s estimated cost  of about $4.8 billion and estimated completion date of September 2016.

In December 2008, DOE approved a revised cost estimate for completing  construction of the MOX project of $4.9 billion and a 1-month delay in the  start of operations to October 2016. From 2009 through 2011, the  estimated cost to complete construction of the MOX project remained at  $4.9 billion. However, the MOX project’s cost and schedule estimate  changed significantly in 2012. That year, at NNSA’s direction to update  the estimate, the MOX contractor submitted a proposal to increase the  cost of the facility to about $7.7 billion—an increase of about $2.8 billion  from the 2008 estimate—with the start of operations delayed by about 3  years, to November 2019.

After receiving the MOX contractor’s revised estimate that indicated  significant cost increases and schedule delays to the project, NNSA  stated in its fiscal year 2014 budget request that pursuing the MOX  approach might be unaffordable and proposed to slow down construction  while the agency assessed alternative approaches for plutonium  disposition. After a series of reviews, DOE ultimately concluded that  pursuing an alternative disposition approach—referred to as “dilute and  dispose”—could significantly reduce the life-cycle cost of the Plutonium  Disposition program, compared with continuing the program using the  MOX approach. Following the identification of a potentially less costly  approach to plutonium disposition, in February 2016, DOE’s fiscal year  2017 budget request proposed terminating the MOX project in favor of  pursuing the dilute and dispose approach. Congress appropriated  funding for the MOX project for fiscal years 2017 and 2018 and directed  DOE to continue work on the project.

In August 2016, DOE issued a revised cost estimate of approximately  $17.2 billion to complete construction of the MOX project by 2048. In the  face of this significant cost increase, the National Defense Authorization  Act for Fiscal Year 2018 authorized the Secretary of Energy to terminate  the MOX project if, among other things, he could certify that the remaining  life-cycle cost for an alternative option for carrying out plutonium  disposition would be less than approximately half of the estimated  remaining life-cycle cost of carrying out the MOX project. In May 2018,  DOE completed this certification and notified Congress of its intention to  terminate construction of the MOX project and to instead pursue the dilute  and dispose option. The Secretary of Energy reported that the life-cycle  cost estimate was $19.9 billion for the dilute and dispose option  compared to $49.4 billion for the MOX project. In October 2018, NNSA  terminated the project. Additional information on the history and status of  the MOX project is in appendix II.

Roles of DOE, NNSA, and the Contractor in Managing the MOX Project

DOE and NNSA are responsible for providing overall direction to, and  oversight of, the contractor for the MOX project. The contractor, MOX  Services, is responsible for the design, construction, and operation of the  MOX facility.

DOE. The Office of Project Management participates in a number of  the MOX project’s oversight activities. In particular, the office has led  independent reviews of the MOX project to validate its cost and  schedule estimates and has conducted certification and surveillance  reviews of the MOX contractor’s earned value management (EVM)  system.

NNSA. Subsequent to its establishment in 2000, several NNSA  offices have provided overall direction to, and oversight of, the  contractor for the MOX project, including the Office of Fissile Materials  Disposition and the Office of Defense Nuclear Nonproliferation. In  November 2011, after starting to place increased emphasis on  improving its management of projects, the newly created Office of  Acquisition and Project Management began providing overall direction  to, and oversight of, the contractor for the MOX project. In March  2013, the Office of Acquisition and Project Management established  the NNSA MOX Project Management Office at the Savannah River  Site to lead the onsite project and contract management direction,  administration, and oversight of the MOX project.

MOX Services. As the contractor for the MOX project, MOX Services  is responsible for designing, constructing, and operating the MOX  facility. MOX Services has also subcontracted work to complete  certain construction activities, such as the fabrication of specific types  of equipment, including the complex gloveboxes needed for handling  plutonium and the heating, ventilation, and air conditioning systems.

Figure 2 depicts the roles of, and interrelation among and between, DOE,  NNSA, and the MOX contractor in overseeing the MOX project.

According to key practices that we and others have identified for both  program and project management, it is important to identify and apply  lessons learned from programs, projects, and missions to limit the chance  of recurrence of previous failures or difficulties. As such, the use of  lessons learned—such as project management lessons learned—is a  principal component of an organizational culture committed to continuous  improvement. Lessons learned, therefore, serve to communicate  knowledge more effectively and to ensure that beneficial information is  factored into planning, work processes, and activities. They also provide a  powerful method of sharing ideas for improving work processes, facility or  equipment design and operation, quality, and cost-effectiveness.  Moreover, as we and others have previously found, agencies can learn  lessons from an event and make decisions about when and how to use  that knowledge to change behavior. Key practices of a lessons-learned  process include collecting, analyzing, saving or archiving, and sharing  and disseminating information and knowledge gained on positive and  negative experiences (see fig. 3).

DOE and NNSA Have Faced Recurring Project Management Problems

For more than 2 decades, we and others have reported on the recurring  nature of the problems affecting DOE’s and NNSA’s ability to manage  contracts and projects effectively. Many of these problems have related to  DOE’s and NNSA’s struggles with managing projects, such as the MOX  project, within their initial cost and schedule estimates, including the  following:  In 1999, the National Academy of Science’s National Research  Council reported that recurring problems with project management  had raised questions about the credibility of DOE’s conceptual  designs and cost estimates.

In a March 2007 report, we found that 9 of 12 major projects we  reviewed—including the MOX project—had exceeded their original  cost estimates, schedule estimates, or both, principally because of  ineffective project oversight and contractor management.

In a November 2014 report, the Congressional Advisory Panel on the  Governance of the Nuclear Security Enterprise (Augustine-Mies  Panel) stated that NNSA’s inability to estimate costs and execute  projects according to plan has been a major source of dissatisfaction  among the national leadership and had significantly undermined  NNSA’s credibility.

Further, in April 2015, we found that NNSA has had a long history of  identifying corrective actions for problems and declaring them  successfully resolved, only to then identify additional actions needed to  address the problems. As we found, the recurrence of such problems  suggests that NNSA did not have a full understanding of the root causes  of its contract- and project-management challenges. Moreover, our 2017  high-risk report found that DOE had taken several important steps that  demonstrate its commitment to improving contract and project  management, but that DOE’s efforts had not fully addressed several  areas where the department continues to have shortcomings. Areas with  shortcomings include acquisition planning for major contracts and the quality of enterprise-wide cost information available to DOE managers  and key stakeholders. Additional information on our prior work  highlighting selected DOE and NNSA project management problems is in  appendix III.

NNSA Recognized Certain Indicators of Cost and Schedule Problems after Strengthening Its Oversight in 2010 and 2011

Prior to 2011, NNSA project staff had failed to recognize and fully resolve  certain cost and schedule problems that indicated that the MOX project  would not be completed on time or within its approved cost estimates.  However, after taking actions to strengthen its project management  oversight in late 2010 and 2011, NNSA recognized indicators of a number  of problems with the MOX project that contributed to NNSA’s decision to  terminate the project.

NNSA Failed to Recognize and Fully Resolve Certain Cost and Schedule Problems Affecting the MOX Project Prior to 2011

Prior to 2011, NNSA’s staff responsible for overseeing the MOX project  failed to recognize and fully resolve certain cost and schedule problems  that indicated that the project would not be completed on time or within its  approved cost estimates. The NNSA staff responsible for overseeing the  MOX project at that time were generally inexperienced in overseeing  complex nuclear construction projects. From 2007 through 2011, staff  overseeing the MOX project were primarily familiar with large  programmatic initiatives and operations but had little experience in  managing large, complex first-of-a-kind nuclear construction projects,  according to a May 2014 root cause analysis. Although information  available to the NNSA staff showed that there were cost and schedule  problems that indicated the increasing likelihood that the project would  not be completed within its approved total cost estimate of $4.9 billion, the  staff did not recognize and fully resolve four key problems.

First, information about the contractor’s use of inaccurate rates to  estimate the time needed to complete certain construction activities— commonly referred to as unit rates or planned production rates—indicated  that the project would not be completed within its approved cost estimate.  These rates are used to reflect levels of productivity during construction  and to help develop projects’ cost and schedule estimates, including  updates to annual forecasted estimates. Following the start of  construction in August 2007, the MOX contractor began to experience  lower-than-estimated productivity rates for key construction activities,  according to the May 2014 root cause analysis report. Despite this  issue, the contractor did not incorporate more realistic assumptions  regarding the unit and production rates, such as by updating the  estimated costs and time needed to complete specific construction  activities, when developing the contractor’s annual forecasted estimates  of the project’s total cost for 2008 through 2011. MOX contractor  representatives told us that the unit rates they used to develop cost and  schedule estimates were realistic based on assumptions at that time and  that DOE was involved in the development of the unit rates. In addition,  the MOX contractor’s representatives told us that expected improvements  in unit rates did not materialize because of higher than expected levels of  worker turnover.

NNSA staff overseeing the project at that time did not recognize that the  unit rates for calculating and updating unit rate estimates should be  realistic and reflect levels of productivity during construction, as called for  in project management principles, or resolve the issues. As a result, the  staff did not take action to resolve the MOX contractor’s continued use of  unrealistic unit rates that did not reflect actual construction progress being  made. Furthermore, NNSA staff did not recognize the extent to which  decreased productivity by the contractor created future cost increases and schedule delays or resolve the issue. Consequently, from 2008 to  2011, the MOX contractor continued to use its overly optimistic and  unrealistic unit rate estimates when developing its annual forecasted cost  estimates.

Second, the MOX contractor’s annual forecasted estimates for the project  consistently increased from 2008 through 2011, and the level of  confidence in those estimates decreased, indicating that the project would  not be completed within its approved cost estimate. Beginning in 2008,  the MOX contractor submitted an annual update to its forecasted estimate  for the project. These estimates increased each year, rising by about  $140 million to $280 million annually, with the estimated total project cost  increasing from about $4.1 billion in 2008 to about $4.7 billion in 2011 (an  increase of about 15 percent). The MOX contractor’s representatives said  they attempted to mitigate the increases, such as by identifying cost  savings on the project. Additionally, as the May 2014 root-cause analysis  report stated, the level of confidence for completing the MOX project  within the approved $4.9 billion total project cost estimate declined each  year, from an 85 percent likelihood of completing the project within the  estimate in 2009 to 45 percent in 2011. Both the annual increases in  forecasted estimates and the annual decline in level of confidence  illustrated the increasing likelihood that the MOX contractor would not  complete the project for $4.9 billion.

As a result of inexperience, the NNSA staff overseeing the project at that  time did not adequately examine the potential consequences of such cost  performance trends over the future schedule and through project  completion or resolve the issues. As the May 2014 root-cause analysis  report stated, NNSA staff did not fully recognize how the risks and  challenges the MOX project faced negatively affected not only the  project’s performance but also its cost and schedule. For example, that  report found that the staff were unable to determine that there were  fundamental problems with completing the MOX project’s design and with  maintaining construction efficiency and progress; both of which  contributed to schedule delays and cost increases. The May 2014 root- cause analysis report stated that because of inexperience in project  management, NNSA staff did not direct the MOX contractor to develop a  more realistic and achievable forecasted estimate for the total cost to  complete the MOX project until January 2012.

Third, information about procuring materials out of sequence and the  resulting rework indicated that the project would not be completed on  schedule or within its approved cost estimate. According to NNSA  officials, the MOX contractor’s method for measuring earned value  incentivized the contractor to purchase and procure materials early and,  in a number of cases, out of sequence, as this helped demonstrate  progress. For example, figure 4 shows outdoor “laydown yards” and an  offsite warehouse storing large amounts of commodities, such as pipes  and electrical panels, that NNSA officials said the MOX contractor  procured earlier than needed. The May 2014 root-cause analysis report  stated that between 2007 and 2011, the equipment and material procured  out of sequence resulted in the need for rework in some cases because  later design changes required changes to the equipment or the need to  procure different items, leading to additional costs for the project.

The MOX contractor’s representatives told us they disagreed with  NNSA’s characterization that they procured material too early. According  to the contractor representatives, they purchased materials in support of  both the project schedule and planned construction end date of 2016, as  well as to achieve the efficiencies through bulk pricing or reduced delivery  charges from procuring larger quantities of items or multiple items at the  same time. Additionally, the MOX contractor representatives disagreed  that they structured the methods for measuring earned value performance  to claim earned value in ways that did not reflect actual progress. In  particular, the MOX contractor representatives said that NNSA staff were  involved in the development of the original methods used for measuring  earned value.

NNSA staff did not take steps to resolve the issues with the  disproportionate value earned by the MOX contractor for purchasing,  procuring, and placing certain commodities until 2015 when the MOX  contractor revised its methods for measuring earned value.  Consequently, the reported commodity installation data based on the  MOX contractor’s methods for measuring claimed earned value inflated  the amount of progress being made on the construction of the MOX  project compared with the amount of work completed.

Fourth, information about the use of management reserve funds early in  the project indicated that the project would not be completed within its  approved baseline. To address cost increases experienced early in the  project, the MOX contractor began to use the project’s management  reserve funds. A May 2010 surveillance review of the MOX contractor’s  EVM system prepared for DOE by an independent contractor identified  this issue and concluded that the rate at which the MOX contractor was  using its management reserve indicated that it was unlikely that there  would be any reserve left to address any risks that were expected to be  encountered later in the project. DOE’s June 2011 follow-up review of  the MOX contractor’s EVM system found that the MOX contractor was no  longer covering cost variances by using management reserve; however,  the MOX contractor’s previous use of management reserve to cover cost  overruns had resulted in inaccurate, inflated cost performance and  understated forecasted cost estimates. The MOX contractor’s  representatives told us they disagreed with the premise that the  management reserve was used to obscure cost performance. Moreover,  they noted that NNSA’s cost-accounting and management staff worked  with the contractor on all EVM issues, including the use of management  reserve.

NNSA staff did not recognize and resolve issues with the contractor’s use  of the management reserve to mitigate cost overruns or the effect on the  project’s cost performance and forecasted cost estimates in part because,  as the May 2014 root cause analysis report stated, the staff possessed  little experience in project management. According to project  management principles, management reserve should be prevented from  being consumed too early so as to ensure that enough reserve remains  available to address any problems that may arise late in the project. The  inexperienced NNSA staff also did not recognize that certain problems  were creating cost overruns because, as stated in the May 2010  surveillance review, the MOX contractor’s use of the management  reserve to cover such overruns hid the problems and did not alleviate  their root causes. As a result of not recognizing or resolving the MOX  contractor’s inappropriate use of the management reserve earlier, NNSA  reported inaccurate measurements of cost performance to DOE and other  stakeholders.

DOE’s Project Management Changes Strengthened Oversight of the MOX Project

In late 2010 and 2011, DOE began to implement actions to strengthen  project management across the department, including NNSA. These  actions, which agency officials said were primarily undertaken in  response to project management problems we and others had identified,  contributed to the steps NNSA began to take to strengthen its project  management and oversight of the MOX project. Changes that  strengthened NNSA’s oversight of the MOX project included: (1) initiating  project peer reviews and (2) making several organizational changes to  improve project oversight. These changes to DOE’s and NNSA’s  oversight of the MOX project contributed to the decision to terminate the  project.

First, in its November 2010 update to requirements for capital asset  projects, DOE established a requirement to conduct peer reviews at least  once a year for large or high-visibility projects with a total project cost of  $100 million or greater. The update required peer reviews more frequently  for complex projects or those experiencing performance challenges.  According to DOE and NNSA officials, they added the requirement in  response to a recommendation in our May 2008 report. According to  NNSA officials, as a result of this requirement, NNSA began conducting  peer reviews of the MOX project in 2011. These reviews led NNSA to  identify significant cost and schedule problems at the MOX project and  included a number of recommendations to improve project performance.  For example, a March 2012 NNSA peer review found that the MOX  project’s total cost may have been understated by anywhere from $600  million to $900 million, in part because the contractor’s estimated unit  rates and planned production rates were not reflective of the actual  performance at that time. Moreover, the peer review found that the  estimated completion date of October 2016 was also at risk. As a result,  the peer review team recommended, among other things, that the MOX  contractor develop an update to its formal cost and schedule estimate.

As a result of the findings and recommendations from its peer reviews,  NNSA requested and the MOX contractor submitted in September 2012 a  proposal that included a revised cost estimate for the MOX project of  about $7.7 billion and an estimated completion date of November 2019.  In response to the significant cost increases, schedule delays, and project  risks captured in the MOX contractor’s updated cost and schedule  estimate, NNSA proposed a slowdown of MOX project construction  activities in its fiscal year 2014 budget request to begin assessing  alternative plutonium disposition strategies.

Second, NNSA carried out several organizational changes starting in  2011 that led to improved oversight of the MOX project in some areas and the continued identification of cost and schedule problems.  Specifically, NNSA transitioned management and oversight of the MOX  project from the Office of Defense Nuclear Nonproliferation to the Office  of Acquisition and Project Management, an office newly created in  January 2011 to improve project oversight through the application of  project management principles. In 2013, the Office of Acquisition and  Project Management created the MOX Project Management Office at the  Savannah River Site to provide project and contract management  oversight for the MOX project.

After establishing the MOX Project Management Office, the Office of  Acquisition and Project Management sought to better address long- standing staffing challenges. For example, a May 2006 external  independent review conducted for DOE found that, among other things,  NNSA understaffed the oversight of the MOX project and recommended  that DOE acquire sufficient personnel with the proper skills to manage  and perform oversight of the project. However, NNSA did not address  this issue until after the creation of the Office of Acquisition and Project  Management. The Office of Acquisition and Project Management  increased the number of staff with specific project management skillsets  at the MOX Project Management Office from 20 for fiscal years 2010 to  2012 to 36 (18 federal employees and 18 support service contractors) for  fiscal years 2016 to 2018.

As a result of the staffing changes, the NNSA MOX Project Management  Office strengthened its oversight of the MOX project, which contributed to  the identification of additional problems, as described below.

Conducted more in-depth assessments of the MOX contractor’s  EVM system. After initially certifying the MOX contractor’s EVM  system in May 2008, a May 2010 surveillance review of the MOX  contractor’s EVM system prepared for DOE by an independent  contractor identified a number of issues. The MOX contractor  addressed the issues, according to DOE’s June 2011 review, resulting  in the recertification of the EVM system at that time. According to  NNSA officials, NNSA’s MOX Project Management Office conducted  more in-depth assessments of the MOX contractor’s EVM system  starting in 2013. These assessments led NNSA staff to identify a  number of concerns with the contractor’s EVM system, such as  earned value data errors; overstatements of the data on the  percentage of work completed in certain areas; and in one instance,  about $300 million in known cost growth that was not incorporated  into the MOX project’s forecasted estimate of total project cost.

According to NNSA officials, in March 2016, the NNSA federal project  director requested an in-depth review of the contractor’s EVM system  because of the continued identification of issues with the system, and  the MOX contractor not adequately addressing them. According to its  October 2016 review, DOE’s Office of Project Management identified  significant deficiencies representing systematic and material internal  control weaknesses and concluded that the MOX contractor’s EVM  system could not be relied upon to provide credible and reliable cost  and schedule performance data for either the project’s current status  or its forecasted cost and schedule estimates. As a result, DOE’s  Office of Project Management rescinded the MOX contractor’s EVM  system certification because the system was no longer in compliance  with the relevant standards.

Implemented a more rigorous invoice review process. According  to NNSA officials, prior to 2014, NNSA did not have a rigorous  process in place to review the contractor’s invoices. The officials said  that NNSA staff did not review all invoices and, for the reviews that  were completed, they did not always thoroughly examine the details  behind the invoices, such as reviewing invoices to verify that costs  were allowable under DOE regulations. The NNSA officials told us  that as part of their efforts to improve oversight of the MOX  contractor’s invoice submissions, NNSA’s MOX Project Management  Office staff developed a more rigorous invoice review process that  resulted in a September 2014 guide. In addition, the NNSA MOX  Project Management Office assigned an additional staff member to (1)  help conduct invoice reviews due to the volume of work needed to  review the MOX contractor’s invoices and (2) ensure that payments  were made within the 14 days generally required by regulation.  According to NNSA officials, as a result of the changes implemented  by the office, NNSA identified a number of potentially unallowable  costs ranging from less than $1,000 to more than $2 million.

Reviewed the MOX contractor’s annual incurred costs. NNSA  officials said that incurred cost audits were supposed to be conducted  at least annually for the MOX project and that the Defense Contract  Audit Agency was supposed to conduct the audits. However, these  officials explained that due to a significant backlog, the Defense  Contract Audit Agency did not complete all of the required audits. In  light of the Defense Contract Audit Agency’s significant backlog—as  well as a requirement prohibiting the agency from conducting non- defense agency audits—the NNSA MOX Project Management Office  arranged to have a third party conduct an audit of the MOX  contractor’s fiscal year 2010 incurred costs. This third-party audit  identified more than $30 million in potentially unallowable costs.

The significant cost and schedule problems that NNSA staff identified  after strengthening its oversight of the MOX project contributed to NNSA’s  decision to terminate it. Project management principles state that effective  project management helps organizations to, among other things, increase  the chances of success; resolve problems and issues; and identify,  recover, or terminate failing projects. After NNSA’s project peer reviews  and the MOX contractor’s proposed update to the project’s cost and  schedule estimate showed the significant likelihood of additional cost  growth and schedule delays, NNSA proposed slowing down construction  of the MOX facility in 2013 and ultimately terminated the project in  October 2018.

DOE Has Requirements for Documenting and Sharing Lessons Learned, but They Do Not Ensure Consistent or Timely Documentation or the Evaluation of Corrective Actions

As outlined in DOE Order 413.3B, DOE requires that project management  staff document and share project management lessons learned on capital  asset projects like MOX but does not require that all project management  lessons learned from capital asset projects be documented consistently  or shared in a timely manner. Moreover, DOE Order 413.3B does not  require the evaluation of the results of corrective actions taken in  response to lessons learned that are identified during the course of  capital asset projects such as the MOX project to ensure that the  problems experienced are resolved department-wide.

DOE’s Requirements for Documenting and Sharing Lessons Learned for Capital Asset Projects

DOE’s requirements for capital asset projects, as outlined in Order  413.3B, specify that project management lessons learned should be  captured—that is, documented—throughout the continuum of a project.  According to the order, there are five critical decisions (CD) that structure  the life of a project. The CDs, which are summarized in figure 5, include  approving: mission need (CD-0); alternative selection and cost range  (CD-1); project performance baseline (CD-2); the start of construction or  execution (CD-3); and the start of operations or project completion (CD- 4). DOE Order 413.3B requires project staff to submit project  management lessons learned to DOE’s Office of Project Management  within 90 days of two critical decision points: (1) upfront planning and  design lessons learned are to be submitted within 90 days of CD-3  approval and (2) project execution and facility startup lessons learned are  to be submitted within 90 days of CD-4 approval.

DOE Order 413.3B also requires that lessons learned for capital asset  projects be collected, analyzed, and disseminated by project  management support offices. These offices consist of DOE or NNSA staff  who provide support to federal project directors and are established  exclusively to oversee and manage the activities associated with projects.  Additionally, DOE Order 413.3B states that the Project Management Risk  Committee should support project management activities within DOE by  enabling the sharing of lessons learned on a routine basis.

DOE and NNSA officials told us that program and project offices  document and save project management lessons learned for capital asset  projects in different ways. In particular, DOE and NNSA officials told us  that peer reviews, which are saved in DOE’s Project Assessment and  Reporting System (PARS II) database, are a primary source of project  management lessons learned. The officials also said that project  management lessons learned are saved through monthly project reports,  monthly staff meetings, Project Management Risk Committee meeting  notes, and project management workshops and training courses. In  addition, DOE and NNSA officials told us that some lessons learned are  shared through informal person-to-person discussions that allow lessons  learned to be shared among staff. Further, the officials said that they  address project management problems identified in lessons learned by  making changes to DOE Order 413.3B.

In addition, while not required, DOE may capture some lessons learned  for projects during the project review process. For example, DOE’s  standard-operating procedures for conducting external independent  reviews state that the scope of such reviews can include assessing  whether project teams are documenting and sharing lessons learned from  their projects internally and externally. However, as noted in the  standard-operating procedures, this is an example of an area that can be  included as part of an external independent review, although there is no  requirement to do so.

DOE’s Lessons-Learned Requirements for Capital Asset Projects Do Not Ensure Consistent or Timely Documentation and Sharing or the Evaluation of Corrective Actions

DOE Order 413.3B requires project management lessons learned for  capital asset projects to be documented throughout the life of a project  but does not specifically require lessons learned to be documented and  saved in a consistent manner or shared routinely or in a timely manner.  Moreover, the order does not require all corrective actions related to  these lessons learned to be evaluated for effectiveness.

DOE Does Not Require That Lessons Learned for Capital Asset Projects Be Documented and Saved Consistently

Although DOE and NNSA use multiple means to document and save  lessons learned, we found DOE and NNSA program and project offices  do not document and save such lessons consistently so that they are  readily accessible by other staff. For example, NNSA uses an internal  database to save project management lessons learned for its projects.  However, NNSA officials told us that DOE staff outside of NNSA must  request access to the database before they can read and examine the lessons learned that are documented and saved in the database. Officials  from DOE’s Office of Science told us that their office submits some  lessons learned to the PARS II database and maintains some project  management lessons-learned reports on a publicly available webpage. A  senior official from DOE’s Office of Environmental Management told us  that some lessons learned from its projects are sent to its staff through  monthly lessons-learned bulletins, but the bulletins are not entered into  PARS II. In addition, DOE and NNSA officials said that project staff can  enter specific lessons learned gleaned from their project in a lessons- learned repository within PARS II. For example, as of November 2017,  PARS II contained 20 entries for project management lessons learned  from the MOX project.

According to key practices for lessons learned identified by us and the  Center for Army Lessons Learned, a central component of a successful  lessons-learned process is to ensure that lessons learned are stored in a  logical, organized manner. Specifically, as we have previously found,  lessons learned should be stored in a manner—such as an electronic  database—that allows users to perform information searches using key  words and functional categories. Moreover, information in the database  should be updated regularly and provide a logical system for organizing  information that is easily retrievable and made available to any requester.  We have also found that relying on person-to-person discussions to share  lessons learned can be problematic because personal networks can  dissolve—for example, through attrition or retirement—and informal  information sharing does not ensure everyone is benefiting from the  lessons that are gleaned. Further, by not documenting and saving all  lessons learned (e.g., those shared through person-to-person  exchanges), there is also generally no way to ensure the validation of the  information shared. This is not consistent with the key practice from the  Center for Army Lessons Learned, which states that by documenting and  saving project management lessons learned in a logical, organized  manner such as an electronic database, lessons learned can be archived,  managed, and made available for review by other projects and applied to  them at a future date.

Because DOE Order 413.3B does not indicate where all project  management lessons learned should be documented and saved in a  consistent manner, the department cannot ensure that future capital asset  projects will be able to take advantage of experiences from past projects.  We found that DOE and NNSA did not document all lessons learned in a  consistent manner, and DOE officials acknowledged that DOE Order  413.3B does not require documenting or saving lessons learned that are  presented through various formal or informal means in a common  location. By developing requirements that clearly define how and where  all project management lessons learned should be documented and  saved to make them readily accessible across the department, such as in  a database, DOE—including NNSA—could improve the agency’s existing  lessons-learned process.

DOE Does Not Require That Lessons Learned for Capital Asset Projects Be Submitted and Shared Routinely or in a Timely Manner

DOE Order 413.3B’s requirements for project management lessons  learned do not require that all lessons learned be shared routinely or in a  timely manner. In particular, the order does not require that lessons  learned be submitted and shared routinely until CD-3—the start of  construction. Consequently, DOE and NNSA staff are not required to  submit lessons learned during the CD-0, CD-1, and CD-2 phases of a  project. These earlier phases, which involve upfront planning and design  for the selected project, often occur many years before the approval and  start of construction. Notably, both the MOX and Uranium Processing  Facility (UPF) projects took about 10 years to reach the start of  construction (CD-3) and experienced cost increases and schedule  delays.

We and others have previously found that lessons learned should be  submitted in a timely manner so as to ensure that key information is  available to identify and address problems or incorporate successful  activities as early and quickly in the process as possible. For example, we  found that lessons-learned reports (i.e., reports documenting lessons- learned reviews) should be prepared promptly so that knowledgeable  personnel are available to contribute to the reports, important details are  recalled accurately, and there are no delays in the dissemination of  lessons learned. Moreover, according to the Center for Army Lessons  Learned, the guiding principle in executing a sharing strategy for lessons  learned is to get the right information to the right person at the right  time. Such a strategy can entail developing a process for creating  timelines for sharing lessons learned that are tied to the urgency of the  information and a means to disseminate that information.

Because DOE Order 413.3B does not require lessons learned to be  submitted prior to CD-3, the department is limiting its ability to promptly  evaluate and address early issues with projects and apply such lessons  learned to other projects department-wide. This approach could affect the  successful completion of capital asset projects, particularly those that  experience prolonged upfront planning and design phases similar to  those the MOX and UPF projects experienced. By developing  requirements for sharing project management lessons learned from early  in the CD phases of projects (i.e., prior to CD-3) routinely and in a timely  manner to improve the ability to identify and evaluate problematic  practices and positive experiences, DOE—including NNSA—could help  improve the success of future capital asset projects and avoid the  problems encountered overseeing the MOX project.

DOE Does Not Require the Evaluation of the Effectiveness of Corrective Actions Taken

DOE Order 413.3B does not require the evaluation of the results of  corrective actions taken to address project management lessons learned  that are identified during the course of capital asset projects such as  MOX. According to DOE guidance and statements, officials track whether  lessons identified through reviews or other efforts are implemented. For  example, according to DOE’s standard-operating procedures for  conducting external independent reviews and officials from DOE’s Office  of Project Management, DOE staff conducting external independent  reviews of projects should assess whether project teams are reviewing  and incorporating applicable lessons learned. In addition, DOE project  management officials told us that peer review recommendations and the  corrective actions to be taken to address them are tracked until the  closure of each recommendation. However, DOE has not evaluated  whether corrective actions taken have led to the resolution of the  problematic practices identified in the lessons learned because DOE  Order 413.3B does not require this type of evaluation.

According to key practices for lessons learned identified by the Center for  Army Lessons Learned and us, a central component of a successful  lessons-learned process is to establish a means to ensure that issues are  being resolved as intended. The Center for Army Lessons Learned states  that while not all issues require a formal process to resolve, there should be a process in place to identify and prioritize the most important things  that need to be fixed. For example, this process could entail addressing  only those problems that may necessitate the need for department-wide  improvements, as some issues may be narrowly focused and be specific  to one project or site. The Center for Army Lessons Learned further  states that an organization’s ability to change behavior by implementing a  lesson is ineffective unless the organization observes changes in  behavior and verifies that the lesson is learned. Additionally, we have  found that if agency management decides to take action to apply an  identified lesson, then it should take subsequent action to observe that  the change in behavior actually occurred and collect additional  information to verify that the change had the desired effect.

Although DOE Order 413.3B does not require DOE to evaluate the  effectiveness of corrective actions other than those associated with peer  reviews, other DOE orders and guidance require the evaluation of the  effectiveness of other types of corrective actions. For example, DOE  Order 226.1B requires that DOE’s organizations and contractors  implement oversight processes that ensure they evaluate and correct  relevant quality assurance problems on a timely basis to prevent their  recurrence. In addition, DOE’s order and guide for implementing an  effective quality assurance program highlight the importance of  undertaking corrective actions to prevent the recurrence of problems,  including determining the effectiveness of the corrective actions for  significant problems. By developing requirements for evaluating the  effectiveness of corrective actions taken in response to project  management problems in capital asset projects, particularly those that necessitate the need for department-wide improvements, DOE—including  NNSA—could verify that changes made as a result of lessons learned  had the intended outcome as the agency does for contractors.

DOE and NNSA made changes that strengthened oversight of large  capital asset projects. These changes helped NNSA better identify cost  and schedule problems affecting the MOX project and contributed to  NNSA’s decision to ultimately terminate the project.

DOE’s Order 413.3B includes certain requirements for documenting and  sharing project management lessons learned. However, the requirements  in DOE Order 413.3B do not fully incorporate several key practices for  lessons learned. For example, the order does not require that DOE or  NNSA document project management lessons learned for capital asset  projects consistently or that such lessons learned are shared in a timely  manner. By developing requirements that clearly define how and where  all project management lessons learned should be documented and  saved to make them readily accessible across the department, such as in  a database, DOE—including NNSA—could improve the existing lessons- learned process and enable future projects across the department to take  advantage of experiences from past projects.

In addition, because DOE Order 413.3B does not require lessons learned  for capital asset projects to be submitted prior to the start of construction  (CD-3), the department is limiting its ability to promptly evaluate and  address early issues with projects as well as applying such lessons  learned to other projects department-wide. By developing requirements  for sharing project management lessons learned from the beginning of a  project routinely and in a timely manner to improve DOE’s ability to  identify and evaluate problematic practices and positive experiences,  DOE—including NNSA—could help improve the success of future capital  asset projects and avoid the problems the agency encountered on the  MOX project.

Moreover, while DOE tracks the implementation of certain project  management lessons learned for capital asset projects, DOE Order  413.3B does not require that DOE—including NNSA—evaluate corrective  actions identified outside the peer review process and taken in response  to lessons identified to verify that the changes made had the desired  effect. By developing requirements for evaluating the effectiveness of  corrective actions taken in response to project management problems in  capital asset projects, particularly those that necessitate the need for  department-wide improvements, DOE could verify that changes made as  a result of lessons learned had the intended outcome as the agency does  for contractors.

We are making the following three recommendations to DOE:

The Secretary of Energy, in coordination with DOE’s Office of Project  Management and NNSA’s Office of Acquisition and Project  Management, should develop requirements that clearly define how  and where project management lessons learned for capital asset  projects should be documented and saved to make them readily  accessible across the department. (Recommendation 1)

The Secretary of Energy, in coordination with DOE’s Office of Project  Management and NNSA’s Office of Acquisition and Project  Management, should develop requirements for sharing project  management lessons learned for capital asset projects from the  beginning of a project (i.e., prior to the start of construction at CD-3)  routinely and in a timely manner to improve DOE’s ability to identify  and evaluate problematic practices and positive experiences.  (Recommendation 2)

The Secretary of Energy, in coordination with DOE’s Office of Project  Management and NNSA’s Office of Acquisition and Project  Management, should develop requirements for evaluating the  effectiveness of corrective actions taken in response to project  management problems for capital asset projects, with a focus on  those lessons that necessitate the need for department-wide  improvements. (Recommendation 3)

Agency Comments, Third-Party Views, and Our Evaluation

We provided a draft of this report to DOE, NNSA, and MOX Services for  review and comment. In written comments, which are reproduced in full in  appendix IV, DOE concurred with the report’s recommendations and  described actions that it intends to take in response to our  recommendations.

In response to our first recommendation, DOE intends to issue a policy  memorandum by December 2019 and revise DOE Order 413.3B to  identify the project management lessons learned repository and outline  the kinds of information the repository will collect. In response to our  second recommendation, DOE intends to issue a policy memorandum by  December 2019 and revise DOE Order 413.3B to collect lessons learned  as part of its peer review process. Because DOE Order 413.3B requires  that peer reviews for projects of $100 million or greater be conducted  once between CD-0 and CD-1, annually between CD-1 and CD-2, at least  annually between CD-2 and CD-4, and more frequently for the most  complex projects or those experiencing performance challenges, this  action is responsive to our recommendation and should help DOE begin  to identify lessons learned in a more routine and timely manner. In  response to our third recommendation, DOE plans to revise the Project  Management Risk Committee charter by assigning it the responsibility to  qualitatively evaluate the effectiveness of corrective actions taken in  response to project management lessons learned from projects with a  total cost greater than $750 million having department-wide implications.  We are encouraged that DOE agrees with our recommendation and view  this change as a positive first step. However, this action may not fully  address the recommendation. For example, the planned action states that  the Project Management Risk Committee would evaluate the  effectiveness of corrective actions for projects with total costs of $750  million or more, but there may be some lessons learned with applicability  department-wide from projects that do not meet this cost threshold.  Additionally, DOE’s planned action as described in its response does not  discuss who would be responsible for evaluating the effectiveness of  corrective actions or a timeline for performing the assessments. The  Project Management Risk Committee has typically served as a review  group and has not itself performed such evaluations.

DOE and MOX Services also provided technical comments, which we  incorporated in our report as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Energy, the Administrator of NNSA, and  other interested parties. In addition, the report is available at no charge on  the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.

Our report examined (1) when the National Nuclear Security  Administration’s (NNSA) project management oversight processes  recognized cost and schedule problems at the Mixed Oxide Fuel  Fabrication Facility (MOX) project and the actions the agency took to  address them and (2) the extent to which the Department of Energy  (DOE) requires that project management lessons learned from MOX and  other projects be documented and shared.

To address both objectives, we reviewed relevant documents from DOE,  NNSA, and MOX Services, LLC (MOX Services), the contractor  constructing the MOX project. We reviewed past reports by GAO and the  National Academy of Sciences’ National Research Council to examine  previously identified weaknesses in DOE project management, contractor  performance, and federal oversight of individual projects, as well as  DOE’s efforts to make improvements. We also reviewed DOE reports  focused on analyzing the root causes of contract- and project- management issues affecting DOE and NNSA and identifying potential  corrective actions and other general improvements. We visited the  Savannah River Site to tour the MOX project while it was under  construction and interviewed officials from NNSA’s MOX Project  Management Office, including the federal project director, and  representatives from MOX Services. We also monitored the status of the  MOX project.

To examine when NNSA’s project management oversight processes  recognized cost and schedule problems at the MOX project and the  actions the agency took to address them, we identified and reviewed  DOE and NNSA documents outlining the agencies’ management and  oversight roles and responsibilities and the processes the agencies used  to monitor the cost and schedule of the MOX project. We also examined  NNSA guidance and memorandums detailing the 2011 transition of  oversight responsibilities for the construction of the MOX project from  NNSA’s Office of Defense Nuclear Nonproliferation to its Office of  Acquisition and Project Management and the effect this change had on  NNSA’s efforts to oversee the project. In addition, we reviewed DOE,  NNSA, and MOX Services documents, as well as independent reviews  and assessments, concerning the performance and status of the MOX  project. In particular, we reviewed a May 2014 report prepared for DOE  that identified and analyzed the root causes behind the cost increases  that affected the MOX project through 2012, after the formal approval of  its cost and schedule estimates in 2007. We also reviewed surveillance  reviews and a May 2013 assessment of the MOX contractor’s earned  value management (EVM) system, which the contractor and NNSA used  to monitor project performance and status, including cost and schedule,  after construction began. Moreover, we examined project cost and  budget information that DOE, NNSA, MOX Services, and others  developed—such as the contractor’s September 2012 baseline change  proposal and DOE’s August 2016 revised cost and schedule estimate—to  determine when they began to identify the MOX project’s cost increases  and schedule delays and why such problems might have occurred. We  also reviewed reports by GAO and DOE’s Office of Inspector General that  identified and discussed cost and schedule problems affecting the MOX  project. Additionally, we interviewed officials from DOE and NNSA to  discuss how and when they identified the MOX project’s cost and  schedule problems.

To examine the extent to which DOE requires that project management  lessons learned from MOX and other projects be documented and  shared, we reviewed DOE’s Order 413.3B, which outlines the primary  set of project management requirements governing DOE and NNSA  capital asset projects that have a total project cost of greater than $50  million. We also reviewed DOE guidance documents, such as those  related to DOE Order 413.3B, to further understand DOE’s suggested  approaches for meeting its existing lessons learned requirements.  Similarly, we reviewed documents from NNSA and DOE’s Offices of  Environmental Management and Science, such as those found in  business-operating procedures and standard-operating policies and  procedures, to examine how those documents supplement the lessons  learned requirements included in DOE Order 413.3B. In addition, we  collected examples of capital asset project-management lessons learned  from DOE and NNSA, including those from the MOX project, from a  variety of sources, such as lessons-learned reports, project peer reviews,  entries stored in DOE’s Project Assessment and Reporting System  (PARS II) and NNSA’s internal databases, monthly lessons-learned  bulletins, and presentations, among others. To better understand lessons  learned and their role within project management, we reviewed reports by  GAO, the U.S. Army’s Center for Army Lessons Learned, and the  Project Management Institute that identify and discuss key practices for  lessons learned. We selected these sources because they are widely  recognized for key practices on lessons learned. We then compared the  project management lessons learned requirements outlined in DOE Order  413.3B against these key practices. We also discussed project  management lessons learned requirements and processes with officials  from DOE’s Offices of Environmental Management, Project Management,  and Science and NNSA’s Office of Acquisition and Project Management.

We conducted this performance audit from May 2017 to December 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Timeline of Selected Information and Events Pertaining to the MOX Project

Appendix II: Timeline of Selected Information  and Events Pertaining to the MOX Project Details  DOE announced a plan to dispose of surplus, weapons-grade plutonium through a dual  approach that would include constructing a facility for the purposes of converting the  plutonium into mixed-oxide fuel for use in modified commercial reactors.  The initial estimate for the MOX project—that is, not an approved baseline—totaled $1.4  billion, with completion of construction expected to be in September 2004.

DOE awarded the contract for designing, constructing, and operating a MOX facility to the  consortium of Duke, Cogema, Stone & Webster, LLC—now MOX Services, LLC, or MOX  Services.  According to a December 2005 DOE Inspector General report, in 1999, an independent team  reviewed the MOX contract and warned of the potential for escalating costs because the  contractor had no incentives to minimize costs nor penalties for overruns or poor  performance.

DOE announced that it would construct the MOX project (as well as two other facilities) at the  Savannah River Site located in Aiken, South Carolina.

A February 2001 independent cost estimate of the MOX contractor’s preliminary cost estimate  for the MOX project concluded that it would cost about $2.4 billion to construct and operate it.  The independent cost estimate concluded that it would cost about $1.1 billion to construct the  facility.

The National Nuclear Security Administration’s (NNSA) February 2002 report to Congress on  the disposition of surplus defense plutonium at the Savannah River Site concluded that the  facility component of the mixed-oxide fuel option identified would cost about $2.2 billion to  implement over about 20 years. According to the report, about $1 billion of these costs would  be for designing and constructing the facility, with construction being completed during fiscal  year 2007.

According to DOE’s fiscal year 2004 budget request, a preliminary estimate of the MOX  project’s total cost totaled about $1.8 billion.

A July 2004 independent review found that the MOX project had experienced a cost increase  of about 300 percent for the design and development phase compared to what was  preliminarily planned for in 1999, in part to due to a number factors, including design changes  and underestimates. Moreover, the report cited the MOX project as an example of a DOE  project greater than $500 million that should have had an approved performance baseline  many years prior given that it had reached critical decision (CD)-1 approval, or the approval of  alternative selection and cost range, in 1997.

According to a December 2005 report by DOE’s Office of Inspector General, as of July 2005,  NNSA’s not-yet-validated estimate for the design and construction of the MOX project was  about $3.5 billion ($2.8 billion for construction).

Details  In February 2006, DOE’s fiscal year 2007 budget request reported a preliminary estimate for  the MOX project totaling about $3.6 billion, but the department reiterated that the estimate  would be finalized following the completion of the project’s performance baseline. The request  also noted that design costs for the MOX project increased from $243 million to $765 million,  primarily due to the decision to fund some design work for gloveboxes and enhanced aqueous  polishing during the design phase as opposed to the construction phase and increased design  work to adapt the facility to handle and treat several tons of pure plutonium resulting from the  cancelation of plutonium immobilization, which would have entailed incorporating plutonium  into a corrosion-resistant ceramic matrix and then encasing the immobilized plutonium in  glass along with highly radioactive nuclear wastes that already existed at DOE sites, thereby  rendering the plutonium as inaccessible and unattractive for reuse in nuclear weapons.  However, NNSA canceled this approach in 2002.  A July 2006 external independent review of the MOX project’s preliminary cost and schedule  estimate projected the MOX project’s total cost to be about $4.7 billion, with the project  expected to be completed in April 2016. The review’s estimated total project cost reflected an  increase of $352 million over the proposed total project cost of $4.3 billion due to increases in  the cost of some construction activities and contingency.

In February 2007, DOE’s fiscal year 2008 budget request reported that the revised total cost  for the MOX project totaled about $4.7 billion and that the estimate was in the final stages of  validation as part of the department’s critical decision process. The request stated that the  revised cost was a change from the prior not-yet-validated $3.6 billion estimate in DOE’s fiscal  year 2007 budget request, with over 50 percent of the $1.1 billion cost increase attributed to  an increase in contingency funds for the project during construction and cold startup.  Also in February 2007, responsibility for the MOX contract was officially transferred to the  Savannah River Site Office.

In April 2007, DOE formally approved a cost estimate, or baseline, for the MOX project of $4.8  billion and start of operations in September 2016.  In August 2007, construction of the MOX project began.

In May 2008, DOE certified the MOX contractor’s earned value management (EVM) system.  A July 2008 independent project review identified a number of concerns, including that only  one person was dedicated to the development and upkeep of the MOX project’s procurement  status information and that the project’s procurement strategy would require additional  procurement and engineering staff to meet future demands.  In December 2008, as a result of funding reductions for fiscal year 2008, DOE approved a  revised cost estimate for the MOX project of $4.9 billion and a 1-month delay in the start of  operations to October 2016.

According to a July 2009 report, the MOX contractor’s 2009 annual forecasted estimate for  completing the MOX project totaled approximately $4.4 billion, an increase of about $283.8  million from the 2008 annual forecasted estimate.

In May 2010, an independent review of the MOX contractor’s EVM system found that the  contractor’s performance data could not be used to accurately assess the cost performance of  the project, in part because the contractor was inappropriately using management reserve  funds to cover cost overruns. The MOX contractor began to implement a number of corrective  actions in response to the report’s findings.  According to an August 2010 report, the MOX contractor’s 2010 annual forecasted estimate  for completing the MOX project totaled approximately $4.6 billion, an increase of about $207.1  million from the 2009 annual forecasted estimate.

Details  In February 2011, DOE’s Office of Acquisition and Project Management—now the Office of  Project Management—changed the overall status of the MOX project from green to yellow,  indicating that the project was at risk of breaching its approved cost estimate (i.e.,  performance baseline).  A May 2011 project peer review found that the MOX project faced expected cost growth and  would be challenged in identifying approximately $364 million in cost savings necessary to  deliver the project at its total project cost (of $4.9 billion).  A June 2011 follow-on to the May 2010 independent review of the MOX contractor’s EVM  system found that the project was likely to exceed the total project cost by anywhere from  $104 million to $699 million, with an estimated most likely cost overrun of $493 million.  Nonetheless, DOE recertified the MOX contractor’s EVM system after the MOX contractor  completed a number of corrective actions.  According to a July 2011 report, the MOX contractor’s 2011 annual forecasted estimate for  completing the MOX project totaled approximately $4.7 billion, an increase of about $142.4  million from the 2010 annual forecasted estimate.

In January 2012, NNSA directed the MOX contractor to add additional scope for plutonium  metal oxidation capability and to include updates to the project’s current cost and schedule  projections, with a baseline change proposal due by the end of May 2012.  A March 2012 project review found that the MOX project’s cost and schedule baselines had a  very low probability of being met, and estimated that the total project cost was likely  underestimated by anywhere from $600 to $900 million when compared to the project’s  approved total cost of $4.9 billion. The review team recommended that the project should  develop an updated and more realistic baseline.  Also in March 2012, DOE changed the overall status of the MOX project from yellow to red,  indicating that the project was expected to breach its approved cost estimate (i.e., its  performance baseline).  A July 2012 project peer review found that the MOX project’s likely total project cost would fall  within the range of $6.9 billion to $7.3 billion as opposed to the project’s approved total cost of  $4.9 billion.  In September 2012, the MOX contractor submitted its revised baseline change proposal to  update the MOX project’s cost and schedule projections, including additional scope of work  that would provide the MOX project with a plutonium metal oxidation capability, referred to as  direct metal oxidation. According to the contractor’s proposal, it would cost about $7.4 billion  to complete the MOX project without the direct metal oxidation by November 2019. The  addition of the direct metal oxidation scope of work would cost an additional $262.3 million,  which would be completed in June 2023 after the completion of MOX project and the start-up  of operations by November 2019.

Details  In April 2013, DOE’s fiscal year 2014 budget request proposed a slowdown of construction of  the MOX project while NNSA took steps to assess alternative plutonium disposition strategies.  According to the request, NNSA cited the increase to the contractor’s total estimated cost for  the project and the budget environment as factors in its decision to pursue a slowdown of the  MOX project while conducting an assessment of potential alternative plutonium disposition  strategies.  According to NNSA, a May 2013 estimate prepared by the U.S. Army Corps of Engineers  estimated that, not including contractor fee, it would cost $9.4 billion to construct the MOX  project by 2024 at an annual funding level of $630 million.  According to NNSA, a June 2013 estimate prepared by the MOX contractor estimated that it  would cost between $8.5 and $9.7 billion to construct the MOX project, with completion from  2023 to 2032 depending on whether the annual funding level totaled $350 million or $500  million.  In September 2013, NNSA estimated it would cost about $10.5 billion to construct the MOX  project by 2027 at an annual funding level of $500 million.  According to NNSA, a November 2013 estimate prepared by the U.S. Army Corps of  Engineers estimated that it would cost from $10 to $11.7 billion to construct the MOX project,  with completion from 2026 to 2036 depending on whether the annual funding level totaled  $350 million or $500 million.

In March 2014, DOE’s fiscal year 2015 budget request stated that ongoing analysis led to the  determination that the MOX project would be significantly more expensive than anticipated  and concluded that, due to cost increases, the MOX approach was not viable within available  resources. The request, therefore, called for placing the facility in cold stand-by so NNSA  could further study more efficient options for plutonium disposition.  A May 2014 root cause analysis report found that some of the cost drivers that contributed to  the MOX project’s cost increases since 2007 included not having sufficiently experienced  project teams in place, basing the approved cost and schedule estimates on incomplete front- end planning, not sufficiently developing designs to support the project’s fast-track  procurement and construction, experiencing greater than expected inefficient execution of  construction activities, not implementing effective corrective actions, and not adequately  applying federal oversight to identify and address project performance issues  Also in May 2014, the DOE Office of Inspector General reported continuing concerns about  the achievability of the estimated cost and completion date for the MOX project. The report  also noted that the MOX project no longer had an approved cost and schedule estimate and  in light of the project continuing to receive significant funding, recommended that the MOX  contractor develop a new cost and schedule estimate.  In September 2014, in light of certain insufficient project data, NNSA directed the MOX  contractor to conduct a review to determine and validate the work completion status—that is,  state of completeness—for all commodities being installed in the MOX project.  In December 2014, both the Carl Levin and Howard P. McKeon National Defense  Authorization Act for Fiscal Year 2015 and the Consolidated and Further Continuing  Appropriations Act, 2015 directed DOE to continue construction and project or program  support activities related to the MOX project. However, the National Defense Authorization Act  also directed DOE to report on, among other things, alternatives to the MOX project, including  cost estimates for each alternative, and how such alternatives would conform to the Plutonium  Management and Disposition Agreement.

Details  In February 2015, DOE’s fiscal year 2016 budget request called for the continued construction  of the MOX project, in part because all four congressional committees of jurisdiction directed  that construction on the MOX project continue in fiscal year 2015 while NNSA conducted  additional cost studies and technology alternative studies.  In March 2015, NNSA’s MOX Project Management Office assessed the MOX contractor’s use  of level of effort versus the discrete method of earned value and determined a  disproportionate use of level of effort—around 56 percent—was masking the performance of  the contractor’s discrete work and therefore affecting the accurate measurement of the  project’s progress.  In April 2015, the Aerospace Corporation completed a report on the MOX project and  estimated that the MOX project’s total cost would be about $21.5 billion, with projected  completion in 2045 at an annual funding level of $500 million.  In June 2015, the MOX contractor finished its completeness verification review and found that  it had over-reported on the results of certain commodities being installed in the MOX project.  As a result of this review, the MOX contractor revised the amount of earned value claimed for  these commodities to address the over-reporting and provide a more realistic accounting of  the selected commodities.

In February 2016, DOE’s fiscal year 2017 budget request proposed terminating the MOX  project in favor of the dilute and dispose option as the path forward for the disposition of the  nation’s surplus, weapons-grade plutonium. According to the request, the MOX project was  found to be significantly more expensive than anticipated and would require approximately  $800 million to $1 billion annually for decades.  A May 2016 report prepared for the MOX contractor by High Bridge Associates, Inc.,  estimated that completing the construction of the MOX project could cost about $5.2 billion  and be completed in 10 years, with an annual funding level of about $520 million.  In July 2016, the MOX contractor submitted its annual forecasted estimate for completing  construction of the MOX project and estimated the total project cost to be about $10 billion,  with completion in 2029, with an annual funding level of $350 million.  In August 2016, DOE issued an updated performance baseline estimating that it would cost  approximately $17.2 billion to complete construction of the MOX project by 2048 assuming an  annual funding level of $350 million. DOE further estimated that it would cost about $14.3  billion to complete construction of the MOX project by 2035 assuming an annual funding level  of $500 million.  In October 2016, DOE rescinded the MOX contractor’s EVM system certification of  compliance in response to an August 2016 surveillance review that identified material non- compliances such as the overstatement of earned value and percentage complete.

Details  A February 2017 report by the U.S. Army Corps of Engineers found that there is likely to be a  substantial amount of rework at the MOX project but noted that the magnitude of the likely  rework has yet to be determined. The report stated that some of the rework is attributed to  design constructability issues as well as procuring, fabricating, and completing work out of  sequence.  In May 2017, DOE’s fiscal year 2018 budget request reiterated for the second consecutive  year, a plan to terminate the MOX project in favor of pursuing the dilute and dispose option for  plutonium disposition.  Also in May 2017, a DOE Office of Inspector General report stated that NNSA was not aware  of the total cost of rework at the MOX project because the time and cost of rework were not  definitively tracked prior to fiscal year 2014.  In December 2017, section 3121 of the National Defense Authorization Act for Fiscal Year  2018 authorized the Secretary of Energy to terminate the MOX project if, among other things,  the Secretary certified that the remaining life-cycle cost for an alternative option for carrying  out plutonium disposition would be less than approximately half of the estimated remaining  life-cycle cost of carrying out the plutonium disposition approach utilizing the MOX project.

In February 2018, DOE’s fiscal year 2019 budget request reiterated for the third consecutive  year a plan to terminate the MOX project in favor of pursuing the dilute and dispose option for  plutonium disposition.  In May 2018, the Secretary of Energy waived existing requirements to continue MOX  construction, but the state of South Carolina obtained an injunction in federal district court  temporarily blocking the waiver in June, which NNSA subsequently appealed.  In October 2018, a federal appellate court granted a stay of the federal district court’s  injunction that prohibited termination of the MOX contract and cessation of construction  operations. NNSA subsequently issued a notice of termination to the MOX contractor.

Appendix III: Selected GAO Recommendations from Prior Reports

We have made numerous agency recommendations in prior reports to  improve contract and project management in the Department of Energy  (DOE) and the National Nuclear Security Administration (NNSA). Some  reports contain recommendations for department and agency policies,  and others address project management problems for specific projects or  also address other agencies besides NNSA. A description of some of our  key recommendations, with the status of implementation as of December  2018, is provided below in table 2. For the most up-to-date status of these  agency recommendations, see our website: http://www.gao.gov.

Appendix IV: Comments from the Department of Energy

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Hilary Benedict (Assistant  Director), Rodney Bacigalupo, Antoinette Capaccio, Tara Congdon,  Pamela Davidson, Richard P. Johnson, Eleni Orphanides, Kevin  Remondini, Karen Richey, Sara Sullivan, and Tatiana Winger made key  contributions to this report.

In 2007, the Department of Energy's nuclear security agency started building a facility to dispose of unneeded weapons-grade plutonium. After cost increases, schedule delays, and nearly $6 billion in spending, Energy cancelled the project in 2018.

While this project was running, Energy took steps to address inexperience among its project management staff. As oversight improved, the project contractor's performance problems became clearer.

Energy has historically struggled to complete projects—in addition to this one—within cost and schedule estimates. We made 3 recommendations to improve how Energy uses the lessons learned from its projects.