Hanford Waste Treatment Plant: DOE Needs to Take Further Actions to Address Weaknesses in Its Quality Assurance Program

DOE and its contractor are building the WTP—which consists of multiple facilities—to treat a large portion of nuclear waste at Hanford. The project has faced persistent challenges, including quality assurance problems that have delayed it by decades and more than tripled its costs, to nearly $17 billion. DOE's quality assurance framework aims to ensure that all problems are identified and do not recur.

Senate Report 114-49 accompanying the National Defense Authorization Act for Fiscal Year 2016 included a provision for GAO to carry out an ongoing evaluation of the WTP. This first report examines (1) the actions DOE has taken to identify and address WTP quality assurance problems, (2) the extent to which DOE has ensured that quality assurance problems have been identified and do not recur, and (3) the extent to which DOE's organizational structure at ORP provides the Quality Assurance Division with independence to effectively oversee the contractor's quality assurance program. GAO reviewed DOE documents and obtained the insights of ORP's internal experts on WTP quality assurance efforts and outcomes.

The Department of Energy (DOE) has taken several actions to identify and address quality assurance problems at the Waste Treatment and Immobilization Plant (WTP) at its Hanford site in Washington. Among the actions taken is the implementation of the Managed Improvement Plan by DOE's Office of River Protection (ORP) and the WTP contactor. The plan is intended to ensure that the WTP can operate in compliance with DOE-approved safety and quality requirements. The contractor has stated that the plan is fully implemented, but GAO found that a number of key activities may be incomplete and ORP officials will not be able to verify the extent of implementation until December 2018.

According to DOE documents that GAO reviewed and ORP quality assurance experts GAO spoke with, ORP has not ensured that all WTP quality assurance problems have been identified and some previously identified problems are recurring. For example, a 2016 DOE report found quality assurance problems, such as engineering errors and construction deficiencies, that neither ORP nor the contractor had identified when the work was conducted. ORP quality assurance experts GAO spoke with reiterated the issues identified in reports. In addition, DOE audits have found that previously identified quality assurance problems have recurred in key areas, such as the procurement of items that do not meet requirements or perform as specified. These problems were also raised by several of the ORP quality assurance experts GAO interviewed. According to these experts, such recurring problems may lead to significant rework at WTP facilities in the future if work is not stopped and the issues addressed. ORP's quality assurance framework requires the contractor to determine the extent to which quality assurance problems exist in all WTP structures, systems, and components when such problems are identified, and allows ORP to stop work at a facility if recurring issues arise. However, ORP has neither directed the contractor to make this determination nor stopped work when problems recur because it has confidence in the Managed Improvement Plan.

ORP's organizational structure may not provide its Quality Assurance Division with sufficient independence from the office's upper management to oversee the contractor's quality assurance program effectively. GAO has previously found that an oversight organization should be structurally distinct and separate from program offices responsible for cost and schedule performance to avoid conflict between mission objectives and safety. However, a 2017 DOE headquarters assessment found that ORP's Quality Assurance Division's effectiveness has been limited. This is because in some cases ORP upper management had mischaracterized its findings, and in other instances, ORP upper management had not used this division to evaluate the extent of potential quality assurance problems. ORP quality assurance experts GAO spoke to were also concerned that ORP's organizational structure does not always ensure the independence of the division. For example, two of these experts described instances when ORP upper management had downgraded the division's findings so that the contractor could take less stringent corrective measures. By providing the Quality Assurance Division adequate independence, DOE can better ensure that compliance with nuclear safety requirements will not be subordinated to other project management goals, such as meeting cost and schedule targets.

GAO recommends that DOE direct the WTP contractor to determine the extent of problems in WTP structures, systems, and components and order work stops when problems recur, and DOE should direct ORP to revise its organizational structure to ensure the independence of the Quality Assurance Division. DOE generally agreed with GAO's recommendations.

The Department of Energy (DOE) is responsible for one of the world’s  largest environmental cleanup programs: the treatment and disposal of  millions of gallons of radioactive and hazardous waste at its 586-square- mile Hanford Site in southeastern Washington State. This waste was  created as a by-product of producing plutonium and other special nuclear  materials for nuclear weapons, starting during World War II and  continuing until the late 1980s. The most hazardous and radioactive  waste is stored in 177 large underground tanks. However, as of January  2017, none of the tank waste had been treated for disposal because the  facilities intended to treat a significant portion of this waste—known  collectively as the Waste Treatment and Immobilization Plant (WTP)— have faced persistent technical and management challenges that have  increased the project’s cost by billions of dollars and added decades to  the waste treatment schedule. Among these challenges, DOE has had  difficulty ensuring that WTP engineering, procurement, and construction  work meets nuclear safety and quality requirements.

In 2000, DOE awarded a contract to Bechtel National, Incorporated, to  design, construct, and commission the WTP. The WTP is a complex, first- of-a-kind project and is being constructed under a design-build contract.  In the design-build approach, technology development activities, plant  design, and construction occur simultaneously rather than sequentially.

DOE no longer allows the use of the design-build approach for complex,  first-of-a-kind facilities but has continued to use it for the WTP.

In December 2012, the WTP Engineering Division of DOE’s Office of  River Protection (ORP) issued a memorandum that recommended that all  activities affecting engineering design, construction, and installation of  structures, systems, and components be stopped because the division  had found that it could not verify that completed work met nuclear safety  and quality requirements. According to the memorandum, stopping work  would help the department avoid future nuclear safety and quality  compromises and substantial rework. Instead of stopping all work at the  WTP, ORP management stopped work only on those facilities that faced  the most significant technical challenges. Around this time, ORP issued  nine “Priority Level One” findings—findings reflecting problems that if  uncorrected could have serious effects on safety, quality, and operability  of a nuclear facility, as well as on the environment, according to ORP’s  quality assurance guidance. Seven of the nine findings were issued by  ORP’s WTP Engineering Division in 2012 and focused on engineering  and nuclear safety. The other two were issued by ORP’s Quality  Assurance Division in 2013 and focused on the contractor’s quality  assurance program.

Federal regulations require DOE contractors to establish DOE-approved  quality assurance programs. Under an approved quality assurance  program, the contractor must establish and implement processes to  detect and prevent quality problems; identify, control, and correct items,  services, and processes that do not meet established requirements;  identify the causes of problems and work to prevent recurrence as a part  of correcting the problems; and plan and conduct independent  assessments to measure the adequacy of work performance and to  promote improvement. ORP’s two Priority Level One findings related to  quality assurance deficiencies found that the contractor’s quality  assurance program did not effectively ensure that work was being  completed to meet these requirements. Significantly, one of the two  findings concluded that the contractor’s overall quality assurance program  was not fully effective. The other finding concluded that the contractor’s  Corrective Action Program to address quality assurance problems was  not fully effective. Overall, ORP determined that the deficiencies in the  contractor’s quality assurance program were the most serious type— those that could result in a systemic breakdown in safety or quality or that  could have a serious effect on worker health and safety, the public and  environment, and facility operations. In addition, the report stated that the  deficiencies warranted immediate attention by the contractor and required  corrective measures and a high degree of ORP oversight.

We have reported on the need for effective oversight of nuclear safety— including quality assurance programs—across the DOE complex, finding  in 2008 that a strong management and oversight program is needed to  ensure that DOE’s nuclear operations are carried out in a safe and  environmentally acceptable manner. Also in 2008, we identified key  elements that any nuclear safety oversight organization should have in  order for it to provide effective independent oversight. For example, we  found that the organization should be structurally distinct and separate  from DOE program offices to avoid management interference or conflict  between program office mission objectives, such as cost and schedule  performance, and safety.

Senate Report 114-49 accompanying the Senate version of the National  Defense Authorization Act for Fiscal Year 2016 includes a provision for  GAO to carry out an ongoing evaluation of issues related to the WTP.  This first report (1) describes the actions DOE has taken to identify and  address quality assurance problems at the WTP, (2) examines the extent  to which DOE has ensured that all quality assurance problems have been  identified and will not recur, and (3) examines the extent to which DOE’s  organizational structure at ORP provides the independence to effectively  oversee the WTP contractor’s quality assurance program.

To describe the actions DOE has taken to identify and address quality  assurance problems at the WTP, we obtained and reviewed DOE  documents and reports that describe ORP and contractor actions,  including the contractor’s root cause analyses of quality assurance  problems at the WTP. In addition, we reviewed DOE orders, ORP  procedures, and documents that describe the requirements DOE is to  follow to ensure that work meets quality assurance requirements and that  the contractor has implemented corrective measures for identified quality  assurance problems.

To examine the extent to which DOE has ensured that all quality  assurance problems have been identified and will not recur, we reviewed  internal and external assessments, audits, and reviews of ORP’s and the  contractor’s quality assurance programs. In addition, we reviewed  technical and management issues that were reported to the contractor’s  Corrective Action Management Program database from January 1, 2014,  when the contractor started to implement corrective measures, to  September 31, 2017, to identify significant quality assurance problems  and issues.

To examine the extent to which DOE’s organizational structure at ORP  provides the independence to effectively oversee the WTP contractor’s  quality assurance program, we obtained and reviewed internal and  external reports and assessments conducted by ORP, DOE’s Office of  Environmental Management’s Office of Standards and Quality Assurance,  DOE’s Office of Enterprise Assessment, DOE’s Office of Inspector  General, and the Defense Nuclear Facilities Safety Board, on the portions  of WTP engineering, procurement, and construction that have been  subject to review and examination for adherence to quality assurance  requirements. In addition, we examined ORP’s record of implementing  DOE headquarters office’s audit and assessment recommendations  designed to improve the WTP quality assurance program dating back to  January 2012.

To address all of our objectives, we also conducted semi-structured  interviews, which we used to obtain information from staff with ORP’s  Quality Assurance, WTP Construction Oversight and Assurance, and  WTP Engineering Divisions who are involved in the oversight of the  contractor’s quality assurance program. The group consisted of all nine  subject matter experts that ORP management identified as employees  primarily responsible for WTP quality assurance oversight. In this report,  we refer to this group as ORP quality assurance experts. During the semi- structured interviews, we asked a series of questions on DOE’s and the  contractor’s compliance with quality assurance requirements, as well as  questions on the resolution of problems with the contractor’s quality  assurance program and DOE’s oversight of the program. In several  cases, all nine ORP quality assurance experts were not able to answer  our questions for various reasons, such as not having recent experience  working in a particular quality assurance area. In addition to conducting  semi-structured interviews, for all objectives we interviewed DOE officials  with ORP; the Inspector General’s Office at Hanford; and DOE  headquarters offices, including the Office of Environmental  Management’s Office of Standards and Quality Assurance and the Office  of Enterprise Assessments. We also interviewed WTP contractors,  officials from the Washington State Department of Ecology, and officials  from the Defense Nuclear Facilities Safety Board.

We conducted this performance audit from February 2016 to April 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

This section describes DOE’s tank waste treatment approach at Hanford  and DOE’s quality assurance framework and requirements.

Cleanup of the Hanford Site is governed by two main compliance  agreements: (1) the 1989 Hanford Federal Facility Agreement and  Consent Order, or Tri-Party Agreement, an agreement between DOE, the  Washington State Department of Ecology, and the Environmental  Protection Agency, and (2) a 2010 consent decree. The Tri-Party  Agreement was signed in May 1989 and lays out a series of legally  enforceable milestones for completing major activities in Hanford’s waste  treatment and cleanup process. The Tri-Party Agreement has been  amended a number of times to establish additional enforceable  milestones for certain WTP construction and tank waste retrieval  activities, among other things. Under the Tri-Party Agreement, DOE must  complete waste treatment at the Hanford Site by 2047.

The overall mission of the WTP is to treat and immobilize a large part of  54 million gallons of radioactive and chemical waste stored in 177  underground storage tanks. The WTP is the most technically complex  and largest construction project within DOE’s Office of Environmental  Management, occupying 65 acres of the Hanford Site. Some of DOE’s  tank waste is highly radioactive material—known as high-level waste— mixed with hazardous waste. Under current law, this waste must be  vitrified—a process in which the waste is immobilized in glass—prior to  disposal. Low-activity waste is DOE’s term for the portion of the tank  waste at Hanford with low levels of radioactivity. Low-activity waste is  primarily the liquid portion of the tank waste that remains after as much  radioactive material as technically and economically practical has been  removed.

The WTP consists of the following set of facilities that are designed to  separate waste into low-activity and high-level waste streams and, once  completed, treat these waste streams in separate facilities using  vitrification.

Pretreatment Facility. This facility is to receive the waste from the  tanks and separate it into high-level and low-activity waste streams.

Low-Activity Waste Facility. This facility is to receive the low-activity  waste from the Pretreatment facility and immobilize it by vitrification.  The canisters of vitrified waste will be permanently disposed of at  another facility at Hanford.

High-Level Waste Facility. This facility is to receive the high-level  waste from the Pretreatment Facility and immobilize it by vitrification.  The canisters of vitrified waste will be stored on-site until a final  repository is established.

Effluent Management Facility. The Effluent Management Facility is  being built to evaporate much of the secondary waste produced  during low-activity waste processing and vitrification at the Low- Activity Waste Facility.

Analytical Laboratory. This facility will conduct analyses as needed,  such as testing samples of the vitrified waste to ensure that it meets  certain criteria and regulatory requirements for disposal.

Balance of Facilities. These facilities consist of the 22 support facilities  that make up the plant infrastructure, such as cooling water systems  and silos that hold vitrifying materials.

In part because of the 2012 work stoppage at the WTP’s Pretreatment  and High-Level Waste Facilities, in 2012 DOE adopted a phased waste  treatment strategy through which the department aims to begin treating  some of the low-activity waste before resolving all WTP technical  issues. During the first phase of this strategy, DOE plans to implement a  Direct Feed Low-Activity Waste (DFLAW) approach to transfer some low- activity waste from the tanks to the WTP’s Low-Activity Waste Facility for  vitrification before the Pretreatment Facility is completed. The approach  relies on construction of a new facility—the Low-Activity Waste  Pretreatment System—designed to remove highly radioactive particles  from liquid tank waste before sending the waste stream to the Low- Activity Waste Facility. During later phases, DOE intends to complete the  WTP Pretreatment Facility and High-Level Waste Facilities. DOE also  plans to construct a Tank Waste Characterization and Staging Facility  under a different contract to stage, mix, sample, and characterize high- level waste from the tanks prior to delivery to the Pretreatment Facility.  Figure 1 illustrates WTP and other facilities planned for Hanford tank  waste treatment.

DOE’s Quality Assurance Framework and Requirements

A set of federal regulations, DOE orders, and ORP procedures  collectively make up DOE’s quality assurance framework that aims to  ensure that all WTP quality assurance problems can be identified and that  identified problems do not recur. DOE’s quality assurance regulations  require DOE contractors to establish DOE-approved quality assurance  programs. The regulations specify that under an approved program, the  contractor’s quality assurance program must, among other things, (1)  establish and implement processes to detect and prevent quality  problems; (2) identify, control, and correct items, services, and processes  that do not meet established requirements; (3) procure items and services  that meet established requirements and perform as specified; (4) plan  and conduct independent assessments to measure item and service  quality, to measure the adequacy of work performance, and to promote  improvement; and (5) maintain items to prevent damage, loss, or  deterioration. In addition, DOE Order 226.1B requires that DOE’s  organizations and contractors implement oversight processes that ensure  that relevant quality assurance problems are evaluated and corrected on  a timely basis to prevent recurrence.

The WTP contract requires compliance with these regulations and  requirements. The WTP contract specifies that as the owner of the WTP  project, DOE is responsible for providing quality assurance oversight of  the WTP. ORP’s Quality Assurance Division provides such oversight,  for example, by doing the following:

Reviewing a sampling of the contractor’s documentation on the WTP’s  engineering, procurement, and construction.

Conducting audits and assessments to ensure that the contractor’s  work complies with applicable quality assurance requirements.

Assessing the effectiveness of the contractor’s Corrective Action  Management Program, which involves identifying, documenting,  planning, addressing, and tracking actions required to resolve or  correct problems.

Both the contractor’s and ORP’s quality assurance programs require that  corrective actions to address significant problems with the quality of the  work must include a determination of the extent to which the problematic  conditions exist (known as an extent-of-condition review) as well as the  underlying causes of those conditions. If corrective actions do not  address the conditions, ORP’s quality assurance policy allows the office  to call for a suspension of work. ORP’s stop work procedure includes  the process ORP is to follow when the Quality Assurance Division  Director, in consultation with ORP management, determines that work  needs to be suspended as a result of the occurrence or reoccurrence of  significant quality assurance problems. ORP updated this procedure in  February 2016 to describe the type of quality assurance deficiencies that  should trigger consideration of work stoppage. According to the updated  procedure, characteristics of a deficiency that can trigger an order to stop  work include, but are not limited to, problems that will result in $25 million  or more in loss of productivity, construction rework, or environmental  damage or a significant quality problem that if left uncorrected can result  in construction delays or create adverse safety conditions. Until February  2016, ORP did not have precise criteria describing the conditions under  which it should evaluate work for possible stoppage, according to a DOE  headquarters report.

ORP Has Taken Several Actions to Identify and Address Quality Assurance Problems at the WTP, but All Planned Actions Have Not Been Completed

ORP has taken several actions to identify and address quality assurance  problems at the WTP, but all planned actions have not been completed.  In 2013 ORP conducted a comprehensive audit, which resulted in several  actions, including when the office had the contractor begin implementing  a Managed Improvement Plan (MIP) in 2014. The MIP is intended to  ensure that the WTP could operate in compliance with DOE-approved  safety and quality requirements. Implementation of the MIP was to be  completed by April 2016. Although the contractor reported that the  implementation was complete, some of the plan’s corrective measures  have not been fully implemented, according to contractor documents we  reviewed and quality assurance experts we spoke to. In addition, ORP’s  effort to verify the extent to which the contractor has implemented MIP  corrective measures is not scheduled to be complete until at least  December 2018.

Actions Taken to Identify and Address Quality Assurance Problems at the WTP

ORP has taken several actions to identify and address quality assurance  problems at the WTP. After the partial work stoppage in 2012, ORP  conducted an audit in 2013 to evaluate the adequacy, implementation,  and effectiveness of the contractor’s quality assurance program. The  audit found that the contractor’s quality assurance program was generally  adequate. However, it also found that the contractor’s quality assurance  program was not fully effective in several areas. In response to the  audit, ORP and the WTP contractor took the following actions:

Developed compensatory measures. At ORP’s request, in 2013, the  contractor started implementing “compensatory measures” to ensure  that ongoing WTP work during a 2-year performance improvement  period would meet DOE quality and safety requirements. For  example, in September 2013, the contractor implemented a measure  requiring senior management review of all condition reports and their  associated levels of significance. According to ORP officials, the  compensatory measures were intended to be additional, temporary  internal controls to ensure that work at the WTP did not result in new  or recurring quality assurance problems.

Initiated the MIP. To systematically integrate compensatory  measures, the contractor developed the MIP to address all quality  assurance problems identified in the two Priority Level One findings  and the seven Priority Level One findings associated with engineering  and nuclear safety. In August 2014, the contractor started  implementing the MIP. The MIP is a set of 52 corrective measures  intended to establish processes, procedures, and metrics to produce  an overall quality program that ensures that the WTP can safely  operate in compliance with DOE-approved nuclear safety  requirements, according to the contractor. The measures include the  following:

Actions to enhance external independent oversight. This measure  calls for the contractor to conduct assessments using external  subject matter experts to evaluate the ability of the contractor’s  quality assurance program to identify precursors to potential  problems and their causes. This measure responds to the 2013  audit in which DOE concluded that the contractor’s quality  assurance program could not ensure compliance with  requirements. Specifically, the audit found that the contractor’s  quality assurance program was not fully effective in several areas,  including, but not limited to, design, software quality, procurement,  and ensuring that identified problems are corrected.

Actions to ensure that procured items and services meet  requirements and perform as specified. This measure is intended  to ensure that the contractor’s processes and procedures to  identify and ensure the quality of technical products meet  requirements. The nuclear industry uses “commercial grade  dedication” to refer to the process by which the contractor or  subcontractor verifies that an item (e.g., an electric switch) or  service (e.g., design of an electrical system) can meet commercial  quality and safety requirements and be approved for use in a  nuclear facility. It requires the contractor to perform source  verification, perform inspections and tests, and assess the  processes that control the quality of purchased items and services  to help ensure that critical components of procured items and  services are designed, fabricated, assembled, installed, and  tested with appropriate documentation to support their compliance  with WTP safety requirements. This measure also responds to  DOE’s 2013 audit, which found that the contractor had inadequate  control over the quality of purchased items and services.

Actions to control and correct items and processes that do not  meet requirements. This measure is intended to allow the  contractor to identify and ensure that materials and equipment that  have been received, and that will be received in the future, meet  requirements. The contractor is to conduct comprehensive  reviews of previously received material and equipment, as well as  all future deliveries, to help ensure the verification, accuracy, and  completeness of documentation for materials and equipment  received from suppliers. This measure also responds to DOE’s  2013 audit, which found that the contractor had received  components that did not comply with safety requirements.

Performed targeted audits to test compensatory measures and the  implementation of the MIP. To assess the effectiveness of the  compensatory measures and the MIP, ORP performed targeted  audits. For example, to assess the extent to which the contractor has  addressed quality assurance program deficiencies, in early 2017  ORP’s Quality Assurance Division conducted a “vertical slice audit.”  This audit reviewed engineering, procurement, and construction of a  key system that will be needed for initial WTP operations.

Because of the long-standing quality assurance problems at the WTP,  DOE required ORP to closely monitor the contractor’s implementation of  the MIP. Specifically, as a result of a DOE Office of Enforcement  investigation into the contractor’s quality assurance and corrective action  management programs, DOE entered into a Consent Order with the  contractor in 2015. The Consent Order required the contractor to  complete the actions identified in the MIP to the extent necessary to  restore quality assurance program to full effectiveness by April 30,  2016. The Consent Order does not preclude DOE from reopening the  investigation or issuing an enforcement action if there is a recurrence of  nuclear safety deficiencies similar to those identified in the Consent Order  or the if contractor fails to complete actions required by the Consent  Order in a timely and effective manner to prevent recurrence of the  identified issues.

Corrective Measures to Address Quality Assurance Problems at the WTP Have Not Been Fully Implemented

The contractor has not fully implemented corrective measures for all  identified quality assurance problems, according to contractor documents  we reviewed. In August 2017, the contractor reported that it had finished  its actions to implement the MIP. However, according to the contractor’s  MIP status update accompanying the contractor’s report, 13 of the 52  corrective measures specified in the MIP had not been fully implemented.  Our review of these 13 MIP corrective measures we found that 9 were  intended to exclusively or partially address weaknesses in the  contractor’s quality assurance program. For example, the two corrective  measures to ensure that WTP facilities’ computer software meets  requirements were not complete, according to the MIP status update.  These corrective measures included improving the software procurement  process and revising the quality assurance manual.

In addition, of the 39 measures that the contractor considers complete,  some do not appear to be fully implemented, according to one ORP  quality assurance expert that we spoke to. For example, one ORP quality  assurance expert disagreed with the contractor’s assessment that a  corrective measure for documentation pertaining to radiographic film— which is needed for conducting quality assurance reviews of certain  equipment—was fully implemented. This corrective measure calls for the  contractor to review purchase orders for radiographic film and then store  the radiographic film as documentation of compliance with nuclear quality  standards. According to the expert, radiographic film reviews are still not  consistently conducted, and radiographic film documentation is still not  consistently stored. In cases where such documentation is incomplete or  missing, the contractor is at times forced to re-create the documentation  at considerable cost to DOE. According to ORP’s MIP oversight plan, it  will take the office until at least December 2018 to verify the extent to  which the contractor has implemented each of the 52 MIP corrective  measures.

ORP Has Not Ensured That All Quality Assurance Problems Have Been Identified, and Some Previously Identified Problems Are Recurring

According to DOE documents we reviewed and ORP quality assurance  experts we spoke with, ORP’s actions have not ensured that all quality  assurance problems have been identified at the WTP, and some  previously identified problems are recurring. Specifically, according to  DOE documents and the experts we spoke with, ORP’s oversight has not  ensured that the contractor has identified all quality assurance problems  in structures, systems, and components that were completed and  installed before the 2012 work stoppage or identified all such problems in  newer structures, systems, and components needed for initial WTP  operations. In addition, according to the documents we reviewed and  experts we interviewed, previously identified quality assurance problems  are recurring.

ORP Oversight Has Not Ensured That the Contractor Has Identified All Quality Assurance Problems

Recent DOE reviews have found that ORP has not ensured that all  quality assurance problems have been identified at the WTP. First, a  2016 DOE Office of Enterprise Assessment report found quality  assurance deficiencies that neither ORP nor the contractor had identified  at the time the work was conducted.  The report identified numerous  construction deficiencies, procurement and supplier deficiencies,  engineering errors, maintenance issues, and materials with expired shelf  lives. For example, the report identified welding deficiencies on tanks  designed to hold nuclear waste that were identified in a WTP facility  several years after the tanks were installed. The report concluded that the  contractor is aware that significant quality assurance problems likely exist  in older structures, systems, and components. This report noted that  much of the equipment in older structures, systems, and components was  manufactured and delivered to the project from 5 to 10 years ago—and  some of this equipment was supplied by vendors or manufacturers that  are no longer in business—which could lead to costly rework.

Second, a 2015 DOE Inspector General report found that the contractor  had procured $4 billion in parts and materials through fiscal year 2014,  but ORP and the contractor had not always identified problems with the  quality of procured items in a timely manner. For example, the report  found that in about 45 percent of the nearly 1,400 procurement problems  reviewed, the contractor did not identify the problems until at least 2 years  after the items arrived on site. The report also found that in many cases  the contractor canceled its efforts to recover the costs to resolve the  problems because of the length of time that had passed. The report  concluded that these problems were caused by weaknesses in the  contractor’s quality assurance program and that the contractor’s  procedures to prevent or identify problems with procured items were not  always followed effectively.

The findings of these reports are consistent with the views of ORP quality  assurance experts we spoke with who stated that ORP oversight has not  ensured that the contractor has identified all quality assurance problems  in structures, systems, and components—particularly those that were  completed and installed before the 2012 work stoppage. These quality  assurance experts said that because quality assurance problems have  not been identified, they expect significant rework will be needed for work  that was completed before 2012. Specifically, most of the ORP quality  assurance experts (seven of the nine) told us that they expect rework will  be needed for existing WTP facilities, such as the Pretreatment and High- Level Waste Facilities. One of these seven quality assurance experts  noted that the contractor does not have a complete record of the  documentation for key systems and equipment, which is required for  demonstrating compliance with nuclear safety standards and eventual  permitting of WTP facilities for operation. According to this expert, the  extent of this shortcoming is not known, but fixing it—that is, creating a  complete record of required documentation—may lead to years of delays.

ORP Quality Assurance Division officials told us that because ORP’s  focus is on ensuring that facilities needed for initial operations will be  ready to operate by December 2023, they have not been directed by ORP  management to focus on identifying all quality assurance problems for  work completed before 2012 for facilities needed for later phases of WTP  operations, such as structures, systems, and components of the  Pretreatment and High-Level Waste Facilities. In addition, they stated that  there may be significant changes to these facilities needed for the WTP’s  later phases, making it unnecessary for them to review the extent of  quality assurance problems until it is known what parts of the facilities will  remain and which parts will not.

However, similar problems appear to exist in WTP facilities needed for  initial operations. ORP quality assurance experts that we interviewed also  stated that ORP oversight has not always ensured that all quality  assurance problems in facilities needed for the initial WTP operations, or  DFLAW, have been identified. Five experts told us that issues such as  identifying problematic items, services, and processes had not been fully  resolved. Specifically, these ORP quality assurance experts told us that  when quality assurance problems are identified in structures, systems, or  components needed for DFLAW, ORP does not always ensure that the  contractor identifies the extent to which such problems may exist in other  areas affected by the same structures, systems, or components. For  example, an ORP quality assurance expert cited an instance in which an  ORP quality assurance team reviewed a sample of 25 procurement  “packages” (out of thousands) for a DFLAW facility and identified 143  problems—significantly more problems than the team expected for such a  small sample. Consistent with ORP quality assurance requirements, this  ORP quality assurance expert recommended to ORP upper management  that the contractor determine the extent to which such problems could  affect other structures, systems, and components needed for DFLAW.  However, according to an ORP memo, ORP upper management did not  require the contractor to implement this recommendation, instead citing  “extenuating circumstances” and requiring a lesser corrective action than  what was recommended. Three ORP quality assurance experts told us  that they believe that because problems have not been comprehensively  assessed, there may be equipment and systems within DFLAW that will  fail to meet their intended functions.

We also found that although ORP conducted its vertical slice audit in  2017 to test its compensatory measures and the MIP to improve quality  assurance, the audit report notes that it was focused on only one system  within the Low-Activity Waste Facility. According to ORP officials, there  are numerous structures, systems, and components in facilities needed  for DFLAW that have not been audited or reviewed in a manner similar to  the vertical slice audit. Both the contractor’s and ORP’s quality assurance  programs require that corrective actions to address significant problems  with the quality of the work include a determination of the extent to which  the problematic conditions exist as well as the underlying causes of those  conditions. Until ORP requires the WTP contractor to determine the full  extent to which problems exist in all WTP structures, systems, and  components, DOE lacks a comprehensive understanding of all potential  quality assurance problems at all WTP facilities.

Previously Identified Quality Assurance Problems Are Recurring

DOE requires its program offices, such as ORP, and contractors to have  oversight processes to ensure that quality assurance problems are  evaluated and corrected in a timely basis to prevent recurrence. However,  several DOE documents we reviewed show that previously identified  quality assurance problems have recurred in recent years, including the  following:  In 2015, an ORP audit report identified recurring weaknesses in  quality assurance for the contractor’s process for procuring  commercial items for use in a nuclear facility. For example, ORP  found that the contractor’s internal controls for this process were not  consistently performed; did not consistently comply with procedural  requirements; and, in many cases, did not establish reasonable  assurance that procured systems, services, and components acquired  from 2010 to 2014 would perform their intended safety functions.

In a 2015 report on the design and operability of key systems and  components for the Low-Activity Waste Facility, ORP found that the  quality of computer systems software was not in full compliance with  DOE requirements, leading to conditions where personnel and the  environment may not be adequately protected. ORP had identified a  similar problem in 2008, when it found that the contractor’s computer  programs used in engineering calculations were not always verified to  show that they produced correct solutions within defined limits for all  parameters, as required by the contractor’s quality assurance manual.  ORP had also previously identified WTP computer software quality  problems in 2010 when it issued a Priority Level Two finding on  software procedures and another Priority Level Two finding on  software testing.

In 2017, ORP’s Quality Assurance Division issued a report that  examined the contractor’s quality assurance program and found  problems in quality assurance areas that had been previously  identified. The report noted that in 6 of 19 quality assurance  program areas, the contractor’s performance was marginal—and in  need of improvement—or indeterminate. These 6 areas included  identifying, controlling, and correcting items, services, and processes  that do not meet established requirements; maintaining items to  prevent damage, loss, or deterioration; and procuring items and  services that meet established requirements and perform as specified.

ORP quality assurance experts that we spoke with also stated that  previously identified quality assurance problems are recurring, including  some in areas where the contractor had implemented corrective  measures. These quality assurance experts told us that quality assurance  problems are recurring in several key areas, including those areas  identified in the documents described above: (1) procurement of items  and services that do not meet established requirements or perform as  specified; (2) software that does not meet established requirements; and  (3) a maintenance program that does not prevent damage, loss, or  deterioration of WTP structures, systems, and components. For example,  see the following.

Procurement of items and services that do not meet requirements or  perform as specified. Four out of the five ORP quality assurance  experts we interviewed who had recent experience with the  procurement of items and services told us that problems with  procured items and services that do not meet established  requirements or perform as specified are not fully resolved. One of  these ORP quality assurance experts stated that an ORP team  recently reviewed a random sample of 45 of the roughly 30,000  procurements the contractor had made for the WTP and identified a  number of instances where materials did not meet requirements,  which resulted in one Priority Level Two finding—which represents a  serious issue that indicates an adverse condition, such as a  noncompliance or breakdown of a management system—and five  Priority Level Three findings. The expert noted that this was many  more deficiencies than the team expected for such a small sample.

Settlement of Allegations of Contractors  Knowingly Mischarging Costs at the Waste  Treatment and Immobilization Plant (WTP)  In November 2016, the WTP contractor and  certain subcontractors agreed to pay $125  million to resolve allegations under the False  Claims Act that they made false statements  and claims to the Department of Energy (DOE)  by charging DOE for deficient nuclear quality  materials, services, and testing that were  provided to the WTP at DOE’s Hanford Site.  The contract required materials, testing, and  services to meet certain nuclear quality  standards. The Department of Justice alleged  that the defendants violated the False Claims  Act by charging the government the cost of  complying with these standards when they  failed to do so. In particular, the Department of  Justice alleged that the defendants improperly  billed the government for materials and  services from vendors that did not meet quality  control requirements, for piping and waste  vessels that did not meet quality standards,  and for testing from vendors that did not have  compliant quality programs. As part of the  settlement, the contractors admitted no  wrongdoing, and the United States did not  concede that its claims were not well founded.

Software that does not meet requirements. ORP quality assurance  experts told us that problems are recurring in certain areas where  items and processes do not meet requirements, such as computer  software quality assurance, despite the contractor developing two MIP  corrective measures in this area. Two ORP quality assurance experts  reported that problems with software quality are recurring. One ORP  quality assurance expert added that the contractor often fails to  develop software quality documentation that is needed to demonstrate  compliance with quality requirements when permitting facilities for  operation. As a result, the contractor will have to re-create this  documentation at some cost.

A maintenance program that does not prevent damage, loss, or  deterioration. Each of the three ORP quality assurance experts with  knowledge in this area told us that the contractor had not established  a fully effective WTP maintenance program, particularly for the  Pretreatment and High-Level Waste Facilities, and as a result,  structures, systems, and components at these facilities have  deteriorated and been damaged. Such statements are consistent  with findings of the Defense Nuclear Facilities Safety Board, which  reported in April 2016 that systems and components stored in an  outdoor storage yard were not properly covered and showed signs of  being affected by water, sand, or animals. In March 2016, ORP  reported significant water intrusion into several areas of the High- Level Waste Facility. As a result, some of the facility’s structures,  systems, and components had deteriorated and will require costly  rework. The contractor notified DOE in April 2017 that because DOE’s  focus is on completing facilities needed for initial WTP operations, it  would submit a proposal to change the WTP contract to account for  the increased scope, cost, and schedule of long-term maintenance,  storage, and management of procured and partially installed  structures, systems, and components at those facilities not needed for  initial WTP operations.

Consistent with its quality assurance procedures, ORP can use its  authorities—such as those under the Consent Order and its quality  assurance policy—to stop work if corrective measures do not prevent  quality assurance problems from recurring. However, ORP has not used  such authorities. ORP senior officials told us that they did not consider it  necessary to stop work because of the recurrence of problems in certain  areas because they plan to evaluate the extent of the contractor’s  implementation of MIP corrective measures over the next year and have  allowed work to continue because they believe that the contractor’s  quality assurance program is generally adequate. Without directing ORP  to use its authorities to stop work in areas where quality assurance  problems are recurring until it can verify that the problems are corrected  and will not recur, DOE may face future rework that could increase costs  and schedule delays for the WTP.

ORP’s Organizational Structure May Not Provide Sufficient Independence for Effective Oversight of the WTP Contractor’s Quality Assurance Program

A 2017 assessment from DOE headquarters and our interviews with nine  ORP quality assurance experts suggest that ORP’s organizational  structure does not provide the quality assurance function with sufficient  independence from upper management—which includes the ORP  Manager and the WTP Federal Project Director—to effectively oversee  the contractor’s quality assurance program. Our prior work has found that  to be independent, an oversight organization should be structurally  distinct and separate from program offices responsible for achieving the  program’s mission to avoid management interference or conflict between  program office mission objectives and safety. At ORP, however, the  Quality Assurance Division is not fully separate and independent from the  upper management of the WTP project, which manages cost and  schedule performance. We believe that such a structure has the potential  to create a conflict of interest.

Specifically, we found that ORP’s Quality Assurance Division performs  assessments of the contractor’s quality assurance program, among other  things, and reports its findings to ORP upper management, including the  ORP Manager, who has the discretion to determine whether and to what  extent to require the contractor to take action in response to findings.  When quality assurance issues are identified, ORP upper management  must balance its mission of meeting cost and schedule targets with its  responsibility to ensure that nuclear safety and quality standards are met.  However, these are two potentially conflicting responsibilities because  meeting WTP cost and schedule targets may be threatened if serious  quality assurance problems are identified.

A February 2017 external assessment from DOE headquarters noted that  ORP’s Quality Assurance Division’s effectiveness has been limited  because, in some instances, its findings have been mischaracterized by  ORP upper management, and in others, ORP upper management has not  used this division effectively to evaluate the extent of potential quality  assurance problems. This assessment found that ORP had not  performed adequate oversight of the contractor’s MIP and that some  critical quality assurance areas were not receiving the necessary scrutiny  from ORP. Further, the assessment found that ORP management  sometimes mischaracterized the seriousness of the Quality Assurance  Division’s findings and, as a result, did not require the contractor to  conduct extent-of-condition review for significant quality assurance  problems. While this assessment stated that ORP had an effective quality  assurance program, it concluded that three of the eight quality assurance  areas the assessment team reviewed were not fully effective, including  ORP’s ability to conduct assessments of the contractor’s quality  assurance program.

A Cautionary Tale: Quality Assurance  Problems Doom Commercial Nuclear  Power Plant  In the commercial nuclear industry, there is a  notable example of a construction project that  faced significant quality assurance challenges.  In the 1970s and early 1980s, Cincinnati Gas  & Electric attempted to construct a commercial  nuclear power plant, known as the Zimmer  Plant, near Moscow, Ohio. After 10 years of  construction and more than $2 billion spent,  the company abandoned its effort to construct  the plant. An independent review mandated by  the Nuclear Regulatory Commission in 1982  concluded that several issues impeded  successful construction of the Zimmer Plant as  a commercial nuclear power plant. These  issues included (1) the company’s failure to  elevate its commitment to quality and quality  assurance to an equal status with cost and  schedule, (2) the regulator’s failure to hold the  company accountable for quality in design and  construction, and (3) the company’s  inadequate quality assurance procedures. To  recoup some of the $2 billion spent in  attempting to construct this commercial  nuclear power plant, Cincinnati Gas & Electric  later converted facilities built at the site for use  in a coal-fired power plant. management and the contractor place cost and schedule performance  above identifying and resolving quality assurance issues. One quality  assurance expert specified that ORP’s culture does not encourage  staff to identify quality assurance problems or ineffective corrective  measures. This expert said that people who discover problems are  not rewarded; rather, their findings are met with resistance, which has  created a culture where quality assurance staff are hesitant to identify  quality assurance problems or problems with corrective measures.  This expert added that quality assurance is subordinate to cost and  schedule—that is, senior managers responsible for approving quality  assurance findings are more concerned with whether WTP  construction meets schedule milestones than identifying and resolving  quality assurance issues. This expert compared the WTP to the  Zimmer Power Plant—a power plant in Ohio that was designed to be  a nuclear power plant but that was never licensed because of  unresolved quality assurance problems and a focus on schedule over  construction quality.

As stated earlier, in October 2008, we identified key elements that any  nuclear safety oversight organization should have in order for it to provide  effective independent oversight. For example, we found that an  organization should be structurally distinct and separate from DOE  program offices to avoid management interference or conflict between  program office mission objectives, such as cost and schedule  performance and safety. We also found that the organization should have  sufficient authority to require program offices to effectively address its  findings and recommendations.

ORP’s Assistant Manager for Technical and Regulatory Support and ORP  senior quality assurance staff told us that ORP’s organizational structure  ensures that the quality assurance function is sufficiently independent of  ORP management. These officials and the ORP Quality Assurance  Program Description state that the Quality Assurance Division is  structured to report directly to the ORP Assistant Manager for Technical  and Regulatory Support and the ORP Manager. They also cited the ORP  Quality Assurance Program policy, which states that the Quality  Assurance Division has the authority and overall responsibility to  independently audit the contractor’s quality assurance program to verify  the achievement of quality. According to these officials, this organizational  structure ensures independence from cost and schedule considerations  and ensures objectivity in quality assurance evaluations, and they added  that the ORP Manager evaluates differing opinions without any  hindrances or organizational bias.

Given that some previously identified problems are recurring at the WTP,  including some in areas where the contractor had implemented corrective  measures, and given the findings of the 2017 headquarters assessment  and the statements of ORP’s quality assurance experts outlined above,  we are concerned that ORP’s organizational structure may not entirely  ensure that the Quality Assurance Division meets key elements for a  nuclear safety oversight organization to provide effective independent  oversight. According to ORP reports and officials, in ORP’s current  organizational structure, upper level management retains discretion in  how to resolve quality assurance problems. As a result, the Quality  Assurance Division does not have sufficient authority to ensure that its  findings are addressed and its recommendations are implemented. By  revising ORP’s organizational structure so that the quality assurance  function is independent of ORP upper-level management, DOE can have  better assurance that compliance with nuclear safety requirements will  not be subordinated to meeting cost and schedule targets.

For years DOE has faced quality assurance problems at the WTP. Upon  learning in 2012 that it could not verify that engineering, procurement, and  construction at the WTP met nuclear safety and quality requirements,  ORP directed the contractor to implement quality assurance corrective  measures to ensure that problems would be identified and prevented from  recurring. However, 5 years later, the contractor has not fully  implemented all planned corrective measures. Moreover, in some areas  where the contractor has stated that corrective measures are now in  place, ORP continues to encounter quality assurance problems similar to  those it encountered in the past. When and where problems have  recurred, ORP has not always required the contractor to determine the  extent to which the problems may affect all parts of the WTP. By directing  ORP to require the WTP contractor, where quality assurance problems  have been identified, to determine the full extent to which problems exist  in all WTP structures, systems, and components, DOE will gain a  comprehensive understanding of all quality assurance problems at all  WTP facilities. In addition, ORP has not always used its authorities to  stop work when problems are detected before they are fully corrected.  Without directing ORP to use its authorities to stop work in areas where  quality assurance problems are recurring until it can verify that the  problems are corrected and will not recur, DOE may face future rework  that could increase costs and schedule delays for the WTP.

Also of concern is the potential lack of sufficient independence of ORP’s  Quality Assurance Division from ORP’s upper management. This has  resulted in ORP upper management not always allowing its own experts  to fully examine the contractor’s work even when problems have recurred.  At other times, this has resulted in the significance of identified  problems—and strength of associated corrective measures—being  reduced. DOE’s ability to effectively self-regulate a high-hazard nuclear  facility not only depends on vigorous oversight of the contractor by the  program office but also on active oversight by an independent group. The  WTP is the largest and most technically complex cleanup project  managed by DOE, and we recognize that meeting its cost and schedule  targets places immense pressure on ORP upper management. However,  meeting those targets is further threatened when quality assurance  problems are downgraded. By revising ORP’s organizational structure so  that the quality assurance function is independent of ORP upper  management, DOE can have better assurance that compliance with  nuclear safety requirements will not be subordinated to meeting cost and  schedule targets.

We are making the following three recommendations to DOE:  The Secretary of Energy should direct ORP to require the WTP contractor  to determine the full extent to which problems exist in all WTP structures,  systems, and components.

The Secretary of Energy should direct ORP to use its authorities to stop  work in areas where quality assurance problems are recurring until ORP’s  Quality Assurance Division can verify that the problems are corrected and  will not recur.

The Secretary of Energy should revise ORP’s organizational structure so  that the quality assurance function is independent of ORP upper  management.

We provided DOE with a draft of this report for its review and comment. In  its written comments, reproduced in appendix I, DOE generally agreed  with the findings in the report and its recommendations. DOE agreed with  our first two recommendations and described actions it has under way  and planned to address them. In addition, DOE agreed with our third  recommendation—to revise ORP’s organizational structure so that the  quality assurance function is independent of ORP upper management—in  principle. While DOE states that it believes that the current ORP quality  assurance reporting relationship meets all established requirements, it  also states that the report identifies instances that indicate that ORP  could be strengthened to improve the effectiveness and independence of  its quality assurance functions. In response to our recommendation, DOE  plans to direct ORP to assess the quality assurance functional reporting  lines, responsibilities, and processes to enhance the independence of the  quality function from cost and schedule influences and to strengthen and  clarify quality assurance reporting to the ORP Manager. This planned  action is a positive first step toward implementing our recommendation.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Energy; and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff members who made major  contributions to this report are listed in appendix II.

Appendix I: Comments from the Department of Energy

In addition to the contact named above, Nathan Anderson (Assistant  Director), Mark Braza, Scott Fletcher, Ellen Fried, Richard Johnson, Paul  Kazemersky, and Peter Ruedel made key contributions to this report.

The nuclear waste treatment plant DOE and its contractor are building at DOE’s Hanford site in Washington has faced persistent challenges, and the cost of the project has more than tripled to nearly $17 billion.

We found that DOE's quality assurance efforts did not always ensure detection of problems such as engineering errors and construction deficiencies, and some problems are recurring.

We recommended that DOE direct the contractor to determine the full extent of plant problems, stop work when problems recur, and establish a more independent quality assurance effort.