Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost Schedule and Performance Goals

The Navy's Columbia class ballistic missile submarines will replace the 14 Ohio class that currently provide the sea-based leg of the U.S. nuclear triad, slated to begin retiring in 2027. The first Columbia must begin patrols in 2031 to prevent a gap in deterrent capabilities; the class will ultimately carry up to 70 percent of the nation's strategic nuclear capability. The program is a top Navy priority with an expected cost of $267 billion over its life cycle, including $128 billion to research, develop, and buy 12 submarines.

House Report 114-102 included a provision for GAO to examine the Columbia class program. Among other things, this review examines (1) the status of key Columbia class technologies; and (2) potential risks with the Navy's planned approach for design and construction.

GAO reviewed the Navy's technology readiness assessment, technology development plan, and the status of key prototyping efforts, and compared efforts with GAO's identified best practices for shipbuilding programs and technology readiness assessments. GAO also assessed the status of design maturity and the Navy's acquisition strategy and interviewed relevant officials.

Additional development and testing are required to demonstrate the maturity of several Columbia class submarine technologies that are critical to performance, including the Integrated Power System, nuclear reactor, common missile compartment, and propulsor and related coordinated stern technologies (see figure). As a result, it is unknown at this point whether they will work as expected, be delayed, or cost more than planned. Any unexpected delays could postpone the deployment of the lead submarine past the 2031 deadline.

Further, the Navy underrepresented the program's technology risks in its 2015 Technology Readiness Assessment (TRA) when it did not identify these technologies as critical. Development of these technologies is key to meeting cost, schedule, and performance requirements. A reliable TRA serves as the basis for realistic discussions on how to mitigate risks as programs move forward from the early stages of technology development. Not identifying these technologies as critical means Congress may not have had the full picture of the technology risks and their potential effect on cost, schedule, and performance goals as increasing financial commitments were made. The Navy is not required to provide Congress with an update on the program's progress, including its technology development efforts, until fiscal year 2020—when $8.7 billion for lead ship construction will have already been authorized. Periodic reporting on technology development efforts in the interim could provide decision makers assurances about the remaining technical risks as the Navy asks for increasing levels of funding.

Consistent with GAO's identified best practices, the Navy intends to complete much of the submarine's overall design prior to starting construction to reduce the risk of cost and schedule growth. However, the Navy recently awarded a contract for detail design while critical technologies remain unproven—a practice not in line with best practices that has led to cost growth and schedule delays on other programs. Proceeding into detail design and construction with immature technologies can lead to design instability and cause construction delays. The Navy plans to accelerate construction of the lead submarine to compensate for an aggressive schedule, which may lead to future delays if the technologies are not fully mature before construction starts, planned for 2021.

GAO had suggested a matter for congressional consideration related to additional reporting on the Columbia class technologies, but removed it because of recent legislation that implements this requirement. Department of Defense comments on the draft were incorporated as appropriate in this report.

The Navy plans to buy 12 Columbia class nuclear-powered ballistic  missile submarines (called SSBNs) armed with nuclear warheads as the  replacement to the Ohio class, the current sea-based leg of the nation’s  strategic nuclear deterrent. The Columbia class fleet will replace the 14  Ohio class SSBNs as they begin to retire in 2027 after over 42 years,  longer than any prior class of submarine. The Navy plans to use new  technologies related to submarine propulsion, missile tubes, and stealth  to ensure that the Columbia class will remain functional throughout its  planned 42.5-year service life. The Navy has identified the Columbia  class program as its top acquisition priority and is investing significantly in  this effort—approximately $267 billion (then-year dollars) in total,  including $128 billion (then-year dollars) to research, develop, and buy 12  submarines. In fiscal year 2017, the Navy began buying materials and  starting detail design, with plans to begin construction of the lead  submarine in fiscal year 2021. To avoid a gap in the nation’s sea-based  deterrent as the Ohio class SSBNs retire, the lead ship will need to make  its first patrol in fiscal year 2031. Given the criticality of the program’s  deterrence mission and the magnitude of the cost and schedule  pressures, any challenges could have far-reaching consequences for the  nation’s defense.

In light of the Columbia class investment requirements, the House Armed  Services Committee report for the National Defense Authorization Act for  Fiscal Year 2016 included a provision that we examine the program. This  report examines (1) the status of key Columbia class technologies and  related reporting requirements; (2) risks, if any, with the Navy’s planned  approach for design and construction; and (3) whether expected funding  levels for the Columbia class will be adequate moving forward.

To assess the status of development of key Columbia class technologies,  we reviewed the Navy’s technology development plan and the status of  key prototyping efforts, and compared them with GAO’s identified best  practices for shipbuilding programs. We also reviewed the program’s  Technology Readiness Assessment and compared it to criteria in GAO’s  Technology Readiness Assessment guide. GAO’s guide draws heavily  from the Department of Defense (DOD), Department of Energy (DOE),  and National Aeronautics and Space Administration (NASA) best  practices, and establishes a methodology based on those best practices  that can be used across the federal government for evaluating technology  maturity, particularly as it relates to determining a program or project’s  readiness to move past key decision points that typically coincide with  major commitments of resources. We interviewed Navy officials and  analyzed available documentation related to the Navy’s technical efforts.  We also examined acquisition laws, regulations, and policies to determine  the reporting requirements for the Columbia class program following the  program’s Milestone B decision, which occurred in January 2017.

To assess the risks, if any, with the Navy’s planned approach for design  and construction of the Columbia class, we compared the status of design  maturity with Navy and shipyard plans to identify any delays. We also  compared planned design maturity and schedule projections with those of  prior U.S. submarine programs to assess realism of these estimates. We  assessed the program’s acquisition strategy, including plans to accelerate  the start of submarine construction and manage shipyard workload  across the Columbia and ongoing Virginia class submarine programs  (which will be built in the same shipyards) to identify factors related to  potential cost, schedule, and oversight risks. Our assessment leverages,  among other things, our prior work on shipbuilding programs.

To assess whether expected funding levels for the Columbia class will be  adequate moving forward, we compared program cost estimates to  historical data on lead ship cost performance and the Navy’s expected  budget for the program to assess the realism of these estimates. We also  reviewed the program’s life-cycle cost estimate and independent cost  estimate.

For all objectives, we interviewed officials from the Navy’s Columbia class  submarine program office; the Office of the Chief of Naval Operations- Undersea Warfare; Naval Sea Systems Command Naval Nuclear  Propulsion Program; Naval Undersea Warfare Center; Naval Surface  Warfare Center Carderock Division; Office of the Secretary of Defense  (OSD) Director Operational Test and Evaluation; Office of Naval  Intelligence; OSD Acquisition Technology and Logistics (AT&L); OSD  Cost Analysis and Program Evaluation (CAPE); and the prime contractor  shipyard General Dynamics Electric Boat and its sub-contractor  Huntington Ingalls Industries Newport News Shipbuilding, among others.

Appendix I presents a more detailed description of the scope and  methodology of our review.

We conducted this performance audit from May 2016 to December 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The U.S. strategic nuclear deterrent is spread among three legs, as  depicted in figure 1.

DOD has continued to reinforce the high priority of the Columbia class  program to the nation’s long-term defense. SSBNs are designed to  maximize stealth to remain undetected while on patrol at sea. This  survivability gives the United States a credible ability to retaliate if faced  with an attack targeting other legs of the triad, and explains DOD’s  decision to ultimately deploy up to 70 percent of the nation’s nuclear  warheads on SSBNs.

As stated in its April 2010 Nuclear Posture Review Report, DOD  determined that ensuring a survivable U.S. deterrent requires continuous  at-sea deployments of SSBNs in both the Atlantic and Pacific oceans, as  well as the ability to surge additional submarines in crisis. Currently, 14  Ohio class SSBNs provide the sea-based strategic deterrent. The Navy  commissioned the lead ship of this fleet in 1981. The first Ohio class  SSBN to retire—SSN 730—will leave service in 2027 and plans are to  retire one per year following this. When these submarines retire, they will  have been in service over 40 years, longer than any previous  submarines. Navy officials have stated that the legacy Ohio fleet cannot  be life-extended any longer than what is planned due to aging issues.

The U.S. Strategic Command (STRATCOM) retains operational control of  the strategic triad and determines how many SSBNs are needed to patrol  on a day-to-day basis. STRATCOM and the Navy have determined that  10 operationally available SSBNs are needed to meet mission  requirements. As a result, the lead Columbia class submarine must be  available for its first deterrent patrol in the first quarter of fiscal year 2031  to coincide with the planned 2031 retirement of SSN 734, or the Navy will  not have 10 operationally available SSBNs, thereby requiring DOD to  identify other steps to ensure it can meet current deterrent requirements.

The Navy expects that it can meet mission requirements with 12  Columbia class submarines carrying 16 missile tubes (equating to a total  of 192 available tubes) in lieu of 14 Ohio class submarines carrying 24  tubes (336 total available tubes). Currently, it takes 14 Ohio class  submarines to provide 10 operationally available SSBNs due to  maintenance needs that can take up to 4 submarines out of the patrol  rotation at any given time. The Navy plans to reduce the number and  duration of required maintenance periods for the Columbia class, allowing  just 12 Columbia class submarines to provide the required 10 operational  submarines at all times. Between fiscal year 2031-2040, the Navy plans  to have a mix of 10 operationally available Columbia and Ohio class  submarines. In fiscal year 2041, with the retirement of the final Ohio class  submarine, this is to increase to 11 Columbia class, and finally to 12  operationally available Columbia class submarines by fiscal year 2042.

The Columbia class program is comprised of several major lines of  effort—hull and supporting systems, the strategic weapons system; and  the nuclear reactor-based propulsion plant—which are managed by  different program offices, as depicted in figure 2.

The Navy is introducing new technologies to improve capabilities where  required while leveraging systems from existing submarine programs— the Virginia and Seawolf attack submarines and the Ohio class SSBNs— in order to ensure commonality with the submarine fleet and reduce  development needs for the Columbia class to limit technical risk. For  example, the program is re-using over 19,000 Virginia class standard  parts including fittings, valves, and switches and leveraging the Navy’s  Submarine Warfare Federated Tactical System program, which integrates  more than 40 independent electronics systems into a common combat  system for use by multiple program offices.

The Navy has identified several key technical efforts for the Columbia  class program: (1) the Common Missile Compartment, (2) Integrated  Power System, (3) Stern Area System, and (4) propulsor. Other systems  that we consider key technical efforts include the nuclear reactor and the  coordinated stern, a system-of-systems that includes the propulsor and  submarine maneuvering components. These areas are depicted in Figure  3 and defined below.

Since 2008, the United States and the United Kingdom (U.K.) have been  jointly developing a common system to house the tubes that will carry  submarine launched ballistic missiles. Columbia class SSBNs and U.K.

SSBNs will carry the Trident II D-5 missile for the first portion of their  respective operational lives; the U.S. missiles armed with nuclear  warheads which are maintained by the Department of Energy (DOE).  Figure 4 shows a notional example of the CMC.

In addition to the missile tubes, the CMC also provides systems to  support the missiles and the launch equipment, including power, cooling,  gas venting, and launch hardware and software. The Navy’s Strategic  Systems Program is responsible for CMC development efforts.

Integrated Power System (IPS) and Nuclear Reactor

The IPS includes an electric drive system to propel the submarine  through the water, unlike other current U.S. submarines which use a  mechanical drive system. IPS is powered by the nuclear reactor, which is  a separate system. As shown in figure 5, with a nuclear electric drive  system, steam from the nuclear reactor turns a turbine creating electricity,  which is then directly used to power electric motors. This is in contrast  with a nuclear mechanical propulsion system, where steam from the  nuclear reactor turns a turbine creating high-speed rotation; a reduction  gear then slows the speed of this rotation to a speed that is suitable for  use by the propulsor.

To provide power to the electric drive, the Columbia class nuclear  propulsion plant relies on a life-of-the-ship reactor core—called S1B—that  is planned to remain in service without refueling, almost 10 years longer  than current U.S. Navy nuclear reactors. The Virginia class also uses a  life-of the-ship reactor core, but the Columbia class reactor needs to be  more powerful to drive the larger submarine, and needs to last longer to  allow for the 42.5-year Columbia class service life of versus 33 years for  the Virginia class. By using a life-of-the-ship reactor, the Columbia class  will not require a mid-life refueling. This will reduce the mid-life  maintenance period from 27 months for Ohio class to 16 months for  Columbia class. This reactor is being developed by the Naval Nuclear  Propulsion Program (also known as Naval Reactors) and the Naval  Nuclear Laboratory (operated by Bechtel Marine Propulsion Corporation).

SAS is a technical feature of the stern that is comprised of three  subcomponents; details of which are classified.

The Columbia class will use a propulsor instead of a propeller to drive the  submarine through the water. The design of the propulsor relies on  several other technical features that form a system-of-systems,  sometimes referred to as the coordinated stern. The coordinated stern is  where the rudder and other control surfaces are mounted; these control  surfaces are used for submarine maneuvering and are critical to  submarine performance. The coordinated stern consists of interrelated  technology elements, including the propulsor and advanced propulsor  bearing, the stern control surface configuration, and the propulsor shaft  and bearing.

The propulsion shaft and bearing connects the propulsion system to the  propulsor, transferring energy from the propulsion system to the propulsor  to drive the submarine through the water. The Navy plans to use a new  design “X-stern” configuration instead of the cruciform stern used in other  submarines. Figure 6 depicts the major components of the coordinated  stern, omitting a depiction of the classified Stern Area System.

Acquisition Strategy for the Columbia Class

The Navy expects to require over $267 billion (then-year dollars) in total  life-cycle costs for the Columbia class program. Figure 7 shows the  break-down of this amount between operations and support costs and  acquisition costs, as well as the elements comprising the $128 billion in  acquisition costs.

The approximately $128 billion total acquisition cost includes funding the  Navy expects it will need to research, develop, and build its Columbia  class SSBN.

Due to their size and complexity, submarines require funding for design,  long-lead materials (such as nuclear propulsion plant components), and  construction over many years. To accomplish these activities, the Navy  awards contracts over several phases of design and construction. Figure  8 outlines major acquisition plans for the Columbia class.

In 2014, Congress created a National Sea-based Deterrence Fund to  provide DOD with greater discretion to fund the design, construction, and  purchase of the Columbia class. Since then, Congress has provided the  Navy with enhanced acquisition authorities to buy and construct  submarines and certain key components early, in bulk, and continuously.

The Columbia class program entered the Technology Development  phase of the defense acquisition process in January 2011. The schedule  to acquire the Columbia class was shifted in 2011 when the Navy decided  to delay the start of construction of the lead submarine by 2 years—from  2019 to 2021—due to budget constraints. The first patrol date for the lead  ship was also shifted from fiscal year 2029 to fiscal year 2031. In January  2017, the Columbia class program achieved Milestone B—considered the  official start of a DOD acquisition program—and moved into the  Engineering and Manufacturing Development phase of the acquisition  process. The program does not envision holding a Milestone C, which  typically denotes a program’s approval to enter the production and  deployment phase as shown in figure 9, but does plan to have an OSD- level review prior to authorizing the construction of the lead ship.

Shipbuilding programs have slightly different decision points than other  DOD weapon systems, partly because of the timing of the Milestone B  decision for ships. Milestone B for ship programs usually occurs after  development of ship specifications and system diagrams is well under  way.

As part of the Columbia class Milestone B decision, OSD approved a Low  Rate Initial Production quantity of 12 submarines, the total quantity  expected for the class. According to the Navy, the program awarded a  $5.1 billion detail design contract to Electric Boat in September 2017 for  work including design completion, component and technology  development, and prototyping efforts. Detail design is typically funded  with Shipbuilding and Conversion, Navy funds (the Navy’s procurement  fund for buying ships) and represents a further refinement of the design of  the ship and ultimately generation of work instructions needed by the  shipyard in advance of lead ship construction. The program was granted  approval to begin early detail design work in January 2017.

In shipbuilding, the design phase generally encompasses three activities:  basic design, functional design, and detail design. These steps occur  after the Navy sets the technical requirements for the ship. At a high level: basic design serves to outline the steel structure of the ship;  functional design routes distributive systems—such as electrical or  piping systems—throughout the ship; a three-dimensional (3D)  computer-aided design model is often generated; and detail design completes the design work for even the lowest-level  items, and ultimately furnishes the work instructions for the shipyard  workers to use in constructing the ship. During this phase, all aspects  of the ship are defined, and two-dimensional paper or 3D electronic  drawings (also called work instructions) are generated.

For the Columbia class program, the Navy defines design in two phases:  arrangements, which program officials describe as a combination of basic  and functional design; and disclosures, which they describe as a  combination of detail design and generation of work instructions. Figure  10 shows the phases of design for the program as compared with typical  surface ship terminology.

Two shipbuilders—General Dynamics Electric Boat and Huntington  Ingalls Industries Newport News—are responsible for designing and  building nuclear submarines. For the Columbia class program, Electric  Boat is the prime contractor for design and construction, with Newport  News as a subcontractor. Similar to the Virginia class program, each  shipyard will build modules of the submarine, but Electric Boat will be  responsible for final delivery of the submarine to the Navy.

For more than a decade, our work on major acquisitions has shown that  part of an effective management process is assessing how far a  technology has matured and how it has been demonstrated, which  indicates the technology’s readiness to be integrated into a system and  the degree of program risk. DOD acquisition instruction requires that  programs complete a technology readiness assessment (TRA) at  Milestone B. A TRA is a systematic, evidence-based process that  evaluates the maturity of hardware and software technologies critical to  the performance of a larger system or the fulfillment of the key objectives  of an acquisition program. A reliable TRA illuminates concerns and  serves as the basis for realistic discussions on how to mitigate potential  risks as programs move from the early stages of technology  development.

TRAs do not eliminate technology risk but, when done well, can illuminate  concerns and serve as the basis for realistic discussions on how to  mitigate potential risks as programs move from the early stages of  technology development, where resource requirements are relatively  modest, to system development and beyond, where resource  requirements are often substantial. In addition, TRAs help legislators,  government officials, and the public hold government program managers  accountable for achieving their technology performance goals.

A main element of a TRA is the identification of critical technology  elements (CTE) and assessment of the appropriate Technology  Readiness Level (TRL), used to measure the readiness of technologies to  be incorporated into a weapon or other type of system. TRLs range from  1 (least mature) to 9 (most mature), as shown in table 1.

Current DOD guidance assigns the program manager responsibility for  identifying CTEs. The program manager identifies possible technologies,  then, in consultation with officials from the Assistant Secretary of Defense  for Research and Engineering—ASD(R&E)—and with the program  executive office and component acquisition executive approval, identifies  the subject matter experts needed to perform the TRA. For the  Columbia class TRA, the expert team was comprised of Navy program  management and technical personnel. ASD(R&E) reviews the list of  critical technologies provided by the program manager and recommends  technologies to add or delete. Ultimately, the program submits the TRA  report to ASD(R&E), who independently assesses the maturity of the  technologies. The ASD(R&E) prepares a memorandum based on the  assessment that is transmitted to the milestone decision authority, along  with the TRA Report.

The TRA is also an element of the Milestone B approval process. Section  2366b, title 10, U.S. code states that a major defense acquisition program  may not receive Milestone B approval until the milestone decision  authority has, among other things, certified that the CTE has been  demonstrated at a TRL 6. A program may request a waiver from OSD if  the maturity provision cannot be met. The statute requires that:

Every waiver determination must be submitted in writing to the  congressional defense committees within 30 days after the waiver  request by the program is authorized.

The milestone decision authority reviews the program not less often  than annually until the milestone decision authority determines that  the program satisfies all certification and determination components.

In addition, in 2015 Congress required program acquisition strategies to  include a comprehensive approach to risk management, including the  consideration of techniques such as technology demonstrations and  decision points for disciplined transition of planned technologies into  programs or the selection of alternative technologies.

Recognizing the importance of the TRA to risk management, in 2016,  GAO developed a Technology Readiness Assessment Guide. This  guide has two purposes: (1) to describe generally accepted best practices  for conducting effective evaluations of technology developed for systems  or acquisition programs; and (2) to provide program managers,  technology developers, and governance bodies with the tools they need  to more effectively mature technology, determine its readiness, and  manage and mitigate risk. As noted above, we developed the guide by  drawing heavily from DOD, DOE, and NASA best practices, terminology,  examples, and credible resources, materials, and tools developed and  applied by experts and organizations in order to capture the current  thinking on technology readiness and maturity. In our guide, we identify  criteria for a CTE, namely that it is a technology that is “new or novel, and  needed for a system to meet its anticipated operational performance  requirements; or that poses major cost, schedule, or performance risk  during design or demonstration”. According to our guide, re-used  existing technologies can also become critical if they are being used in a  different form, fit, or function—as is the case with the propulsor and  coordinated stern.

Major Funding Commitments Planned, but Reporting on the Progress of Several Key Immature Technologies Is Not Required

Several key technical efforts remain immature as the Columbia class  program moves into its design phase—a practice counter to best  practices we have previously identified. These efforts include the  integrated power system, nuclear reactor, propulsor/coordinated stern,  stern area system, and common missile compartment. While the Navy  made progress in some areas—such as prototyping efforts for the missile  compartment and nuclear reactor—all of these systems continue to  require development and testing to mature them to TRL 7, the point at  which GAO’s technology readiness guide considers a technology mature.  Any challenges in development could put the program at risk of costing  more, taking longer to develop, or jeopardizing the program’s ability to  meet its expected performance requirements. However, the Navy  identified only two of the submarine’s technologies as “critical” in the  program’s 2015 TRA, thereby underrepresenting the technology risk in  the program. Underreporting technical risks can hinder Congress’ and  other decision makers’ full understanding of the program’s progress. This  is especially important because the Navy has already requested $1.6  billion for advanced procurement and recently awarded the detail design  contract. Moreover, there is no requirement that the Navy report to  Congress on its progress in developing and testing the technologies until  after the program completes its production readiness review in May 2020  after the Navy requests another $8.7 billion in funding for the construction  of the lead submarine.

Several Technologies Remain Immature as Detail Design Begins

Demonstrating Technology Maturity  Based on our work on best practices in  weapon system acquisitions, we have  previously recommended that programs fully  mature technologies to TRL 7—versus TRL 6  as required by DOD—prior to passing  Milestone B and entering the engineering and  manufacturing development phase. TRL 7  represents a major step up from TRL 6,  requiring demonstration of an actual system  prototype in an operational environment such  as in an aircraft, vehicle, or space. We have  previously identified that demonstrating  technologies in an operational environment  provides a higher level of technology  understanding and reduces risk prior to  starting product development. DOD has  historically disagreed with this recommended  practice. added that modeling and simulation should be considered appropriate in  some cases in lieu of actual prototype testing.

While the Navy has made progress in reducing technical risks in many  areas, such as starting construction of the first CMC, the program  (according to the Navy) awarded a detail design contract in September  2017, with several key technologies not yet at a TRL 7.

The nuclear reactor, IPS, propulsor and coordinated stern, and SAS all  have potentially significant effects on design and construction of the  Columbia class because they encompass much of the design and  physical structure of the submarine. Based on our analysis, we found that  IPS, SAS, the propulsor and coordinated stern are not yet at a TRL 7, as  depicted in figure 11. The nuclear reactor and CMC are further along in  prototyping work but still require testing in an operational environment to  achieve a TRL 7.

If any of these systems do not develop as planned, the Navy and the  shipyards could be required to complete some redesign, or, if risks  manifest later, they may force costly workarounds or construction rework.  In addition, these systems also enable many performance attributes  ranging from weapon launch to speed and maneuverability, so  performance could be negatively affected. The status of these  technologies is discussed in detail below.

According to officials from Naval Reactors, the permanent magnet motor- based electric drive system—a key component of IPS for the Columbia  class—is at a TRL 6, below the TRL 7 recommended by our work on best  practices. Naval Reactors has yet to develop an IPS prototype that is  near or at the planned operational system configuration (integrated and  full-size) and has been tested in an operational environment. The Navy  has experimented with electric drive technology on submarines in the  past with two now-decommissioned nuclear-powered attack submarines,  but these submarines used different motor technology than what is  planned for the Columbia class, and thus are not representative. The T- AKE 1 Lewis and Clark class of dry-cargo ammunition ships and DDG  1000 Zumwalt class destroyer are current U.S. Navy electric drive ships  in operation, but these two systems are somewhat different than what is  planned for the Columbia class and neither is powered by a nuclear  reactor. The Navy is currently developing the IPS and producing a  number of pre-production prototypes.

Naval Reactors officials told us that they are confident that the IPS will  meet requirements based on 20 years of development and testing of the  underlying permanent magnet motor technology. They also noted that this  technology is proven based on testing of the smaller-scale prototype  motor to validate the main propulsion motor design. However, Naval  Reactors is still developing and producing the system’s major  components. Testing of a full-scale prototype under full power, which we  would consider evidence that the technology is mature, is not scheduled  to occur until fiscal years 2018-2020. In a land-based test facility, the  Navy plans to integrate all the IPS systems in a ship-representative  layout. Successful completion of this testing is an important step in  mitigating risk. In contrast, the DDG 1000 program only tested its electric  drive system at the land based test facility at one-half of the ship’s power  generation and electric propulsion system configuration, and as a result  performance problems were not discovered until well after installation and  when system testing on the ship was run at full power. Thus, the Navy’s  planned full-scale prototype testing for Columbia class should prevent a  similar experience, since it will test a full-sized and full-power system  rather than a partial system.

According to officials from Naval Reactors, as a result of its statutory  mandate, its programs follow a different development process than typical  DOD programs and do not use documents typical of other Navy  programs, such as an Integrated Master Schedule or a Test and  Evaluation Master Plan. Instead, officials from Naval Reactors told us that  they use a rigorous process to assess, manage and control technical risk  during development and testing to manage its day-to-day technical  efforts. Based on descriptions provided by Naval Reactors officials, the  Navy has been operating a Columbia-like experimental reactor in a land- based environment for many years to demonstrate some Columbia class  submarine systems. Naval Reactors officials said that this experience  gives them confidence that the Columbia class reactor will be delivered to  the shipyard on time and will meet all requirements.

Naval Reactors has design and development work remaining before it  awards the contract for reactor core production in fiscal year 2019. Naval  Reactors budget documentation shows that reactor design work is  planned to be 65 percent complete in fiscal year 2018. While we  recognize that it would not be realistic to expect Naval Reactors to test  the reactor in a submarine to achieve a TRL 7, a completed design would  still be required to produce a final configuration to demonstrate  technology maturity.

Neither the propulsor nor other related components of the coordinated  stern have been demonstrated through testing in a near or planned  operational system configuration, a key element for achieving TRL 7.  Navy officials told us that the propulsor effort is based on prior experience  with propulsors and that it will resemble the Virginia-class propulsor  design. However, according to Navy documentation, the propulsor will be  different in form, fit, and function than prior propulsors, and the final  configuration has yet to be selected or tested. Specifically, the following  components require additional design work and testing prior to  demonstrating a representative prototype:

Propulsor: The Navy is working with various partners to refine two  different high-level propulsor designs. The program also faced a year  delay in completing the first phase of design work, which  subsequently delayed large-scale vehicle testing. Further, the Navy  still has to complete large-scale prototype testing of different propulsor  designs that are being evaluated for an eventual down-select to one  vendor for production.

Propulsor shaft: The system that connects the propulsion to the  motors—which the Navy states is similar to shafting systems used on  previous submarine classes but with different materials and size and  weight—is still in concept and preliminary design phases. Main shaft  design development and testing is being performed to select materials  and inform design efforts.

Advanced propulsor bearing: The Navy has yet to complete the  preliminary design of the advanced propulsor bearing, with prototype  test in a full scale configuration planned to begin in fiscal year 2019.  Navy officials told us that they believe that the final design and  material selections will exceed the reserved weight and size margins  of the shafting or bearing system.

X-stern: the final X-stern configuration has yet to be tested with a final  design propulsor.

Our assessment of the propulsor and coordinated stern system design  indicates that it is not yet mature enough to provide the basis for a  prototype in final form, fit, and function—key elements of achieving TRL  7.

The Navy identified the SAS as a TRL 4 at Milestone B. The preliminary  design review for the SAS is planned for March 2018. This review  establishes the baseline (hardware, software, human/support systems)  and underlying architectures to ensure that the system has a reasonable  expectation of satisfying requirements within the current budget and  schedule. The critical design review—a technical review that ensures that  a system can proceed into fabrication and demonstration and can meet  stated performance requirements within cost, schedule, and risk—is not  planned until March 2020.

A TRL 4 represents a relatively low level of maturity compared to the  eventual system. At this low level of maturity, there are no assurances  that the SAS will work as planned, which would likely result in the  Columbia class not meeting certain requirements or in cost and schedule  increases. The Navy plans to hold a critical design review for SAS in  fiscal year 2019. The Navy has identified existing fleet technologies as  backups for two SAS components, but officials noted that if these are  used the submarine will not meet current requirements. According to the  program office, there is no backup technology for one other SAS  component, and, if that element—currently a TRL 4—does not develop as  planned, it will be omitted, meaning that the program will lack that  capability.

Specific details of SAS are classified and cannot be included in this  report.

The shipbuilders and the Navy have described CMC as complex to build.  The Navy and the two shipyards—with consultation from the United  Kingdom, which will also leverage the CMC design on its new SSBN— have conducted risk-reducing prototyping work and are building a  representative CMC to demonstrate production processes. In fact,  Columbia class representative missile tubes will be first installed on a  United Kingdom submarine, scheduled for mid-2020. The Navy has plans  for a robust land-based test procedure for both the missile tubes and the  CMC as a system that will provide an operationally similar environment to  a submarine; however, this testing has yet to start and will not conclude  for several years.

The Navy Has Not Appropriately Identified Technologies as Critical, Which Underrepresents the Program’s Technical Risk

While the Navy conducted the 2015 Columbia class TRA in accordance  with a DOD-approved plan, it did not follow our identified best practices  for identifying all critical technology elements (CTE), resulting in an  underrepresentation of the technical risk facing the program. Specifically,  the TRA only identified 2 CTEs: the SAS and a carbon dioxide removal  system. CTEs are required to be at TRL 6 at Milestone B (the official start  of a program). For the Columbia class program, OSD approved Milestone  B in January 2017. The Navy received a waiver at Milestone B for the  SAS because the system was still immature, as discussed above. The  carbon dioxide removal system has matured since the TRA following  demonstration on an operational submarine, and no longer requires  active risk mitigation efforts.

We compared the Navy’s 2015 Columbia class TRA to criteria  documented in GAO’s TRA Guide and DOD’s own guidance. In doing so,  we found that 4 additional key technical efforts—IPS, nuclear reactor, and  propulsor/coordinated stern, and the CMC—meet the criteria for a CTE.  Since the Navy did not identify these technologies in the TRA, it also did  not assign them a TRL. Their exclusion is significant because the 2015  TRA represents a key independent review and technical risk assessment  used by DOD to certify to Congress that the Columbia class program’s  technologies had been demonstrated in a relevant environment (TRL 6)  at Milestone B. Because not all of the CTEs were identified, DOD and  Congress lack an important oversight tool for assessing technology  maturity and evaluating program risk. Further, this certification is the only  required reporting on technology development prior to the Navy  requesting authorization for construction of the lead ship. Some of the  concerns that we identified are discussed in detail below.

Conflicting Criteria for Identifying Critical Technologies

The team responsible for preparing the 2015 Columbia class TRA did not  identify all appropriate CTEs because it used a more restrictive definition  of a CTE than that recommended in our best practices guide and DOD’s  2011 TRA guide. Table 2 compares the criteria in the three sources.

As reflected in table 2, not only does the Navy’s TRA definition require a  technology to meet a number of criteria to be considered a CTE, it also  has to be considered a technology development effort. According to the  Columbia class program office, the TRA team based this definition on a  2011 OSD AT&L memorandum issued contemporaneously with the 2011  TRA guidance that states: “TRAs should focus only on technology  maturity, as opposed to engineering and integration risk.” However, our  analysis of this memo found that it also directs programs to use DOD’s  TRA guidance and CTE definition, which are broader and more consistent  with our definition of a CTE. The 2015 Columbia class TRA does not  further define what constitutes a technology development effort, with the  Navy applying this as a criterion without defining what the criteria actually  meant. Moreover, the TRA does not provide any definition or criteria for  what it considers engineering and integration risk. We determined that the  Navy under-identified program technical risks because the Navy’s criteria  were more restrictive than GAO’s CTE definition.

Several Critical Technologies Not Identified

We further assessed the specific technologies in the Columbia class  program against our technology readiness criteria for a CTE, as shown in  table 3.

As shown in table 3, by applying the additional “technology development  effort” criteria in the 2015 Columbia class TRA, the TRA team eliminated  several systems from CTE consideration without criteria or a definition of  what constitutes a technology development effort. Some of these systems  were previously identified as CTEs in other recent Navy documentation.  The TRA team did not identify the nuclear reactor as a CTE because this  system is under the cognizance of Naval Reactors and not the Columbia  class program office. Officials from Naval Reactors told us that they do  not conduct TRAs, but rather follow a different and more iterative process  to manage their technology development efforts.

While the Navy did not identify all of the program’s CTEs as compared  with the TRA criteria in our guide, it is tracking these efforts to manage  technology risks. For example, 3 of the 4 CTEs we identified are also  identified in Navy documents as “key technical efforts” with active risk  mitigation plans. We will continue to track the progress of these efforts  in our future work.

Required Report to Congress on Technology Efforts Will Not Occur Until after Lead Ship Authorization

As the Columbia class program moves into its detail design and  construction phase, it will be more than 2 years before the next  requirement for a formal DOD report to Congress on the progress of the  technology efforts. This will occur at some point after the program’s  Production Readiness Review is completed in May 2020. In the  meantime, the Navy plans to request another $8.7 billion (in addition to  the $1.6 billion already requested) for lead ship construction. If a typical  budget schedule is followed, this request will come before Congress in  February 2020. The Navy plans to begin construction of the lead  submarine starting in fiscal year 2020. Congress will be asked to approve  lead ship construction absent key information on the maturity of the  critical technologies that, at present, are not up to the maturity levels that  would provide assurance they will work as intended. Without additional  updates on the progress of technology maturity between now and 2020,  we believe Congress will not have information it needs to evaluate  technical risk in advance of the Navy’s requests for considerable  increases in program funding. As previously discussed, there is currently  no DOD requirement to submit such reports to congressional oversight  committees.

The Navy Plans to Leverage Completed Design to Mitigate Aggressive Schedule, but Ongoing Technology Development Likely to Undermine This Goal

The Navy is prioritizing design completion before starting construction,  which is a good practice that is in accordance with our work on best  practices because it helps reduce cost and schedule challenges in  construction. However, since some of the key technologies are not fully  matured, detail design work is proceeding with notional or placeholder  data representing these key systems. As a result, the design will likely  remain immature once construction starts in fiscal year 2021. We have  previously reported that concurrency of technology development and  design increases the risk of design rework—or having to make  modifications to design drawings to accommodate any changes needed  as a result of technologies changing size, shape, or weight as they  mature—and potentially can result in negative cost and schedule  impacts. Further, the Navy faces an aggressive production schedule in  order to deliver the lead submarine by fiscal year 2031, which will be  required to prevent a gap in U.S. nuclear deterrent capabilities. According  to our analysis of previous submarine program schedules, the Columbia  class program’s schedule is aggressive in its expected short duration to  build the lead submarine. The program office intends to mitigate this  schedule challenge, in part, by starting construction of portions of the  submarine earlier than initially planned. If this early construction occurs  and the Navy does not alter design plans, construction of some parts of  the lead submarine could outpace a finalized design for developing other  components, which increases the risk of rework during construction and  could further delay completion.

Consistent with Best Practices, Program Has Prioritized Design Completion, but Immature Technologies May Compromise Design Maturity

The Columbia class program is prioritizing a high level of design  completion prior to the start of construction of the lead submarine of the  class. The program plans to complete 100 percent of design  arrangements and 83 percent of design disclosures prior to the start of  construction of the lead submarine. In our 2009 report on best practices in  shipbuilding, we identified design maturity as important step in reducing  cost and schedule risk. As such, we recommended that the design be  stabilized through completion of basic and functional design and 3D  product modeling prior to the start of construction for a new ship.  Because, as mentioned previously, the Navy defined design  arrangements on the Columbia class program as being equivalent to  basic and functional design, having 100 percent of the arrangements  completed prior to the start of Columbia class construction would meet  the intent of our prior recommendation.

Further, our analysis found that the Columbia class program’s planned  level of design completion prior to starting construction is much higher  than most recent Navy shipbuilding programs. For example, the Virginia  class attack submarine program started construction with only 43 percent  of the design complete compared with a planned 83 percent completion  for the Columbia class. The Columbia class program also plans to have a  52 week buffer between the completion of design for an area of the  submarine and the start of construction on that area, which is intended to  allow time to address any challenges that may arise and thus minimize  schedule delays. Additionally, the Navy plans to have all components fully  developed 8 months before they are required in the shipyard, which will  provide some additional schedule buffer to address challenges before the  components are actually needed for construction.

To facilitate design completion, the Navy made a commitment at the start  of the program to set realistic and reasonable requirements and to keep  those requirements stable throughout the program. This approach is also  in keeping with our previously identified best practices, which highlight the  importance of demonstrating balance among program requirements,  technology demands, and cost considerations. The Columbia class  program has not had any significant requirements changes since DOD’s  Joint Requirements Oversight Council validated the Capability  Development Document in 2015. Setting realistic and reasonable  requirements also permitted the Navy and shipyards to reuse some  design elements for components of the submarine that are similar in  design and function to the Virginia class instead of requiring new design  work. Similarly, the program has worked to keep stable ship specifications  to minimize design disruptions.

The technical specifications for the ship have been set since 2014, and  the program manager maintains personal visibility and accountability over  any proposed deviations or changes to the specifications. According to  the program manager, to date there have been minimal changes made to  the technical baseline. These steps help to minimize design rework that  can be caused by changing requirements, as was seen on the Littoral  Combat Ship program, and that can lead to cost increases or scheduled  delays. The program has also conducted some prototyping efforts— including building representative portions of the submarine to  demonstrate that its design tool can send the correct information to the  shop floor to build the ship—and has plans for more.

However, based on our analysis of the program’s current technology  development plan and status, it is unlikely that the Navy’s planned 83  percent of design disclosures will be finalized at the time construction  begins for the lead ship in 2021. Similar to many shipbuilding programs,  the Columbia class program plans to continue to mature technologies into  their final form while detail design is underway. As we have previously  reported, to offset this risk, shipbuilding programs, including the Columbia  class, often include design “reservations” for space, weight, power,  cooling, and other key attributes to reserve a footprint for components. As  contractors or government employees develop and refine technologies or  systems, they provide vendor furnished information (VFI) or government  furnished information (GFI) to the shipyards to update the design.  Completion of the detail design of the submarine—and subsequent  achievement of design stability to support a properly sequenced  construction phase—requires shipbuilders to have final information on the  form and fit of each system that will be installed on the ship, including the  system’s weight and its demand for power, cooling, and other supporting  elements.

As development proceeds on a new technology, initial assumptions about  size, shape, weight, and power and cooling requirements can change,  potentially significantly. These changes in VFI or GFI—if not resolved  early in the design phase—can introduce considerable volatility to the  design process for a lead ship. As such, in our May 2009 report, we  recommended that, to attain the level of knowledge needed to retire  design risk and reduce construction disruptions, complete—versus  notional—VFI or GFI must be incorporated for the design to be truly  stable. DOD concurred with this recommendation. We have previously  reported that other Navy programs have run into difficulties, including out- of-sequence or more costly construction work, when space, weight,  power, and cooling reservations are based on immature or ill-defined  technologies or components that have changed in size, weight, or other  attributes when they are finalized. Ramifications from such changes can  ripple through much of the ship design. For example, we reported in 2009  that during construction of the Seawolf-class attack submarine, the  AN/BSY-2 combat system did not fit into the space and weight  reservations that the Navy had allocated within the submarine’s design.  As a result, a portion of the submarine had to be redesigned at additional  cost.

However, the Navy has entered the detail design phase for the Columbia  class with incomplete technical data on several key components that are  either significant in size relative to the submarine or spread throughout a  number of spaces of the submarine. These components include IPS, the  nuclear reactor, the propulsor and coordinated stern, and SAS. This  situation is problematic because even if the Columbia class design is 83  percent complete, if it contains many reservations for systems that are not  fully developed the design will continue to be immature and subject to  change. Thus, the 83 percent completion metric may be somewhat  meaningless since elements of the design are uncertain and could  change because of the incomplete technology development efforts.

As shown in figure 12, the Columbia class program has entered the detail  design phase with a number of technologies still in development or design  finalization, which means that the VFI/GFI for these systems are not yet  final. This figure also depicts our recommended knowledge points for  shipbuilding programs, which align with contract award for detail design  and the start of lead ship construction. The concurrency depicted  between phases could be further exacerbated if the Navy pursues plans  to start construction of some components early.

As is shown in figure 12, the Navy plans to continue technology  development while executing detail design; this concurrency may  potentially extend through construction if the Navy pursues its plans for  early construction. For example, the Navy and the shipyards are currently  designing the stern of the submarine—with 95 percent of stern  arrangements planned to be complete by December 2017—but the final  configuration of the propulsor has yet to be determined. As currently  planned, the Navy will not complete prototype testing until the third  quarter of fiscal year 2020, and development and design of the SAS is  planned to continue until the end of fiscal year 2021—almost a year after  the start of lead ship construction. The Navy believes it is managing this  stern risk by controlling the interfaces through an Interface Control  Document that identifies set design constraints. According to Navy  officials, all aspects of the propulsor design that could impact the overall  ship design such as size, weight, and arrangements of major sub- assemblies of the propulsor are already finalized, and that the systems  are currently tracking to the reservation allowances. However, until a final  representative prototype is tested as a system, the possibility of design  changes and broader design impacts remains. Although the Navy plans  to have arrangements for the stern 100 percent complete at construction  start, the VFI or GFI for these important systems will not be finalized until  later after these systems finish development.

Additionally, the electric drive of IPS has already experienced  manufacturing problems that could compromise its ability to meet its  schedule if further challenges arise. According to Naval Reactors officials,  a manufacturing defect was identified in February 2017 that affected the  assembly of the first production-representative propulsion motor intended  for installation in the land-based test facility to prove out the integration of  all the electric drive components. The officials explained that the vendor  responsible for the motor is in the process of repairing the defect—a  process that will take up to 9 months to complete. As a result, Naval  Reactors is now executing a schedule recovery plan to regain some  schedule margin. Part of this plan involves using a smaller scale  prototype motor in initial land-based test facility testing to prove out  system integration. This plan means that initial full-scale system testing  will be conducted with a different motor, albeit one with the same  electromagnetic properties. Further, this delay will leave less margin to  account for any unexpected challenges encountered in developmental  testing.

Aggressive Construction Schedule for Lead Submarine Unprecedented

The Columbia class program has an aggressive schedule to deliver the  lead submarine in time to begin patrols in fiscal year 2031. The Navy  plans for 84 months, or 7 years, to build the lead submarine. While  imperatives associated with our nation’s nuclear deterrent are driving this  planned schedule, our analysis shows that it is significantly shorter than  what the Navy has achieved on any recent lead submarine construction  effort—including during high levels of Cold War submarine production.  The Navy expects that the Columbia class will be built in the same  timeframe as was planned for the lead Virginia class submarine—a  submarine that is one and a half times smaller and has less estimated  construction man hours than the Columbia class. Figure 13 shows the  estimated and actual timeframes for constructing prior lead submarines  as compared with the 84 month estimate for the Columbia class lead  submarine.

Further, there are industrial base implications to this aggressive schedule.  The Navy and the two shipyards will be trying to attain this level of  unprecedented schedule performance with the lead submarine while the  shipbuilders are also starting work on the first few Virginia class  submarines built in a new Block V configuration. Virginia class program  officials told us that the ramp-up to building two attack submarines per  year has resulted in recent cost and schedule growth at the shipyards.  The addition of Block V and Columbia-class will likely create additional  schedule pressures with the increase in workload required to build those  submarines compared with non-Block V version submarines.

In an effort to mitigate the risks associated with its aggressive delivery  schedule, the Navy is planning to start construction of a number of parts  of the structure of the lead submarine years earlier than the date of lead  ship authorization in fiscal year 2021. This plan, called advanced  construction, would use expanded acquisition authorities provided by  Congress in the National Sea-Based Deterrence Fund. The Navy and  its shipbuilders intend to start construction as early as 2019 on numerous  areas of the submarine’s structure. Specifically, the Navy and shipyards  plan to start building the stern, bow and missile command and control  module as early as 6 months before fiscal year 2021, citing the disruptive  effects of delays to these three “super-modules” that are also critical to  ensuring an on-time delivery. These super-modules also comprise vital  areas of the submarine, including the CMC, IPS and the coordinated  stern. The shipyards have proposed moving 500,000-600,000 labor hours  of construction work to before ship authorization. Figure 14 shows the  super-modules of the submarine that the Navy plans to start early.

However, the Navy has yet to finalize or fund the approach for this type of  early work. Starting construction early for the lead and follow submarines  provides schedule relief to the Navy and shipbuilders, but these plans  may further exacerbate the existing overlap of technology development  and design and construction, which was discussed above. Moving  construction earlier could challenge the Navy’s goal to have all  components developed 7 months before they are required in the  shipyard. Further, the shipbuilders acknowledge that early construction  plans will result in increased overlap between various stages of design  activities in certain areas, including the bow and stern. If Congress funds  the Navy’s plans to fund advanced construction work, this incomplete  VFI/GFI situation will likely be worsened and could disrupt the optimal  build strategy.

We have previously reported that programs starting construction of the  lead ship of a class without a mature, stable design has been a major  source of cost growth and schedule delays in Navy programs. We have  also reported that when a schedule is set that cannot accommodate  program scope, delivering an initial capability is often delayed and higher  costs are incurred because problems typically occur that cannot be  resolved within compressed, optimistic schedules. The Navy’s Columbia  class plans put the program at risk of cost and schedule growth.  However, its options for reducing concurrency are, at this point, limited  due to the schedule imperatives driven by the lead ship patrol deadline.

Columbia Class Is Not Funded Adequately to Address Program Risks

Our analysis determined that it is more likely than not that the Columbia  class program will exceed the Navy’s $128 billion (then-year dollars)  estimate of total acquisition cost to which the program will be funded.  Specifically, the program’s 2017 Milestone B cost estimates are optimistic  because they do not account for a sufficient amount of program risk due  to ongoing technology development, as well as the likely costs to design  and construct the submarines. In addition, the Navy has budgeted the  program to a confidence level for the program that is lower than what  experts recommend, with a particularly optimistic estimate for the lead  ship. While there may be situations when this would be appropriate, this  is not the case for the Columbia class program due to the technical and  design risks that we identified above. As a result, program costs will more  likely than not exceed requested funding, particularly for lead ship  construction. Due to the significant level of funding required for this  program, even a small percentage of cost growth could have far-reaching  consequences on the Navy’s long-range plans to fund construction of its  future fleet. For this review we conducted an initial analysis of the Navy’s  cost estimate but did not assess if it was conducted in accordance with all  of the best practices identified in our cost estimating guide. We plan to  more fully assess the Navy’s life-cycle cost estimate for the entire  Columbia class, including the program’s risk analyses, in future work.

Confidence Levels and the Navy’s Estimate

Confidence Levels  A confidence level is stated as a percentage  depicting the probability that the program’s  cost will actually be at some value or lower,  calculated after conducting a risk analysis to  identify and quantity program risks and  determine the effects of these risks on its  point estimates.

From early on, the Navy recognized the need to control costs for the  Columbia class. In fact, the program’s cost estimates have decreased  significantly since the program’s inception due to Navy decisions early in  the program to trade off some capabilities and the incorporation of  updated actual cost data from the continued procurement of Virginia class  submarines. At Milestone B, OSD determined that Columbia class  procurement costs had fallen almost 40 percent since the program’s  original estimate. However, while the Navy did conduct a risk analysis  for its recent Columbia class cost estimates, the confidence level of the  Navy’s estimate at Milestone B for acquisition of the entire class is 45  percent. This means that it is more likely than not that actual costs to  research, develop, and buy the submarines will exceed the Navy’s $128  billion estimate.

This situation is particularly apparent at this point with regard to costs to  design the class and build the lead submarine. Any difficulties in ongoing  technology development efforts would likely worsen the picture. At  Milestone B, the Navy’s point estimate to develop the technologies,  design the class, and build the lead Columbia was at a 43 percent  confidence level.

Experts agree that programs should be budgeted to at least the 50  percent confidence level, but budgeting to a higher level (e.g., 70 to 80  percent, or the mean) is a common practice to cover increased costs  resulting from unexpected design complexity and technology uncertainty,  among other things. Navy cost guidance recommends using the “risk  adjusted mean” for the cost for the program, which usually lies between  50 and 60 percent. If the Navy budgeted to an estimate at a higher  confidence level like the risk adjusted mean, its Milestone B point  estimates—meaning the selected estimate of cost—would be higher,  reducing the probability of overruns occurring. According to Navy cost  analysts, the program’s total acquisition cost, which the Navy estimated at  Milestone B at $128 billion (then-year dollars), would exceed $131 billion  (then-year dollars) at 50 percent confidence, which is the bottom range of  the risk adjusted mean confidence level.

Cost Growth Potential Based on the Navy’s Estimate

Even if the Navy budgeted to the 90 percent—a “worst-case” scenario  where significant programmatic challenges are realized and the  probability of cost overruns is low—confidence level, Columbia class lead  ship costs would not be dissimilar to cost outcomes on other lead ship  programs. We have observed in prior work that cost growth for recent  lead ships across the Navy’s shipbuilding portfolio is 28 percent on  average. For example, the Navy’s lead Virginia class submarines (SSN  774 and SSN 775)—the most similar class to Columbia in terms of  technology and component development as well as aspects of its design  and build plans—experienced 15 and 24 percent budget growth  respectively, with average cost growth of 28 percent for the three most  recent lead submarines (see figure 15).

The 28 percent cost growth we have observed is slightly more than the 22  percent cost increase between the Navy’s point estimate and the 90  percent confidence level, meaning that even if the Navy budgeted the  program to the 90 percent confidence level there would still be historical  shipbuilding precedence for further cost growth. In particular, if costs to  build the lead Columbia class submarine grow similar to the lead Seawolf  and Virginia class submarines, the cost to construct the submarine would  exceed the Navy’s Milestone B estimate by more than $2.5 billion. This  would represent a total approaching $12 billion (then-year dollars) versus  the current estimate of $9.2 billion for the lead submarine. Due to the  magnitude of the Columbia class program’s expected cost, any cost  growth, including for design and construction of the lead ship could  impact the availability of funds for other Navy priorities.

The Congressional Budget Office (CBO) and CAPE also analyzed  Columbia class program costs. CBO predicted higher costs than the Navy  estimate. In its 2017 assessment of the Navy’s long-term shipbuilding  plans, CBO concluded that the Navy underestimated the cost of the total  Columbia class procurement by $8 billion (2017 dollars).

CAPE estimated a lower cost, but also identified areas where reliable cost  data were unavailable. The independent cost estimate prepared by CAPE  in support of the program’s Milestone B reflects a 3 percent lower total  program life-cycle cost (2017 dollars) than the Navy estimate. In setting  the program baseline in January 2017, DOD pragmatically opted to use  the Navy’s higher estimate ($7.3 billion) instead of CAPE’s $7 billion  estimate for the average unit cost to procure a Columbia class submarine  (calendar year 2017 dollars). According to CAPE officials, this difference  in estimates is largely due to CAPE incorporating more recent Virginia  class actual cost data into its estimate than the Navy. However, CAPE  also identified that there is a lack of reliable cost data on some contractor- furnished materials and government furnished equipment (GFE) for the  Columbia class program, which limited the quality of the estimate. GFE  comprises critical areas of the Columbia class submarine, including the  strategic weapon system managed by Strategic Systems Program and  the IPS developed by Naval Reactors.

The Columbia class submarine will be a significant DOD acquisition for  the next several decades due to cost and mission importance in  guaranteeing the nation’s strategic deterrence. Failure to meet the  aggressive patrol dates required of the program could challenge the Navy  in effectively meeting strategic patrol requirements, and not delivering the  required level of performance could compromise the Navy’s plan to  operate this class through 2080.

Given the risks facing the program and the significance of potential delays  or cost growth, we believe this program warrants increased attention to  and scrutiny over what we consider to be its critical technologies  (inclusive of the program’s stated technology development efforts),  several of which remain immature. Specifically, technologies such as IPS  and the propulsor and coordinated stern demand more specific  congressional visibility to ensure they stay on track. These areas also  warrant specific assurances from the Navy that they will be delivered on  time and will perform as required. This assurance could augment the  Milestone B certifications which were predicated on a TRA that was not  representative of the technical risk facing the program. Further, such  information would help bolster confidence for Congress that the program  technologies will be matured in time to support construction, which is  especially important as the Navy pursues plans to start construction of the  lead ship early. Without putting in place a requirement for the Navy to  provide these assurances on a periodic basis, Congress will not have the  information until after the Navy has asked for another $8.7 billion in  funding for lead ship construction. It is also important for Congress to be  informed of the impact on performance requirements if technologies are  delayed or fail to mature as planned.

The Columbia class program is also facing risks from its aggressive and  concurrent schedule as a result of the continued and pressing need for it  to meet the Navy’s nuclear deterrent requirements as the legacy  submarine fleet that cannot be life extended any longer. Typically  addressing risks of such concurrency is accomplished by, among other  things, delaying milestones until more knowledge is obtained. Doing so  helps reduce concurrency and bring more stability to the design before  construction activities begin. Recognizing the mission imperatives that are  driving Columbia class’s aggressive and concurrent schedule it is unlikely  that the Navy will have the ability to slow the pace of the program in order  to reduce cost and schedule risk. Therefore, additional reporting to  decisionmakers on the status of key technologies could help ensure they  fully understand the risks of such an approach and account for such risks  when making programmatic decisions.

In our draft report we had suggested a matter for congressional  consideration related to additional Navy reporting on the Columbia class  technologies, but we have since removed it because the recently passed  National Defense Authorization Act (NDAA) for 2018 includes Navy  reporting requirements for the Columbia class program that would  achieve the intent of our matter.

We provided a draft of this product to DOD for comment. The Navy  provided technical comments earlier in the review process which we  incorporated where appropriate. In its written comments, reproduced in  appendix III, DOD’s position was that there is not a need for additional  congressional reporting on the Columbia class program because there  are new reporting requirements in the conference report accompanying  the NDAA for fiscal year 2018. We agree that the reporting requirements  in the section 231 of the NDAA for Fiscal Year 2018 meet the intent of our  matter for congressional consideration. These new reporting  requirements for the Navy became law on December 12, 2017, after we  sent the report to DOD and appropriate congressional committees. We  agree that the reporting requirements meet the intent of our matter for  congressional consideration. Accordingly, we have removed our matter  from this report.

In addition, DOD also disagreed with our characterization of technical  risks facing the Columbia class program and its TRA. Specifically, DOD  stated that the program is meeting statutory and DOD maturity standards  and met or exceeded DOD technology maturity requirements. DOD also  stated that the program’s TRA was conducted in accordance with a 2011  DOD policy memo that directed TRA’s should focus only on “technology  maturity, as opposed to engineering and integration risk.” However,  neither this policy memo nor the Columbia class TRA define what  constitutes engineering and integration risk and it is unclear what criteria  the Navy used in making these determinations. Our report acknowledges  that DOD followed statutory and DOD requirements for the two  technologies that the Navy identified as critical technologies in the  program’s TRA. However, our report also identifies several other  technologies that we believe should have also been subject to these  requirements had the Navy conducted a TRA in accordance with our  identified best practices. By applying our identified best practices, we  believe these efforts would have been considered critical technologies  and would have been subject to an evaluation of technology maturity  levels, additional reporting requirements and, potentially, identification of  additional risk mitigation efforts.

DOD also disagreed with our criteria for identifying a critical technology  and assessing maturity. DOD asserted that applying our criteria would  result in nearly every system on a submarine becoming a critical  technology. We disagree. Our criteria are consistent with DOD’s own  criteria for identifying critical technologies, and only focus on those that  are most significant to a program. Given the program’s cost and schedule  risks and operational imperatives, we believe that appropriately identifying  the critical technologies is an important step in acknowledging and  mitigating program risk. DOD also stated that achieving a TRL 7 by  milestone B would be unrealistic because of the difficulties in testing  some systems in an operational environment prior to launching the  submarine. We agree that in some cases testing at sea is not practical  and testing in a relevant environment may be sufficient to demonstrate  maturity. However, achieving a TRL 7 is not only based on the test  environment; it is also based on demonstrating a prototype near or at the  planned operational system configuration, which requires a design  resembling the final configuration. The Columbia class program has yet to  complete this type of prototype for the key systems we identified. As we  stated in the report some systems, like the propulsor, do not yet have a  final design. While we do not expect the Navy to test every critical  technology on a submarine at sea to demonstrate maturity, we would  expect testing of a prototype near or at the planned operational system  configuration prototype in a relevant environment. For example, prototype  testing of the electric drive at a land-based test facility would demonstrate  maturity—but is not planned for several years—well after the submarine’s  design and potentially construction is underway. While such concurrency  introduces cost, schedule and technical risk, we have previously reported  that programs may choose to move forward with these risks, but should  acknowledged and appropriately resource the program to address the  risks should they materialize. As we stated in the report, this is not the  case for the Columbia class program: some risks have not been properly  identified and the cost estimate does not fully account for the margin of  technical and schedule risks facing the program.

DOD also provided a table of Columbia class practices, reprinted with our  comments in appendix III.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense, the Secretary of the Navy, and  other interested parties. This report will also be available at no charge on  GAO’s website at http://www.gao.gov.

If you or your staff members have any questions regarding this report,  please contact me at (202) 512-4841 or oakleys@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made key  contributions to the report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This report examines (1) the status of key Columbia class technologies  and congressional reporting requirements on this status, (2) risks, if any,  with the Navy’s planned approach for design and construction, and (3)  whether expected funding levels for the Columbia class will be adequate  moving forward.

To assess the status of key Columbia class technologies, we reviewed  the Navy’s technology development plan and the planned technical  approach and the status of key prototyping efforts to all of the systems  that comprise the program, focusing on the technology readiness level of  the major components that are key to enabling program success and that  are key cost and schedule drivers. We also compared technology  development efforts with program requirements and with GAO’s identified  best practices for shipbuilding programs. We also evaluated the  program’s Technology Readiness Assessment, which included applying  the GAO-developed criteria documented in GAO’s Technology  Assessment Guide. GAO’s guide draws heavily from the Department of  Defense (DOD), Energy (DOE), and National Aeronautics and Space  Administration (NASA) best practices, and establishes a methodology  based on those best practices that can be used across the federal  government for evaluating technology maturity, particularly as it relates to  determining a program or project’s readiness to move past key decision  points that typically coincide with major commitments of resources. We  also interviewed relevant officials from the Navy’s Columbia class  submarine program office; the Office of the Chief of Naval Operations- Undersea Warfare; Naval Sea Systems Command Naval Nuclear  Propulsion Program; Navy Strategic Systems Program; Naval Undersea  Warfare Center Newport; Naval Surface Warfare Center Carderock  Division; Office of the Secretary of Defense (OSD) Director Operational  Test and Evaluation; OSD Acquisition, Technology, and Logistics (AT&L);  OSD Cost Analysis and Program Evaluation (CAPE); and the prime  contractor shipyard General Dynamics Electric Boat and their sub- contractor Huntington Ingalls Industries Newport News Shipbuilding. To  determine the congressional reporting requirements on this status we  reviewed relevant DOD acquisition instructions and statute.

To assess the risks, if any, with the Navy’s planned approach for design  and construction, we compared the status of design maturity with Navy  and shipyard plans to identify any delays, and compared planned design  maturity and schedule projections with those of prior U.S. submarine  efforts (the Virginia, Seawolf, and Ohio classes) to assess realism of  Columbia class estimates. We also interviewed and analyzed available  documentation from Naval Reactors (NAVSEA 08) related to nuclear  reactor and Integrated Power System status. We also interviewed  relevant officials from the Navy’s Columbia class submarine program  office; Naval Sea Systems Command Naval Nuclear Propulsion Program;  Naval Surface Warfare Center Carderock Division, and the prime  contractor shipyard General Dynamics Electric Boat and their sub- contractor Huntington Ingalls Industries Newport News Shipbuilding. We  also assessed the Navy’s acquisition strategy and the Integrated  Enterprise Plan that tracks shipyard workload across the Columbia and  Virginia class submarines and the Ford class aircraft carrier to identify  any factors related to potential schedule challenges.

To assess whether expected funding levels for the Columbia class will be  adequate moving forward, we compared program cost estimates  prepared at Milestone B to historical data on lead ships and submarine  estimates and actuals to assess the realism of these requirements. We  also analyzed program documentation to identify risk factors, if any,  related to cost projections, including the program’s Independent Cost  Estimate created by the OSD Cost Analysis and Program Evaluation, and  the Navy’s Service Cost Position and Program Life Cycle Cost Estimate.  This evaluation leverages, among other things, prior GAO work on cost  estimating and the Navy’s acquisition of lead ships.

We also interviewed relevant officials from the Navy’s Columbia class  submarine program office; the Office of the Chief of Naval Operations- Undersea Warfare; Naval Sea Systems Command Naval Nuclear  Propulsion Program; Naval Undersea Warfare Center; Naval Surface  Warfare Center Carderock Division; OSD Director Operational Test and  Evaluation; OSD AT&L; CAPE; and the prime contractor shipyard  General Dynamics Electric Boat and their sub-contractor Huntington  Ingalls Industries Newport News Shipbuilding.

We conducted this performance audit from May 2016 to December 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Department of Defense Technology Readiness Levels

Appendix II: Department of Defense  Technology Readiness Levels Description  Lowest level of technology readiness. Scientific research begins to be translated into  applied research and development (R&D). Examples might include paper studies of a  technology’s basic properties.

Invention begins. Once basic principles are observed, practical applications can be  invented. Applications are speculative and there may be no proof or detailed analysis to  support the assumptions. Examples are limited to analytic studies.

Active R&D is initiated. This includes analytical studies and laboratory studies to  physically validate the analytical predictions of separate elements of the technology.  Examples include components that are not yet integrated or representative.

Basic technological components are integrated to establish that they will work together.  This is relatively “low fidelity” compared with the eventual system. Examples include  integration of “ad hoc” hardware in the laboratory.

Fidelity of breadboard technology increases significantly. The basic technological  components are integrated with reasonably realistic supporting elements so they can be  tested in a simulated environment. Examples include “high-fidelity” laboratory  integration of components.

Representative model or prototype system, which is well beyond that of TRL 5, is tested  in a relevant environment. Represents a major step up in a technology’s demonstrated  readiness. Examples include testing a prototype in a high-fidelity laboratory  environment or in a simulated operational environment.

Prototype near or at planned operational system. Represents a major step up from TRL  6 by requiring the demonstration of an actual system prototype in an operational  environment (e.g., in an aircraft, in a vehicle, or in space.

Technology has been proven to work in its final form and under expected conditions. In  almost all cases, this TRL represents the end of the true system development.  Examples include developmental test and evaluation of the system in its intended  weapon system to determine if it meets design specifications.

Actual system proven through  successful mission operations.

Actual application of the technology in its final form and under mission conditions, such  as those encountered in operational test and evaluations. Examples include using the  system under operational conditions.

Appendix III: Comments from the Department of Defense

DOD also provided the above table of Columbia class practices. These  practices align with GAO’s identified best practices in shipbuilding—stable  requirements, design maturity at construction start, and manufacturing  readiness. However, we have several observations on the DOD’s  statements:  Stable Operational and Technical Requirements:

We have previously identified maintaining stable requirements as a  best practice; in this report we note that the Navy has provided a  stable basis for the Columbia class program by adhering to this  practice.

High Design Maturity at Construction Start:

While we give credit to the program for striving for a high level of  design maturity at construction start for the Columbia class program,  we identify in this report that we have concerns about the Navy’s  ability to stabilize design drawings while technology development  continues.

As we point out in this report, we are concerned with the maturity of  the Columbia class design due to the unknowns with key  technologies. In this table the Department identifies that the program  is leveraging proven Virginia class technology for the propulsor, which  it identifies as a TRL 9. Although this technology is indeed mature in  the context of Virginia class submarines (i.e., not new or novel), it is  nevertheless novel in the context of Columbia class submarines and  should thus be considered a CTE to be evaluated and risk managed.   As such, we dispute the Navy’s assertion that the Virginia class  propulsor is TRL 9 in the context of the Columbia class program,  since the Navy has yet to complete a design for the propulsor nor has  it tested a production representative prototype, which would achieve a  TRL 6 or 7 (depending on the test environment).

Manufacturing and Construction Readiness:

We have not conducted adequate work in this area to comment on  DOD’s statements of manufacturing and construction readiness; we  plan to address this in future work.

While the Navy has made significant progress in reducing potential  costs for the Columbia class program, we believe that the risks  identified in this report, coupled with the optimistic cost estimate and  aggressive schedule, could result in cost growth that reduces the  actual savings identified by the program.

Appendix IV: GAO Contact and Staff Acknowledgments

In addition the contact name above, the following staff members made  key contributions to this report: Diana Moldafsky, Assistant Director; C.  James Madar; Jacob Leon Beier; Brian Bothwell; Herb Bowsher; Kurt  Gurka; Stephanie Gustafson; Tim Persons; and Robin Wilson.

Getting the first Columbia class submarine out on patrol by 2031 is essential to maintain an important U.S. nuclear capability. But doing so will be challenging and expensive—over $267 billion to develop, buy and operate 12 submarines.

Several technologies critical to Columbia class performance need more development and testing; and,

Starting to design and build vessels before technologies are proven can lead to increased costs and schedule delays.

We had suggested that Congress consider requiring additional reporting on these technologies but, before this report issued, legislation passed that did so.