DOD Major Automated Information Systems: Adherence to Best Practices Is Needed to Better Manage and Oversee Business Programs

DOD's MAIS programs are intended to help the agency sustain its key operations. In April 2017, recognizing that MAIS programs met different mission needs, DOD categorized its MAIS programs into business and non-business systems.

The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on DOD's MAIS programs annually through March 2018. GAO's objectives, among others, were to (1) assess DOD's policies for managing and overseeing MAIS programs and (2) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance goals. To address these objectives, GAO compared DOD's policies for managing and overseeing all 34 MAIS programs (24 non-business programs and 10 business programs) to leading IT management practices. GAO also compared 15 selected programs' initial cost, schedule, and performance baselines to their current acquisition program estimates.

The strength of Department of Defense's (DOD) policies for managing and overseeing major automated information system (MAIS) programs varies. Specifically, the policy for managing 24 non-business MAIS programs adheres to leading information technology (IT) management practices, but the policy for managing 10 MAIS business programs does not always do so (see table).

When DOD categorized 10 of the 34 MAIS programs as MAIS business programs, it also directed these programs to adhere to DOD's business systems policy (DOD Instruction 5000.75). However, the department directed those programs to use a policy for the management and oversight of MAIS business programs that was not fully comprehensive. Until DOD updates its business systems policy to address gaps in establishing performance information such as baseline estimates on program cost and schedule goals, identifying thresholds to identify high risk, and requiring periodic reports to be provided to stakeholders at regular intervals, stakeholders will likely not have all the information they need to manage and oversee MAIS business programs.

While all 15 business and non-business MAIS programs had either increased or decreased their planned cost estimates and the majority had delays in their planned schedule estimates, the majority of the 9 programs that had performance targets met those performance goals. Specifically, the decreases and increases in cost estimates ranged from a decrease of $1.6 billion (-41 percent) to an increase of $1.5 billion (163 percent). The decreases in planned cost were largely due to scope reduction, while cost increases were due to underestimating levels of effort and contracting issues. The slippages in schedule estimates ranged from a delay of 5 years to 5 months; these delays were caused by unrealistic expectations or unplanned changes. Six of the 9 programs that had performance targets met all of them, while the other 3 met several but not all of their performance targets. The other 6 programs were in the early stages of system development and had not begun performance testing.

GAO is making three recommendations, including that DOD update its policy for managing MAIS business programs to include baseline estimates. DOD partially concurred with this recommendation, and fully concurred with the other two recommendations. GAO continues to believe that all the recommendations are warranted.

The Department of Defense (DOD) is one of the largest and most  complex organizations in the world. To meet its mission to protect the  security of our nation and to deter war, it relies heavily on the use of  information technology (IT) and information systems to support  warfighters. In this regard, according to DOD’s IT investment portfolio for  fiscal year 2017, the department spent approximately $33.4 billion for IT  investments. Of this amount, approximately $2.5 billion was spent on  major automated information system (MAIS) programs, which include  business and non-business information systems that help the department  sustain its key operations in communications and command and control,  and provide it with access to information to organize, plan, and monitor  mission operations.

A DOD IT investment that falls within one of the following categories is  designated as a MAIS program when: (1) program costs in any single  year exceed $40 million, (2) total program acquisition costs exceed $165  million, or (3) total life-cycle costs exceed $520 million. The Secretary of  Defense could also use discretion to designate a program as a MAIS  program if it did not meet these cost thresholds.

The National Defense Authorization Act for Fiscal Year 2012 includes a  provision that we select, assess, and report on DOD MAIS programs  annually through March 2018. GAO satisfied the statutory mandate by  submitting a draft of the report to congressional committees on March 29,  2018. This final version of the report is the sixth and last report in the  series of annual mandated assessments. Our specific objectives for this  review were to (1) assess DOD’s policy for the management and  oversight of MAIS programs; (2) describe the extent to which selected  MAIS programs have changed their planned cost and schedule estimates  and met performance targets; and (3) assess the extent to which selected  MAIS programs have used leading IT acquisition practices, including  requirements and risk management.

To address the first objective, we assessed DOD’s memorandums and  policies for managing MAIS programs. To evaluate DOD’s approach in  managing and overseeing MAIS programs, we identified leading IT  management practices in GAO’s Information Technology Investment  Management guide and compared DOD’s policies to those practices. We  also interviewed an acquisition official responsible for the department’s  plan to update its policies for administering how MAIS programs are to be  managed and monitored.

To address the second objective, we used DOD’s official list of 34 MAIS  programs, as of April 18, 2017, to establish a basis for selecting  programs. From this list, we identified those MAIS programs based on our  criteria that programs must be unclassified and have a first acquisition  program baseline that could be used as a reference point for evaluating  cost, schedule, and technical performance characteristics. Based on  these criteria, we selected 15 of the 34 MAIS programs for our review.

We then compared each program’s cost (in then-year dollars) and  schedule estimates established in the first acquisition baseline to the  latest total life-cycle cost and schedule estimates. In addition, to  determine whether system performance targets were met, we identified  nine MAIS programs that had conducted performance tests. We then  compared each program’s initial and most recent baseline performance  targets.

To address the third objective, we selected three programs that had not  been included in our last assessment, while seeking to ensure that we  had representation from at least one military service and at least one  defense agency. Using these criteria, we selected the Navy’s  Consolidated Afloat Networks and Enterprise Services; Defense Logistics  Agency’s Defense Agencies Initiative, Increment 2; and Defense Health  Agency’s Department of Defense Healthcare Management System  Modernization.

We then identified requirements and risk management practices in the  Software Engineering Institute’s Capability Maturity Model® Integration  for Acquisition (CMMI®-ACQ) and assessed each of the three programs  against these criteria. Specifically, regarding risk management practices,  we analyzed each program’s key documents, such as the risk register  logs, risk management plans, and other artifacts, and compared them to  the leading practices. Regarding requirements management, we  compared requirements documents, such as the requirements  management plan, traceability matrix, and procedural tools to the leading  practices. We also conducted follow-up interviews with project officials  regarding the management practices of each program.

To assess the reliability of the data we used to support the findings in this  report, we corroborated program office responses with relevant program  documentation and interviews with agency officials. We determined that  the data were sufficiently reliable for our reporting purposes. Since we  selected a nonprobability sample of MAIS programs, the results of our  analysis are not generalizable to all MAIS programs. See appendix I for a  more detailed discussion of our objectives, scope, and methodology.

We conducted this performance audit from April 2017 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

DOD’s organizational structure includes the Office of the Secretary of  Defense, the Joint Chiefs of Staff, the military departments, numerous  defense agencies and field activities, and various unified combatant  commands that contribute to the oversight of DOD’s acquisition  programs. Prior to February 2018, the former Under Secretary of Defense  for Acquisition, Technology, and Logistics also served as the principal  acquisition official of the department and was the acquisition advisor to  the Secretary of Defense.

The former Under Secretary also served as the Defense Acquisition  Executive and was the official responsible for supervising the acquisition  of MAIS programs. The former Under Secretary’s authority included  directing the military services and defense agencies on acquisition  matters and making milestone decisions for MAIS and other programs.  This official also had policy and procedural authority for the defense acquisition system, which establishes the steps that DOD programs  generally take to plan, design, acquire, deploy, operate, and maintain the  department’s information systems.

However, as of February 2018, the department changed the way it  conducts business and operations with the statutory elimination of the  Office of the Under Secretary of Defense for Acquisition, Technology, and  Logistics. The statute contains a provision that required DOD to establish  a new Office of the Under Secretary of Defense for Research and  Engineering to be responsible for driving innovation and acceleration of  the advancement of warfighting capability. In addition, a new Office of the  Under Secretary of Defense for Acquisition and Sustainment was created  to focus on delivering proven technology more quickly. The creation of  these offices within the department is intended to shift the principal focus  of the Office of the Secretary of Defense from a role of program oversight  to that of directing major department investments. Further, the statutory  creation of a Chief Management Officer to replace the former Deputy  Chief Management Officer is intended to improve the quality and  productivity of the department’s business operations.

DOD’s Acquisition Guidance and Framework for Managing MAIS Acquisitions

In January 2015, DOD updated its guidelines that outline the framework  for MAIS programs. This framework consists of six models for acquiring  and deploying a program, including two hybrid models that each describe  how a program may be structured based on the type of product being  acquired (e.g., software-intensive programs and hardware-intensive  programs). A generic acquisition model that shows all of the program life- cycle phases and key decision points is depicted in figure 1 and  described below.

Materiel solution analysis: Refine the initial system solution (concept)  and create a strategy for acquiring the solution. A decision—referred to as  Milestone A—is made at the end of this phase to authorize entry into the  technology maturation and risk reduction phase.

Technology maturation and risk: Determine the preferred technology  solution and validate that it is affordable, satisfies program requirements,  and has acceptable technical risk. A decision—referred to as Milestone  B—is made at the end of this phase to authorize entry of the program into  the engineering and manufacturing development phase and award  development contracts. An acquisition program baseline is first  established at the Milestone B decision point. A program’s first acquisition  program baseline contains the original life-cycle cost estimate (which  includes acquisition and operations and maintenance costs), the schedule  estimate (which consists of major milestones and decision points), and  performance parameters that were approved for that program by the  milestone decision authority. The first baseline is established after the  program has refined user requirements and identified the most  appropriate technology solution that demonstrates that it can meet users’  needs.

Engineering and manufacturing development: Develop a system and  demonstrate through testing that the system meets all program  requirements. A decision—referred to as Milestone C—is made during  this phase to authorize entry of the system into the production and  deployment phase or into limited deployment in support of operational  testing.

Production and deployment: Achieve an operational capability that  meets program requirements, as verified through independent operational  tests and evaluation, and implement the system at all applicable  locations.

Operations and support: Operationally sustain the system in the most  cost-effective manner over its life cycle.

Leading Practices for Managing IT Investments and Acquisition Programs

We have developed and identified leading practices for governing IT  investments to help guide organizations to better manage and oversee  their projects. GAO’s Information Technology Investment Management  guide states that good performance data and stakeholder oversight are  elements that can lead to positive outcomes, such as helping to ensure a  project is keeping to its initial cost, schedule, and performance goals. The  guide also states that projects should be reviewed at regular intervals to  monitor performance so that stakeholders can be aware of and review  any differences between actual outcomes and goals.

In addition, we and other entities, such as the Software Engineering  Institute at Carnegie Mellon University, have identified leading practices  to help guide organizations to effectively plan and manage their  acquisitions of major IT systems. Our prior reviews have shown that  proper implementation of such practices can significantly increase the  likelihood of delivering promised system capabilities on time and within  budget. These practices include, but are not limited to:  Requirements management: Requirements establish what the system is  to do, how well it is to do it, and how it is to interact with other systems.  Appropriate requirements management involves eliciting and developing  customer and stakeholder requirements, and analyzing them to ensure  that they will meet users’ needs and expectations. It also consists of  validating requirements as the system is being developed to ensure that  the final systems to be deployed will perform as intended in an  operational environment.

Risk management: Risk management is a process for anticipating  problems and developing plans to take appropriate steps to mitigate risks  and minimize their impact on program commitments. It involves  identifying and documenting risks, categorizing them based on their  estimated impact, prioritizing them, developing risk mitigation strategies,  and tracking progress in executing the strategies.

DOD’s Policies for Managing MAIS Programs Do Not Always Adhere to Leading IT Management Practices

According to GAO’s Information Technology Investment Management  guide, leading practices for managing IT projects include:  instituting the investment board, which is the process for creating and  defining the membership, guiding policies, operations, roles,  responsibilities, and authorities within the organization;  identifying decision authorities for making important acquisition  decisions; providing oversight whereby the organization monitors each project on  its performance progress (e.g., establishing and tracking baseline  estimates on cost and schedule goals, and thresholds to identify high  risk on cost and schedule); and  capturing and providing performance information about a particular  investment (project) to decision makers at regular intervals (e.g.,  quarterly and annually).

To align MAIS programs with the functions they perform, DOD recently  made changes in how it characterizes its MAIS programs and, as a result,  different programs must follow different management policies.  Specifically, in April 2017, DOD identified 10 of 34 total MAIS programs  as business programs and the Director, Acquisition Resources and  Analysis announced that these programs would adhere to DOD’s  Instruction 5000.75 policy for management and oversight. Further, in  November 2017, the former Under Secretary of Defense for Acquisition,  Technology, and Logistics announced that non-business MAIS  programs would adhere to DOD’s Instruction 5000.02 policy for  management and oversight.

However, the policies used for MAIS business programs are not  consistent in their adherence to leading IT management practices. For  example, while the policy for non-business MAIS programs is consistent  in its adherence to all four of the leading IT management practices, the  policy for MAIS business programs is consistent in its adherence to only  two of the four practices. Table 1 shows our analysis of DOD’s policies for  non-business MAIS programs and MAIS business program and their  adherence to the leading IT management practices.

As shown in the table, DOD’s policy for non-business MAIS programs  adheres to all four leading IT management practices. For example, the  policy requires non-business MAIS programs to report the status of each  program’s cost, schedule, and technical performance information  quarterly and annually. The policy also designates specific decision  makers who are responsible for monitoring and overseeing the progress  of non-business system MAIS programs. Further, the policy requires each  program to establish and report their initial baseline estimates and current  estimates on cost and schedule so their performance can be tracked and  monitored. In addition, to identify when programs may be at risk of  significant cost or schedule increases, the policy requires programs to  predetermine cost and schedule threshold estimates as an early warning  indicator on when programs reach the point where they are at increased  risk.

In contrast, DOD’s policy for MAIS business programs policy only  adheres to two of the four practices. Specifically, the policy adheres to the  practice of instituting an investment board with processes for creating and  defining the membership, policies, operations, roles, responsibilities, and  authorities within the organization. In addition, the policy identifies  decision authorities for making important executive-level acquisition  decisions. However, the policy does not specify the establishment of  initial and current baseline estimates on cost and schedule, and does not  specify the reporting of threshold cost and schedule estimates to identify  the point when programs may be at high risk. In addition, the policy does  not adhere to leading practices requiring the periodic (quarterly and  annual) reporting of performance information to stakeholders.

To help address the need for improved guidance, the former Under  Secretary of Defense for Acquisition, Technology and Logistics  established a cross-functional team that is to examine the future of non- business MAIS programs and MAIS business programs from a policy,  organization, management, and reporting perspective. The team was  expected to provide its recommendations to the Under Secretary of  Defense for Acquisition and Sustainment by March 15, 2018. However,  because no final decisions had been made by the Under Secretary as of  that date, it is unclear what specific actions the department will take  regarding its policy recommendations, among other recommendations, to  improve the management of non-business MAIS programs and MAIS  business programs.

Until DOD updates its policy for MAIS business programs to adhere to  leading practices on the establishment of baseline estimates on cost and  schedule to include threshold estimates on cost and schedule to identify  when programs may be at high risk, stakeholders may not have the  information they need to manage and oversee MAIS business programs.  Further, unless the department updates its policy for MAIS business  programs to adhere to the leading practice for periodically (quarterly and  annually) reporting essential performance information, stakeholders may  not have the information they need to make informed decisions for  managing and overseeing MAIS business programs.

All Selected MAIS Programs Had Changes in Cost and Schedule Estimates, and Most Programs Had Met Performance Targets

All of the 15 selected MAIS programs had either increased or decreased  their planned cost estimates, and 10 of them had delays in their planned  schedule estimates when comparing the first acquisition program  baseline to the most recent acquisition program baseline estimates. The  changes in the cost estimates ranged from a decrease of $1.6 billion (-41  percent) to an increase of $1.5 billion (163 percent), and slippages in the  schedule estimates ranged from a delay of 5 years to a delay of 5  months.

Further, 9 of the 15 selected programs had conducted testing in which we  could report on the number of performance targets met for each  program. Of those 9, 6 programs reported that they had met all of their  performance targets. The remaining 3 programs reported that they met  several but not all performance targets. The following table shows the  extent of changes in planned cost and schedule estimates for the  selected MAIS programs since the first baseline estimate, as well as the  number of performance targets met.

All Selected MAIS Programs Had Either Increases or Decreases in Their Planned Cost

All 15 selected MAIS programs had experienced increases or decreases  in their planned cost estimates when comparing the initial, or first,  baseline estimate to the current estimate. Specifically, 10 programs had  decreases in their cost estimates that ranged from $1.2 million (less than  -1 percent) for the Defense Agencies Initiative, Increment 2 program to  $1.6 billion (-41 percent) for the Air Force’s Base Information Transport  Infrastructure Wired program.

Program officials reported that reductions in planned cost estimates were  due to changes in program scope. Specifically, the reasons for reduction  in cost include:

Program scope changes. Officials for the Air Force’s Joint Space  Operations Center Mission System Increment 2 program reported that  its 12 percent cost decrease was due to a reduction in its estimate for  operations and support that was changed from 20 years to 10 years.  Officials for the Defense Information Systems Agency’s Global  Combat Support System–Joint Increment 8 program reported that its  20 percent cost decrease was due to a reduction in the program’s  scope for the number of development hours required to meet the  logistics and operational needs. In addition, officials for the Defense  Information Systems Agency’s Teleport Generation 3 program  reported that its 22 percent cost decrease was due to a revised scope  in terms of what is needed at the Milestone C decision point for low  rate production.

Design reconfiguration. Officials for the Air Force’s Base Information  Transport Infrastructure Wired program reported that its 41 percent  cost decrease was due to a reduction in the program’s scope when  they changed from a base network system to a critical core  configuration.

In addition, 5 of the programs had experienced cost increases. These  cost increases ranged from $2.9 million (less than 1 percent) for the  Army’s Logistics Modernization Program Increment 2 to $1.5 billion (163  percent) for the Army’s Tactical Mission Command program. Program  officials reported a variety of reasons for the increases in planned cost  estimates. These reasons included the following:

Underestimating schedule. Officials for the Air Force’s Defense  Enterprise Accounting and Management System Increment 1 program  attributed its 60 percent cost increase to underestimating the level of  effort that was needed to develop the system within the estimated  schedule. For example, the program did not account for software  upgrades and, when this effort was added to the schedule to account  for the work, the cost increased.

Contractor issues. Officials from the National Security Agency’s Key  Management Infrastructure Increment 2 program attributed its cost  increase of 14 percent to schedule delays caused by the contractor  and, as a result, increased funding at the Milestone C decision point.

Underestimating development and test efforts. Officials from the  Army’s Tactical Mission Command program attributed the cost  increase of 163 percent to higher than expected costs to conduct  research and developmental tests.

Ten Selected MAIS Programs Had Delays in Their Planned Schedule Estimates

Ten of the 15 selected MAIS programs had experienced changes in their  planned schedule estimates, and 5 programs had no changes to their  schedule estimates. The changes consisted of schedule slippages that  ranged from 5 months for both the Army’s Logistics Modernization  Program Increment 2 and the Defense Health Agency’s Department of  Defense Healthcare Management System program, to 5 years for the  Defense Enterprise Accounting and Management System Increment 1  program.

Program officials reported that delays in the planned schedule estimates  were due to unplanned budget reductions or unrealistic expectations  regarding project milestones. Specifically, the reasons for these schedule  slippages included:

Aggressive schedule, funding reduction, and contract issues.  Officials for the Air Force’s Joint Space Operations Center Mission  System Increment 2 program attributed its schedule slippage of 2  years and 11 months to funding reductions of $18.9 million in fiscal  years 2013 and 2014. In addition, the officials noted that an  aggressive schedule for a Milestone B decision, contracting issues in  the earlier acquisition phase, and longer than expected time to obtain  personnel had contributed to the slippage.

Longer than expected time to reach deployment. Officials for the  Air Force’s Defense Enterprise Accounting and Management System  Increment 1 program reported that its schedule slippage of 5 years  occurred because of a change in the approach to deliver the system  in multiple increments, thereby increasing the amount of time it would  take to reach the deployment decision milestone. Also, officials for the  Defense Information Systems Agency’s Teleport Generation 3  program reported a 3-year and 2-month slip. This schedule delay was  due to the program’s inability to develop the mobile user and system  interface capability by the estimated deployment milestone. Further,  program officials for the Navy’s Consolidated Afloat Networks and  Enterprise Services program attributed its schedule slip of 2 years and  6 months to a longer than expected maintenance period for the test  platform and to a lengthy budget approval process, resulting in a  slippage in the deployment date.

Unplanned procurement fund reduction. Officials for the Army’s  Global Combat Support System-Army program reported that its  schedule delay of 11 months was due, in part, to a $16 million dollar  decrease to the fiscal year 2016 budget. This unplanned reduction in  procurement fund affected their ability to field the system as originally  planned.

Contractor staffing issues. Officials for the National Security  Agency’s Key Management Infrastructure Increment 2 program  reported significant schedule delays due to the contractor’s inability to  staff the program with software developers that had the required  security clearances. As a result, a critical change was reported in  January 2012 that led to a new independent cost estimate, which  extended program development by 10 months. The new estimate  included additional time to improve the governance structure, such as  increasing discipline across the oversight process, adding more  stakeholder interaction, and improving the use of metrics.

Six of Nine MAIS Programs Had Met All Performance Targets

Among other information, DOD uses key performance parameters as a  metric to report on programs’ progress toward meeting system  performance targets. This information includes a description of the  performance characteristics, the objective and threshold value for each  target and, importantly, whether the target has been met in demonstrating  performance.

Of the nine programs we evaluated, six programs reported that they met  all of their performance targets. For example, the Navy’s Common  Aviation Command and Control System, Increment 1 program reported in  May 2017 that both of its technical performance targets had been met.  According to the program, these targets were related to the readiness of  the system to fully support all operational activities and satisfy all  technical requirements for military operations and the fusion of all kinds of  data onto any workstation.

In another example, the Army’s Logistics Modernization Program  Increment 2 program reported in June 2017 that all seven of its  performance targets had been met. According to the program, these  targets were related to the system’s ability to support military operations,  exchange information in the network, provide system and information  assurance in a disaster recovery scenario, and be operationally available.

Further, three programs reported that they met several, but not all, of their  performance targets. For example, the Navy’s Consolidated Afloat  Networks and Enterprise Services program reported that it met eight of  nine performance targets. According to program officials, the remaining  target (i.e., network shall fully support joint critical operational activities)  had not been met because the program lacked an operational platform  that was required to demonstrate its performance.

The Defense Information Systems Agency’s Teleport Generation 3  program reported that it met 8 of 12 performance targets. According to  programs officials, the remaining 4 targets (i.e., coverage to allow  warfighter communications, capacity to provide 100 percent of the  required services, and interoperability with military and commercial  frequencies and wave forms) had not been met because the program  needed to field multiple systems and perform solution testing, which they  expect to be completed in fiscal year 2018. Further, the Air Force’s  Defense Enterprise Accounting and Management System Increment 1  program officials reported that it met 3 of 4 targets (i.e., compliance with  requirements, network ready, and sustainment to ensure materiel  availability). The officials reported that the program did not meet the  remaining target because it was waiting for an evaluation of cyber test  results before proceeding.

Selected Programs Fully Implemented Most, but Not All, of the Leading Practices for Managing Requirements and Risk

According to the Software Engineering Institute’s Capability Maturity  Model Integration® for Acquisition (CMMI®-ACQ), an appropriate  requirements management process involves establishing an agreed-upon  set of requirements, ensuring traceability between requirements and work  products, and managing any changes to the requirements in collaboration  with stakeholders. Likewise, an effective risk management process  identifies potential problems before they occur, so that risk-handling  activities may be planned and invoked, as needed, across the life of the  project in order to mitigate the potential for adverse impacts.

Leading requirements management practices help organizations to better  manage the design, development, and delivery of systems within  established cost and schedule time frames. These practices include developing an understanding with the requirements providers of the  meaning of the requirements, obtaining commitment to requirements from project participants, managing changes to requirements as they evolve during the project, maintaining bidirectional traceability among requirements and work, ensuring that project plans and work products remain aligned with  requirements.

An effective risk management process includes the following leading  practices determining risk sources and categories; defining parameters used to analyze and categorize risks and to  control the risk management effort; establishing and maintaining the strategy to be used for risk  identifying and documenting risks; evaluating and categorizing each identified risk using defined risk  categories and parameters, and determining its relative priority; developing a risk mitigation plan in accordance with the risk monitoring the status of each risk periodically and implementing the  risk mitigation plan as appropriate.

The three selected MAIS programs that we evaluated had fully  implemented most, but not all, of the five leading practices for managing  requirements and the seven leading practices for managing risks.  Specifically, two of three programs implemented all of the requirements  management practices, while one program implemented most, but not all,  of the practices.

Further, one of three programs implemented all of the risk management  practices, while two programs implemented most, but not all of the  practices. Table 3 shows the extent to which practices were implemented  by the three selected programs.

Two Programs Fully and One Program Partially Implemented Leading Practices for Managing Requirements

Two of the three programs had fully implemented the requirements  management practices. The other program had partially implemented two  practices and fully implemented three practices.

Navy — Navy Consolidated Afloat Networks and Enterprise Services  The Navy had fully implemented the five requirements management  practices for the Consolidated Afloat Networks and Enterprise Services  program. For example, the program developed an understanding with  requirements providers of the meaning of the requirements. Specifically,  there was a plan for documenting, managing, and controlling changes to  requirements throughout the system lifecycle. This plan served as the  primary guidance for integrating the management of all specified and  derived requirements for the Consolidated Afloat Networks and Enterprise  Services system program.

In addition, the program had established criteria for determining  requirements providers. Specifically, roles and responsibilities for  requirements management had been identified. Further, the program  managed changes to requirements as they evolved during the project.  For example, the program provided evidence that it maintains a  requirements change history, including the rationale for changes.

Defense Logistics Agency — Defense Agencies Initiative, Increment  2  The Defense Logistics Agency had fully implemented the five  requirements management practices for the Defense Agencies Initiative,  Increment 2. For example, the program had established objective criteria  for the evaluation and acceptance of requirements. Specifically, there was  a process in place to develop and finalize deliverables in support of the  business requirements identified by the stakeholders, ensure that  requirements management activities were performed in a timely manner  throughout the life of the project, and review and approve requirements  deliverables. Further, throughout the process, the requirements manager  tracked requirements changes and maintained traceability of end user  needs to the system performance specification.

The Defense Health Agency had fully implemented three and partially  implemented two of the five requirements management practices for the  Defense Healthcare Management System Modernization program. For  example, the program had established objective criteria for the evaluation  and acceptance of requirements. Specifically, any new or updated  requirements were presented to a Configuration Steering Board for  review and approval prior to any changes being made. Further,  throughout the process, the requirements manager tracked requirements  changes and maintained traceability to ensure they were documented.

The program has not developed an understanding with the requirements  providers on the specific meaning of the requirements. For example,  although the program had developed a requirements management plan  which provided guidance in this area, according to program officials, the  plan was not signed and approved based on the recent shift of the  program from a non-business MAIS program to a MAIS business program  operating under DOD Instruction 5000.75. Program officials stated that  the requirements management plan is not expected to be complete until  final guidance is provided by the Office of the Secretary.

Regardless of this recent shift, the program should have already had an  approved requirements management plan in place since program  initiation. In the absence of an approved plan, the program lacks  assurance that it can effectively communicate and manage requirements  practices.

Further, the program had not demonstrated that it identified any changes  that should be made to plans and work products resulting from changes  to the requirements baseline. Programs officials stated that efforts to  review modifications to the plan due to requirements changes had not  been conducted, but they expected the review and approval to be done at  some future date. However, they could not provide a specific time frame.  According to CMMI®-ACQ, until project plans and work products are  updated to coincide with changes in requirements, the program will not be  able to effectively identify inconsistencies between requirement changes  and project plans and work products, and initiate corrective actions to  resolve them.

One Program Fully and Two Programs Partially Implemented Applicable Risk Management Leading Practices

One program had fully implemented the risk management practices, while  two had fully implemented all but one practice.

Navy — Navy Consolidated Afloat Networks and Enterprise Services  The Navy had fully implemented six and partially implemented one risk  management practice for the Consolidated Afloat Networks and  Enterprise Services system program. For example, the program’s risks  defined consistent criteria for evaluating and quantifying risk likelihood  and severity levels. Specifically, the program provided a risk exposure  (e.g., a risk source used to examine and oversee changes that impact the  project), which is the value that is given to a risk event, a product, or the  overall program based on the analysis of the probability and  consequences of the event. Further, the program’s Risk Management  Guide outlined risk performance, cost, and schedule criteria. In addition,  the program demonstrated that it included the cost and benefits of  implementing risk mitigation plans. Specifically, a risk’s description  provided the cost impacts associated with the risk, which in turn provided  evidence that cost and benefits were considered during risk evaluation.

However, the Navy partially implemented one practice. Specifically,  although the program provided its failover/recovery plan that is intended  to return the program to a state of readiness after a failure, the plan did  not explicitly identify environmental elements. A program official stated  that environmental factors, such as risks that could negatively affect their  work, is understood, but these factors had not been documented in the  plan. Further, the official stated that the program should update the plan  accordingly, but did not provide a time frame to complete this effort. Until  all potential issues, hazards, threats, and vulnerabilities that could  negatively affect work efforts have been identified in the plan, successful  risk management cannot be ensured.

Defense Logistics Agency—Defense Agencies Initiative, Increment 2  The Defense Logistics Agency had fully implemented all seven risk  management practices for the Defense Agencies Initiative, Increment 2.  For example, the program identified program risks, including risk sources,  categories, and stakeholders. In addition, Defense Agencies Initiative,  Increment 2 risks followed consistent criteria for evaluating and  quantifying risk likelihood and severity levels. Specifically, risk level was  based on a combination of factors to include both likelihood and  consequence. In all instances, consensus on the risk levels was required  between the risk owner and the customer counterpart. Further, the  program’s contingency plan provided guidance when outages fell into one  of three disaster categories including natural disasters, man-made  disasters, and technological disasters.

Defense Health Agency—Defense Healthcare Management System  Modernization  The Defense Health Agency had fully implemented six and partially  implemented one of the seven risk management practices for the  Defense Healthcare Management System Modernization program. For  example, the program’s risks followed consistent criteria for evaluating  and quantifying risk likelihood and severity levels. Specifically, the  program’s Risk and Issue Management Plan described how to assess the  impact level in each risk area (performance, project and program  schedules, and cost). Further, the program prioritized risks for mitigation.  For example, risks were categorized and charted as low, medium, or  high, and grouped accordingly in the program’s risk register. Further, the  program’s Disaster Recovery Plan provides processes that allowed rapid  support recovery for critical operations during a disaster, including  environmental disasters such as tornadoes.

Regarding the partially implemented practice, the program provided an  example of a risk mitigation plan. However, the program indicated that  costs and benefits were not quantified within the program-level risk  mitigation plans. According to CMMI®-ACQ, risk mitigation activities  should be examined for benefits they provide versus resources they will  expend. Just like any other design activity, alternative plans may need to  be developed and costs and benefits of each alternative assessed.

However, the program does not require that costs and benefits be  included as part of its risk mitigation planning efforts. As a result, the  information for making an informed decision on cost and benefits of risk  mitigation solutions is limited. Program officials did not indicate whether  they have plans to implement this practice, and did not provide an  explanation as to why they are unable to provide this information. Until  the program quantifies costs and benefits, it will not be able to effectively  select the most appropriate risk mitigation plan to address each risk.

While DOD’s policy for non-business MAIS programs adheres to all four  leading IT management practices, the department’s policy for MAIS  business programs does not adhere to two leading practices on  establishing initial and current baseline estimates on cost and schedule  and predetermining threshold estimates, as well as reporting periodically  on performance information to stakeholders. Until DOD adheres to these  practices in its policies that govern MAIS business programs, it cannot  ensure that stakeholders will have the information they need to manage  and oversee their investments.

Following leading IT acquisition practices on requirements and risk  management is essential to help programs effectively plan and direct their  development and acquisition efforts. All of the leading IT acquisition  practices for requirements and risk management had been fully or  partially implemented by three programs that we reviewed. However, the  Defense Health Agency’s Defense Healthcare Management System  Modernization has not finalized its requirements management plan nor  has it identified changes that should be made to plans and work products  resulting from changes to the requirements baseline. Until the program  addresses these practices, it will lack a comprehensive plan for managing  its requirements and it may not be able to effectively identify  inconsistencies and initiate corrective actions.

Further, the Navy Consolidated Afloat Networks and Enterprise Services  program did not fully identify and document risks that could negatively  affect work efforts. In addition, the Defense Health Agency’s Defense  Healthcare Management System Modernization did not quantify costs  and benefits of risk mitigation within its program-level risk mitigation  plans. As a result, successful risk management for avoiding, reducing,  and controlling the probability of risk occurrence cannot be ensured.

We are making the following three recommendations to the Secretary of  Defense to direct:

The Under Secretary of Defense for Acquisition and Sustainment to  update the policy or guidance for MAIS business programs.  Specifically, the update should include the following elements: establishment of initial and current baseline cost and schedule predetermined threshold cost and schedule estimates to identify  the point when programs may be at high risk, and quarterly and annual reports on the performance of programs to  stakeholders. (Recommendation 1)

The Director of the Defense Health Agency to direct the program  manager for the Defense Healthcare Management System  Modernization program to:  finalize and approve its requirements management plan,  identify and document changes that should be made to plans and  work products resulting from changes to the requirements  baseline, and quantify costs and benefits of risk mitigation within its program- level risk mitigation plans. (Recommendation 2)

The Secretary of the Navy to direct the program manager for the Navy  Consolidated Afloat Networks and Enterprise Services program to:  identify and document, in the failover/recovery plan, all potential  external environmental issues, such as hazards, threats, and  vulnerabilities that could negatively affect work efforts.  (Recommendation 3)

DOD provided written comments on a draft of this report, which are  reproduced in appendix II. In its comments, the department partially  concurred with our first recommendation and concurred with the two other  recommendations.

DOD partially concurred with the first recommendation on updating the  policy or guidance for MAIS business programs. Specifically, the Under  Secretary of Defense for Acquisition and Sustainment stated that  regarding establishing baselines, the DOD Instruction 5000.75 requires  establishment of cost, schedule, and performance parameters for each  release before development or delivery. The 5000.75 also requires  consideration of program progress against baselined cost, schedule, and  performance as a criterion at the limited deployment and full deployment  decision points. A baseline requirement thus exists in DOD Instruction  5000.75 but it is not described as an acquisition program baseline, which  may be familiar to readers of DOD Instruction 5000.02. However, the  Under Secretary added that the Army’s implementation guidance includes  guidance that states each increment must have an acquisition program  baseline with its own set of threshold and objective values set by the  user.

While we agree that the existing policy requires such parameters to be  captured and included in the department’s decision making process, we  found the policy to be vague in its discussion of these parameters and to  not clearly define what a baseline is, or which baselines are to be used or  reported for comparison purposes. For example, the policy does not  make a distinction between the initial acquisition program baseline,  current baseline, and baseline deviations. Yet, such information is  important because it provides a basis for decision makers to identify the  extent to which a program may have deviated from its initial cost,  schedule, or technical performance baseline. By making these  distinctions in the policy, the department’s policy for its MAIS business  programs will be more consistent with its other policy for non-business  MAIS programs with regard to the way an acquisition program baseline is  defined and the elements that should be captured and reported to its  decision makers. In turn, the program managers who prepare these  reports and the decision makers who rely on them will have information  that is consistently and succinctly prepared for making credible decisions.

Regarding adding provisions in its policy for the establishment of  predetermined thresholds, the Under Secretary stated that the 5000.75  states that the milestone decision authority is responsible for delivery  within cost, schedule, and performance parameters, and the milestone  decision authority is to do this by establishing oversight controls for  programs, including procedures to report and address variances. The  Under Secretary added that the 5000.75 does not suggest the practice of  establishing a predetermined threshold for the variance, and DOD will  consider the addition of this feature to the 5000.75 update.

Finally, regarding providing periodic annual and quarterly reports to the  department’s leadership, the Under Secretary stated that such a periodic  report would add value only if there had been no recent communication of  program status from the program office to the leadership or stakeholders  communities. While such communication is expected to occur frequently,  its regularity is not specified in current policy or guidance. The Under  Secretary stated that DOD will consider adding a provision for a report to  leadership and functional stakeholder if such communication has not  occurred within the past 3 or 4 months.

DOD concurred with the second and third recommendations related to the  department’s implementation of selected IT management practices.  Regarding the second recommendation, the Under Secretary of Defense  for Acquisition and Sustainment agreed to direct the Defense Healthcare  Management System Modernization program manager to update and  approve the requirements management plan, identify and document  changes to the requirements baseline, and quantify the costs and benefits  in the risk mitigation plans. Further, regarding the third recommendation,  the Secretary of the Navy agreed to direct the program manager to  identify and document all potential external environmental issues that  could negatively affect work efforts for the Navy’s Consolidated Afloat  Networks and Enterprise Services program. By taking these steps, these  programs should be better positioned to effectively identify  inconsistencies in managing changes to their requirements, and be more  responsive to the potential for environmental issues.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Army, Navy,  and Air Force; the Under Secretary of Defense for Acquisition and  Sustainment; the Director of the Defense Health Agency; and other  interested parties. This report also is available at no charge on the GAO  website at http://www.gao.gov.

Should you or your staffs have any questions on information discussed in  this report, please contact me at (202) 512-4456 or harriscc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The National Defense Authorization Act for Fiscal Year 2012 mandated  that we select, assess, and report on selected major automated  information systems (MAIS) programs annually through March 2018.  GAO satisfied the statutory mandate by submitting a draft of this report to  the congressional committees on March 29, 2018. This final version of the  report is the sixth and last report in the series of annual mandated  assessments.

Our objectives were to: (1) assess the Department of Defense’s (DOD)  policy for the management and oversight of MAIS programs; (2) describe  the extent to which selected MAIS programs have changed their planned  cost and schedule estimates and met performance targets; and (3)  assess the extent to which selected MAIS programs have used leading  information technology (IT) acquisition practices, including requirements  and risk management.

To address the first objective, we identified four leading IT management  practices in GAO’s Information Technology Investment Management  guide and compared DOD’s policy adherence to those practices. These  leading practices are:  instituting the investment board, which is the process for creating and  defining the membership, guiding policies, operations, roles,  responsibilities, and authorities within the organization;  identifying decision authorities for making important acquisition  decisions; providing oversight whereby the organization monitors each project on  its performance progress (e.g., establishing and tracking baseline  estimates on cost and schedule goals, and thresholds to identify high  risk on cost and schedule); and  capturing and providing performance information about a particular  investment (project) to decision makers at regular intervals (e.g.,  quarterly and annually).

We then compared DOD’s policies used to manage and oversee the  department’s non-business MAIS programs and MAIS business programs against these leading IT management practices. The department’s policy  documents for managing and overseeing non-business MAIS programs  and MAIS business programs include the:

Memorandum by the Under Secretary of Defense for Acquisition,  Technology, and Logistics, dated November 17, 2017, regarding the  regulatory response to the repeal of title 10, United States Code,  Chapter 144A, Major Automated Information System Programs.

Memorandum by the Under Secretary of Defense for Acquisition,  Technology, and Logistics, dated April 24, 2017, regarding the  transition of programs to business system categories.

DOD Instruction 5000.75, Business Systems Requirements and  Acquisition, effective February 2, 2017.

DOD Instruction 5000.02, Operation of the Defense Acquisition  System, effective February 2, 2017.

We also interviewed an official from the former Office of the Under  Secretary of Defense for Acquisition, Technology, and Logistics, who was  responsible for the development of plans and policy for the administration  regarding the management and monitoring of non-business MAIS  programs and MAIS business programs  To address the second objective, we used DOD’s official list of 34  business and non-business MAIS programs, as of April 18, 2017, to  establish a basis for selecting programs. Of the 34 programs, we selected  the 15 business and non-business MAIS programs that met our criteria:  programs must be unclassified and have an initial acquisition program  baseline that could be used as a reference point for evaluating cost,  schedule, and technical performance characteristics.

We then collected and analyzed key documents, reports, and artifacts for  each program and summarized the information on estimated cost,  schedule, and technical performance goals, including their latest program  status in meeting those estimated goals. Next, we analyzed and  compared each selected program’s first acquisition program baseline cost  estimate to the latest estimate to determine the extent to which planned  program costs had changed. Specifically, we used the total life-cycle cost  estimate and analyzed and compared them to the latest estimate to  determine the extent to which planned program costs had changed.  Similarly, to determine the extent to which these programs changed their  planned schedule estimates, we compared each program’s first  acquisition program baseline schedule to the latest schedule.

To determine whether the selected programs met their performance  targets, we analyzed each program’s self-identified system performance  targets and compared them against actual system performance metrics  and latest test reports. We also reviewed additional information on each  program’s cost, schedule, and performance, including program  documentation, such as DOD’s MAIS annual and quarterly reports,  acquisition program baselines, system test reports, and our prior reports.  We then aggregated and summarized the results of these analyses  across the programs.

To address the third objective, we started with the list of the 15 programs  from the second objective as a basis for selecting three MAIS programs  as case studies. We used a combination of the following criteria to select  the MAIS programs to review.

Programs used in a most recent MAIS review were eliminated from  consideration.

The program was not designated as classified.

Based on these criteria, we chose the following systems:

Navy Consolidated Afloat Networks and Enterprise Services;

Defense Logistics Agency’s Defense Agencies Initiative, Increment 2;

Defense Health Agency’s Defense Healthcare Management System  Modernization.

We then analyzed each selected program’s IT acquisition documentation  and compared it to key requirements and risk management and leading practices—including Software Engineering Institute’s Capability Maturity  Model® Integration for Acquisition (CMMI- ACQ) practices—to determine  the extent to which the programs were implementing these practices. In  particular, the requirements management practices we reviewed were: develop an understanding with the requirements providers on the  meaning of the requirements, obtain commitment to requirements from project participants, manage changes to requirements as they evolve during the project, maintain bidirectional traceability among requirements and work, and ensure that project plans and work products remain aligned with  requirements.

Specifically, we analyzed program requirements documentation, including  requirements management plans, requirements traceability matrices,  requirements change forms, technical performance assessments, and  requirements board meeting minutes. Additionally, we interviewed  program officials to obtain additional information about their requirements  management practices. The conclusions reached for this objective are not  generalizable to the larger population of 34 business and non-business  MAIS programs.

We also reviewed the following risk management practices: determine risk sources and categories; define parameters used to analyze and categorize risks and to control  the risk management effort; establish and maintain the strategy to be used for risk management;  identify and document risks; evaluate and categorize each identified risk using defined risk  categories and parameters, and determine its relative priority; develop a risk mitigation plan in accordance with the risk management  strategy; and monitor the status of each risk periodically and implement the risk  mitigation plan as appropriate.

Specifically, we analyzed program risk documentation, including risk  reports, risk-level assignments, risk management plans, risk mitigation  plans, and risk board meeting minutes. Additionally, we interviewed  program officials to obtain additional information about their risks and risk  management practices.

To assess the reliability of the data of these programs we used to support  the findings in this report, we corroborated program office responses with  relevant program documentation and interviews with agency officials. We  found no data reliability issues and determined that the data used in this  report were sufficiently reliable for our reporting purposes. We have also  made appropriate attribution indicating the sources of the data.

We conducted this performance audit from April 2017 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact name above, the following staff also made key  contributions to this report: Eric Winter (Assistant Director), John Ortiz  (Analyst in Charge), Alex Bennett, Neha Bhatt, Chris Businsky, and  Rebecca Eyler.


Coast Guard: Improved Acquisition Portfolio Management Could Help Address Aging Assets and Capability Gaps

The Coast Guard, a component within DHS, is spending billions of dollars to acquire assets, such as cutters and aircraft. This portfolio of major acquisition programs is intended to help the Coast Guard accomplish its missions—including interdicting illegal drugs and search and rescue missions. GAO's extensive prior work on Coast Guard acquisitions has found that the Coast Guard's reliance on its annual budget process to manage its portfolio is a major management challenge. In the report issued today, GAO discusses particular challenges with the Coast Guard's approach in managing its acquisition portfolio, such as not performing a collective assessment of the portfolio to ensure affordability.

This statement addresses the challenges the Coast Guard faces in (1) managing its overall acquisition portfolio, and (2) sustaining aging assets. This statement is based on GAO's extensive body of published and ongoing work examining the Coast Guard's acquisition efforts over several years.

The Coast Guard's approach of relying on the annual budget process and the 5-year Capital Investment Plan (CIP) to manage its acquisition portfolio does not provide the best basis for making decisions to develop a more balanced and affordable portfolio in the long term. Specifically, the Coast Guard's annual budget-driven trade-off approach creates constant churn as program baselines must continually re-align with budget realities instead of budgets being formulated to support program baselines. Further, Coast Guard officials have told GAO the CIP reflects trade-off decisions made as part of the annual budget process, but it does not describe the effects of those trade-offs because including such information is not statutorily required. This short-term approach has also left the Coast Guard with a bow wave of near-term unfunded acquisition programs, putting future missions at risk. Until these trade-offs are transparent to all stakeholders and decision makers, the effectiveness of Coast Guard's long-term acquisition portfolio planning is limited.

Until new assets being acquired become available, the Coast Guard plans to rely on aging assets, many of which are already past their intended service lives—the time an asset is expected to operate. For example, the Coast Guard plans to replace the Medium Endurance Cutters (see figure) with the Offshore Patrol Cutters beginning in 2023, but the Medium Endurance Cutters exhausted their intended service lives in 2014.

The Coast Guard plans to extend service lives for some of the Medium Endurance Cutters to keep them operating longer; however, maintenance for these vessels is becoming more expensive, and some systems are obsolete. GAO will continue to monitor the maintenance effort for the Medium Endurance Cutter and the Offshore Patrol Cutter acquisition in an annual review of Department of Homeland Security (DHS) major acquisition programs.

The report on which this statement is primarily based ( GAO-18-454 ) recommends that the Coast Guard work with Congress to include in its annual CIP a discussion of how trade-off decisions could affect other acquisition programs. DHS agreed with this recommendation. GAO has also made other recommendations in this area in the past, as discussed in this testimony.

I am pleased to be here today to discuss key challenges the Coast Guard  faces as it seeks to modernize its aging assets, a process referred to as  Coast Guard recapitalization, including the management of the overall  Coast Guard acquisition portfolio. The Coast Guard continues to rely on  the annual budget process and its 5-year Capital Investment Plan (CIP)  for long-term acquisition planning, processes which we found have  contributed to capability gaps and funding shortfalls. For example, as the  Coast Guard continues this short-sighted approach, it is currently  experiencing a gap in its polar icebreaking capability, because the Coast  Guard does not have assets available to conduct its missions in both the  Arctic and Antarctic regions year round. The Coast Guard identified a  need for three heavy and three medium icebreakers in 2010, but to date it  has only one active heavy and medium icebreaker. Exacerbating this  capability gap is the condition of the Coast Guard’s only operating heavy  polar icebreaker, the Polar Star, which we found has experienced longer  than expected maintenance periods in 2016 and 2017 to prepare for its  annual mission to Antarctica.

My statement today will address challenges in the (1) management of the  overall Coast Guard acquisition portfolio, and (2) sustainment of certain  aging assets. This statement is based on our extensive body of work  examining the Coast Guard’s acquisition efforts spanning the past several  years, including our report on Coast Guard acquisitions released today in  conjunction with this statement. We also include preliminary information based on our ongoing review of the Coast Guard’s heavy polar icebreaker  acquisition.

For the reports cited in this statement, among other methodologies, we  analyzed Coast Guard guidance, data, and documentation, and  interviewed Coast Guard officials at its headquarters and field units to  determine the total cost of the Coast Guard’s acquisition portfolio and  how the Coast Guard manages its acquisition portfolio. Further detailed  information on our scope and methodology can be found in the reports  cited in this statement. For our ongoing work on the polar icebreaker, we  assessed the status of the Coast Guard’s efforts to recapitalize its heavy  polar icebreaking fleet.

We conducted the work on which this statement is based in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe the evidence obtained provides  a reasonable basis for our findings and conclusions based on our audit  objectives.

Coast Guard Faces Challenges in Effectively Managing its Acquisition Portfolio

Short-term Prioritization through the Annual Budget Process and the 5-Year Capital Investment Plan Limit Effective Planning

We found in September 2012, and in our July 2018 review, that the Coast  Guard’s approach of relying on the annual budget process and the 5-year  CIP to manage portfolio affordability does not provide the best basis for  making decisions to develop a more balanced and affordable portfolio in  the long term. Further, in June 2014, we found that there is no evidence  that short-term budget decisions will result in a good long-term strategy,  and the Coast Guard’s annual budget-driven trade-off approach creates  constant churn as program baselines must continually re-align with  budget realities instead of budgets being formulated to support program  baselines. This situation results in trade-off decisions between capability  and cost being pushed into the future. For example, since 2010, the  Coast Guard has a stated requirement for three medium polar  icebreakers, but it has only one operational medium icebreaker, the  Healy, which has an expected end of service life—the total period for  which an asset is designed to operate—in 2029. Despite the requirement  for three medium polar icebreakers, Coast Guard officials said they are  not currently assessing acquisition of the medium polar icebreakers  because they are focusing on the heavy icebreaker acquisition and plan  to assess the costs and benefits of acquiring medium polar icebreakers at  a later time.

As required by statute, the Coast Guard has, since 2012, prepared a 5- year CIP that it is required to update and submit annually with the  administration’s budget request. The 5-year CIP is the Coast Guard’s  key acquisition portfolio planning tool. However, in our July 2018 review,  we found that shortcomings of that plan that limit its effectiveness.  Specifically, we found that the Coast Guard’s 5-year CIPs continue to  demonstrate a pattern of certain ineffective planning practices, such as  not identifying priorities or trade-offs between acquisition programs and  not providing information about the effect of current decisions on the  overall affordability of the acquisition portfolio.

These shortcomings limit the Coast Guard’s ability to manage the  affordability of its acquisition portfolio. Coast Guard officials said the CIP  reflects the highest priorities of the department within the given top  funding level and that prioritization and trade-off decisions are made as  part of the annual budget cycle. However, the reasoning behind these  decisions, and the resulting impacts on affected programs, are not  articulated in the CIPs. While the Coast Guard is not required under  statute to identify the effects of trade-off decisions in the CIP, failing to  show which acquisitions would take on more risk—such as delays to  certain recapitalization efforts—so other acquisitions can be prioritized  and adequately funded within budget parameters also makes it difficult for  Congress and other stakeholders, such as Department of Homeland  Security (DHS) and the Office of Management and Budget (OMB), to  understand any other options the Coast Guard considered. GAO’s Cost  Estimating and Assessment Guide states that comparative analyses  showing facts and supporting details among competing alternatives, such  as budget priorities, should consider trade-offs needed to identify  solutions and manage risk. In the report we issued today, we  recommended that the Coast Guard work with Congress to include a  discussion of the acquisition programs it prioritized and describe how  trade-off decisions made could affect other acquisition programs in the  Coast Guard’s annual 5-year CIP. DHS agreed with our recommendation  and plans to include additional information in future CIP reports to  address how trade-off decisions could affect other major acquisition  programs. The Coast Guard plans to implement this recommendation by  March 2020.

In June 2014, we found that the Coast Guard needed to take a more  strategic approach in managing its acquisition portfolio. We  recommended that the Coast Guard develop a 20-year fleet  modernization plan that would identify all acquisitions necessary for  maintaining at least its current level of service and the fiscal resources  necessary to build these assets. DHS concurred with this  recommendation and the Coast Guard is in the process of developing a  20-year Long-Term Major Acquisitions Plan to guide and manage the  affordability of its acquisition portfolio, but DHS has not yet approved the  plan. Such an analysis would facilitate a fuller understanding of the  affordability challenges facing the Coast Guard while it builds the  Offshore Patrol Cutter, among other major acquisitions.

The lack of a long-term plan and continuing to determine priorities and  make trade-off decisions based on the annual budget have rendered the  Coast Guard’s acquisition planning reactive. We found that reactive  planning and the Coast Guard’s constrained budget environment have  created a bow wave of near-term unfunded acquisitions, negatively  affecting future acquisition efforts and potentially affecting future  operations. This bow wave consists of new acquisition programs and  recapitalization efforts, as well as high-cost maintenance projects that use  the same acquisition construction and improvements account, which  continue to put pressure on available resources. These projects include  some that are not currently identified in the 5-year CIP. For instance, the  Coast Guard’s 87-foot patrol boats are forecast to require recapitalization  beginning in 2023. Additionally, the ocean-going 175-foot coastal buoy  tenders are past the point in their service lives when a midlife  maintenance availability would normally have been conducted. In July  2018, we found that that the Coast Guard has historically operated  vessels well past their expected end of service life, and it will likely need  to do so with these assets given limited available acquisition funding.

Executive Oversight Council Has Not Conducted Annual Reviews of All Acquisitions Collectively

The Coast Guard has a management body—the Executive Oversight  Council—in place to conduct oversight of its major acquisition programs;  however, this management body has not conducted oversight across the  entire acquisition portfolio using a comprehensive, collective approach.  Among the Coast Guard’s three cross-directorate groups that have roles  in the acquisition process, we found in July 2018 that the Executive  Oversight Council is best positioned to oversee the portfolio collectively  and has the potential to implement key portfolio-wide management  practices, including conducting formal reviews and issuing reports. This  council has cross-directorate senior-level management representation,  access to information on acquisition programs, and support from the other  two cross-directorate groups (the Systems Integration Team and the  Resource Councils). However, this council has not carried out these  portfolio-wide practices.

In 2014, the Coast Guard updated the Executive Oversight Council’s  charter, in response to our September 2012 recommendation, adding the  responsibility for portfolio-wide oversight to include conducting an annual  review to assess and oversee acquisitions collectively. However, in our  July 2018 review, we found that the Coast Guard revised the council’s  charter in June 2017, removing this responsibility. According to  Executive Oversight Council officials, this responsibility was removed  from the 2017 charter because the council did not conduct these annual  reviews. Instead, Executive Oversight Council officials indicated that the  council facilitates a balanced and affordable portfolio of acquisition  programs through the individual program-level reviews. Best practices  states that successful organizations assess product investments in  aggregate, rather than as independent projects products or programs.  For example, by considering the requirements, acquisition, and budget  processes collectively, it helps organizations prioritize their product  investments.

Further, we found that the Executive Oversight Council has not engaged  in overseeing or reporting on the acquisition portfolio collectively and  annually. OMB’s 2017 Capital Programming Guide outlines a capital  programming process, including how agencies should effectively and  collectively manage a portfolio of capital assets. This OMB guidance  states that a senior-level executive review committee should be  responsible for reviewing the agency’s entire capital asset portfolio on a  periodic basis and for making decisions or priorities on the proper  composition of agency assets needed to achieve strategic goals and  objectives within the budget limits. In the case of the Coast Guard, only  the Executive Oversight Council has members at the senior-level  executive level and has the responsibility for oversight of its major  acquisition programs. Without conducting comprehensive, collective  portfolio reviews at the senior management level, the Coast Guard does  not have sufficient cross-directorate information to determine needed  trade-offs in the major acquisitions realm, considering budget realities. It  is also limiting its ability to make strategic decisions on future  requirements and capability gaps in a timely manner within the acquisition  portfolio. In our July 2018 report on Coast Guard recapitalization efforts,  we recommended that the Commandant of the Coast Guard should  require the Executive Oversight Council, in its role to facilitate a balanced  and affordable acquisition portfolio, to annually review the acquisition  portfolio collectively, specifically for long-term affordability. DHS  disagreed with our recommendation stating that other bodies within the  Coast Guard, such as the Investment Board, Deputies Council, and  Investment Review Board—are responsible for making decisions  regarding out-year funding, while the Executive Oversight Council works  outside of the annual budget process. DHS also stated that, to meet the  spirit of our recommendation, the Coast Guard will update the Executive  Oversight Council’s charter to require a review of the collective acquisition  portfolio, specifically evaluating long-term planning. We believe that  updating the Executive Oversight Council’s charter to include long-term- planning is a positive step. However, we continue to believe that in  addition to long-term planning, the Executive Oversight Council should  include the major acquisition portfolio’s budget realities faced by the  Coast Guard in its reviews, or long-term affordability. If the planning  accounts for long-term funding considerations to achieve the Coast  Guard’s acquisition goals and objectives, we believe the intent of our  recommendation would be met.

Coast Guard’s Heavy Polar Icebreaker Program’s Optimistic Schedule Is Driven by Capability Gap Rather Than Knowledge-Based Analysis

The Coast Guard’s short-term planning focus has, in part, driven the  acquisition of its heavy polar icebreaker program to its current situation— trying to meet a highly optimistic schedule. The heavy polar icebreaker  program is intended to field three new icebreakers to replace the Coast  Guard’s sole operational heavy polar icebreaker, the Polar Star. The  Polar Star is expected to reach the end of its service life between 2020  and 2023 while the first heavy polar icebreaker is expected to be  delivered in fiscal year 2023, with the second and third icebreakers  expected to be delivered in 2025 and 2026, respectively. Figure 1 shows  the potential icebreaking capability gap.

We are currently conducting a review of the heavy polar icebreaker  acquisition, and, preliminarily, we have found that the Coast Guard set an  optimistic schedule baseline for the delivery dates for new polar  icebreakers based on the ice-breaking capability gap rather than an  analysis of what is realistic and feasible. Rather than building a schedule  based on knowledge—such as determining realistic schedule targets and  analyzing how much time to include in the schedule to buffer against  potential delays, and comprehensively assessing schedule risks—the  Coast Guard used the estimated end date of the Polar Star’s service life  as the primary driver to set the lead icebreaker’s objective (or target)  delivery date of September 2023 and threshold (latest acceptable)  delivery date of March 2024. Design study information provided by  several shipbuilders estimated that it could take up to 3.5 years to build  the lead icebreaker, but the Coast Guard is planning for a more optimistic  estimate of 2.5 years for the delivery date. Our best practices for  developing project schedules state that estimating how long an activity  takes should be based on the effort required to complete the activity and  the resources available and not driven by a specific completion date.

In addition, preliminary findings indicate the Coast Guard did not conduct  analysis to identify a reasonable amount of margin or time to include in  the program schedule baseline to account for any delays in the program.  The current heavy polar icebreaker’s schedule includes only 6 months of  margin between the Coast Guard’s target and latest acceptable delivery  dates. However, our analysis of recent shipbuilding acquisitions shows  that longer schedule delays, whether they are in the program’s control or  not, should be expected. For example, among the 12 selected  shipbuilding acquisition programs active in the last 10 years that we  analyzed, the Navy and the Coast Guard have delayed delivery of all but  one lead ship from their original planned delivery dates, with delays  ranging from 9 to 75 months. We have found in our past shipbuilding work  that delays have resulted from a number of issues, including redesign  work to address discoveries during pre-delivery testing, and key system  integration problems, and design quality issues among others. However,  Coast Guard officials told us such risks are not accounted for in the  Heavy Polar Icebreaker schedule. We plan to issue a report on the Coast  Guard’s heavy polar icebreaker acquisition this summer. In addition, we  will continue to review this program in our annual assessment of major  acquisition programs.

Coast Guard Faces Sustainment Challenges for the Polar Star and 270- foot Medium Endurance Cutters

We found in July 2018 that the Coast Guard’s heavy polar icebreaker  Polar Star and the Medium Endurance Cutters are currently either  approaching or operating beyond the end of their design service lives.  These cutters are in need of major maintenance overhauls—or Service  Life Extension Projects (SLEP)—in order to continue providing  capabilities to operators. According to Coast Guard officials, SLEPs are  necessary because the Coast Guard does not have the funds available to  initiate a new major acquisition program to recapitalize these assets in the  short term, or because a significant amount of maintenance work is  required to keep these assets operational until replacements are fielded.  These planned SLEPs involve several risks including scheduling and  funding.

Heavy Icebreaker Polar Star has Required More Maintenance than Planned to Remain Operational

After being placed in a nonoperational status in 2006 due to equipment  problems, the Coast Guard conducted reactivation work on the Polar Star  from 2010 to 2013, and the icebreaker resumed its primary mission for  the annual deployment to the National Science Foundation’s McMurdo  Research Facility in Antarctica in 2014. Further, our July 2018 review  indicated that the Coast Guard is planning a SLEP on the Polar Star to  keep it operational until the first and second new heavy polar icebreakers  are delivered in order to bridge a potential operational gap. This  approach, according to Coast Guard officials, would allow the Coast  Guard to operate a minimum of two heavy icebreakers once the first polar  icebreaker is delivered and provide the Coast Guard with a self-rescue  capability—the ability for one icebreaker to rescue the other if it became  incapacitated while performing icebreaking operations.

However, we found that the Coast Guard’s plans to conduct this SLEP  during its annual depot-level maintenance periods—that is, maintenance  that is beyond the capability of the crew of a cutter or other asset—may  not be feasible given the amount of maintenance already required on the  cutter. Specifically, the Polar Star’s mission capable rating (an asset’s  availability to conduct operations) has been decreasing in recent years  and reached a low point of 29 percent—well below the target of 41  percent—from October 2016 to September 2017. Based on mission  capable data, we found this was mostly due to additional time spent in  depot-level maintenance, which has increased in recent years from about  6 months in 2015 to more than 8 months in 2017. Additionally, the Polar  Star has required extensions of about 3 months for its annual dry dock  periods—the period of time when a cutter is removed from the water so  that maintenance can be conducted—in 2016 and 2017 to complete  required maintenance activities. These dry docks were originally planned  to last between 2 1/2 months and 4 months. We found in July 2018 that  these delays and extensions are likely to continue as the cutter ages.  According to Coast Guard officials, the Polar Star’s SLEP work will be  conducted during the annual dry dock periods by adding an additional 1  or 2 months to the annual dry docks. However, if the work is unable to be  completed during this timeframe, it could force the Coast Guard to miss  its commitment to conduct its annual Antarctica mission. Coast Guard  maintenance officials stated that until the Polar Star completes the SLEP,  its repairs will likely continue to get more expensive and time consuming.

As we found in July 2017, the Polar Star SLEP effort has a rough-order  cost estimate of $75 million, which is based on the reactivation work  completed in 2013. However, we found this estimate may be unrealistic  based on assumptions the Coast Guard used, such as that it would  continue to use parts from the Coast Guard’s other heavy polar  icebreaker, the Polar Sea, which has been inactive since 2010. The  Coast Guard’s recent assessment of the Polar Star’s material condition— the physical condition of the cutter, which includes the hull structure,  habitability, major equipment systems, and spare parts availability—was  completed in January 2018. The material assessment stated that many  of the available parts from the Polar Sea have already been removed and installed on the Polar Star. As a result of the finite parts available from the  Polar Sea, the Coast Guard may have to acquire new parts for the Polar  Star that could increase the $75 million SLEP estimate. The Polar Star’s  recent material assessment will form the basis to determine which  systems will be overhauled during the SLEP and for a more detailed cost  estimate. The Coast Guard expects the Polar Star SLEP to begin by June  2020, at which time the Polar Star could reach the end of its current  useful service life (currently projected to be between 2020 to 2023). This  timeline contains risk that the Polar Star could be rendered inoperable  before the cutter is able to undergo a SLEP. We will continue to monitor  the Polar Star’s SLEP through our annual review of DHS programs.

Coast Guard Is Developing Plans to Extend Medium Endurance Cutters’ Service Lives

The Coast Guard operates two fleets of Medium Endurance Cutters (270- foot and 210-foot cutters) and both are either approaching or have  exceeded their design service lives. According to Coast Guard officials,  there are no plans to extend the service lives of the 210-foot Medium  Endurance Cutters due to the age of the vessels (some of the cutters will  be over 60 years old when they are expected to be removed from  service). However, we found in July 2018 that, according to Coast Guard  maintenance officials, the primary problem facing the 270-foot Medium  Endurance Cutters is obsolescence of parts. The cutters have several  systems that are no longer manufactured, and in many cases the original  manufacturer no longer makes parts for the systems, such as the  generators, fire pumps, and main diesel engines. To sustain the 270-foot  Medium Endurance Cutters until the replacement cutters—the Offshore  Patrol Cutters—are delivered, the Coast Guard is planning to conduct a  SLEP. Coast Guard officials stated they are evaluating how many of the  13 270-foot cutters will undergo the SLEP.

According to Coast Guard officials, the Offshore Patrol Cutter acquisition  program is on track to meet its cost and schedule goals. The Coast Guard  is in the process of completing the design of the cutter before starting  construction, which is in-line with GAO-identified shipbuilding best  practices. In addition, Coast Guard officials stated that the program is  using state-of-the-market technology that has been proved on other ships  as opposed to state-of-the-art technology, which lowers the risk of the  program. The Coast Guard expects to start construction of the first  Offshore Patrol Cutter in fiscal year 2019 and procure a total of 25 ships,  with plans to initially fund one cutter per year and eventually two cutters  per year until all 25 cutters are delivered. Further, Coast Guard officials  have stated that if the Offshore Patrol Cutter program experiences any  delays, it will likely decrease the Coast Guard’s operational capacity  because the legacy Medium Endurance Cutters will likely require  increased downtime for maintenance and other issues, reducing their  availability. As we indicated earlier, short-term planning limits the Coast  Guard’s ability to identify and consider tradeoffs with its acquisition  portfolio.

The Coast Guard is evaluating how long the 270-foot Medium Endurance  Cutters should remain in service. According to Coast Guard officials, this  decision is at least partially dependent on the delivery of the Offshore  Patrol Cutters—specifically the shipbuilder’s ability to deliver 2 cutters per  year, which is expected to start in fiscal year 2024 with the 4th and 5th  cutters. Officials stated that the Coast Guard does not plan to operate any  Medium Endurance Cutters once all 25 Offshore Patrol Cutters are  operational, yet the fiscal year 2018 through 2022 CIP report indicates  that 7 of the 270-foot Medium Endurance Cutters will still be in service  when all 25 Offshore Patrol Cutters are delivered and operational.  Officials said this is a contingency plan in case not all Offshore Patrol  Cutters are delivered on time. Figure 2 shows the planned delivery dates  for the Offshore Patrol Cutters and the proposed decommissioning dates  for the legacy Medium Endurance Cutters.

The fiscal year 2018 through 2022 CIP shows that there is little, if any,  gap between when the 210-foot and 270-foot Medium Endurance Cutters  will be removed from service and when the Offshore Patrol Cutters will be  operational. However, both Medium Endurance Cutter classes will be well  past their end of service lives by the time they are decommissioned. For  instance, in our July 2012 report, we found that the 210-foot Medium  Endurance Cutter Dependable reached its end of service life in 2006.  Nevertheless, based on the fiscal year 2018 through 2022 CIP, we found  that the Coast Guard plans for the cutter to operate for an additional 23  years (until 2029) without any major sustainment work to extend its  service life. While it is not unusual for the Coast Guard to operate cutters  for longer than originally planned, the lack of a more comprehensive,  collective portfolio management approach, in part, will result in some of  the Medium Endurance Cutters operating over 60 years, which is 30  years beyond their original design service lives.

In addition, the Coast Guard’s own assessments indicate likely  challenges. For instance, the Coast Guard’s February 2017 Sustainability  Assessment of the 210-foot Medium Endurance Cutters, it rated 5 of the  14 cutters as a high risk for sustainability, which reflects either a poor  material condition or high maintenance costs. Moreover, the most  recent material condition assessments for the Medium Endurance  Cutters, completed in 2015, found that

210-foot Medium Endurance Cutters cannot be expected to meet  operational requirements using the normal depot-level maintenance  funding levels due to the time required to complete maintenance and  the increased maintenance costs in recent years; and mission effectiveness of the 270-foot Medium Endurance Cutters will  continue to degrade without a near-continuous recapitalization of  older sub-systems.

In July 2012, we found that as assets age beyond their design service  lives, they can negatively affect the Coast Guard’s operational capacity to  meet mission requirements as the cutters require more maintenance.

We will continue to monitor the Medium Endurance Cutters’ SLEP and  the Offshore Patrol Cutter acquisition in our annual review of major  acquisition programs.

In conclusion, as the Coast Guard continues modernizing its fleet and  sustaining existing assets for longer than planned, it is important that it  develops a more strategic and comprehensive approach for managing its  portfolio so that future requirements and capability gaps can be  addressed in a timely manner. The Coast Guard has a history of using its  annual budgets to plan its acquisition portfolio, which leads to ever  changing priorities and creates deferred acquisitions and a bow wave of  future funding requirements. This bow wave has begun and the Coast  Guard will continue to add to it until it begins to have a longer-term focus,  such as with the creation of the 20-year Long Term Major Acquisition  Plan that we recommended in 2014. The Coast Guard has an opportunity  with this plan to lay the foundation for the success of the future acquisition  portfolio by showing what assets are needed and how much it is expected  to cost, and it will position itself to provide decision makers with critical  knowledge needed to prioritize its constrained acquisition funding. In the  meantime, the Coast Guard would benefit from describing in the 5-year  CIP how the annual trade-off decisions that are made could affect other  acquisition programs. This would help decision makers understand the  needs of the Coast Guard so that they can know how to better allocate  taxpayer dollars as they invest in new more capable Coast Guard assets.

Chairman Hunter, Ranking Member Garamendi, and Members of the  Subcommittee, this concludes my prepared statement. I would be  pleased to respond to any questions.

If you or your staff have any questions about this statement, please  contact Marie A. Mak, (202) 512-4841 or makm@gao.gov. In addition,  contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this statement. Individuals who  made key contributions to this testimony include Rick Cederholm,  Assistant Director; Peter W. Anderson; John Crawford; Claire Li; Roxanna  Sun; and David Wishard.

This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.

The Coast Guard is modernizing its fleets of aircraft, cutters, and other assets, but it continues to lack a long-term plan to guide these efforts. Instead, it relies on a short-term focus of yearly budgets and its 5-year plan. This has led to unpredictable funding and capability gaps.

For example, the Coast Guard has identified a need for three heavy and three medium icebreakers, but has only one of each.

In the report on which our testimony is based, we recommended that, in its annual reports to Congress, the Coast Guard describe how budget trade-off decisions will affect the other programs in its acquisition portfolio.