Data Center Optimization: Continued Agency Actions Needed to Meet Goals and Address Prior Recommendations

In December 2014, Congress enacted federal IT acquisition reform legislation that included provisions related to ongoing federal data center consolidation efforts. OMB's Federal Chief Information Officer launched DCOI to build on prior data center consolidation efforts; improve federal data centers' performance; and establish goals for inventory closures, cost savings and avoidances, and optimization performance.

The 2014 legislation also included a provision for GAO to annually review agencies' data center inventories and strategies. Accordingly, GAO reviewed agencies' data center closures to date and plans for further closures; evaluated agencies' progress in achieving consolidation savings and described their plans for future savings; and assessed agencies' progress against OMB's data center optimization targets. To do so, GAO assessed the 24 DCOI agencies' data center inventories as of August 2017; reviewed their reported cost savings documentation; evaluated their data center optimization strategic plans; and assessed 22 agencies' progress against OMB's established optimization targets. Two agencies did not have a basis to report planned optimization milestones.

OMB and the 24 DCOI agencies provided mixed responses to GAO's findings on the progress made towards initiative goals. GAO continues to believe that implementation of the recommendations made previously will help the agencies meet OMB's targets for cost savings and optimization of performance.

The 24 agencies participating in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI) reported mixed progress toward achieving OMB's goals for closing data centers by September 2018. Over half of the agencies reported that they had either already met, or planned to meet, all of their OMB-assigned goals by the deadline. This would result in the closure of 7,221 of the 12,062 centers that agencies reported in August 2017. However, 4 agencies reported that they do not have plans to meet all of their assigned goals and 2 agencies are working with OMB to establish revised targets.

With regard to agencies' progress in achieving cost savings, 20 agencies reported, as of August 2017, that they had achieved $1.04 billion in cost savings for fiscal years 2016 and 2017. In addition, the agencies' DCOI strategic plans identify an additional $0.58 billion in planned savings—for a total of $1.62 billion for fiscal years 2016 through 2018. This total is approximately $1.12 billion less than OMB's DCOI savings goal of $2.7 billion (see figure). This shortfall is the result of 12 agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans, as compared to the savings targets established for them by OMB.

The 24 agencies reported limited progress against OMB's five data center optimization targets for server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. As of August 2017, 1 agency had met four targets, 1 agency had met three targets, 6 agencies had met either one or two targets, and 14 agencies reported meeting none of the targets. Further, as of August 2017, most agencies were not planning to meet OMB's fiscal year 2018 optimization targets. Specifically, 4 agencies reported plans to meet all of their applicable targets by the end of fiscal year 2018; 14 reported plans to meet some of the targets; and 4 reported that they do not plan to meet any targets.

In 2016 and 2017, GAO made a number of recommendations to OMB and the 24 DCOI agencies to help improve the reporting of data center-related cost savings and to achieve optimization targets. As of March 2018, 74 of these 81 recommendations had not been fully addressed.

The federal government’s demand for information technology (IT) is ever  increasing. In recent years, as federal agencies have modernized their  operations, put more of their services online, and improved their  information security profiles, their need for computing power and data  storage resources has grown. Accordingly, this growing demand led to a  dramatic rise in the number of federal data centers and a corresponding  increase in operational costs.

To reduce data center duplication and costs, the Office of Management  and Budget’s (OMB) Federal Chief Information Officer (CIO) launched  two initiatives. The first, launched in 2010, was the Federal Data Center  Consolidation Initiative (FDCCI), which aimed to reduce the number of  data centers that were outdated or duplicative. The second initiative—the  Data Center Optimization Initiative (DCOI)—was launched in August 2016  and superseded FDCCI. This initiative shifted the focus to optimizing  agencies’ remaining data centers by requiring, among other things, that  agencies consolidate inefficient infrastructure, optimize existing facilities,  and transition to more efficient infrastructure, such as cloud services.

Further, Congress has recognized the importance of reforming the  government-wide management of IT and, in December 2014, enacted  Federal Information Technology Acquisition Reform provisions  (commonly referred to as FITARA) as a part of the Carl Levin and Howard  P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year  2015. Among the requirements related to federal data center  consolidation, the law requires:

Covered departments and agencies (agencies) to annually report to  OMB about federal data center inventories and strategies to achieve  consolidation, including yearly calculations of investment and cost  savings.

OMB to develop goals for the amount of planned cost savings and  optimization improvements that agencies are to achieve through  FDCCI. OMB is to make the goals publicly available and compare  progress against the goals.

In addition to these requirements, FITARA included a provision for GAO  to annually review and verify the quality and completeness of covered  agencies’ federal data center inventories and strategies for consolidation.  Accordingly, our specific objectives were to (1) review agencies’ data  center closures to date and plans for further closures, (2) evaluate  agencies’ progress in achieving data center consolidation savings and  describe plans for future savings, and (3) assess agencies’ progress  against OMB’s data center optimization targets.

On November 13, 2017, we briefed congressional committee staff on the  results of our work. This report officially transmits the results of our review  to the committees.

To review agency closures to date and plans, we obtained and analyzed  August 2017 data center inventory documentation from the 24 DCOI  agencies. We compared information on the agencies’ completed and  planned data center closures to OMB’s fiscal year 2018 consolidation  goals, as documented in its August 2016 memorandum (M-16-19). We  determined the number of data center that had been closed by adding the  closures from fiscal year 2010 through August 2017, as reported by the  agencies. We identified future closures by counting data centers that  agencies reported as planned closures, as of August 2017 through fiscal  year 2019.

To verify the quality, completeness, and reliability of the agencies’ data  center inventories, we compared the information on completed and  planned data center closures to similar information reported on OMB’s IT  Dashboard—a public website that provides information on federal  agencies’ major IT investments. We determined that the data were  sufficiently reliable to report on agencies’ consolidation progress and  planned closures.

In order to evaluate agencies’ progress in and plans for achieving data  center cost savings, we reviewed August 2017 cost savings and  avoidance documentation submitted by the 24 DCOI agencies in  response to OMB’s March 2013 and August 2016 memorandums. This  documentation included the agencies’ quarterly reports of cost savings  and avoidances posted to their digital services websites and discussed in  their DCOI strategic plans. We determined cost savings achieved by  adding agencies’ reported savings and avoidances from the start of fiscal  years 2012 through August 2017, as found in the August 2017 quarterly  reports posted to their digital services websites. We identified future  planned savings by totaling the agencies’ projected savings and  avoidances from fiscal years 2016 through 2018, as reported in their  DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data  center consolidation cost savings information, we used the latest version  of each agency’s update of the August 2017 quarterly cost savings report  and DCOI strategic plan. We also reviewed the quarterly reports and  DCOI strategic plans for missing data and other errors, such as missing  cost-savings information. In addition, we compared agencies’ reported  cost savings and avoidances with data from our most recently issued  report on data center consolidation. Further, we obtained written  responses from agency officials regarding the steps they took to ensure  the accuracy and reliability of their cost savings data. As a result, we  determined that the data were sufficiently reliable to report on agencies’  data center consolidation cost-savings information.

To assess agencies’ progress against OMB’s data center optimization  targets, we obtained the August 2017 data center optimization progress  information of 22 DCOI agencies, as reported on the IT Dashboard. We  then compared the agencies’ optimization progress information to OMB’s  fiscal year 2018 optimization targets, as documented in its August 2016  memorandum.

To assess the reliability of agencies’ optimization progress information on  OMB’s IT Dashboard, we reviewed the information for errors or missing  data, compared agencies’ optimization progress information across  multiple reporting quarters to identify any inconsistencies in agencies’  reported progress, and discussed with OMB and agency officials the  steps they took to ensure the accuracy and reliability of the reported  optimization progress. We determined that the data were sufficiently  reliable to report on agencies’ optimization progress.

In addition, to assess the reliability of the DCOI strategic plans, we  reviewed agencies’ documentation to identify any missing or erroneous  data. We also compared the planned data center optimization milestones  contained in agencies’ documentation against current optimization  progress information obtained from the IT Dashboard; we then discussed  any discrepancies or potential errors that we identified with agency  officials to determine the causes or request additional information. As a  result of these efforts, we determined that the agencies’ strategic plan  information was sufficiently reliable for reporting on plans to meet or not  meet OMB’s fiscal year 2018 optimization targets. Appendix I provides  greater details regarding our objectives, scope, and methodology.

We conducted this performance audit from July 2017 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The federal government’s increasing demand for IT has led to an  increase in the number of federal data centers and a corresponding  increase in operational costs. According to OMB, the federal government  had 432 data centers in 1998, 2,094 in July 2010, and 9,995 in August  2016.

Operating such a large number of centers has been, and continues to be,  a significant cost to the federal government. For example, in 2007, the  Environmental Protection Agency (EPA) estimated that the electricity  costs to operate federal servers and data centers across the government  were about $450 million annually. According to the Department of Energy  (Energy), a typical data center has 100 to 200 times the energy use  intensity of a commercial building. In 2009, OMB reported that server  utilization rates as low as 5 percent across the federal government’s  estimated 150,000 servers were a factor driving the need to establish a  coordinated, government-wide effort to improve the efficiency,  performance, and environmental footprint of federal data center  activities.

OMB and the Federal CIO Established FDCCI

Concerned about the size of the federal data center inventory and the  potential to improve the efficiency, performance, and the environmental  footprint of federal data center activities, OMB’s Federal CIO established  FDCCI in February 2010. This initiative’s four high-level goals were to  reduce the overall energy and real estate footprint of government data  centers; reduce the cost of data center hardware, software, and  operations; increase the overall IT security posture of the government;  and shift IT investments to more efficient computing platforms and  technologies.

In February 2010, OMB required all of the agencies participating in the  FDCCI to submit a data center inventory and a consolidation plan. In  October 2010, OMB also clarified the definition of a data center and noted  that, for the purposes of FDCCI, a data center was to be defined as any  room used for the purpose of processing or storing data that is larger than  500 square feet and meets stringent availability requirements. Under this  definition, OMB reported that agencies had identified 2,094 data centers  as of July 2010.

However, in 2011, the Federal CIO expanded the definition to include a  facility of any size and OMB published its revised definition in March  2012. Based on the revised definition, OMB estimated that there were a  total of 3,133 federal data centers in December 2011. In addition, its goal  was to consolidate approximately 40 percent, or 1,253 of these data  centers, for a savings of approximately $3 billion by the end of 2015.  Figure 1 provides an example of data center server racks at the Social  Security Administration’s (SSA) National Support Center.

The number of federal data centers reported by agencies has continued  to grow since 2011. In March 2016, we reported that agencies had  collectively identified a total of 10,584 data centers as of November  2015—an increase of about 7,500 data centers compared to OMB’s  October 2011 estimate. According to the Federal CIO, the increase in  the number of data centers was primarily due to the expanded definition  of a data center and improved inventory reporting by the agencies.

Further, OMB placed greater emphasis on data center optimization when  it issued memorandum M-13-09 in March 2013. Specifically, OMB  stated that, to more effectively measure the efficiency of an agency’s data  center assets, agencies would also be measured by the extent to which  their primary data centers are optimized for total cost of ownership by incorporating metrics for data center energy, facility, labor, and storage,  among other things. Subsequently, in May 2014, OMB issued  memorandum M-14-08, which established a set of data center  optimization metrics to measure agency progress, along with target  values for each metric. All agencies were expected to achieve the target  values by the end of fiscal year 2015.

IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts

Recognizing the importance of reforming the government-wide  management of IT, Congress enacted FITARA in December 2014.  Among other things, the law required agencies to:

Submit to OMB a comprehensive inventory of the data centers owned,  operated, or maintained by or on behalf of the agency.

Submit, by the end of fiscal year 2016, a multi-year strategy to  achieve the consolidation and optimization of the agency’s data  centers. The strategy was to include performance metrics that were  consistent with the government-wide data center consolidation and  optimization metrics.

Report progress towards meeting government-wide data center  consolidation and optimization metrics on a quarterly basis to OMB’s  Administrator of the Office of Electronic Government.

In addition, according to FITARA, the Office of Electronic Government at  OMB was to:

Establish metrics applicable to the consolidation and optimization of  data centers (including server efficiency), ensure that agencies’  progress toward meeting government-wide data center consolidation  and optimization metrics is made publicly available, review agencies’  inventories and strategies to determine whether they are  comprehensive and complete, and monitor the implementation of  each agency’s strategy.

Develop and make publicly available not later than December 19,  2015, a goal broken down by year, for the amount of planned cost  savings and optimization improvements to be achieved through  FDCCI and, for each year thereafter until October 1, 2020, compare  reported cost savings and optimization improvements against those  goals.

In August 2016, OMB issued memorandum M-16-19, which established  DCOI and included guidance on how to implement the data center  consolidation and optimization provisions of FITARA. Among other  things, the guidance required agencies to consolidate inefficient  infrastructure, optimize existing facilities, improve their security posture,  and achieve cost savings. For example, each agency was required to  maintain a complete inventory of all data center facilities owned,  operated, or maintained by or on its behalf, and measure progress toward  defined optimization performance metrics on a quarterly basis as part of  its data center inventory submission.

OMB’s memorandum also directed each agency to develop a DCOI  strategic plan that defined its data center strategy for fiscal years 2016  through 2018. Among other things, this strategy was to include a timeline  for agency consolidation and optimization activities with an emphasis on  cost savings and optimization performance benchmarks that the agency  could achieve between fiscal years 2016 and 2018. For example, each  agency was required to develop cost savings targets due to consolidation  and optimization actions and report any realized cost savings. OMB  required each agency to publicly post its DCOI strategic plan to its  agency-owned digital strategy website by September 30, 2016, and to  post subsequent strategic plan updates by April 14, 2017 and April 13,  2018.

Further, the memorandum stated that OMB was to maintain a public  dashboard (referred to as the IT Dashboard) to display government-wide  and agency-specific progress in areas such as planned and achieved  data center closures, consolidation-related cost savings, and data center  optimization performance information. In this regard, OMB began  including data center consolidation and optimization progress information  on the IT Dashboard in August 2016.

OMB’s memorandum also provided new guidance for the classification of  a physical data center, expanding the definition of a data center.  According to the revised definition, a room with at least one server that  provides services (whether in a production, test, staging, development, or  any other environment) should be considered a data center, while a room  containing only print servers, routing equipment, switches, security  devices (such as firewalls), or other telecommunication components, was  not to be considered a data center.

In light of this new definition, OMB directed each agency to perform a  comprehensive review of its data centers and maintain a complete and  updated data center inventory. Further, OMB directed each agency to  categorize each of its data centers as either a tiered data center or a non- tiered data center. OMB’s memorandum defined a tiered data center as  one that uses each of the following: a separate physical space for IT infrastructure; an uninterruptible power supply; a dedicated cooling system or zone; and a backup power generator for a prolonged power outage.

According to the memorandum, all other data centers are to be  considered non-tiered.

Moreover, OMB guidance included a series of performance metrics in the  areas of data center closures, cost savings, and optimization progress.

Data center closures: Agencies are expected to close at least 25  percent of tiered data centers government-wide, excluding those  approved as inter-agency shared services providers, by the end of  fiscal year 2018. Further, agencies are to close at least 60 percent  of non-tiered data centers government-wide by the end of fiscal year  2018. OMB’s guidance further notes that, in the long term, all  agencies should continually strive to close all non-tiered data centers,  noting that server rooms and closets pose security risks and  management challenges and are an inefficient use of resources.

Cost savings: Agencies are expected to reduce government-wide  annual costs attributable to physical data centers by at least 25  percent, resulting in savings of at least $2.7 billion for fiscal years  2016 through 2018.

Data center optimization: Agencies are expected to measure progress  against a series of new data center performance metrics in the  areas of server utilization, energy metering, power usage, facility  utilization, and virtualization. Further, OMB’s guidance established  target values for each metric that agencies are to achieve by the end  of fiscal year 2018.

OMB’s guidance further noted that agency progress against these  performance metrics is to be measured by OMB on a quarterly basis,  using agencies’ data center inventory submissions and OMB-defined  closures, cost savings, and optimization targets.

Agencies Have Taken Limited Action to Address GAO Recommendations from Prior Reports on Agencies’ Consolidation and Optimization Efforts

Since the enactment of FITARA in December 2014, we have annually  reviewed and verified the quality and completeness of each covered  agency’s inventory and DCOI strategy. We have also published reports  documenting the findings from each of these reviews. In addition, we  have examined and reported on agencies’ efforts to optimize their data  centers, as well as the challenges encountered and successes  achieved.

In a report that we issued in March 2016, we noted that agencies had  reported significant data center closures—totaling more than 3,100  through fiscal year 2015—with the Departments of Agriculture  (Agriculture), Defense (Defense), the Interior (Interior), and the Treasury  (Treasury) accounting for 84 percent of the total. Although agencies fell  short of OMB’s fiscal year 2015 consolidation goal, their plans identified  about 2,100 additional centers planned for closure through fiscal year  2019.

Agencies also reported significant consolidation cost savings and  avoidances—totaling about $2.8 billion through fiscal year 2015, and  expected to increase to over $8.0 billion in future years. The Departments  of Commerce (Commerce), Defense, Homeland Security (DHS),  Transportation (Transportation), and the Treasury accounted for 96  percent of the total planned savings. However, we pointed out that many  agencies lacked complete cost savings goals for the next several years  despite having closures planned.

In addition, we reported that 22 agencies had made limited progress  against OMB’s fiscal year 2015 data center optimization performance  metrics, such as the utilization of data center facilities. Accordingly, we  recommended that the agencies take actions to complete their cost  savings targets and improve optimization progress. Of the 24 agencies to  which we made recommendations, 14 agreed with our recommendations,  4 did not state whether they agreed or disagreed, and 6 stated that they  had no comments. As of March 2018, 26 of the 32 recommendations from  this report had yet to be fully addressed.

In May 2017, we reported that the agencies continued to report significant  data center closures—totaling more than 4,300 through August 2016— with Agriculture, Defense, Interior, and the Treasury accounting for 84  percent of the total. The agencies’ plans for 2016 had identified more  than 1,200 additional centers planned for closure through fiscal year  2019.

Agencies also reported significant consolidation and optimization cost  savings and avoidances, which totaled about $2.3 billion through August  2016. However, reductions in the amount of achieved savings reported to  OMB, particularly by the Treasury, resulted in a net decrease of more  than $400 million in these savings, compared to amounts we previously  reported in 2015.

Further, our report noted that, as of December 2016, agencies’ total  planned cost savings of about $656 million were more than $3.3 billion  less, compared to the amounts that we reported in 2015, and more than  $2 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7  billion. This reduction in planned savings was the result of eight agencies  reporting less in planned cost savings and avoidances in their DCOI  strategic plans compared to the savings amounts previously reported to  us in November 2015. The reduction also reflected the absence of cost  savings information for one agency (Defense) that did not submit its  strategic plan in time for our review.

In addition, our May 2017 report identified weaknesses in agencies’ DCOI  strategic plans. Of the 23 agencies that had submitted their strategic  plans at the time of our review, 7 agencies—Agriculture, the Department  of Education (Education), DHS, and the Department of Housing and  Urban Development (HUD); the General Services Administration (GSA);  the National Science Foundation (NSF); and the Office of Personnel  Management (OPM)—had addressed all five required elements of a  strategic plan, as identified by OMB (such as providing information related  to data center closures and cost savings metrics). The remaining 16  agencies that submitted their plans either partially met or did not meet the requirements. We also pointed out that there were inconsistencies in the  reporting of cost savings in the strategic plans of 11 agencies.

Given these findings, we recommended that OMB improve its oversight of  agencies’ DCOI strategic plans and their reporting of cost savings and  avoidances. We also recommended that 16 agencies and Defense (which  did not submit a plan in time for our review) complete the missing  elements in their strategic plans, and that 11 agencies ensure the  reporting of consistent cost savings and avoidance information to OMB.  Of the 25 agencies (including OMB) to which we made recommendations,  12 agreed with our recommendations, 2 disagreed, and 11 did not state  whether they agreed or disagreed. As of March 2018, 29 of the 30  recommendations had not been fully addressed.

In a subsequent report that we issued in August 2017, we noted that 22 of  the 24 agencies required to participate in the OMB DCOI collectively had  reported limited progress against OMB’s fiscal year 2018 performance  targets for the five optimization metrics. The 2 remaining agencies,  Education and HUD, did not have agency-owned data centers and,  therefore, did not have a basis to report on progress. Specifically, for  each of the five targets, no more than 5 agencies reported that they had  met or exceeded that specific target. This limited progress against OMB’s  optimization targets was due, in part, to agencies not fully addressing our  prior recommendations in this area.

In addition, we noted in the report that most agencies had not yet  implemented automated monitoring tools to measure server utilization, as  required by the end of fiscal year 2018. Specifically, 4 agencies reported  that they had fully implemented such tools, 18 reported that they had not  yet done so, and 2 did not have a basis to report on progress because  they did not have agency-owned data centers.

We also noted that, although federal standards emphasize the need to  establish plans to help ensure goals are met, none of the 18 agencies  had fully documented plans for implementing automated monitoring tools.  Accordingly, we recommended that OMB formally document a  requirement for agencies to include plans, as part of existing OMB  reporting mechanisms, to implement automated monitoring tools at their  agency-owned data centers. We also recommended that the 18 agencies  without fully documented plans take action, within existing OMB reporting  mechanisms, to complete plans describing how they intend to achieve  OMB’s requirement to implement automated monitoring tools at all  agency-owned data centers by the end of fiscal year 2018. Of the 19  agencies (including OMB) to which we made recommendations, 10  agreed with our recommendations, 3 partially agreed, and 6 did not state  whether they agreed or disagreed. As of March 2018, none of the 19  recommendations had been fully addressed.

Agencies Report Mixed Results Relative to Achieving OMB’s Data Center Closure Targets

As previously mentioned, in August 2016, OMB established a goal to  close at least 25 percent of each agency’s tiered data centers and at least  60 percent of each agency’s non-tiered data centers by the end of fiscal  year 2018. Related to doing so, agencies’ August 2016 inventories  reported a total of 9,995 data centers and, in August 2017, 14 agencies  reported an additional 2,067 facilities, for a total of 12,062 data centers.  Based on this revised inventory, agencies will need to close 6,306 data  centers (665 tiered and 5,641 non-tiered) to meet OMB’s goals by the end  of fiscal year 2018.

Toward this end, the 24 agencies participating in DCOI collectively have  made progress on their data center closure efforts. Specifically, as of  August 2017, the agencies reported that they had closed 5,805 tiered and  non-tiered centers (48 percent). Figure 2 provides a summary of the total  number of federal data centers and closures reported from 1998 to  August 2017.

Figure 3 provides a further breakdown of agencies’ data center  inventories, as of August 2017, in terms of the total number of data  centers that were closed, planned for closure, or not planned for closure.

Nevertheless, while the agencies collectively had made progress toward  OMB’s closure goals, the 24 agencies’ individual reported progress and  plans showed mixed results when compared with OMB’s goal for each  agency to close at least 25 percent of tiered data centers and at least 60  percent of non-tiered centers. Specifically, as of August 2017, 13  agencies reported that they had already met the goal of closing 25  percent of their tiered data centers, another 4 agencies reported that they  plan to meet the goal by the end of fiscal year 2018, and 5 agencies  reported that they do not currently plan to meet the goal.

Further, as of August 2017, 7 agencies reported having already met the  goal for closing 60 percent of their non-tiered centers, 6 agencies  reported that they planned to meet the goal by the end of fiscal year  2018, and 10 agencies reported that they did not plan to meet the goal.  Table 1 displays a breakdown of the number of reported tiered and non- tiered data centers and completed and planned closures by agency, as of  August 2017.

As shown in the table, the reported closures of Agriculture (2,233 data  centers), Defense (834), and Treasury (1,713) together accounted for  4,780 (or 82 percent) of the 5,805 data center closures. However, no  other agency accounted for more than 187 (or 3 percent) of those  closures. In addition, the remaining 1,416 planned closures are to be  carried out across 20 agencies. Further, among the agencies, 7 reported  that they do not plan to meet one of their tiered or non-tiered closure  goals, and 4 reported that they do not plan to meet either of the goals by  the end of fiscal year 2018.

Officials from these 11 agencies that do not plan to meet one or both of  their goals provided various reasons for why they currently do not plan to  do so. For example, officials in USAID’s Office of the Chief Information  Officer stated that their agency had reported a number of server closets in  overseas locations as non-tiered data centers to comply with OMB’s data  center definition provided in its August 2016 guidance. The officials said  that, as long as USAID maintains those locations, the agency needs the  services provided in those server closets and will not be able to close  them to meet OMB’s goal. However, the officials also said that the agency  is exploring ways to replace the server closets using cloud services.

Several agencies that viewed their goals as unattainable indicated that  they were seeking revised closure goals. Specifically, officials from  Interior’s Office of the Chief Information Officer stated that a number of  the department’s non-tiered data centers were either mission critical or  not cost effective to close. Thus, the officials said Interior was working  with OMB to establish a revised closure goal. Similarly, Transportation’s  Director for IT Compliance stated that the department was working with  OMB to establish a revised closure goal. The department reported having  186 tiered data centers in Federal Aviation Administration control towers  that it believes should be excluded from its count of data centers when  OMB sets the department’s goal for closures.

Further, our analysis determined that it may not always be realistic for an  agency to meet OMB’s targets. For example, NSF, which does not have  any tiered data centers, reported that it plans to close one (50 percent) of  its two non-tiered data centers. However, the only way to meet OMB’s 60  percent threshold would be for the agency to close both of its non-tiered  centers, which may not be an option, depending on the services provided  by that one remaining center.

Although OMB’s stated time frame for closing data centers currently  remains as the end of fiscal year 2018, the recent extension of FITARA’s  data center consolidation and optimization provisions through fiscal year  2020 provides agencies additional time to work toward meeting OMB’s  closure targets. However, in some cases, these efforts may require  significant restructuring of an agency’s business operations, or, as  reported by several agencies, a revision of OMB’s goals in consideration  of the agencies’ specific needs.

Given that agencies had been working toward OMB’s DCOI goals for  approximately one year as of August 2017, and because the extension of  FITARA’s data center provisions pushes the sunset for these efforts out  through fiscal year 2020, we are not making any related  recommendations to those agencies that have not met the closure goals  at this time. We plan to continue to monitor the agencies’ progress toward  meeting the goals in our future work.

Agencies’ Planned Savings Are Not Expected to Meet OMB’s Goal

Since 2013, federal agencies have been required to report on data center  cost savings. In this regard, OMB provided guidance regarding how  agencies were to report cost savings and avoidances. Specifically, it  required agencies to report both data center consolidation cost savings  and avoidances, among other areas, as part of a quarterly data collection  process known as the integrated data collection. FITARA also called for  each agency to submit a multi-year strategy for achieving the  consolidation and optimization of data centers that includes year-by-year  calculations of investment and cost savings through fiscal year 2018,  which has now been extended to 2020.

In addition, in August 2016, OMB M-16-19 provided guidance on how  agencies should implement the requirements of FITARA. Specifically,  agencies were to develop a strategic plan that included information on  historical cost savings and avoidances due to data center consolidation  and optimization through fiscal year 2015. This guidance stated that  agency strategic plans were also to include year-by-year calculations of  target and actual agency-wide spending and cost savings on data centers  from fiscal years 2016 through 2018. Further, the guidance established a  DCOI government-wide cost savings goal of $2.7 billion for all federal  agencies to achieve from fiscal years 2016 through 2018. This overall  goal is then broken down into agency-specific targets on the IT  Dashboard.

As of August 2017, 20 agencies had reported through the integrated data  collection that they had achieved $1.04 billion in cost savings for fiscal  years 2016 and 2017, while 4 agencies reported that they had not  achieved any savings. Further, the 20 agencies’ DCOI strategic plans  identified an additional $0.58 billion, for a total of $1.62 billion in planned  savings from fiscal years 2016 through 2018. Nevertheless, this total is  about $1.12 billion below OMB’s goal of $2.7 billion for DCOI savings.  Figure 4 provides a comparison of the 24 agencies’ total reported savings  for fiscal years 2016 and 2017, and the planned savings through 2018,  against OMB’s DCOI savings goal for fiscal years 2016 through 2018.

Among the 24 participating DCOI agencies that reported achieving about  $1.04 billion in savings, Commerce ($594.28 million), Defense ($141.36  million), and DHS ($106.51 million) were responsible for approximately  $842 million (about 81 percent) of that total. No other agency reported  saving more than $54.40 million. Table 2 provides specific data related to  each agency’s planned and achieved savings for fiscal years 2016 and  2017, as of August 2017.

In addition, the 24 agencies reported that they have planned an additional  $0.58 billion in DCOI cost savings (for a total of $1.62 billion) through  fiscal year 2018. However, as noted earlier, this total is approximately  $1.12 billion below OMB’s $2.7 billion goal for the initiative. Table 3  provides a more detailed comparison between each agency’s planned  savings, as reported in its DCOI strategic plan, and OMB’s agency- specific targets, as reported on the IT Dashboard.

As shown in table 3, 6 agencies identified planned savings that are  expected to meet or exceed their OMB targets, while 2 agencies that did  not have an OMB target also identified planned savings. In contrast, 12  agencies reported that they are not currently planning to meet their  targets, and 4 agencies did not have a savings target and are not  planning any savings.

These findings align with what we reported in March 2016, when we  noted that 10 agencies had not established planned cost savings goals  for fiscal years 2016 through 2018, even though they had closures  planned during that time period. Accordingly, in that report, we  recommended that these agencies complete their planned data center  cost savings targets for fiscal years 2016 through 2018. Most of the  agencies agreed with the recommendations or had no comments.  Nonetheless, agencies continued to be challenged in identifying and  reporting their cost savings. As of August 2017, 5 of the agencies had  implemented our recommendations in this area.

In the absence of consistent and full reporting of fiscal years 2016  through 2018 planned savings in agencies’ DCOI strategic plans, as  required by FITARA and OMB, agencies’ total planned savings will likely  continue to be understated. With less than a year for agencies to meet  OMB’s current planned savings targets, we are re-emphasizing the need  for agencies to implement our prior recommendations related to  establishing and meeting their planned data center cost savings targets.

Agencies Continue Reporting Limited Progress against OMB’s Data Center Optimization Metrics

FITARA required OMB to establish metrics to measure the optimization of  data centers, including server efficiency, and ensure that agencies’  progress toward meeting the metrics is made public. Pursuant to FITARA,  OMB’s August 2016 memorandum established a set of five data center  optimization metrics intended to measure agencies’ progress in the areas  of server utilization and automated monitoring, energy metering, power  usage effectiveness, facility utilization, and virtualization. According to  OMB, while the server utilization and automated monitoring metric applies  to agency-owned tiered and non-tiered data centers, the four remaining  metrics apply only to agency-owned tiered centers.

OMB’s memorandum also established a target value for each of the five  metrics, which agencies are expected to achieve by the end of fiscal year  2018. OMB measures agencies’ progress against the optimization targets  using the agencies’ quarterly data center inventory submission and  publicly reports this progress information on its IT Dashboard. Table 4  provides a description of the five data center optimization metrics and  target values.

As of August 2017, 22 of the 24 DCOI agencies continued to report  limited progress in meeting OMB’s fiscal year 2018 data center  optimization targets identified on the IT Dashboard. As noted earlier, the  remaining 2 agencies—Education and HUD—reported that they did not  have any agency-owned data centers in their inventory and, therefore, did  not have a basis to measure and report optimization progress.

With regard to the data center optimization targets, the most progress  was reported for the power usage effectiveness and virtualization metrics,  with 5 and 6 agencies, respectively, reporting that they had met OMB’s  targets. However, only 3 agencies or fewer reported meeting the energy  metering, facility utilization, and server utilization and automated  monitoring metrics. Figure 5 summarizes the 24 agencies’ progress in  meeting each optimization target, as of August 2017.

As of August 2017, SSA and EPA reported the most progress among the  22 agencies with a basis to report against OMB’s metrics targets— meeting 4 and 3 targets, respectively. Six agencies met either one or two  targets, and 14 agencies reported meeting none of the targets.

Further, of the 22 agencies, 9 were not able to report any progress  against either the server utilization metric or power usage effectiveness  metric, or both, because their data centers lacked the required monitoring  tools to measure progress in these areas. OMB began requiring the  implementation of these monitoring tools in August 2016; however, as of  August 2017, these 9 agencies had not reported that they had  implemented the tools at any data centers. The remaining 13 agencies  reported that they had implemented the tools in at least one data center.  Table 5 depicts the agencies and whether they met or did not meet each  OMB target.

Agencies’ limited progress against OMB’s optimization targets is due, in  part, to not fully addressing our prior recommendations in this area. As  discussed earlier, in March 2016, we reported on weaknesses in  agencies’ data center optimization efforts, including that 22 agencies did  not meet OMB’s fiscal year 2015 optimization targets. We noted that  this was partially due to the agencies facing challenges in optimizing their  data centers, including their decentralized organizational structures that  made consolidation and optimization difficult, and competing priorities for  resources. In addition, consolidating certain data centers was problematic  because the volume or type of information involved required the data  center to be close in proximity to the users. Accordingly, we  recommended that the agencies take action to improve optimization  progress, to include addressing any identified challenges. Most agencies  agreed with our recommendations or had no comments.

In response to our recommendation, 19 of the 22 agencies submitted  documentation to us that described steps they intended to take to  improve their data center optimization efforts. The planned steps included  completion dates ranging between April 2016 and September 2019.  Among the steps described by the agencies was developing internal  scorecards to track and report on optimization progress, including  progress at their component agencies, and launching more aggressive  efforts to optimize data centers using virtualization and cloud computing  solutions. However, as of February 2018, only 1 of the 22 agencies  (Education) had fully addressed our recommendation.

Agencies’ Planned Optimization Progress Will Not Meet OMB’s Fiscal Year 2018 Targets

In addition to reporting current optimization progress on the IT  Dashboard, OMB requires agencies’ DCOI strategic plans to include,  among other things, planned performance levels for fiscal years 2017 and  2018 for each optimization metric.

However, according to the 24 agencies’ DCOI strategic plan information  as of August 2017, most are not planning to meet OMB’s optimization  targets by the end of fiscal year 2018. More specifically, of the 24  agencies, only 4—Commerce, EPA, NSF, and USAID—reported plans to  fully meet their applicable targets by the end of fiscal year 2018. Of the  remaining agencies, 14 reported plans to meet some, but not all, of the  targets; 4 reported that they do not plan to meet any targets; and—as  already discussed—Education and HUD do not have a basis to report planned optimization milestones because they do not report having any  agency-owned data centers. Figure 6 summarizes agencies’ progress,  as of August 2017, in meeting OMB’s optimization targets and planned  progress to be achieved by September 2018.

The limited progress made by agencies in optimizing their data centers,  combined with the lack of established plans to improve progress, makes it  unclear whether agencies will be able to achieve OMB’s optimization  targets by the end of fiscal year 2018. Considering that OMB is expecting  at least $2.7 billion in cost savings from agencies’ optimization efforts, the  ability of agencies to meet the optimization targets will be critical to  meeting this savings goal. However, only four agencies are planning to  meet all of their applicable targets. If the remaining agencies take steps to  implement the prior recommendations we have made in this area, it  should increase the likelihood that DCOI can achieve the expected  benefits of optimization and the resulting cost savings.

We requested comments on a draft of this report from OMB and the 24  other agencies that we reviewed. Of these, 5 agencies indicated that they  agreed with our report and 20 acknowledged receiving the draft, but did  not state whether they agreed or disagreed with the report. Agencies also  offered various comments in support of the DCOI effort and actions taken  to improve performance. Additionally, multiple agencies provided  technical comments, which we have incorporated, as appropriate.

The following five agencies agreed with our report:  In emails received from Agriculture, Energy, VA, and NRC, the  agencies agreed with the findings in the draft report. In written  comments, SSA agreed with the draft report’s characterization of the  agency’s DCOI efforts. SSA’s comments are reprinted in appendix II.

In addition, the following 20 agencies did not state whether they agreed or  disagreed with the report and offered other comments:   In written comments, HUD and Transportation did not agree or  disagree with the draft report. The departments’ comments are  reprinted in appendices III and IV, respectively.

In emails received from Commerce, Defense, Education, HHS, DHS,  Justice, Labor, State, Treasury, EPA, NASA, NSF, OPM, SBA, and  USAID, the agencies did not agree or disagree with the draft report.

In written comments, Interior did not agree or disagree with the draft  report’s findings.

In addition, Interior provided a technical comment related to our  calculation of the department’s achieved optimization savings for  fiscal years 2016 and 2017. Specifically, the department provided  updated savings figures for those years, which were based on  Interior’s February 2018 quarterly submission to OMB.

However, the effective date of the updated data (February 2018) is  outside the scope of our review, which relied solely on data reported  in August 2017, as detailed in appendix I. While we recognize the  department’s efforts to ensure that its past reporting is updated and as  accurate as possible, our report only presents data as reported by  agencies in August 2017. Consequently, we believe that the achieved  cost savings for Interior accurately reflect what the department  reported at the time of our review. The department’s comments are  reprinted in appendix V.

In an e-mail received on March 27, 2018, GSA did not agree or  disagree with the draft report’s findings, and provided comments  questioning the methodology we used to determine the number of  data center closures. Specifically, GSA stated that, by including  facilities that are now closed in our reporting of the federal  government’s inventory of 12,062 data centers, our report gives the  impression that the government is currently operating 12,062 open  data centers. GSA also asserted that we should include in our counts  facilities identified as using cloud computing providers.

In our report, we consistently state that the 12,062 data centers reflect  the overall count of federal facilities identified since the launch of  FDCCI in 2010. Further, we clearly identify the portion of the overall  count of data centers that have since closed, or that are planned to  close. By doing so, we provide a perspective of consolidation  progress against the overall inventory over the past 8 years.

Further, in regard to facilities using cloud providers, OMB’s August  2016 guidance specifically states that such facilities are not to be  considered data centers. As such, we did not include them in our  totals. Accordingly, we maintain that our methodology is reasonable  and continue to believe that our report accurately reports the status of  federal data center consolidation efforts.

GSA also suggested that we confirm with OMB our statement that the  FITARA Enhancement Act of 2017’s extension of the sunset provides  agencies with an additional 2 years to accomplish the goals of DCOI.  It is true that OMB has not issued guidance that extends the existing  August 2018 DCOI deadlines. However, it is appropriate to note that  the 2017 law extends the DCOI sunset date for the data center  requirements that govern agencies.

Finally, GSA expressed concern that the data on optimization metrics  that we drew from the IT Dashboard were not representative of the  data provided by agencies in their August 2017 quarterly submissions  to OMB. Further, GSA noted that such a selection of mixed data  sources may present an incomplete and inaccurate picture and  recommended that we note the date on which we accessed the IT  Dashboard data, note that those data are frequently updated, and  specify the date of other sources of data that we used for our analysis.

In this report, we present an analysis of agencies’ progress against  OMB’s data center optimization metrics, using data that were taken  from the IT Dashboard. These data are posted to the Dashboard after  being automatically calculated from agencies’ quarterly submissions  to OMB. For the purposes of this report, we pulled the data from the  Dashboard a week after agencies’ August 2017 submissions were  due to OMB. We then confirmed with the agencies that the data we  collected were consistent with their August 2017 submissions to OMB  and we labeled the data as being effective as of August 2017.

We also analyzed other data from the agencies’ August 2017  quarterly submissions, such as the status of data center closures and  associated cost savings, and showed the effective date of our  analysis as being August 2017. Based on these actions, which are  also discussed in the description of the scope and methodology of our  audit work found in app. I, we believe that our presentation of the data  on optimization metrics is accurate, appropriately labeled, and  correctly reflects the status of agency efforts at a specific moment in  time.

In an e-mail received on April 18, 2018, OMB did not agree or  disagree with our report, but offered several comments on our  findings. Specifically, OMB noted that some agencies may be  reporting their planned savings incorrectly in their DCOI strategic  plans in that agencies may be reporting annual savings figures  instead of the required cumulative figures. OMB further described  plans to update the IT Dashboard in the near future to more  accurately reflect planned DCOI savings and added that, as a result,  the data in our report likely will not match the data OMB intends to  publish.

In conducting our analysis, on two occasions, we requested and  received agencies’ validation of the results of our analysis of their  planned cost savings. This process resulted in minor technical  changes to some agencies’ data. We believe that our continued  efforts to validate these data provide reasonable assurance as to the  accuracy of the agency reported information that we analyzed. The  implementation of OMB’s proposed changes to the IT Dashboard  should provide yet another tool that can be used to improve how  agencies report their cost savings—an important measure of DCOI’s  success.

Additionally, the comments noted that GAO and OMB use a different  basis to calculate agencies’ data center closure targets, with OMB  using a baseline inventory from the beginning of DCOI in 2016, which  does not recognize changes in agencies’ inventories since that point.  OMB asserted that, because GAO’s calculations account for changes  in agencies’ inventories since the beginning of DCOI, OMB’s targets  differ from GAO’s calculations.

We recognize the difference in approach for calculating data center  closures that we used, as compared to that used by OMB. As detailed  in our report, when DCOI was launched in August 2016, agencies  reported an overall inventory of 9,995 data centers. One year later, in  August 2017, agencies reported more than 2,000 additional facilities,  for a total of 12,062 centers.

Because OMB’s closure targets required agencies to close a certain  percentage of their data centers, basing agency goals on an outdated  inventory (that did not include the additional facilities) does not give a  true picture of progress towards consolidation of data centers.  Conversely, our methodology (detailed in app. I) takes into account  inventory growth and uses the same percentage-based closure goals  defined in OMB’s guidance. Our methodology presents a more  accurate status of progress based on the growth of the inventory over  time. Consequently, we believe that our methodology allows us to  present a reasonable status of agencies’ progress against OMB’s  goals.

We are sending copies of this report to interested congressional  committees, the Director of OMB, the secretaries and heads of the  departments and agencies addressed in this report, and other interested  parties. In addition, the report will be available at no charge on GAO’s  website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-9286 or pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this engagement were to: (1) review agencies’ data  center closures to date and plans for further closures, (2) evaluate  agencies’ progress in achieving data center consolidation savings and  describe plans for future savings, and (3) assess agencies’ progress  against the Office of Management and Budget’s (OMB) data center  optimization targets.

To address the first objective, we obtained and analyzed August 2017  data center inventory documentation from the 24 departments and  agencies (agencies) that participate in OMB’s Data Center Optimization  Initiative (DCOI). To determine data center closures to date, we totaled  agency reported closures from fiscal year 2010 through August 2017 and,  to identify future closures, we totaled agency reported planned closures  through fiscal year 2019. We also compared agencies’ completed and  planned closures to OMB’s fiscal year 2018 consolidation goals, as  documented in its August 2016 memorandum (M-16-19).

To verify the quality, completeness, and reliability of the agencies’ data  center inventories, we compared information on completed and planned  data center closures to similar information reported on OMB’s Information  Technology (IT) Dashboard—a public website that provides information  on federal agencies’ major IT investments. We also checked for missing  data and other errors, such as missing closure status information.  Further, we obtained written responses from agency officials regarding  actions taken to ensure the reliability of their inventory data, and  discussed any discrepancies or potential errors identified to determine the  causes or to request additional information. We determined that the data  were sufficiently complete and reliable to report on agencies’  consolidation progress and planned closures.

For the second objective, we obtained and analyzed cost savings and  avoidance documentation from the 24 DCOI agencies. This  documentation is required by OMB’s March 2013 and August 2016  memorandums and included the agencies’ quarterly reports of cost  savings and avoidances posted to their digital services websites and their  DCOI strategic plans. To determine cost savings achieved, we totaled  agencies’ reported savings and avoidances from the start of fiscal years  2012 through August 2017, as found in the August 2017 quarterly reports  posted to the agencies’ digital services websites. To identify future  planned savings, we totaled the agencies’ projected savings and  avoidances from fiscal years 2016 through 2018, as reported in their  DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data  center consolidation cost savings information, we used the latest version  of each agency’s update of the August 2017 quarterly cost savings report  and DCOI strategic plan. We also reviewed the quarterly reports and  DCOI strategic plans for missing data and other errors, such as missing  cost-savings information. In addition, we compared agencies cost savings  and avoidances with data from our most recent data center consolidation  report. Further, we obtained written responses from agency officials  regarding the steps taken to ensure the accuracy and reliability of their  cost savings data. As a result, we determined that the data were  sufficiently complete and reliable to report on agencies data center  consolidation cost-savings information.

For our third objective, we analyzed the August 2017 data center  optimization progress information of the 24 DCOI agencies. This progress  information was obtained from the IT Dashboard—an OMB public website  that provides information on federal agencies’ major IT investments. We  then compared the agencies’ optimization progress information against  OMB’s fiscal year 2018 optimization targets, as documented in its  August 2016 memorandum. Although OMB’s memorandum establishes  a single optimization target value for the server utilization and automated  monitoring metric, the IT Dashboard displays agencies’ progress for  tiered and non-tiered data centers separately. To report consistently with  OMB’s implementation memorandum, we combined the progress  information for tiered and non-tiered data centers into a single  assessment in this report.

To assess the reliability of agencies’ optimization progress information on  OMB’s IT Dashboard, we reviewed the information for errors or missing  data, such as progress information that was not available for certain  metrics. We also compared agencies’ optimization progress information  across multiple reporting quarters to identify any inconsistencies in  agencies’ reported progress. We discussed with staff from OMB’s Office  of the Federal Chief Information Officer any discrepancies or potential  errors identified to determine the causes.

In addition, we interviewed OMB officials to obtain additional information  regarding the steps taken to ensure the reliability of and validate the  optimization data on the Dashboard. Moreover, we obtained written  responses from agency officials regarding the steps taken to ensure the  accuracy and reliability of the reported optimization progress. We  discussed with agency officials any discrepancies or potential errors  identified during our reviews to determine the causes or request  additional information. We determined that the data were sufficiently  reliable to report on agencies’ optimization progress.

To assess the reliability of the DCOI strategic plans, we reviewed  agencies’ documentation to identify any missing data or errors. We also  compared the planned data center optimization milestones in agencies’  documentation against current optimization progress information obtained  from the IT Dashboard. In addition, we discussed with agency officials  any discrepancies or potential errors identified during our reviews of the  DCOI strategic plans to determine the causes or request additional  information. As a result of these efforts, we determined that the agencies’  strategic plan information was sufficiently reliable for reporting on plans to  meet or not meet OMB’s fiscal year 2018 optimization targets.

We conducted this performance audit from July 2017 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Comments from the Social Security Administration

Appendix III: Comments from the Department of Housing and Urban Development

Appendix IV: Comments from the Department of Transportation

Appendix V: Comments from the Department of the Interior

Appendix VI: GAO Contact and Staff Acknowledgments

In addition to the contact named above, individuals making contributions  to this report included Dave Hinchman (Assistant Director), Justin Booth  (Analyst-in-Charge), Chris Businsky, Nancy Glover, Linda Kochersberger,  and Jonathan Wall.


Coast Guard Health Records: Timely Acquisition of New System Is Critical to Overcoming Challenges with Paper Process

In 2010, the Coast Guard initiated an effort—known as IHiS—to replace its aging EHR system with a new system that was to modernize various health care services for its nearly 50,000 military members. However, in October 2015, the Coast Guard announced that the modernization project would be canceled.

GAO was asked to summarize its report that is being released today on the Coast Guard's actions related to its EHR modernization initiative. GAO's testimony specifically addresses Coast Guard's (1) reasons for deciding to terminate further IHiS development; (2) management and oversight actions for the discontinued project and whether lessons learned were identified; (3) current process for managing health records and the challenges it is encountering; and (4) plans for effectively implementing a new EHR system and the current status of its efforts.

In preparing the report on which this testimony is based, GAO reviewed IHiS project expenditures; analyzed key project management documentation; surveyed Coast Guard's Regional Managers and clinical staff; and interviewed key staff.

Financial, technical, schedule, and personnel risks led to the United States Coast Guard's (Coast Guard) decision to terminate the Integrated Health Information System (IHiS) project in 2015. According to the Coast Guard (a military service within the Department of Homeland Security), as of August 2017, $59.9 million was spent on the project over nearly 7 years and no equipment or software could be reused for future efforts. In addition, the Coast Guard could not fully demonstrate the project management actions taken for IHiS, lacked governance mechanisms, and did not document lessons learned for the failed project.

In the absence of an electronic health record (EHR) system, the Coast Guard currently relies on a predominately paper health record management process to document health care services. Currently, the Coast Guard's clinical staff perform various manual steps to process each paper health record. Coast Guard Regional Managers and clinic and sick bay administrators informed GAO of the many challenges encountered in returning to a paper process. These challenges include the inability for some clinics to adequately track vital information such as medications—potentially causing harm to members if they take medications that have dangerous interactions.

To help alleviate several of these challenges, the Coast Guard has developed alternative work-around processes. However, these alternative processes may not provide sustained solutions to overcoming these challenges.

In February 2016, the Coast Guard initiated the process for acquiring a new EHR system. As of November 2017, agency officials had conducted research and recommended a solution based on performance, risk, cost, and schedule advantages. However, 2 years after canceling IHiS and moving toward a predominately manual process, the agency has not yet made a final determination on this. Successfully and quickly implementing an EHR system is vital to overcoming the challenges the Coast Guard currently faces in managing paper health records. The expeditious implementation of such a system can significantly improve the quality and efficiency of care to the thousands of Coast Guard active duty and reserve members that receive health care.

In the report being released today, GAO is recommending that the Coast Guard (1) expeditiously and judiciously pursue a new EHR system, and in doing so (2) ensure key processes are implemented; (3) establish project governance boards; and (4) document lessons learned from the IHiS project. The Department of Homeland Security concurred with GAO's recommendations.

I am pleased to be here today to participate in your hearing on the United  States Coast Guard’s (Coast Guard) electronic health record (EHR)  system. As part of its mission, the Coast Guard is tasked with providing  health care to active duty and reserve members and ensuring the  availability of quality, cost-effective health care for all eligible  beneficiaries. To assist with this task, the agency has historically relied on  EHR systems to perform such functions as scheduling patient  appointments, documenting patient consults and referrals, and tracking  prescribed medications.

In 2010, the Coast Guard’s Health Safety and Work-Life Directorate  (HSWL) initiated an effort to replace the agency’s aging EHR systems  with a new system called the Integrated Health Information System (IHiS).  This system was to modernize various health care services and provide  additional functionality, such as a document management system, which  was previously lacking. The project consisted of various contracts with 25  different vendors and was estimated to cost approximately $56 million to  implement. However, in October 2015, the Coast Guard announced that  the modernization project would be canceled.

As requested, my statement summarizes our report that is being released  today on the Coast Guard’s actions related to its EHR modernization  initiative and its current health records management process.  Specifically, the statement addresses the Coast Guard’s (1) reasons for  deciding to terminate further IHiS development and how much it spent on  the project; (2) management and oversight actions for the discontinued  EHR modernization project and whether lessons learned were identified;  (3) current process for managing health records and the challenges it is  encountering; and (4) plans for effectively implementing a new EHR  system and the current status of its efforts.

Among other steps, in conducting our work, we reviewed IHiS project  expenditures; analyzed key project management documentation;  surveyed Regional Managers and clinical staff regarding challenges they  face in managing paper health records and any mitigation strategies; and  interviewed knowledgeable staff about the project. Our related report  includes a detailed explanation of the scope and methodology for our  work.

We conducted the work on which this testimony is based in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.

The Coast Guard Attributed IHiS Termination to Financial and Other Risks, after Spending Approximately $60 Million on the Project

According to the Director of HSWL, who was appointed to the position in  August 2015, financial, technical, schedule, and personnel risks led the  Coast Guard’s Executive Oversight Council to decide to terminate the  IHiS project in October 2015:

Financial risks. Internal investigations were initiated in January 2015  and May 2015 to determine whether the HSWL Directorate had  violated the Antideficiency Act by using incorrect funding sources and  incorrect fiscal year funds for the IHiS project. The Coast Guard  ordered project management and contractor staff to cease work on  IHiS until a determination was made regarding the antideficiency  violation.

Technical risks. IHiS lacked an independent security assessment  and full interface testing to ensure security and data integrity. In  addition, key functionality for the system, such as user verification,  had not been completed.

Schedule risks. The HSWL Director stated that she requested that  the Department of Defense’s (DOD) Defense Health Agency Solution  Delivery Information Technology (IT) team independently validate the  IHiS timelines and the status of the project in 2015 because of the  identified technical risks and concerns as to whether the system  would be ready to be piloted in the fall of 2015. According to the  Director, the Defense Health Agency team projected the timeline for  the first clinic implementation to be approximately 1 year later than  originally estimated due, in part, to incomplete interfaces and  workflows.

Personnel risks. Although HSWL staff had been managing the IHiS  project since it was initiated in 2010, Command, Control,  Communications, Computers, and Information Technology (C4&IT)  was directed to assume the oversight responsibilities for IHiS  implementation in May 2015. This action was due to concerns about  the project’s adherence to established governance processes raised  by the internal investigators looking into the potential Antideficiency  Act violations. By August 2015, the key HSWL project management  personnel that had overseen the project since 2010 had been  removed. As a result of the changes in staff, one vendor noted that it  was unclear as to who were the stakeholders, responsible parties,  and decision makers.

According to an analysis conducted by the Coast Guard, which included  obligations and expenditures from September 2010 to August 2017, the  agency had obligated approximately $67 million for the IHiS project and,  of that amount, had spent approximately $59.9 million at the time of its  cancelation. In addition, over 2 years after the project’s cancelation, the  Coast Guard continued to pay vendors. In this regard, it paid  approximately $6.6 million to vendors between November 2017 and  February 2018 to satisfy existing contractual obligations for services such  as leased equipment that was damaged or missing; software licensing  and support; a data storage center; and removal and shipment of  equipment. Further, according to staff in Coast Guard’s Office of Budget  and Programs, no equipment or software from the IHiS project could be  reused for future efforts.

The Coast Guard Could Not Demonstrate Effective Project Management, Lacked Governance Mechanisms, and Did Not Document Lessons Learned for the IHiS Project

The Coast Guard could not demonstrate that it effectively managed and  oversaw the IHiS project prior to its discontinuance, and did not document  and share valuable lessons learned from the failed project. Specifically,  although the Coast Guard was to follow its System Development Life  Cycle (SDLC) Practice Manual to guide its management and oversight of  the project, the agency could not provide complete evidence that it had  addressed 15 of the 30 SDLC practices we selected for evaluation. For  example, the Coast Guard could not demonstrate that it had conducted  IHiS system testing, although the agency granted an authority to operate  (ATO) and indicated in the ATO memorandum that the system had  undergone some form of testing. The Coast Guard’s SDLC specifies that  system testing is to take place prior to the issuance of an ATO.

Project team members provided inconsistent explanations regarding  whether or not documentation existed to demonstrate the actions taken to  manage and oversee the project. The absence of the various documents  and other artifacts that would support the required SDLC activities raises  doubts that the Coast Guard took the necessary and appropriate steps to  ensure effective management of the IHiS project.

Further, although the Coast Guard developed charters for various  governance boards to provide project oversight and direction, the boards  were not active and the Chief Information Officer (CIO) was not included  as a member of the boards. Taking steps to fully implement governance  boards that include the CIO will be important to the Coast Guard’s  oversight efforts in implementing a future EHR system and may decrease  the risk of IT project failure.

Lastly, although Coast Guard officials stated that lessons learned had  been identified throughout the process of developing IHiS, as of 2 years  after its cancelation, the agency had not documented and shared any  lessons learned from the project and did not have established plans for  doing so. Until the Coast Guard takes steps to document and share  identified lessons learned with individuals charged with developing and  acquiring its IT systems, opportunities to protect future systems against  the recurrence of mistakes that contributed to the failure of IHiS will likely  be missed.

The Coast Guard Is Managing Health Records Using a Predominately Paper Process, but Many Challenges Hinder Service Delivery

In the absence of an EHR system, the Coast Guard is relying on a  predominately paper health record management process to document  health care services for its nearly 50,000 military members. Currently, the  Coast Guard’s clinical staff perform various manual steps to process each  paper health record. For example, clinical staff schedule appointments for  patients using Microsoft Outlook’s calendar feature and provide the  patient with paper forms for completion upon his or her arrival. In addition,  clinical staff must handwrite clinical notes in the paper health record  during the appointment, as well as handwrite prescriptions, among other  manual processes.

In response to our survey, the 12 HSWL Regional Managers identified a  number of challenges that clinics and sick bays in their regions had  experienced in managing and maintaining paper health records. These  challenges were grouped into 16 categories. Further, the 120 clinic and  sick bay administrators that subsequently responded to a separate survey  reported varying degrees to which they viewed each category as  challenging. Figure 1 provides the clinic and sick bay respondents’  views of the top four challenges.

With regard to these top four challenges to managing and maintaining  paper health records, clinic and sick bay respondents offered the  following examples:  Incomplete records. Ninety-eight (82 percent) of the respondents  reported incomplete records as challenging. In this regard, 34 of the  survey respondents reported that not all records from the Coast Guard  legacy EHR systems were printed out and included in patients’ paper  health records as required before the systems were retired. Thus, they  had no way to ensure the patients’ paper records were complete.

Penmanship. Among the 91 (76 percent) survey respondents that  reported penmanship as challenging, several respondents noted that it is  difficult for staff to read illegible handwritten medical notes. This, in turn,  results in difficulty determining the accurate diagnosis, the required  prescription, or a referral.

Tracking medications. According to 89 (76 percent) of the respondents,  it is challenging to track medications without an EHR. For example, one  administrator stated that staff members rely heavily on patients to  remember what medications they are taking—potentially causing harm if  patients cannot remember what medications they are taking and the  medications have dangerous interactions.

Amount of time to manage records. According to 86 (72 percent) of the  respondents, managing paper health records is challenging and requires  more time for staff to complete and file paperwork. Several respondents  stated that the size of the paper health records has increased, resulting in  additional time required to review and file records.

The responding clinic and sickbay administrators described a range of  alternative work-around processes that they have developed to help  alleviate several of the challenges. Specifically, they reported having  developed additional forms, tracking methods, and alternative processes,  as well as having notified Coast Guard HSWL management of the  challenges they face. However, these alternative processes may not  provide sustained solutions to overcoming these challenges. Until Coast  Guard implements a new EHR solution, the challenges inherent in a  predominantly paper process will likely remain.

The Coast Guard Intends to Acquire a New EHR System, but Has Not Yet Chosen a Solution

The Coast Guard has begun taking steps to acquire a new EHR system  referred to as the Electronic Health Record Acquisition (eHRa). The  Coast Guard plans to manage and oversee the acquisition of eHRa  through its non-major acquisition process (NMAP), as described in its  Non-Major Acquisition Process (NMAP) Manual. NMAP requires formal  approval reviews at three discrete knowledge points called acquisition  decision events (ADE) and includes three phases to assess the readiness  and maturity of the acquisition.

The Coast Guard formally identified the need for a new EHR system on  February 1, 2016, and obtained approval for the first of three ADE’s on  February 13, 2016. It subsequently initiated market research activities by  collecting cost, schedule, and capabilities information from commercial  and government solution providers, including DOD and the Department of  Veterans Affairs.

The Coast Guard used the providers’ responses to develop an  alternatives analysis report that was completed in October 2017. The  report recommended a solution based on performance, risk, cost, and  schedule advantages. The report indicated that the Coast Guard plans to  use the results of the alternatives analysis to refine the acquisition  strategy, and to support the development of artifacts which are required to  successfully achieve the ADE-2 milestone. Staff within the Acquisitions  Directorate stated that they were also in the process of finalizing a life  cycle cost estimate and a project plan for eHRa—documents necessary  for ensuring that appropriate business decisions will be made regarding  eHRa’s logistics, affordability, and resources, among other things.

As of December 2017, the Coast Guard had not yet made a final  determination as to which option would be chosen as the solution for the  eHRa acquisition.

Implementation of Our Recommendations Should Better Position Coast Guard to Overcome Challenges with Paper Health Records

Our report that is being released today contains four recommendations to  the Coast Guard. Specifically, we recommend that the Coast Guard: expeditiously and judiciously pursue the acquisition of a new EHR ensure established processes required for the future acquisition or  development of an EHR are effectively implemented and adequately  documented; direct the Chief Information Officer and the Chief Acquisition Officer to  establish and fully implement project governance boards for the future  EHR effort that include the Chief Information Officer; and document any lessons learned from the discontinued IHiS project,  share them with the new project management team, and ensure  lessons learned are utilized for the future EHR effort.

The Department of Homeland Security concurred with our four  recommendations and identified actions being taken or planned to  implement them. If the Coast Guard fully and effectively implements our  recommendations, many of the challenges faced by its clinics and sick  bays and the thousands of Coast Guard members utilizing its health  services could be diminished.

In summary, given the numerous challenges inherent with managing and  maintaining paper health records, it will be important for the Coast Guard  to prioritize obtaining an EHR for its thousands of members. Until a  solution for its EHR system is chosen and successfully implemented, the  agency is likely to continue to face these challenges. In addition, ensuring  established project management and governance processes are  effective, as well as documenting and sharing lessons learned, will be  essential in avoiding past mistakes and helping to ensure a successful  implementation of a future EHR solution at the Coast Guard.

Chairman Hunter, Ranking Member Garamendi, and Members of the  Subcommittee, this concludes my prepared statement. I would be  pleased to respond to any questions that you may have.

If you or your staff have any questions about this testimony, please  contact David A. Powner, Director, Information Technology Management  Issues, at (202) 512-9286 or pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this testimony statement. GAO staff who made key  contributions to this statement are Nicole Jarvis (Assistant Director),  Ashfaq Huda (Analyst in Charge), Sharhonda Deloach, Rebecca Eyler,  Monica Perez-Nelson, and Scott Pettis.

This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.