Missile Defense: Air Force Report to Congress Included Information on the Capabilities, Operational Availability, and Funding Plan for Cobra Dane

First fielded in 1976 on Shemya Island in Alaska, the Cobra Dane radar faces growing sustainment challenges that DOD plans to address through modernization projects. Anticipating future needs, DOD began investing in new radar systems that share capabilities with Cobra Dane to support ballistic missile defense and space surveillance, including the LRDR (Alaska), the Space Fence (Marshall Islands), and the Pacific Radar (location to be determined).

The conference report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 included a provision that GAO review the Air Force's report to Congress on the operation and sustainment of Cobra Dane. This report identifies information included in the Air Force's report and describes additional information that GAO reviewed on (1) the capabilities of the Cobra Dane radar and other planned radars to meet DOD's mission requirements, (2) Cobra Dane's operational availability and the plan to mitigate the effect on those missions when Cobra Dane is not available, and (3) DOD's funding plan and project cost estimates for the operation and sustainment of Cobra Dane and its site at Shemya Island. GAO reviewed the Air Force report and related documentation, and interviewed relevant officials.

In its January 2018 report to Congress, the Air Force reported how the Cobra Dane radar and the Long Range Discrimination Radar (LRDR) have shared and unique capabilities to support ballistic missile defense and space surveillance missions. The report noted that the respective locations of both radar systems affect their ability to provide those capabilities. The Department of Defense (DOD) also has other radar investments—the Pacific Radar and the Space Fence, which, according to DOD officials, may reduce DOD's reliance on Cobra Dane to provide ballistic missile defense and space surveillance capabilities.

The Air Force's report to Congress noted that Cobra Dane met its requirement for operational availability, which refers to the percentage of time that the radar is able to meet its missions. GAO found that the Air Force has developed procedures to mitigate risks when Cobra Dane is not available. For example, U.S. Northern Command and Missile Defense Agency (MDA) officials stated that they can mitigate risks when Cobra Dane is not available by using the Sea-Based X-band radar to provide support for ballistic missile defense. The Air Force would face some limitations in its ability to conduct space surveillance if Cobra Dane were not available, as Cobra Dane tracks objects no other radar can track. However, MDA officials noted there are no plans to take Cobra Dane offline long enough to compromise space surveillance.

The Air Force and MDA plan to contribute total funding of $278.6 million for the operation and sustainment of Cobra Dane, according to their fiscal year 2019 budget plans. Specifically, the Air Force and MDA plan to share funding for the operation and maintenance of the Cobra Dane radar and for three modernization projects that make up their sustainment plan for the radar. Further, the Air Force report noted that the Air Force also plans to provide $140 million in funding for the sustainment and maintenance of operational access to Cobra Dane's site at Shemya Island. In addition, GAO found that the Air Force developed a total cost estimate for one project—known as the transmitter group replacement—but not for its other two projects. Air Force officials plan to complete cost estimates for those two projects in conjunction with their fiscal year 2020 budget submission.

The Cobra Dane radar faces growing sustainment challenges that the  Department of Defense (DOD) plans to address through modernization  projects. The Air Force fielded Cobra Dane in 1976 on Shemya Island,  Alaska. In subsequent years, Cobra Dane has undergone upgrades and  changes to its mission. From 1977 to 1994, the Air Force used Cobra  Dane to support a primary mission of intelligence gathering, and to  provide early warning of missile threats and conduct space surveillance  (tracking objects in space) as additional missions. In 1994, the Air Force  stopped using Cobra Dane to support missile warning. The Air Force also  ceased using Cobra Dane for space surveillance for 5 years, but  continued to operate the radar in a limited capacity to support intelligence  gathering. In 2004, the Missile Defense Agency (MDA) completed  upgrades to Cobra Dane so that it could contribute to a new ballistic  missile defense mission. Since MDA completed the upgrades, the Air  Force has used Cobra Dane to support ballistic missile defense and  space surveillance missions. In anticipating future needs, DOD began  investing in new radar systems that will provide varying mission  capabilities. However, DOD does not intend for any one of these systems  to be a complete replacement for Cobra Dane. Radar investments include  the Long Range Discrimination Radar (LRDR) at Clear Air Force Station  in Alaska; the Space Fence at Kwajalein Atoll, Marshall Islands; and the  Pacific Radar. See figure 1 for a photo of the Cobra Dane radar.

Congress has expressed interest in DOD’s plan to use Cobra Dane to  meet various mission requirements and how it would maximize the  radar’s reliability and minimize life-cycle costs. Specifically, the  conference report accompanying a bill for the National Defense  Authorization Act for Fiscal Year 2018 (“the Act”) included a provision that  the Air Force, in coordination with MDA and U.S. Northern Command,  submit a report on the Cobra Dane radar to the congressional defense  committees with its fiscal year 2019 budget request. The report was to  outline the capabilities of Cobra Dane and alternative radars, DOD’s plan  for the operation and sustainment of Cobra Dane, and the costs to  sustain Cobra Dane and its site at Shemya Island. The Air Force  submitted the report in January 2018.

The conference report accompanying the Act also included a provision  that we review the Air Force’s report to Congress. This report identifies  information included in the Air Force’s report and describes additional  information that we reviewed on (1) the capabilities of the Cobra Dane  radar and other planned radars to meet DOD’s mission requirements, (2)  Cobra Dane’s operational availability and the plan to mitigate the effect on  those missions when Cobra Dane is not operationally available, and (3)  DOD’s funding plan and project cost estimates related to the operation  and sustainment of Cobra Dane and its site at Shemya Island.

For all objectives, we reviewed the Air Force report to Congress and  identified information related to requirements outlined in the conference  report. We interviewed officials from Headquarters Air Force, Air Force  Space Command, the Air Force Life Cycle Management Center, MDA,  U.S. Northern Command, and U.S. Strategic Command to discuss  information contained within the Air Force report, and to identity other  information and documentation relevant to each of our three objectives.

For objective one, we also reviewed documentation that outlined the  capabilities of Cobra Dane and other radar investments. Our review  included MDA’s reports on the LRDR and its analysis of alternatives on  ballistic missile defense sensors. We also reviewed a U.S. Strategic  Command briefing for information on the capabilities of Cobra Dane and  alternative radar systems. We did not independently review the quality of  the technical materials or the process by which MDA or U.S. Strategic  Command assessed the capabilities of Cobra Dane or other radar  systems because doing so was beyond the scope of our review to  describe additional information on the capabilities of Cobra Dane and  other radar investments.

For objective two, we also reviewed documentation related to the  operational availability of Cobra Dane and the associated effects if Cobra  Dane is not available, including Air Force data on the operational  availability and space surveillance performance of Cobra Dane. We  asked cognizant Air Force officials questions about the reliability of their  operational availability data and reviewed relevant documentation on how  they calculate that data. We determined that we could use these data for  the limited purpose of describing the reported operational availability of  Cobra Dane compared to its requirement. We also reviewed U.S.  Northern Command’s analysis on the abilities of various radar systems to  track missile threats targeting U.S. locations.

For objective three, we also reviewed documentation related to DOD’s  plans to fund the operation and sustainment of Cobra Dane and to  identify project cost estimates. Specifically, we reviewed documents from  the Air Force and MDA, such as budget submissions, funding profiles,  and cost estimates. We did not independently assess the underlying  budget data or cost estimates from the documentation we reviewed  because doing so was beyond the scope of our review to describe  additional information on DOD’s funding plan and project cost estimates.

We conducted this performance audit from June 2018 to November 2018,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on audit objectives.

Cobra Dane and other radar systems can provide capabilities that  contribute to a range of missions, such as ballistic missile defense, space  surveillance, and intelligence-gathering missions. DOD uses Cobra Dane  and other radar systems to provide information over a short period of time  to ground-based interceptors so they can hit their targets. Such radar  systems contribute to ballistic missile defense by tracking incoming  missile threats, classifying the missile threat, and determining if a threat  was intercepted successfully. In addition, some radar systems can  provide discrimination capabilities, which allow for that radar to identify a  warhead when a missile threat deploys decoys at the same time. Radar  systems can also have the capability to contribute to a space surveillance  mission, which provides an awareness of space objects within or near the  Earth’s orbit and their movements, capabilities, and intent. Finally, radars  can also contribute intelligence-gathering capabilities. Each radar  system’s ability to contribute to various missions can be dependent on  that radar’s inherent capabilities and physical location.

See table 1 for a description of selected radar systems that can provide  some or all of these capabilities.

Various offices within the Air Force, in coordination with MDA, are  responsible for the operation and sustainment of the Cobra Dane radar.  Since 2013, Air Force Space Command has overseen the operation of  Cobra Dane, and contributes to the sustainment of Cobra Dane’s site at  Shemya Island. The Air Force Life Cycle Management Center has overall  responsibility of the sustainment of the Cobra Dane radar. In addition,  MDA works in coordination with the Air Force and combatant commands  to develop, test, and field ballistic missile defense assets. MDA also  shares funding with the Air Force to operate and sustain Cobra Dane.

U.S. Northern Command and U.S. Strategic Command define priorities  for the overall radar infrastructure and establish the various missions that  those radar systems are intended to meet. U.S. Northern Command  oversees the homeland ballistic missile defense mission, and establishes  operational objectives for radar systems operating in its region. U.S.  Northern Command officials told us that they are the end user for Cobra  Dane. U.S. Strategic Command has established a ballistic missile  defense and a space surveillance mission, both of which are supported  by Cobra Dane. Further, U.S. Strategic Command’s components  coordinate global missile defense and space operations planning.

Air Force Reported That Cobra Dane and LRDR Can Contribute to Various Missions, and We Found That Additional Radar Investments May Reduce Reliance on Cobra Dane

Air Force Reported That Cobra Dane and LRDR Contribute Both Shared and Unique Capabilities to Their Respective Missions

In its January 2018 report to Congress, the Air Force described how  Cobra Dane and LRDR can meet mission requirements through their  shared and unique capabilities, as well as how their locations affect their  ability to provide those capabilities for DOD’s ballistic missile defense  mission. MDA studies we reviewed found that locating LRDR at Clear Air  Force Station allows for operational advantages and cost savings.

Ballistic Missile Defense and Space Surveillance Missions

The Air Force included information in its report to Congress on the  ballistic missile defense capabilities of Cobra Dane and LRDR, and the  effects of each radar’s location on those capabilities. Specifically, the Air  Force report stated that both radars have the capabilities to track and  classify missile threats. However, the report incorrectly stated that both  radar systems have the inherent capability to determine if a missile threat  is successfully intercepted. MDA documentation that we reviewed shows  that Cobra Dane does not yet have this capability. When we shared our  finding with Air Force and MDA officials, they agreed that this reported  capability was incorrectly identified in the Air Force report to Congress.  MDA officials also told us that Cobra Dane could provide this capability in  the future if it implements software changes, but they are unlikely to do  this until calendar year 2025.

The Air Force report also noted that LRDR would have a unique  capability, once it is operational, to discriminate missile threats from any  deployed decoys. See table 2 for a summary of what the Air Force  reported for the ballistic missile defense capabilities of Cobra Dane and  LRDR.

In addition to identifying ballistic missile defense capabilities of each  radar, the Air Force report noted that both Cobra Dane and LRDR will  have the inherent capabilities to support space surveillance and  intelligence-gathering missions. DOD officials we spoke to confirmed that  they have plans to use those inherent capabilities to support these other  missions. For example, U.S. Strategic Command identified that DOD  needs Cobra Dane to support its space surveillance mission. Further, Air  Force and MDA officials told us that they use Cobra Dane to track small  objects that no other radar system can track. MDA officials told us that  LRDR could be used for space surveillance. However, Air Force and U.S.  Strategic Command officials stated that there are no plans to use LRDR’s  space surveillance capabilities as a replacement for Cobra Dane.  Additionally, Air Force officials told us that neither Cobra Dane nor LRDR  is required to support an intelligence-gathering mission.

The Air Force also included information in its report on how the locations  of Cobra Dane and LRDR affect their abilities to contribute to the ballistic  missile defense mission. For example, the Air Force reported that Cobra  Dane’s location at Shemya Island, Alaska, allows it to track missile  threats from North Korea earlier in their trajectories than LRDR would be  able to track at Clear Air Force Station, Alaska. This is consistent with an  MDA analysis that we reviewed that outlined additional advantages  provided by Cobra Dane’s location at Shemya Island. According to that  analysis, Cobra Dane can begin tracking missile threats approximately  210 seconds earlier than LRDR. Air Force officials told us that the  additional time to track missile threats allows the warfighter an earlier  opportunity to intercept a missile threat and deploy additional interceptors  if the first attempt fails. Further, the MDA analysis described a tracking  gap between the areas covered by LRDR—once it is operational at Clear  Air Force Station—and the two sets of AN/TPY-2 radars that are currently  located in Japan. Without Cobra Dane’s coverage of this gap, the  analysis found that the warfighter would have a more limited opportunity  to intercept a missile threat from North Korea. Figure 2 shows how Cobra  Dane covers a gap between the LRDR (once operational) and the two  AN/TPY-2 radars in Japan.

The Air Force report also noted that LRDR’s geographic location has its  own advantages in contributing to ballistic missile defense compared to  Cobra Dane’s location. For example, the Air Force report noted that  LRDR’s location would allow it to track missile threats later in their  trajectories beyond Cobra Dane’s coverage as those threats make their  way to the continental United States. We also found that MDA has  determined LRDR will have other advantages due to its location. For  example, an MDA analysis that we reviewed found that LRDR’s location  will allow for the radar system to contribute to ballistic missile defense  from North Korean and Iranian threats. Absent LRDR, this analysis  determined that there are no other radar systems that are located in a  position to provide the capability to discriminate missile threats and  determine if a threat was successfully intercepted.

In addition to what the Air Force reported, we found that DOD decided to  locate LRDR at Clear Air Force Station in Alaska after considering the  advantages and disadvantages of other locations. For example, MDA  completed studies that examined how LRDR could perform at various  locations in Alaska, and the cost-effectiveness of constructing and  sustaining the radar at those sites. In a June 2015 analysis, MDA  compared how LRDR could perform in discriminating missile threats  when co-locating it with Cobra Dane at Shemya Island or placing it at  Clear Air Force Station. MDA determined that LRDR could provide more  real-time discrimination information for missile threats targeting Alaska  and the continental United States if it constructed the radar at Clear Air  Force Station versus Shemya Island. Additionally, MDA identified in an  October 2016 study that the department could obtain operational  advantages and cost savings by constructing LRDR at Clear Air Force  Station, Alaska, when compared to constructing it at Shemya Island,  Alaska. Specifically, MDA determined that Clear Air Force Station could  provide better results for 11 of the 13 factors it reviewed compared to  Shemya Island. For example, MDA determined that locating LRDR at  Clear Air Force Station would result in lower costs and enhanced system  performance.

DOD Has Made Other Investments in Radar Systems That May Reduce Its Reliance on Cobra Dane to Meet Mission Requirements

According to DOD officials and documents we reviewed, other radar  investments may reduce the department’s reliance on Cobra Dane for  ballistic missile defense and space surveillance, given that U.S. Northern  Command identified it has a need for Cobra Dane after DOD begins  operating LRDR in fiscal year 2021. Specifically, the Pacific Radar and  Space Fence may reduce DOD’s reliance on Cobra Dane to support  ballistic missile defense and space surveillance, respectively.

Pacific Radar: According to DOD officials, the department may no  longer need Cobra Dane to meet the ballistic missile defense mission  after MDA fields a new radar in the Pacific region in fiscal year 2025.  MDA began developing the Pacific Radar to provide additional missile  threat tracking and discrimination capabilities. According to U.S.  Northern Command and MDA officials, the Pacific Radar may fill the  gap in tracking missile threats currently covered by Cobra Dane.

Space Fence: The Air Force has also determined it will no longer  have a requirement for Cobra Dane to provide space surveillance  once the Space Fence is fully operational. The Air Force plans for the  Space Fence to be operational in fiscal year 2019. According to a  U.S. Strategic Command briefing, the Space Fence will provide the  same capabilities as Cobra Dane. Air Force officials noted that they  want to continue relying on Cobra Dane for space surveillance when  the Space Fence is operational, as long as the radar is available and  used to contribute to ballistic missile defense.

Air Force Reported That Cobra Dane Generally Meets Its Requirements for Operational Availability, and We Found That the Air Force Can Mitigate Radar Downtime for Its Missions

In its January 2018 report to Congress, the Air Force noted that Cobra  Dane met its requirement for operational availability—i.e., the percentage  of time that the radar system is able to meet its ballistic missile defense  and space surveillance missions. Specifically, the Air Force report noted  that Cobra Dane had been available an average of 91 percent of the time  over a 2-year period (January 2016 through December 2017), which  exceeded the 90 percent requirement for operational availability.

Information that we reviewed from a more recent 2-year period (August  2016 through July 2018) showed that Cobra Dane’s 2-year average for  operational availability had declined to approximately 88 percent—below  the 90 percent requirement. Air Force officials stated that the decline in  the operational availability over the more recent two-year period was due  to a few instances where they needed to take Cobra Dane off-line for  extended periods of scheduled downtime (e.g., regular operations and  maintenance, calibration of instruments). Further, they noted that when  Cobra Dane is not operationally available, the reason is usually due to  scheduled downtime.

Officials also told us there was one instance of unscheduled downtime  (e.g., part or system failure) in that 2-year period which required  emergency maintenance on the radar’s mission control hardware. We  also reviewed Air Force data on the frequency of unscheduled downtime  between August 2016 and July 2018, which show that Cobra Dane is able  to contribute to its missions without unscheduled downtime 99.7 percent  of the time.

According to U.S. Northern Command and MDA officials, they can  mitigate the effect on the ballistic missile defense mission if they know far  enough in advance that Cobra Dane will not be operationally available— such as during scheduled downtime. Officials stated that they do this by  moving a transportable radar, known as the Sea-Based X-band radar, to  specific locations in the Pacific Ocean to provide additional tracking  coverage of missile threats. A U.S. Northern Command analysis that we  reviewed describes how DOD can deploy the Sea-Based X-band radar at  particular locations in the Pacific Ocean to supplement Cobra Dane.  This analysis found that U.S. Northern Command can lose the ability to  track some missile threat trajectories if Cobra Dane is not available and  the Sea-Based X-band radar is not deployed.

We also reviewed Air Force data on space surveillance, which shows that  the Air Force would face some limitations in its ability to complete its  space surveillance mission when Cobra Dane is not operationally  available. According to the data, Cobra Dane tracks 3,300 space objects  each day that cannot be tracked by any other radar system. Air Force  officials noted that when Cobra Dane is not operationally available for  space surveillance for short periods (less than 24 hours), they can  overcome that downtime without losing track of those unique objects.  However, officials told us that it would take six months to reacquire all of  the small space objects that Cobra Dane tracks, if they encounter any  significant scheduled or unscheduled downtime. MDA officials told us  there are no scheduled plans to take Cobra Dane down long enough to  compromise DOD’s ability to conduct space surveillance.

Air Force Reported That DOD Has Plans to Fund Cobra Dane and Its Site, and We Found That It Has Developed Cost Estimates for Some Projects

Air Force Reported the Funding for the Operation and Sustainment of Cobra Dane, and We Found That DOD Has Developed Cost Estimates for Some Modernization Projects

In its January 2018 report to Congress, the Air Force projected that the  Air Force and MDA would contribute total funding of $278.6 million based  on their fiscal year 2019 budget plans for the operation and sustainment  of Cobra Dane. According to the report, the Air Force and MDA plan to  share funding for the operation and maintenance of the Cobra Dane  radar, and for three modernization projects that make up their  sustainment plan for the radar. Table 3 outlines the plan for how the Air  Force and MDA will share funding for the operation and maintenance of  Cobra Dane.

In addition, the Air Force included information in its report on how the Air  Force and MDA plan to share funding to support Cobra Dane’s three  modernization projects. Specifically, the Air Force and MDA plan to  redesign parts for three sets of obsolete systems: (1) mission system  replacement; (2) traveling wave tubes; and (3) transmitter groups. The  Air Force has identified that it no longer has vendors that manufacture  some critical parts, and failure of any of the three systems could result in  Cobra Dane not being available to meet mission requirements. As such,  the Air Force determined that it could sustain these three systems more  effectively if they were redesigned. Table 4 summarizes the reported  funding for the three projects that make up the Cobra Dane sustainment  plan.

In addition to what the Air Force reported, we identified that the Air Force  developed a total cost estimate for its transmitter group replacement, but  not for its other two projects. For the other two projects, Air Force officials  stated that they plan to complete estimates for the total costs in  conjunction with their fiscal year 2020 budget submission. In August  2016, the Air Force estimated that the transmitter group replacement  would have a total cost of $91.2 million, but reported it would fund this  project at $94.0 million through fiscal year 2023 (see table 4). Air Force  officials plan to request the transfer of any unused funding to the other  projects once it completes the transmitter group project. The Air Force  also completed a partial cost estimate for the traveling wave tube  redesign—covering the redesign of the parts and replacement of 1 of 12  groups of parts—estimating that the first phase would cost $16.0 million.  Further, Air Force officials told us that they have not yet developed a total  cost estimate for the mission system replacement.

We also found that the Air Force and MDA expedited Cobra Dane’s  mission system replacement project, but Air Force officials told us they face challenges in expediting the other two projects without compromising  Cobra Dane’s operational availability. For the mission system  replacement, MDA requested additional funding in fiscal year 2018. Air  Force and MDA officials told us that the additional funding they received  allowed them to prioritize the mission system replacement and advance  its timeline earlier that year. Air Force officials stated that they explored  ways to expedite the two other projects: the traveling wave tubes and  transmitter groups. However, they stated that replacing too many parts at  the same time will result in their having to take Cobra Dane off-line for  longer periods of time. According to Air Force and MDA officials, they  may look for opportunities to expedite timeframes for their other two  projects as long as the amount of scheduled downtime is kept to  acceptable levels.

The Air Force Reported the Funding for the Operation and Sustainment of Shemya Island

In its report to Congress, the Air Force identified that it plans to provide  $140 million in funding for the sustainment and maintenance of  operational access to Cobra Dane’s site at Shemya Island based on its  fiscal year 2019 budget plans. According to the report, the Air Force is  solely responsible for funding all work related to the operation and  sustainment of Shemya Island, shared between two of its major  commands: Air Force Space Command and Pacific Air Forces. Table 5  summarizes the information the Air Force included in its report on how  funding will be shared for Shemya Island.

We also reviewed a support agreement between Air Force Space  Command and Pacific Air Forces that identifies how they will sustain the  site and the calculation for sharing costs. The agreement describes the  specific work to sustain the site, including maintaining the airfield, support  facilities, and communication infrastructure. Air Force officials told us that  they are constantly addressing challenges related to operational access  to the site at Shemya Island, but Air Force Space Command and Pacific  Air Forces work together to address those challenges.

We provided a draft of this report to DOD for review and comment. DOD  told us that it had no comments on the draft report.

We are sending copies of this report to the Secretary of Defense; the  Under Secretary of Defense for Acquisitions and Sustainment; the  Secretary of the Air Force; the Director of the Missile Defense Agency;  and the Commanders of U.S. Northern Command and U.S. Strategic  Command. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  Joe Kirschbaum at (202) 512-9971 or kirschbaumj@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made  contributions to the report are listed in Appendix I.

Appendix I: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Kevin O’Neill (Assistant Director),  Scott Bruckner, Vincent Buquicchio, Martin De Alteriis, Amie Lesser, and  Richard Powelson made key contributions to the report.

DOD plans to invest over $278 million to operate, maintain, and modernize its Cobra Dane radar system—which helps defend against incoming missiles and tracks space objects such as satellites and debris. These investments are needed because the system has been operating for over 40 years, and key components are obsolete.

The Air Force occasionally has to take Cobra Dane offline to address failing parts or for routine maintenance, but we found that DOD has a mobile radar system it can use when Cobra Dane is scheduled to be offline.

We also found that DOD is planning to invest in new radar systems that may reduce its reliance on Cobra Dane.


Managing for Results: Government-wide Actions Needed to Improve Agencies' Use of Performance Information in Decision Making

To reform the federal government and make it more efficient and effective, agencies need to use data about program performance. The benefit of collecting performance information is only fully realized when it is used by managers to make decisions aimed at improving results.

GAO was asked to review agencies' use of performance information. This report assesses, among other things, the extent to which: (1) 24 agencies' reported use of performance information and related leading practices has changed since 2013 and (2) the Executive Branch has taken actions to enhance the use of performance information.

To address the first objective, GAO analyzed results from its 2017 survey of federal managers, and compared them to 2013 results. The survey covered a stratified random sample of 4,395 managers from the 24 Chief Financial Officers Act agencies. The survey had a 67 percent response rate and results can be generalized to the population of managers government-wide and at each agency. For the second objective, GAO reviewed agency documents and interviewed staff from OMB and the PIC.

Agencies' reported use of performance information to make decisions, and leading practices that can promote such use, generally has not improved since GAO's last survey of federal managers in 2013. However, GAO's survey results continue to point to certain practices that could help agencies improve managers' use of performance information. For example, as shown in the table below, GAO's survey found that managers whose programs were subject to data-driven reviews (regular reviews used to assess progress on select agency goals) to a greater extent reported statistically significantly greater use of performance information to make decisions.

The Executive Branch has begun taking steps to improve the use of performance information within agencies and across the government. For example,

In the President's Management Agenda and government-wide reform plan, released in March and June 2018 respectively, the administration acknowledged the need to do more, and announced a goal, among other actions, to improve the use of data in federal decision making. However, the Office of Management and Budget (OMB) and others responsible for this goal have yet to fully develop action plans to hold agencies accountable for achieving it.

The Performance Improvement Council (PIC), which is chaired by OMB, has undertaken efforts to improve the use of performance information by, for example, creating a working group on agency performance reviews. But it has not yet taken a systematic approach to identify and share proven practices that led to, or challenges that may be hampering, increased use of performance information by managers. GAO's survey results identified agencies that may have insights into such practices and challenges.

More fully developing action plans for the new goal, and identifying and sharing proven practices and challenges, could help ensure the Executive Branch takes further steps to improve the use of performance information by managers within agencies and across the federal government.

To improve the use of performance information within agencies and across the federal government, GAO recommends that OMB work with (1) fellow goal leaders to more fully develop action plans for the new goal to improve the use of data and (2) the PIC to prioritize efforts to identify and share proven practices and challenges. OMB had no comments on this report.

The federal government is one of the world’s largest and most complex  entities. It oversees a broad array of programs central to protecting the  health, safety, and security of the American public. As the nation  confronts both new and long-standing challenges, the public expects the  government to be more transparent and accountable. This requires  federal agencies to focus on their performance and improving results. To  modernize the federal government and improve agencies’ abilities to  achieve results, the Office of Management and Budget (OMB) released  the President’s Management Agenda and its plan to reform and  reorganize the federal government. These initiatives propose a range of  reform efforts and priorities designed to address the complex and  interconnected challenges facing federal agencies.

To successfully implement reforms and improve their operations and  results, agencies will need to robustly manage their performance. This  involves not only measuring progress toward goals, but also using that  information to identify and correct problems, improve program  implementation, and make other important management and resource  allocation decisions. However, for more than two decades, our work has  highlighted weaknesses in the use of performance information in decision  making across the federal government. For example, in September  2017, we found that the use of performance information (i.e., data collected to measure progress toward agency goals) in various decision- making activities generally was unchanged or even declined government- wide, as reported by federal managers on a periodic survey we have  conducted since 1997. These persisting weaknesses could hinder  agencies’ abilities to achieve critical results.

Based on the aggregated, government-wide level survey results  presented in our September 2017 report, you asked us to review  individual agencies’ use of performance information in decision making.  This report assesses the extent to which:  the reported use of performance information and related leading  practices at 24 agencies has changed compared to our prior survey in  2013; being subject to data-driven reviews related to managers’ reported  use of performance information and leading practices; and  the Executive Branch has taken actions to enhance agencies’ use of  performance information in various decision-making activities.

To address the first and second objectives, we analyzed relevant results  from our 2017 survey of federal managers on organizational performance  and management issues. We administered the web-based survey to a  stratified random sample of 4,395 managers at 24 agencies between  November 2016 and March 2017. We received usable questionnaires  from about 67 percent of the eligible sample. The weighted response rate  at each agency generally ranged from 57 percent to 82 percent, except  for the Department of Justice, which had a weighted response rate of 36  percent. The maximum margin of error for the government-wide  estimates was less than or equal to 6 percentage points, and the  maximum margin of error for agency estimates was less than 20  percentage points. These results are generalizable to the population of  managers government-wide and at each individual agency. The 2017  results are comparable to those from other surveys we conducted in  1997, 2000, 2003, 2007, and 2013.

To further address the first objective, we conducted various statistical  analyses on selected survey questions related to the use of performance  information and practices that can promote such use. For the use of  performance information, we selected the same set of 11 survey  questions used in our last report on our 2013 survey that focused on  agency-level survey results. Using 2017 survey data, we conducted  statistical analyses and ensured that these 11 questions were still  positively correlated and no changes were necessary. As in that  September 2014 report, we updated our index to create a single score for  each agency to approximate its reported use of performance information  in decision making as of 2017. We also reviewed our 2017 survey and  identified 22 questions that reflect five management practices—identified  by our past work—that can promote the use of performance information in  decision making. Of those 22 questions, we selected 10 questions that  our analyses showed had statistically significant associations with greater  reported use of performance information.

For the index and each question described above, we calculated results  for each agency based on the 2017 survey results. We then determined if  each agency’s results were statistically significantly different from (1) their  prior results in 2013 and (2) the government-wide average results in  2017. We focus on statistically significant results throughout the report  because these indicate that observed relationships between variables  and differences between groups are likely to be valid, after accounting for  the effects of sampling and other sources of survey error. Because our  audit work was limited to analyzing and summarizing the results of our  2017 survey, we did not conduct any audit work at the agencies to  determine what may have caused any statistically significant changes.  Appendix I presents the relevant survey results for each of the 24  agencies.

To further address the second objective, we used 2017 survey results to  identify managers who reported they were familiar with their agency’s  data-driven reviews. For those who reported familiarity with the reviews,  we identified the extent managers reported that their programs had been  subject to those reviews. Based on the extent they were subject to  reviews, we assessed differences among managers’ (1) index scores and  (2) responses on the 10 questions that reflect leading practices that can  promote the use of performance information.

For the third objective, we identified and reviewed various resources (i.e.,  guidance, guides, and playbooks) developed by OMB and the  Performance Improvement Council (PIC) that could support agencies’ use  of performance information. We also reviewed our past work that  assessed Executive Branch activities to enhance the use of performance  information. Lastly, we reviewed documents on recent cross-agency  efforts to improve the use of data in federal decision making, and  interviewed OMB and PIC staff about any actions they have taken, or  planned to take, to further support the use of performance information  across the federal government. Appendix II provides additional details on  the objectives, scope, and methodology of this report.

We conducted this performance audit from October 2017 to September  2018 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Use of Performance Information in the Federal Government

Concerned that the federal government was more focused on program  activities and processes than the results to be achieved, Congress  passed the Government Performance and Results Act of 1993 (GPRA).  GPRA sought to focus federal agencies on performance by requiring  agencies to develop long-term and annual goals, and measure and report  on progress towards those goals annually. Based on our analyses of the  act’s implementation, we concluded in March 2004 that GPRA’s  requirements had laid a solid foundation for results-oriented  management. At that time, we found that performance planning and  measurement had slowly yet increasingly become a part of agencies’  cultures. For example, managers reported having significantly more  performance measures in 2003 than in 1997, when GPRA took effect  government-wide.

However, the benefit of collecting performance information is fully realized  only when that information is actually used by managers to make  decisions aimed at improving results. Although our 2003 survey found  greater reported availability of performance information than in 1997, it  also showed managers’ use of that information for various management  activities generally had remained unchanged. Based on those results,  and in response to a request from Congress, in September 2005, we  developed a framework intended to help agencies better incorporate  performance information into their decision making. As shown in figure  1, we identified  five leading practices that can promote the use of performance  information for policy and program decisions; and  four ways agency managers can use performance information to  make program decisions aimed at improving results.

Our September 2005 report also highlighted examples of how agencies  had used performance information to improve results. For example, we  described how the Department of Transportation’s National Highway  Traffic Safety Administration used performance information to identify,  develop, and share effective strategies that increased national safety belt  usage—which can decrease injuries and fatalities from traffic accidents— from 11 percent in 1985 to 80 percent in 2004.

Subsequently, the GPRA Modernization Act of 2010 (GPRAMA) was  enacted, which significantly expanded and enhanced the statutory  framework for federal performance management. The Senate  Committee on Homeland Security and Governmental Affairs report  accompanying the bill that would become GPRAMA stated that agencies  were not consistently using performance information to improve their  management and results. The report cited the results of our 2007  survey of federal managers. That survey continued to show little change  in managers’ use of performance information. The report further stated  that provisions in GPRAMA are intended to address those findings and  increase the use of performance information to improve performance and  results. For example, GPRAMA requires certain agencies to designate a  subset of their respective goals as their highest priorities—known as  agency priority goals—and to measure and assess progress toward those  goals at least quarterly through data-driven reviews.

Our recent work and surveys suggest that data-driven reviews are having  their intended effect. For example, in July 2015, we found that agencies  reported that their reviews had positive effects on progress toward  agency goals and efforts to improve the efficiency of operations, among  other things. In addition, for those managers who were familiar with their  agencies’ data-driven reviews, our 2013 and 2017 surveys showed that  the more managers viewed their programs as being subject to a review,  the more likely they were to report their agencies’ reviews were driving  results and conducted in line with our leading practices. Recognizing the  important role these reviews were playing in improving data-driven  decision making, our management agenda for the presidential and  congressional transition in 2017 included a key action to expand the use  of data-driven reviews beyond agency priority goals to other agency  goals.

More broadly, our recent surveys of federal managers have continued to  show that reported government-wide uses of performance information  generally have not changed or in some cases have declined. As we  found in September 2017, and as illustrated in figure 2, the 2017 update  to our index suggests that government-wide use of performance  information did not improve between 2013 and 2017. In addition, it is  statistically significantly lower relative to our 2007 survey, when we  created the index.

Moreover, in looking at the government-wide results on the 11 individual  survey questions that comprise the index, we found few statistically  significant changes in 2017 when compared to (1) our 2013 survey or (2)  the year each question was first introduced. For example, in comparing  2013 and 2017 results, two questions had results that were statistically  significantly different:

The percentage of managers who reported that employees who report  to them pay attention to their agency’s use of performance information  was statistically significantly higher (from 40 to 46 percent).

The percentage of managers who reported using performance  information to adopt new program approaches or change work  processes was statistically significantly lower (from 54 to 47 percent).

As we stated in our September 2017 report, the decline on the latter  question was of particular concern as agencies were developing plans to  improve their efficiency, effectiveness, and accountability, as called for by  an April 2017 memorandum from OMB.

The Administration’s Plans for Federal Performance Management

In early 2017, the administration announced several efforts intended to  improve government performance. OMB issued several memorandums  detailing the administration’s plans to improve government performance  by reorganizing the government, reducing the federal workforce, and  reducing federal agency burden. As part of the reorganization efforts,  OMB and agencies were to develop government-wide and agency reform  plans, respectively, designed to leverage various GPRAMA provisions.  For instance, the April 2017 memorandum mentioned above stated that  OMB intends to monitor implementation of the reforms using, among  other things, agency priority goals. While many agency-specific  organizational improvements were included in the President’s fiscal year  2019 budget, released in February 2018, OMB published additional  government-wide and agency reform proposals in June 2018.

The President’s Management Agenda (PMA), released in March 2018,  outlines a long-term vision for modernizing federal operations and  improving the ability of agencies to achieve outcomes. To address the  issues outlined in the PMA, the administration established a number of  cross-agency priority (CAP) goals. CAP goals, required by GPRAMA, are  to address issues in a limited number of policy areas requiring action  across multiple agencies, or management improvements that are needed  across the government.

The PMA highlights several root causes for the challenges the federal  government faces. Among them is that agencies do not consistently apply  data-driven decision-making practices. The PMA states that smarter use  of data and evidence is needed to orient decisions and accountability  around service and results. To that end, in March 2018, the administration  established the Leveraging Data as a Strategic Asset CAP goal to  improve the use of data in decision making to increase the federal  government’s effectiveness.

Federal Performance Management Leadership Roles and Responsibilities

Over the past 25 years, various organizations, roles, and responsibilities  have been created by executive action or in law to provide leadership in  federal performance management. At individual agencies and across the  federal government, these organizations and officials have key  responsibilities for improving performance, as outlined below.

OMB: At least every four years, OMB is to coordinate with other  agencies to develop CAP goals—such as the one described earlier on  leveraging data as an asset—to improve the performance and  management of the federal government. OMB is also required to  coordinate with agencies to develop annual federal government  performance plans to define, among other things, the level of  performance to be achieved toward the CAP goals. Following  GPRAMA’s enactment, OMB issued guidance for initial  implementation, as required by the act, and continues to provide  updated guidance in its annual Circular No. A-11, additional  memorandums, and other means.

Chief Operating Officer (COO): The deputy agency head, or  equivalent, is designated as the COO, with overall responsibility for  improving agency management and performance through, among  other things, the use of performance information.

President’s Management Council (PMC): The PMC is comprised of  OMB’s Deputy Director for Management and the COOs of major  departments and agencies, among other individuals. Its  responsibilities include improving overall executive branch  management and implementing the PMA.

Performance Improvement Officer (PIO): Agency heads designate  a senior executive as the PIO, who reports directly to the COO. The  PIO is responsible for assisting the head of the agency and COO to  ensure that agency goals are achieved through, among other things,  the use of performance information.

Performance Improvement Council (PIC): The PIC is charged with  assisting OMB to improve the performance of the federal  government. It is chaired by the Deputy Director for Management at  OMB and includes PIOs from each of the 24 Chief Financial Officers  Act agencies, as well as other PIOs and individuals designated by the  chair. Among its responsibilities, the PIC is to work to resolve  government-wide or cross-cutting performance issues, and facilitate  the exchange among agencies of practices that have led to  performance improvements.

Previously, the General Service Administration’s (GSA) Office of  Executive Councils provided analytical, management, and administrative  support for the PIC, the PMC, and other government-wide management  councils. In January 2018, the office was abolished and its functions,  staff, and authorities, along with those of the Unified Shared Services  Management Office, were reallocated to GSA’s newly created Shared  Solutions and Performance Improvement Office.

Agencies’ Use of Performance Information in Decision Making and Related Leading Practices Generally Has Not Improved

Reported Use of Performance Information in Decision Making Generally Has Not Improved at Individual Agencies Since 2013

As at the government-wide level—where, as described earlier, the use of  performance information did not change from 2013 to 2017—managers’  reported use of performance information at most agencies also did not  improve since 2013 (illustrated in figure 3).

At the agency level, 3 of the 24 agencies had statistically significant  changes in their index scores—1 increase (National Science Foundation)  and 2 decreases (Social Security Administration and the Office of  Personnel Management). Also, in 2017, 6 agencies had results that  were statistically significantly different—4 higher and 2 lower—than the  government-wide average (see sidebar).

Throughout the report, we highlight two different types of statistically  significant results—changes from our last survey in 2013 and differences  from the 2017 government-wide average. The former indicates when an  agency’s reported use of performance information or leading practices  has measurably improved or declined. The latter indicates when it is  statistically significantly higher or lower than the rest of government.  These results suggest agencies have taken actions that led to  improvements in their use of performance information. For example,  when a result is a statistically significant increase since 2013, as with the  National Science Foundation index score in 2017, this suggests that the  agency has adopted practices that led to a measurable increase in the  use of performance information by managers.

When a result is statistically significantly higher than the government-wide  average, like GSA’s 2017 index score, this suggests that the agency’s  use of performance information is among the highest results when  compared to the rest of government. These agencies could also have  insights into practices that led to relatively high levels of performance  information use. Finally, when a result is a statistically significant  decrease since 2013, as with the Social Security Administration’s index  score in 2017, or statistically significantly lower than the government-wide  average, like the Department of Homeland Security’s 2017 index score,  this suggests the agencies face challenges that are hampering their  ability to use performance information. Appendix III provides each  agency’s index scores from 2007, 2013, and 2017 to show changes  between survey years.

When we disaggregated the index and analyzed responses from the 11  questions that comprise the index—which could help pinpoint particular  actions that improved the use of performance information—we similarly  found relatively few changes in agencies’ recent results. Specifically, we  identified 16 instances where agency responses on individual questions  were statistically significantly different from 2013 to 2017—10 increases  and 6 decreases. This represents about 6 percent of the total possible  responses to the 11 survey questions from each of the agencies.

In addition, we found 12 instances where an agency’s result on a  question was statistically significantly higher (11) or lower (1) than the  government-wide average in 2017. For example, the percentage of Social  Security Administration (SSA) managers reporting that their peers use  performance information to share effective approaches was statistically  significantly higher than the government-wide average. Although SSA’s  index score had a statistically significant decline in 2017 compared to  2013, the agency’s index score remains relatively high, as it has in prior  years. The scope of our work has not allowed us to determine definitively  what factors caused the decline in SSA’s index score and whether the  decline is likely to continue, although its result on this particular question  may indicate a continued strength. Each agency’s results on the 11  questions that comprise the index are presented in appendix I. The  agencies’ respective statistically significant results are identified in figure  4.

While some agencies had statistically significant improvements on  individual questions, and could point to actions that led to improvements  in their use of performance information, these improvements should be  considered in relation to the range of agency results and the government- wide average. In figure 4, there are five agencies with statistically  significant increases on responses to individual questions, where those  results were not statistically significantly higher than the government-wide  average (see arrows without plus signs for the Departments of  Agriculture, Defense, and Justice; the Environmental Protection Agency;  and the National Science Foundation). While these represent  improvements, they should be considered in relation to the range of  agency results and the government-wide average (provided in detail in  the agency summaries in appendix I). For example, in 2017, the  percentage of managers at the Department of Agriculture who reported  that upper management use performance information to inform decisions  about program changes was statistically significantly higher than in 2013.  However, the department’s 2017 result (37 percent) was relatively lower  when compared to the maximum agency result on that question (60  percent). Appendix I presents the results on the index and the 11  questions that comprise it for each of the 24 agencies.

Individual Agencies’ Reported Use of Leading Practices Generally Remains Unchanged

When we compared government-wide and agency-level results on  selected survey questions that reflect practices that promote the use of  performance information, we found that results between 2013 and 2017  generally remained unchanged. As described earlier, there are 10 survey  questions that both reflect the five leading practices identified in our past  work and had statistically significant associations with higher index  scores. As shown in figure 5, government-wide results on 2 of the 10  questions were statistically significantly different, both increases, from  2013 to 2017. Despite these two increases, the overall results suggest  these practices are not widely followed government-wide. On most of the  10 questions, only about half (or fewer) of the managers reported their  agencies were following them to a “great” or “very great” extent.

When we analyzed agency-level responses to these 10 questions, we  also found relatively few changes in recent results. Specifically, our  analysis found 20 instances—16 increases and 4 decreases—where  agencies’ responses on individual questions were statistically significantly  different from 2013 to 2017. This represents about 8 percent of the total  possible responses to the 10 survey questions from each of the  agencies.

In addition, we found 10 instances where an agency’s result on a  question was statistically significantly higher (8) or lower (2) than the  government-wide average in 2017. Each agency’s results on these 10  questions are presented in appendix I, and the statistically significant  results are identified in figure 6.

Those agencies with results on individual questions that are either  statistically significantly higher than 2013, higher than the 2017  government-wide average, or both may have taken actions in line with our  leading practices for promoting the use of performance information. For  example, the National Science Foundation had both types of statistically  significant results on a question about having sufficient information on the  validity of their performance data. Here, the agency’s result increased 27  percentage points from 2013 to 2017. While the scope of our review does  not allow us to definitively determine the reasons for the National Science  Foundation’s higher results, they suggest the agency has taken recent  actions that greatly improved the availability and accessibility of  information on the validity of performance data. In both 2013 and 2017,  our analyses found this particular question to be the strongest predictor of  higher performance information use when we tested for associations  between the questions that reflect leading practices and our index.

Managers Whose Programs Were Subject to Data- Driven Reviews Reported Greater Use of Performance Information and Leading Practices

Our 2017 survey results show that managers who reported their  programs were subject to data-driven reviews also were more likely to  report using performance information in decision making to a greater  extent (see figure 7). For the 35 percent of managers who reported being  familiar with data-driven reviews, those who reported their programs had  been subject to data-driven reviews to a “great” or “very great” extent had  index scores that were statistically significantly higher than those whose  programs were subject to these reviews to a lesser extent.

Similarly, we found that being subject to data-driven reviews to a greater  extent was also related to greater reporting of agencies following  practices that can promote the use of performance information. As figure  8 shows, managers who reported their programs were subject to these  reviews to a “great” or “very great” extent more frequently reported that  their agencies followed the five leading practices that promote the use of  performance information, as measured by the 10 related survey questions  associated with higher scores on the index. For example, of the estimated  48 percent of managers who reported their programs were subject to  data-driven reviews to a “great” or “very great” extent, 72 percent also  reported that managers at their level (peers) effectively communicate  performance information on a routine basis to a “great” or “very great”  extent. Conversely, for the 24 percent of managers who reported their  programs were subject to data-driven reviews to a “small” or “no” extent,  only 30 percent reported that managers at their level do this to a “great”  or “very great” extent.

Opportunities Exist for the Executive Branch to Increase the Use of Performance Information within Agencies

Disparities Exist in the Use of Performance Information by Senior Agency Leaders and Managers at Lower Levels

Our past work has found that the Executive Branch has taken steps to  improve the use of performance information in decision making by senior  leaders at federal agencies. However, our survey results indicate those  steps have not led to similar improvements in use by managers at lower  levels. Through its guidance to implement GPRAMA, OMB developed a  framework for performance management in the federal government that  involves agencies setting goals and priorities, measuring performance,  and regularly reviewing and reporting on progress. This includes  expectations for how agency senior leaders should use performance  information to assess progress towards achieving agency priority goals  through data-driven reviews, and strategic objectives through strategic  reviews. For example, GPRAMA requires, and OMB’s guidance  reinforces, that data-driven reviews should involve the agency head,  Chief Operating Officer, Performance Improvement Officer, and other  senior officials responsible for leading efforts to achieve each goal.

OMB’s guidance also identifies ways in which agency leaders should use  the results of those reviews to inform various decision-making activities,  such as revising strategies, formulating budgets, and managing risks. Our  past work also found that agencies made progress in implementing these  reviews and using performance information. In July 2015, we found that  agencies generally were conducting their data-driven reviews in line with  GPRAMA requirements and our related leading practices, including that  agency leaders used the reviews to drive performance improvement. In  addition, in September 2017, we reported on selected agencies’  experiences in implementing strategic reviews and found that the reviews  helped direct leadership attention to progress on strategic objectives.

Despite those findings, our survey results continue to show that the  reported use of performance information by federal managers has  generally not improved, and actually declined at some agencies. This  could be because of the two different groups of agency officials covered  by our work. GPRAMA’s requirements, and the federal performance  management framework established by OMB’s guidance, apply at the  agency-wide level and generally involve senior leaders. Our past work  reviewing implementation of the act therefore focused on improvements  in the use of performance information by senior leaders at the agency- wide level. In contrast, our surveys covered random samples of mid- and  upper-level managers within those agencies, including at lower  organizational levels such as component agencies. Their responses  indicate that the use of performance information more broadly within  agencies—at lower organizational levels—generally has not improved  over time. The exception to this was managers whose programs were  subject to the data-driven reviews required by GPRAMA. As described  above, those managers were more likely to report greater use of  performance information in their agencies. This reinforces the value of the  processes and practices put in place by GPRAMA. Our survey results  suggest that limited actions have been taken to diffuse processes and  practices related to the use of performance information to lower levels  within federal agencies, where mid-level and senior managers make  decisions about managing programs and operations.

Although OMB staff agreed that diffusing processes and practices to  lower levels could lead to improved use of performance information, they  told us they have not directed agencies to do so for a few reasons. First,  OMB staff expressed concerns about potentially imposing a “one-size-fits- all” approach on agencies. They stated that agencies are best positioned  to improve their managers’ use of performance information, given their  individual and unique missions and cultures, and the environments in  which they operate. We agree that it makes sense for agencies to be able  to tailor their approaches for those reasons. OMB’s existing guidance  provides an overarching framework that recognizes the need for flexibility  and for agencies to tailor their approaches. Moreover, given the long- standing and cross-cutting nature of this challenge, a government-wide  approach also would provide a consistent focus on improving the use of  performance information more extensively within agencies.

OMB staff also told us that they believed it would go beyond their  mandate to direct agencies to extend GPRAMA requirements to lower  levels. GPRAMA requires OMB to provide guidance to agencies to  implement its requirements, which only apply at the agency-wide level. As  noted earlier, however, GPRAMA also requires OMB to develop cross- agency priority (CAP) goals to improve the performance and  management of the federal government. The President’s Management  Agenda established a CAP goal to leverage data as a strategic asset, in  part, to improve the use of data for decision making and accountability  throughout the federal government. This new CAP goal presents an  opportunity for OMB and agencies to identify actions to expand the use of  performance information in decision making throughout agencies.

Plan for New CAP Goal Does Not Yet Contain Required Elements for Successful Implementation

As of June 2018, the action plan for implementing the Leveraging Data as  a Strategic Asset CAP goal is limited. According to the President’s  Management Agenda and initial CAP goal action plan, the goal primarily  focuses on developing and implementing a long-term, enterprise-wide  federal data strategy to better govern and leverage the federal  government’s data. It is through this strategy that, among other things, the  administration intends to improve the use of data for decision making and  accountability. However, the strategy is under development and not  expected to be released until January 2019, with a related plan to  implement it expected in April 2019.

The existing action plan, released in March 2018 and updated in June  2018, does not yet include specific steps needed to improve the use of  data—including performance information—more extensively within  agencies. According to the action plan for the goal, potential actions  currently under consideration focus on establishing agency “learning agendas” that prioritize the development  and use of data and other evidence for decision-making; building agency capacity to use data and other evidence; and  improving the timeliness of performance information and other data,  and making that information available to decision makers and the  public.

Although developing learning agendas and building capacity could help  improve the use of performance information in agencies, improving  availability of data may be less effective. For example, as our past survey  results have shown, increasing the availability of performance information  has not resulted in corresponding increases in its use in decision  making.

We recognize that the CAP goal was created in March 2018.  Nonetheless, it is important that OMB and its fellow goal leaders develop  the action plan and related federal data strategy consistent with all key  requirements to better ensure successful implementation. The action plan  does not yet include complete information related to the following  GPRAMA requirements: performance goals that define the level of performance to be achieved  each year for the CAP goal;  the various federal agencies, organizations, programs, and other  activities that contribute to the CAP goal; performance measures to assess overall progress towards the goal  as well as the progress of each agency, program, and other activity  contributing to the goal; and  clearly defined quarterly targets.

Consistent with GPRAMA, Standards for Internal Control in the Federal  Government identifies information that agencies are required to include in  their plans to help ensure they achieve their goals. The standards state  that objectives—such as improving the use of data in decision making— should be clearly defined to enable the identification of risks. Objectives  are to be defined in specific terms so they can be understood at all levels  of the entity—in this case, government-wide as well as within individual  agencies. This involves defining what is to be achieved, who is to achieve  it, how it will be achieved, and the time frames for achievement.

Ensuring that future updates to the new CAP goal’s action plan includes  all required elements is particularly important, as our previous work has  found that some past CAP goal teams did not meet all planning and  reporting requirements. For example, in May 2016 we found that most  of the CAP goal teams we reviewed had not established targets for all  performance measures they were tracking. This limited the transparency  of their efforts and the ability to track progress toward established goals.  We recommended that OMB, working with the Performance Information  Council (PIC), report on actions that CAP goal teams are taking, or plan  to take, to develop such targets and performance measures. OMB staff  generally agreed and, in July 2017, told us they were working, where  possible, to assist the development of measures for CAP goals. However,  the recommendation has not been addressed and OMB staff said the  next opportunity to address it would be when the administration  established new CAP goals (which took place in March 2018). Following  the initial release of the new CAP goals, CAP goal teams are to more fully  develop the related action plans through quarterly updates. Given the  ongoing importance of meeting these planning and reporting  requirements, we will continue to monitor the status of actions to address  this recommendation as implementation of the new CAP goals proceeds.

Our Survey Results Identify Additional Opportunities for the PIC to Improve Federal Use of Performance Information

While the PIC, which is chaired by OMB, has contributed to efforts to  enhance the use of performance information, our survey results identify  additional opportunities to further those efforts. The PIC’s past efforts  have included hosting various working groups and learning events for  agency officials to provide performance management guidance, and  developing resources with relevant practices. For example, the PIC  created a working group focused on agency performance reviews, which  was used to share recommendations for how agencies can implement  reviews, along with a guide with practices for effectively implementing  strategic reviews. In January 2018, staff supporting the PIC joined with  staff from another GSA office to create a new group called Fed2Fed  Solutions. This group consults with agencies and provides tailored  support, such as data analysis and performance management training for  agency officials, to help them address specific challenges related to  organizational transformation, data-driven decision making, and other  management improvement efforts.

Our survey results identify useful information related to potential  promising practices and challenges that OMB and the PIC could use to  inform efforts to enhance the use of performance information more  extensively within agencies (e.g., at lower levels). As was previously  described, the PIC has responsibilities to (1) facilitate the exchange  among agencies of proven practices, and (2) work to resolve government- wide or cross-cutting performance issues, such as challenges. Our  analyses of 2017 survey results identified instances where agencies may  have found effective ways to enhance the use of performance information  by agency leaders and managers in decision making, as well as  instances where agencies (and their managers) face challenges in doing  so.

Specifically, based on analyses of our survey responses, we identified 14  agencies that may have insights into specific practices that led to recent  improvements in managers’ use of performance information, or ways that  they maintain relatively high levels of use by their managers when  compared to the rest of the government. Figure 9 summarizes the  agencies identified earlier in the report that had statistically significant  increases, or results higher than the government-wide average, on our  index or individual survey questions. As the figure shows, several  agencies had statistically significant results across all three sets of  analyses and therefore may have greater insights to offer: the General  Services Administration, National Aeronautics and Space Administration,  and the National Science Foundation.

In addition, our analyses identified nine agencies where results suggest  managers face challenges that have hampered their ability to use  performance information. Figure 10 summarizes the agencies identified  earlier in the report that had statistically significant decreases, or results  lower than the government-wide average, on our index or individual  survey questions. As the figure shows, the Office of Personnel  Management had statistically significant decreases in all three sets of  analyses.

Four agencies—the Departments of the Treasury and Veterans Affairs,  the Nuclear Regulatory Commission, and the Social Security  Administration—were common to both of the figures above. That is, they  had results that indicate they may have insights on some aspects of using  performance information and face challenges in other aspects. As was  mentioned earlier, to provide proper context, these results should be  considered in relation to the range of agency results and the government- wide average (provided in detail in the agency summaries in appendix I).

Given the prioritization of other activities, such as the recent creation of  the Fed2Fed Solutions program, the PIC has not yet undertaken a  systematic approach that could improve the use of performance  information by managers at lower levels within agencies. Such an  approach would involve identifying and sharing practices that have led to  improved use, as well as identifying common or cross-cutting challenges  that have hampered such use. The results of our analyses could help the  PIC do so, and in a more targeted manner. By identifying and sharing  proven practices, the PIC could further ensure that agency leaders and  managers are aware of effective or proven ways they can use  performance information to inform their decisions across the spectrum of  activities they manage within their agencies. Those proven practices also  may help agency leaders and managers resolve any identified  challenges.

Furthermore, in September 2017, we found that, for the estimated 35  percent of managers who reported familiarity with data-driven reviews,  the more they viewed their programs being subject to a review, the more  likely they were to report the reviews were driving results and were  conducted in line with our leading practices for using performance  information. Despite the reported benefits of and results achieved through  data-driven reviews, they were not necessarily widespread. As noted  above, GPRAMA requires agencies to conduct such reviews for agency  priority goals, which represent a small subset of goals, and they are  required at the departmental level. These reasons may explain why most  managers reported they were not familiar with the reviews.

As a result, we recommended that OMB should work with the PIC to  identify and share among agencies practices for expanding the use of  data-driven reviews. OMB staff agreed with our recommendation but have  yet to address it. In June 2018, OMB updated its annual guidance to  agencies to explicitly encourage them to expand data-driven reviews to  include other goals, priorities, and management areas as applicable to  improve organizational performance. However, as of June 2018, OMB  and the PIC have yet to take any steps to identify and share practices for  expanding the use of these reviews in line with our recommendation.  Given the additional analyses we conducted for this report—which show  that being subject to data-driven reviews is related to greater reported use  of performance information and leading practices that promote such  use—we continue to believe these further actions would help agencies  implement these reviews more extensively. We reiterate the importance  of the September 2017 recommendation and will continue to monitor  OMB’s progress to address it.

For more than 20 years, our work has highlighted weaknesses in the use  of performance information in federal decision making. While the  Executive Branch has taken some actions in recent years, such as  establishing a framework for performance management across the  federal government, our survey results underscore that more needs to be  done to improve the use of performance information more extensively  within agencies and government-wide. The President’s Management  Agenda and its related CAP goal to leverage data as a strategic asset  present an opportunity to do so, as it aims to improve data-driven  decision making. As OMB and its fellow goal leaders more fully develop  the action plan for achieving this goal, providing additional details for its  plans to improve data-driven decision making would help provide  assurance that it can be achieved.

As part of those initiatives, our survey results could provide a useful guide  for targeting efforts. Officials at each agency could use these results to  identify areas for additional analysis and potential actions that could help  improve the use of performance information across the agency and at  lower levels. Similarly, OMB and the PIC could use the results to identify  broader issues in need of government-wide attention. It will also be  important, however, for OMB and the PIC to go beyond this analysis and  work with agencies to identify and share proven practices for increasing  the use of performance information at lower levels within agencies, as  well as challenges that may be hampering agencies’ ability to do so.

We are making the following two recommendations to OMB:  The Director of OMB should direct the leaders of the Leveraging Data as  a Strategic Asset CAP Goal to ensure future updates to the action plan,  and the resulting federal data strategy, provide additional details on  improving the use of data, including performance information, more  extensively within federal agencies. The action plan should identify  performance goals; contributing agencies, organizations, programs, and  other activities; those responsible for leading implementation within these  contributors; planned actions; time frames; and means to assess  progress. (Recommendation 1)

The Director of OMB, in coordination with the PIC, should prioritize efforts  to identify and share among agencies proven practices for increasing,  and challenges that hamper, the use of performance information in  decision making more extensively within agencies. At a minimum, this  effort should involve the agencies that our survey suggests may offer  such insights. (Recommendation 2)

We provided a draft of this report to the Director of the Office of  Management and Budget for review and comment. We also provided a  draft of the report to the heads of each of the 24 federal agencies covered  by our survey.

OMB had no comments, and informed us that it would assess our  recommendations and consider how best to respond.

We are sending copies of this report to congressional requesters, the  Director of the Office of Management and Budget, the heads of each of  the 24 agencies, and other interested parties. This report will also be  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mcneilt@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix  IV.

Appendix I: Summaries of Agency Survey Results

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

10 Questions Related to 5 Leading Practices That Promote the Use of Performance Information

U.S. Agency for International Development

National Aeronautics and Space Administration

Appendix II: Objectives, Scope, and Methodology

This report responds to a request that we analyze agency-level results  from our 2017 survey of federal managers at the 24 agencies covered by  the Chief Financial Officers (CFO) Act of 1990, as amended, to determine  the extent agencies are using performance information. This report  assesses the extent to which:  1.  the reported use of performance information and related leading  practices at 24 agencies has changed compared to our prior survey in  2013;  2.  being subject to data-driven reviews related to managers’ reported  use of performance information and leading practices; and  3.  the Executive Branch has taken actions to enhance agencies’ use of  performance information in various decision-making activities.

From November 2016 through March 2017, we administered our online  survey to a stratified random sample of 4,395 individuals from a  population of 153,779 mid- and upper-level civilian managers and  supervisors at the 24 CFO Act agencies. The management levels  covered general schedule (GS) or equivalent schedules at levels  comparable to GS-13 through GS-15, and career Senior Executive  Service (SES) or equivalent. We obtained the sample from the Office of  Personnel Management’s Enterprise Human Resources Integration  database as of September 30, 2015—the most recent fiscal year data  available at the time. The sample was stratified by agency and whether  the manager or supervisor was a member of the SES. To help determine  the reliability and accuracy of the database elements used to draw our  sample of federal managers for the 2017 survey, we checked the data for  reasonableness and the presence of any obvious or potential errors in  accuracy and completeness and reviewed our past analyses of the  reliability of this database. We concluded in our September 2017 report  that the data used to draw our sample were sufficiently reliable for the  purpose of the survey. For the 2017 survey, we received usable  questionnaires from about 67 percent of the eligible sample. The  weighted response rate at each agency generally ranged from 57 percent  to 82 percent, except the Department of Justice, which had a weighted  response rate of 36 percent. The overall survey results are generalizable  to the population of managers government-wide and at each individual  agency.

To assess the potential bias from agencies with lower response rates, we  conducted a nonresponse bias analysis using information from the survey  and sampling frame as available. The analysis confirmed discrepancies in  the tendency to respond to the survey related to agency and SES status.  The analysis also revealed some differences in response propensity by  age and GS level; however, the direction and magnitude of the  differences on these factors were not consistent across agencies or  strata. Our data may be subject to bias from unmeasured sources for  which we cannot control. Results, and in particular estimates from  agencies with low response rates such as the Department of Justice,  should be interpreted with caution. However, the survey’s results are  comparable to five previous surveys we conducted in 1997, 2000, 2003,  2007, and 2013.

To address the first objective, we used data from our 2017 survey to  update agency scores on our use of performance information index.   This index, which was last updated using data from our 2013 survey,  averages managers’ responses on 11 questions related to the use of  performance information for various management activities and decision  making. Using 2017 survey data, we conducted statistical analyses to  ensure these 11 questions were still positively correlated. That analysis  confirmed that no negative correlations existed and therefore no changes  to the index were needed. Figure 11 shows the questions that comprise  the index.

After calculating agency index scores for 2017, we compared them to  previous results from 2007 and 2013, and to the government-wide  average for 2017, to identify any statistically significant differences. We  focus on statistically significant results because these indicate that  observed relationships between variables and differences between  groups are likely to be valid, after accounting for the effects of sampling  and other sources of survey error. For each of the 11 questions that  comprise the index, we identified individual agency results, excluding  missing and no basis to judge responses, and determined when they  were statistically significantly different from (1) the agency’s results on the  same question in 2013, or (2) the government-wide average results on  the question in 2017. In this report, we analyzed and summarized the  results of our 2017 survey of federal managers. Due to the limited scope  of the engagement, we did not conduct additional audit work to determine  what may have caused statistically significant changes between our 2017  and past survey results.

To further address this objective we completed several statistical  analyses that allowed us to assess the association between the index and  22 survey questions that we determined relate to leading practices we  previously found promote the use of performance information. See figure  12 for the 22 specific questions related to these five practices that we  included in the analysis.

When we individually tested these 22 survey questions (bivariate  regression), we found that each was statistically significantly and  positively related to the index in 2017. This means that each question,  when tested in isolation from other factors, was associated with higher  scores on the index. However, when all 22 questions were tested  together (multivariate regression), we found that 5 questions continued to  be positively and significantly associated with the index in 2017, after  controlling for other factors.

To conduct this multivariate analysis, we began with a base model that  treated differences in managers’ views of agency performance  management use as a function of the agency where they worked. We  found, however, that a model based on agency alone had little predictive  power (R-squared of 0.04). We next examined whether managers’  responses to these questions reflecting practices that promote the use of  performance information related to their perceptions of agency use of  performance information, independent of agency. The results of this  analysis are presented in table 1 below. Each coefficient reflects the  increase in our index associated with a one-unit increase in the value of a  particular survey question.

Our final multivariate regression model had an R-squared of 0.67,  suggesting that the variables in this model explain approximately 67  percent of the variation in the use index. We also tested this model  controlling for whether a respondent was a member of the SES and found  similar results.

As shown above in table 1, five questions related to three of the leading  practices that promote agencies’ use of performance information were  statistically significant in 2017. These results suggest that, when  controlling for other factors, certain specific efforts to increase agency use  of performance information—such as providing information on the validity  of performance data—may have a higher return and lead to higher index  scores.

With respect to aligning agency-wide goals, objectives, and  measures, we found that each increase in terms of the extent to which  individuals felt that managers aligned performance measures with  agencywide goals and objectives was associated with a 0.08 increase  in their score on the use index.

In terms of improving the usefulness of performance information, we  found that having information on the validity of performance data for  decision making was the strongest predictor in our model (0.18). As  measured here, taking steps to ensure the performance information is  useful and appropriate was associated with almost as large a change  in a managers’ index score (0.16).

In terms of developing agency capacity to use performance  information, we found that having sufficient analytical tools to collect,  analyze, and use performance information (0.07), and providing or  paying for training that would help link their programs to achievement  of agency strategic goals (0.10), were also statistically significantly  related to a manager’s reported use of performance information.

When we combined these results with what we previously found through  a similar analysis of 2013 survey results in September 2014, we identified  10 questions that have had a statistically significant association with  higher index scores. This reinforces the importance of the five leading  practices to promote the use of performance information. For each of  these questions, which are outlined in figure 13 below, we determined  when agency results were statistically significantly different from 2013  results or the 2017 government-wide average.

For the second objective, we examined, based on the extent they  responded their programs had been subject to agency data-driven  reviews, differences in managers’ use index scores and responses on  questions related to practices that promote the use of performance  information. We grouped managers based on the extent they reported  their programs had been subject to these reviews, from “no extent”  through “very great extent.” We then calculated the average index scores  for the managers in each of those five categories.

We also examined differences in how managers responded to the 10  questions reflecting practices that can promote the use of performance  information based on the extent they reported their programs had been  subject to data-driven reviews. We grouped managers into three  categories based on the extent they reported their programs had been  subject to these reviews (no-small extent, moderate extent, great-very  great extent). We then compared how these groups responded to the ten  questions.

For the third objective, we reviewed our past work that assessed Executive Branch activities to enhance  the use of performance information;  various resources (i.e., guidance, guides, and playbooks) developed  by the Office of Management and Budget (OMB) and the Performance  Improvement Council (PIC) that could support agencies’ use of  performance information; and  the President’s Management Agenda, and related materials with  information on cross-agency efforts to improve the use of data in  federal decision making.

Lastly, for the third objective we also interviewed OMB and PIC staff  about any actions they have taken, or planned to take, to further support  the use of performance information across the federal government.

We conducted this performance audit from October 2017 to September  2018 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix III: Comparison of 2007, 2013, and 2017 Agency Use of Performance Information Index Scores

Appendix IV: GAO Contact and Staff Acknowledgments

In addition to the above contact, Benjamin T. Licht (Assistant Director)  and Adam Miles (Analyst-in-Charge) supervised this review and the  development of the resulting report. Arpita Chattopadhyay, Caitlin Cusati,  Meredith Moles, Dae Park, Amanda Prichard, Steven Putansu, Alan  Rozzi, Shane Spencer, and Khristi Wilkins also made key contributions.  Robert Robinson developed the graphics for this report. Alexandra  Edwards, Jeff DeMarco, Mark Kehoe, Ulyana Panchishin, and Daniel  Webb verified the information presented in this report.

Results of the Periodic Surveys on Organizational Performance and  Management Issues  Managing for Results: Further Progress Made in Implementing the GPRA  Modernization Act, but Additional Actions Needed to Address Pressing  Governance Challenges. GAO-17-775. Washington, D.C.: September 29,  2017.

Supplemental Material for GAO-17-775: 2017 Survey of Federal  Managers on Organizational Performance and Management Issues.  GAO-17-776SP. Washington, D.C.: September 29, 2017.

Program Evaluation: Annual Agency-wide Plans Could Enhance  Leadership Support for Program Evaluations. GAO-17-743. Washington,  D.C.: September 29, 2017.

Managing for Results: Agencies’ Trends in the Use of Performance  Information to Make Decisions. GAO-14-747. Washington, D.C.:  September 26, 2014.

Managing for Results: Executive Branch Should More Fully Implement  the GPRA Modernization Act to Address Pressing Governance  Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Managing for Results: 2013 Federal Managers Survey on Organizational  Performance and Management Issues, an E-supplement to GAO-13-518.  GAO-13-519SP. Washington, D.C.: June 26, 2013.

Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation  in Program Management and Policy Making. GAO-13-570. Washington,  D.C.: June 26, 2013.

Government Performance: Lessons Learned for the Next Administration  on Using Performance Information to Improve Results. GAO-08-1026T.  Washington, D.C.: July 24, 2008.

Government Performance: 2007 Federal Managers Survey on  Performance and Management Issues, an E-supplement to  GAO-08-1026T. GAO-08-1036SP. Washington, D.C.: July 24, 2008.

Results-Oriented Government: GPRA Has Established a Solid  Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.:  March 10, 2004.

Managing for Results: Federal Managers’ Views on Key Management  Issues Vary Widely Across Agencies. GAO-01-592. Washington, D.C.:  May 25, 2001.

Managing for Results: Federal Managers’ Views Show Need for Ensuring  Top Leadership Skills.GAO-01-127. Washington, D.C.: October 20, 2000.

The Government Performance and Results Act: 1997 Governmentwide  Implementation Will Be Uneven. GAO/GGD-97-109. Washington, D.C.:  June 2, 1997.

Information about whether and how well agencies are reaching their goals is essential to making the federal government more effective. This only works, however, when federal managers use that data to make decisions.

Despite various efforts aimed at increasing managers’ use of data in decision making, our work has found little change over the past 20 years. The Office of Management and Budget has begun taking steps to address this issue, and our work identified additional practices—such as data-driven reviews—that can lead to increased use of data.

We recommended that OMB do more to further improve managers' use of performance information.