Managing for Results: Further Progress Made in Implementing the GPRA Modernization Act, but Additional Actions Needed to Address Pressing Governance Challenges

Full implementation of GPRAMA could facilitate efforts to reform the federal government and make it more effective. GPRAMA includes a provision for GAO to review the act's implementation. This report assesses how GPRAMA implementation has affected the federal government's progress in resolving key governance challenges in (1) addressing cross-cutting issues, (2) ensuring performance information is useful and used, (3) aligning daily operations with results, and (4) building a more transparent and open government.

To address these objectives, GAO reviewed statutory requirements, OMB guidance, and GAO's recent work related to GPRAMA implementation and the key governance challenges. GAO also interviewed OMB staff and surveyed a stratified random sample of 4,395 federal managers from 24 agencies on various performance and management topics. With a 67 percent response rate, the survey results are generalizable to the government-wide population of managers.

The Office of Management and Budget (OMB) and agencies have made some progress in more fully implementing the GPRA Modernization Act (GPRAMA), but GAO's work and 2017 survey of federal managers highlight numerous areas where improvements are needed.

Cross-cutting issues: Various GPRAMA provisions are aimed at addressing cross-cutting issues, such as cross-agency and agency priority goals and related data-driven reviews of progress towards those goals. To ensure alignment with the current administration's priorities, OMB's 2017 guidance removed the priority status of those goals, which stopped quarterly data-driven reviews and related public progress reports until new goals are published. OMB plans to resume implementation of these provisions in February 2018. GPRAMA also requires OMB and agencies to implement an inventory of federal programs, which could help decision makers better identify and manage fragmentation, overlap, and duplication. OMB and agencies implemented the inventory once, in May 2013. In October 2014, GAO found several issues limited the usefulness of that inventory. Since then, OMB has postponed updating the inventory, citing among other reasons the passage of subsequent laws. OMB has yet to develop a systematic approach for resuming implementation of the inventory and specific time frames for doing so. A systematic approach to developing the inventory could help ensure it provides useful information for decision makers and the public.

Performance information: Survey results show federal managers generally reported no improvements in their use of performance information in decision making for various management activities, or practices that can enhance such use, since GAO's 2013 survey. For example, the use of performance information to streamline programs to reduce duplicative activities (an estimated 33 percent in 2017) is statistically significantly lower relative to 2013 (44 percent). In contrast, managers who were familiar with and whose programs were subject to quarterly data-driven reviews reported that those reviews were used to make progress toward agency priority goals. Identifying and sharing practices to expand the use of such reviews—for other performance goals and at lower levels within agencies—could lead to increased use of performance information.

Daily operations: Agencies have made progress in developing results-oriented cultures but need to take additional actions. GAO's past work found that high-performing organizations use performance management systems to help individuals connect their daily activities to organizational goals. In 2017, about half of federal managers reported using performance information when setting expectations with employees (no change from GAO's last survey in 2013).

Transparent and open government: GAO's past work identified a number of needed improvements to Performance.gov, the central government-wide website required by GPRAMA. The site is to provide quarterly updates on priority goals in effect through September 2017, but those updates stopped in December 2016. According to OMB, the existing information for cross-agency priority goals is the final update, and agencies should publish final updates on their priority goals in annual performance reports. Performance.gov does not provide users with this information, thereby limiting the transparency and accessibility of those results.

In addition to following through on plans to resume implementation of key GPRAMA provisions, GAO recommends that OMB (1) consider a systematic approach to developing the program inventory, (2) revise guidance to provide specific time frames for inventory implementation, (3) identify and share practices for expanding the use of data-driven reviews, and (4) update Performance.gov to explain that reporting on priority goals was suspended and provide the location of final progress updates. OMB staff agreed with these recommendations.

The performance planning and reporting framework originally put into  place by the Government Performance Results Act of 1993 (GPRA), and  significantly enhanced by the GPRA Modernization Act of 2010  (GPRAMA), provides important tools that can help decision makers  address challenges facing the federal government. Full and effective  implementation of GPRAMA could facilitate efforts to reform the federal  government and make it more efficient, effective, and accountable. In  April 2017, the Office of Management and Budget (OMB) announced that  agencies are to develop and submit to OMB reform plans by September  2017, and OMB will develop crosscutting proposals that are to leverage  many of GPRAMA’s “performance tracking and accountability” tools.  OMB is to work with agencies to finalize agency reform plans and release  a final government-wide plan as part of the President’s fiscal year 2019  budget request.

The federal government faces a number of significant budget,  management, and performance challenges as it seeks to achieve diverse  and complex results. For example, since 2011, our series of annual  reports has identified 724 actions for Congress or executive branch  agencies to address fragmentation, overlap, and duplication; achieve  other cost savings; or enhance revenue in 249 different areas. In  addition, weaknesses in management capacity, both government-wide  and in individual agencies, impair efficient and effective government  operations. In the latest update to our High-Risk List, we identified 34  areas that need broad-based transformation or are vulnerable to fraud,  waste, abuse, or mismanagement. Addressing these challenges will  require tough choices in setting priorities and reforming programs and  management practices.

GPRAMA includes a statutory provision for us to periodically evaluate and  report on (1) how implementation of the act is affecting performance  management at the 24 major departments and agencies subject to the  Chief Financial Officers (CFO) Act of 1990, as amended, including  whether performance management is being used to improve the  efficiency and effectiveness of agency programs; and (2) crosscutting  goal implementation. Since 2012, we have issued over 30 products in  response to this provision; this is the third summary report. This report  assesses how implementation of GPRAMA has affected the federal  government’s progress in resolving key governance challenges in (1)  addressing crosscutting issues, (2) ensuring performance information is  useful and used in decision making, (3) aligning daily operations with  results, and (4) building a more transparent and open government.

We reviewed relevant statutory requirements, related OMB guidance, and  our recent work related to GPRAMA implementation and the four key  governance challenges included in our reporting objectives. Since our last  summary report in September 2015, we examined various aspects of  GPRAMA implementation in 12 products that covered 34 agencies,  including the 24 CFO Act agencies. (See figure 1.) We also interviewed  OMB and Performance Improvement Council (PIC) staff to obtain (1) their  perspectives on GPRAMA implementation and progress on the four  governance challenges and (2) updates on the status of our past  recommendations.

To supplement this review, we administered and analyzed the results of  our periodic survey of federal managers on organizational performance  and management issues. We surveyed a stratified random sample of  4,395 individuals from a population of 153,779 mid-level and upper-level  civilian managers and supervisors at the 24 CFO Act agencies. We  obtained the sample from the Office of Personnel Management’s  Enterprise Human Resources Integration (EHRI) database as of  September 30, 2015, which was the most recent fiscal year data available  at the time. We administered the web-based survey between November  2016 and March 2017. The overall survey results are generalizable to the  population of managers government-wide. The survey’s results are  comparable to other surveys we conducted in 1997, 2000, 2003, 2007,  and 2013.

Concurrently with this report, we are issuing online supplemental material  that shows responses to all survey items at the government-wide and  individual agency levels. For the 2017 survey, we received usable  questionnaires from about 67 percent of the eligible sample. The  weighted response rate at each agency generally ranged from 57 percent  to 82 percent, except the Department of Justice, which had a weighted  response rate of 36 percent.

Since each sample could have provided different estimates, we express  our confidence in the precision of our particular sample’s results as a 95  percent confidence interval. This is the interval that would contain the  actual population value for 95 percent of the samples we could have  drawn. The percentage estimates presented in this report based on our  sample for the 2017 survey have 95 percent confidence intervals within  plus or minus 5.5 percentage points of the estimate itself, unless  otherwise noted. We also note in this report when we are 95 percent  confident that changes from 1997 or 2013 relative to 2017 are statistically  significant. The supplemental material also shows the percentage  estimates and associated 95 percent confidence intervals for each item  for each agency and government-wide.

To help determine the reliability and accuracy of the EHRI database  elements used to draw our sample of federal managers for the 2017  survey, we checked the data for reasonableness and the presence of any  obvious or potential errors in accuracy and completeness and reviewed  our past analyses of the reliability of this database. We believe the data  used to draw our sample are sufficiently reliable for the purpose of the  survey. Appendix I provides additional information about our objectives,  scope, and methodology.

We conducted this performance audit from January 2016 to September  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

GPRAMA significantly enhances GPRA, the centerpiece of a statutory  framework that Congress put in place during the 1990s to help resolve  longstanding performance and management problems in the federal  government and provide greater accountability for results. Congress  passed GPRAMA in 2010 to address a number of persistent federal  performance challenges, including focusing attention on crosscutting  issues and enhancing the use and usefulness of performance  information.

OMB and agencies are to establish various government-wide and  agency-specific performance goals, in line with GPRAMA requirements or  OMB guidance. These include the following:

Cross-agency priority (CAP) goals: CAP goals are crosscutting and  include outcome-oriented goals covering a limited number of policy  areas as well as goals for management improvements needed across  the government. OMB is to coordinate with agencies to establish CAP  goals at least every 4 years. OMB is also required to coordinate with  agencies to develop annual federal government performance plans to,  among other things, define the level of performance to be achieved  toward the CAP goals.

Strategic objectives: A strategic objective is the outcome or impact  the agency is intending to achieve through its various programs and  initiatives. Agencies establish strategic objectives in their strategic  plans and may update the objectives during the annual update of  performance plans.

Agency priority goals (APG): At the agency level, every 2 years,  GPRAMA requires that the heads of certain agencies, in consultation  with OMB, identify a subset of agency performance goals as APGs.  These goals are to reflect the agencies’ highest priorities. They should  be informed by the CAP goals as well as consultations with relevant  congressional committees and other interested parties.

In a schedule established by GPRAMA, OMB and agencies are to  develop and publish new CAP goals, APGs, and strategic plans (with  updated strategic objectives) in February 2018.

GPRAMA and related OMB guidance require agencies to regularly  assess their progress in achieving goals and objectives through  performance reviews.

Data-driven reviews: Agency leaders and managers are to use  regular meetings, at least quarterly, to review data and drive progress  toward key performance goals and other management-improvement  priorities. For each APG, GPRAMA requires agency leaders to  conduct reviews at least quarterly to assess progress toward the goal,  determine the risk of the goal not being met, and develop strategies to  improve performance. Similarly, the Director of OMB, with relevant  parties, is to review progress toward each CAP goal.

Strategic reviews: OMB guidance directs agency leaders to annually  assess progress toward achieving each strategic objective using a  broad range of evidence.

GPRAMA establishes certain senior leadership positions and a council,  as described below.

Chief Operating Officer (COO): The deputy agency head, or  equivalent, is designated COO, with overall responsibility for  improving agency management and performance.

Performance Improvement Officer (PIO): Agency heads are to  designate a senior executive within the agency as the PIO. The PIO  reports directly to the COO and assists the agency head and COO  with various performance management activities.

Goal leaders: Goal leaders are responsible for developing strategies  to achieve goals, managing execution, and regularly reviewing  performance. GPRAMA requires goal leaders for CAP goals and  agency performance goals, including APGs. OMB guidance directs  agencies to designate goal leaders for strategic objectives.

Performance Improvement Council (PIC): The PIC is charged with  assisting OMB to improve the performance of the federal government  and achieve the CAP goals. The PIC is chaired by the Deputy  Director for Management at OMB and includes agency PIOs from  each of the 24 CFO Act agencies as well as other PIOs and  individuals designated by the chair. Among its responsibilities, the PIC  is to work to resolve government-wide or crosscutting performance  issues, and facilitate the exchange among agencies of practices that  have led to performance improvements within specific programs,  agencies, or across agencies.

GPRAMA includes several provisions related to providing the public and  Congress with information, as described below.

Performance.gov: GPRAMA calls for a single, government-wide  performance website to communicate government-wide and agency  performance information. Among other things, the website— implemented by OMB as Performance.gov—is to include (1) quarterly  progress updates on CAP goals and APGs; (2) an inventory of all  federal programs; and (3) agency strategic plans, annual performance  plans, and annual performance reports.

Reporting burden: GPRAMA establishes a process to reexamine the  usefulness of certain existing congressional reporting requirements.  Specifically, GPRAMA requires an annual review (including  congressional consultation), based on OMB guidance, of agencies’  reporting requirements to Congress. Additionally, OMB is to include in  the budget a list of plans and reports determined to be outdated or  duplicative and may submit legislation to eliminate or consolidate such  plans or reports.

The Administration’s Plans for Federal Performance Management

In early 2017, the administration announced several efforts that are  intended to improve government performance. The 2018 Budget Blueprint  states that the President’s Management Agenda will seek to improve the  federal government’s effectiveness by using evidence-based approaches,  balancing flexibility with accountability to better achieve results, improving  mission support functions, and developing and monitoring critical  performance measures. In addition, OMB issued several memoranda  detailing the administration’s plans to improve government performance  by reorganizing the government, reducing the federal workforce, and  reducing federal agency burden.

A number of these efforts, which are to leverage GPRAMA and our past  work, have the potential to further progress in addressing key governance  challenges. As part of reorganization efforts, OMB and agencies are  developing government-wide and agency reform plans, respectively, that  are to leverage various GPRAMA provisions. For example, an April 2017  memorandum states that OMB intends to monitor implementation of the  reform plans using CAP goals, APGs, annual strategic reviews, and  Performance.gov. The government-wide plan also is to include  crosscutting reform proposals, such as merging agencies or programs  that have similar missions. To that end, the memorandum states agencies  should consider our reports, including our work on fragmentation, overlap,  and duplication, as well as inspectors general reports.

Despite Progress in Selected Areas, the Executive Branch Needs to Take Additional Actions to Manage Crosscutting Issues

Agencies Have Made Progress in Some Areas, but Continued Attention Is Needed to Better Manage Crosscutting Issues

Many of the meaningful results that the federal government seeks to  achieve, such as those related to ensuring public health, providing  homeland security, and promoting economic development, require the  coordinated efforts of more than one federal agency, level of government,  or sector. For more than 2 decades, we have reported on agencies’  missed opportunities for improved collaboration through the effective  implementation of GPRA and, more recently, GPRAMA. Our reports  also have demonstrated that collaboration across agencies is critical to  address issues of fragmentation, overlap, and duplication as well as many  of the areas on our High-Risk List.

Fragmentation, Overlap, and Duplication: Since 2011, our annual  reports have identified 133 crosscutting areas that require the coordinated  effort of more than one federal organization, level of government, or  sector. For instance, for the area of federal grant awards, we found in  January 2017 that the National Park Service (NPS), Fish and Wildlife  Service, Food and Nutrition Service, and Centers for Disease Control and  Prevention (CDC) had not established guidance and formal processes to  ensure their grant-management staff review applications for potential  duplication and overlap among grants in their agencies before awarding.  We recommended that these agencies do so, and they agreed. As of  August 2017, these agencies had taken several actions to address the  recommendation. For example, the Department of the Interior (Interior)  provided documentation showing that the Fish and Wildlife Service now  requires discretionary grant applicants to provide a statement that  addresses whether there is any overlap or duplication of proposed  projects or activities to be funded by the grant. Fish and Wildlife also  updated its guidance to grant awarding offices instructing them to perform  a potential overlap and duplication review of all selected applicants prior  to award. Our Action Tracker provides details on the status of actions  from our annual reports.

Within the 133 crosscutting areas, since 2011 we have identified 315  targeted actions where opportunities exist to better manage  fragmentation, overlap, and duplication, including 29 new actions in our  most recent report issued in April 2017. We found that the executive  branch and Congress addressed 145 (46 percent) of the 315 actions. For  example, in November 2014, we recommended that the U.S. Coast  Guard and Consumer Product Safety Commission establish a formal  approach to coordination (such as a memorandum of understanding) to  facilitate information sharing; better leverage their resources; and address  challenges, including those related to fragmentation and overlap that we  identified. In response to this recommendation, the two agencies signed  a formal policy document to govern their coordination in May 2015. This  policy document outlined procedures for determining jurisdictional authority for recreational boat-associated equipment and marine safety  items. Specifically, the procedures clarified that upon receiving notice of a  possible defect, the agency receiving such notice shall determine whether  the item properly falls within its jurisdiction, and if not, initiate discussions  to determine the appropriate jurisdiction. These new procedures should  help the agencies share information and leverage each other’s resources  so they can better ensure that recreational boat-associated equipment  and marine safety items are fully regulated.

However, more work is needed on the remaining 170 actions (54 percent)  that have not been fully addressed. For example, in July 2016, we  reported that four federal agencies—the Departments of Defense,  Education, Health and Human Services, and Justice—manage at least 10  efforts to collect data on sexual violence, which differ in target population,  terminology, measurements, and methodology. We found that data  collection efforts use 23 different terms to describe sexual violence. Data  collection efforts also differed in how they categorized particular acts of  sexual violence, the context in which data were collected, data sources,  units of measurement, and time frames. We recommended that OMB  convene an interagency forum to better manage fragmentation of efforts  to collect sexual violence data. In commenting on that report, OMB stated  it would consider implementing the action in the future but did not believe  it was the most effective use of resources at that time, in part because the  agencies were not far enough along in their research. In response, we  stated that given the number of federal data collection efforts, the range  of differences across them, and the potential for causing confusion, it  would be beneficial for agencies to discuss these differences and  determine whether they are, in fact, necessary. As of July 2017, OMB had  not provided an update on the status of this recommendation.

High-Risk List: Since the early 1990s, our high-risk program has focused  attention on government operations with greater vulnerabilities to fraud,  waste, abuse, and mismanagement or that are in need of transformation  to address economy, efficiency, or effectiveness challenges. As of  February 2017, there were 34 high-risk areas covering a wide range of  issues including human capital management, modernizing the U.S.  financial regulatory system, and ensuring the security of federal  information systems and cyber critical infrastructure. Many of these high- risk areas require a coordinated response from more than one branch of  government, agency, or sector.

In the time between our 2015 and 2017 High-Risk Updates, many of  these high-risk areas on our list demonstrated solid progress. During that  period, 15 high-risk areas fully met at least one of the five criteria required  for removal from the High-Risk List. In many cases, progress was  possible through the joint efforts of Congress and leadership and staff in  agencies. For example, Congress passed over a dozen laws following our  2015 High-Risk Update to help address high-risk issues. In addition, in  2017, we removed one high-risk area on managing terrorism-related  information, because significant progress had been made to strengthen  how intelligence on terrorism, homeland security, and law enforcement is  shared among federal, state, local, tribal, international, and private sector  partners. Despite this progress, continued oversight and attention is also  warranted given the issue’s direct relevance to homeland security as well  as the constant evolution of terrorist threats and changing technology.

Our February 2017 High-Risk Update also highlighted a number of long- standing high-risk areas that require additional attention. We also added  three new crosscutting areas to incorporate the management of federal  programs that serve tribes and their members, the government’s  environmental liabilities, and the 2020 decennial census. Based on our  body of work on federal programs that serve tribes and their members,  we concluded that federal agencies had (1) ineffectively administered  Indian education and health care programs and (2) inefficiently fulfilled  their responsibilities for managing the development of Indian energy  resources. For example, we identified numerous challenges facing  Interior’s Bureau of Indian Education (BIE) and Bureau of Indian Affairs,  and the Department of Health and Human Services’ (HHS) Indian Health  Service (IHS), in administering education and health care services. We  concluded that these challenges put the health and safety of American  Indians served by these programs at risk. In May 2017, we issued two  additional reports on accountability for school construction and safety at  schools funded by BIE. Although these agencies have taken some  actions to address recommendations we made related to Indian  programs, about 50 recommendations have yet to be fully resolved. We  are monitoring federal efforts to address the unresolved  recommendations. We also are reviewing IHS’s workforce, and tribal  nations’ management and use of their energy resources.

The Executive Branch Could Better Leverage GPRAMA Implementation to Work across Organizational Boundaries

Many of the crosscutting areas highlighted by our annual reports on  fragmentation, overlap, and duplication and designated as high-risk would  benefit from enhanced collaboration among the federal agencies involved  in them. GPRAMA establishes a framework aimed at taking a more  crosscutting and integrated approach to focusing on results and  improving government performance. Our survey results and past work  demonstrate that agencies continue to face difficulties when working  together on crosscutting issues, but also that implementing certain  GPRAMA requirements can have a positive effect on collaboration.

An item related to coordination in our survey of federal managers is  statistically significantly lower in 2017, relative to our previous survey in  2013 and our initial survey in 1997. In 2017, an estimated 43 percent of  managers agreed that they use information obtained from performance  measurement to a great or very great extent when coordinating program  efforts with internal or external organizations (compared to an estimated  50 percent in 2013 and an estimated 57 percent in 1997). Moreover, our  past work has found that agencies face a variety of challenges when  working across organizational boundaries to deliver programs and  improve performance. For example, our work has found that  interagency groups have, at times, encountered difficulty clarifying roles  and responsibilities or developing shared outcomes and performance  measures.

In contrast, our past work demonstrates that implementing GPRAMA  provisions can improve collaboration. For example, in May 2016, we  found that OMB and the PIC updated the governance structure for CAP  goals to include both agency-level and Executive Office of the President  goal leaders and held regular, senior-level reviews on CAP goal  progress. Moreover, CAP goal teams told us that the CAP goal  designation increased leadership attention and improved interagency  collaboration on their crosscutting issues. Furthermore, our prior work has  found that priority goals and related data-driven reviews have also been  used to help manage crosscutting issues and enhance collaboration.

Priority Goals and Related Reviews Can Help Address Crosscutting Issues

Various GPRAMA requirements are aimed at improving agencies’  coordination of efforts to address crosscutting issues. As with our 2013  survey, our 2017 survey continues to show that CAP goals, APGs, and  related data-driven reviews—also called quarterly performance reviews  (QPR)—are associated with reported higher levels of collaboration with  internal and external stakeholders. For example, our 2017 survey data  indicate that about half of federal managers (an estimated 54 percent)  reported they were somewhat or very familiar with CAP goals. Among  these individuals, those who viewed their programs as contributing to  CAP goals to a great or very great extent (36 percent) were more likely to  report collaborating outside their program to a great or very great extent  to help achieve CAP goals (62 percent), as shown in figure 2. Our  analysis shows a similar pattern exists for APGs and QPRs.

Our past work also has highlighted ways in which OMB and agencies  could better implement GPRAMA’s crosscutting provisions—many of  which have been addressed. A continued focus on fully and effectively  implementing these provisions will be important as OMB and agencies  establish new CAP goals and APGs, and assess progress toward them  through related QPRs.

Cross-agency priority (CAP) goals: In May 2012 and June 2013, we  found that OMB had not always identified relevant agencies and program  activities as contributors to the initial set of CAP goals. OMB took  actions in response to our recommendations to include relevant  contributors. Our most recent review, in May 2016, found that all relevant  contributors had been identified for a subsequent set of CAP goals. In  that report, we also found that OMB and the PIC had improved  implementation of the CAP goals, in part, by helping agencies build their  capacity to contribute to implementing the goals. Appendix II summarizes  our past recommendations related to GPRAMA and the actions agencies  have taken to address them.

Agency priority goals (APGs): In April 2013, we found that agencies did  not fully explain the relationship between their APGs and crosscutting  efforts.

Identify contributors: Similar to OMB’s responsibilities with the CAP  goals, agencies are to identify the various organizations and programs  that contribute to each of their performance goals, including APGs.  We found that agencies identified internal contributors for their APGs,  but did not list external contributors in some cases. We recommended  that the Director of OMB ensure that agencies adhere to OMB’s  guidance for website updates by providing complete information about  the organizations, program activities, regulations, tax expenditures,  policies, and other activities—both within and external to the agency— that contribute to each APG. In response, in April 2015, OMB asked  agencies to identify organizations, program activities, regulations,  policies, tax expenditures, and other activities contributing to their  2014-2015 APGs. Based on an analysis of the final quarterly updates  for those APGs, published in December 2015, we found that agencies  made progress in identifying external organizations and programs for  their APGs.

Describe how agency goals contribute to CAP goals: Agencies  generally did not identify how their APGs contributed to CAP goals.  We recommended that OMB direct agencies to describe in their  performance plans how the agency’s performance goals—including  APGs—contribute to any of the CAP goals as required by GPRAMA.  In response, in July 2013, OMB updated its guidance directing  agencies to include a list of the CAP goals to which the agency  contributes and explain the agency’s contribution to them in their  strategic plans, performance plans, and performance reports.

Data-driven reviews: For their data-driven reviews of agency priority  goals, agencies are to include, as appropriate, relevant personnel within  and outside the agency who contribute to the accomplishment of each  goal. However, in February 2013, we found that most Performance  Improvement Officers (PIO) we surveyed (16 of 24) indicated that there  was little to no involvement in these reviews from external officials who  contribute to achieving agency goals. We recommended that OMB and  the PIC help agencies extend their QPRs to include, as relevant,  representatives from outside organizations that contribute to achieving  their APGs. OMB staff told us that they generally concurred with the  recommendation, but believed it would not always be appropriate to  regularly include external representatives in agencies’ data-driven  reviews, which they considered to be internal management meetings.

In a subsequent review, we found in July 2015 that PIOs at 21 of the 22  agencies we surveyed said that their data-driven reviews had a positive  effect on collaboration among officials from different offices or programs  within the agency. Despite the positive effects, most agency PIOs (17)  indicated that there continued to be little to no involvement in the reviews  from external officials who contribute to achieving agency goals. In May  2016, OMB and PIC staff reported that, in response to our earlier  recommendation, they were working with agencies to identify examples  where agencies included representatives from outside organizations in  data-driven reviews, and to identify promising practices based on those  experiences. PIC staff told us they would disseminate any promising  practices identified through the PIC Internal Reviews Working Group and  other venues. In August 2017, OMB staff told us they plan to hold a  summit with agencies later in the year to discuss implementing various  performance management requirements, which could include agencies  highlighting experiences and promising practices related to involving  external officials in their data-driven reviews. We continue to believe data- driven reviews should include any relevant contributors from outside  organizations and will continue to monitor progress.

Despite the important role priority goals and related reviews can play in  addressing crosscutting issues and enhancing collaboration, OMB  recently removed the priority status of the current sets of priority goals.  According to OMB staff, removing the priority designation from CAP goals  and APGs returned them to regular performance goals, which are not  subject to quarterly data-driven reviews or updates on the results of those  reviews on Performance.gov. In a June 2017 memorandum, OMB stated  that CAP goals and APGs are intended to focus efforts toward achieving  the priorities of current political leadership, and therefore reporting on the  priority goals of the previous administration on Performance.gov was  discontinued for the remainder of the period covered by the goals  (through September 30, 2017, the end of fiscal year 2017). The  memorandum further noted that agencies and teams working on those  goals should continue working on the current goals where they align with  the priorities of the current administration. Moreover, the memorandum  states that agencies have flexibility in structuring their data-driven reviews, but they should continue such reviews focused on agency  priorities.

When asked about these actions, OMB staff told us that they believed  they were working in line with the intentions of GPRAMA, which realigned  the timing of goal setting with presidential terms, to better take into  account changes in priorities. This is the first presidential transition since  GPRAMA was enacted, and OMB staff told us they thought the act was  unclear on how to handle priority goals during the changes in  administrations and priorities. They stated that it was not practical to  continue reporting on the priority goals of the prior administration as  agencies worked to develop new strategic plans and priority goals for  publication in February 2018. Hence, they told us OMB ended the current  round of CAP goals and directed agencies to remove the priority  designation from the APGs, returning them to regular performance goals.  OMB staff further told us that although the guidance was published in a  June 2017 memorandum, these decisions had been made and previously  communicated to agencies during the transition in administrations.  Therefore, reporting on the fiscal year 2014-2017 CAP goals, fiscal year  2016-2017 APGs, and related reviews stopped much earlier in the year,  well before goal cycles were planned to be completed on September 30,  2017.

OMB staff further stated that although the goals no longer had priority  designations, work towards them largely continued in 2017. For example,  one of the prior administration’s CAP goals was to modernize the federal  permitting and review process for major infrastructure projects. OMB  staff told us that they and agencies have continued many of the activities  intended to achieve that goal, but they are no longer subject to quarterly  data-driven reviews or updates on the results of these reviews on  Performance.gov. Moreover, they expect most of this work will continue  towards a new and refocused CAP goal on infrastructure permitting  modernization.

OMB staff reaffirmed to us their intentions to resume implementation of  CAP goals, APGs, and related data-driven reviews when the new  planning and reporting cycle begins in February 2018. This is in line with  stated plans to leverage various GPRAMA provisions to track progress of  proposed government-wide and agency-specific reforms, as outlined in  OMB’s April 2017 memorandum on the reform plans. In addition, OMB’s  July 2017 update to its guidance for implementing GPRAMA similarly  focuses on continued implementation of the act.

Strategic Reviews and Program Inventory Also Could Help with Crosscutting Issues

Additional aspects of GPRAMA implementation could similarly help  improve the management of crosscutting issues.

Strategic reviews: OMB’s 2012 guidance implementing GPRAMA  established a process in which agencies, beginning in 2014, were to  conduct leadership-driven, annual reviews of their progress toward  achieving each strategic objective established in their strategic plans.  As we found in July 2015, effectively implementing strategic reviews  could help identify opportunities to reduce, eliminate, or better manage  instances of fragmentation, overlap, and duplication. Under OMB’s  guidance, agencies are to identify the various organizations, program  activities, regulations, tax expenditures, policies, and other activities that  contribute to each objective, both within and outside the agency. Where  progress in achieving an objective is lagging, the reviews are intended to  identify strategies for improvement, such as strengthening collaboration to  better address crosscutting challenges, or using evidence to identify and  implement more effective program designs. If successfully implemented  in a way that is open, inclusive, and transparent—to Congress, delivery  partners, and a full range of stakeholders—this approach could help  decision makers assess the relative contributions of various programs to  a given objective. Successful strategic reviews could also help decision  makers identify and assess the interplay of public policy tools that are  being used to ensure that those tools are effective and mutually  reinforcing, and that results are being efficiently achieved.

In July 2017, OMB released guidance which updated the status of the  2017 strategic reviews. Because agencies are currently developing new  strategic goals and objectives, OMB stated that agencies may forego the  reporting and categorization requirements for any current strategic  objectives that an agency determines will be substantively different or no  longer aligned with the current administration’s policy, legislative,  regulatory, or budgetary priorities. In addition, OMB stated that while  there will be no formal meetings between OMB and the agencies to  discuss findings and related progress from the 2017 strategic reviews, it  expects that agencies will continue to conduct strategic reviews or assess  progress made toward strategic goals and objectives aligned with  administration policy. Furthermore, OMB stated that during this  transition year, updates of progress on agency strategic objectives will  only be published in the agency’s annual performance report and will not  be reported to Performance.gov. Full reporting through Performance.gov  is to resume after new agency strategic plans are published in February  2018. Agencies are to include a progress update for strategic objectives  as part of their progress update in their fiscal year 2017 annual  performance reports. Agencies also must address next steps for performance improvement as part of their fiscal year 2019 annual  performance plans.

Program inventories: GPRAMA requires OMB to publish a list of all  federal programs, along with related budget and performance information,  on a central government-wide website. Such a list could help decision  makers and the public fully understand what the federal government  does, how it does it, and how well it is doing. An inventory of federal  programs could also be a critical tool to help decision makers better  identify and manage fragmentation, overlap, and duplication across the  federal government.

Agencies developed initial program inventories in May 2013, but since  then have not updated or more fully implemented these inventories. In  October 2014, we found several issues limited the completeness,  comparability, and usefulness of the May 2013 program inventories.  OMB and agencies did not take a systematic approach to developing  comprehensive inventories. For example, OMB’s guidance in Circular No.  A-11 presented five possible approaches agencies could take to define  their programs and noted that agencies could use one or more of those  approaches in doing so. We found that because the agencies used  inconsistent approaches to define their programs, the comparability of  programs was limited within agencies as well as government-wide. In  addition, we found that the inventories had limited usefulness for decision  making, as they did not consistently provide the program and related  budget and performance information required by GPRAMA. Moreover, we  found that agencies did not solicit feedback on their inventories from  external stakeholders—which can include Congress, state and local  governments, third party service providers, and the public. Doing so  would have provided OMB and agencies an opportunity to ensure they  were presenting useful information for stakeholder decision making. We  concluded that the ability to tag and sort information about programs  through a more dynamic, web-based presentation could make the  inventory more useful. In October 2014, we made several  recommendations to OMB to update relevant guidance to help develop a  more coherent picture of all federal programs and to better ensure  relevant information is useful for decision makers. For example, we  recommended that OMB revise its guidance to direct agencies to consult  with relevant congressional committees and stakeholders on their  approach to defining and identifying programs when developing or  updating their inventories.

OMB staff generally agreed with these recommendations, but have not  yet taken any actions to implement them. OMB’s guidance for the  program inventory has largely remained unchanged since 2014, when  OMB postponed further development of the program inventory and  eliminated portions of the guidance. For example, the guidance no longer  describes, or provides directions for agencies to meet, GPRAMA’s  requirements for presenting related budget or performance information for  each program. OMB decided to postpone implementing a planned May  2014 update to the program inventory in order to coordinate with the  implementation of the public spending reporting required by the Digital  Accountability and Transparency Act of 2014 (DATA Act). OMB  subsequently stated that it would not begin implementing the program  inventory until after the DATA Act was implemented in May 2017, despite  requirements for regular updates to the program inventory to reflect  current budget and performance information.

The DATA Act is now being implemented, but OMB has postponed  resuming the development of the program inventory. In July 2017, OMB  staff told us that they are now considering how to align GPRAMA’s  program inventory provisions with future implementation of the Program  Management Improvement Accountability Act (PMIAA). This was  reflected in OMB’s July 2017 update to its guidance, which states that  OMB is working with agencies to determine the right strategy to merge  the implementation of the DATA Act and PMIAA with GPRAMA’s program  inventory requirements to the extent possible to avoid duplicating  efforts. For example, PMIAA requires OMB to coordinate with agency  Program Management Improvement Officers to conduct portfolio reviews  of agency programs to assess the quality and effectiveness of program  management.

GPRAMA requires OMB to issue guidance for implementing the program  inventory requirements, among other things. Moreover, federal internal  control standards state that organizations should clearly define what is to  be achieved, who is to achieve it, how it will be achieved, and the time  frames for achievement. As described above, OMB’s current guidance  for the program inventory lacks some of those details—such as  describing and providing direction to meet GPRAMA’s requirements for  budget and performance information—in part because OMB is working  with agencies to determine a strategy for implementation. Ensuring all  GPRAMA requirements are covered and taking action on our past  recommendations would help OMB improve its guidance to more fully  implement the program inventory and improve its usefulness.

To that end, in a report issued earlier this month, we identified a series of  iterative steps that OMB could use in directing agencies to develop a  useful inventory, as described in figure 3. A useful inventory would  consist of all programs identified, information about each program, and  the organizational structure of the programs. Our work showed that the  principles and practices of information architecture—a discipline focused  on organizing and structuring information—offer an approach for  developing such an inventory to support a variety of uses, including  increased transparency for federal programs. Such a systematic  approach to planning, organizing, and developing the inventory that  centers on maximizing the use and usefulness of information could help  OMB ensure the inventory is implemented in line with GPRAMA  requirements and meets the needs of decision makers and the public,  among others.

OMB’s guidance also lacks specific time frames, with associated  milestones for resuming implementation of the program inventory  requirements. As part of PMIAA’s requirements, OMB is to issue  standards, policies, and guidelines for program and project management  for agencies by December 2017. OMB staff told us that, within a year  after that, they expect to issue further guidance on moving forward with  resuming the program inventory. However, that general time frame was  not reflected in the July 2017 update to OMB’s guidance. Providing  specific time frames and associated milestones would bring the program  inventory guidance in line with other portions of OMB’s guidance for  implementing GPRAMA requirements, which contains a timeline of  various performance planning and reporting requirements, including  specific dates for meeting those requirements and related descriptions of  required actions. For example, OMB’s July 2017 guidance identifies over  30 actions agencies should take between June 2017 and December 2018  to implement various GPRAMA provision. More specific time frames  and milestones related to the program inventory requirements would help  agencies prepare for resumed implementation by allowing them to know  what actions they would be expected to take and by when. Moreover,  publicly disclosing planned implementation time frames and associated  milestones also would help ensure that external stakeholders are  prepared to engage with agencies as they develop and update their  program inventories.

The Executive Branch Does Not Systematically Assess the Results Achieved by Tax Expenditures, Which Represent Over $1 Trillion in Annual Forgone Revenue

Effectively implementing various GPRAMA tools could help inform  assessments of the performance of tax expenditures, which are  reductions in tax liabilities that result from preferential provisions (figure  4). In fiscal year 2016, tax expenditures represented an estimated $1.4  trillion in forgone revenue, an amount greater than total discretionary  spending that year. Despite the magnitude of these investments, our  work has also shown that little has been done to determine how well  specific tax expenditures work to achieve their stated purposes and how  their benefits and costs compare to those of spending programs with  similar goals.

GPRAMA requires OMB to identify tax expenditures that contribute to the  CAP goals. In addition, OMB guidance directs agencies to identify tax  expenditures that contribute to their strategic objectives and APGs.  However, our past work reviewing GPRAMA implementation found that  OMB and agencies rarely identified tax expenditures as contributors to  these goals. Fully implementing our recommendation to identify how tax  expenditures contribute to various goals could help the federal  government establish a process for evaluating the performance of tax  expenditures.

To that end, in May 2017, we provided the Director of OMB with three  priority recommendations that require attention:

Develop framework for reviewing performance: In June 1994, and  again in September 2005, we recommended that OMB develop a  framework for reviewing tax expenditure performance. We explained  that the framework should (1) outline leadership responsibilities and  coordination among agencies with related responsibilities, (2) set a  review schedule, (3) identify review methods and ways to address the  lack of credible tax expenditure performance information, and (4)  identify resources needed for tax expenditure reviews. Since their  initial efforts in 1997 and 1999 to outline a framework for evaluating  tax expenditures and preliminary performance measures, OMB and  the Department of the Treasury (Treasury) have ceased to make  progress and retreated from setting a schedule for evaluating tax  expenditures.

Inventory tax expenditures: In October 2014, we found that OMB  had not included tax expenditures in the federal program inventory,  and therefore was missing an opportunity to increase the  transparency of tax expenditures and the outcomes to which they  contribute. We recommended that OMB should designate tax  expenditures as a program type in relevant guidance, and develop, in  coordination with the Secretary of the Treasury, a tax expenditure  inventory that identifies each tax expenditure and provides a  description of how the tax expenditure is defined, its purpose, and  related budget and performance information. OMB staff said they  neither agreed nor disagreed with these recommended actions. As  noted earlier, OMB has not resumed updates to the program  inventory. Therefore, OMB had not taken any actions in response to  this recommendation, according to OMB staff as of July 2017.

Identify contributions to agency goals: In July 2016, we found that  agencies had made limited progress identifying tax expenditures’  contribution to agency goals, as directed by OMB guidance. As of  January 2016, 7 of the 24 CFO Act agencies identified tax  expenditures as contributing to their missions or goals. The 11 tax  expenditure they identified—out of the 169 tax expenditures included  in the President’s Budget for Fiscal Year 2017—represented  approximately $31.9 billion of the $1.2 trillion in estimated forgone  revenues for fiscal year 2015. (See figure 5.) To help address this  issue, we recommended that OMB, in collaboration with the  Department of the Treasury, work with agencies to identify which tax  expenditures contribute to their agency goals, as appropriate. In  particular, we recommended that they identify which specific tax  expenditures contribute to specific strategic objectives and APGs. In  July 2017, OMB staff said they had taken no actions to address the  recommendation.

Our July 2016 report also identified options for policymakers to further  incorporate tax expenditures into federal budgeting processes, several of  which options align with the recommendations discussed above. These  options could help achieve various benefits, but we also reported that  policymakers would need to consider challenges and tradeoffs in deciding  whether or how to implement them. For example, one option was to  require that all tax expenditures, or some subset of them, expire after a  finite period. This option could result in greater oversight, requiring  policymakers to explicitly decide whether to extend more or all tax  expenditures. One consideration with this option is that it could lead to  frequent changes in the tax code, such as from extended or expired tax  expenditures, which can create uncertainty and make tax planning more  difficult.

Long-standing Weaknesses Persist in Ensuring Performance Information Is Useful and Used; Expanded Use of Data-Driven Reviews Could Help Agencies Better Achieve Results

Federal Managers Generally Did Not Report Improvements in Their Use of Performance Information in Decision Making

Our previous work has shown that using performance information in  decision making is essential to improving results. Performance  information can be used across a range of management activities, such  as setting priorities, allocating resources, or identifying problems to be  addressed. However, our work continues to show that agencies can  better use performance information in decision making, as shown in the  example in the text box below.

Department of Justice (DOJ) Could Better Analyze Performance  Information to Reduce Backlog in Immigration Courts  In June 2017, we found that the case backlog—cases pending from previous  years that remain open at the start of a new fiscal year—at DOJ’s Executive  Office for Immigration Review (EOIR) courts more than doubled from fiscal  years 2006 through 2015. Stakeholders identified various factors that  potentially contributed to the backlog, including continuances—temporary case  adjournments until a different day or time. Our analysis of continuance records  showed that the use of continuances increased by 23 percent from fiscal years  2006 through 2015.  We found that EOIR collects continuance data but does not systematically  assess them. Systematically analyzing the use of continuances could provide  EOIR officials with valuable information about challenges the immigration courts  may be experiencing, such as with operational issues like courtroom technology  malfunctions, or areas that may merit additional guidance for immigration  judges. Further, using this information to potentially address operational  challenges could help that office meet its goals for completing cases in a timely  manner.  We recommended that the Director of EOIR systematically analyze immigration  court continuance data to identify and address any operational challenges faced  by courts or areas for additional guidance or training. EIOR agreed with this  recommendation. EOIR stated that it supports conducting additional analysis of  immigration court continuance data and recognizes that additional guidance or  training regarding continuances may be beneficial to ensure that immigration  judges use continuances appropriately in support of EOIR’s mission to  adjudicate immigration cases in a careful and timely manner. We will monitor  EOIR’s progress in taking these actions.

Our 2017 survey of federal managers shows little change in their reported  use of performance information. Using a set of survey questions, we  previously developed an index that reflects the extent to which managers  reported that their agencies used performance information for various  management activities and decision making. The index suggests that  government-wide use of performance information did not change  significantly between 2013 and 2017, and it is statistically significantly  lower relative to our 2007 survey, when we created the index. Figure 6  shows the questions included in the index and the government-wide  results.

In regard to individual survey items, in 2017 federal managers reported  no changes or decreases in their use of performance information when  compared to our last survey and when those survey items were first  introduced. These results are generally consistent with our last few  surveys. For example, in 2008 we found that there had been little  change in federal managers’ reported use of performance information  government-wide from 1997 to our 2007 survey. Citing those results, the  Senate Committee on Homeland Security and Governmental Affairs  report accompanying the bill that would become GPRAMA stated that  agencies were not consistently using performance information to improve  their management and results. The report further stated that provisions  in GPRAMA are intended to address those findings and increase the use  of performance information to improve performance and results.  However, five items that were highlighted in our 2008 statement on the  2007 survey results generally show no improvement when compared to  the 2017 results, as shown in figure 7.

The one exception is for managers’ reported use of performance  information to refine program performance measures. While this item was  statistically significantly higher in 2013 relative to 2007—an estimated 46  percent to 53 percent—the 2017 result (43 percent) is a statistically  significant decrease relative to 2013 and is not statistically different from  the 2007 results. Another item, the use of performance information to  adopt new program approaches or change work processes, also was  statistically significantly lower in 2017 (47 percent) when compared to  2007 and 2013 (53 and 54 percent, respectively). This is of particular  concern as agencies are developing their reform plans. Moreover, when  compared to our 1997 survey, the 2017 results show four of the five items  are statistically significantly lower, and the remaining item—allocating  resources—has not changed.

Similarly, we found there was no improvement in 2017 for more recent  survey items on other uses of performance information compared to the  years in which they were introduced, as shown in figure 8.

Although one item, on the use of performance information to develop  program strategy, was statistically significantly higher in 2013 relative to  2007 (an estimated 58 and 51 percent, respectively), the 2017 result (53  percent) does not represent a statistically significant change from either of  those years. Another item, on the use of performance information to  streamline programs to reduce duplicative activities, is statistically  significantly lower relative to 2013, when it was introduced (from 44 to 33  percent in 2017). This is especially concerning because streamlining and  reducing duplication are to be key parts of agencies’ reform plans.

There is one area in the survey where we saw improvement: an  estimated 46 percent of managers agreed to a great or very great extent  that employees who report to them pay attention to their agency’s use of  performance information in management decision making. That is  statistically significantly higher relative to 2013 (40 percent), as well as  when compared to when the item was introduced in 2007 (37 percent).  For a new and related item in the 2017 survey that asked managers the  amount of attention their employees pay to the use of performance  information in decision making when compared to 3 years ago, we found an estimated 48 percent reported that employees pay about the same

33 percent reported that employees pay somewhat or a great deal  more attention.

Federal Managers Generally Did Not Report Changes in Applying Management Practices That Promote the Use of Performance Information

In September 2005, we identified five practices that agencies can apply to  enhance the use of performance information in their decision making and  improve results: demonstrating management commitment;  communicating performance information frequently and efficiently;  improving the usefulness of performance information, such as by  ensuring the accessibility of the information; developing the capacity to use performance information; and aligning agency-wide goals, objectives, and measures.

Many of the requirements put in place by GPRAMA reinforce the  importance of these practices. Our 2017 survey of federal managers  includes a number of items related to these practices. However, the 2017  results suggest that managers have not effectively adopted them. In the  following sections, we examine several of the practices to enhance the  use of performance information and their related survey items further. In  doing so, we also highlight a subset of six survey items related to these  practices that, while separate from those in our use of performance  information index, we found in September 2014 to have a statistically  significant and positive relationship with it.

The commitment of agency leaders to results-oriented management is  critical to increased use of performance information for policy and  program decisions. GPRAMA requires top leadership involvement in  performance management, including leading data-driven performance  reviews. However, we have previously reported that improvements are  needed to strengthen leadership’s commitment to use performance  information, as discussed in the text box below.

Department of Defense Should Strengthen Leadership Responsibilities for  Using Performance Information  In January 2005, we designated the Department of Defense’s (DOD) approach  to business transformation as high-risk because DOD had not taken the  necessary steps to achieve and sustain business reform on a broad, strategic,  department-wide, and integrated basis. In the February 2017 update to our  High-Risk List, we found that DOD had taken some positive steps to improve its  business transformation efforts.continuing to hold business function leaders accountable for diagnosing  performance problems and identifying strategies for improvement, and  leading regular DOD performance reviews regarding transformation goals  and associated metrics and ensuring that business function leaders attend  these reviews to facilitate problem solving.

In July 2017, DOD officials told us that the department’s performance reviews  have been put on hold until after the new Agency Strategic Plan is issued. We  will review DOD’s updated Agency Strategic Plan when it is issued (expected in  February 2018, as required by GPRAMA) to see if it addresses continuing to  hold business function leaders accountable for diagnosing performance  problems and identifying strategies for improvement. We will continue to  monitor the status of these actions.

GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on  Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017).

Results from our 2017 survey show no statistically significant difference  relative to 2013 in managers’ perceptions of leaders’ and supervisors’  attention and commitment to the use of performance information. (See  figure 9.)

Three items are statistically significantly different from the years when  they were introduced. Two items increased between 1997 and 2017:  changes by management to my program(s) are based on results-oriented  information (from an estimated 16 to 25 percent), and the individual I  report to periodically reviews with me the outcomes of my program(s)  (from 42 to 54 percent). For the third item, top leadership demonstrates  a strong commitment to using performance information to guide decision  making, results decreased from 49 percent in 2007 to 42 percent in  2017.

New items in the 2017 survey show some improvement in management  commitment to the use of performance information in decision making. An  estimated 36 percent of federal managers reported that, when compared  to 3 years ago, the individual they report to pays somewhat or a great  deal more attention to the use of performance information in decision  making, while 46 percent said they pay about the same amount of  attention. Additionally, an estimated 21 percent of federal managers  said that, when compared to 3 years ago, the head of their agency pays  somewhat or a great deal more attention to the use of performance  information in decision making, while 33 percent said they pay about the  same amount of attention.

Communicating performance information frequently and effectively  throughout an agency can help to achieve the agency’s goals.  GPRAMA includes requirements for communicating performance  information, such as reporting progress updates for APGs at least  quarterly. However, our prior work has found that some agencies could  continue to improve in the communication of performance information, as  illustrated by the example in the text box below.

Department of Education (Education) Could Better Share Effective  Practices across States in Grant Program  Education awards 21st Century Community Learning Centers grants to states,  which in turn competitively award funds to local organizations that use them to  offer academic enrichment and other activities to improve students’ academic  and behavioral outcomes. In April 2017, we found that states are experiencing  substantial difficulty in sustaining their programs after 21st Century funding  ends. We further found that Education was missing opportunities in its  monitoring efforts to collect information on states’ strategies and practices for  program sustainability—information that could be useful for sharing promising  practices across states.  We recommended that Education use the information it collects from its  monitoring visits and ongoing interactions with states to share effective  practices across states for sustaining their 21st Century programs once  program funding ends. Education neither agreed nor disagreed with the  recommendation but outlined steps it is taking to address it. We will continue to  monitor progress on the implementation of this recommendation.

There is no difference for two survey items on federal managers  communicating performance information relative to 2013 or since those  items were introduced in 2007. In 2017, we estimate that 44 percent of  federal managers agreed to a great or very great extent that agency  managers at their level effectively communicate performance information  on a routine basis. In addition, 34 percent agreed to a great or very  great extent that managers at their level use performance information to  share effective program approaches with others.

Our 2017 survey data also indicate that agencies may not be effectively  communicating to their employees about contributions to CAP goals or  progress toward achieving APGs. Of the estimated 54 percent of federal  managers who indicated they were familiar with CAP goals, 23 percent  reported that their agency has communicated to its employees on those  goals to a great or very great extent. Of the 74 percent of federal  managers who indicated familiarity with APGs, 44 percent reported that  their agency has communicated on progress toward achieving those  goals to great or very great extent.

Improving the Usefulness of Performance Information

Our prior work has shown that agencies should consider users’ differing  needs—for accessibility, accuracy, completeness, consistency, ease of  use, timeliness, and validity, among others things—to ensure that  performance information will be both useful and used. GPRAMA  introduced several requirements that could help to address aspects of  usefulness, such as requiring agencies to disclose more information  about the accuracy and validity of their performance data and actions to  address limitations to the data. However, agencies face challenges in  ensuring their performance information is useful, with one instance from  our past work described in the text box below.

The Environmental Protection Agency (EPA) Could Improve Usefulness of  Information in Planned Grantee Portal  EPA monitors performance reports and program-specific data from grantees to  ensure that grants achieve environmental and other program results. However,  in July 2016, we found that EPA’s 2014 internal analysis of its grants  management business processes identified improvements that, if implemented  into EPA’s planned web-based portal, could improve the accessibility and  usefulness of information in grantee performance reports for EPA, grantees,  and other users.  We recommended, among other actions, that EPA incorporate expanded  search capability features, such as keyword searches, into its proposed web- based portal for collecting and accessing performance reports to improve their  accessibility. EPA agreed with our recommendation but stated that it is a long- term initiative, subject to the agency’s budget process and replacement of its  existing grants management system. As of May 2017, EPA officials said that  they have not begun work on the web-based portal project, which is subject to  the availability of funds.

Federal managers generally responded similarly in 2017 on a variety of  survey items related to usefulness, relative to earlier surveys. On a  broadly worded item, less than half of managers agreed to a great or very  great extent that agency managers at their level take steps to ensure that  performance information is useful and appropriate. At an estimated 43  percent in 2017, this represents no statistically significant change  compared to our last surveys in 2013 or 2007, when the item was  introduced.

Responses to four survey items indicate no changes in hindrances  related to the usefulness of performance information. There is no  statistically significant change in managers reporting hindrances  compared to 1997 or 2013, as shown in figure 10.

In addition, there was a statistically significant increase when compared  to 2013 on only one of six items about managers’ views on the usefulness  of performance information, as shown in figure 11.

As the figure shows, approximately one-third to half of managers agreed  to a great or very great extent on each item related to the usefulness of  performance information. Although less than half of managers reported  having sufficient information on validity of performance data used to make  decisions, this represents a statistically significant increase to an  estimated 42 percent in 2017 compared to 36 percent in 2013, and from  28 percent in 2000, when this item was introduced. This is a notable  improvement because our September 2014 report found that the  strongest driver of the use of performance information was whether  federal managers had confidence in its validity.

Our analysis suggests that easy access to performance information is  related to the effective communication of performance information. Of the  estimated 49 percent of federal managers in 2017 who agreed to a great  or very great extent that performance information is easily accessible to  managers at their level, 63 percent also agreed that agency managers at  their level effectively communicate performance information on a routine  basis to a great or very great extent. Conversely, of the 20 percent that  agreed to a small or no extent that performance information is easily  accessible to managers at their level, 12 percent also agreed that agency  managers at their level effectively communicate performance information  on a routine basis to a great or very great extent.

Developing the Capacity to Use Performance Information

Our prior work has shown that building capacity—including analytical  tools and staff expertise—is critical to using performance information in a  meaningful manner. GPRAMA lays out specific requirements that  reinforce the importance of staff capacity to use performance information.  GPRAMA directed the Office of Personnel Management (OPM) to take  certain actions to support agency hiring and training of performance  management staff. Specifically, by January 2012, OPM was to identify  skills and competencies needed by government personnel for setting  goals, evaluating programs, and analyzing and using performance  information for improving government efficiency and effectiveness. By  January 2013, OPM was to incorporate these skills and competencies  into relevant position classifications and to work with each agency to  incorporate the identified skills into employee training.

In April 2013, we found that OPM had completed its work on the first two  responsibilities and taken steps to work with agencies to incorporate  performance management staff competencies into training. However,  OPM did not assess competency gaps among agency performance  management staff to inform its work. Without this information, OPM,  working with the PIC, was not well-positioned to focus on the most- needed resources and help other agencies use them. We recommended  that the Director of OPM, in coordination with the PIC and the Chief  Learning Officer Council, work with agencies to take the following three  actions:  1.  Identify competency areas needing improvement within agencies.  2.  Identify agency training that focuses on needed performance  management competencies.  3.  Share information about available agency training on competency  areas needing improvement.

In July 2017, PIC staff stated they have not focused on identifying  competency areas because the competencies do not resonate strongly  with the performance community. Instead, staff said they identified a need  for introductory training on performance management, which they have  developed and piloted. They said that they are not sure when they will  implement the training, since the PIC is reviewing priorities with its new  executive director. We continue to believe that identifying the competency  areas would be useful, and will monitor the PIC’s efforts to identify and  share training.

The need for performance management training is further highlighted by  our survey results. Our 2017 survey shows no statistically significant  change in managers’ responses about the availability of training on  various performance management activities relative to 2013, including the  use of performance information to make decisions. However, the  response to each of the six questions related to specific training is  statistically significantly higher relative to the year in which it was  introduced, as shown in figure 12.

Similarly, in 2017 there was no statistically significant change on four  survey items related to agencies’ analysis and evaluation tools and staff’s  skills and competencies when compared to 2013 or when these items  were introduced. We estimate that in 2017

29 percent of managers agreed to a great or very great extent that  their agencies were investing in resources to improve the agencies’  capacity to use performance information;

28 percent of managers agreed to a great or very great extent that  their agencies were investing the resources needed to ensure that  performance data are of sufficient quality;

33 percent of managers reported that they agreed to a great or very  great extent that their agencies have sufficient analytical tools for  managers at their levels to collect, analyze, and use performance  information; and

33 percent of managers reported that they agree to a great or very  great extent that the programs they are involved with have sufficient  staff with the knowledge and skills needed to analyze performance  information.

Conducting Additional Data-Driven Reviews Could Increase the Use of Performance Information in Decision Making

Performance reviews can serve as a strategy to bring leadership and  other responsible parties together to review performance information and  identify important opportunities to drive performance improvements.  Our prior work has examined how different types of performance  reviews—strategic reviews, data-driven reviews, and retrospective  regulatory reviews—can contribute to agencies assessing progress  toward desired results.

Strategic reviews: As previously mentioned, in implementing GPRAMA,  OMB established a review process in which agencies are to annually  assess their progress in achieving each strategic objective in their  strategic plans, known as strategic reviews. Given the long-term and  complex nature of many outcomes, the strategic review should be  informed by a variety of evidence regarding the implementation of  strategies and their effectiveness in achieving outcomes. OMB’s guidance  states that the strategic review process should consider multiple  perspectives and sources of evidence to understand the progress made  on each strategic objective. It further states that the results of these  reviews should inform many of the decision-making processes at the  agency, as well as decision making by the agency’s stakeholders, in  areas such as long-term strategy, budget formulation, and risk  management. In 2017, agencies are completing their fourth round of  these reviews.

Our prior work has identified ways in which agencies can effectively  conduct these reviews and leverage the results that come from them. In  July 2015, we identified seven practices federal agencies can employ to  facilitate effective strategic reviews. (See sidebar.) In addition, earlier  this month we reported on selected agencies’ experiences in  implementing these reviews. Specifically, we found that (1) strategic  reviews helped direct leadership attention to progress on strategic  objectives, (2) agencies used existing management and performance  processes to conduct the reviews, and (3) agencies refined their reviews  by capturing lessons learned.

Data-driven reviews: GPRAMA requires agencies to review progress  toward APGs at least once a quarter. The Senate Committee on  Homeland Security and Governmental Affairs report accompanying the  bill that would become GPRAMA stated that this approach is aimed at  increasing the use of performance information to improve performance  and results. In February 2013, we identified nine leading practices to  promote successful data-driven performance reviews in the federal  government. (See sidebar.) In July 2015, we found that most of the 24  CFO Act agencies were conducting their reviews in line with GPRAMA  requirements and our leading practices. Moreover, agencies reported  that their data-driven performance reviews had positive effects on  progress toward agency goals, collaboration between agency officials, the  ability to hold officials accountable for progress, and efforts to improve the  efficiency of operations.

Our 2017 survey shows that federal managers remain largely unfamiliar  with their agency’s data-driven performance reviews, also known as  quarterly performance reviews (QPRs). An estimated 35 percent of  managers reported familiarity with their agency’s QPRs. Survey results  show that a greater percentage of Senior Executive Service (SES)  managers than non-SES managers reported that they were familiar with  QPRs. Approximately 50 percent of SES managers reported being  somewhat or very familiar with QPRs; 34 percent of non-SES reported  the same.

However, for the estimated 35 percent of managers who reported  familiarity with QPRs, the more they viewed their programs being subject  to a QPR, the more likely they were to report their agency’s QPRs were  driving results and conducted in line with our leading practices. Figure 13  shows several illustrative examples of these survey items. For example,  of the estimated 48 percent of federal managers who reported their  programs being subject to QPRs to a great or very great extent, 83  percent also reported their agencies use QPRs to identify problems or  opportunities associated with agency performance goals. Conversely,  for the 24 percent of managers who reported their programs were subject  to QPRs to a small or no extent, 22 percent also reported the reviews  were used for these purposes to a great or very great extent.

Being subject to a QPR is also positively related to viewing QPRs as  having led to similar meetings at lower levels. An estimated 62 percent of  federal managers who reported being subject to QPRs to a great or very  great extent also reported their agencies have similar meetings at lower  levels to a great or very great extent. An estimated 16 percent of federal  managers subject to QPRs to a small or no extent reported the same.

Despite the reported benefits of and results achieved through QPRs, as  found by our past work and survey data, these reviews are not  necessarily widespread. GPRAMA requires agencies to conduct QPRs  for APGs, which represent a small subset of goals—generally 2 to 8  priority goals at each designated agency, with approximately 100 total  government-wide. Moreover, these required reviews are at the  department (or major independent agency) level. These reasons may  explain why most managers reported they were not familiar with the  reviews.

As was described previously, our 2017 survey data show that the  reported use of performance information in decision making generally has  not improved and in some cases is lower than it was 20 years ago.  Survey data also show that managers generally have not reported  increases in their employment of practices that further promote the use of  performance information in decision making. This suggests that agencies  could increase the use of performance information in decision making and  the likelihood of achieving desired results by going beyond the specific  GPRAMA requirements and expanding their use of data-driven  performance reviews—in line with leading practices—to more broadly  cover other agency-wide performance goals, as well as goals at lower  levels within the agency. For example, such reviews at the program level  could help inform the previously mentioned portfolio reviews required by  the Program Management Improvement Accountability Act (PMIAA).

We have already suggested expanding reviews to other performance  goals. Our management agenda for the presidential and congressional  transition includes a key action to expand the use of data-driven  performance reviews to assess progress toward meeting agency  performance goals. Our prior work has stated that although GPRAMA’s  requirements apply at the agency-wide level, they can also serve as  leading practices at other organizational levels, such as component  agencies, offices, programs, and projects. In addition, federal internal  control standards call for the design of appropriate control activities, such  as top-level reviews of actual performance and reviews by management  at the functional or activity level. The standards also recommend that  management design control activities at the appropriate levels in the  organizational structure.

The July 2017 update to OMB’s guidance states that agency leaders,  including various chief officer positions, are to conduct frequent data- driven reviews to drive improvements on various management  functions. For example, the agency Chief Human Capital Officer is to  conduct quarterly data-driven reviews (known as HRStat) to monitor the  progress of human capital goals and measures contained in the human  capital operating plan. Beyond these management areas, OMB’s  guidance also states that agencies may expand quarterly progress  reviews beyond APGs to include other goals and priorities. However,  OMB’s guidance does not identify practices for agencies to expand the  use of these reviews to other goals, such as other agency-wide  performance goals or those at lower levels within the agency. As  mentioned previously, one of the responsibilities of the Performance  Improvement Council (PIC) is to facilitate the exchange among agencies  of practices that have led to performance improvements within specific  programs, agencies, or across agencies. By working with the PIC to  identify and share among agencies practices to expand the use of data- driven reviews, OMB could help agencies increase the use of  performance information in decision making and achieve results.

Retrospective regulatory reviews: In retrospective reviews, agencies  evaluate how existing regulations are working in practice and whether  they are achieving expected outcomes. GPRAMA requires agencies to  identify and assess how their various program activities and other  activities, including regulations, contribute to APGs. However, in April  2014, we found that agencies reported mixed experiences linking  retrospective analyses to APGs. We recommended that OMB  strengthen these reviews by issuing guidance for agencies to take actions  to ensure that contributions made by regulations toward achieving APGs  are properly considered, and improve how retrospective regulatory  reviews can be used to help inform assessments of progress toward  these APGs. OMB staff agreed with this recommendation and stated  that the agency was working on strategies to help facilitate agencies’  ability to use retrospective reviews to inform APGs.

To that end, in April 2017, OMB issued guidance to agencies that, among  other things, emphasized the importance of performance measures  related to evaluating and improving the net benefits of their respective  regulatory programs. OMB included explicit references to section 6 of  Executive Order 13563, which directed agencies’ efforts to conduct  retrospective regulatory reviews. Specifically, the updated guidance  encourages agencies to establish and report “meaningful performance  indicators and goals for the purpose of evaluating and improving the net  benefits of their respective regulatory programs.” The guidance further  states that agencies’ efforts to improve such net benefits may be  conducted as part of developing agency strategic and performance plans  and priority goals. In July 2017, OMB confirmed that the updated  guidance was issued, in part, to address our April 2014  recommendation.

Evidence-Based Tools Can Help Federal Agencies Use Performance Information for Decision Making

For several years, OMB has encouraged agencies to expand their use of  evidence—performance measures, program evaluation results, and other  relevant data analytics and research studies—in budget, management,  and policy decisions with the goal of improving government  effectiveness. In particular, OMB has encouraged agencies to  strengthen their program evaluations—systematic studies that use  research methods to address specific questions about program  performance. Evaluation is closely related to performance  measurement and reporting. Evaluations can be designed to better  isolate the causal impact of programs from other external economic or  environmental conditions in order to assess a program’s effectiveness.  Thus, an evaluation study can provide a valuable supplement to ongoing  performance reporting by measuring results that are too difficult or  expensive to assess annually, explaining the reasons why performance  goals were not met, or assessing whether one approach is more effective  than another.

Despite the valuable insights and information that program evaluations  can provide, we continue to find that most federal managers lack access  to or awareness of such studies. Our 2017 survey shows that an  estimated 40 percent of managers reported that an evaluation had been  completed within the past 5 years of any program, operation, or project in  which they were involved—comparable to the results in our 2013 survey,  when questions about program evaluations were added. In recent  years, OMB has encouraged agencies to explore evidence-based tools to  strengthen agency and grantee evaluation capacity, consider the  effectiveness of their programs, and foster innovation rooted in research  and rigorous evaluation. During the past 2 years, we examined several of  those tools, as described below.

Pay for success: Also known as social impact bonds, pay for success is  a contracting mechanism under which investors provide the capital the  government uses to provide a social service. The government specifies  performance outcomes in pay for success contracts and generally  includes a requirement that a program’s impact be independently  evaluated. The evaluators also are to regularly review performance data,  while those managing and investing in a project focus on performance  and accountability, as shown in the figure 14.

In September 2015, we found that the federal government’s involvement  in pay for success had been limited. In addition, a formal mechanism  for federal agencies to collaborate on pay for success did not exist. We  concluded that, given the evolving nature of pay for success, a  mechanism for federal agencies to collaborate would increase access to  leading practices. We therefore recommended that OMB establish a  formal means for federal agencies to collaborate on pay for success.  OMB concurred and, in February 2016, announced that it had developed  the Pay for Success Interagency Learning Network with representatives  from 10 federal agencies to share lessons, hone policy, and strengthen  implementation.

Tiered evidence grants: Tiered evidence grants seek to incorporate  evidence of effectiveness into grant making. Federal agencies establish  tiers of grant funding based on the level of evidence grantees provide on  their approaches to deliver social, educational, health, or other services.  (See figure 15.)

Smaller awards are used to test new and innovative approaches, while  larger awards are used to scale up approaches that have strong evidence  of effectiveness. This creates incentives for grantees to use approaches  supported by evidence and helps them build the capacity to conduct  evaluations.

In September 2016, we found that interagency collaboration had helped  federal agencies that administer tiered evidence grants address  challenges and share lessons learned. At that time, such collaborative  efforts relied on informal networks. We recommended that OMB establish  a formal means for agencies to collaborate on tiered evidence grants.  OMB had no comment on the recommendation. In July 2017, OMB staff  told us that they had established an interagency working group and other  mechanisms to facilitate collaboration and disseminate information on  tiered evidence grants.

Performance partnerships: Performance partnerships allow federal  agencies to provide grant recipients flexibility in how they use funding  across two or more programs along with additional flexibilities. In  exchange, the recipient commits to improve and assess progress toward  agreed-upon outcomes. Figure 16 provides an overview of the  performance partnership model.

In April 2017, we examined two performance partnership initiatives  authorized by Congress: the Environmental Protection Agency’s  Performance Partnership Grants and the Performance Partnership Pilots  for Disconnected Youth, which allows funding from multiple programs  across multiple agencies to be combined into pilot programs serving  disconnected youth. For the Performance Partnership Pilots for  Disconnected Youth, we found that the agencies involved in the initiative  had not fully identified the key financial and staff resources each agency  would need to contribute over the lifetime of the initiative in line with  leading practices for interagency collaboration. This was because  agencies primarily had been focused on meeting near-term needs to  support design and implementation. We also found that agencies had not  developed criteria to help determine whether, how, and when to  implement the flexibilities tested by the pilots in a broader context. (This is  known as scalability.) Officials involved in the pilots told us it was too  early in pilot implementation to determine such criteria. However, by not  identifying these criteria while designing the pilots, they were risking not  collecting needed data during pilot implementation. We recommended  that OMB coordinate with federal agencies to identify (1) agency resource  contributions needed for the lifetime of the pilots and (2) criteria and  related data for assessing scalability. OMB neither agreed nor disagreed  with these recommendations. We continue to monitor progress on these  recommendations.

Agencies Have Made Some Progress in Aligning Daily Operations with Results, but Could Take Additional Actions

Agencies Could Take Additional Actions to Further Develop Results- Oriented Cultures

In 2003, we identified nine key practices for effective performance  management that collectively create a “line of sight” between individual  performance and organizational success. (See sidebar on next page.)  Our recent work and the results of our 2017 survey of federal managers  highlight areas where agencies have made progress but could take  additional action to better reflect several of these practices, thereby better  instilling results-oriented cultures.

Align individual performance expectations with organizational goals:  Our 2003 report found that high-performing organizations use their  performance management systems to help individuals see the connection  between their daily activities and organizational goals. The executive  branch has taken several steps to link individual and organizational  results. For example, in October 2000, OPM issued guidance to link SES  performance expectations with GPRA-required goals. In January 2012,  OPM and OMB released a government-wide SES performance appraisal  system that provided agencies with a standard framework to manage the  performance of SES members.

However, our work continues to identify areas for improvement.

Goal leaders and deputy goal leaders are responsible for achieving  APGs, but our July 2014 review found that the performance plans for  a sample of goal and deputy goal leaders generally did not link their  individual performance and the broader goal. We recommended  that OMB ensure that those plans demonstrate a clear connection  with APGs. OMB staff generally agreed with our recommendation. In  July 2017, OMB staff stated that components of both OMB and OPM  guidance support accountability for agency priority goals. Despite this,  we continue to believe that ensuring an explicit connection in  performance plans to APGs will improve accountability, and that  additional action is needed to do so.

In May 2016, we found that the Federal Emergency Management  Agency (FEMA) had not aligned Federal Disaster Recovery  Coordinators’ performance expectations with its organizational goals  for implementing the National Disaster Recovery Framework. We  concluded that without this linkage, FEMA could not evaluate how  effectively the coordinators performed in implementing the framework.  We recommended that FEMA align performance expectations  consistent with leading practices. The Department of Homeland  Security concurred with our recommendation. In July 2017, FEMA  stated that it is preparing the Field Leader Manual, which will define  the core competencies and duties of coordinators. We will continue to  monitor FEMA’s actions to implement this recommendation.

Our 2017 survey also shows that this linkage could be improved for other  federal employees. An estimated 58 percent of federal managers  reported using performance information to a great or very great extent in  setting expectations for employees they manage or supervise. The  2017 responses do not represent a statistically significant change when  compared to our last survey in 2013 (62 percent) or to 1997 (61 percent),  the year this survey item was introduced.

Address organizational priorities: Our prior work showed that, by  requiring and tracking follow-up actions on performance gaps, high- performing organizations underscore the importance of holding  individuals accountable for making progress on their priorities. Our past  and 2017 surveys have identified differences in responses between SES  and non-SES managers reporting being held accountable for results.  For example, in 2017, our survey results indicate that there was a  statistically significant difference between SES and non-SES managers  reporting to a great or very great extent that they were held accountable  for results of the programs for which they are responsible. However, our  2017 survey shows no change compared to our last survey in either SES  or non-SES managers reporting they were held accountable for results.  There are statistically significant increases when compared to 1997, when  these survey items were introduced. For example, an estimated 79  percent of SES managers and 64 percent of non-SES managers reported  being held accountable to a great or very great extent for results of the  programs for which they are responsible in 2017. This does not represent  a statistically significant change from our 2013 survey (80 percent and 67  percent, respectively), but it is statistically significantly higher than the 62  percent of SES managers and 54 percent of non-SES managers in 1997.  (See figure 17.)

Similarly, as shown in figure 18, an estimated 71 percent of SES  managers reported being held accountable to a great or very great extent  for accomplishing agency strategic goals in 2017. This represents no  statistical change since 2013 (73 percent), but it is a statistically  significant increase compared to when this item was introduced in 2003  (61 percent). Additionally, as figure 18 shows, a gap between being held  accountable for strategic goals and having the decision-making authority  needed to help accomplish those goals has nearly closed, due to an  increase in the latter survey item. The estimated 69 percent of SES  managers who reported having such authority to a great or very great  extent in 2017 is a statistically significant increase relative to both 2013  (61 percent) and 1997 (51 percent).

As noted earlier, GPRAMA requires goal leaders for CAP goals and  APGs. Our past work has generally found that they are in place.  GPRAMA also requires agencies to identify an agency official responsible  for resolving major management challenges, which can help ensure  accountability. (See sidebar.) However, in June 2016 we found that 17  of the 24 CFO Act agencies had not identified an agency official  responsible for resolving each of their challenges, partly because OMB  guidance was not clear that major management challenges should be  identified in agency performance plans. We recommended that the 17  agencies identify such officials in their performance plans, and that OMB  clarify its guidance. OMB revised its guidance accordingly in July 2016,  and, as of July 2017, 7 of the 17 agencies had identified officials  responsible for resolving major management challenges.

Link pay to individual and organizational performance: High- performing organizations seek to create pay, incentive, and reward  systems that clearly link employee knowledge, skills, and contributions to  organizational results. Our work has found that agencies have made  progress in this area. For example, in July 2013, we found that the  Securities and Exchange Commission (SEC) lacked mechanisms to  monitor how supervisors used its performance management system to  recognize and reward performance. To help enhance the credibility of  SEC’s performance management system, we recommended that it create  mechanisms to monitor how supervisors use the performance  management system. In a subsequent (December 2016) report, we found  that, in response to our recommendation, SEC began monitoring how  supervisors provide feedback, recognize and reward staff, and address  poor performance.

However, federal managers generally reported no change on three items  related to recognizing and rewarding employee performance since our  last survey in 2013 (figure 19). One of those items—managers agreeing  to a great or very great extent that employees in their agency receive  positive recognition for helping the agency to accomplish its strategic  goals—had a statistically significant increase between 1997 and 2017  (from an estimated 26 percent to 46 percent).

Make meaningful distinctions in performance: Effective performance  management requires the organization’s leadership to meaningfully  distinguish between acceptable and outstanding performance of  individuals and to appropriately reward those who perform at the highest  level. For example, in January 2015, we found disparities in  performance ratings for SES among agencies. Across the 24 CFO Act  agencies, the percent of SES rated at the highest level ranged from about  22 percent to 95 percent in fiscal year 2013. To help address these  disparities, we recommended that the Director of OPM consider the need  to refine the performance certifications guidelines addressing distinctions  in performance. To address this recommendation, OPM informed us, in  June 2015, that it had convened a cross-agency working group that  developed a standard template for agencies to complete and post on a  website to more transparently justify their SES ratings distributions.

In May 2016, we found that about 74 percent of non-SES employees  under a five-level appraisal system—the most commonly used system— were rated in the top two of five performance categories in 2013. We  explored this issue further in our December 2016 review of human capital  challenges at the Veterans Health Administration (VHA), which illustrates  the importance of making meaningful distinctions in performance for non- SES employees. We found that in fiscal year 2014, about 73 percent of  VHA employees were rated in the top two of five performance categories.  This may have been due, in part, to a policy that did not require standards  to be defined for each level of performance. We recommended that VHA  ensure that meaningful distinctions are being made in employee  performance ratings by reviewing and revising performance management  policies consistent with leading practices, among other actions. The  Department of Veterans Affairs partially concurred with our  recommendation. In May 2017, the department stated that it had begun  piloting a new performance management process and would analyze  results at the end of fiscal year 2017.

Additional OMB Actions Could Help Address Long- Standing Performance Measurement Issues

One key aspect of connecting daily operations to results is aligning  program performance measures to agency-wide goals and objectives.  However, in 2017, an estimated 50 percent of federal managers agreed  to a great or very great extent that managers at their level took steps to  create such an alignment. There has been no statistically significant  change since this item was introduced in 2007.

In addition, GPRAMA calls for agencies to develop a balanced set of  performance measures, which reinforces the need for agencies to have a  variety of measures across program areas. Our 2017 survey shows that  managers have not reported any difference in the availability of  performance measures for their programs when compared to the 2013  results. However, the 2017 result (an estimated 87 percent) represents a  statistically significant increase when compared to 1997 (76 percent).  When asked about the availability of certain types of performance  measures, three of the five types (outcome, output, and efficiency) were  statistically significantly higher in 2017 when compared to our initial 1997  survey. However, when comparing 2017 results to those in 2013, two of  the five types (output and quality) showed a statistically significant  decrease, and the other types did not change. These are illustrated in  figure 20.

Beyond the survey results, our work has found that some agencies had  not developed or used outcome measures, but have taken steps to do so.  Agencies have been responsible for measuring program outcomes since  GPRA was enacted in 1993. The text box below describes two illustrative  examples from our past work.

Examples of Agencies That Did Not Develop or Use Outcome Measures  Patient access to electronic health information: In March 2017, we found that the  Department of Health and Human Services (HHS) had invested over $35 billion  since 2009 to enhance patient access to electronic health information, among  other things. HHS had not developed outcome measures to gauge the  effectiveness of these efforts, which meant the department did not have  information to determine whether the efforts were contributing to its overall  goals. We recommended that HHS develop relevant outcome measures and  HHS concurred.  Safety interventions: According to the Federal Motor Carrier Safety  Administration (FMCSA), between 2011 and 2015, over 4,000 people died in  crashes involving motor carriers each year.

GAO, Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety  Interventions, GAO-17-49 (Washington, D.C.: Oct. 27, 2016).

Further OMB actions could also help agencies make progress in  measuring the performance of different program types. In our June 2013  report on initial GPRAMA implementation, we found that agencies  experienced common issues in measuring the performance of various  types of programs, such as contracts and grants. We recommended  that OMB work with the PIC to develop a detailed approach to examine  those difficulties. Although they took some actions, OMB and the PIC  have not yet developed a comprehensive and detailed approach to  address these issues. We concluded that, without such an approach, it  would be difficult for the PIC and agencies to fully understand these  measurement issues and develop a crosscutting approach to help  address them. In August 2017, OMB staff stated that efforts related to the  future implementation of the Program Management Improvement  Accountability Act (PMIAA) could help address this recommendation. As  highlighted in table 1, our work continues to show why it is important for  OMB and the PIC to take actions to more fully address our  recommendation.

Increased Transparency and Public Engagement Could Improve Government Oversight and Foster Innovation

Further OMB Action Could Improve the Transparency of Government-wide Performance and Financial Data

Congress has passed legislation to increase the transparency and  accessibility of federal performance and financial data. For example,  GPRAMA modernized agency reporting requirements to ensure that they  make timely, relevant data available to inform decision making by  Congress and agency officials as well as improve transparency for the  public. Results of our 2017 survey, however, show the need for  improvements in the public availability of agency performance  information. An estimated 17 percent of managers reported that their  agency’s performance information is easily accessible to the public to a  great or very great extent, the same percentage as in 2013. Moreover,  of the 87 percent of managers that reported there are performance  measures for the programs they are involved in, 25 percent reported that  they use information obtained from performance measurement when  informing the public about how programs are performing to a great or very  great extent. This is not statistically different from the 30 percent  estimated in 2013.

The DATA Act, enacted in 2014, built on previous transparency legislation  by expanding what federal agencies are required to report regarding their  spending. The act significantly increases the types of data that must be  reported, requires government-wide data standards, and regular reviews  of data quality to help improve the transparency and accountability of  federal spending data. OMB provides websites and guidance to make  agency performance and financial information available to the public;  however, our prior work has identified a number of areas related to  Performance.gov and the DATA Act where OMB action is needed to  improve the transparency and accessibility of this information.

Performance.gov: Since 2013, our work has identified a number of  issues with Performance.gov, the website intended to serve as a central  source of information on the federal government’s goals and  performance. Over time, we have recommended that OMB take a number  of specific actions to improve the website. For example, in June 2013, we  found that the website offered an inconsistent user experience and  presented accessibility and navigation challenges. To clarify the purpose  of the website and enhance its usability, we recommended that OMB take  steps to systematically collect customer input.

In August 2016, we reported that OMB was not meeting all of the  reporting requirements for Performance.gov, and did not have a plan to  develop and improve the website. We recommended that OMB ensure  that information presented on Performance.gov consistently complies with  reporting requirements and develop a plan for the website that includes,  among other things, a customer outreach plan. OMB agreed with these  recommendations and, in July 2017, OMB staff informed us that they will  be partnering with a vendor to redesign Performance.gov to improve the  accessibility of information on the website. To inform this redesign, OMB  staff said that they will consider our previous recommendations and plan  to engage a wide group of stakeholders, including Congress, agency  staff, and interested members of the public and outside organizations.  OMB staff anticipated releasing updated agency reporting guidance in the  fall of 2017 and the redesigned website in February 2018.

Under GPRAMA, OMB is required to make available, through  Performance.gov, quarterly updates on progress toward CAP goals and  APGs. As described earlier, in June 2017 OMB announced that reporting  to Performance.gov has been discontinued through the end of fiscal year  2017 as agencies develop new priority goals. However,  Performance.gov does not state that it will not be updated, nor does it  provide the location of the final progress updates for these goals. OMB’s  guidance states that agencies should report the results of progress on  their previous APGs in their annual performance reports for fiscal year  2017. Moreover, OMB staff told us that the existing updates on  Performance.gov for CAP goals, last updated in December 2016,  represent the final updates on those goals, although they are not labeled  as such on the website. As a result, those interested in progress updates  and reported results for the previous priority goals may not know where  they will be able to find this information, limiting the transparency and  accessibility of those results for decision makers and the public.

DATA Act: The DATA Act requires federal agencies to disclose their  spending and link this to program activities so that policymakers and the  public can more effectively track federal spending. The act has the  potential to improve the accuracy and transparency of federal spending  information and increase its usefulness for government decision making  and oversight. Since the DATA Act became law, OMB and Treasury have  taken significant steps to make more complete and accurate federal  spending data available. These have included standardizing data element  definitions to make it easier to compare different federal agencies’  financial information, and issuing guidance to help agencies submit  required data. In May 2017, federal agencies started to report data under  the standardized definitions developed under the act.

We have made a number of recommendations to address challenges that  could affect the consistency and quality of the data. Addressing these  recommendations could help ensure that financial data are provided to  the public in a transparent and useful manner. For example, in January  2016, we found some standardized data element definitions were  imprecise or ambiguous, which could result in inconsistent or potentially  misleading reporting. We recommended that OMB provide agencies with  additional guidance to address potential issues with the clarity,  consistency, and quality of reported data. OMB released guidance in  May and November 2016, but in April 2017 we found that additional  guidance was needed to help agencies implement certain data definitions  to produce data that would be consistent and comparable across  agencies. We are in the process of examining the quality of the data  that was submitted by agencies in May 2017 and was made available to  the public on an early version of the USAspending.gov website. We  expect to issue the results of this work in fall 2017.

More Complete Public Reporting of Performance Information Could Enhance Oversight and Accountability

Our past work also identified a number of actions agencies need to take  to make performance information more transparent. Increasing the  accessibility of this information could enhance oversight and  accountability of agency performance and results.

CAP goals: In May 2016, we found that while selected CAP goal teams  were working to develop performance measures to track progress, they  were not consistently reporting on their efforts to develop these  measures. We recommended that OMB report on Performance.gov the  actions that CAP goal teams are taking to develop performance  measures and quarterly targets to help ensure that measures are aligned  with major activities, and ensure that it is possible to track teams’  progress toward establishing measures. While OMB agreed with this  recommendation, it did not address it before reporting on the CAP goals  was discontinued, as discussed earlier.

Customer service standards: As we described earlier, in 2017, an  estimated 48 percent of federal managers that indicated they have  performance measures for the programs they are involved in also agreed  to a great or very great extent that they have customer service  performance measures. There has been no statistically significant change  relative to our last survey in 2013, or the initial survey in 1997.  Relatedly, in October 2014, we reviewed customer service standards at  five federal agencies. Customer service standards inform customers  about what they have a right to expect when they request services, and  the standards should include goals for the quality and timeliness of a  service an agency provides to its customers. They should also be easily  available to the public so that customers know what to expect, when to  expect it, and from whom. In our review of standards at five agencies,  however, we found that only Customs and Border Protection had  standards that were easily available to the public. We recommended the  other four agencies—the United States Forest Service, Federal Student  Aid, the National Park Service (NPS), and the Veterans Benefits  Administration (VBA)—make their standards more easily accessible to  the public. As of July 2017, only VBA had done so.

Major management challenges: In June 2016, we found that 14 of the  24 CFO Act agencies did not describe their major management  challenges in their performance plans, as required by GPRAMA.  Furthermore, 22 of the 24 agencies reviewed did not report complete  performance information for each of their major management challenges,  including performance goals, milestones, indicators, and planned actions  that they have developed to address such challenges. As a result, it was  not always transparent what these agencies considered to be their major  management challenges or how they planned to resolve these  challenges. We recommended that the 22 agencies describe their major  management challenges in their agency performance plans and include  goals, measures, milestones, and information on planned actions and  responsible officials. As of August 2017, 8 agencies—the U.S. Agency for  International Development, Small Business Administration, Nuclear  Regulatory Commission, OPM, National Aeronautics and Space  Administration (NASA), and the Departments of Education, State, and  Veterans Affairs—had fully implemented our recommendations; the other  14 agencies had not.

Quality of performance information: In September 2015, we found that  six selected agencies reported limited information on the actions they are  taking to ensure the quality of their performance information for selected  APGs, as required by GPRAMA. We recommended that all six of the  agencies work with OMB to fully report this information. In response, the  Department of Homeland Security and NASA described how they ensure  reliable performance information is reported to external audiences. As of  June 2017, the Departments of Agriculture, Defense, the Interior, and  Labor had not yet taken actions to address this recommendation by  providing more specific explanations of how they ensure reliable  performance information is reported for their APGs.

Unnecessary reports: GPRAMA requires that OMB guide an annual  review of agencies’ plans and reports for Congress and include in the  President’s budget a list of those plans and reports determined to be  outdated or duplicative. However, in July 2017, we found that OMB did  not implement the report review process on an annual basis, as  required. We also found that OMB published the list of agency plans  and reports on Performance.gov, rather than in the President’s annual  budget, where they may be more visible and useful to congressional  decision makers and others. Therefore, we recommended that OMB  instruct agencies to identify outdated or duplicative reports on an annual  basis and submit or reference the list of identified plans and reports with  the President’s annual budget. OMB agreed with these  recommendations. In July 2017, OMB stated it would include a list of  report modification proposals in the President’s fiscal year 2019 budget  as required by GPRAMA.

For all of the unimplemented recommendations described above, we will  continue to monitor agencies’ actions.

Open Innovation Can Help Agencies Engage the Public to Achieve Results, but Guidance for Implementing Initiatives Should Be Improved

In addition to providing access to performance and financial information,  federal agencies can directly engage and collaborate with citizens,  nonprofits, academic institutions, and other levels of government using  open innovation strategies. Open innovation involves using various tools  and approaches to harness the ideas, expertise, and resources of those  outside an organization to address an issue or achieve specific goals. In  October 2016, we found that in recent years agencies had frequently  used five open innovation strategies—singularly or in combination—to  collaborate with citizens and encourage their participation in agency  initiatives. (See figure 21.)

Our October 2016 report found that agencies can use these strategies for  a variety of purposes.

To develop new ideas, solutions to specific problems, or new  products: For example, from April 2015 to November 2016, the  Department of Energy held a prize competition to create more efficient  devices that would double the energy captured from ocean waves.  According to the competition’s website, the winning team achieved a  five-fold improvement.

To enhance collaboration and agency capacity by leveraging  external resources, knowledge, and expertise: For example, every  2 years since 2009, the Federal Highway Administration has regularly  engaged stakeholders to identify and implement innovative ideas that  have measurably improved the execution of highway construction  projects.

To collect the perspectives and preferences of a broad group of  citizens and external stakeholders: For example, the Food and  Drug Administration used in-person and online dialogue to engage  outside stakeholders in the development of an online platform  designed to make key datasets easily accessible to the public.

Subsequently, in June 2017, we found that OMB, the Office of Science  and Technology Policy (OSTP), and the General Services Administration  (GSA) developed resources to support the use of open innovation  strategies by federal agencies. These resources included guidance,  staff to assist agencies in implementing initiatives, and websites to  improve access to relevant information. For example, GSA developed a  step-by-step implementation guide, program management team, and  website to help agency staff carry out prize competitions and challenges.  Agencies have also developed their own resources, including guidance,  staff positions, and websites, to reach specific audiences and to provide  tailored support for open innovation strategies they use frequently. For  example, NASA’s Solve website provides a central location for the public  to find the agency’s challenges and citizen science projects, as well as  links to relevant resources.

We also evaluated key government-wide guidance for the five strategies  listed above to determine the extent to which the guidance reflects  leading practices for effectively implementing open innovation initiatives.

We identified these practices in our October 2016 report. We found that  the guidance for each strategy reflected these practices to differing  extents, as shown in figure 22.

We made 22 recommendations to GSA, OMB, and OSTP to enhance the  guidance. GSA and OMB generally agreed with these recommendations  and OSTP neither agreed nor disagreed. We will monitor their progress  toward implementing these recommendations.

GPRAMA provides important tools that can help decision makers better  achieve results and address the federal government’s significant and  long-standing governance challenges. Although OMB and agencies have  made progress in improving implementation of the act over the years, our  work has highlighted numerous opportunities for further improvements.

In 2017, OMB removed the priority designation of CAP goals and APGs.  For those goals, this action stopped related data-driven reviews and  quarterly updates of progress on Performance.gov until new priority goals  are published next year. What OMB considers to be the final results of  CAP goals for fiscal years 2014 to 2017 already are on Performance.gov  (although not labeled as such). In addition, agencies may report on their  former APGs in their annual fiscal year 2017 performance reports.  However, Performance.gov does not state that it will not be updated or  provide the location of the final progress updates for these goals, limiting  transparency and its value to the public. OMB has stated its plans to  restart implementation of those provisions in February 2018, with the start  of a new goal cycle. We believe it is critical for OMB to do so, given the  important role those tools play in addressing key governance challenges  and the results we have seen in better managing crosscutting areas and  driving performance improvements across the government.

In addition, OMB has postponed implementation of the federal program  inventory. To date, the inventory has only been developed once, in 2013,  despite requirements for regular updates to reflect current budget and  performance information. OMB has given a variety of reasons for the  delays over the past 4 years—most recently, to determine the right  strategy to merge implementation of the DATA Act and PMIAA with  GPRAMA’s program inventory requirements. Although OMB staff told us  that they expect to issue guidance by the end of 2018 to resume  implementation of the program inventory requirements, they have not  provided more specific time frames and milestones related to the program  inventory requirements. Doing so would help agencies prepare for  resumed implementation. Moreover, publicly disclosing planned  implementation time frames and associated milestones would help  ensure that interested stakeholders, such as federal decision makers and  the public, are prepared to engage with agencies as they develop and  update their program inventories, which in turn could help ensure the  inventories meet stakeholders’ needs.

A well-developed inventory would provide key program, budget, and  performance information in one place to help federal decision makers  better understand the federal investment and results in given policy  areas, and better identify and manage fragmentation, overlap, and  duplication. Information architecture offers one approach to developing an  inventory. As OMB determines a strategy for implementing the program  inventory and develops its guidance, considering such a systematic  approach to planning, organizing, and developing the inventory that  centers on maximizing the use and usefulness of information could help it  ensure the inventory meets GPRAMA requirements as well as the needs  of decision makers and the public. Moreover, such an approach could  also help OMB implement our past recommendations related to the  program inventory, which are intended to ensure the inventory provides  more complete information and is useful to various stakeholders.

Our survey of federal managers continues to generally show no  improvement in their reported use of performance information in decision  making, nor in the employment of practices that can enhance such use.  One area where our survey data and past work show promise is through  the use of regular, leadership-driven reviews of performance data at  agencies, especially when conducted in line with related leading  practices. However, GPRAMA only requires these data-driven reviews for  APGs, which represent a small subset of goals, both within individual  agencies as well as across the government. This is probably why most  federal managers were not familiar with the reviews. Identifying and  sharing practices for expanding the use of those reviews—such as for  additional agency-wide performance goals and at lower levels within  agencies—could significantly enhance the use of performance  information and drive to better and greater results.

We are making the following four recommendations to OMB:  The Director of OMB should update Performance.gov to explain that  quarterly reporting on the fiscal year 2014 through 2017 CAP goals and  fiscal year 2016 and 2017 APGs was suspended, and provide the  location of final progress updates for these goals. (Recommendation 1)

The Director of OMB should revise and publicly issue OMB guidance— through an update to its Circular No. A-11, a memorandum, or other  means—to provide time frames and associated milestones for  implementing the federal program inventory. (Recommendation 2)

The Director of OMB should consider—as OMB determines its strategy  for resumed implementation of the federal program inventory—using a  systematic approach, such as the information architecture framework, to  help ensure that GPRAMA requirements and our past recommendations  for the inventory are addressed. (Recommendation 3)

The Director of OMB should work with the Performance Improvement  Council to identify and share among agencies practices for expanding the  use of data-driven performance reviews beyond APGs, such as for other  performance goals and at lower levels within agencies, that have led to  performance improvements. (Recommendation 4)

We provided a draft of this report to the Director of the Office of  Management and Budget for review and comment. In comments provided  orally and via email, OMB staff agreed with the recommendations in this  report.

OMB staff also asked us to (1) consider revising the draft title of the  report, to better reflect progress in GPRAMA implementation, and (2)  clarify our recommendations on issuing guidance for implementing the  federal program inventory and expanding the use of data-driven  performance reviews, by describing possible actions that could be taken  to implement them. We agreed and made revisions accordingly.

We are sending copies of this report to interested congressional  committees, the Director of the Office of Management and Budget, and  other interested parties. This report will also be available at no charge on  the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The GPRA Modernization Act (GPRAMA) includes a statutory provision  for us to periodically evaluate implementation of the act. Since 2012, we  have issued over 30 products in response to this provision; this is the  third summary report. This report assesses how implementation of  GPRAMA has affected the federal government’s progress in resolving key  governance challenges in (1) addressing crosscutting issues, (2) ensuring  performance information is useful and used in decision making, (3)  aligning daily operations with results, and (4) building a more transparent  and open government.

We reviewed relevant statutory requirements, related Office of  Management and Budget (OMB) guidance, and our recent work related to  GPRAMA implementation and the four key governance challenges  included in our reporting objectives. Specifically, since our last summary  report in September 2015, we examined various aspects of GPRAMA  implementation in 12 products that covered 35 agencies, including the 24  agencies covered under the Chief Financial Officers (CFO) Act of 1990,  as amended (identified in table 2). We interviewed OMB and Performance  Improvement Council staff to obtain (1) their perspectives on GPRAMA  implementation and progress on the four governance challenges, and (2)  updates on the status of our past recommendations. We also received  updates from other agencies on the status of our past recommendations  to them related to GPRAMA implementation.

To supplement this review, we administered our periodic survey of federal  managers on organizational performance and management issues from  November 2016 through March 2017. This survey is comparable to five  previous surveys we conducted in 1997, 2000, 2003, 2007, and 2013.  We selected a stratified random sample of 4,395 people from a  population of approximately 153,779 mid-level and upper-level civilian  managers and supervisors working in the 24 executive branch agencies  covered by the CFO Act, as shown in table 2. We obtained the sample  from the Office of Personnel Management’s (OPM) Enterprise Human  Resources Integration (EHRI) database as of September 30, 2015, which  was the most recent fiscal year data available at the time. We used file  designators indicating performance of managerial and supervisory  functions. In reporting survey data, we use the term “government-wide”  and the phrases “across the government” or “overall” to refer to the 24  CFO Act executive branch agencies. We use the terms “federal  managers” and “managers” to collectively refer to both managers and  supervisors.

We designed the questionnaire to obtain the observations and  perceptions of respondents on various aspects of results-oriented  management topics. These topics include the presence and use of  performance measures, any hindrances to measuring performance and  using performance information, agency climate, and program evaluation  use. To assess implementation of GPRAMA, the questionnaire included  questions to collect respondents’ views on various provisions of  GPRAMA, such as cross-agency priority goals, agency priority goals, and  related quarterly performance reviews.

Similar to the five previous surveys, the sample was stratified by agency  and by whether the manager or supervisor was a member of the Senior  Executive Service (SES). The management levels covered general  schedule (GS) or equivalent schedules at levels comparable to GS-13  through GS-15 and career SES or equivalent. Stratifying the sample in  this way ensured that the population from which we sampled covered at  least 90 percent of all mid- to upper-level managers and supervisors at  the departments and agencies we surveyed.

Most of the items on the questionnaire were closed-ended, meaning that  depending on the particular item, respondents could choose one or more  response categories or rate the strength of their perception on a 5-point  extent scale ranging from “no extent” to “very great extent.” On most  items, respondents also had an option of choosing the response category  “no basis to judge/not applicable.” A few items had other options, such as  “yes,” “no,” or “do not know,” or a 3-point familiarity scale (“not familiar,”  “somewhat familiar,” and “very familiar”).

We asked many of the items on the questionnaire in our earlier surveys,  though we introduced a number of new items in 2013, including the  sections about GPRAMA and program evaluations. For 2017, we added a  new question on use of performance information (question 12) and a new  question on program evaluation (question 24). Before administering the  survey, questions were reviewed by our staff, including subject matter  experts, a survey specialist, and a research methodologist. We also  conducted pretests of the new questions with federal managers in several  of the 24 CFO Act agencies. We changed the wording of subquestions or  added clarifying examples based on pretester feedback.

To administer the survey, we e-mailed managers in the sample to notify  them of the survey’s availability on our website and we included  instructions on how to access and complete the survey. To follow up with  managers in the sample who did not respond to the initial notice, we  emailed or called multiple times to encourage survey participation or  provide technical assistance, as appropriate.

Similar to our last survey, we worked with OPM to obtain the names of  the managers and supervisors in our sample, except for those within  selected subcomponents whose names were withheld from the EHRI  database. Since Foreign Service officials from the Department of State  (State) are not in the EHRI database, we drew a sample for that group  with the assistance from State. We worked with officials at the  Department of Homeland Security (DHS) and the Department of the  Treasury (Treasury) to gain access to these individuals to maintain  continuity of the population of managers surveyed from previous years.  The Department of Justice (DOJ) was concerned about providing  identifying information (e.g., names, e-mail addresses, and phone  numbers) of federal agents to us, so we administered the current survey  to DOJ managers in our sample through DOJ officials. To identify the  sample of managers whose names were withheld from the EHRI  database, we provided DOJ with the last four digits of Social Security  numbers, the subcomponent, duty location, and pay grade information.  To ensure that DOJ managers received the same survey administration  process as the rest of the managers in our sample to the extent possible,  we provided DOJ with text for the survey activation and reminder e-mails  similar to ones we emailed to managers at other agencies. DOJ  administered the survey to these managers and emailed them one  reminder to complete the survey.

To help determine the reliability and accuracy of the EHRI data elements  used to draw our sample of federal managers, we checked the data for  reasonableness and the presence of any obvious or potential errors in  accuracy and completeness and reviewed past analyses of the reliability  of this database. For example, we identified cases where the managers’  names were withheld and contacted OPM to discuss this issue. We also  checked the names of the managers in our selected sample provided by  OPM with the applicable agency contacts to verify these managers were  still employed with the agency. We noted discrepancies when they  occurred and excluded them from our population of interest, as  applicable. On the basis of these procedures, we believe the data we  used from the EHRI database are sufficiently reliable for the purpose of  the survey.

Of the 4,395 managers selected for the 2017 survey, we found that 388 of  the sampled managers had retired, separated, or otherwise left the  agency or had some other reason that excluded them from the population  of interest. These exclusions included managers that the agency could  not locate, and therefore we were unable to request that they participate  in the survey. We received usable questionnaires from 2,726 sample  respondents, for a weighted response rate of about 67 percent of the  remaining eligible sample. The weighted response rate across 23 of the  24 agencies ranged from 57 percent to 82 percent, while DOJ had a  weighted response rate of 36 percent. See the supplemental material for  each agency’s response rate.

We conducted a nonresponse bias analysis using information from the  survey and sampling frame as available. The analysis confirmed  discrepancies in the tendency to respond to the survey related to agency  and SES status. The analysis also revealed some differences in response  propensity by age and GS level; however, the direction and magnitude of  the differences on these factors were not consistent across agencies or  strata. Our data may be subject to bias from unmeasured sources for  which we cannot control. Results, and in particular estimates from  agencies with low response rates such as DOJ, should be interpreted  with caution because these estimates are associated with a higher level  of uncertainty.

The overall survey results are generalizable to the government-wide  population of managers as described above. The responses of each  eligible sample member who provided a usable questionnaire were  weighted in the analyses to statistically account for all members of the  population. All results are subject to some uncertainty or sampling error  as well as nonsampling error, including the potential for nonresponse bias  as noted above. Because we followed a probability procedure based on  random selections, our sample is only one of a large number of samples  that we might have drawn.

The magnitude of sampling error will vary across the particular surveys,  groups, or items being compared because we (1) used complex survey  designs that differed in the underlying sample sizes, usable sample  respondents, and associated variances of estimates, and (2) conducted  different types of statistical analyses. For example, the 2000 and 2007  surveys were designed to produce agency-level estimates and had  effective sample sizes of 2,510 and 2,943, respectively. However, the  1997 and 2003 surveys were designed to obtain government-wide  estimates only, and their sample sizes were 905 and 503, respectively.  Consequently, in some instances, a difference of a certain magnitude  may be statistically significant. In other instances, depending on the  nature of the comparison being made, a difference of equal or even  greater magnitude may not achieve statistical significance.

Because each sample could have provided different estimates, we  express our confidence in the precision of our particular sample’s results  as a 95 percent confidence interval. This is the interval that would contain  the actual population value for 95 percent of the samples we could have  drawn. The percentage estimates presented in this report based on our  sample for the 2017 survey have 95 percent confidence intervals within  plus or minus 5.5 percentage points of the estimate itself, unless  otherwise noted. We also note in this report when we are 95 percent  confident that changes from 1997 or 2013 relative to 2017 are statistically  significant. Online supplemental material shows the questions asked on  the survey along with the percentage estimates and associated 95  percent confidence intervals for each item for each agency and government-wide. In a few instances, we report estimates with larger  margins of error because we deemed them reliable representations of  given findings due to the statistical significance of larger differences  between comparison groups. In all cases, we report the applicable  margins of error.

In addition to sampling errors, the practical difficulties of conducting any  survey may also introduce other types of errors, commonly referred to as  nonsampling errors. For example, difficulties in how a particular question  is interpreted, in the sources of information available to respondents, or in  how the data were entered into a database or analyzed can introduce  unwanted variability into the survey results. With this survey, we took a  number of steps to minimize these nonsampling errors. For example, our  staff with subject matter expertise designed the questionnaire in  collaboration with our survey specialists. As noted earlier, the new  questions added to the survey were pretested to ensure they were  relevant and clearly stated. When the data were analyzed, a second  independent analyst on our staff verified the analysis programs to ensure  the accuracy of the code and the appropriateness of the methods used  for the computer-generated analysis. Since this was a web-based survey,  respondents entered their answers directly into the electronic  questionnaire, thereby eliminating the need to have the data keyed into a  database, thus avoiding a source of data entry error.

To supplement descriptive analysis of the survey questions, we  generated an index to gauge government-wide use of performance  information. The index, which was identical to one we reported in 2014,  averaged manager’s responses to 11 questions deemed to relate to the  concept of performance information use. The index runs from 1  (corresponding to an average value of “to no extent”) to 5 (corresponding  to an average value of “to a very great extent”). We used Cronbach’s  alpha to assess the internal consistency of the scale. Our government- wide index score weights each agency’s contribution equally, and  provides a relative measure of the use of performance information over  time rather than an absolute indicator of the government-wide level of use  of performance information.

We conducted this performance audit from January 2016 to September  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix II: Recommendations from GAO’s Work Related to the GPRA Modernization Act

The Office of Management and Budget (OMB) and agencies have taken  some actions to address our recommendations related to implementation  of the GPRA Modernization Act of 2010 (GPRAMA); however, the  majority of recommendations remain open. Since GPRAMA was enacted  in January 2011, we have made 100 recommendations in 18 reports to  OMB and agencies aimed at improving the act’s implementation (table  3). Of those 100, OMB and the agencies have implemented 42  recommendations. Fifty-eight recommendations require additional action.

Nearly half (47) of our recommendations are directed to OMB. For the 23  recommendations that OMB has implemented, many represent revisions  to guidance to better reflect GPRAMA’s requirements or to enhance  implementation. Many of the 24 recommendations to OMB that are not  implemented deal with long-standing or complex challenges, on which  OMB has taken limited action to date. Of those, we have designated 3 as  priorities for OMB to address. Agencies have also taken some action on  our recommendations, implementing 19 of the 53 recommendations we  have made.

The following tables present each of the 100 recommendations along with  a summary of actions taken to address it. Tables 4 and 5 provide  information about our recommendations to OMB that are implemented  and not implemented, respectively. Tables 6 and 7 provide information  about our recommendations to other agencies that are implemented and  not implemented, respectively.

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the above contact, Benjamin T. Licht (Assistant Director)  and Shannon Finnegan (Assistant Director) supervised this review and  the development of the resulting report. Leah Q. Nash (Assistant  Director), Elizabeth Fan (Analyst-in-Charge), and Adam Miles (Analyst-in- Charge) supervised the development and administration of the Federal  Managers Survey and the resulting supplemental material. Peter Beck,  Valerie Caracelli, Karin Fangman, Steven Flint, Robert Gebhart, Ricky  Harrison Jr., John Hussey, Jill Lacey, Won Lee, Krista Loose, Meredith  Moles, Anna Maria Ortiz, Steven Putansu, Alan Rozzi, Cindy Saunders,  Stephanie Shipman, Shane Spencer, Andrew J. Stephens, and Brian  Wanlass also made key contributions. Ann Czapiewski and Donna Miller  developed the graphics for this report. John Ahern, Divya Bali, Jeff  DeMarco, Alexandra Edwards, Ellen Grady, Jyoti Gupta, Erinn L. Sauer,  and Katherine Wulff verified the information presented in this report.

Prior Summary Reports on the Government Performance and Results Act (GPRA) Modernization Act (GPRAMA) Implementation

Managing for Results: Implementation of GPRA Modernization Act Has  Yielded Mixed Progress in Addressing Pressing Governance Challenges.  GAO-15-819. Washington, D.C.: September 30, 2015.

Managing For Results: Executive Branch Should More Fully Implement  the GPRA Modernization Act to Address Pressing Governance  Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Results of the Periodic Surveys on Organizational Performance and Management Issues

Supplemental Material for GAO-17-775: 2017 Survey of Federal  Managers on Organizational Performance and Management Issues.  GAO-17-776SP. Washington, D.C.: September 29, 2017.

Program Evaluation: Annual Agency-wide Plans Could Enhance  Leadership Support for Program Evaluations. GAO-17-743. Washington,  D.C.: September 29, 2017.

Managing for Results: Agencies’ Trends in the Use of Performance  Information to Make Decisions. GAO-14-747. Washington, D.C.:  September 26, 2014.

Managing for Results: Executive Branch Should More Fully Implement  the GPRA Modernization Act to Address Pressing Governance  Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Managing for Results: 2013 Federal Managers Survey on Organizational  Performance and Management Issues, an E-supplement to GAO-13-518.  GAO-13-519SP. Washington, D.C.: June 26, 2013.

Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation  in Program Management and Policy Making. GAO-13-570. Washington,  D.C.: June 26, 2013.

Government Performance: Lessons Learned for the Next Administration  on Using Performance Information to Improve Results. GAO-08-1026T.  Washington, D.C.: July 24, 2008.

Government Performance: 2007 Federal Managers Survey on  Performance and Management Issues, an E-supplement to  GAO-08-1026T. GAO-08-1036SP. Washington, D.C.: July 24, 2008.

Results-Oriented Government: GPRA Has Established a Solid  Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.:  March 10, 2004.

Managing for Results: Federal Managers’ Views on Key Management  Issues Vary Widely Across Agencies. GAO-01-592. Washington, D.C.:  May 25, 2001.

Managing for Results: Federal Managers’ Views Show Need for Ensuring  Top Leadership Skills. GAO-01-127. Washington, D.C.: October 20,  2000.

The Government Performance and Results Act: 1997 Governmentwide  Implementation Will Be Uneven. GAO/GGD-97-109. Washington, D.C.:  June 2, 1997.

Reports Related to GPRAMA Implementation

Federal Programs: Information Architecture Offers a Potential Approach  for Inventory Development. GAO-17-739. Washington, D.C.: September  28, 2017.

Managing for Results: Selected Agencies’ Experiences in Implementing  Strategic Reviews. GAO-17-740R. Washington, D.C.: September 7, 2017.

Federal Reports: OMB and Agencies Should More Fully Implement the  Process to Streamline Reporting Requirements. GAO-17-616.  Washington, D.C.: July 14, 2017.

Open Innovation: Executive Branch Developed Resources to Support  Implementation, but Guidance Could Better Reflect Leading Practices.  GAO-17-507. Washington, D.C.: June 8, 2017.

Performance Partnerships: Agencies Need to Better Identify Resource  Contributions to Sustain Disconnected Youth Pilot Programs and Data to  Assess Pilot Results. GAO-17-208. Washington, D.C.: April 18, 2017.

Open Innovation: Practices to Engage Citizens and Effectively Implement  Federal Initiatives. GAO-17-14. Washington, D.C.: October 13, 2016.

Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early  Implementation and Inform Future Federal Efforts. GAO-16-818.  Washington, D.C.: September 21, 2016.

Performance.gov: Long-Term Strategy Needed to Improve Website  Usability. GAO-16-693. Washington, D.C.: August 30, 2016.

Tax Expenditures: Opportunities Exist to Use Budgeting and Agency  Performance Processes to Increase Oversight. GAO-16-622.  Washington, D.C.: July 7, 2016.

Managing for Results: Agencies Need to Fully Identify and Report Major  Management Challenges and Actions to Resolve them in their Agency  Performance Plans. GAO-16-510. Washington, D.C.: June 15, 2016.

Managing for Results: OMB Improved Implementation of Cross-Agency  Priority Goals, But Could Be More Transparent About Measuring  Progress. GAO-16-509. Washington, D.C.: May 20, 2016.

Managing for Results: Greater Transparency Needed in Public Reporting  on the Quality of Performance Information for Selected Agencies’ Priority  Goals. GAO-15-788. Washington, D.C.: September 10, 2015.

Pay for Success: Collaboration among Federal Agencies Would Be  Helpful as Governments Explore New Financing Mechanisms.  GAO-15-646. Washington, D.C.: September 9, 2015.

Managing for Results: Practices for Effective Agency Strategic Reviews.  GAO-15-602. Washington, D.C.: July 29, 2015.

Managing for Results: Agencies Report Positive Effects of Data-Driven  Reviews on Performance but Some Should Strengthen Practices.  GAO-15-579. Washington, D.C.: July 7, 2015.

Program Evaluation: Some Agencies Reported that Networking, Hiring,  and Involving Program Staff Help Build Capacity. GAO-15-25.  Washington, D.C.: November 13, 2014.

Government Efficiency and Effectiveness: Inconsistent Definitions and  Information Limit the Usefulness of Federal Program Inventories.  GAO-15-83. Washington, D.C.: October 31, 2014.

Managing for Results: Selected Agencies Need to Take Additional Efforts  to Improve Customer Service. GAO-15-84. Washington, D.C.: October  24, 2014.

Managing for Results: Enhanced Goal Leader Accountability and  Collaboration Could Further Improve Agency Performance. GAO-14-639.  Washington, D.C.: July 22, 2014.

Managing for Results: OMB Should Strengthen Reviews of Cross-Agency  Goals. GAO-14-526. Washington, D.C.: June 10, 2014.

Managing for Results: Implementation Approaches Used to Enhance  Collaboration in Interagency Groups. GAO-14-220. Washington, D.C.:  February 14, 2014.

Managing for Results: Leading Practices Should Guide the Continued  Development of Performance.gov. GAO-13-517. Washington, D.C.: June  6, 2013.

Managing for Results: Agencies Should More Fully Develop Priority Goals  under the GPRA Modernization Act. GAO-13-174. Washington, D.C.:  April 19, 2013.

Managing for Results: Agencies Have Elevated Performance  Management Roles, but Additional Training Is Needed. GAO-13-356.  Washington, D.C.: April 16, 2013.

Managing for Results: Data-Driven Performance Reviews Show Promise  But Agencies Should Explore How to Involve Other Relevant Agencies.  GAO-13-228. Washington, D.C.: February 27, 2013.

Managing for Results: A Guide for Using the GPRA Modernization Act to  Help Inform Congressional Decision Making. GAO-12-621SP.  Washington, D.C.: June 15, 2012.

Managing for Results: GAO’s Work Related to the Interim Crosscutting  Priority Goals under the GPRA Modernization Act. GAO-12-620R.  Washington, D.C.: May 31, 2012.

Managing for Results: Opportunities for Congress to Address  Government Performance Issues. GAO-12-215R. Washington, D.C.:  December 9, 2011.

The GPRA Modernization Act was designed to help the federal government address longstanding performance and management problems. Among other things, it requires agency leaders to set goals and use data to review progress toward them.

We surveyed more than 4,000 federal managers and found overall use of performance data in decision making has dropped since 2007. However, the survey shows that data-driven reviews agencies conducted for a subset of goals did improve program performance.

We recommended that OMB work with agencies to identify and share practices for conducting performance reviews for additional goals.