CMS Innovation Center: Model Implementation and Center Performance

The Patient Protection and Affordable Care Act created the Innovation Center within CMS to test new approaches to health care delivery and payment—known as models—for use in Medicare, Medicaid, or CHIP. The Innovation Center became operational in November 2010. In 2012, GAO reported on the early implementation of the Innovation Center. GAO found that, during the first 16 months of operations, the Innovation Center focused on implementing 17 new models and developed preliminary plans for evaluating the effects of each model and for assessing the center's overall performance.

GAO was asked to update its previous work. In this report, GAO: (1) describes the status of payment and delivery models implemented and the resources used; (2) describes the center's use of model evaluations; and (3) examines the center's assessment of its own performance. GAO reviewed available documentation, such as model fact sheets and frequently asked questions, and evaluation reports for models that have been implemented. GAO reviewed obligation data and performance information for the time period for which complete data or information were available. GAO also interviewed officials from the Innovation Center and CMS's Office of the Actuary.

The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate.

As of March 1, 2018, the Center for Medicare and Medicaid Innovation (Innovation Center) had implemented 37 models that test new approaches for delivering and paying for health care with the goal of reducing spending and improving quality of care. These models varied based on several characteristics, including the program covered—Medicare, Medicaid, the Children's Health Insurance Program (CHIP), or some combination of the three—and the nature of provider participation—voluntary or mandatory. Going forward, the Innovation Center indicated that the center plans to continue focusing on the use of voluntary participation models and to develop models in new areas, including prescription drugs, Medicare Advantage, mental and behavioral health, and program integrity. Through fiscal year 2016, the Innovation Center obligated $5.6 billion of its $10 billion appropriation for fiscal years 2011 through 2019.

The Innovation Center has used evaluations of models (1) to inform the development of additional models, (2) to make changes to models as they are implemented, and (3) to recommend models for expansion. For example, Innovation Center officials noted that, for some instances where evaluations have shown reduced spending with maintained or improved quality of care, the center has developed new models that build upon the approaches of earlier models, but with adjustments intended to address reported limitations. In addition, the Innovation Center used evaluations to recommend two models to the Centers for Medicare & Medicaid Services (CMS) Office of the Actuary for certification for expansion. According to CMS officials, a model evaluation and a certification for expansion differ in that a model evaluation assesses the impact of a delivery and payment approach for model participants only, while a certification for expansion assesses the future impact on program spending more broadly across all beneficiaries, payers, and providers who would be affected by the expanded model. As a result, the Office of the Actuary used the results of the evaluation and other information, such as Medicare claims data and published studies, to certify the expansion of both models.

To assess the center's overall performance, the Innovation Center established performance goals and related measures and reported meeting its targets for some goals in 2015, the latest year for which data were available (see table below).

Innovation Center officials told GAO that the center also recently developed a methodology to estimate a forecasted return on investment for its model portfolio. The center is in the early stages of refining the methodology and applying it broadly across its models.

Federal spending on health care in the United States—driven primarily by  Medicare and Medicaid expenditures—is expected to reach over $1  trillion in 2018 and to continue increasing and exerting pressure on the  federal budget. At the same time, studies have found that higher levels  of spending do not reliably lead to enhanced quality of care. The Centers  for Medicare & Medicaid Services (CMS), within the Department of Health  and Human Services (HHS), has sought to both reduce spending and  improve quality of care for beneficiaries enrolled in Medicare, Medicaid,  and the Children’s Health Insurance Program (CHIP) by testing new ways  for delivering and paying for health care services. To further such testing,  the Patient Protection and Affordable Care Act (PPACA) established the  Center for Medicare and Medicaid Innovation (Innovation Center) within  CMS under section 1115A of the Social Security Act.

In establishing the Innovation Center, the law provided CMS with  additional authority when testing new health care delivery and payment  approaches, known as models. For example, CMS may expand the  duration and scope of models tested by the Innovation Center through  rulemaking instead of needing the enactment of legislation, which was  required to expand the demonstrations that CMS frequently conducted in  the past. In addition, the law provided a dedicated appropriation for  testing models—$10 billion for the Innovation Center’s activities for the  period of fiscal years 2011 through 2019 and $10 billion per decade  beginning in fiscal year 2020.

In November 2012, we reported on the early activities of the Innovation  Center. We found that, during the first 16 months of operations, the  Innovation Center focused on implementing 17 new models while  assuming responsibility for 20 demonstrations that CMS began before the  start of the center. We also reported that the Innovation Center developed  preliminary plans for evaluating the effects of each model on spending  and quality of care and assessing the center’s overall performance.

At the time of our 2012 report, however, it was too early to consider  certain questions raised by members of Congress about Innovation Center operations, including the use of its dedicated funding, the impact  of the models tested, and the center’s overall performance. Given the  amount of time that has passed—the Innovation Center has been in  operation for over 7 years—you asked us to update our previous work to  provide information on the activities of the center and to report on any  results of the testing. This report examines  1.  the status of the Innovation Center’s testing of models and the  resources used for such activities;  2.  the use of model evaluations; and  3.  the Innovation Center’s assessment of its performance.

To determine the status of model testing and the resources used by the  Innovation Center for such activities, we reviewed Innovation Center  documentation, including information on models the center was  implementing or had announced, as well as web pages, model fact  sheets, and frequently asked questions. We obtained and analyzed  Innovation Center data on the amounts of the Innovation Center’s  appropriations obligated. We also interviewed and obtained written  responses from Innovation Center officials. Our work focused on models  tested and funded through appropriations under section 1115A of the  Social Security Act, as enacted by PPACA, which established the center  and provided its dedicated appropriations. In general, our work covered  the period during which the Innovation Center first became operational  (fiscal year 2011) through the most recent time period for which complete  information was available. For the status of model testing, we considered  information through March 1, 2018. For the resources used, we analyzed  data on the amounts of the Innovation Center’s appropriations obligated  through fiscal year 2016. We assessed the reliability of the obligation data  by comparing it to publicly reported amounts and discussing the data with  center officials. We determined these data were sufficiently reliable for the  purposes of our objectives.

To determine how the Innovation Center used evaluations of models, we  interviewed officials from the center, CMS’s Office of the Actuary,  evaluation contractors, and subject matter experts to discuss the use of  evaluations, in general, as well as for five selected models specifically.  We selected models based on a nonprobability sample that included both  Medicare and Medicaid models; ongoing and completed models; models  that fell under the responsibility of different Innovation Center staffing  groups; and one model evaluated for expansion. Because we used a  nonprobability sample, our results are not generalizable beyond the  models we reviewed; however, they provide insight into how CMS uses  the evaluations of its models. We also analyzed publicly available  evaluation reports and other model documentation publicly available from  the Innovation Center and the Office of the Actuary.

To describe the Innovation Center’s assessment of its performance, we  reviewed information reported on the center’s targeted and actual  performance available in CMS’s Congressional Budget Justifications for  fiscal years 2012 through 2018. Information on the center’s targets was  available for performance years 2014 through 2018. Complete  information on the center’s actual performance was available for 2015.  Partial information was available for 2014 and 2016, and no information  was available for 2017 and 2018. We also interviewed Innovation Center  officials regarding the assessment of performance.

We conducted this performance audit from February 2017 to March 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Requirements for Innovation Center Models Implemented under Section 1115A

Section 1115A establishes certain requirements for the Innovation Center  that relate to the selection of models, use of resources, and evaluation of  models. These requirements include:  consulting with representatives of relevant federal agencies, as well  as clinical and analytical experts in medicine or health care  management, when carrying out its duties as described in the law; ensuring models address deficits in care that have led to poor clinical  outcomes or potentially avoidable spending; making no less than $25 million of the Innovation Center’s dedicated  funding available for model design, implementation, and evaluation  each fiscal year starting in 2011; evaluating each model to analyze its effects on spending and quality  of care, and making these evaluations public; and modifying or terminating a model any time after testing and evaluation  has begun unless it determines that the model either improves quality  of care without increasing spending levels, reduces spending without  reducing quality, or both.

Under section 1115A, certain requirements applicable to previous CMS  demonstrations are inapplicable to models tested under the Innovation  Center. For example, while prior demonstrations generally required  congressional approval in order to be expanded, section 1115A allows  CMS to expand Innovation Center models—including on a nationwide  basis—through the rulemaking process if the following conditions are met:  (1) the agency determines that the expansion is expected to reduce  spending without reducing the quality of care, or improve quality without  increasing spending; (2) CMS’s Office of the Actuary certifies that the  expansion will reduce or not increase net program spending; and (3) the  agency determines that the expansion would not deny or limit coverage or  benefits for beneficiaries. In addition, certain requirements previously  cited by the Medicare Payment Advisory Commission as administrative  barriers to the timely completion of demonstrations are inapplicable.  Specifically, section 1115A provides the following:

HHS cannot require that an Innovation Center model initially be  budget neutral—that is, designed so that estimated federal  expenditures under the model are expected to be no more than they  would have been without the model—prior to approving a model for  testing.

Certain CMS actions in testing and expanding Innovation Center  models cannot be subject to administrative or judicial review.

The Paperwork Reduction Act—which generally requires agencies to  submit all proposed information collection efforts to the Office of  Management and Budget (OMB) for approval and provide a 60-day  period for public comment when they want to collect data on 10 or  more individuals—does not apply to Innovation Center models.

Innovation Center Staffing and Organization

The Innovation Center uses a combination of staff and contractors to test  models. Since the center became operational in November 2010, the  number of staff increased steadily through the end of fiscal year 2016.  (See fig. 1.) As of September 30, 2017, there were 617 staff—a slight  decrease in the number of staff from the end of the prior fiscal year.  Officials indicated that, in the future, changes in the model portfolio may  require additional staff to manage and support model development and  implementation. However, officials do not anticipate needing to increase  staffing levels at the same pace as they did between fiscal years 2011  and 2016. Additionally, the Innovation Center uses third-party contactors  to perform functions related to the implementation of models and to  perform evaluations of the changes in the quality of care furnished and  program spending under a model.

The Innovation Center has organized its 617 staff members primarily into  eight groups and the Office of the Director. Four of the eight groups are  responsible for coordinating the development and implementation of  models. Staff in these four groups primarily lead efforts in developing  model designs and obtaining approval for their models from CMS and  HHS. Once a model is approved, staff coordinate the remaining  implementation steps, including soliciting and selecting participants and  overseeing the model during the testing and evaluation period. The other  four groups perform key functions that support model development and  implementation, such as reviewing ideas submitted for consideration as  possible models, overseeing the evaluations of models, providing  feedback to model participants about their performance, disseminating  lessons learned across models, and monitoring budget resources. The  Office of the Director, in general, has oversight responsibilities for the  models led by these groups. Table 1 provides information on the staffing  groups within the Innovation Center.

Innovation Center Process for Model Development and Implementation

The Innovation Center has developed internal agency guidance that  outlines a general process used by the four model groups for developing  and implementing models. (See fig. 2.) Appendix I provides additional  information about the general process for implementing models.

The Innovation Center has organized its models into seven categories  based on delivery and payment approaches tested and program  beneficiaries covered. The seven categories are as follows:

Accountable Care. This category includes models built around  accountable care organizations (ACOs)—groups of coordinated  health care providers who are held responsible for the care of a group  of patients. The models are designed to encourage ACOs to invest in  infrastructure and care processes for improving coordination,  efficiency, and quality of care for Medicare beneficiaries.

Episode-based payment initiatives. This category includes models  in which providers are held accountable for the Medicare spending  and quality of care received by beneficiaries during an “episode of  care,” which begins with a health care event (e.g., hospitalization) and  continues for a limited time after.

Initiatives Focused on Medicare-Medicaid Beneficiaries. This  category includes models focused on better serving individuals  eligible for both Medicaid and Medicare in a cost-effective manner.

Initiatives Focused on Medicaid and CHIP Populations. This  category includes models administered by participating states to lower  spending and improve quality of care for Medicaid and CHIP  beneficiaries.

Initiatives to Accelerate the Development and Testing of New  Payment and Service Delivery Models. This category includes  models where the Innovation Center works with participants to test  state-based and locally developed models, covering Medicare  beneficiaries, Medicaid beneficiaries, or both.

Initiatives to Speed the Adoption of Best Practices. This category  includes models in which the Innovation Center collaborates with  health care providers, federal agencies, and other stakeholders to test  ways of disseminating evidence-based best practices that improve  Medicare spending and quality of care for beneficiaries.

Primary Care Transformation. This category includes models that  use advanced primary care practices—also called “medical homes”— to emphasize prevention, health information technology, care  coordination, and shared decision-making among patients and their  providers.

For certain categories, the Innovation Center assigns primary  responsibility for developing and implementing models to a single model  group; for some other categories, the responsibility is shared across  different groups. For example, the center assigned responsibility for  models in the ACO and the Primary Care Transformation categories to  the Seamless Care Model Group, whereas the responsibility for models in  the Initiatives to Accelerate the Development and Testing of New  Payment and Service Delivery Models categories were assigned across  all four model groups. Appendix II provides a summary of the number of  models organized under each category and a description of each model.

The Innovation Center Implemented 37 Models That Test Varying Delivery and Payment Approaches, and Obligated over $5.6 Billion

The Innovation Center Implemented 37 Models and Announced an Additional 2; Models Varied by Delivery and Payment Approaches Tested, Beneficiaries Covered, and Other Characteristics

As of March 1, 2018, the Innovation Center had implemented 37 models  under section 1115A of the Social Security Act. (See fig. 3.) Of those 37  models, the testing period has concluded for 10 of them. In addition, the  Innovation Center has announced two models to begin testing in 2018.

Innovation Center models varied based on several characteristics,  including delivery and payment approaches tested and program(s)  covered. Delivery and payment approaches varied across all  implemented and announced models—even models organized by the  Innovation Center under the same model category. For example, the six  models that tested an episode-based payment approach varied in terms  of how episodes were defined, including the clinical and surgical episodes  to which models applied. In addition, some models included multiple  approaches for achieving changes in health care delivery or payment.  Models also differed in terms of the programs covered, with 22 models  covering Medicare only, 9 models covering Medicare and Medicaid, one  model covering Medicaid and CHIP, and 7 models covering all three  programs. Other characteristics by which models varied include the  nature of model participation for providers (voluntary or mandatory) and  the source of innovation (i.e., federal, state, or local initiatives). See table  2 for a breakdown of models across selected characteristics. Appendix II  provides a full description of all models implemented and announced by  the Innovation Center.

In September 2017, the Innovation Center provided some insight into its  future plans when it issued an informal “request for information” that  identified guiding principles under which models will be designed going  forward, described focus areas for new models, and requested feedback  from stakeholders. One of the guiding principles focused on voluntary  models—a principle consistent with a final rule published in December  2017 canceling four mandatory participation models in development and  making participation in a fifth mandatory model voluntary for some  geographic areas. Other guiding principles included promoting  competition based on quality, outcomes, and costs; empowering  beneficiaries, their families, and caregivers to take ownership of their  health; and using data-driven insights to ensure cost-effective care that  also leads to improvements in beneficiary outcomes. In addition, the  Innovation Center indicated the following focus areas for new model  development: additional advanced alternative payment models;  consumer-directed care and market-based innovation models; physician  specialty models; prescription drug models; Medicare Advantage  innovation models; state-based and local innovation, including Medicaid- focused models; mental and behavioral health models; and program  integrity.

The Innovation Center Obligated over 55 Percent of Its Initial Multiyear Appropriation through Fiscal Year 2016

According to Innovation Center documentation, through September 30,  2016, the center obligated over $5.6 billion of the $10 billion appropriated  for fiscal years 2011 through 2019 under section 1115A of the Social  Security Act. The obligated amounts for individual models during this  period ranged from $8.4 million to over $967 million, and varied based on  model scope and design. For example, a model where the Innovation  Center used its waiver authority to provide additional flexibility to  participants (rather than additional funding) required only $8.4 million in  obligations for the evaluation of the model and implementation activities.  In contrast, a model where the Innovation Center awarded funding to a broad set of partners, including providers, local government, and public- private partnerships, to test their own care delivery and payment models  required more than $870 million in obligations for payments to awardees  and used over $95 million for contractor evaluations and other activities  that supported model development and implementation.

Innovation Center spending falls into three categories: model programs,  innovation support, and administration.

Model programs include obligations that directly support individual  models and delivery system reform initiatives.

Innovation support includes center-wide operational expenses that are  not directly attributable to a single model.

Administration includes permanent federal full-time equivalent payroll  expenses, administrative contracts, administrative interagency  agreements, and general administrative expenses.

As the Innovation Center implemented additional models each year, total  annual obligations increased steadily from approximately $95 million in  fiscal year 2011 to more than $1.3 billion in fiscal year 2015, but  decreased slightly in fiscal year 2016. (See fig. 4) Most of these total  obligations were for model programs, which followed a similar pattern,  increasing from $51 million in 2011 to about $1.1 billion in fiscal year  2015, with a slight decrease in fiscal year 2016. According to officials, the  2016 decrease in obligations for model programs was due in part to some  of the earlier, expensive models ending and to newer models being less  costly than the older models. Officials noted, for example, that a number  of newer models incorporated basic program infrastructure used in  previously implemented models, which allowed for reduced model costs.  Officials also indicated that the decrease in obligations may be due to  newer models using payment approaches that are funded by the  Medicare Trust Fund, rather than funded by the Innovation Center’s  dedicated appropriation. The center’s obligations for both innovation  support and administration increased from around $20 million for each  category in fiscal year 2011 to about $163 million for innovation support  and $119 million for administration in fiscal year 2016. Officials told us  that as obligations for model programs grew, so did obligations for  innovation support and administration, which includes indirect costs and  contractor assistance.

Evaluations Inform the Development of Models and Decisions to Certify Certain Models for Expansion

The Innovation Center Has Used the Results from Evaluations to Inform the Development of Additional Models and to Make Changes to Implemented Models

The Innovation Center has used the results from model evaluations to  generate ideas for new models. For some of the early implemented  models, evaluation results showed reduced spending and maintained or  improved quality of care, but also identified model design limitations that  could affect those results. According to officials, in some of these  instances, the Innovation Center has developed new models that build  upon the approaches of earlier models, but include adjustments intended  to address identified limitations (see text box).

Evaluations of Implemented Models  The evaluation of each model is performed by  a third-party contractor, who generally  determines the effect of a model on quality of  care and program spending by comparing  data for model participants to those of a  comparison group of providers and  beneficiaries with characteristics similar to  model participants. For purposes of the  evaluation, the Innovation Center has the  authority to require the collection and  submission of necessary data by model  participants. Accordingly, the third-party  contractor collects both quantitative and  qualitative data. The quantitative data are  used to assess program spending and quality  of care and the qualitative data are used to  provide the context needed to understand the  quantitative results.

Example of A Model That Tests the Same General Delivery and Payment Approach  of a Previously Implemented Model While Addressing Limitations  Bundled Payment for Care Improvement (BPCI) Model 2 tested an episode-based  delivery and payment approach in which the Innovation Center set a benchmark, or  target, price for all Medicare services a beneficiary might receive during a clinical  episode—defined by BPCI Model 2 as the initial hospital stay and all services received up  to 90 days after discharge. If the total spending for Medicare services during an episode  was lower than the target price, participating hospitals would receive payments in  addition to the normal fee-for-service payments. If the total spending for Medicare  services during an episode was higher than the target price, participating hospitals would  have to reimburse Medicare. Participants could select up to 48 different clinical episodes  under the model.   The evaluation of BPCI Model 2 found that orthopedic surgery episodes—of which  approximately 90 percent were hip and knee joint replacement surgeries—may have  resulted in reduced program spending and improved quality of care. However, the  evaluation also identified limitations affecting those results. For example, the target prices  for hip and knee replacement surgeries did not account for potential differences in  Medicare spending between elective surgeries and surgeries required after a fracture. As  a result of this limitation, hospitals could attempt to control spending by limiting the  number of episodes associated with higher cost beneficiaries (i.e., those requiring  surgery due to a fracture).  In part to address the design issue identified under BPCI Model 2, Innovation Center  officials told us they developed the Comprehensive Care for Joint Replacement (CJR)  model. Implemented in April 2016, the CJR model tests the same general delivery and  payment approach used in BPCI Model 2, but focuses specifically on hip and knee joint  replacement surgical episodes and adjusts the target price to account for the higher  spending related to hip and knee joint replacement surgeries following a fracture. As of  March 1, 2018, no evaluations of the CJR model have been publicly released.

The Innovation Center has also used the results from evaluations as one  way to improve the operational and participant support for new models.  According to officials, evaluations have helped them identify lessons  learned regarding support systems, such as which types of systems work  well with which types of models, and then the center incorporated those  lessons when designing the systems for new models. For example,  officials noted that the experience with the learning system from the  Bundled Payments for Care Improvement (BPCI) models informed the  learning system for the Comprehensive Care for Joint Replacement  (CJR) model. The lessons learned helped the Innovation Center better  identify where participants would need additional support and the learning  activities—such as webinars and implementation guides—to provide the  needed support during the early stages of model implementation.  Innovation Center officials told us that these lessons from evaluations  helped ensure that each successive model built upon the collective  experience of models implemented by the center.

The Innovation Center also has used evaluation results to make periodic  changes to models during the testing period. According to officials, these  changes include adjustments to the delivery and payment approaches  tested, such as refining the target population, broadening the geographic  focus, and refinements of spending calculations. Innovation Center  officials noted that, in general, such changes were limited to minimize  their effects on the evaluation of program spending and quality of care.  Officials also identified changes to operational and participant support  systems, which have included changes to the timing of participant data  reporting, revisions to how data are collected from participants, and  changes to the way learning materials are delivered to participants.  According to officials, these types of changes are generally intended to  help improve the experience of participants.

According to Innovation Center officials, evaluation results may also be  used in making a decision to terminate a model prior to the end of its  planned testing period. However, officials stated that the Innovation  Center has not terminated any models prior to the conclusion of their  testing periods, either based on the results of an evaluation or for other  reasons.

Evaluations Informed Innovation Center Decisions to Recommend Two Models be Certified for Expansion

The Innovation Center used evaluation results in recommending two  models be certified for expansion. According to Innovation Center  officials, the evaluation of each model adequately demonstrated that the  delivery and payment approach tested reduced Medicare spending while  maintaining or improving quality of care. Based on these results, the  Innovation Center formally requested that CMS’s Office of the Actuary  analyze the financial impact of a potential expansion of each model. The  two models were:

Pioneer ACO. Pioneer ACO tested an ACO delivery and payment  approach that gave providers an opportunity to be paid a relatively  greater share of savings generated, compared to participants in other  ACO models, in exchange for accepting financial responsibility for any  losses. In year 3 of the model, ACOs that met certain levels of savings  in the first two years could elect to receive a portion of their Medicare  fee-for-service payments in the form of predetermined, per beneficiary  per month payments.

YMCA of the USA Diabetes Prevention Program (Diabetes  Prevention Program). The Diabetes Prevention Program applied a  lifestyle change program recognized by the Centers for Disease  Control and Prevention to reduce to the risk of Type 2 diabetes for at- risk Medicare beneficiaries. The Diabetes Prevention Program was a  part of the Health Care Innovation Awards Round One model.

When assessing the Pioneer ACO and Diabetes Prevention Program  models for expansion, the officials from the Office of the Actuary  considered the model evaluation results that were available and  information from other sources. For example, the assessment of  Pioneer ACO used historical shared savings calculations and beneficiary  attribution data from ACOs in the Medicare Shared Saving Program and  Pioneer ACO; Medicare claims and enrollment data; and published  studies. According to CMS officials, a model evaluation and a certification  for expansion differ in that a model evaluation assesses the historical  impact of a delivery and payment approach for model participants only,  while a certification for expansion assesses the future impact on program  spending across all beneficiaries, payers, and providers who would be  affected by the expanded model.

Based on its assessments, the Office of the Actuary certified both models  for expansion and steps have been taken to expand them. In certifying  Pioneer ACO, the Office of the Actuary concluded that because ACOs, in  general, have been shown to produce savings relative to Medicare fee- for-service, an expansion of Pioneer ACO would generate further savings  to the Medicare program. According to officials, CMS expanded Pioneer  ACO by incorporating elements of the model—through rulemaking—as  one of the options that providers may choose under the Medicare Shared  Savings Program. For the Diabetes Prevention Program, the Office of  the Actuary concluded that certain changes considered as part of the  expansion would, in the near term, improve upon the original savings  achieved as part of the Health Care Innovation Awards as well as savings  achieved in similar diabetes prevention programs. The Innovation Center  has expanded—through rulemaking—the Diabetes Prevention Program  under a new, nationwide model to be implemented in April 2018.

In addition, officials from the Innovation Center and the Office of the  Actuary discussed potentially assessing whether Partnership for Patients  should be certified for expansion. Partnership for Patients is a model that  leveraged federal, state, local, and private programs to spread proven  practices for reducing preventable hospital-acquired conditions and  readmissions across acute care hospitals. According to officials, the  Innovation Center shared the results for Partnership for Patients—which  showed improved quality of care in the form of reduced preventable  hospital-acquired conditions and readmissions—with the officials from the  Office of the Actuary. After discussing these issues, Innovation Center  officials decided not to request a formal analysis for certification of  expansion.

The Innovation Center Established Performance Goals and Related Performance Measures and Reported Meeting Its Targets for Some Goals

To assess is own performance, the Innovation Center established three  center-wide performance goals and related measures.

Goal 1: Reduce the growth of healthcare costs while promoting  better health and health care quality through delivery system reform.  This goal has three performance measures that focus on ACOs. As  shown in table 3, the Innovation Center has reported mixed results in  achieving the targets set. According to agency reported data, the  Innovation Center met the targets for 2 of its 3 Goal 1 performance  measures for 2015. For the remaining measure—the percentage of ACOs  that shared in savings—the center did not meet its target during either of  the two years for which data were available. According to officials, when  results fall short of targets, they examine the causes and make  appropriate adjustments to the program. Officials stated that the missed  target was driven by the high growth in the number of ACOs that were  new—and therefore would not yet be expected to achieve a level of  savings in which they could share—and not by ACO performance deficits.  As a result, officials decided that no adjustments were required to the  Medicare Shared Savings Program or other ACO Models to help improve  performance. However, as shown in table 3, the Innovation Center set a  target for 2016 that was lower than the 2015 target. For 2017, the  Innovation Center lowered the expectation for growth compared to  previous years, setting a target that was 1 percent higher than the 2016  target. Moving forward, CMS believes that as more ACOs gain  experience, more will share in savings. Additionally, the agency expects  that with additional performance years, the targets for the measure will  become more refined.

Goal 2: Identify, test, and improve payment and service delivery  models. This goal has one performance measure, which identifies the  number of models that currently indicate (1) cost savings while  maintaining or improving quality or (2) improving quality while maintaining  or reducing cost. As of September 30, 2016, the Innovation Center  reported that four section 1115A model tests have met these criteria (see  table 4).

Goal 3: Accelerate the spread of successful practices and models.  For this goal, the first performance measure focuses on the number of  states developing and implementing a health system transformation and  payment reform plan. The second measure focuses on increasing the  percentage of active model participants who are involved in Innovation  Center or related learning activities. As shown in table 5, the Innovation  Center reported meeting its target for the first measure for both fiscal  years 2015 and 2016, but not meeting its target for the second measure.  For the second measure, the Innovation Center noted in its report to  Congress that although the results for fiscal year 2016 showed a slight  decrease in overall participation in Innovation Center or related learning  activities, the majority of models performed higher than their individual  targets. Several models underperformed, however, bringing down the  overall percentage rate.

In addition to the Goal 3 performance measures, the Innovation Center  identifies two related contextual indicators—which according to officials  are measures that provide supporting information to help understand  trends or other information related to the goal. The first contextual  indicator provides a snapshot of Medicare beneficiary participation at a  given point in time for all models operational for more than 6 months. In  fiscal year 2016, CMS reported that over 3.6 million Medicare fee-for- service beneficiaries participated in models, representing approximately 9  percent of Medicare fee-for-service beneficiaries. The second contextual  indicator provides information to help understand the level of interest and  participation among providers in the Innovation Center’s model portfolio.  In fiscal year 2016, the Center estimates that 103,291 providers  participated in Innovation Center payment and service delivery models.

In addition to the three goals established by the Innovation Center, CMS  has established an agency-wide goal related to the center’s performance.  In 2015, CMS announced goals to help drive Medicare, and the health  care system at large, toward rewarding the quality of care instead of the  quantity of care provided to beneficiaries. One of these goals was to shift  Medicare health care payments from volume to value using alternative  payment models established under the Innovation Center. This agency- wide goal has one performance measure, which is to increase the  percentage of Medicare fee-for-service payments tied to alternative  payment models, such as ACOs or bundled payment arrangements. As  shown in table 6, CMS reported meeting its target for 2015 and 2016.

Looking forward, officials told us that the Innovation Center has  developed a methodology to estimate a forecasted return on investment  for the model portfolio, and is in the early stages of refining the  methodology and applying it broadly across the portfolio in 2018. As part  of the development efforts, the Innovation Center expects to utilize  standard investment measures used in the public and private sectors.

We provided a draft of this report to HHS for comment. The Department  provided technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of Health  and Human Services. In addition, the report will be available at no charge  on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or kingk@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix IV.

Appendix I: Center for Medicare and Medicaid Innovation’s General Process for Implementing Models

Appendix I: Center for Medicare and Medicaid  Innovation’s General Process for Implementing  Models An agency may issue a request for information for planning purposes.

Appendix II: Models Implemented or Announced by the Center for Medicare and Medicaid Innovation under Section 1115A

As of March 1, 2018, the Center for Medicare and Medicaid Innovation  (Innovation Center) organized its models into seven categories based on  delivery and payment approaches tested and program beneficiaries  covered. Table 8 provides the number of models implemented and  announced, organized under each category.

The Innovation Center organized seven of its models under the  Accountable Care category. (See table 9.)

The Innovation Center organized seven of its models under the Episode- Based Payment Initiatives category. (See table 10.)

The Innovation Center organized three of its models under the Initiatives  Focused on Medicare-Medicaid Enrollees category. (See table 11.)

The Innovation Center organized one of its models under the category,  Initiatives Focused on the Medicaid and Children’s Health Insurance  Program Population. (See table 12.)

The Innovation Center organized 14 of its models under the category,  Initiatives to Accelerate the Development and Testing of New Payment  and Service Delivery Models. (See table 13.)

The Innovation Center organized three of its models under the category,  Initiatives to Speed the Adoption of Best Practices. (See table 14.)

The Innovation Center organized four of its models under the category,  Primary Care Transformation. (See table 15.)

Appendix III: Models Required by Different Provisions of the Patient Protection and Affordable Care Act

In addition to models required by section 1115A of the Social Security  Act, as added by the section 3021 of Patient Protection and Affordable  Care Act, the Center for Medicare and Medicaid Innovation implemented  six models under different provisions of the Patient Protection and  Affordable Care Act. (See table 16.)

Appendix IV: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Greg Giusto (Assistant Director),  Aaron Holling (Analyst-in-Charge), Ashley Dixon, and Rachel Rhodes  made key contributions to this report. Also contributing to the report were  Sam Amrhein, Muriel Brown, and Emily Wilson.

Federal spending on health care—driven primarily by Medicare and Medicaid—is expected to top $1 trillion in 2018.

The Affordable Care Act created the CMS Innovation Center to test new approaches to health care delivery, known as models, that could curb spending while providing better care. We reviewed the center's efforts.

Since 2010, the center has tested or is in the process of testing 37 models. In evaluating these models, the Innovation Center has recommended the expansion of 2 Medicare models that reduced costs and maintained or improved care quality. Evaluations have also helped the center improve new model design and operation.


Information Technology: OMB Needs to Report On and Improve Its Oversight of the Highest Priority Programs

The federal government plans to spend almost $96 billion on IT investments in fiscal year 2018; however, as GAO has previously found, too often these investments have cost overruns and schedule delays. To enhance oversight of IT programs, for 2015, Congress directed OMB to identify the 10 highest priority IT programs that are under development across federal agencies and report on their status each quarter. Further, for 2016, Congress directed USDS to provide a quarterly report of current USDS projects, including the top 10 high priority programs.

GAO was asked to review OMB's oversight of high priority programs. This review evaluated (1) OMB's process for identifying, overseeing, and reporting on the high priority IT investment programs and (2) USDS's process for identifying and prioritizing its projects, including its consideration of the high priority programs, and its reporting on the projects. GAO analyzed OMB memorandums and reports to Congress and interviewed OMB staff, including from USDS. In addition, GAO compared USDS's processes with IT management best practices.

The Office of Management and Budget's (OMB) Office of E-Government and Information Technology (E-Gov) reported that it undertook a structured approach to identify the top 10 high priority IT programs in reports to Congress in June 2015 and June 2016. Specifically, OMB staff stated that they chose the top 10 programs from a longer list of agency programs requiring additional oversight (referred to as high impact programs). E-Gov staff reported that the high priority programs were already receiving greater oversight than what was provided to the other major programs due to their high impact designation. This additional oversight included frequent meetings with agency Chief Information Officer (CIO) leadership and quarterly meetings with OMB staff. However, the Federal CIO was not directly involved in this oversight. According to E-Gov staff, the Federal CIO does not typically get involved in individual programs due to the large number of programs. However, past experience has shown that Federal CIO involvement has had a significant impact. For example, Federal CIO-led reviews of troubled projects, known as TechStat reviews, resulted in $3 billion in savings in 2010. Until OMB ensures that the Federal CIO is more directly involved in the oversight of these high priority programs, it may be missing a key opportunity to improve accountability and achieve positive results.

OMB's 2015 and 2016 reports to Congress on the top 10 high priority programs identified the status of the programs and major milestones. However, the reports were not issued on a quarterly basis, as directed. E-Gov staff stated that they were unable to do so because of other competing reporting requirements and the limited resources available to draft and fully review the report on a quarterly basis. In addition, OMB stopped issuing the reports on the top 10 high priority IT programs after June 2016. OMB stated that Congress' 2016 direction to the U.S. Digital Service (USDS)—an OMB component—to provide a quarterly report that described the status of USDS teams and projects, including the top 10 high priority programs, meant that OMB should only report on USDS projects considered to be high priority. However, continued identification and reporting on the top 10 high priority programs, and not just USDS projects, would further enhance congressional oversight by providing congressional stakeholders information that is not readily available on those programs in the greatest need of attention.

USDS issued reports to Congress on the status of its key projects in December 2016 and July 2017; however, the reports did not address the top 10 high priority programs as directed by Congress, according to OMB staff, because of OMB's interpretation of Congress's direction. In addition, the reports were not issued quarterly, as directed. USDS staff attributed this to the time and effort needed to review and prepare the report. However, continuing to identify and report on the top 10 high priority programs while also reporting on USDS's projects would help to enhance congressional oversight and current administration IT governance entities' efforts by providing stakeholders with information that is not readily available.

GAO is making three recommendations to OMB for enhancing the oversight of high priority programs and continuing to report on both these programs and USDS projects. OMB neither agreed nor disagreed with the recommendations but disagreed with several of GAO's conclusions, which GAO continues to believe are valid as discussed in the report.

The federal government plans to spend almost $96 billion on information  technology (IT) in fiscal year 2018. However, we have previously found  that the effective and efficient acquisition of IT investments has been a  longstanding challenge in the federal government and that IT  expenditures have often resulted in significant cost overruns, schedule  delays, and questionable mission-related achievements. Due to these  and other issues, in February 2015, we added improving the  management of IT acquisitions and operations to our list of high-risk  areas of concern for federal government agencies.

In addition, to achieve increased oversight of federal IT programs, in  December 2014, Congress stated in its explanatory statement  accompanying the Consolidated and Further Continuing Appropriations  Act, 2015, that the Office of Management and Budget (OMB) is to identify  the 10 highest priority IT investment programs (hereafter referred to as  the top 10 high priority programs) that are under development across  federal agencies and report on their status each quarter. Further, in an  explanatory statement accompanying the Consolidated Appropriations  Act, 2016, Congress stated that the U.S. Digital Service (USDS), a  component of OMB, is to provide a quarterly status report on current  USDS projects, including the top 10 high priority programs.

You asked us to review OMB’s efforts to provide oversight of the top 10  high priority IT programs. Our specific objectives were to evaluate (1)  OMB’s process for identifying, overseeing, and reporting on the high  priority IT investment programs and (2) USDS’s process for identifying  and prioritizing its projects, including its consideration of the high priority  programs, and its reporting on the projects.

To address the first objective, we obtained and reviewed reports that  OMB sent to Congress in June 2015 and June 2016 on the high priority IT  programs to discern any information about the process that was used for  identifying and overseeing these programs. In the absence of  documented procedures specific to the high priority programs, we also  reviewed OMB memorandums addressing the identification and oversight  of high impact programs, which are programs that OMB has determined  merit additional support and oversight by OMB and/or agency leadership  due to factors such as their size and cost. According to OMB staff, these  high impact programs are a broader set of programs from which OMB  selects the top 10 high priority programs.

Further, we reviewed an OMB work plan charting the process that OMB’s  Office of E-Government and Information Technology (E-Gov) used to  identify and oversee high impact programs. We also reviewed  screenshots of a tool that OMB uses to analyze agency IT investment  data as a part of their process to identify and oversee high impact  programs. The tool draws information from IT portfolio summaries that  agencies submit to OMB as part of the budget process. In addition, we  interviewed relevant E-Gov office staff responsible for preparing the top  10 high priority reports, including the Unit Chief for the Agency Oversight  and Implementation team, regarding the processes they used for  identifying the programs and the oversight they provide to such  programs.

We corroborated what we learned from OMB about its oversight of high  priority and high impact programs by obtaining information from eight  agencies identified as having one of the top 10 high priority IT programs  in the June 2016 report to Congress. We asked relevant officials within  these agencies, among other things, how the oversight that OMB  provided to their high priority IT programs had differed from that provided  to other major investments. In addition, we asked the officials whether this  oversight included the programs being discussed with OMB during  quarterly portfolio review meetings known as PortfolioStat. Finally, we  analyzed OMB’s June 2015 and June 2016 reports to Congress  identifying the top 10 high priority IT, which were issued in response to  Congress’s explanatory statement for fiscal year 2015.

To address the second objective, we reviewed USDS’s documentation,  including its January 2017 master project list, and interviewed staff,  including the former Deputy Administrator. In addition, we discussed with  USDS staff the findings from a previous GAO engagement that assessed  the organization’s project selection process to determine any changes  to their process since August 2016. Upon determining that the process  had not changed, we relied on the prior engagement’s evaluation of the  process to determine if it addressed the best practice identified in GAO’s  Information Technology Investment Management Framework. This  framework calls for organizations to use their defined selection process,  including predefined selection criteria, to select their projects. We also  analyzed OMB’s December 2016 and July 2017 reports to Congress  that were issued in response to the explanatory statement for fiscal year  2016, which stated that OMB was to report on the status of USDS’s  projects, including the top 10 high priority IT programs.

We conducted this performance audit from July 2016 to November 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The federal government plans to invest about $96 billion in fiscal year  2018 for IT that is critical to the health, economy, and security of the  nation. However, prior IT expenditures have often resulted in significant  cost overruns, schedule delays, and questionable mission-related  achievements. For example,

The Department of Health and Human Services’ website,  Healthcare.gov, and its supporting systems, which were to facilitate  the establishment of a federal health insurance marketplace by  January 2014, encountered significant cost increases, schedule slips,  and delayed functionality. In a series of reports, we identified  numerous planning, oversight, security, and system development  challenges faced by this program.

For almost two decades, the Department of Veterans Affairs (VA) has  undertaken numerous initiatives with the Department of Defense  (DOD) that were intended to advance the ability of the two  departments to share electronic health records. In our report of the  departments’ efforts in 2015, we reported that the departments had  not identified outcome-oriented goals and metrics to clearly define  what they aimed to achieve from their interoperability efforts, resulting  in numerous failures. During most of the last 20 years, VA has also  been planning to modernize its system separately from DOD.  Recently, the Secretary of VA announced that the department plans to  use the same electronic health record system that DOD is in the  process of acquiring. However, the significant challenges that have  confronted VA in its efforts contributed to our designation of VA health  care as a high risk area.

The Department of Homeland Security’s U.S. Citizenship and  Immigration Services’ Transformation Program, which was initiated to  address processing inefficiencies and transform the agency’s current  paper-based system into an electronic account-based system, has  faced continual management and development challenges, limiting its  progress and ability to achieve its goals of enhanced national security  and system integrity, better customer service, and operational  efficiency. The U.S. Citizenship and Immigration Services estimates  that the program’s cost increased by approximately $1 billion and its  schedule was delayed by over 4 years from its initial approved  baseline.

The Office of Personnel Management’s Retirement Systems  Modernization program, which was intended to improve the efficiency  and effectiveness of its retirement claims processing, was canceled in  February 2011 after the agency had spent approximately $231 million  on a third attempt to automate the processing of federal employee  retirement claims.

As previously stated, due to the challenges associated with acquiring IT  across the federal government, in 2015, we added improving the  management of IT acquisitions and operations to our list of high-risk  areas. We recently issued an update to our high-risk report and  determined that, while progress has been made in addressing the high- risk area of IT acquisitions and operations, significant work remains to be  completed. For example, as of May 2017, OMB and federal agencies  had implemented 380 (or about 47 percent) of the 803 recommendations  that we had made from fiscal years 2010 through 2015 related to IT  acquisitions and operations.

OMB’s Mission and Oversight of Federal IT Programs

By law, OMB is to oversee federal agencies’ management of information  and information technology. Within OMB, primary responsibility for  oversight of federal IT has been given to the Administrator of the Office of  Electronic Government and Information Technology, who is also called  the Federal Chief Information Officer (Federal CIO). According to OMB,  this oversight responsibility covers about 800 major and nearly 5,700  non-major IT investments across the federal government. As a part of  the oversight, the E-Gov office develops policy and reviews federal  agencies’ IT strategic plans. In addition, OMB establishes processes to  analyze, track, and evaluate the risks and results of IT investments made  by executive agencies, and issues guidance on processes for selecting  and overseeing agency privacy and security protections for information  and information systems.

OMB has also implemented a series of initiatives to improve the oversight  of underperforming investments and more effectively manage IT. These  initiatives include the following:

Federal IT Dashboard. In June 2009, to further improve the  transparency into and oversight of federal agencies’ IT investments,  OMB deployed the Federal IT Dashboard, a public website with  information on the performance of these investments. OMB provided  guidance to the agencies on the information they should maintain and  update on the dashboard. Currently, the dashboard displays  information on the cost, schedule, and performance of close to 800  major IT investments at 26 federal agencies. In addition, agencies are  to submit ratings from their CIOs to the dashboard, which, according  to OMB’s instructions, should reflect the level of risk facing an  investment relative to that investment’s ability to accomplish its goals.  The public display of these data is intended to allow OMB, other  oversight bodies, and the general public to hold agencies accountable  for mission-related outcomes. Over the past 7 years, we have issued  a series of reports that have noted both significant steps OMB has  taken to enhance the oversight, transparency, and accountability of  federal IT investments by creating the dashboard, as well as issues  with the accuracy and reliability of the data it contains.

TechStat reviews. In January 2010, the Federal CIO began leading  TechStat reviews—face-to-face meetings to discuss whether to  terminate or turn around IT investments that are in danger of failing or  are not producing results. These meetings involved OMB and agency  leadership and were intended to increase accountability and improve  performance. OMB reported that federal agencies achieved over $3  billion in cost savings or avoidances as a result of these reviews in  2010. Subsequently, it empowered agency CIOs to begin holding  their own TechStat reviews by June 2012. OMB’s 2015 guidance  specified that the TechStat reviews were to be held with agency  leadership, not led by OMB as in the past, and that agencies were  only required to notify OMB of the meetings’ occurrence and report  the results.

In November 2015, we testified that OMB had conducted only one  TechStat review between March 2013 and October 2015, and had not  listed any related savings in its quarterly reporting to Congress since  June 2012. In April 2017, OMB reported that it had not led a  TechStat review since 2015. A report issued by the Federal CIO  Council in January 2017, entitled The State of Federal Information  Technology, noted that shifting TechStat reviews from OMB to  agencies had diminished the executive scrutiny and impact of the  initiative.

PortfolioStat sessions. To better manage existing IT systems, in  2012, OMB launched the PortfolioStat initiative, which required  agencies to conduct an annual, agency-wide portfolio review to,  among other things, reduce commodity IT spending and demonstrate  how their investments aligned with the agency’s mission and business  functions. These reviews were to be held between the Federal CIO  and agency leadership. In 2014 and 2015, OMB’s PortfolioStat  guidance also called for it and agencies to identify high impact IT  programs that merited additional support and oversight by OMB  and/or agency leadership, and for these programs to be discussed  during a PortfolioStat session. The 2015 guidance also changed the  frequency of the PortfolioStat sessions from annually to quarterly, and  the level of participation to no longer require attendance by the  Federal CIO or the agency’s Deputy Secretary.

The Federal Information Technology Acquisition Reform provisions  (commonly referred to as FITARA) enacted as a part of the Carl Levin  and Howard P. ‘Buck’ McKeon National Defense Authorization Act for  Fiscal Year 2015 aimed to improve federal IT acquisition and operations  and recognized the importance of these OMB initiatives by incorporating  certain requirements into the law. For example, among other things, the  act requires OMB to publicly display investment performance information  and review federal agencies’ IT investment portfolios.

Further, as previously mentioned, the December 2014 explanatory  statement for the Consolidated Appropriations Act, 2015, stated that OMB  was to identify the top 10 high priority IT programs under development in  the federal government and report on their status quarterly. Additionally,  in December 2015, in the explanatory statement for the Consolidated  Appropriations Act, 2016, Congress stated that USDS, an OMB  component, was to provide a quarterly status report on its current  projects, including the top 10 high priority programs.

The current mission of USDS is to deliver better government services to  the American people through technology and design. USDS is focused  on, among other things, improving the nation’s most important digital  services used by the public, and modernizing procurement processes and  practices for the digital era. To execute its mission, USDS recruits  private sector experts, such as IT engineers and designers and leading  civil servants, and deploys small teams to federal agencies. It selects  which projects it will apply resources to and generally initiates the effort  with the federal agency that owns the IT projects. In August 2016, we  reported that USDS had developed procedures and criteria for selecting  and prioritizing projects to work on.

Current Administration’s Efforts to Improve Federal IT

The current administration has initiated additional efforts aimed at  improving federal IT, including digital services. Specifically, in March  2017, the administration established the Office of American Innovation,  which has a mission to, among other things, make recommendations to  the President on policies and plans aimed at improving federal  government operations and services and modernizing federal IT. In doing  so, the office is to consult with both OMB and the Office of Science and  Technology Policy. Further, in May 2017, the administration established  the American Technology Council, which has a goal of helping to  transform and modernize federal agency IT and how the federal  government uses and delivers digital services. The Federal CIO and the  USDS Administrator are members of this council.

OMB Identified High Priority IT Programs, but Did Not Ensure Federal CIO Oversight or Continued and Timely Reporting

OMB reported that it undertook a structured approach to identifying the  top 10 high priority IT programs it reported to Congress in June 2015 and  June 2016. Specifically, staff in the E-Gov office, including the Unit Chief  for the Agency Oversight and Implementation team, stated that they  chose the top 10 programs from a list of high impact programs that OMB  separately maintains. They added that their approach was not guided by  any documented procedures or scoring techniques to distinguish the  programs.

Analysts in the E-Gov office told us that, to identify the high impact  programs from which the top 10 high priority programs were selected,  they used program information from IT portfolio summaries, monthly IT  dashboard updates, and quarterly integrated data collection submissions  that OMB receives from agencies. They also considered several  additional factors, such as risk exposure, public impact, public use,  criticality to agency mission, size, and cost. In addition, they considered  input from USDS leadership and OMB budget examiners, as well as our  reports, inspectors general reports, and CIO risk ratings.

Further, the analysts stated that OMB sought input from officials of the 24  Chief Financial Officers Act agencies to gain a better understanding of  the importance of each program. In the end, based on all of the  information considered, E-Gov staff made a judgment call regarding  which of the agencies’ programs to identify as being high impact.  According to these staff, on average, two high impact programs are  identified for each of the 24 agencies, with at least one program being  identified for each agency, and as many as four programs being identified  for some larger agencies. Further, according to the staff, the Federal CIO  approved the process for selecting the high impact programs.

In addition, in determining the top 10 high priority programs, the E-Gov  staff stated that they identified on the high impact list those programs that  they believed were representative of the most important IT programs  across federal agencies. They also considered other factors, such as  whether a program had generated legislative interest or had performance  issues. They added that the Federal CIO then approved the list of top 10  programs that they had selected.

OMB subsequently issued two reports to Congress—in June 2015 and  June 2016—that identified the top 10 high priority programs. Along with  the status of each program, the reports identified the IT investments that  were a part of each program as well as total IT spending, average CIO  risk rating, major milestones, and the level of involvement USDS has with  the program. Table 1 lists the programs that OMB identified in the two  reports.

As shown in the table, OMB made two changes to the top 10 list in 2016.  Specifically, it replaced the Social Security Administration’s Service  Modernization program with the agency’s Disability Case Processing  System due to a change in focus within the agency. In addition, it  replaced VA’s Medical Appointment Scheduling System with VA’s  Medical 21st Century Development Core program, which is a larger effort  that encompasses the scheduling system.

OMB Oversight Efforts Did Not Ensure Federal CIO Involvement

E-Gov staff reported that the high priority programs were already  receiving greater oversight than what was provided to the other major  programs due to their high impact designation. Specifically, the staff  stated that OMB provides additional oversight to high impact IT programs  by regularly communicating about them through quarterly, monthly,  weekly, or daily meetings with agency CIOs and program staff, depending  on the risk and profile of the program. In contrast, communication is much  less frequent for other major IT systems. The staff added they also  discuss high impact programs with agencies during quarterly PortfolioStat  meetings. Further, they stated that they can request that agencies  perform a TechStat review, if needed, for a troubled high impact program.

Most agencies that owned the high priority programs identified in the June  2016 report confirmed that OMB had already provided increased  oversight for the high priority programs when they were originally  designated as high impact programs. Specifically, officials from six of the  eight agencies stated that the programs were discussed during quarterly  PortfolioStat sessions, and officials from five agencies stated that OMB  had provided action items for them to address with regard to their  programs. Further, officials from six agencies stated that OMB’s  oversight included periodic meetings (e.g., daily, weekly, or bi-weekly). Its  oversight also included participation in a TechStat review of a high impact  program performed by the Social Security Administration.

Nevertheless, while additional OMB oversight of the high impact  programs (and, accordingly, the identified high priority programs) is a  positive step, the Federal CIO was not directly involved in this oversight.  According to the E-Gov staff, the Federal CIO does not typically get  involved with overseeing individual IT programs due to the large number  of programs. However, the results of past CIO-led TechStat reviews  suggest that the Federal CIO’s involvement in overseeing such programs  does have significantly positive results. Specifically, as previously  mentioned, CIO-led TechStat reviews of IT investments performed in  2010 resulted in $3 billion in savings and cost avoidance.

Further, during a September 2016 Comptroller General forum to explore  challenges and opportunities for improving federal IT acquisitions and  operations, current and former CIOs and other participants pointed to the  importance of OMB’s oversight and guidance, and specifically, to the role  of the Federal CIO in helping to ensure effective IT governance. The  participants cited specific OMB initiatives undertaken by the Federal CIO,  including the TechStat reviews that had resulted in greater accountability  and positive results. Thus, without the involvement of the Federal CIO  more directly and regularly in the oversight of high impact and high  priority programs, including leading TechStat reviews for the programs,  OMB is likely to miss significant opportunities to improve accountability for  and achieve positive results from the federal government’s IT  investments.

OMB Has Stopped Reporting on High Priority IT Programs

OMB issued two reports on the high priority programs—one in 2015 and  another in 2016. While these reports provided the requested information  they were not issued quarterly. According to the E-Gov staff, OMB was  not able to report quarterly on the programs because of other competing  reporting requirements, limited resources available to draft the report, and  the amount of time it takes to get its reports fully reviewed.

Moreover, the staff stated that they stopped issuing the high priority  reports in 2016 because they believe the explanatory statement to the  Consolidated Appropriations Act, 2016, no longer directed them to  continue reporting on the programs. Specifically, the explanatory  statement directed USDS to provide a quarterly report to the Committees  on Appropriations of the House and Senate describing the status of  current USDS teams and projects, including the top 10 high priority  programs, a list of USDS accomplishments, and agency project  proposals. Both E-Gov staff and USDS staff said they determined this to  mean that OMB should report on the USDS projects considered to be  high priority given USDS’s responsibilities. In addition, USDS staff stated  that they did not receive any feedback from congressional stakeholders  indicating otherwise.

However, continued identification and reporting on the top ten high priority  programs, and not just USDS projects, would further enhance  congressional oversight by providing congressional stakeholders with  information on high priority programs that is not readily available. Such  information could also be useful for current administration IT governance  entities such as the Office of American Innovation and the American  Technology Council, to assist them with prioritizing their efforts to  modernize federal agency IT.

USDS Has a Defined Project Selection Process, but its Reporting Did Not Address High Priority Programs As Requested and Was Not Timely

USDS has developed a process for identifying and prioritizing the IT  projects to which it provides support. Moreover, as we have previously  reported, its project selection process is consistent with best practices,  which state that organizations should establish and implement  procedures for prioritizing projects that include identifying selection  criteria to help consistently select projects based on their contributions to  the strategic goals of the organization.

In explaining the process used to identify projects, USDS staff stated that  they obtained input from various sources, including the Federal IT  Dashboard, leadership of the relevant federal agency, E-Gov analysts,  and GAO reports. In addition, the staff said they considered the high  impact programs identified by the E-Gov office; they also coordinated with  E-Gov analysts through monthly meetings to incorporate issues of  significance to E-Gov into their selection process.

To further facilitate the selection process, the USDS staff established  three questions as criteria for prioritizing agencies’ IT projects, in the  following order of importance: (1) What will do the greatest good for the  greatest number of people in the greatest need? (2) How effective and  cost efficient will the USDS investment be? (3) What potential exists to  scale or reuse a technological solution across the government? The staff  said they used the criteria to create a list of all potential projects, including  their descriptions and information on resource needs; they updated the  list when they identified additional projects that met the criteria. This list  was subsequently used by USDS leadership to make decisions about  which projects to pursue. According to the staff, an important  consideration when selecting a project was whether there was executive  sponsorship from the agency. They added that executive sponsorship  ensures that USDS has the help it needs to make changes to the  projects, and it affects the efficiency and cost-effectiveness of USDS’s  investment. Projects that were not selected went into a backlog that is to  be used to select a project when a team becomes available to work on a  new engagement.

USDS has issued two reports to Congress on the status of its projects— one in December 2016 and the other in July 2017. The report issued in  December 2016 summarized the status of 11 projects that USDS is  engaged in at federal agencies, and 3 broader initiatives that are intended  to improve the performance and cost-effectiveness of government digital  services. According to USDS staff, the 11 projects had broad impact and  had made the most significant progress. For example, these included the  VA’s Vets.gov project, which is intended to assist the department in  developing a new digital application for healthcare, and DOD’s Defense  Travel System, a system that facilitates travel for all DOD employees.  Further, USDS reported on the status of their efforts for these three  initiatives: modernizing procurement processes, development of federal  shared services, and hiring top technical talent.

The July 2017 report included summaries for 10 projects, including 5 new  projects. Specifically, the new projects were the General Service  Administration’s Login.gov project, the Small Business Administration’s  effort to modernize small business certification for government  contractors, DOD’s Advisor Network system, DOD’s Defense Personal  Property System, and transforming federal IT procurement through digital  acquisition training. In addition, 6 projects that were discussed in the  December 2016 report were not included in the July 2017 report.

Table 2 provides a complete list of the projects identified in the two  reports.

Our analysis determined that four of the projects identified in USDS’s  reports were among the high priority programs that OMB’s E-Gov office  had identified in its June 2015 and June 2016 reports to Congress. These  projects were Healthcare.Gov, Disability Claim Processing, modernizing  the immigration system at the Department of Homeland Security, and  improving the Visa program at the Department of State. However,  USDS’s report does not specify this or provide an update on the status of  the other high priority programs. USDS staff stated that they did not  address all of the top 10 high priority programs because, as stated earlier  in this report, they interpreted Congress’s 2016 request as being focused  on USDS’s priority projects and not on the programs previously identified  by E-Gov. As mentioned earlier, however, continuing to identify and  report on the top 10 high priority IT programs while also reporting on  USDS’s projects would further enhance congressional oversight and  current administration IT governance entities’ efforts by providing  stakeholders with information on high priority programs and USDS  projects that is not readily available.

Further, although USDS was directed to report quarterly, it did not do so.  Instead, it issued a report in December 2016, nearly a year after  Congress’s direction, and in July 2017, nearly 7 months after its first  report. In discussing this matter, USDS staff said they were not able to  report quarterly due to the time and effort needed to prepare and review a  report. As a result, USDS’s reporting did not provide congressional  stakeholders with the timely information needed to support their oversight  responsibilities.

While OMB’s 2015 and 2016 reports to Congress on the top 10 high  priority programs included the status of the programs, their total IT  spending, and other information to assist Congress in monitoring the  progress of critical programs, OMB did not issue the reports quarterly as  directed in the explanatory statement. In addition, OMB does not plan on  continuing to issue the top 10 high priority reports because it believes  that, in 2016, Congress directed the agency to instead focus on providing  a status of USDS’s most important projects. Further, OMB’s December  2016 and July 2017 reports on USDS’s projects did not address the top  10 high priority programs across the government. However, continued  reporting on the top 10 high priority programs would further enhance  congressional oversight by providing congressional stakeholders with  information that is not readily available on those programs in the greatest  need of attention. Reporting on the top 10 high priority programs could  also be useful for IT governance entities such as the Office of American  Innovation and the American Technology Council, to assist them with  prioritizing their efforts to modernize federal agency IT.

Moreover, OMB did not issue the high priority programs and USDS  reports on a quarterly basis as requested. Without OMB’s quarterly  reporting on the progress of both the top 10 high priority programs and  the status of the USDS projects, congressional stakeholders and others  may lack the timely information they need to support their oversight and  other responsibilities.

Finally, while additional OMB oversight of the high impact programs (and,  accordingly, the identified high priority programs) is a positive step, the  Federal CIO was not directly involved in the oversight of these programs.  Based on the positive impact of direct Federal CIO involvement in leading  investment reviews in the past, such involvement could significantly  improve program outcomes.

We are making the following three recommendations to OMB:

The Director of OMB should continue to identify and report to  Congress on the status of the top 10 high priority IT programs and the  extent to which USDS is involved in the programs, as was done in  June 2015 and June 2016. In doing so, the Director should ensure  that these reports are issued quarterly. (Recommendation 1)

The Director of OMB should ensure that the Federal CIO is directly  involved in the oversight of high priority programs. (Recommendation  2)

The Director of OMB should continue to report on the status of USDS  projects. In doing so, the Director should ensure that the reports are  issued quarterly. (Recommendation 3)

We received comments on our draft report via e-mail from the OMB  liaison to GAO. In the comments, OMB did not specifically state whether it  agreed or disagreed with our recommendations. Rather, OMB stated that  it has concerns with GAO’s alternative interpretations of law and that  GAO’s findings and conclusions are rooted in an incorrect legal  interpretation of OMB’s annual appropriation. Specifically, it stated that  GAO considers reporting requirements specific to an annual appropriation  to apply for all future annual appropriations.

However, OMB’s characterization is incorrect, as we did not assert this  legal conclusion. As stated in our report, in the explanatory statement  accompanying the Consolidated and Further Continuing Appropriations  Act, 2015, Congress directed OMB to identify the 10 highest priority IT  investment programs (referred to in our report as the top 10 high priority  programs) that are under development across federal agencies and  report on their status each quarter. Subsequently, the explanatory  statement accompanying the Consolidated Appropriations Act, 2016,  directed USDS to provide a quarterly status report on, among other  things, current USDS projects, including the top 10 high priority programs.  Our report does not conclude that the language of either explanatory  statement establishes a legally binding requirement, whether applicable  only to the subject fiscal year or beyond.

GAO’s conclusions are based on the view that continued identification  and reporting on the top 10 high priority programs, and not just USDS  projects, would help to enhance congressional oversight. Identifying and  reporting on the top 10 high priority programs is important because such  information is not readily available. We have revised relevant statements  in the report to clarify our message in this regard. Further, while GAO did  not assert a legal conclusion, we have, nonetheless, removed all  references to our “interpretation” of the explanatory statement so as to  avoid the inference that we are making legal conclusions.

OMB also said our conclusion that it has stopped reporting altogether is  incorrect. However, our report does not state that OMB has stopped  reporting altogether. Rather, our report states that OMB stopped issuing  reports on the top 10 high priority IT programs due to its interpretation of  the 2016 explanatory statement and, instead, switched to reporting on the  status of USDS’s projects.

In addition, OMB stated that, while it agreed that the reports have not  been submitted on a quarterly basis, it provided Congress with the  requested information for the four quarters of the relevant fiscal years. In  addition, OMB stated that USDS is currently discussing with  congressional stakeholders whether providing quarterly briefings, instead  of reports, would address the quarterly reporting requirement. As noted in  our report, timely (i.e., quarterly) information would enhance  congressional and other stakeholders’ oversight responsibilities.  Therefore, we maintain that our recommendation for reporting quarterly is  appropriate.

Finally, OMB stated that it disagreed with our conclusion that the lack of  personal Federal CIO involvement in high priority IT programs had  resulted in inadequate oversight. While we state that the Federal CIO was  not directly involved in overseeing the high priority programs, we did not  conclude that this resulted in inadequate oversight. Rather, we stated that  the results of past CIO-led TechStat reviews suggest that more direct and  regular involvement of the Federal CIO would improve accountability and  achieve positive results for the federal government’s investments. Thus,  we continue to believe that our recommendation to ensure that the  Federal CIO is directly involved in overseeing high priority programs is  appropriate.

OMB also provided technical comments, which we have incorporated into  the report as appropriate.

We are sending copies of this report to interested congressional  committees, the Director of the Office of Management and Budget, and  other interested parties. In addition, the report will be available at no  charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-9286 or pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix I.

Appendix I: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Sabine Paul (Assistant Director),  Scott Borre (Analyst in Charge), Nancy Glover, Lori Martinez, Bradley  Roach, and Marshall Williams, Jr. made key contributions to the report.