Critical Infrastructure Protection: Progress and Challenges in DHS's Management of Its Chemical Facility Security Program

Thousands of facilities have hazardous chemicals that could be targeted or used to inflict mass casualties or harm surrounding populations in the United States. In accordance with the DHS Appropriations Act, 2007, DHS established the CFATS program in 2007 to, among other things, identify and assess the security risk posed by chemical facilities. DHS inspects high-risk facilities after it approves facility security plans to ensure that the facilities are implementing required security measures and procedures.

This statement summarizes progress and challenges related to DHS's CFATS program management. This statement is based on prior products GAO issued from July 2012 through June 2017, along with updates conducted in June 2018 on DHS actions to address prior GAO recommendations. To conduct the prior work, GAO reviewed relevant laws, regulations, and DHS policies for administering the CFATS program, how DHS assesses risk, and data on high-risk chemical facilities. GAO also interviewed DHS officials and reviewed information on DHS actions to implement its prior recommendations.

The Department of Homeland Security (DHS) has made progress addressing challenges that GAO's past work identified to managing the Chemical Facility Anti-Terrorism Standards (CFATS) program. The following summarizes progress made and challenges remaining in key aspects of the program.

Identifying high-risk chemical facilities. In July 2015, GAO reported that DHS used self-reported and unverified data to determine the risk of facilities holding toxic chemicals that could threaten surrounding communities if released. GAO recommended that DHS should better verify the accuracy of facility-reported data. DHS implemented this recommendation by revising its methodology so it now calculates the risk of toxic release, rather than relying on facilities to do so.

Assessing risk and prioritizing facilities. In April 2013, GAO reported weaknesses in multiple aspects of DHS's risk assessment and prioritization approach. GAO made two recommendations for DHS to review and improve this process, including that DHS enhance its risk assessment approach to incorporate all of the elements of consequence, threat, and vulnerability associated with a terrorist attack involving certain chemicals. DHS launched a new risk assessment methodology in October 2016 and is currently gathering new or updated data from about 27,000 facilities to (1) determine which facilities should be categorized as high-risk because of the threat of sabotage, theft or diversion, or a toxic release and (2) assign those facilities deemed high risk to one of four risk-based tiers. GAO has ongoing work assessing these efforts and will report later this summer on the extent to which they fully address prior recommendations.

Reviewing and approving facilities' site security plans . DHS is to review security plans and visit facilities to ensure their security measures meet DHS standards. In April 2013, GAO reported a 7 to 9 year backlog for these reviews and visits. In July 2015, GAO reported that DHS had made substantial progress in addressing the backlog—estimating that it could take between 9 and 12 months for DHS to review and approve security plans for the approximately 900 remaining facilities. DHS has since taken additional action to expedite these activities and has eliminated this backlog.

Inspecting facilities and ensuring compliance. In July 2015, GAO reported that DHS conducted compliance inspections at 83 of the 1,727 facilities with approved security plans. GAO found that nearly half of the inspected facilities were not fully compliant with their approved security plans and that DHS did not have documented procedures for managing facilities' compliance. GAO recommended that DHS document procedures for managing compliance. As a result, DHS has developed an enforcement procedure and a draft compliance inspection procedure and expects to finalize the compliance inspection procedure by the end of fiscal year 2018.

GAO has made various recommendations to strengthen DHS's management of the CFATS program, with which DHS has generally agreed. DHS has implemented or described planned actions to address most of these recommendations.

Thank you for the opportunity to discuss our past work on the Department  of Homeland Security’s (DHS) efforts to manage its Chemical Facility  Anti-Terrorism Standards (CFATS) program. Thousands of facilities that  produce, use, or store hazardous chemicals could be of particular interest  to terrorists who are intent on using toxic chemicals to inflict mass  casualties in the United States. These chemicals could be released from  a facility to cause harm to surrounding populations; they could be stolen  and used as chemical weapons or as their precursors (the ingredients for  making chemical weapons); or they could be stolen and used to build an  improvised explosive device. Past incidents remind us of the danger that  these chemicals pose, including the 2013 ammonium nitrate explosion at  a fertilizer storage and distribution facility in West, Texas, which killed at  least 14 people and damaged or destroyed at least 200 homes, and the  1995 domestic terrorist attack on the federal building in Oklahoma City,  Oklahoma, where 168 people were killed using ammonium nitrate  fertilizer mixed with fuel oil.

The Department of Homeland Security Appropriations Act, 2007, required  DHS to issue regulations to establish risk-based performance standards  (performance standards) for securing high-risk chemical facilities. DHS  subsequently established the CFATS program in 2007 to, among other  things, identify high-risk chemical facilities and assess the risk posed by  them; place facilities considered to be high risk into one of four risk-based  tiers (with tier 1 being the highest risk tier and 4 being the lowest); assess  facility security; approve security plans prepared by facilities; and inspect  facilities to ensure compliance with regulatory requirements. DHS’s  CFATS rule established 18 performance standards that identify the areas  for which a facility’s security posture are to be examined, such as  perimeter security, access control, and cyber security. To meet these  standards, facilities are free to choose whatever security programs or  processes they deem appropriate so long as DHS determines that the  facilities achieve the requisite level of performance in each of the  applicable areas. The Protecting and Securing Chemical Facilities from  Terrorist Attacks Act of 2014 (CFATS Act of 2014) enacted in December  2014, in effect, reauthorized the CFATS program for an additional 4  years, while also imposing additional implementation requirements on  DHS for the program. DHS’s National Protection and Programs  Directorate’s Infrastructure Security Compliance Division manages the  CFATS program. According to DHS, the Department received  approximately $911 million for the CFATS program for the period  beginning fiscal year 2007 through fiscal year 2018.

My testimony today summarizes our past work examining DHS’s  management of the CFATS program, and provides updates on actions  DHS has taken to address our prior recommendations. This testimony is  based on our reports issued from July 2012 through June 2017. For  these reports, we reviewed applicable laws and regulations, DHS policies  and procedures, DHS data on tiered facilities, information on the  approach DHS used to determine a facility’s risk and process for  reviewing security plans. We also interviewed DHS officials about facility  tiering, how DHS assesses risk, and how it processes security plans.  Additional details on the scope and methodology are available in our  published reports. In addition, this statement contains updates as of June  2018 from DHS on actions it has taken to address the recommendations  made in our prior reports. The work upon which this statement is based  was conducted in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.

DHS Has Made Progress Addressing Past Challenges, But Some Actions are Still In Progress

Our past work has identified progress and challenges in a number of  areas related to DHS’s management of the CFATS program including (1)  the process for identifying high risk chemical facilities; (2) how it assesses  risk and prioritizes facilities; (3) reviewing and approving facility security  plans; (4) how it conducts facility compliance inspections; and (5) efforts  to conduct stakeholder outreach and gather feedback. DHS has made a  number of programmatic changes to CFATS in recent years that may also  impact its progress in addressing our open recommendations; these  changes are included as part of our ongoing review of the program.

Identifying High-Risk Chemical Facilities

In May 2014, we found that more than 1,300 facilities had reported having  ammonium nitrate to DHS. However, based on our review of state data  and records, there were more facilities with ammonium nitrate holdings  than those that had reported to DHS under the CFATS program. Thus,  we concluded that some facilities that were required to report may have  failed to do so. We recommended that DHS work with other agencies,  including the Environmental Protection Agency (EPA), to develop and  implement methods of improving data sharing among agencies and with  states as members of a Chemical Facility Safety and Security Working  Group. DHS agreed with our recommendation and has since addressed  it. Specifically, DHS compared DHS data with data from other federal  agencies, such as EPA, as well as member states from the Chemical  Facility Safety and Security Working Group to identify potentially  noncompliant facilities. As a result of this effort, in July 2015, DHS  officials reported that they had identified about 1,000 additional facilities  that should have reported information to comply with CFATS and  subsequently contacted these facilities to ensure compliance. DHS  officials told us that they continue to engage with states to identify  potentially non-compliant facilities. For example, as of June 2018, DHS  officials stated they have received 43 lists of potentially noncompliant  facilities from 34 state governments, which are in various stages of review  by DHS. DHS officials also told us that they recently hired an individual to  serve as the lead staff member responsible for overseeing this effort.

DHS has also taken action to strengthen the accuracy of data it uses to  identify high risk facilities. In July 2015, we found that DHS used self- reported and unverified data to determine the risk categorization for  facilities that held toxic chemicals that could threaten surrounding  communities if released. At the time, DHS required that facilities self- report the Distance of Concern—an area in which exposure to a toxic  chemical cloud could cause serious injury or fatalities from short-term  exposure—as part of its Top-Screen. We estimated that more than  2,700 facilities with a toxic release threat had misreported the Distance of  Concern and therefore recommended that DHS (1) develop a plan to  implement a new Top-Screen to address errors in the Distance of  Concern submitted by facilities, and (2) identify potentially miscategorized  facilities that could cause the greatest harm and verify that the Distance  of Concern of these facilities report is accurate. DHS has fully  addressed both of these recommendations. Specifically, DHS  implemented an updated Top-Screen in October 2016 and now collects  data from facilities and calculates the Distance of Concern itself, rather  than relying on the facilities’ calculation. In response to our second  recommendation, in November 2016, DHS officials stated they completed  an assessment of all Top-Screens that reported threshold quantities of  toxic release chemicals of interest and identified 158 facilities with the  potential to cause the greatest harm. As of May 2017, according to ISCD  officials, 156 of the 158 facilities submitted updated Top-Screens and 145  of the 156 Top-Screens had undergone a quality assurance review  process.

Assessing Risk and Prioritizing Facilities

DHS has also taken actions to better assess regulated facilities’ risks in  order to place the facilities into the appropriate risk tier. In April 2013, we  reported that DHS’s risk assessment approach did not consider all of the  elements of threat, vulnerability, and consequence associated with a  terrorist attack involving certain chemicals. Our work showed that DHS’s  risk assessment was based primarily on consequences from human  casualties, but did not consider economic consequences, as called for by  the National Infrastructure Protection Plan (NIPP) and the CFATS  regulation. We also found that (1) DHS’s approach was not consistent  with the NIPP because it treated every facility as equally vulnerable to a  terrorist attack regardless of location or on-site security and (2) DHS was  not using threat data for 90 percent of the tiered facilities—those tiered for  the risk of theft or diversion—and using 5-year-old threat data for the  remaining 10 percent of those facilities that were tiered for the risks of  release or sabotage. We recommended that DHS enhance its risk  assessment approach to incorporate all elements of risk and conduct a  peer review after doing so. DHS agreed with our recommendations and  has made progress towards addressing them.

Specifically, with regard to our recommendation that DHS enhance its risk  assessment approach to incorporate all elements of risk, DHS worked  with Sandia National Laboratories to develop a model to estimate the  economic consequences of a chemical attack. In addition, DHS worked  with Oak Ridge National Laboratory to devise a new tiering methodology,  called the Second Generation Risk Engine. In so doing, DHS revised the  CFATS threat, vulnerability, and consequence scoring methods to better  cover the range of CFATS security issues. Additionally, with regard to our  recommendation that DHS conduct a peer review after enhancing its risk  assessment approach, DHS conducted peer reviews and technical  reviews with government organizations and facility owners and operators,  and worked with Sandia National Laboratories to verify and validate the  new tiering approach. We are currently reviewing the reports and data  that DHS has provided about its new tiering methodology as part of our  ongoing work and will report on the results of this work later this summer.

To further enhance its risk assessment approach, in fall 2016, DHS also  revised its Chemical Security Assessment Tool (CSAT), which supports  DHS efforts to gather information from facilities to assess their risk.  According to DHS officials, the new tool—called CSAT 2.0—is intended to  eliminate duplication and confusion associated with DHS’s original CSAT.  DHS officials told us that they have improved the tool by revising some  questions in the original CSAT to make them easier to understand;  eliminating some questions; and pre-populating data from one part of the  tool to another so that users do not have to retype the same information  multiple times. DHS officials also told us that the facilities that have used  the CSAT 2.0 have provided favorable feedback that the new tool is more  efficient and less burdensome than the original CSAT. Finally, DHS  officials told us that as of June 2018, DHS has completed all notifications  and has processed tiering results for all but 226 facilities. DHS officials  stated they are currently working to identify correct points of contact to  update registration information for these remaining facilities. We are  currently assessing DHS’s efforts to assess risk and prioritize facilities as  part of our ongoing work and will report on the results of this work in our  report later this summer.

Reviewing and Approving Facility Site Security Plans

DHS has also made progress reviewing and approving facility site  security plans by reducing the time it takes to review these plans and  eliminating the backlog of plans awaiting review. In April 2013, we  reported that DHS revised its procedures for reviewing facilities’ security  plans to address DHS managers’ concerns that the original process was  slow, overly complicated, and caused bottlenecks in approving plans.  We estimated that it could take DHS another 7 to 9 years to review the  approximately 3,120 plans in its queue at that time. We also estimated  that, given the additional time needed to do compliance inspections, the  CFATS program would likely be implemented in 8 to 10 years. We did not  make any recommendations for DHS to improve its procedures for  reviewing facilities’ security plans because DHS officials reported that  they were exploring ways to expedite the process, such as reprioritizing  resources and streamlining inspection requirements. In July 2015, we  reported that DHS had made substantial progress in addressing the  backlog—estimating that it could take between 9 and 12 months for DHS  to review and approve security plans for the approximately 900 remaining  facilities. DHS officials attributed the increased approval rate to  efficiencies in DHS’s review process, updated guidance, and a new case  management system. Subsequently, DHS reported in its December 2016  semi-annual report to Congress that it had eliminated its approval  backlog.

Finally, we found in our 2017review that DHS also took action to  implement an Expedited Approval Program (EAP). The CFATS Act of  2014 required that DHS create the EAP as another option that tier 3 and  tier 4 chemical facilities may use to develop and submit security plans to  DHS. Under the program, facilities may develop a security plan based  on specific standards published by DHS (as opposed to the more flexible  performance standards using the standard, non-expedited process). DHS  issued guidance intended to help facilities prepare and submit their EAP  security plans to DHS, which includes an example that identifies  prescriptive security measures that facilities are to have in place.  According to committee report language, the EAP was expected to  reduce the regulatory burden on smaller chemical companies, which may  lack the compliance infrastructure and the resources of large chemical  facilities, and help DHS to process security plans more quickly. If a tier 3  or 4 facility chooses to use the expedited option, DHS is to review the  plan to determine if it is facially deficient, pursuant to the reporting  requirements of the CFATS Act of 2014. If DHS approves the EAP site  security plan, it is to subsequently conduct a compliance inspection.

In 2017, we found that DHS had implemented the EAP and had reported  to Congress on the program, as required by the CFATS Act of 2014. In  addition, as of June 2018 according to DHS officials, only 18 of the 3,152  facilities eligible to use the EAP opted to use it. DHS officials we  interviewed attributed the low participation to several possible factors  including:

DHS had implemented the expedited program after most eligible  facilities already submitted standard (non-expedited) security plans to  DHS;  facilities may consider the expedited program’s security measures to  be too strict and prescriptive, not providing facilities the flexibility of  the standard process; and  the lack of an authorization inspection may discourage some facilities  from using the expedited program because this inspection provides  useful information about a facility’s security.

We also found in 2017 that recent changes made to the CFATS program  could affect the future use of the expedited program. As discussed  previously, DHS has revised its methodology for determining the level of  each facility’s security risk, which could affect a facility’s eligibility to  participate in the EAP. DHS continues to apply the revised methodology  to facilities regulated under the CFATS program and but it is too early to  assess the impact on participation in the EAP.

Inspecting Facilities and Ensuring Consistent Compliance

In our July 2015 report, we found that DHS began conducting compliance  inspections in September 2013, and by April 2015, had conducted  inspections of 83 of the 1,727 facilities that had approved security plans.  Our analysis showed that nearly half of the facilities were not fully  compliant with their approved site security plans and that DHS had not  used its authority to issue penalties because DHS officials found it more  productive to work with facilities to bring them in compliance. We also  found that DHS did not have documented processes and procedures for  managing the compliance of facilities that had not implemented planned  measures by the deadlines outlined in the plans. We recommended that  DHS document processes and procedures for managing compliance to  provide more reasonable assurance that facilities implement planned  measures and address security gaps. DHS agreed and has taken steps  toward implementing this recommendation. DHS updated its CFATS  Enforcement Standard Operating Procedure (SOP) and has made  progress on the new CFATS Inspections SOP. Once completed these  two documents collectively are expected to formally document the  processes and procedures currently being used to track noncompliant  facilities and ensure they implement planned measures as outlined in  their approved site security plans, according to ISCD officials. DHS  officials stated they expect to finalize these procedures by the end of  fiscal year 2018. We are examining compliance inspections as part of our  ongoing work and will report on the results of our work in our report later  this summer.

In April 2013, we reported that DHS took various actions to work with  facility owners and operators, including increasing the number of visits to  facilities to discuss enhancing security plans, but that some trade  associations had mixed views on the effectiveness of DHS’s outreach.  We found that DHS solicited informal feedback from facility owners and  operators in its efforts to communicate and work with them, but did not  have an approach for obtaining systematic feedback on its outreach  activities. We recommended that DHS take action to solicit and document  feedback on facility outreach consistent with DHS efforts to develop a  strategic communication plan. DHS agreed and implemented this  recommendation by developing a questionnaire to solicit feedback on  outreach with industry stakeholders and began using the questionnaire in  October 2016.

Chairman Shimkus, Ranking Member Tonko, and Members of the  Subcommittee, this completes my prepared statement. I would be  pleased to respond to any questions that you may have at this time.

If you or your staff members have any questions about this testimony,  please contact me at (404) 679-1875 or curriec@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this statement. Other individuals making key  contributions to this work include John Mortin, Assistant Director; and  Brandon Jones, Analyst-in-Charge; Michael Lennington, Ben Emmel, and  Hugh Paquette.

This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.


Defense Health Care: Expanded Use of Quality Measures Could Enhance Oversight of Provider Performance

The National Defense Authorization Act for fiscal year 2016 contains provisions for GAO to review DOD's plans to (1) improve the experience of beneficiaries who receive care through military hospitals and clinics or from civilian providers and (2) reduce variation in the quality of care.

In this report, GAO examines (1) measures DOD uses to assess the quality of direct and purchased care, and (2) the extent to which DOD has established performance standards related to the measures and corrective action requirements for providers who do not meet those standards.

GAO reviewed the measures in DOD's Core Dashboard for direct care and Purchased Care Dashboard for purchased care. It also reviewed DOD documents and reports to Congress, and interviewed MHS officials, including officials from the Army, Navy, and Air Force. GAO also compared the quality measures DOD uses to those used in Medicare and by private insurers, which have been vetted by multiple stakeholders. GAO assessed DOD's use of performance standards and corrective action requirements in the context of federal internal control standards.

The National Defense Authorization Act for fiscal year 2016 (NDAA 2016) directed the Department of Defense (DOD) to align its measures of health care quality used in the Military Health System (MHS) to improve beneficiary experience and reduce variation in the quality of care. GAO reviewed the quality measures DOD identified in March 2017 in response to the NDAA 2016; DOD senior leadership tracks these measures on dashboards to gauge progress on MHS strategic goals. GAO found that DOD does not use a common set of measures on its dashboards to assess the quality of care provided by either military hospitals and clinics—known as direct care—or networks of civilian hospitals and other providers, known as purchased care. (See figure.) As a result, DOD's senior leadership has limited information on the extent to which MHS beneficiaries receive consistently high quality care across the MHS.

Furthermore, for both direct and purchased care, DOD uses measures on its dashboards that track a limited range of quality care areas and medical conditions compared to the measures adopted by Medicare and by private health insurers. For example, whereas civilian hospitals report to Medicare information on 11 measures of patients' self-reported experience in hospitals, Military hospitals report only 1 such measure. By using a limited range of quality measures, DOD may not detect key quality issues. Further, when selecting quality measures, the MHS does not prioritize using common measures across direct and purchased care or expanding the range of measures it uses.

GAO also found that for direct care DOD has established performance standards and corrective action requirements for military hospitals or clinics that do not meet those standards in direct care. The performance standards indicate the level of performance providers should meet on the various quality measures DOD tracks on its dashboards, and the corrective action requirements instruct providers to take steps to improve care. However, for purchased care, DOD has not established similar performance standards for individual providers. Without consistent performance standards and corrective action requirements, DOD is limited in its ability to address variation in the quality of care delivered and help ensure that its beneficiaries receive consistent high quality care across the MHS.

The MHS should (1) prioritize, as appropriate, selecting quality measures common for both direct and purchased care that expand the range of quality areas covered by the measures and (2) establish consistent performance standards and corrective action requirements for direct and purchased care providers. DOD concurred with both recommendations.

The Department of Defense’s (DOD) Military Health System (MHS) offers  a full range of health care services to over 9 million eligible beneficiaries,  including active duty servicemembers and their dependents, medically- eligible National Guard and Reserve members and their dependents, and  retirees and their dependents and survivors, among others. These  beneficiaries may receive care from military treatment facilities (MTFs)— known as direct care—or from civilian hospitals, physicians, and other  clinicians participating in DOD-sponsored health plans that are  administered by contractors—known as purchased care. While MTFs  take the lead in delivering health care to active duty servicemembers,  eligible beneficiaries can receive a wide array of primary and specialty  care from the MTFs, purchased care providers, or both. Overall, about 60  percent of all the health care services that beneficiaries received in 2017  were delivered through purchased care.

In 2014, the Secretary of Defense ordered a comprehensive review of the  MHS that found considerable variation in the quality of care delivered. For  example, the review found that 8 of 17 MTFs with high volume surgery  programs had higher than expected rates of surgical complications.  Furthermore, in the case of purchased care, the review found that DOD  had limited information on the quality of care delivered by those civilian  providers.

The information DOD has on the quality of care in the MHS comes from  what it collects on various quality measures. In general, health care  quality measures are standard, evidence based metrics—such as the  percentage of patients receiving a screening or the rate of hospital  readmissions or surgical complications—that health care systems such as  the MHS use to quantify health care processes, outcomes, and other  aspects of care. Health care quality measures can be linked to  performance standards established for providers. For example, a quality  measure may indicate the percentage of patients who receive a diabetes  screening, while a related performance standard may be that providers  are expected to ensure that at least a certain percentage of applicable  patients receive the screening each year.

Section 730 of the National Defense Authorization Act for Fiscal Year  2016 (NDAA 2016) directed DOD to develop plans to enhance the  experience of beneficiaries receiving care under the MHS and eliminate  variation in the quality of care that beneficiaries receive across direct and  purchased care. The Act stipulated that DOD align the measures used to  assess the quality of direct and purchased care to improve care across  the MHS as a whole. In March 2017, DOD issued its response to section  730, reporting that it had established two sets of quality measures— a set  of measures for direct care described as “core measures” and another set  of measures for purchased care. The MHS reported to Congress that the  core direct care measures and purchased care measures are the key  quality measures DOD’s senior health care leadership use to track DOD’s  progress towards achieving the department’s overall strategic goals of  providing high quality care across the MHS as a whole. DOD reported  that it aligned these quality measures across direct and purchased care  where possible.

Section 730 of the NDAA 2016 also includes a provision for us to assess  the strengths and limitations of DOD’s plans for achieving the Act’s  objectives. In this report, we examine  1.  the core direct care measures and purchased care measures DOD  uses to assess the quality of care in the MHS; and  2.  the extent to which DOD has established performance standards  related to its core direct care measures and purchased care measures  and corrective action requirements for providers that do not meet  these performance standards.

To examine the core direct care measures and purchased care measures  DOD uses to assess the quality of care in the MHS, we focused our  review on the measures listed on the Core Dashboard and Purchased  Care Dashboard, which are used in direct care and purchased care,  respectively. We focused on the dashboard measures because the MHS  reported to Congress that these are the measures that DOD health care  leaders rely on to establish accountability throughout the MHS and  identify areas where quality improvement is needed. The measures on  the dashboards are periodically updated, and we examined the measures  included on the dashboards as of March 31, 2018 and the extent to which  the measures on one dashboard aligned with those on the other. We also  examined the range of quality care areas and medical conditions  assessed by the dashboard measures and compared these to the range  of quality care areas and medical conditions assessed by the measures  included in the Hospital Compare measure set for inpatient care and the  Core Quality Measure Collaborative (CQMC) measure sets for outpatient  care. Because the Hospital Compare and CQMC measures for inpatient  and outpatient care have been vetted and reviewed by multiple health  care stakeholders and widely adopted, we determined that they  represented appropriate benchmarks for assessing the scope of the  MHS’s quality assessment. We also reviewed DOD documents to  understand DOD senior health leadership’s decisions to select and track  the measures on the Core and Purchased Care Dashboards and not  others. Specifically, we systematically reviewed minutes from 335  meetings of the MHS governance bodies—which represent the three  military services that provide health care services under the MHS (Army,  Navy, and Air Force) and other components of the MHS—that make final  decisions on the measures to be included on the Core and Purchased  Care Dashboards. We supplemented this document review with  interviews of the MHS officials responsible for managing direct care as  well as those responsible for overseeing the MHS’s contracting of  purchased care. We also interviewed representatives of the medical  commands for each of the three military services to obtain their  perspectives on the measures that the MHS had selected to assess the  quality of care provided to their active duty members and other  beneficiaries. In addition, we interviewed the Chair of the Health Care  Subcommittee of the Military Coalition, a group of 32 military, veterans,  and uniformed services organizations. Although not representative, this  allowed us to obtain the perspectives of beneficiaries on the MHS’s  quality measurement efforts.

To examine the extent to which DOD has established performance  standards related to its core direct care measures and purchased care  measures and corrective action requirements for providers that do not  meet performance standards, we reviewed policies related to the MHS’s  oversight of provider performance on the measures on its Core and  Purchased Care Dashboards. We also interviewed MHS officials and  reviewed relevant MHS documents to determine whether the MHS sets  specific performance standards for MTFs and civilian providers related to  the quality measures on the Core and Purchased Care Dashboards. We  also examined whether or to what extent MHS has established  requirements for initiating corrective actions when providers do not meet  those performance standards. The relevant documents we reviewed  include the minutes of the MHS governance body meetings noted above;  the MHS reports to Congress on direct and purchased care; documents  related to the purchased care contracts; and reports that contractors  submitted to the MHS for purchased care. Additionally, in our review of  the MHS governance body meeting minutes, we examined MHS officials’  decisions, their rationale, and any actions taken in response to the trends  being monitored on the Core and Purchased Care Dashboard quality  measures. We compared these MHS efforts to federal standards for  internal control related to monitoring.

We conducted this performance audit from June 2017 to September 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

As we have previously reported in reviews of health care quality outside  of the MHS, health care quality measures are standard, evidence-based  metrics designed to assess the extent to which patients receive health  care that increases the likelihood of desired health outcomes and are  consistent with current professional knowledge. These measures may  be used to assess the quality of care in various settings, including  hospitals and physician offices. Health care quality measures are  intended to (1) inform providers about opportunities for potential  improvements in their delivery of care, (2) encourage or incentivize  providers to consistently provide high quality care, and (3) inform  consumers about which providers are most likely to deliver high quality  care. There are broad categories of clinical quality measures that address  various aspects of quality of care. See table 1 for a description of these  broad categories of quality measures.

The data used to calculate the results of health care quality measures can  come from a number of different sources. Some measures often require  detailed clinical information obtained from patient medical records, such  as process measures that indicate whether timely and effective care was  provided in a specific situation, for example, or whether stroke patients  received clot-dissolving medication appropriately. Other measures are  designed to use information on patient demographics and diagnoses that  can be obtained from more readily accessible sources, such as claims  data or other administrative data that have already been collected for  other purposes such as billing. In addition, patients can be asked directly,  usually through surveys, to report on their experiences receiving care.

The MHS Structure and Administration of Direct and Purchased Care

The MHS is a complex organization in which responsibility for the delivery  of health care is primarily shared among the military services—Army,  Navy, and Air Force—and the Defense Health Agency (DHA). The  Army, Navy and Air Force medical commands report through their service  chiefs to their respective military department Secretaries and then to the  Secretary of Defense. DHA reports through the Office of the Assistant  Secretary of Defense for Health Affairs and the Under Secretary of  Defense for Personnel and Readiness to the Secretary of Defense. The  Office of the Assistant Secretary of Defense for Health Affairs manages  the Defense Health Program appropriation, which funds the medical and  health care programs at the medical commands of the military services.  As of fiscal year 2018, most of the MTFs, including military hospitals and  clinics, were under the direction and control of the military services, which  are responsible for staffing, training, and equipping those MTFs to meet  mission requirements. DHA has responsibility for the managed care  support contracts through which the MHS administers its purchased care,  and DHA also administers several MTFs in the vicinity of Washington,  DC. Figure 1 depicts the MHS organizational structure.

Recently enacted changes will affect the administration of the MTFs in  future years. Most notably, DOD will alter administration of the MTFs,  shifting responsibility from the military services to DHA. Section 702 of the  National Defense Authorization Act for Fiscal Year 2017 (NDAA 2017)  directed DOD to give DHA responsibility for the administration of all  MTFs, including budgetary matters, information technology, and health  care administration and management. In the conference report for  NDAA 2017, Congress stated its intention that the creation of a single  agency responsible for all MTFs would improve and sustain readiness,  reduce costs, and increase efficiency. DOD has since prepared a series  of implementation plans as it works to develop the specific policies and  procedures to enable this change to take effect starting October 1, 2018.  The most recent plan issued by DOD in June 2018 envisions a 3-year  transition to be completed October 1, 2021.

For purchased care, DOD contracts with civilian health care contractors to  manage its civilian providers on a regional basis. The primary  responsibilities of these managed care support contractors include the  following: developing civilian provider networks, which include hospitals and ensuring adequate access to health care;  referring and authorizing beneficiaries to receive health care; processing health care claims; educating providers and beneficiaries; and  conducting utilization management and quality management  programs.

There have been several generations of multi-year contracts since 1996.  In July 2016, DOD awarded its fourth generation of managed care  support contracts to two regional contractors, and on January 1, 2018, the  MHS began health care delivery under these contracts.

Selection of Measures to Assess Quality of Direct and Purchased Care

According to our review of DOD documents, the MHS uses a structured  process to select the measures on its dashboards that are used to assess  the quality of direct and purchased care. Specifically, DOD documents  state that the core direct care measures that are on the Core Dashboard  are selected through the MHS’s performance management system called  Partnership for Improvement (P4I), which began in 2015. The  documents show that proposals for potential quality measures are  developed by work groups that focus on different specialized areas, such  as maternity care or mental health. These proposals are reviewed and  approved by the Steering Committee for P4I, which develops the list of  core quality measures for direct care. The Steering Committee then  presents the list of core quality measures to a succession of governance  bodies—each of which incorporates representation from the three military  services plus DHA—for review and approval. DOD documents indicate  that the MHS repeats this process annually as it decides which quality  measures to add, drop, or modify for the coming fiscal year from the Core  Dashboard.

The DOD documents we reviewed lay out a parallel process that the MHS  follows to select which purchased care quality measures will be tracked in  the Purchased Care Dashboard. A work group that specializes on  purchased care issues with representation of DHA and the three military  services develops the proposed list of quality measures for the Purchased  Care Dashboard. This list is then reviewed and approved by the same  succession of governance bodies that decide on the Core Dashboard  measures.

Officials told us and DOD documents confirmed that the MHS and the  purchased care contractors also track additional quality measures that  are not included in the Core and Purchased Care Dashboards. For  example, MHS clinicians who provide maternity care track a set of  measures developed by the National Perinatal Information Center.  Similarly, a number of military hospitals report on surgical quality  measures to the National Surgical Quality Improvement Program. The  MHS also conducts surveys of MHS beneficiaries from which it obtains  data for patient experience measures for both direct and purchased care.  Additionally, the MHS requires the managed care support contractors that  administer the MHS’s networks of civilian providers for purchased care to  monitor several different sets of quality measures or indicators, many of  which focus on patient safety. These include patient safety indicators,  hospital acquired conditions, and serious reportable events. They also  analyze measures selected from Hospital Compare and the Healthcare  Effectiveness Data and Information Set (HEDIS), some of which  correspond to measures included in the Core and Purchased Care  Dashboards.

The Hospital Compare and Core Quality Measures Collaborative Measure Sets Adopted by Medicare and Private Health Insurers

While health care systems in the United States can use a variety of  measures to assess the quality of care, two of the most widely adopted  sets of quality measures include the Hospital Compare measure set  developed by Centers for Medicare & Medicaid Services (CMS) for  inpatient care and the CQMC measure sets jointly developed by CMS  and major private health insurers for outpatient care. Since 2005, CMS  has collected results for individual hospitals on a specific list of health  care quality measures that are posted on a website known as Hospital  Compare. CMS does this to make comparable information on the quality  of care provided by different hospitals publicly available. Hospital  Compare currently covers more than 4,000 hospitals that participate in  the Medicare program. These hospitals supply data to CMS for quality  measures of inpatient and emergency department care. These data  reflect the care provided to all patients treated at these hospitals, not just  those covered by Medicare. Each year CMS goes through a formal  process, including receiving input from experts and stakeholders, to  review and revise the mix of quality measures that these hospitals are  expected to report. The purpose of this review, according to CMS, is to  ensure that the set of measures reported on Hospital Compare provides  meaningful information for quality improvement while reducing  unnecessary administrative burden.

Initiated in 2014, the CQMC is a multi-stakeholder voluntary effort  focused on quality measure alignment that has developed eight sets of  measures for outpatient primary and specialty care, known as the CQMC  measure sets. In developing the measure sets, CMS and private health  insurers negotiate sets of core measures on which they agree to focus on  measuring care quality for certain conditions. Physician specialty  societies, employer groups, consumer groups, and regional collaboratives  also participate in the negotiations. The CQMC measure sets have been  adopted by CMS for Medicare and by 15 major private health insurers for  commercial health plans. Additionally, section 728 of the NDAA 2017  directs the MHS to use, to the extent appropriate, these quality measures  to assess the quality of direct and purchased care. CQMC documents  show that the members of the CQMC intend to continually update these  core measure sets as more meaningful measures are developed over  time. CMS and the private health insurers plan to expand their application  of these measures incrementally, as CMS conducts its annual reviews of  Medicare’s quality measures and the insurers update or renew their  contracts with different providers.

The MHS Does Not Use a Common Set of Quality Measures for Direct and Purchased Care, and the Measures It Uses Assess a Limited Range of Quality Areas

The MHS does not use a common set of measures on its Core and  Purchased Care Dashboards to assess the quality of care provided  through direct and purchased care. In addition, for both direct and  purchased care, the MHS uses measures on its dashboards that assess  a more limited range of quality care areas and medical conditions as  compared to the Hospital Compare and CQMC measures adopted by  Medicare and private health insurers.

The MHS Does Not Use a Common Set of Quality Measures for Direct and Purchased Care

Although the NDAA 2016 directed the MHS to align its quality measures  for direct and purchased care, we found that as of March 31, 2018, the  MHS used separate sets of measures on the Core and Purchased Care  Dashboards to assess the quality of care delivered in direct and  purchased care, respectively. To assess the quality of direct care, the  MHS tracks 43 measures on its Core Dashboard, and to assess the  quality of purchased care, the MHS tracks 18 measures on its Purchased  Care Dashboard. The MHS tracks 8 measures that are the same for both  dashboards, leaving 35 measures tracked only on the Core Dashboard  for direct care and 10 measures tracked only on the Purchased Care  Dashboard for purchased care. (See fig. 2.)

According to MHS officials, since launching the P4I performance  management system in 2015, the MHS has focused on making  systematic improvements in the quality of care across the MTFs in direct  care. As a result, the 43 measures they have chosen for the Core  Dashboard reflect their priorities for quality improvement within direct care  only. In the case of purchased care, MHS officials stated that requiring  civilian providers to report on the same 43 measures that are used on the  Core Dashboard for direct care would add burden, and the MHS had  concerns that this would make civilian providers less likely to participate  in purchased care. Instead, the MHS tracks 18 measures on the  Purchased Care Dashboard that rely on information sources other than  provider reporting, such as claims that the providers submit in the normal  course of receiving payment for their services and surveys that the MHS  conducts of its beneficiaries. MHS officials explained that they try to  minimize the reporting burden for purchased care providers because for  most of these civilian providers, eligible MHS beneficiaries represent only  a small proportion of their patient population.

We also found that for direct care, the MHS uses its quality measures on  the Core Dashboard to assess the quality of care delivered to  beneficiaries served by individual MTFs, such as hospitals or clinics.  However, for purchased care, the MHS uses its quality measures on the  Purchased Care Dashboard to assess the quality of care delivered to the  beneficiary population served by each contractor’s network as a whole– not the quality of care delivered by individual civilian hospitals, clinicians,  or other providers in the network. Specifically:  In direct care, the MHS uses the 43 measures on the Core Dashboard  to track the quality of care delivered by individual MTFs. For example,  on a measure of central line-associated bloodstream infections, the  MHS tracks the incidence of such infections by individual MTF and by  military service (i.e., the incidence of such infections in Army, Navy  and Air Force MTFs).

In contrast, in purchased care, the MHS assesses information on the  18 measures on the Purchased Care Dashboard for all beneficiaries  in each of the networks administered by the two managed care  support contractors. For example, on a measure of the percentage of  beneficiaries with diabetes who have their hemoglobin level tested  annually, the MHS calculates an overall rate of hemoglobin testing  across all the diabetic patients that receive care in each contractor’s  network.

The beneficiary population-level reporting on quality measures on the  Purchased Care Dashboard reflects the nature of the MHS’s relationship  with its managed care support contractors for purchased care. Under the  terms of the contracts that the MHS has negotiated with the contractors  that administer the networks of civilian providers to care for eligible  beneficiaries, the contractors bear responsibility for ensuring the quality of  care delivered by those providers. While the MHS requires the managed  care support contractors to monitor different sets of quality measures or  indicators, such as patient safety indicators, hospital acquired conditions,  and serious reportable events to identify possible cases of individual  patient harm and determine appropriate interventions, the contractors  report this information in annual reports to the MHS for their network as a  whole, as opposed to reporting on individual providers.

Because the MHS largely uses separate measures for direct and  purchased care on its dashboards and tracks the quality of care delivered  by civilian providers in purchased care in the aggregate rather than  individually, the MHS lacks the information it needs to make comparable  assessments of the quality of care delivered across the MHS as a whole.  This, in turn, limits the MHS’s ability to ensure it has the information  needed to determine whether it is achieving the department’s overall  strategic goals of providing high quality care across the MHS as a whole  and ensuring that beneficiaries receive a consistent level of high quality  care regardless of whether that care is delivered in direct or purchased  care. Moreover, using a different set of quality of measures on the  dashboards for direct and purchased care is inconsistent with section 730  of the NDAA 2016, which directs the MHS to align its measures for direct  and purchased care so it can reduce performance variation across the  MHS. MHS officials acknowledge in principle the value of using aligned  measures to assess quality of care in direct and purchased care, but the  officials cited a range of factors that pose challenges to achieving this  objective, such as the large number of civilian providers and the lack of  common health information technology systems.

Based on our review, we found that one way the MHS could have a  common set of quality measures for both direct and purchased care,  without increasing the reporting burden on civilian providers, would be to  use, as appropriate, Hospital Compare and CQMC quality measures.  Notably, the MHS states on its website that almost all of the civilian  hospitals that are in the contractors’ networks for purchased care already  report information on the measures posted on the Hospital Compare  website. As a result, there potentially would be no additional burden for  these purchased care providers to report information on the Hospital  Compare quality measures. Similarly, major health plans report that they  have begun implementing the CQMC measure sets in their contracts with  physicians, meaning that physicians participating in those plans already  report information on CQMC outpatient quality measures. To the extent  that those physicians are also in the MHS contractors’ networks for  purchased care, the information the physicians report on the CQMC  measures could be used by the MHS.

We found the MHS is already using some Hospital Compare and CQMC  measures for inpatient and outpatient care. There are a total of 76  measures that Medicare and private health insurers report to Hospital  Compare and a total of 60 CQMC outpatient measures. Besides the  measures used in the Core and Purchased Care Dashboards, MHS also  collects 24 of 76 Hospital Compare measures and 10 of the 60 CQMC  outpatient measures. For the most part, these measures are not part of  the direct and purchased care dashboards that MHS leadership uses to  assess the performance of direct and purchased care. Furthermore, MHS  officials told us that they have no specific plans to increase the number of  measures that the MHS uses from Hospital Compare for inpatient care  delivered in its hospitals. In the case of outpatient care, our review of  DOD documents shows that the MHS plans on expanding reporting to  only 5 more CQMC quality measures, in large part to minimize its  reporting burden.

The MHS Uses Quality Measures that Assess a Limited Range of Quality Care Areas and Medical Conditions Compared to Measures Adopted by Medicare and Private Health Insurers

We found that the measures the MHS uses on its Core and Purchased  Care Dashboards to assess the quality of direct and purchased care  address only a limited range of quality areas and medical conditions when  compared with the Hospital Compare and CQMC measure sets that are  adopted by Medicare and private health insurers. According to the  National Quality Forum, which plays a central role in developing and  annually reassessing the Hospital Compare measure set and also was  consulted in the development of the CQMC measure sets, the measures  used to assess quality of care should comprise an appropriate mix of  recognized measure types, including outcome measures, process  measures, experience of care measures, and cost and structure  measures. These measures should cover a broad enough range of  measure types and medical conditions so that they provide an accurate  overall assessment of the quality of care patients receive.

Based on our analysis, Table 2 below shows the limited range of  measures on the Core and Purchased Care dashboards used by the  MHS to assess inpatient care, as compared to the range of inpatient  measures that Medicare hospitals report for Hospital Compare. In  general, each of the five types of measures shown in the table below  addresses different aspects of health care quality in hospital settings. For  direct care, the MHS uses no more than one measure on its Core  Dashboard for all of these five measure types except for “Outcome”  measures; for purchased care, the MHS does not use any inpatient care  measures on its Purchased Care Dashboard.

Similarly, based on our analysis, Table 3 below shows the limited range  of measures on the Core and Purchased Care Dashboards used by the  MHS to assess outpatient care, as compared to the range of outpatient  measures that are part of the CQMC measure sets adopted by Medicare  and private health insurers. The MHS uses measures on its dashboards  that assess fewer clinical focus areas and medical conditions as  compared with those measures included in the CQMC measure sets. As  with hospital care, the difference is greatest with respect to purchased  care.

The limitations we found in the quality measures used by the MHS—the  relatively narrow range of measures as well as the relatively few  measures used across direct and purchased care—reflect the MHS’s  priorities in selecting quality measures. In short, the MHS focuses on the  value and impact of implementing individual measures, but does not  prioritize aligning the measures used across direct and purchased care or  expanding the range of medical conditions and quality areas covered in  the aggregate by the measures. The MHS’s annual assessment of quality  measures focuses only the Core Dashboard measures. For each Core  Dashboard measure for which a change is under consideration—such as  dropping, modifying, or adding another quality measure to the Core  Dashboard—MHS officials apply a standard set of criteria involving both  the feasibility of collecting the data needed for that measure and the utility  of that measure for addressing a strategic priority or promoting  performance improvement. When asked about the potential value of  increasing the number of Hospital Compare measures, MHS officials said  they need to make a value-based determination of whether the benefits of  obtaining results for any given Hospital Compare measure justified the  costs of collecting and transmitting the data required for that measure. In  discussions about potential measures for the Purchased Care  Dashboard, MHS officials also focused on the characteristics of specific  measures being considered for inclusion in the dashboard.

Because the MHS does not prioritize expanding the range of medical  conditions and quality areas covered by common measures across direct  and purchased care, the measures the MHS uses provide DOD’s senior  health care leadership with an incomplete picture of the quality of care  across the MHS. As we have noted, the MHS has reported to the  Congress that its DOD health care leaders rely on the Core and  Purchased Care Dashboard measures to establish accountability  throughout the MHS and identify areas where quality improvement is  needed. However, the current approach may not lead to the selection of  quality measures for the two dashboards that would enable MHS officials  to identify the most critical quality of care issues in the MHS. The lack of  that information, in turn, limits the ability of DOD’s senior health care  leadership to target their performance improvement efforts most  effectively in support of DOD’s overall strategic goals of providing high  quality care across the MHS as a whole.

The MHS Has Established Performance Standards and Related Corrective Action Requirements for Individual Providers in Direct Care but Not in Purchased Care

The MHS has established performance standards in direct care related to  the Core Dashboard measures and has corrective action requirements for  MTFs that do not meet the standards. However, the MHS has not  established performance standards related to the Purchased Care  Dashboard measures for individual civilian providers in purchased care  and therefore does not have related corrective action requirements for  these providers.

The MHS Has Established Direct Care Performance Standards and Related Corrective Action Requirements for Military Treatment Facilities

As part of its P4I performance management system for direct care, the  MHS has established specific performance standards that each MTF  must meet in delivering quality care to MHS beneficiaries. These  standards—some of which are under development—specify a minimum  level of performance that each MTF should achieve related to the Core  Dashboard quality measures tracked in direct care. For example, in the  case of the HEDIS All Cause Readmission measure on the Core  Dashboard, the MHS’s performance standard is that MTFs should have a  rate of unplanned acute readmissions within 30 days of an initial hospital  admission that is as good as or better than the national 75th percentile.  This performance standard is based on the readmission rates that the  National Committee for Quality Assurance, the lead entity for that  measure, has observed across U.S. hospitals.

During regularly recurring governance meetings throughout the year,  MHS governance bodies review how MTFs have performed relative to the  performance standards for the Core Dashboard measures. Our review  found that during these meetings, the governance bodies generally do not  examine the circumstances of MTFs that do not perform well on the  performance standards related to the Core Dashboard measures.  Consequently, DOD’s senior health care leadership within the  governance bodies may receive limited information on the challenges  faced by low-performing individual MTFs. However, during these  meetings, officials from the military services and DHA highlight MTFs that  are performing well on the established performance standards, and the  officials share best practices and specific strategies used to achieve high  performance.

We also found that in direct care, the MHS requires MTFs that do not  meet the MHS’s performance standards related to its Core Dashboard  measures to take corrective actions to improve the quality of care they  deliver. The military services—Army, Navy and Air Force—and DHA have  been responsible for implementing this requirement. For example, Navy  officials explained that they periodically review information collected on  the MHS’s Core Dashboard quality measures to analyze areas where  MTFs do not meet established performance standards tied to these  measures and to oversee MTFs’ efforts to correct these deficiencies.  Officials told us that each of the services exercises its discretion to  independently develop and implement the corrective actions that the  service determines best address the performance issues identified  through the use of the MHS’s quality measures. For example, to help  reduce the number of Central Line-Associated Bloodstream Infections  (CLABSI), the Army began financially awarding MTFs that performed well  on the CLABSI measure, whereas the Air Force developed a toolkit to  help providers prevent CLABSI.

As the MHS moves to transfer administration of the MTFs from the  individual military services to DHA as directed by section 702 of the  NDAA 2017, the approach for assessing performance and implementing  corrective actions is likely to change. The MHS’s recently issued  implementation plan as of June 2018 outlines some alterations to the  current performance assessment process. Specifically, MTFs will create  and submit a performance plan that will be reviewed and approved by  DHA. DHA will host monthly review sessions with MTFs to track  performance on the plan. MTFs will be evaluated using a set of measures  aligned to the Quadruple Aim that will include many but not all of the Core  Dashboard measures.

The MHS Has Not Established Performance Standards and Related Corrective Action Requirements for Individual Civilian Providers in Purchased Care

The MHS has not established performance standards related to the 18  Purchased Care Dashboard measures for individual civilian hospitals,  clinicians, or other providers in purchased care. Instead, the MHS has  established performance standards related to the 18 Purchased Care  Dashboard measures that MHS officials use to track the performance of  each of the two managed care support contractors. According to MHS  officials, the MHS does not require the contractors to ensure that each  individual hospital, physician, or other provider in these networks meets  the performance standards related to the Purchased Care Dashboard  measures. For example, in the case of a measure on the use of imaging  for low back pain, the MHS has set a performance standard for each  managed care support contractor, one that aims at avoiding excessive  imaging across the beneficiary population in the contractor’s network.  However, officials told us that the information that the MHS collects on the  measure—the number of beneficiaries in each of the contractors’  networks who receive imaging services for low back pain—does not  indicate the extent to which each individual civilian provider in the  contractor networks meets or fails to meet the performance standard.  Thus, the information the MHS obtains on the quality measure and its  related performance standard does not identify which hospitals, clinicians,  or other providers need to improve their performance in order for all  beneficiaries to receive the expected level of care quality that the  performance standard represents.

Because the MHS has not established performance standards related to  the Purchased Care dashboard measures for individual civilian hospitals,  clinicians, or other providers in purchased care, there are no related  requirements for corrective action. Instead, the MHS requires its  managed care support contractors to undertake other activities to  promote improved quality of care across civilian providers in their  networks. These include investigations of quality issues, focused reviews,  analyses of Hospital Compare data, and value-based purchasing pilots,  as discussed further below. However, our review found that these efforts  are not applied comprehensively across all individual purchased care  providers.

Investigations of Quality Issues. One approach the MHS uses to promote  improved quality of care across purchased care providers is to direct its  managed care support contractors to investigate whether individual  beneficiaries have experienced what the MHS refers to as a quality issue.  Potential quality issues are defined by the MHS as any instance when  there are indications that a purchased care provider has deviated from  what the managed care support contractors deem acceptable standards  of professional practice. The contractors can identify these potential  quality issues through beneficiary complaints; analyses of patient safety  indicators, hospital acquired conditions, and serious reportable events; or  by the MHS or contractor staff. Once potential quality issues are  identified, they are investigated by a clinician, who reviews the patient’s  complete medical record. Based on the clinician’s review of the patient’s  medical records, the clinician verifies whether or not a quality issue has  occurred and, if so, assigns the quality issue a severity level. To  address the quality issue, the managed care support contractors may  take a range of steps, including educating the provider, monitoring the  provider, notifying the appropriate state or federal bodies, and removing  the provider from the MHS’s purchased care provider network. In  practice, however, MHS officials said and documents we reviewed show  that providers are rarely removed from the network. For example, MHS  officials reported that one contractor estimated that one provider was  removed from its network over quality issues every 1 to 2 years.

Focused Reviews. Another way the MHS uses its managed care support  contractors to promote improved quality of care across purchased care  providers is through focused reviews. During these reviews, the managed  care support contractors review the medical records for a selected patient  population to determine the extent to which a specified quality concern is  a widespread problem. For example, in 2015 one contractor reviewed the  medical records of 96 beneficiaries to determine the frequency of  obstetric trauma, an injury related to vaginal deliveries. If a focused  review determines that there is a widespread quality problem, the  contractor may implement a quality improvement initiative designed to  prompt all of its network providers to address that concern, as opposed to  targeting specific providers.

Analyses of Hospital Compare Data. The MHS also requires the  contractors to conduct an annual examination of the performance of  hospitals in their networks on the different quality measures reported on  Medicare’s Hospital Compare. However, the managed care support  contractors have considerable flexibility in deciding how to structure these  analyses and how to follow-up on results. Consequently, the two  managed care support contractors have adopted different analytical  approaches to define and identify hospitals with relatively low  performance. For example, the managed care support contractors  chose to examine different quality measures and use different criteria to  identify hospitals with relatively low performance. In their most recent  annual reports issued during 2017, both managed care support  contractors indicated that they were considering contacting the lower  performing hospitals to prompt remedial action, but because no action  had yet occurred, the reports leave open what steps were ultimately taken  and how these hospitals responded. Nonetheless, these activities  suggest that the managed care support contractors have the ability to use  Hospital Compare to analyze and address individual provider  performance on a standard set of quality measures. However, the MHS  has not specified how this process should proceed, leaving it to the  managed care support contractors to decide what and how much they will  do in conducting these analyses of individual hospitals.

Value-Based Purchasing Pilots. The MHS has recently begun to test  different approaches to incentivize purchased care providers to deliver  high quality care through several value-based purchasing pilots. For  example, in February 2018 the MHS launched a maternity care pilot that  pays providers more for better performance on specified quality  measures. The pilot also implements a ‘steerage model’ approach that  identifies higher performing providers in directories provided to patients  by indicating providers as “Gold Stork” or “Silver Stork.” These pilots may  provide the MHS another way to influence the quality of care provided by  certain subsets of its purchased care providers. MHS officials stated that  although DOD has not arrived at specific goals, it plans to expand these  pilots to cover around 20 to 25 percent of its purchased care services by  2020.

The use of performance standards and corrective action requirements for  individual hospitals, clinicians, or other providers who serve MHS  beneficiaries is consistent with federal internal control standards for  monitoring, which state that management should establish monitoring  activities, evaluate the results, and remediate any deficiencies. While  the MHS has established performance standards related to its Core  Dashboard measures in direct care and has corrective action  requirements for MTFs that do not meet those standards, it has not done  so for individual civilian hospitals, clinicians, or other providers in  purchased care related to its Purchased Care Dashboard measures.  Additionally, if the MHS aligned quality measures on the Core and  Purchased Care Dashboards at the provider level, the MHS could require  its managed care support contractors to monitor the performance of  individual civilian providers relative to set performance standards  comparable to the ones that the MHS has established for MTFs. This  approach would allow the MHS to determine the extent of performance  variability, both among individual civilian providers and across MTFs and  individual civilian providers. By not establishing consistent performance  standards at the provider-level for direct and purchased care and  requiring corrective action requirements to ensure that these standards  are met by providers in both direct and purchased care, the MHS is  limited in its ability to address variation in the quality of care delivered.  This further limits the MHS’s ability to ensure that it is achieving the  department’s overall strategic goals of providing high quality care across  the MHS as whole and ensuring that beneficiaries receive a consistent  level of high quality care regardless of whether that care is delivered in  direct or purchased care.

Congress directed DOD to reduce variation in the quality of care  beneficiaries receive through the MHS. DOD has taken important steps  towards this goal by identifying a set of core measures that DOD senior  health care leadership use to assess quality of care in direct care and  another set of measures that they use to assess quality in purchased  care. DOD health care leaders rely on these measures on their Core and  Purchased Care Dashboards to establish accountability throughout the  MHS and identify areas where quality improvement is needed. However,  with few exceptions, the MHS uses different measures on its Core and  Purchased Care Dashboards to assess the quality of direct and  purchased care, making it difficult to determine the extent to which it is  ensuring consistent quality across the MHS as a whole. Furthermore, for  both direct and purchased care, the MHS uses measures on its  dashboards that assess a limited range of quality areas and medical  conditions when compared to the widely used quality measure sets  adopted by Medicare and private insurers. Without using a broader range  of available quality measures available—measures that many purchased  care providers already report to CMS and private health insurers—DOD is  missing an opportunity to better target the most critical quality of care  issues in the MHS. The limitations we identified in the MHS’s Core and  Purchased Care Dashboard quality measures reflect the fact that in its  annual measure selection process, the MHS does not prioritize aligning  the quality measures across direct and purchased care and expanding  the range of measures it uses across the two systems of care.

Finally, our review shows that while DOD has established performance  standards for the core measures in direct care and corrective action  requirements for MTFs that do not meet these standards, DOD has not  done so for individual purchased care providers. Notably, DOD does not  set clear expectations that individual purchased care providers should  meet the performance standards related to the quality measures on the  Purchased Care Dashboard. Performance standards and related  corrective action requirements are critical for holding both MTFs and  individual civilian providers accountable for providing quality care. Without  consistent standards and related corrective action requirements across  the MHS, DOD is limited in its ability to ensure that beneficiaries  consistently receive high quality care, regardless of whether they receive  that care in the direct or purchased care systems.

We are making two recommendations to the Assistant Secretary of  Defense for Health Affairs.

As MHS governing bodies conduct their recurring reviews of quality  measures selected for MHS’s Core Dashboard and Purchased Care  Dashboards, the Assistant Secretary of Defense for Health Affairs should  direct those bodies to prioritize, as appropriate, the selection of measures  that apply to both direct and purchased care at the provider level and that  expand the range of quality measure types and medical conditions that  are assessed. (Recommendation 1)

The Assistant Secretary of Defense for Health Affairs should establish, as  appropriate, performance standards related to the Purchased Care  Dashboard measures that are consistent with the MHS’s performance  standards for direct care; ensure they are applied to individual purchased  care providers; and take steps, such as amending its managed care  support contracts, if necessary, to require corrective actions to be taken  when providers do not meet those standards. (Recommendation 2)

We provided a draft of this report to DOD for review, and DOD provided  written comments, which are reprinted in appendix I. In its written  comments, DOD concurred with both of our recommendations. With  regards to the first recommendation, DOD stated that it plans to enhance  the process for selecting quality measures that apply to both direct and  purchased care; optimize use of data on Hospital Compare to expand the  types and medical conditions evaluated; augment their governance  reporting structure so that senior leadership can review quality measures  included on the Core Dashboard and Purchased Care Dashboard; and  implement the CQMC measure sets for outpatient care. Additionally, DOD  stated that it has efforts underway to create a library of all quality  measures used across direct and purchased care.

With regards to the second recommendation, DOD acknowledged the  need to strengthen accountability for meeting performance standards that  apply to both direct and purchased care providers.  It also agreed that  measures of individual provider performance in purchased care should be  augmented and consistent with measures in direct care, where possible.  DOD noted, however, that because it works through managed care  support contractors for purchased care, it can hold the contractors  accountable for meeting performance standards but cannot currently take  action against individual providers based solely on performance. Instead,  DOD stated that rather than taking a corrective action approach, it plans  to expand its value-based purchasing efforts and incentivize providers  that meet and exceed certain quality standards.  This raises concerns, as  DOD’s current plans to expand its value-based purchasing efforts would  only be applicable for between 20 and 25 percent of the services MHS  beneficiaries receive from purchased care providers by 2020, as we  noted in our report. Without having all providers managed consistently  and subject to prompt remediation of deficiencies, DOD is missing an  opportunity to improve the quality of purchased care, and it increases the  risk that not all beneficiaries will receive a consistent level of high quality  care across the MHS. Acknowledging DOD’s comment that it cannot  currently take action against individual providers based solely on  performance, we have modified our recommendation to clarify that DOD  should take the steps it determines are necessary, such as amending its  managed care support contracts, to institute corrective action  requirements for purchased care providers.

We are sending copies of this report to the Secretary of Defense and  appropriate congressional committees. The report is also available at no  charge on GAO’s website at http://www.gao.gov.

If you or your staff has any questions regarding this report, please contact  me at (202) 512-7114 or silass@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix II.

Appendix I: Comments from the Department of Defense

Appendix II: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Rashmi Agarwal, Assistant  Director; Eric Peterson Analyst-in-Charge; Muriel Brown; Shaunessye  Curry; Michael Erb; Krister Friday; Jacquelyn Hamilton; and Colbie  Holderness made key contributions to this report.

Defense Health Reform: Steps Taken to Plan the Transfer of the  Administration of the Military Treatment Facilities to the Defense Health  Agency, but Work Remains to Finalize the Plan. GAO-17-791R.  Washington, D.C.: Sep 29, 2017.

Health Care Quality: HHS Should Set Priorities and Comprehensively  Plan Its Efforts to Better Align Health Quality Measures. GAO-17-5.  Washington, D.C.: Oct 13, 2016.

VA Health Care Quality: VA Should Improve the Information It Publicly  Reports on the Quality of Care at Its Medical Facilities. GAO-17-741.  Washington, D.C.: Sep 29, 2017.

DOD’s Military Health System allows patients to use military or civilian health care providers. We examined the measures DOD's leadership reviews to assess providers' quality of care across its health system and found that DOD:

is not using common measures across all providers;

uses measures covering a limited range of medical conditions; and

hasn’t created consistent standards and requirements related to providers' performance on the measures.

We recommended that, for all providers, DOD use common key quality measures, expand their range, and develop consistent performance standards and requirements.