Electronic Health Records: VA Needs to Identify and Report System Costs

VA provides health care services to approximately 9 million veterans and their families and relies on its health information system—VistA—to do so. However, the system is more than 30 years old, is costly to maintain, and does not fully support exchanging health data with DOD and private health care providers. Over nearly 2 decades, VA has pursued multiple efforts to modernize the system. In June 2017, the department announced plans to acquire the same system—the Cerner system—that DOD is implementing. VA plans to continue using VistA during the decade-long transition to the Cerner system.

GAO was asked to review key aspects of VistA and VA's plans for the new acquisition of the Cerner system. The objectives of the review were to (1) determine the extent to which VA has defined VistA, (2) evaluate VA's annual costs to develop and sustain VistA, and (3) describe the actions VA has taken to transition from VistA to the Cerner system.

GAO analyzed documentation that defines aspects of VistA and identifies components to be replaced; evaluated the reliability of cost data, including obligations associated with the development and sustainment of VistA for fiscal years 2015, 2016, and 2017; and reviewed program documentation related to VA's program, governance, and plans to transition to Cerner.

The Department of Veterans Affairs (VA) has various documents and a database that describe parts of the Veterans Health Information Systems and Technology Architecture (VistA); however, the department does not have a comprehensive definition for the system. For example, VA has identified components that comprise VistA, identified interfaces related to the system, and collected system user guides and installation manuals. VA has also conducted analyses to better understand customization of VistA components at various medical facilities. Nevertheless, the existing information and analyses do not provide a thorough understanding of the local customizations reflected in about 130 versions of VistA that support health care delivery at more than 1,500 sites. Program officials stated that they have not been able to fully define VistA due to the decentralization of the development of the system for more than 30 years. Cerner's contract to provide a new electronic health record system to VA calls for the company to conduct comprehensive assessments to identify site-specific requirements where its system is planned to be deployed. Three site assessments have been completed and additional assessments are planned. If these assessments provide a thorough understanding of the 130 VistA versions, the department should be able to define VistA and be better positioned to transition to the new system.

VA identified costs for VistA and its related activities adding up to approximately $913.7 million, $664.3 million, and $711.1 million in fiscal years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion over the 3 years. However, of the $2.3 billion, the department was only able to demonstrate that approximately $1 billion of these costs were sufficiently reliable. In addition, the department omitted VistA-related costs from the total. The lack of a sufficiently reliable and comprehensive total cost for VistA is due in part to not following a well-documented methodology that describes how the department determined the costs for the system. As a result of incomplete cost data and data that could not be determined to be sufficiently reliable, the department, legislators, and the public do not have a complete understanding of how much it has cost to develop and maintain VistA. Further, VA lacks the information needed to make decisions on sustaining the many versions of the system.

VA has initiated a number of actions to prepare for the transition from VistA to the Cerner system. These actions include taking steps to establish and begin to staff a program office, forming a governance structure, conducting assessments at the initial sites, preparing program plans to guide the initial system implementation, and setting a program baseline to help guide implementation at the initial sites. The department's actions in these important areas are ongoing. Additional actions are in progress to address GAO's September 2018 recommendation that VA clearly define the role and responsibilities of the joint Department of Defense (DOD) and VA Interagency Program Office in the department's governance plans for the new electronic health record system. VA intends to continue maturing and fully establishing a program management organization and a program governance structure to track program progress.

GAO is recommending that VA develop and implement a methodology for reliably identifying and reporting the total costs of VistA. VA agreed with the recommendation.

The Department of Veterans Affairs (VA) manages one of the largest  health care delivery systems in the United States, providing a continuum  of health care services to more than 9 million veterans at sites throughout  the United States, the Virgin Islands, Puerto Rico, American Samoa,  Guam, and the Philippines. VA’s health information system—the Veterans  Health Information Systems and Technology Architecture (VistA)—has  been essential to the department’s ability to deliver health care to  veterans. However, this technically complex system has been in  operation for more than 30 years, is costly to maintain, and does not fully  support VA’s need to electronically exchange health records with other  organizations, such as the Department of Defense (DOD) and private  health care providers.

VA has been challenged in the past by its various attempts to modernize  VistA, and its efforts have fallen short of completion at a high cost to the  department. In June 2017, VA’s former Secretary announced that the  department would redirect its modernization efforts to replace the VistA  electronic health record (EHR). In doing so, VA planned to adopt the  same system that DOD is currently acquiring—Cerner Millennium—a  commercial-off-the-shelf (COTS) product.

You asked us to review key aspects of VistA and VA’s plans for acquiring  the new Cerner system. Our specific objectives for this engagement were  to: (1) determine the extent to which VA has defined VistA, (2) evaluate  VA’s annual costs to develop and sustain VistA, and (3) describe the  actions VA has taken to transition from VistA to the Cerner system.

To address the first objective, we examined VA documentation, which  officials familiar with VistA have cited as sources that define the system.  These documents included the VA Monograph, reports from the VA  Systems Inventory, and documents listed in the VA Software Document  Library. We also examined the VistA Product Roadmap, which described  modernization plans and achievements related to VistA.

Further, we reviewed the results of additional efforts undertaken by the  department to define VistA. For example, we reviewed analyses that the  department undertook to identify VistA components to be replaced by the  Cerner System and the department’s visual mapping of VistA. We then  compared the extent to which VA has defined VistA with elements for  defining information technology (IT) systems described in GAO’s  Standards for Internal Control in the Federal Government and our Cost  Estimating and Assessment Guide. In addition, we reviewed Electronic  Health Records Modernization (EHRM) Program documentation related  to site visits that the department and Cerner have conducted at initial  operating capability sites and planned for future sites. Finally, we  supplemented our work with interviews of officials in VA’s Office of  Information and Technology (OIT), Veterans Health Administration (VHA),  and the EHRM program office.

To address the second objective, we examined cost data provided by OIT  and VHA that was associated with the development and sustainment  (operation and maintenance) of VistA for fiscal years 2015, 2016, and  2017. The scope of our work focused on these 3 prior fiscal years  because development and sustainment cost information for full fiscal  years should have been available during the time period in which we  conducted our evaluation.

Specifically, we examined documentation of the total costs for these 3  years, including source data provided by the department for each  category of cost identified, to assess the reliability of the supporting data  consistent with best practices described in GAO’s Cost Estimating and  Assessment Guide. We also examined the documentation and controls  related to the IT systems that VA identified as the sources of these cost data. The systems included OIT’s Budget Tracking Tool and VA’s  Financial Management System.

Further, we discussed with knowledgeable EHRM program officials the  nature of the cost data, the rationale for why each cost line item was  included, and any anomalies (e.g., missing data and calculation errors)  found during our analysis. In addition, we interviewed OIT and VHA  subject matter experts and vendors identified by VA to further understand  the methodology that the department used to identify or estimate VistA  costs.

This report includes specific VistA-related cost totals for which OIT and  VHA were able to sufficiently demonstrate the reliability of the program  data. For other costs in which we were not able to make a reliability  determination, we have summarized and reported those to provide  context for the magnitude of the total costs.

To address the third objective, we examined the department’s decision  memorandums and charters establishing the Office of Electronic Health  Record Modernization and the EHRM program to manage VA’s transition  from VistA to Cerner. We also examined VA’s plans to establish a  structure for governing technical and functional issues and joint decisions  that arise with DOD. To understand how the office intended to manage  the transition from VistA to the commercial system, we reviewed the site  assessment reports to understand how the reports were used to refine  the scope of work. We reviewed the EHRM Program Management Plan  and subordinate plans used to guide the program. We also examined  documentation supporting establishment of the initial program baseline.  We supplemented our documentation reviews with information obtained  through interviews with VA officials including the Executive Director and  Chief Technology and Integration Officer for the EHRM program.  Appendix I provides a more detailed discussion of our objectives, scope,  and methodology.

We conducted this performance audit from August 2017 through July  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

VA’s mission is to promote the health, welfare, and dignity of all veterans  in recognition of their service to the nation by ensuring that they receive  medical care, benefits, social support, and lasting memorials. In carrying  out this mission, the department manages one of the largest health care  delivery systems in the United States that provides enrolled veterans with  a full range of services. These services may include primary care; mental  health care; and outpatient, inpatient, and residential treatment. VHA,  one of the department’s three major components, is responsible for  overseeing the provision of health care at all VA medical facilities.

IT is widely used and critically important to supporting the department in  delivering health care to veterans. As such, VA operates and maintains  an IT infrastructure that is intended to provide the backbone necessary to  meet the day-to-day operational needs of its medical centers and other  critical systems supporting the department’s mission. The infrastructure is  to provide for data storage, transmission, and communications  requirements necessary to ensure the delivery of reliable, available, and  responsive support to all VA staff offices and administration customers,  as well as veterans.

VA Has Begun to Acquire a New System after a Long History of Efforts to Modernize VistA

Over nearly 2 decades, VA pursued multiple efforts to modernize VistA.  However, these efforts were abandoned due to expectations of high costs  and challenges to ensuring interoperability of health data.

Beginning in December 2013, the department initiated VistA Evolution, a  joint program between OIT and VHA that focused on implementing a  collection of projects to improve the efficiency and quality of veterans’  health care. Specifically, it focused on modernizing the VistA system,  increasing the department’s data exchange and interoperability with DOD  and private sector health care partners, and reducing the time it takes to  deploy new health information management capabilities. The VistA 4  Roadmap was the key plan that the department used to guide VistA  Evolution. According to this plan, VistA Evolution was intended to result in  lower costs for system upgrades, maintenance, and sustainment.

As part of VistA Evolution, the department initiated work to, among other  things, standardize VistA instances; expand the use and functionality of  the Joint Legacy Viewer; and release enhancements to legacy  scheduling, pharmacy, and immunization systems. For example, one  focus of the VistA Evolution program over the last several years was to  standardize a core set of the system’s modules which, according to the  department, account for about 60 percent of VistA.

As part of these efforts, the department implemented a process to assess  variances in the system at individual sites. According to OIT officials, this  process led to more standardization of the code, where possible, and also  allowed sites to apply for a waiver if there was a need to continue to  operate a nonstandardized VistA instance.

Although VistA Evolution was intended to modernize aspects of the  system through December 2018, the planned scope of work was reduced  as VA redirected the department’s efforts. Specifically, in June 2017, the  former VA Secretary announced a significant shift in the department’s  approach to modernizing VistA. Rather than continue to use the system,  the Secretary stated that the department planned to acquire the same  EHR system that DOD is acquiring—Cerner Millennium. According to  the department, it has chosen to acquire this product because Cerner  Millennium should allow the entire department’s and DOD’s patient data  to reside in one system, thus, potentially reducing or eliminating the  manual and electronic exchange and reconciliation of data between two  separate systems.

Accordingly, the department awarded an indefinite delivery, indefinite  quantity contract to Cerner in May 2018 for a maximum amount of $10  billion over 10 years. Cerner is to replace the 130 instances of VistA with  a standard COTS system to be implemented across VA. This new system  is to support a broad range of health care functions including acute care,  clinical decision support, dental care, and emergency medicine. When  implemented, the new system will be expected to become the  authoritative source of clinical data to support improved health, patient  safety, and quality of care provided by VA.

The EHRM program is responsible for managing the Cerner contract  implementation. As of June 2019, the department had issued eight task  orders to Cerner to: provide project management and planning support services,  conduct site assessments at the initial operating capability sites, host the Cerner system and supporting data, perform data migration and enterprise interface development, develop a functional baseline, deploy the Cerner system at the initial operating capability sites, analyze, design, and develop a technical baseline, and provide additional interface development.

For fiscal year 2019, the program was appropriated about $1.1 billion for  planning and managing the transition from VistA to Cerner. VA’s Office  of the Deputy Secretary approves spending on EHRM activities according  to the appropriation. Further, according to the department, funds are  tracked as a major IT investment on the Office of Management and  Budget’s Federal IT Dashboard.

According to VA documentation, the EHRM program is to provide  management support and the infrastructure modernization required to  install and operate the new system. Further, the department has  estimated that an additional $6.1 billion in funding, above the Cerner  contract amount, will be needed to fund additional project management  support supplied by outside contractors, government labor costs, and  infrastructure improvements over the 10-year contract period. Each VA  medical facility is expected to continue using VistA until the new system  has been deployed.

VA plans to deploy the new EHR system at three initial operating  capability sites within 18 months of October 1, 2018, with a phased  implementation of the remaining sites over the next decade. The three  initial deployment sites, located in the Pacific Northwest, are the Mann- Grandstaff, American Lake, and Seattle VA Medical Centers and related  clinical facilities that operate the same instances of VistA. These are the  first locations where the system is expected to “go live.”

The task order to deploy the Cerner system at the three initial sites  provides a detailed description of the steps Cerner needs to take in order  to reach initial operating capability at the Mann-Grandstaff site in March  2020, and at the Seattle and American Lake sites in April 2020. According  to the schedule, the initial operating capability sites are expected to be  operational by July 2020.

GAO Has Previously Reported on VA’s Challenges in Managing Health IT and VistA Modernization

In 2015, we designated VA health care as a high-risk area for the federal  government, and we continue to be concerned about the department’s  ability to ensure that its resources are being used cost-effectively and  efficiently to improve veterans’ timely access to health care. In part, we  identified limitations in the capacity of VA’s existing IT systems, including  the outdated, inefficient nature of key systems and a lack of system  interoperability, as contributors to the department’s challenges related to  health care. In our 2019 update to the high-risk series, we stressed that  VA should demonstrate commitment to addressing its IT challenges by  stabilizing senior leadership, building capacity, and finalizing its action  plan for addressing our recommendations and establishing metrics and  mechanisms for assessing and reporting progress.

We have also issued numerous reports over the last decade that  highlighted the challenges facing VA in modernizing VistA and improving  EHR interoperability with DOD. For example,

Between July 2008 and January 2010, we issued a series of reports  related to provisions included in the National Defense Authorization  Act for Fiscal Year 2008 that required VA and DOD to, among other  things, jointly develop and implement fully interoperable EHR systems  or capabilities and establish an Interagency Program Office to be a  single point of accountability for their efforts. These reports  summarized progress made over time to set up the program office,  but also noted that the office was not positioned to function as a single  point of accountability for the delivery of the future interoperable  capabilities that the departments were planning.

In March 2011, the Secretaries of VA and DOD committed the two  departments to the development of a new common integrated  electronic health record (iEHR) system and, in May 2012, announced  their goal of implementing it across the departments by 2017.  However, in February 2014, we reported on the departments’ decision  to abandon their plans for the iEHR. Specifically, we reported that  the Secretaries of VA and DOD, citing challenges in the cost and  schedule for developing the iEHR, had announced that they would not  continue with the new system and would, instead, pursue separate  efforts to modernize or replace their existing systems and work to  ensure interoperability between them.

Further, we reported that the departments had not addressed  management barriers to effectively collaborate on their joint health IT  efforts. We made recommendations regarding, among other things,  developing a plan to describe the schedule, cost, and roles and  responsibilities for the organizations within VA and DOD involved in  acquiring, developing, and implementing the EHR systems. The  departments agreed with these recommendations and took steps to  address them.

We reported in August 2015 that VA and DOD, with guidance from the  Interagency Program Office, had taken actions to increase  interoperability between their EHR systems. However, the office had  not yet specified outcome-oriented metrics and established related  goals that are important to gauging the impact that interoperability  capabilities have on improving health care services for shared  patients. As a result, we made several recommendations to VA and  DOD to address these deficiencies and the departments agreed with  them. VA, DOD, and the Interagency Program Office subsequently  took actions that addressed the recommendations.

In a June 2018 testimony, we noted that VA had undertaken important  analyses to better understand the scope of VistA and identify  capabilities that can be provided by the Cerner system. The  department also had other key activities underway, such as  establishing program governance and EHRM program planning. We  noted that critical success factors could serve as a model of best  practices that VA could apply to enhance the likelihood that the  acquisition of the new system would be successfully achieved.

Further, in a September 2018 testimony, we summarized our  previously reported findings on the establishment and evolution of the  DOD/VA Interagency Program Office, which has been involved in  various approaches to increase health information interoperability  between the departments. We noted that the office had not been  effectively positioned to function as the single point of accountability  for the departments’ EHR system interoperability efforts as called for  in the National Defense Authorization Act for Fiscal Year 2008.

As a result of these findings, we recommended that VA clearly define  the role and responsibilities of the Interagency Program Office within  the governance plans for acquisition of the department’s new EHR  system. The department agreed with the recommendation and stated  that the Joint Executive Council, a joint governance body comprised  of leadership for both VA and DOD, had approved a role for the office.  However, as of June 2019, additional work was ongoing to clarify the  role of the Interagency Program Office in VA’s EHR acquisition.

VA Has Undertaken Efforts to Define VistA, but Additional Work Remains

In order to maintain internal control activities over an IT system and its  related infrastructure, organizations should be able to define physical and  performance characteristics of the system, including descriptions of the  components and the interfaces. Further, consistent with GAO’s Cost  Estimating and Assessment Guide, a comprehensive system definition  should identify customization and the environment in which the system  operates. While defining a complex IT system can be challenging,  having an adequate understanding of its characteristics will better position  the organization to comprehensively project and account for costs over  the life of a system or program as well as identify specific technical and  program risks. Definition of VistA remains important because VA plans to  continue using the system during the department’s decade-long transition  to the Cerner system.

VA maintains multiple documents and a database that describe parts of  VistA, including various components and interfaces. However, despite  these existing sources, OIT officials acknowledged that there is no  comprehensive definition of the VistA system. Consequently, VA has  completed a number of efforts to better define VistA and understand the  environment in which it operates and additional work is planned in the  future.

Specifically, VA has documented descriptions of the system, including the  components that comprise it. These descriptions are documented in  multiple sources: the VA Monograph, VA Systems Inventory, and VA  Document Library.

The VA Monograph is a document maintained by OIT that provides  an overview of VistA and non-VistA applications used by VHA.  According to VHA officials, the VA Monograph is the primary  document that describes the components of the system. The  Monograph describes VistA in terms of modules. For modules  identified, including VistA modules, information such as the  associated business functions, VA Systems Inventory identification  number, and a link to the VA Document Library for additional technical  information are provided.

The VA Systems Inventory is a database maintained by OIT that  identifies current IT systems at the department, including systems and  interfaces related to VistA. For systems identified, the database  includes information such as the system name, the system status (i.e.,  active, in development, or inactive), and related system interfaces.

The VA Document Library is an online resource for accessing  documentation (i.e., user guides and installation manuals) on the  department’s nationally released software applications, including  VistA.

VA has also taken steps to further define the system in its efforts to  understand VistA and the environment in which it operates. For example,  EHRM program officials recognized the need to further understand the  customization of VistA components at the various medical facilities and  have conducted analyses to do so. These analyses include:

Variance analysis: As part of its VistA Evolution program, which has  focused on standardizing a core set of VistA functionality, the  department implemented a process to compare the instances of VistA  installed at sites to the Enterprise Standard version. The results of  this analysis allowed the department to assess the criticality of each  variance, which is expected to help with VA’s transition to the Cerner  system.

Module analysis: EHRM program subject matter experts undertook  an analysis that involved reviewing and assessing capabilities  provided by VistA modules. This analysis enabled department officials  to determine whether the capability provided by a VistA module could  be provided by the Cerner system, or whether another COTS solution  would be required to support this function going forward.

Visual mapping: EHRM program officials also directed an analysis  that involved developing a notional visual mapping of VA’s health care  applications, components, and supporting systems within the health  delivery environment. The results of this analysis provided a  description of the current state of one instance of VistA and the VA  health environment, which is intended to inform the department of  possible opportunities for business process and IT improvements as it  proceeds with the Cerner acquisition.

Nevertheless, even with these analyses, VA has not yet fully defined  VistA, including, for example, identifying performance characteristics of  the system and describing the environment in which it operates. The  department’s three sources that describe VistA and the additional  analyses undertaken do not provide insight into site specific  customizations of the system. For example, the VA Monograph does not  include information on module customization at local facilities. In addition,  according to OIT officials, the systems inventory does not reflect  differences among the 130 different instances of VistA and does not take  into consideration regional and local customizations of related  components. Further, the visual mapping analysis noted that there was  not full insight of the intertwined structure of data and applications or the  various local customizations of VistA.

EHRM program officials stated that they have not been able to fully define  VistA and understand all local customizations due to the decentralization  of the development of the system and its evolution over more than 30  years. They explained that VistA’s complexity is partly due to the various  instances of the system, compounded by local customizations, which  have resulted in differences in VistA instances operating at various  facilities.

According to EHRM program documentation, Cerner’s contract calls for  the company to conduct comprehensive assessments to capture the  current state of technical and clinical operations at specific facilities, as  well as identify site-specific requirements where the Cerner system is  planned to be deployed. As of June 2019, Cerner had completed site  assessments for the three initial operating capability sites in the Pacific  Northwest and had planned additional assessments at future deployment  sites. The initial site assessments included, among other things, an  assessment of the unique VistA instances and the environment in which  the system operates. The continuation of planned site assessments  should provide a thorough understanding of the 130 VistA versions, help  the department better define VistA, and position it for transitioning from  VistA to Cerner’s COTS solution.

VA Identified Total VistA Costs of about $2.3 Billion between 2015 and 2017, but Could Not Sufficiently Demonstrate the Reliability of All Data and Omitted Other Costs

When using public funds, an agency must employ effective management  practices in order to let legislators, management, and the public know the  costs of programs and whether they are achieving their goals. To make  those evaluations for a program or for a system as large and complex as  VistA, a complete understanding of the system and reliable cost  information is required. By following a methodology and utilizing reliable  data, an agency can ensure that all costs are fully accounted for, which in  turn, better informs management decisions, establishes a cost baseline,  and enhances understanding of a system’s performance and return on  investment.

Fundamental characteristics of reliable costs are that they should be  accurate (unbiased, not overly conservative or optimistic), well- documented (supportable with source data, clearly detailed calculations,  and explanations for choosing a particular calculation method), credible  (identifying any uncertainty or biases surrounding data or related  assumptions), and comprehensive (costs are neither omitted nor double  counted). Identification of VistA’s costs remains important because VA  plans to continue using the system during the department’s transition to  the Cerner system over the next decade.

VA identified costs for VistA and its related activities adding up to  approximately $913.7 million, $664.3 million, and $711.1 million in fiscal  years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion  over the 3 years. However, of the $2.3 billion, the department was only  able to demonstrate that approximately $1 billion of these costs were  reliable. The department could not sufficiently demonstrate the reliability  of the remaining approximately $1.3 billion of VistA costs that it identified.  In addition, VA identified other categories of VistA-related costs, but  omitted these costs from the total.

VA Did Not Sufficiently Demonstrate the Reliability of Data for All VistA Costs

Of the $2.3 billion total costs for VistA, VA demonstrated that only  approximately $1 billion of these costs were reliable. Specifically, OIT  officials identified VistA-related costs within seven categories. The  officials were able to sufficiently explain why these categories were  included in the development and sustainment costs for VistA and how  they were documented by the department; the officials also presented  detailed source data for our examination. As a result of our review, we  determined that the cost data for these seven categories were accurate,  well-documented, credible, and comprehensive and, thus, sufficiently  reliable.

Table 1 provides a summary of the program costs identified for VistA by  OIT and VHA for fiscal years 2015 through 2017 that we determined to be  reliable.

As shown in the table, VA identified costs for the following seven  categories for fiscal years 2015 through 2017:

VistA Evolution – The VistA Evolution program costs were  associated with VistA strategy, system design, product development,  and program management. These costs totaled approximately $549.6  million.

Interoperability – The Interoperability program focused on sharing  electronic health data between VA and non-VA facilities, including  private sector providers and DOD. For example, interoperability  costs were associated with architecture, strategy, the Interagency  Program Office, product development, and program management.  These VistA-related costs totaled approximately $140.2 million.

Virtual Lifetime Electronic Record (VLER) Health – This program  focused on streamlining the transition of electronic medical  information between VA and DOD. These VistA-related costs were  associated with product development and program management and  totaled approximately $81.2 million.

Contracts – Contract costs for VistA Evolution included VHA’s  obligations associated with workload management, change  management, clinical requirements, and clinical interoperability.  These VistA-related costs totaled approximately $202.8 million.

Intergovernmental personnel acts – Intergovernmental personnel  acts are agreements for the temporary assignment of personnel  between the federal, state, and local governments; colleges and  universities; Indian tribal governments; federally funded research and  development centers; and other eligible organizations. These costs  accounted for VHA’s need to use outside experts from approved  entities for limited periods of time to work on VistA Evolution  assignments. The total VistA-related costs were approximately $2.4  million.

Memorandums of understanding – According to VHA,  memorandums of understanding are agreements used by the  administration to obtain the services of personnel between VA entities  for VistA-related activities. These agreements accounted for  approximately $2.3 million.

Pay – Costs in this category included salaries for VHA staff who  worked on VistA-related projects as well as travel, training, and supply  costs associated with employment. These costs totaled approximately  $34.1 million.

However, VA was not able to sufficiently demonstrate the reliability of  approximately $1.3 billion in costs related to VistA. Specifically, OIT  officials identified the additional legacy VistA costs of $1.3 billion that  generally fell into three categories:

Legacy VistA: Infrastructure, hosting, and system sustainment –  Legacy VistA costs are generally related to the maintenance of fully  operational items, such as VistA Imaging and Fileman—two key  components related to VistA’s operation. The costs also included  obligations for costs related to hosting health data in both VA and  non-VA facilities. The OIT officials and subject matter experts  estimated these total costs to be approximately $343 million during  fiscal years 2015 through 2017.

However, we were not able to determine the reliability of these costs  because, for example, source data were not well documented;  changes in the cost information provided to us during our review  indicated that the cost data may not be credible; and subject matter  experts were unclear about how to separate VistA costs from non- VistA costs.

Related software – Related software costs are associated with the  software supporting or closely integrated with VistA that were  identified by EHRM officials, yet not tracked directly for one of the  VistA-related programs. Both OIT and VHA identified software  licensing costs as VistA-related obligations. The EHRM program  reported these costs to be approximately $389 million in total during  fiscal years 2015 through 2017.

However, we were not able to determine the reliability of the costs in  this category for a variety of reasons, including that source data were  not well documented. In addition, VA officials were not clear regarding  how the total amounts in each category should be divided between  OIT and VHA. Given this confusion, we were not able to determine if  the costs were fully accurate or credible.

OIT personnel (pay and administrative) – According to EHRM  officials, OIT does not track labor costs by program. Instead, the  department provided estimations of the amount of salaries paid to OIT  government staff working on activities such as VistA Evolution,  program management, and overall support of VistA and related  applications. OIT personnel costs were estimated by the EHRM  program office to be approximately $544 million total during fiscal  years 2015 through 2017.

However, we were not able to determine the reliability of costs in this  category because assumptions made for estimating the personnel  and salary costs were not well documented and could not be verified.

VA Omitted Certain Costs from the Total Cost of VistA

In addition, VA omitted certain VistA costs from the total costs identified  for fiscal years 2015, 2016, and 2017. Specifically, VA omitted the  following costs:

Additional hosting – OIT officials stated that additional costs related  to hosting health data by an outside vendor, as well as hosting backup  VistA instances at each of the medical center sites, should also be  included in the total costs for VistA; however, VA omitted these costs  from the total for fiscal years 2015 through 2017. Specifically,  according to the officials, calculating costs for these hosting activities  requires subject matter experts to identify equipment, space, utilities,  and maintenance costs for resources allocated specifically for VistA.  However, the department has not yet developed a methodology to  calculate the costs. The officials said they were working on identifying  a reliable approach for calculating these costs in the future.

Data standardization and testing – OIT officials stated that  additional costs related to work on clinical terminology mapping and  functional testing were not included in the total costs for VistA for  fiscal years 2015 through 2017. This work related to mapping existing  clinical data to national standards and making updates to VistA or the  Joint Legacy Viewer and included mapping data and building test  scripts and reports. OIT officials noted that this work had been critical  to the VistA Evolution program, but they did not provide actual cost  data in this category.

The lack of sufficiently reliable and comprehensive costs indicates that  the department is not positioned to accurately report the annual costs to  develop and sustain VistA. This is due, in part, to the fact that VA has not  followed a well-documented methodology that describes how the  department determined the total costs for the system. In lieu of a  methodology, OIT officials said that leadership and staff from the program  took efforts to identify and track the cost components and contracts  associated with the system. However, they noted that costs associated  with VistA were not all clearly labeled as VistA in an IT system and it was  necessary to estimate other costs. The officials were also unable to verify  how VistA-related costs were separated from other department costs in  all areas and subject matter experts were not consistently familiar with the  estimation methods employed and how VistA was defined for the  purposes of calculating costs. Further, VA officials noted that they were  still working on the best approach to identifying and calculating omitted  costs.

Without documenting the methodology for what costs are to be included  and how they were identified and calculated, VA’s total does not  accurately reflect the development and sustainment costs for VistA. As a  result, the department, legislators, and the public do not have the  comprehensive, reliable information needed to understand how much it  actually cost to develop and maintain the system. Further, VA does not  have the reliable information needed to make critical management  decisions for sustaining the many versions of VistA over the next 10 years  until the Cerner system is fully deployed.

VA Has Initiated a Number of Activities to Transition from VistA to the Cerner System

VA has initiated a number of actions to prepare for the transition from  VistA to the Cerner system. These actions include (1) taking steps to  establish a program office reporting to senior agency management, (2)  forming a governance structure, (3) conducting assessments at initial  system deployment sites, (4) preparing program plans, and (5) setting an  initial program baseline. These activities represent important initial steps  to prepare for the transition to the new system. The program office is  working to hire staff and establish a joint governance structure to  coordinate with DOD on the departments’ efforts to implement the Cerner  system.

VA Has Taken Steps to Establish a Program Office Reporting to Senior Agency Management and Efforts to Hire Staff Are Ongoing

Strong agency leadership support is a key factor that can increase the  likelihood of a program’s success. For example, senior leadership can  define a vision for the program and intervene when there are difficulties.  Such leadership can come from the establishment of a program office  with staff reporting to senior agency management.

VA took steps to establish a program office, under the leadership of the  VA Deputy Secretary, to support the contract negotiations between the  department and Cerner. Toward this end, in January 2018, the  department moved the EHRM program office from OIT to directly report to  the VA Deputy Secretary. Then, after the contract with Cerner was  awarded in May 2018, a new program office—the Office of Electronic  Health Record Modernization—was established in June 2018 to plan and  implement the EHRM program. The office is intended to coordinate with  OIT and VHA leadership—specifically, VA’s CIO and VHA’s Under  Secretary for Health—under the direction of an Executive Director. The  Executive Director reports directly to the VA Deputy Secretary. Reporting  to the Executive Director is the Deputy Executive Director, whose  responsibilities include supporting the program’s execution and  management, ensuring the program’s direction is in alignment with VA’s  desired outcomes, and identifying strategic challenges related to the  program.

The Office of Electronic Health Record Modernization also includes three  management structures:

The Chief Medical Office is responsible for overseeing strategy and  planning efforts for change management, user testing and training,  and business process re-engineering. It also leads communication  efforts for the end users and deployment.

The Technology and Integration Office is responsible for providing  technical leadership, management, and oversight of IT. As such, the  office approves technical requirements and supports interoperability  with DOD, as well as performs information security, architecture, data  migration and management, configuration management, infrastructure  engineering, transition and data engineering, and development.

The Program Management Office is responsible for, among other  things, providing program control support for the scope, schedule,  quality, and risk management for the EHRM program; human  resources support for the Office of Electronic Health Record  Modernization government staff; financial management for operating  plans, budgets, cost estimates and reporting; test and evaluation  support; and oversight of contracts providing staffing to the EHRM  program.

As of May 2019, VA was still working to fully staff the Office of Electronic  Health Record Modernization. Figure 1 shows the organization of the  Office of Electronic Health Record Modernization.

According to program officials and the Office of Electronic Health Record  Modernization organization chart, the office is expected to be staffed by  289 government employees. These positions are expected to be filled by  April 2020 and represent the staff required for the program to achieve its  initial operational capability. According to the program’s January 2019  hiring plan, the office had begun its process to reassign staff and hire  additional government employees.

VA also awarded a contract for program management support. According  to EHRM program officials, the support contractor is to supplement the  Office of Electronic Health Record Modernization staff with program and  project management support, technical support, community care support,  and executive support and internal communications, among other areas.  The support contractor provides about 370 personnel to deliver project  management support. The contractor reported as of January 2019 that it  had achieved the following accomplishments, among others:

Developed a Project Readiness Assessment Report including roles,  schedules, risk, and measures of success within the Chief Medical  Office.

Developed a survey to identify key clinical priorities for data migration  related to patient safety and clinical quality.

Coordinated the site visit schedule and logistics with initial operating  capability sites and conducted site surveys at eight outpatient clinics.

By establishing a program office reporting to the Deputy Secretary, VA  has begun to build a framework to demonstrate senior agency  management support of the program. Establishing the program office also  focuses oversight and program management of the EHRM program.

VA Has Established Program Governance and Is Working on Developing a Joint Management Structure with DOD

Implementing collaborative governance brings together key agency  executives to discuss investment performance and increases  accountability. In addition, it is critical for program officials to be actively  engaged with stakeholders to ensure the success of a major acquisition.

The department has established a governance structure that includes  multiple levels of governance bodies and stakeholders. In addition, VA  has prepared charters for the governance boards and identified board  membership. According to the charters for the governance bodies, the  structure is intended to address technical and functional issues, as well  as any joint management issues that arise between VA and DOD as both  departments implement the Cerner EHR.

As of January 2019, the EHRM program governance structure was  comprised of a Steering Committee, Governance Integration Board,  Functional Governance Board, Technical Governance Board, and EHR  Councils. EHRM program officials have stated that the charters for these  boards, which describe their membership and responsibilities, will  continue to evolve as the program matures.

The Steering Committee, the highest board in the program  governance structure, advises the VA Secretary on the progress and  performance of the EHRM program toward meeting program goals  and outcomes and providing strategic direction on program  implementation. This committee is chaired by the Deputy Secretary of  VA. Voting members of the committee include, among others, the VA  CIO and the Under Secretary for Health.

According to the draft charter, the Steering Committee is expected to  resolve any items that cannot be resolved at the level of the next  lower-level board and is to meet at least quarterly. However, as of  January 2019, the Steering Committee had not met. According to  program officials, other reviews, such as a monthly program review  with the Deputy Secretary, beginning in November 2018, have  provided executive-level oversight of the EHRM program and have  met the purpose of the Steering Committee.

The Governance Integration Board is responsible for integrating and  communicating efforts across all lower program governance boards  (including the Functional Governance Board and the Technical  Governance Board) to meet program goals and milestones. The  board has three voting members: the Office of Electronic Health  Record Modernization Executive Director, the Assistant Deputy Under  Secretary for Health, and the Principal Deputy Assistant Secretary for  OIT. According to the charter, this board is expected to act as  arbitrator between clinical, technical, and budget priorities and  adjudicate items that cannot be resolved at the lower-level boards.

In addition, the Governance Integration Board serves as the EHRM  program Configuration Control Board. According to the charter, the  board is to meet on a monthly basis. According to program officials  and meeting minutes, as of January 2019, the Governance Integration  Board had met six times.

The Functional Governance Board is responsible for providing  guidance on the functional and business community needs for the  EHR modernization efforts. This board interacts with the Technical  Governance Board as a functional and business advisor. The  Functional Governance Board is chaired by the program office’s Chief  Medical Officer and includes members from a variety of VHA  functional areas (e.g., nursing, community care, and patient safety).  According to the charter, the board is to meet on a biweekly basis and  is to provide guidance to address functional decisions escalated from  the EHR Councils. According to program officials and meeting  minutes, as of January 2019, the Functional Governance Board had  met 10 times.

The Technical Governance Board is responsible and accountable for  all decisions related to EHRM program technical transformation  efforts. The board is expected to provide technical decision  recommendations and collaborate with DOD and other external  partners. The chair of this board is the Office of Electronic Health  Record Modernization’s Chief Technology and Integration Officer.  Other voting members include an OIT CIO representative and  selected technical directors from within the Office of Electronic Health  Record Modernization. The board’s draft charter specifies that it is to  meet on a biweekly basis. According to EHRM program officials, as of  January 2019, the Technical Governance Board had met 16 times.

The EHR Councils are working groups comprised of subject matter  experts from both clinical and functional (i.e., business) domains that  are to work with Cerner to provide input and recommendations for  developing and validating standard workflows. As of October 2018, a  total of 12 councils had been established to address clinical  processes and six councils had been established to address business  processes. A total of 121 VHA field office staff and 100 VHA central  office staff were appointed to these councils.

In addition, the councils have eight planned national workshops and  seven planned local workshops. These workshops are ongoing and  are expected to be completed by October 2019. According to program  officials, the national workshops are intended to establish a national  baseline for workflow configuration decisions. The local workshops  are to review the national baseline and make integration decisions to  suit local needs.

Figure 2 depicts the relationships among VA’s EHRM program  governance bodies.

In addition to the program’s governance, the Secretaries of VA and DOD  issued a joint memorandum in September 2018 asserting the need to  establish a joint management structure, which could have responsibilities  beyond those currently within the purview of the Interagency Program  Office. According to the agency officials, the joint management structure  will be expected to leverage lessons learned by DOD from its experience  in deploying the Cerner system, such as the timing of infrastructure  upgrades. Further, in December 2018, the departments chartered a Joint  Electronic Health Record Modernization Work Group to assess the  departments’ existing EHR modernization strategies and efforts.  According to its charter, the work group is also intended to develop and  design recommended approaches, processes, and organizational  structures to optimize the use of the departments’ resources in pursuit of  EHR interoperability objectives.

The joint working group is to develop short- and long-term  recommendations to support four objectives to provide: a single accountable authority to facilitate decision-making and an organizational structure to support the delivery of a single,  coordinated clinical and business workflows; and a coordinated implementation plan and detailed timelines.

According to EHRM program officials, the joint working group is to define  the joint management structure to be used to coordinate between the  departments. According to the charter, the goal is for the recommended  joint organization to be operational by the end of September 2019.

VA and Cerner Conducted Site Assessments to Refine the Scope of Work

As previously discussed, according to EHRM program officials, the  department determined that site-specific assessments are required to  allow Cerner to appropriately identify the requirements for system  implementation at each site. To refine the scope of work required for  initial operating capability, Cerner and the department conducted  assessments, beginning in July 2018, at the three sites identified to be  part of the initial operating capability of the program. These site  assessments included, among other things, an assessment of the IT  infrastructure at each site and identification of site-specific requirements.

Additional site assessments are planned at every facility before the  Cerner system will be deployed at each location. According to the task  order, the assessments are expected to provide perspective on the  current state of technical and clinical operations of each facility beyond  VA’s current documentation. For example, Cerner is expected to  document all interfaces with medical devices, third-party systems and  other data sets at each site, as well as update monthly a site readiness  checklist to inform comprehensive deployment planning.

According to the assessments of the three initial operating capability  sites, a number of issues have been identified such as updating or  replacing infrastructure and workstations to be compatible with the Cerner  COTS system. In addition, according to the site assessments, the  services offered by the department, such as telehealth and behavioral  health, are generally more expansive than commercial deployments and  will require increased collaboration between VA and Cerner to meet  business and system requirements. Thus, the assessments are intended  to position Cerner and the department to have more information readily  available in order to better plan for site-specific issues prior to actual  implementation.

VA Is Preparing Program Plans for Implementation

Program planning is critical for ensuring effective management of key  aspects of an IT program and serves as the basis for controlling and  managing project performance. These key aspects include, for example,  identification of the program’s scope, responsible organizations, costs,  and schedules.

The Office of Electronic Health Record Modernization Executive Director  approved an initial Program Management Plan for the EHRM program in  November 2018. According to the plan, it is to be used to guide the  management of the EHRM program and defines the program’s policies  and processes necessary to achieve the program’s goals. It briefly  defines the program’s scope and strategy, including the assumptions  made. For example, according to the plan, the EHRM program assumes  that VA and DOD will use a single instance of the Cerner system. Further,  it states that both the legacy VistA data and EHRM data will be available  to both VistA and new system users during the transition.

The Program Management Plan also identifies a series of subordinate  plans that have been developed to further elaborate on specific program  planning and execution activities. For example, the plan summarizes the  Deployment Management Plan, which details the strategy and tasks  required from initial site assessment through configuration, testing,  training, change management, deployment, and transition to sustainment.

The plan also describes the Schedule Management Plan, which defines  the development and maintenance of the integrated master schedule for  the life of the program. Thus, the Program Management Plan provides the  guidance for where to look for key planning information for the  department.

The EHRM program also developed a draft Risk Management Plan,  dated September 2018, that defines how risk and issue planning,  analysis, and management are to be implemented. The draft risk  management process consists of risk identification and mitigation,  including conducting risk management planning, identification, analysis,  response planning, response identification, and monitoring. According to  the plan, management of overall program risk is intended to keep risk  exposure within an acceptable range and maximize the likelihood of  achieving overall objectives.

In addition, the EHRM program developed plans for change  management, communications, and training activities to ensure that VA  clinicians, staff members, volunteers, and veterans understand and are  ready for the changing systems and processes that will impact them. The  initial versions of the plans were delivered by Cerner in November 2018.  The program’s approach is to continue to evolve these plans as the  program matures. By developing these program plans, VA is taking steps  to ensure effective management of key aspects of the EHRM program.

VA Established a Program Baseline for Achieving Initial System Deployments

Baselined program plans act as a guide throughout the life of an  investment to provide a basis for measuring performance, identify who is  accountable for the deliverables, describe the implementation approach  and interdependencies, identify key decisions, and embed quality  assurance and reviews. Ultimately, baseline management demonstrates  that a project is under financial and managerial control.

According to EHRM program officials, on October 30, 2018, the program  conducted a review of the time period from contract award through initial  operating capability. The review validated the scope of the program for  the transition of VistA to the initial operating capability sites, identified an  initial work breakdown structure, and included an integrated master  schedule and a cost baseline. The results of this review established a  baseline for the initial operating capability and changes to the baseline  are subject to change control. Also, as a result of the review, the Office of  Electronic Health Record Modernization is to conduct monthly program  reviews to inform the Deputy Secretary of the status of the EHRM  program.

According to EHRM program officials, upgrades to the IT infrastructure  are to be accomplished by OIT, and the local area network infrastructure  is to be upgraded at all initial operating capability sites prior to  implementation of the new system. As baselined, upgrades of end user  devices are scheduled to be completed at the Mann-Grandstaff site by  September 2019, the American Lake site by October 2019, and the  Seattle site by November 2019. Program officials have stated that the  goal is to have infrastructure upgrades at a site completed 6 months  before the site begins to implement the Cerner system. However, in May  2019, EHRM program officials indicated that infrastructure updates may  be delayed for the initial sites by up to 3 months.

After an evaluation of the initial operating capability, the EHRM program  is to determine whether the minimum operational capabilities have been  achieved. Figure 3 shows a timeline of the baselined implementation  milestones for the initial sites, established at the review held in October  2018.

The baseline review also included identifying and addressing program  risks related to the Cerner system implementation. The review identified  10 program risks, prioritized the risks by probability and impact, and  assigned mitigation plans for the risks. For example, the review identified  the risk that if required infrastructure upgrades were not implemented,  then VA would not be able to deploy a fully operational EHR system. The  program identified development of acquisition strategies to address  infrastructure requirements from the site assessments as an action to  mitigate this risk.

By establishing a program baseline for the initial operating capability, VA  has instituted a basis for measuring actual versus planned program  performance. In addition, the risk mitigation plans provide an approach to  address the identified risks.

VA lacks a comprehensive definition of the VistA system that captures the  complexity of the system, the environment in which it operates, and the  local customizations that have evolved in the VistA instances over many  years. Consequently, VA has engaged in efforts to provide additional  insight into the system. The department plans to continue to conduct  comprehensive site-specific assessments with Cerner to refine its  understanding of the unique VistA instances and the environment in  which the system operates. The continuation of planned site assessments  should help VA better define VistA.

With regard to calculating costs for VistA, the department has identified  reliable costs for approximately $1 billion in development and sustainment  for the system over 3 fiscal years. However, VA was not able to  sufficiently demonstrate the reliability of an additional $1.3 billion of costs  identified and omitted other relevant costs from the total. The cost  deficiencies existed largely because VA officials were uncertain about  what to identify as part of VistA; documentation related to certain  categories of costs was incomplete; and a documented methodology for  identifying and reporting those costs does not exist. As a result, VA lacks  the comprehensive and reliable cost information needed to make critical  management decisions for sustaining the system and ensuring an  accurate basis for reporting on the return on its investment for replacing  VistA.

VA has taken a number of actions to prepare for the transition from VistA  to the Cerner system, such as establishing and beginning to staff a  program office, forming a governance structure, conducting site  assessments at initial sites, preparing program plans to guide the initial  implementation, and setting an initial program baseline to help guide  implementation of the system at three key sites.

The Secretary of VA should direct the Under Secretary for Health and the  Assistant Secretary for Information and Technology/Chief Information  Officer to develop and implement a methodology for reliably identifying  and reporting the total costs of VistA. The methodology should include  steps to identify the definition of VistA and what is to be included in its  sustainment activities, as well as ensure that comprehensive costs are  corroborated by reliable data. (Recommendation 1)

VA provided written comments on a draft of this report. In its comments  (reprinted in appendix II), the department generally agreed with our  conclusions and concurred with our recommendation. The department  stated that it will provide the actions it plans to take to address the  recommendation within 180 days. VA also provided technical comments,  which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of VA, and other interested parties. In addition,  the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-4456 or harriscc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to: (1) determine the extent to which the Department  of Veterans Affairs (VA) has defined the Veterans Health Information  Systems and Technology Architecture (VistA), (2) evaluate VA’s annual  costs to develop and sustain VistA, and (3) describe the actions VA has  taken to transition from VistA to the Cerner system.

To address the first objective, we examined VA documentation including  the VA Monograph, reports from the VA Systems Inventory, and  documents listed in the VA Software Document Library. These  documents were cited by VA officials as sources that define the VistA  system and provide information on modules and interfaces. Our review  and compilation of information from these three sources enabled us to  describe the various sources used at the department to document  information about the VistA system and identify the limitations of each  source. We also examined the VistA Product Roadmap, which described  modernization plans and achievements related to VistA. Further, we  interviewed officials from the Veterans Health Administration (VHA) to  obtain information on additional efforts undertaken by the department to  further understand and define VistA.

In addition, we reviewed program documentation related to three  analyses undertaken by VA to further define VistA. These analyses  included the department’s efforts to ascertain variances between versions  of VistA, identify components of VistA to be replaced by the Cerner  System, and document the current state of a sample instance of VistA.  For example, we examined VA documentation that described software  modules available in the department’s VistA product and program  documentation identifying components of VistA to be replaced by the  Cerner system. In addition, our review of a visual mapping developed for  Electronic Health Record Modernization (EHRM) program officials  depicting the environment in which VistA operates allowed us to describe  the size and complexity of the system and how it is used by the  department.

Further, we compared the extent to which VA has defined VistA with  criteria for defining information technology (IT) systems described in  GAO’s Standards for Internal Control in the Federal Government and our  Cost Estimating and Assessment Guide. In addition, we reviewed EHRM program documentation related to site assessments that have taken  place at initial operating capability sites and are planned for future sites.  Specifically, we reviewed the relevant contract task order to understand  how site assessments were planned and to identify site-specific gaps  between the current VistA system in use and the target future Cerner  system. We supplemented our documentation reviews with information  obtained through interviews with officials from VA’s Office of Information  and Technology (OIT), VHA, and the EHRM program office.

To address the second objective, we examined department  documentation of costs associated with the development and sustainment  (operation and maintenance) of VistA for fiscal years 2015, 2016, and  2017. These 3 fiscal years were selected because development and  sustainment cost information for full fiscal years should have been  available during the time period in which we conducted our evaluation. To  compile the total costs, we examined all categories of costs identified by  VA to determine reliability of the source data. We also discussed the  methodology VA used related to identifying costs and estimating costs  when source data was not available with officials from the EHRM  program. We compared the identified cost data to best practices  described in GAO’s Cost Estimating and Assessment Guide that are the  basis for effectively capturing reliable program costs. The guide also  describes the importance of documenting the methodology by which  costs are included and how they are calculated in detail, step by step, to  provide enough information so that someone unfamiliar with the program  could easily recreate or update cost calculations.

Specifically, we analyzed all cost documentation provided by the  department over the course of our work. For example, OIT officials  identified VistA costs tracked under three programs—VistA Evolution,  Interoperability, and Virtual Lifetime Electronic Record (VLER) Health– and VHA officials reported that costs for the system were tracked  separately from OIT through various types of contracts and agreements  associated with VistA Evolution. In regard to the OIT and VHA program  data, VA provided detailed source data that we analyzed for reliability and  verified the calculations of costs identified over the course of our work.

We also examined the documentation and controls related to the IT  systems VA identified as the source of these cost data. The systems  included OIT’s Budget Tracking Tool and VA’s Financial Management  System. Further, we discussed the nature of the cost data, the rationale  behind why each cost line item was included, and any anomalies found  during our analysis with cognizant OIT and VHA officials. For example,  anomalies included omitted contract numbers or transposed entries in  summary tables. As a result of these efforts, OIT and VHA were able to  sufficiently demonstrate the reliability of the program data for the purpose  of calculating costs for VistA.

Officials from the EHRM program also identified costs that were not  directly tracked under the program areas previously mentioned. OIT and  VHA relied upon subject matter experts or vendors to identify costs or to  calculate estimates for cost categories such as sustainment,  maintenance, co-location, hosting, pay, administrative, and infrastructure  costs related to VistA operations. We analyzed the data provided for  reliability consistent with GAO Cost Estimating and Assessment Guide  over the course of our work.

Further, we discussed the nature of the cost data, the rationale behind  why each cost line item was included, and any anomalies found during  our analysis with cognizant OIT and VHA officials. We also interviewed  OIT and VHA subject matter experts and vendors identified by VA to  examine the rationale or methodology for how the costs were identified  and estimated. During the course of our work, VA continued to revise  these estimates as part of the department’s efforts to identify the costs for  VistA and could not provide a consistent, documented methodology for  how the costs were calculated or provided only summary costs that could  not be analyzed. As such, VA was not able to sufficiently demonstrate the  reliability of legacy VistA, related software, and OIT personnel costs for  our purpose of calculating the total costs for VistA. This report does not  conclude that the data are unreliable, only that a reliability determination  could not be made during the course of our work. However, given the  importance of these related costs to VistA, we have summarized and  reported these costs in the total cost amount for VistA to more accurately  approximate the magnitude of total costs, but have not reported itemized  costs in these areas.

Finally, the department identified that there were additional costs that  should be included in the compilation of the total costs for VistA related to  additional hosting costs and data standardization and testing. However,  the department did not provide such data to include in the total costs for  VistA.

To address the third objective, we examined the department’s decision  memorandums and charters establishing the Office of Electronic Health  Record Modernization and the EHRM program to manage VA’s transition  from VistA to Cerner. We also examined the statement of work for the  program support contract as well as VA’s draft charters, program  briefings, and organization charts that describe plans to govern the  program to acquire the Cerner system. Specifically, we examined VA’s  plans to establish a structure for governing technical and functional issues  and joint decisions that arise with the Department of Defense.

To understand how site assessments were used to refine the scope of  work, we examined the site assessment task order and the site  assessment reports. To understand how the program office plans to  manage the program, we examined the EHRM Program Management  Plan and subordinate plans that guide the management of the program  and describe ongoing efforts to define the policies and processes  necessary to achieve the program’s goals. To address the program’s  establishment of an initial program baseline, we examined the decision  memorandum approving the award of the Cerner contract, the briefings  presented to program stakeholders at the initial program baseline review,  and the documents supporting the program baseline review. We  supplemented our analysis with information obtained through interviews  with relevant department officials including the Executive Director and  Chief Technology and Integration Officer for the EHRM program.

We conducted this performance audit from August 2017 to July 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Comments from the Department of Veterans Affairs

In addition to the contact named above, Mark Bird (Assistant Director),  Jennifer Stavros-Turner (Analyst in Charge), John Bailey, David Blanding,  Chris Businsky, Juaná Collymore, Rebecca Eyler, Jacqueline Mai, Scott  Pettis, and Charles Youman made key contributions to this report.

The VA’s health information system is more than 30 years old and is costly to maintain. Over nearly 2 decades, VA’s multiple modernization efforts have continually fallen short.

In June 2017, VA announced it would buy the same system DOD is implementing. VA plans to continue using its current system during a decade-long transition.

We reviewed the early stages of this transition. Among other things, we found VA lacked insight into the total costs for the current system. This could make it more difficult to make decisions during the transition to the new system.

We recommended VA more reliably identify and report system costs.