VA Disability Benefits: Improved Planning Practices Would Better Ensure Successful Appeals Reform

VA's disability compensation program pays cash benefits to veterans with disabilities connected to their military service. In recent years, the number of appeals of VA's benefit decisions has been rising. For decisions made on appeal in fiscal year 2017, veterans waited an average of 3 years for resolution by either VBA or the Board, and 7 years for resolution by the Board. The Veterans Appeals Improvement and Modernization Act of 2017 makes changes to VA's current (legacy) appeals process, giving veterans new options to have their claims further reviewed by VBA or appeal directly to the Board. The Act requires VA to submit to Congress and GAO a plan for implementing a new appeals process, and includes a provision for GAO to assess VA's plan.

This report examines the extent to which VA's plan (1) addresses the required elements in the Act, and (2) reflects sound planning practices identified in prior GAO work. GAO reviewed and assessed VA's appeals plan and related documents against sound planning practices, and solicited VA's views on its assessments.

The Department of Veterans Affairs' (VA) plan for implementing a new disability appeals process while attending to appeals in the current process addresses most, but not all, elements required by the Veterans Appeals Improvement and Modernization Act of 2017 (Act). VA's appeals plan addresses 17 of 22 required elements, partially addresses 4, and does not address 1. For example, not addressed is the required element to include the resources needed by the Veterans Benefits Administration (VBA) and the Board of Veterans' Appeals (Board) to implement the new appeals process and address legacy appeals under the current process. VA needs this information to certify, as specified under the Act, that it has sufficient resources to implement appeals reform and make timely appeals decisions under the new and legacy processes.

VA's appeals plan reflects certain sound planning practices, but it could benefit from including important details in several key planning areas:

Performance measurement : VA's plan reflects steps taken to track performance, but could articulate a more complete and balanced set of goals and measures for monitoring and assessing performance on a range of dimensions of success. Specifically, the plan reports that VA is developing a process to track timeliness of the new and legacy processes. However, contrary to sound planning practices, the plan does not include timeliness goals for all five appeals options available to veterans, does not include goals or measures for additional aspects of performance (such as accuracy or cost), and does not explain how VA will monitor or assess the new process compared to the legacy process. Unless VA clearly articulates a complete and balanced set of goals and measures, it could inadvertently incentivize staff to focus on certain aspects of appeals performance over others or fail to improve overall service to veterans.

Project management : VA's plan includes a master schedule for implementing the new appeals plan. However, this schedule falls short of other sound practices for guiding implementation and establishing accountability, such as articulating interim goals and needed resources for, and interdependencies among, activities. Unless VA augments its master schedule to include all key activities and reflect sound practices, VA may be unable to provide reasonable assurance that it has the essential program management information needed for this complex and important effort.

Risk assessment : VA has taken steps to assess and mitigate some risks related to appeals reform by, for example, pilot testing two of the five appeals options through its Rapid Appeals Modernization Program (RAMP). However, as designed, RAMP does not include key features of a well-developed and documented pilot test. For example, VA has not articulated how it will assess RAMP before proceeding with full implementation. In addition, RAMP is not pilot testing three options and, as a result, VA will not have data on the extent to which veterans will appeal directly to the Board when given the option. Unless VA identifies and mitigates key risks associated with implementing a new process, VA is taking a chance that untested aspects will not perform as desired.

GAO recommends that VA (1) fully address all legally required elements in its appeals plan, (2) articulate how it will monitor and assess the new appeals process as compared to the legacy process, (3) augment its master schedule for implementation, and (4) address risk more fully. VA agreed with GAO's recommendations and outlined its planned actions to address them.

The Department of Veterans Affairs (VA) provides cash benefits to  veterans for disabling conditions incurred in or aggravated by military  service, paying an estimated $67 billion to about 4.3 million veterans in  fiscal year 2016. If veterans are dissatisfied with VA’s initial decision they  can appeal—first to the Veterans Benefits Administration (VBA) and then,  if not satisfied there, to the Board of Veterans’ Appeals (Board), a  separate agency within VA. In recent years, the number of appeals of  VA’s benefit decisions has been rising. For appeals resolved in fiscal year  2017, veterans waited an average of 3 years from the date they initiated  their appeal to resolution by either VBA or the Board—and a cumulative  average of 7 years for appeals resolved by the Board. Due in part to the  challenges VA faces managing large workloads and deciding disability  claims and appeals in a timely manner, GAO in 2003 designated VA  disability compensation and other federal disability programs as one of  the government’s highest management risks.

In a March 2017 report, we examined VA’s approaches to address  challenges it identified as contributing to lengthy appeals processing  times—including VA efforts to hire staff, propose reform legislation to  Congress in April 2016, and upgrade its information technology (IT)  systems—and the extent to which those approaches were consistent with  sound planning practices. We made five recommendations to improve  VA’s ability to implement its proposed appeals process reform while  addressing a growing appeals workload. VA agreed in principle with our  five recommendations, which remain open as of March 2018. We  recommended, in essence, that VA develop: (1) a detailed workforce  plan, (2) a complete schedule of IT updates, (3) better estimates of future  workloads and timeliness, (4) a robust plan for monitoring appeals reform, and (5) a strategy for assessing whether the new process improves  veterans’ experiences over the current process. We also suggested that  Congress require VA to pilot test appeals reform.

Enacted on August 23, 2017, the Veterans Appeals Improvement and  Modernization Act of 2017 (Act) will make changes to VA’s appeals  process. Specifically, the Act replaces the current appeals process with  one that gives veterans various options to have their claim reviewed  further by VBA or to bypass VBA and appeal directly to the Board. The  Act also requires VA to submit a comprehensive plan for implementing  the new appeals process and processing legacy appeals (appeals that  remain pending in the current process prior to fully implementing appeals  reform) to the appropriate committees of Congress and GAO. The Act  delineates the required elements of this plan, and required VA to submit  its plan within 90 days of enactment. VA submitted its plan on November  22, 2017. The Act also includes a provision for GAO to assess VA’s  appeals plan, including whether the plan comports with sound planning  practices and/or contains gaps.

This report examines the extent to which VA’s appeals plan (1) addresses  the required elements in the Act; and (2) reflects sound planning practices  identified in prior GAO work.

To assess the extent to which VA’s plan addresses the required elements  in the Act, we identified the required elements under section 3(a) and (b)  of the Act; compared VA’s appeals plan and supplemental materials VA  provided at our request to those elements; and made a preliminary  determination as to whether VA’s plan addressed, partially addressed, or  did not address each element. We then shared the results of this review  with VA officials, and considered their comments in arriving at our  assessment.

To address the extent to which VA’s plan reflects sound planning  practices, we compared the appeals plan and supplemental materials  against relevant sound planning practices and other criteria identified in  our prior work.

For both assessments, our analyses focused on the information and  elements VA presented in its appeals plan and in supplemental materials  rather than auditing the underlying information. We shared our findings  related to both assessments with VA. Specifically, in advance of a  January 30, 2018 hearing based on our ongoing work, we provided VA a  draft of our testimony statement for review and comment. At a January  17, 2018 meeting, VA officials provided us with comments on our  findings, including our findings of how VA’s plan compared to sound  practices, which we have incorporated in this report, as appropriate.

We conducted this performance audit from October 2017 to March 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

VA’s Disability Compensation Claims Process

VA’s process for deciding veterans’ eligibility for disability compensation  begins when a veteran submits a claim to VA. The veteran submits his or  her claim to one of VBA’s 56 regional offices, where staff members assist  the veteran by gathering additional evidence, such as military and  medical records, that is needed to evaluate the claim. Based on this  evidence, VBA decides whether the veteran is entitled to compensation  and, if so, how much.

A veteran dissatisfied with the initial claim decision can generally appeal  within 1 year from the date of the notification letter sent by VBA. Under  the current appeals process (now referred to by VA as the legacy  process), an appeal begins with the veteran filing a Notice of  Disagreement. VBA then re-examines the case and generally issues a  Statement of the Case that represents its decision.

A veteran dissatisfied with VBA’s decision can file an appeal with the  Board. In filing that appeal, the veteran can indicate whether a Board  hearing is desired. Before the Board reviews the appeal, VBA prepares  the file and certifies it as ready for Board review. If the veteran requests a  hearing to present new evidence or arguments, the Board will hold a  hearing by videoconference or at a local VBA regional office. The Board’s  members, also known as Veterans Law Judges, review the evidence and  either issue a decision to grant or deny the veteran’s appeal or refer (or  remand) the appeal back to VBA for further work.

The 2017 Act made changes to VA’s legacy appeals process that will  generally take effect no earlier than February 2019, which is  approximately 18 months from the date of enactment. According to its  appeals plan, VA intends to implement the Act by replacing the current  appeals process with a process offering veterans who are dissatisfied  with VBA’s decision on their claim one of five options: two of those  options afford the veteran an opportunity for an additional review of VBA’s  decision within VBA, and the other three options afford them the  opportunity to bypass additional VBA review and appeal directly to the  Board.

Under the new appeals process, the two VBA options will be:  1.  Request higher-level review: The veteran asks VBA to review its  initial decision based on the same evidence but with a higher-level  official reviewing and issuing a new decision.  2.  File supplemental claim: The veteran provides additional  evidence and files a supplemental claim with VBA for a new  decision on the claim.

The three Board options will be:  3. Request Board review of existing record: The veteran appeals  to the Board and asks it to review only the existing record without  a hearing.  4. Request Board review of additional evidence, without a  hearing.  5. Request Board review of additional evidence, with a hearing.

The Act also requires VA to submit to the appropriate committees of  Congress and GAO, within 90 days of the date of enactment, a  comprehensive plan for (1) processing appeals under the legacy process  until there are no more to process, (2) implementing the new appeals  process, (3) processing of claims under the new appeals process in a  timely manner, and (4) monitoring implementation of the new appeals  process. In addition to these four broad elements, the Act lists 18  elements required to be included in the plan that relate to, among other  things:  staffing, IT, and other resources required to implement the plan; estimated timelines for hiring and training VA employees; and a description of risks associated with each element of the plan.

The Act also includes a provision for GAO to assess the plan within 90  days after VA submits it.

The Act also requires VA to provide progress reports to the appropriate  committees of Congress and GAO at least once every 90 days (starting  after VA submits its plan), until the date the Act’s legal changes to the  appeals process generally go into effect and then at least once every 180  days after this date for 7 years.

Rapid Appeals Modernization Program (RAMP)

The Act also authorized VA to carry out a program to test any  assumptions relied upon in developing its comprehensive plan and test  the feasibility and advisability of any facet of the new appeals process. In  its appeals plan, VA reported its decision to pilot test two of the five new  options by allowing veterans with pending appeals in the legacy process  (known as legacy appeals) to elect the VBA higher-level review or VBA  supplemental claim options beginning in November 2017. This program,  which VA refers to as RAMP, is intended to reduce legacy appeals by  providing veterans with a chance for early resolution of their claims within  VBA while the Board focuses on reducing its inventory of legacy appeals,  according to VA. Participation in RAMP is voluntary, but veterans must  withdraw their pending legacy appeal to participate, according to VA.  Veterans dissatisfied with their RAMP decisions must wait until VA fully  implements the new appeals process (in February 2019 at the earliest)  before pursuing an appeal with the Board under the new process,  according to VA officials.

VA’s Plan Addresses Most of the Act’s Required Elements for the New and Legacy Disability Appeals Processes

VA’s appeals plan addresses 17 of the Act’s 22 required elements,  partially addresses 4 related to monitoring implementation and workforce  planning, and does not address 1 element related to identifying total  resources. For example, VA’s appeals plan addresses the required  elements related to, among others, identifying legal authorities for hiring  and removing employees, estimating timelines for hiring and training  employees, and outlining the outreach VA expects to conduct. For the  elements in the Act that VA’s appeals plan partially addresses or does not  address, see table 1. For a detailed list of the 22 required elements in the  Act, see appendix I.

When we provided VA with our preliminary assessment, VA officials said  they disagreed and that their appeals plan addresses all 22 of the  required elements. In general, they said that data are not available, and  VA cannot yet forecast the information required by the Act until aspects of  the new appeals process are tested or implemented. However, in  discussing our assessment at a January 2018 House Committee on  Veterans’ Affairs hearing, the Deputy Secretary of Veterans Affairs stated  that the agency agreed with our assessment and will work with us to  address these gaps in VA’s appeals plan.

Until VA’s appeals plan has complete information on all 22 of the required  elements, Congress does not have the information it needs to fully  conduct oversight of the plan and the agency’s efforts to implement and  administer the new process while addressing legacy appeals. VA also  needs information on resources, among other areas, to certify that the  agency is prepared to carry out timely processing of appeals under the  new and legacy appeals process. Further, as discussed below,  addressing required elements through a more comprehensive plan and  underlying analysis is consistent with sound planning practices and would  better position VA to implement the new appeals process while attending  to legacy appeals. For example, such an appeals plan would provide for  carefully monitoring the new and legacy appeals processes against  balanced goals and metrics, and clearly articulates resources, milestones  and other information needed for effective program management.

VA’s Appeals Plan Reflects Certain Sound Planning Practices, but Could Improve on Others

VA’s appeals plan reflects certain sound planning practices, such as  convening a working group on performance tracking; however, the plan  could benefit from including important details related to three key planning  areas:  1.  articulating a balanced set of goals and related measures to monitor  and assess the performance of the new appeals process, in  conjunction with the legacy process;  2.  developing a high-quality and reliable implementation schedule to  manage key steps and activities of the project; and  3.  assessing key risks in a comprehensive manner, including respective  mitigation strategies, articulating clear criteria and an assessment  plan for RAMP, and more fully testing or analyzing all appeals options.

VA’s Appeals Plan Indicates Steps to Assess Process Changes, but Should Also Include Goals and Measures to Provide Full Picture of Success

VA’s appeals plan reflects steps taken to track performance, but it could  improve its planning practices related to monitoring and assessing  performance on a range of key dimensions of success. Sound planning  practices suggest that agencies develop overall goals tied to meaningful  and balanced performance measures. These measures include a mix of  outcome, output, and efficiency measures to ensure that an  organization’s priorities—as well as government-wide priorities such as  quality, timeliness, and cost of service—are addressed.

VA’s appeals plan reports that the agency convened a working group to  design a process for tracking timeliness of both the legacy appeals and  appeals within the new process. In supporting documentation that we  requested, VA officials stated they are also determining the best way to  measure veterans’ satisfaction with the new appeals process. VA’s  appeals plan and supporting documentation also identify timeliness goals  for the two VBA-only options and one of the three Board options.  Nevertheless, its appeals plan does not articulate a set of goals and  measures that cover all aspects of its new appeals process, such as  accuracy of decisions and cost. The plan also does not provide details on  the metrics the agency will develop, how it will assess if the new appeals  process is an improvement over the legacy appeals process, and how it  will monitor the allocation of resources between legacy and new appeals  claims. More specifically:

VA’s reported timeliness measures are incomplete: VA’s appeals  plan outlines timeliness goals for the two VBA options (average  processing time of 125 days) and for the Board option that does not  include new evidence or a hearing (average processing time of 365  days). However, VA’s plan does not establish timeliness goals for  the other two Board options: Board review of additional evidence  without a hearing and Board review of additional evidence with a  hearing. In commenting on our findings, while VA officials indicated  they expect the new process to be more efficient than the legacy  process (and, therefore, more timely), they also said data to inform  goal setting for all Board options will not be available until VA fully  implements these options. Establishing timeliness goals for all options  would provide a more complete picture of VA’s vision for the new  appeals process, and help VA to develop concrete, objective, and  observable performance measures to show progress in achieving that  vision, as well as inform resource estimates.

VA’s reported measures lack adequate balance: Other than  including certain timeliness goals, VA’s appeals plan does not  articulate additional aspects of performance important for managing  appeals, such as accuracy of decisions, veteran satisfaction with the  process, or cost. We previously reported that VA officials said they  wanted to also use veteran survey results, wait times, and inventories  as sources of information to measure progress under the new appeals  process. Further, VA’s fiscal year 2018 annual performance plan  includes an overall customer satisfaction score for veterans’  benefits. However, these and other potential measures of success  are not specified in VA’s appeals plan for monitoring the new appeals  process as compared with legacy appeals. By not articulating a set of  comprehensive and balanced goals and measures in its appeals plan,  VA could be inadvertently creating skewed incentives by focusing on  one area of program performance to the detriment of other areas  (e.g., processing claims quickly but inaccurately).

In commenting on our findings, VA officials recognized the need to  develop additional goals and measures. They indicated, for example,  that they are developing and testing whether the existing quality  assurance goal—92 percent accuracy—is appropriate for the new  process. According to VA officials, once they have developed these  other goals and measures, VA will communicate this information as  part of the required progress reports to the appropriate committees of  Congress and GAO. Moreover, at a January 2018 House Committee  on Veterans’ Affairs hearing, the Deputy Secretary of Veterans Affairs  acknowledged that their performance goals and measures are not yet  complete and indicated that the agency will address these gaps in  measuring performance.

VA’s plan does not reflect how it will establish baseline data:  VA’s approach for evaluating the efficiency and effectiveness of the  implementation of the new appeals process falls short of sound  practices for using baseline data to assess performance. Our prior  work has demonstrated that by developing and tracking a  performance baseline for all measures, including those that  demonstrate the effectiveness of a program, agencies can better  evaluate progress made and whether or not goals are being  achieved. However, VA’s appeals plan did not provide important  details about what aspects of the new appeals process’ performance  will be compared to what aspects of the legacy process’ performance.

In particular, section 5 of the Act lists a number of metrics VA is  required to report periodically, including some that could be used as  baseline measures. For example, VA is required to periodically  publish on its website the average time that elapsed between the filing  of an initial claim and the final resolution of the claim, for legacy  appeals as well as appeals under the new system, which is consistent  with our prior recommendation. However, VA’s appeals plan does  not explain how or when the agency would collect and use these or  other data about the legacy and new processes’ performance—such  as accuracy, veteran satisfaction, and cost—to assess their relative  performance.

As we had previously reported, VA’s business case for reform in some  instances relied on unproven assumptions and limited analyses of its  legacy process to identify root causes of performance problems.  Specifically, VA determined that the open-ended nature of its legacy  appeals process, whereby a veteran can submit additional evidence  numerous times at any point, can cause additional cycles of re- adjudication, a process VA refers to as “churning.” According to VA,  this re-adjudication can occur multiple times and can add years to the  time needed to reach a final decision on an appeal. Without fully  articulating a plan for collecting and using baseline and trend data, VA  cannot determine the extent to which the new appeals process, which  also allows for multiple appeal opportunities, will achieve final  resolution of veterans’ appeals sooner, on average, than the legacy  process.

In commenting on our assessment, VA indicated that it is working  toward capturing the metrics listed in section 5 of the Act. VA officials  also noted that reporting on the new appeals process will require IT  system functionality that currently does not exist, but stated that  efforts are underway to add this functionality.

VA’s plan does not explain how the agency will monitor  processing of legacy versus new appeals: In addition, VA’s  appeals plan does not fully articulate how the agency will monitor  whether resources are being appropriately devoted to both the new  and legacy appeals process and how it will track both sets of  workloads. An appeals plan that does not specifically articulate how  VA will manage the two processes in parallel exposes the agency to  risk that veterans with appeals in the legacy process may experience  significant delays or otherwise poor results relative to those in the new  appeals process or vice versa. In commenting on our findings, VA  officials noted that VA was not required under section 3 of the Act to  provide a description of its plans to capture metrics listed in section 5.  Even if not required by the Act, developing an approach for carefully  monitoring the management of new and legacy appeals would help  VA track progress being made and achievement of goals.

Until VA establishes complete and balanced goals and measures,  identifies baseline data, and develops a plan for monitoring and  assessing both the new and legacy processes, VA runs the risk of  promoting skewed behaviors, or not fully understanding whether the new  process is an improvement or whether veterans with appeals in the  legacy process are experiencing poor results.

VA’s Appeals Plan Needs a Reliable Implementation Schedule to Manage the Project

VA’s appeals plan reflects certain aspects of sound planning practices  related to managing the implementation of process change; however,  other key components are not addressed. Sound planning practices for  implementing process change suggest establishing a transition team.  Consistent with such practices, VA’s appeals plan states that the agency  convened an agency-wide governance structure to coordinate  implementation of its new appeals process; it is comprised of senior-level  employees with authority to make necessary decisions to keep the project  on track. VA’s appeals plan also includes a copy of a master schedule. In  its plan, VA asserts that the master schedule reflects timelines, interim  goals and milestones, reporting requirements, and established deadlines,  and that it will be used to guide implementation. VA’s appeals plan also  reports that VA is consulting with project management professionals, who  are using the master schedule, among other tools, to monitor  implementation. In addition, VA made progress addressing some of the  issues we previously identified by developing steps and timetables for  updating training in anticipation of implementing the new appeals  process.

However, VA’s master schedule for implementing reform is missing  elements of a high-quality and reliable implementation schedule for key  activities. We have previously reported that having a well-planned  schedule is a fundamental management tool. Generally recognized sound  practices from the Project Management Institute (PMI) and GAO call for  organizations to employ an integrated and reliable master schedule that  defines when work activities will occur, who will complete the work, how  long they will take, how they are related to one another, and the  constraints affecting the start and completion of work elements, as well as  whether resources will be available when they are needed. Such a  project management schedule not only provides a road map for  systematic project execution, but also provides the means by which to  gauge progress, identify and address potential problems, and promote  accountability.

The master schedule VA provided in its appeals plan should have  included other sound practices for project management related to a  reliable schedule. Specifically:

Key activities and their duration are not included: VA’s master  schedule does not capture specific Board-related activities, such as  efforts to develop metrics, and the schedule and other project plans  we reviewed do not go beyond February 2019. For example, the  schedule does not indicate the period of time when VA expects to no  longer be processing legacy appeals. Also, VA’s master schedule  submitted with its November 2017 appeals plan did not include the  Rapid Appeals Modernization Program (RAMP) activities, even  though this pilot test is occurring at the same time VA is preparing for  full implementation of appeals options at VBA and the Board.  However, VA’s updated schedule that accompanied its comments on  our draft report was updated to include RAMP.

When all key and necessary activities are not included, it raises  questions about whether all activities are scheduled in the correct  order, resources are properly allocated, or the estimated completion  dates are reliable. In addition, if the schedule does not fully and  accurately reflect VA’s efforts, it will not serve as an appropriate basis  for analysis and may result in unreliable completion dates and delays.

Sequencing and linkages among activities are not identified: For  the high-level activities VA’s appeals plan identifies, VA’s master  schedule does not indicate whether there were sequencing or  linkages among them, which is not consistent with sound scheduling  practices. Sequencing and linkages would show, for example, if any of  these activities or sub-activities must finish prior to the start of other  activities, or the amount of time an activity could be delayed before  the delay affects VA’s estimated implementation date. For example,  VA cannot train new employees until after it hires them.

The activities VA identifies also do not appear supported by lower- level project schedules. Specifically, when we requested  documentation to support VA’s high-level summary of activities and  milestones, VA officials did not provide intermediate or more detailed  schedules that reflected these practices. In particular, VA’s appeals  plan lacks a complete schedule for IT modifications that clearly  defines what is to be achieved and the time frames for achievement.

We previously recommended that VA develop a schedule for IT  updates that explicitly addresses when and how process reform will  be integrated into new systems and when these systems will be ready  to support the new appeals process at its onset. For example, VA’s  appeals plan references several required IT modifications that do not  appear in its master schedule. Schedules that are defined at too  high a level may disguise risk that is inherent in lower-level  activities.

Interim goals are not reflected: VA officials stated that they have  interim goals and milestones, though VA’s appeals plan and  supporting documentation generally do not include this information.

Sound planning and redesign practices suggest closely monitoring  implementation and developing project goals that include a mix of  intermediate goals to be met at various stages. We previously made a  recommendation that VA develop a more robust plan for closely  monitoring implementation of process reform, including metrics and  interim goals to help track progress, evaluate efficiency and  effectiveness, and identify trouble spots—all of which are consistent  with sound planning practices. Without interim goals and milestones,  VA lacks information to support sequencing of activities and to track  and ensure accountability for steady progress.

Resources are not assigned to all identified activities: The high- level summary schedule that VA provided us also lacks details  regarding the assignment of resources for all activities. Specifically,  while the plan identifies workgroups responsible for coordinating  elements in the plan, such as regulations, training, and outreach, the  schedule does not assign resources to the 40 listed activities. As  discussed previously, VA’s appeals plan also does not provide  information on the total resources required for this reform effort.  Assigning resources to the listed activities, as well as providing other  information, could provide a better indication of the estimated total  resources required to implement the new appeals process and  address legacy appeals.

In commenting on our findings, VA officials stated that the agency is  developing lower-level project schedules for key activities—such as  RAMP and IT requirements—and will provide these schedules as part of  the required progress reports to the appropriate committees of Congress  and GAO. VA officials also noted that future updates will include  additional dependencies and risks, which VBA and the Board are still  developing. Further, in discussing our findings at a January 2018 House  Committee on Veterans’ Affairs hearing, the Deputy Secretary of  Veterans Affairs reiterated VA’s commitment to developing more robust  project plans, particularly for IT. Until VA has a robust integrated master  schedule, supported by detailed project plans that adhere to sound  practices, VA’s appeals plan does not provide reasonable assurance that  decision makers have the essential program management information  needed for this complex and important effort.

VA’s Plan Addresses Some but Not All Key Risks Related to the New Appeals Process

VA’s appeals plan includes an assessment of risks involved in  implementing the new appeals system, but could more comprehensively  reflect key risks posed by such a significant reform effort. VA’s appeals  plan and supplemental materials include a “risk register” that describes  risks associated with many elements of its plan, and the remaining level  of risk after its planned response to these risks. VA’s appeals plan also  states that senior leaders will receive regular updates of risks and  mitigation strategies. However, because VA has not yet articulated a  balanced set of performance goals and measures in its appeals plan, it is  hindered in its ability to identify and assess risks.

Federal internal control standards state, and our previous work at VA and  other agencies demonstrates, that establishing clear performance goals  and objectives is a necessary pre-condition to effectively assessing risk.  Having, for example, more complete timeliness goals, and goals and  measures reflecting other areas of performance, would allow VA to better  identify and target risks associated with managing two processes in  parallel, including the potential that veterans with appeals in the legacy  process may experience significant delays relative to those in the new  appeals process.

Importantly, VA is missing an opportunity to fully benefit from RAMP by  not testing and assessing other aspects of the new appeals process. The  Act authorizes VA to test the feasibility and advisability of any facet of the  new appeals process, and VA is taking a positive step to mitigate some  risks by testing the two review options available within VBA (review of a  claim by a higher-level official based on the same evidence and review of  a supplemental claim with additional evidence) through RAMP. In  November 2017, VA began RAMP by inviting 500 veterans whose  appeals have been pending the longest to participate. According to VA  officials, VA plans to continue offering RAMP to additional eligible  veterans with pending legacy appeals each month until January 2019—a  month before VA anticipates fully implementing the new appeals system.  However, as designed, RAMP does not include features—consistent with  a well-developed and documented pilot test program—that would provide  VA with an opportunity to evaluate fully the soundness of new processes  and practices on a smaller scale. Specifically:

VA’s plan does not clearly define success criteria for RAMP: VA’s  appeals plan states that the agency will collect certain data from  RAMP, such as the rate at which eligible veterans opt into the  process, timeliness of claims processing, and individual employee  productivity. VA also established an overall average processing time  goal of 125 days for the two VBA options; however, the plan and  supporting documentation did not clearly articulate whether RAMP  reviews are expected to meet this timeliness goal. The plan also did  not identify other success criteria for RAMP or the types of results  expected before fully implementing the new appeals process. For  example, VA’s plan does not articulate the expected number and type  of subsequent appeals to the Board that result from RAMP.

In commenting on our findings, VA noted that its intent in  implementing RAMP was to collect data and test aspects of the new  process, and that RAMP was not an initiative in and of itself. However,  developing performance measures and data gathering procedures  and defining success criteria for a pilot test before proceeding to full  implementation are sound practices for process redesign and pilot  testing. In addition, because RAMP was not included in VA’s risk  assessment, we asked VA if it had identified any risks or mitigation  strategies specific to RAMP. In its supplemental materials, VA stated  that the greatest risk to RAMP is a low participation rate among  eligible veterans with legacy claims. VA also indicated that it would  need 10 percent of eligible veterans to opt into RAMP to yield  meaningful results. However, this threshold is not articulated in VA’s  appeals plan as an explicit success criterion or objective. According to  data provided by VA, as of January 22, 2018, 238 veterans opted in.

Of veterans with pending claims in RAMP, two-thirds chose the  higher-level review option. VA also reported that 47 RAMP decisions  have been made so far. As of yet, no appeals of RAMP decisions  have been filed.

VA’s plan does not articulate how it will assess RAMP before  proceeding with full implementation: Although VA’s appeals plan  describes a “close-out” phase in which VA intends to assess the  results of RAMP, it does not detail the conditions that would have to  be met (or not met) to trigger changes. For example, VA’s plan does  not explain when or how it might respond to low opt-in rates for  RAMP—other than stating it will increase outreach to eligible  veterans—or to unexpectedly high appeal rates to the Board resulting  from RAMP decisions. Sound redesign and change management  practices both suggest that pilot tests be rigorously monitored and  evaluated, and that further roll-out occur only after an agency’s  transition team takes any needed corrective action and determines  that the new process is achieving previously identified success  criteria. Without fully articulating its plan for deciding how and when to  roll out changes more broadly, it is not clear whether VA would be  prepared to fully implement a new appeals process that achieves its  aim of better serving veterans.

RAMP does not test all aspects of the new appeals process:  RAMP provides an opportunity to learn about experiences at VBA  under the new system, such as the rate at which eligible veterans  choose those options and the resources that will be required to  process their appeals. However, RAMP was not designed to test how  many veterans would choose to appeal directly to the Board and,  therefore, it will not provide comparable information on the Board  appeals options. Sound workforce planning practices suggest that  agencies identify the total resources needed to manage the risk of  implementing new processes and conduct scenario planning to  determine those needs.

In addition, although we previously recommended VA conduct additional  sensitivity analyses to inform projections of future appeals inventories,  VA’s appeals plan does not reflect VA’s use or intended use of sensitivity  analyses when projecting staffing needs for new appeals options at the  Board. In commenting on our findings, VA officials said they do not plan  to conduct additional sensitivity analyses to project future workloads until  they have more information from RAMP to inform their assumptions. As a  result, VA will lack data on scenarios in which veterans may  overwhelmingly choose options available at the Board over those at VBA  when the appeals plan is fully implemented. This presents a risk that VA’s  early production projections and initial resource allocations may not be  properly balanced between the Board and VBA. This, in turn, may result  in an unexpectedly large number of appeals pending with the Board, and  corresponding lengthy average wait and decision times for some, if not  all, Board options.

Having information on the number of veterans who are likely to appeal to  the Board is particularly critical, given that similar efforts to create  additional review options at VBA did not achieve their goals of reducing  the percentage of appeals that continue on to the Board. In 2001, VA  established the Decision Review Officer (DRO) process—in which senior  staff have the authority to overturn an initial disability claim decision  without any new evidence—to resolve more appeals at the regional level  and avoid long waits at the Board. However, we reported in 2011 that,  although the DRO process helped some veterans get additional benefits  at the regional office level, it did not accomplish the program’s primary  goal of reducing the percentage of appeals continuing on to the Board.

In responding to our findings, VA officials reiterated their plans to increase  outreach in the event of low opt-in rates for RAMP and indicated they  recently began to send follow-up RAMP invitation letters. With respect to  assessing all appeals options, VA officials stated that, while no legal bar  prevents testing of the Board options, the Board is focused on reducing  its inventory of pending appeals while RAMP provides early resolution of  appeals within the new VBA-only options. Officials conceded that this  approach means they cannot collect data on the rate at which veterans  opt to appeal directly to the Board (e.g., bypassing additional VBA review)  until the new process is fully implemented. However, they noted that they  can collect some data on the rate at which veterans whose appeals go  through RAMP file subsequent appeals to the Board, even though the  Board will not begin processing those appeals until full implementation. At  a January 2018 House Committee on Veterans’ Affairs hearing, the  Deputy Secretary of Veterans Affairs stated that given our assessment,  VA will adjust its approach to identify and mitigate risks associated with  implementing all of the appeals options, including those at the Board.

Until VA pursues an approach that identifies and mitigates significant  risks associated with implementing a new process, VA is taking a chance  that untested aspects will not perform as desired. The Act provides VA  authority to pilot aspects of the process and flexibility on the timing of  implementing the new process, which could allow some additional time  for VA to carefully measure performance under RAMP and determine  whether any corrective actions are necessary. If VA does not take full  advantage of this authority, it risks moving forward without knowing  whether the new appeals process improves experiences for veterans, and  potentially implementing a process that is more expensive or results in  longer wait times than originally anticipated.

In implementing appeals reform after the enactment of the Veterans  Appeals Improvement and Modernization Act of 2017, VA is undertaking  a complex endeavor that has the potential to affect the lives of hundreds  of thousands of veterans with service-connected disabilities. Such an  endeavor demands a commensurate level of planning to be successful.  While the Act required VA to submit its plan within 90 days of enactment,  VA had proposed and began to plan for appeals reform much earlier, and  had our March 2017 recommendations to guide its planning efforts from a  foundation of sound practices.

VA’s November 2017 appeals plan is a positive step forward. Certain  elements of the plan—such as establishing an agency-wide governance  structure to oversee implementation and testing aspects of reform prior to  full implementation—are notable gains since our March 2017 report. At  the same time, the plan partially addresses or does not address five of  the required elements called for by the Act, such as delineating the total  resources required by VBA and the Board to implement and administer  the new appeals process and address legacy appeals. The plan also is  not fully responsive to our past recommendations and does not reflect a  number of sound planning practices that are essential for gauging  progress, establishing accountability, and linking resources to results.

One such key practice is articulating a desired “end state”—a vision for  what successful implementation would look like for the new appeals  process as well as the wind-down of the legacy process, such as  accurate and timely processing of appeals while ensuring veteran  satisfaction. Without establishing a complete and balanced set of goals  and related performance measures to achieve this end state, and  monitoring and assessing progress along the way, VA risks falling short of  its overarching objective—to improve timeliness of appeals decisions for  veterans overall. By not fully articulating how it plans to monitor workloads  and devote resources to both the new and legacy processes, VA runs the  risk of disadvantaging veterans with legacy appeals relative to those in  the new process, or vice versa.

Just as important is establishing a robust integrated master schedule— rather than a high-level timeline—that is built upon and clearly reflects  extensive detailed planning and includes all activities necessary to  execute the program and interdependencies between these activities.  Without such a road map, VA’s appeals plan does not provide reasonable  assurance that decision makers have the essential information needed to  manage this complex and important program.

We are encouraged that VA has taken some steps toward assessing  risks, including establishing a risk register and implementing RAMP to  collect information on the two VBA appeals options. However, unless VA  assesses risks against a balanced set of goals and measures, VA may  not be fully aware of risks that may impede successful implementation of  appeals reform. Further, although VA will undoubtedly learn from the  RAMP experience, it may not learn all that it should from its efforts  without (1) establishing clear criteria for what success looks like (or the  circumstances that would cause VA to consider making course  corrections), and (2) building in time to take stock of the lessons learned  before moving to full implementation.

VA’s plan places a lot of weight on RAMP to, among other efforts,  mitigate risk and generate estimates of the resources needed for  successful implementation after fiscal year 2018, even though RAMP  does not fully test options for appealing to the Board that will be available  to veterans after full implementation. Unless VA addresses key risks  associated with fully implementing appeals reform—by either testing or  conducting sensitivity analyses for all five appeals options, to better  understand potential workloads at the Board—VA runs the risk of fully  implementing the process without knowing if it is improving the process  for veterans.

We are making the following four recommendations to VA:  The Secretary of Veterans Affairs should address all of the required  elements in the Act in VA’s appeals plan to Congress—including  delineating resources required for all VBA and Board appeals options— using sensitivity analyses and RAMP results, where appropriate and  needed. (Recommendation 1)

The Secretary of Veterans Affairs should clearly articulate in VA’s appeals  plan how VA will monitor and assess the new appeals process compared  to the legacy process, including specifying a balanced set of goals and  measures—such as timeliness goals for all VBA appeals options and  Board dockets, and measures of accuracy, veteran satisfaction, and  cost—and related baseline data. (Recommendation 2)

The Secretary of Veterans Affairs should augment the master schedule  for VA’s appeals plan to reflect all activities—such as modifications to IT  systems—as well as assigned responsibilities, interdependencies, start  and end dates for key activities for each workgroup, and resources, to  establish accountability and reduce overall risk of implementation failures.  (Recommendation 3)

The Secretary of Veterans Affairs should ensure that the appeals plan  more fully addresses risk associated with appeals reform—for example,  by assessing risks against a balanced set of goals and measures,  articulating success criteria and an assessment plan for RAMP, and  testing or conducting sensitivity analyses of all appeals options—prior to  fully implementing the new appeals process. (Recommendation 4)

We provided a draft of this report to the Department of Veterans Affairs  (VA) for comment. In its comments, reproduced in appendix II, VA  outlined its planned actions to address the recommendations. VA clarified  in a subsequent communication that the agency agreed with all our  recommendations. VA did not provide technical comments.

With respect to our first recommendation that VA address all the required  elements in the Act in VA’s appeals plan (including delineating total  resources), VA stated that both VBA and the Board will use existing  resources to implement the new appeals process. VA also stated it plans  to take additional steps to determine resource requirements for  addressing workloads under both the legacy and new appeals process.  For VBA, VA stated that it will continue to rely on RAMP to project  resource requirements, while acknowledging the need to augment its  analysis of RAMP data by adopting additional strategies to project  resource requirements. VA did not describe these strategies, but stated  that it will share them with Congress and GAO in the near future.  Meanwhile, VA noted that its 2019 budget request includes 605 additional  FTEs for VBA to process appeals, but did not indicate how it developed  this budget request. For the Board, VA stated that it plans to develop  better predictions regarding resource allocations among dockets by  leveraging project management and other support within the agency. We  will continue to monitor VA’s efforts to delineate needed resources,  including how it uses the results of pilot tests and prediction analysis.

With respect to our second recommendation to articulate how it will  monitor and assess the new appeals process compared to the legacy  process, VA stated that it is working to develop a complete and balanced  set of measures for the new appeals process, and timeliness goals for all  appeals options. Further, VA indicated it will track performance for Board  appeal options using an existing process. We are encouraged by VA’s  proposed actions, which will provide a more complete picture of VA’s  vision for the new appeals process. However, VA does not detail whether  or how the agency will develop a baseline or compare performance of the  new appeals process to the legacy process. Until VA develops a baseline  and a plan for monitoring and assessing both the new and legacy  processes—using a complete and balanced set of goals and measures— VA risks not fully understanding whether the new process is an  improvement.

With respect to our third recommendation to augment its master  implementation schedule to manage the project, VA provided an updated  schedule with additional key activities and responsibilities, such as  RAMP. Moreover, VA restated its plans to use an agency-wide  governance structure to coordinate and track implementation of its new  appeals process. We are encouraged by VA’s efforts to develop a more  detailed implementation schedule. However, the updated schedule VA  provided does not include major activities, such as integrated IT system  testing, and completion dates for major activities, for example, adding  functionality to VA’s primary claims processing system. In addition, VA  provided an updated calendar for six major IT activities through the end of  calendar year 2018. However, we continue to believe that VA will need to  develop a longer term schedule that projects when processes will be  integrated into new systems and when new systems will be ready to  support the new appeals process. The schedule also does not indicate  whether activities are interrelated, such that a delay in one activity could  affect other activities and thereby affect VA’s estimated implementation  date. This sound planning practice is especially important because VA  stated the agency is concurrently executing many of the activities.

With respect of our fourth recommendation to more fully assess risks  associated with appeals reform prior to its full implementation, VA stated  that it will assess risks against a balanced set of goals it plans to select.  Moreover, using its existing risk management process, VA stated it has  identified additional risks and mitigation strategies after submitting its  November 2017 plan. For example, VA states that it is addressing the  continued low opt-in rate for RAMP, which is testing the new VBA-only  options. VA is also acknowledging that delays in the development of IT  required to implement the appeals process may prevent the agency from  certifying readiness in January 2019. Importantly, VA states that the  Board is exploring the development of a pilot program to identify needs  and concerns related to full implementation—including all Board appeals  options—and to make predictions about timeliness and productivity under  the new appeals process. However, VA did not define success criteria for  its current pilot test, RAMP, or clearly articulate how the agency will  assess results of either RAMP or a new test of Board appeals options  before proceeding to full implementation. Implementing our  recommendation in a complete and timely manner is important because it  would improve VA’s ability to identify and mitigate significant risks  associated with implementing a new process.

We will continue to monitor the status of VA’s actions to address our  recommendations and how they are implemented, to help ensure that VA  is undertaking a level of planning appropriate to implementing a complex  endeavor, and thereby improving VA’s chance of success.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, and other interested  parties. In addition, the report is available at no charge on GAO’s website  at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  Elizabeth Curda at (202) 512-7215 or curdae@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who contributed to this  report are listed in appendix III.

Appendix I: Our Assessment of VA’s Appeals Plan Against Required Elements in the Act

To assess the extent to which VA’s appeals plan addresses the required  elements in the Veterans Appeals Improvement and Modernization Act of  2017 (Act), we first identified and developed a checklist reflecting each  required element for VA’s appeals plan (including sub-parts) under  section 3(a) and (b) of the Act. To compare the required elements and  their sub-parts against VA’s appeals plan and supplemental materials  provided by VA, we developed decision rules for determining whether  VA’s appeals plan addressed, partially addressed, or did not address  each required element (see table 2). Specifically, we concluded that VA’s  plan addressed (or partially addressed) a required element if the plan  included information related to all (or some) subparts of the requirement.  We focused on the plan as presented, rather than auditing the information  VA relied on in developing the plan. For example, the Act’s section  3(b)(10) required VA’s plan to include a description of the modifications to  the information technology (IT) systems required to carry out the new  appeals system, including cost estimates and a timeline. We concluded  that VA’s plan addressed all sub-parts of this element because it provided  a description of required IT modifications, a reference to costs included in  the Appeals Modernization IT budget, and a timeline. However, our  determination that VA addressed this element should not be construed to  necessarily mean that VA fully identified or described all IT requirements,  or provided complete estimated costs and timelines associated with those  requirements, or that the information in VA’s appeals plan comported with  sound planning practices. This type of assessment was outside the scope  of this objective.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

Elizabeth H. Curda, (202) 512-7215 or curdae@gao.gov.

In addition to the contact above, the following staff members made  significant contributions to this report: Michele Grgich (Assistant Director),  James Whitcomb (Analyst in Charge), and Rachael Chamberlin.

In addition, key support was provided by Susan Aschoff, Mark Bird, David  Chrisinger, Daniel Concepcion, Clifton Douglas, Alex Galuten, Nisha  Hazra, Melissa Jaynes, Benjamin Licht, Patricia McClure, Sheila McCoy,  Lorin Obler, Gloria Proa, Almeta Spencer, James Sweetman, Walter  Vance, and Greg Whitney.