Social Security Disability: Additional Measures and Evaluation Needed to Enhance Accuracy and Consistency of Hearings Decisions

Individuals who do not agree with the initial decision on a claim for Social Security disability benefits can ultimately appeal the decision by requesting a hearing before one of SSA's approximately 1,500 administrative law judges. However, the rate at which these judges have allowed benefits has varied, raising questions about the reasons for this variation. GAO was asked to review aspects of SSA's oversight of judges' decisions.

This report examines (1) to what extent allowance rates vary across administrative law judges, and factors associated with this variation; and (2) the extent to which SSA has processes to monitor the accuracy and consistency of hearings decisions.

GAO developed a statistical model to analyze SSA data on adult disability decisions made by administrative law judges from fiscal years 2007 through 2015, the most current data available at the time of GAO's analysis; reviewed relevant federal laws, regulations, and agency documents; and interviewed SSA officials and chief judges in SSA's 10 regions, as well as officials from organizations representing judges, disability claimants, and claimant representatives.

Allowance rates—the rate at which Social Security Administration (SSA) administrative law judges allowed disability benefits to be paid when claimants appealed—varied across judges, even after holding constant certain characteristics of claimants, judges, hearing offices, and other factors that could otherwise explain differences in allowance rates. Specifically, GAO estimated that the allowance rate could vary by as much as 46 percentage points if different judges heard a typical claim (one that was average in all other factors GAO analyzed). SSA officials said that this level of variation is not surprising, given the complexity of appeals and judicial discretion. Nonetheless, the variation declined by 5 percentage points between fiscal years 2007 and 2015 (see figure), a change officials attributed to enhanced quality assurance efforts and training for judges. GAO also identified various factors that were associated with a greater chance that a claimant would be allowed benefits. In addition to characteristics related to disability criteria, such as the claimant's impairment and age, GAO found that claimants who had representatives, such as an attorney or family member, were allowed benefits at a rate nearly 3 times higher than those without representatives. Other factors did not appear related to allowance rates, such as the percentage of backlogged claims in a hearing office.

SSA has various reviews to monitor the accuracy and consistency of hearings decisions by administrative law judges, but some of these reviews may overlap and SSA has not systematically evaluated them. Specifically, SSA conducts five types of quality assurance reviews of hearings decisions, several of which have similar goals and may look at similar claims. SSA has not evaluated the efficiency or effectiveness of these reviews, despite spending at least $11 million on them in fiscal year 2016. Moreover, the agency has struggled to sustain all of its quality reviews due to competing priorities—two of the five reviews were curtailed in 2016 because SSA reassigned staff to help expedite claims decisions. By evaluating which quality assurance reviews are most effective and efficient in improving accuracy and consistency, SSA would be better positioned to meet its goals within its resources.

GAO is making two recommendations, including that SSA systematically evaluate its quality assurance reviews and take steps to reduce or better manage any unnecessary overlap among them. SSA concurred and plans to address them through a comprehensive assessment of its oversight.

The Social Security Administration (SSA) manages two disability benefit  programs—Disability Insurance (DI) and Supplemental Security Income  (SSI)—that together provide about 16 million Americans with about $200  billion in benefits annually. Overall, more than 6 percent of the U.S.  working-age population receives disability benefits from one or both of  these programs, although the rate varies by state (ranging from about 4  percent to more than 12 percent), according to SSA. A claimant who is  dissatisfied with the initial decision on his or her application can ultimately  appeal at a hearing, where an administrative law judge (ALJ) reviews the  case and any new evidence submitted by the claimant. About 30 percent  of the claimants who are ultimately allowed benefits are granted benefits  at the hearings level or beyond.

For more than 15 years, the number of people applying for disability  benefits has generally increased, along with the number of appeals. At  the same time, the percentage of appealed claims that have been allowed  benefits has declined. However, these rates—known as allowance  rates—can vary widely across judges and hearings offices, raising  questions about the reasons for this variation.

Moreover, reports of improper decisions made by certain judges and a  scheme between a judge and an attorney in one state to commit fraud,  have raised questions about SSA’s oversight of ALJs. You asked us to  examine allowance rates and SSA’s oversight of ALJs. This report  examines (1) the extent to which allowance rates vary across  administrative law judges, and any factors that are associated with this  variation, and (2) the extent to which SSA has processes to monitor the  accuracy and consistency of hearings decisions.

To address these objectives, we conducted a statistical analysis of data  on adult disability decisions made by administrative law judges from fiscal  years 2007 through 2015 and the factors that may be associated with  variation in allowance rates. Specifically, we considered the following  factors:

Claimant characteristics relevant to the disability determination  process (e.g., age and primary impairment),

Participants in the process other than claimants (e.g., claimant  representatives and medical and vocational experts),  Judge characteristics (e.g., year of appointment as a judge, and any  prior related SSA experience),

SSA administrative characteristics (e.g., hearing office where the case  was decided and whether the hearing was conducted by  videoconference), and

Economic characteristics (unemployment and poverty rates in the  claimant’s state).  (Appendix I provides more detail on our statistical analysis, including a  complete list of the factors we analyzed.)

We assessed the reliability of SSA’s administrative data on disability  decisions and judge characteristics by (1) performing electronic testing of  required data elements, (2) reviewing existing information about the data  and the system that produced them, and (3) interviewing agency officials  knowledgeable about the data. We determined that the data were  sufficiently reliable for the purposes of this report. (Appendix I describes  our data in more detail.)

We reviewed relevant federal laws, regulations, and documentation and  collected testimonial evidence from SSA officials to describe and evaluate  the processes SSA uses to monitor hearings decisions, detect variation,  and improve accuracy. We assessed these monitoring efforts against  federal internal control standards and our management and evaluation  guide for assessing fragmentation, overlap, and duplication in  government programs. We also reviewed SSA’s annual performance  plans from fiscal years 2006 through 2017 to describe the performance  measures the agency has established to improve the accuracy and  consistency of its hearings decisions. We evaluated the current  performance measures using key attributes of performance measures  identified in prior GAO work and federal internal control standards.

We conducted this performance audit from December 2015 to December  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

The Social Security Administration’s (SSA) Disability Insurance (DI) and  Supplemental Security Income (SSI) programs are the two largest federal  programs providing cash assistance to people with disabilities.

The DI program, established in 1956, provides monthly payments to  working-age adults (and their dependents or survivors) who are  unable to work due to a long-term disability.

The SSI program, established in 1972, is a means-tested income  assistance program that provides monthly payments to adults or  children who are aged, blind, or have other disabilities and whose  income and assets fall below a certain level.

Individuals with low incomes and assets who also have a sufficient  work history may qualify for the DI and SSI programs concurrently. In  this case, the individual’s SSI payment is generally offset by the  amount of the DI payment.

In fiscal year 2016, according to SSA, about 10.8 million disabled workers  and their family members received about $143 billion in DI benefits, and  an estimated 8.2 million individuals received almost $59 billion in SSI  benefits (of those, 2.6 million received SSI in addition to DI or Old-Age  and Survivors benefits).

Although DI and SSI have different purposes and target populations, the  disability criteria for adults are the same for both programs. To be  considered eligible for either program as an adult, a person must have a  medically determinable physical or mental impairment that (1) has lasted  or is expected to last for at least a continuous period of 1 year or result in  death, and (2) prevents them from engaging in any substantial gainful  activity (SGA).

The disability decision-making process includes five sequential steps (see  fig. 1). First, SSA determines if a claimant is working and screens out  (denies) claimants who earn over a specified amount. Second, SSA  determines whether the claimant has an impairment severe enough to  significantly limit his or her ability to do basic work activities and expected  to last more than 12 months or result in death, and denies claimants who  do not meet these criteria. At the third step, SSA determines whether a  claimant’s impairment meets or is equivalent to an impairment listed in  SSA’s Listings of Impairments. If a claimant “meets” or “equals” one of the  listed impairments, they are allowed benefits. If not, SSA proceeds to the  last two steps and assesses whether a claimant, given their impairment,  can do their past work (step four) or other work that exists in significant  numbers in the national economy (step five).

Over time, more of SSA’s disability decisions have been made at the last  two steps in the process, which require more judicial discretion than  decisions made at steps 1 through 3, according to SSA. In 2000, 29  percent of decisions were made at steps 4 and 5, according to an SSA  report. By 2014, nearly half—49 percent—of all decisions were made at  these steps.

Disability Application and Appeals Process

To apply for benefits, a claimant must file an application online, by  telephone, or mail, or in person at a local Social Security office. If field  office staff determine that the claimant meets the nonmedical eligibility  criteria, they forward the claim to the appropriate state Disability  Determination Services (DDS) office. DDS staff—generally a team  comprised of disability examiners and medical consultants—review  medical and other evidence provided by the claimant, obtaining additional  evidence as needed, and make the initial disability determination. In fiscal  year 2016, SSA received more than 2.5 million disability claims.

If the claimant is not satisfied with this determination, in most states he or  she may request a reconsideration of the decision within the same DDS  office. If the claimant is dissatisfied with the reconsideration, he or she  may request a hearing before an administrative law judge (ALJ). In one of  several initiatives to improve the disability determination process, SSA  has eliminated the reconsideration step of the process in 10 states,  allowing the claimant to appeal the initial decision directly to an ALJ. In  fiscal year 2016, claimants appealed more than 698,000 decisions to the  hearings level, and SSA issued more than 637,000 dispositions (including  allowances, denials, and dismissals). (See fig. 2).

Within SSA’s Office of Disability Adjudication and Review (ODAR), there  are approximately 1,500 ALJs who are located in 166 hearing offices  across the country, as well as at five National Hearing Centers. In  general, cases are randomly assigned to ALJs within the area each  hearing office serves, in the order in which the requests for a hearing are  received. The ALJ reviews the claimant’s file, including any additional  evidence the claimant submitted after the initial determination, and  generally conducts a hearing. At the hearing, the ALJ may hear testimony  from the claimant, medical experts on the claimant’s medical condition,  and vocational experts regarding the claimant’s past work and jobs  currently available in significant numbers in the national economy. The  majority of claimants are represented at these hearings by an attorney or  nonattorney representative, such as a professional disability  representative, relative, or social worker.

If the claimant is not satisfied with the ALJ decision, he or she may  request a review by SSA’s Appeals Council, which is the final  administrative appeal within SSA. The Appeals Council may grant,  deny, or dismiss a request for review. If it agrees to review the case, the  Appeals Council may uphold, modify, or reverse the ALJ’s decision, or it  may remand the case back to the ALJ to hold another hearing and issue  a new decision. In fiscal year 2016, the Appeals Council reviewed more  than 154,000 ALJ decisions and remanded 13 percent of them.

Hearings Backlogs and Processing Times in Recent Years

Hearings-level backlogs and processing times have increased between  fiscal years 2010 and 2016. The number of annual requests for a  hearing before an ALJ peaked in fiscal 2011, and declined in each  subsequent year, through fiscal year 2016. Despite this decline, SSA has  not been able to keep pace with the demand, in terms of dispositions— the number of cases the agency decided or dismissed—in each of those  years after 2010 (see figure 3). By the end of fiscal year 2016, SSA  reported there were about 1.1 million pending cases. Average processing  times for hearings-level decisions also increased during this same time  period, from 426 days to 543 days.

During these years, the number of ALJs declined, along with the number  of case dispositions per month. For example, SSA reported it employed  1,356 ALJs in fiscal year 2013, and these judges had an average of 48  case dispositions per month. In fiscal year 2015, 1,265 judges had an  average of 44 case dispositions per month. Also during this time period,  SSA reduced its reliance on senior attorney adjudicators (SAA) to make  fully-favorable, on-the-record decisions (that is, decisions in which a  hearing is not necessary because the documentary evidence alone  supports a decision that is fully favorable to the claimant).

According to SSA, its backlog will be eliminated when the national  average processing time for a hearing decision is 270 days. In January  2016, SSA issued a plan to achieve this goal by the end of fiscal year  2020. However, in its fiscal year 2018 performance plan, SSA set a goal  for processing hearings decisions in 600 days (up from a target of 485  days in fiscal year 2010). SSA reported that the increase in average  processing times is due to the increase in the number of pending cases.  Since SSA generally processes cases in the order in which they are  received, they focus on the oldest cases first, which increases the  average processing time for closed cases.

Requirements for Hiring, Overseeing, and Disciplining SSA Administrative Law Judges

The role of ALJ was created by the Administrative Procedure Act, which  was enacted in 1946 to ensure fairness and due process in federal  agency proceedings involving rulemaking and adjudications. ALJs serve  in a number of executive branch agencies, although SSA employs the  vast majority. ALJs preside and make decisions at formal adjudicatory  proceedings.

One of the primary goals behind the creation of the ALJ position is to  ensure that judges can conduct hearings free from influence or coercion  from the agency. Although ALJs are hired by and serve as employees of  executive branch agencies like SSA, the Office of Personnel  Management (OPM) is responsible for the initial examination, certification  for selection, and implementation of the three levels of basic pay of ALJs.  As part of its responsibilities, OPM sets the minimum qualifications for  ALJs, which are that they generally must be licensed attorneys with a  minimum of 7 years of experience in litigation and/or administrative law  and pass the competitive examination.

The Administrative Procedure Act gave ALJs qualified decisional  independence, with some oversight from agencies. Decisional  independence means that ALJs can make decisions independently.  Federal law also excludes ALJs from performance evaluations and  generally requires that disciplinary actions against ALJs be for good  cause established and determined by the Merit Systems Protection Board  (MSPB). While ALJs have qualified decisional independence, they must  follow their agency’s policies and procedures when making decisions.  The Administrative Procedure Act also authorized agencies to review ALJ  decisions. If SSA determines that an ALJ has not followed its policies and  procedures, it can issue a directive to the ALJ to comply and, if that is  unsuccessful, bring a disciplinary action before the MSPB.

Allowance Rates Vary Across Judges, Even for Typical Claims

Allowance Rates Have Varied Across Judges and Hearing Offices in Recent Years, Even After Holding Constant a Range of Factors Relevant to the Appeals Process

Allowance rates varied across administrative law judges from fiscal years  2007 through 2015. We defined the “allowance rate” for each judge as the  number of claims in which a judge granted the claimant Disability  Insurance (DI) and/or Supplemental Security Income (SSI) benefits  divided by the total number of decisions issued by the judge (excluding  claims that were dismissed). We analyzed about 3.3 million decisions  made by administrative law judges on adult Social Security disability  appeals over this period.

The average allowance rate across judges fell 15 percentage points over  this period—from a peak of 70 percent in 2008 to 55 percent in 2015—but  the range in allowance rates across judges remained fairly constant (see  fig. 4). Specifically, the range—the difference between judges with high  allowance rates (those at the 95th percentile) and judges with low  allowance rates (at the 5th percentile)—was 55 percentage points over  this period.

This variation in allowance rates persisted, but fell modestly over time,  even when we used multivariate statistical methods to hold constant a  variety of factors related to the disability appeals process. These factors  included characteristics of claimants, judges, and hearing offices, as well  as other factors such as the unemployment rates in a claimant’s state,  that could otherwise explain differences in allowance rates. Specifically,  for the years 2007 through 2015 combined, our analysis estimated that  the allowance rate would vary by 46 percentage points for a typical  claim, depending on the judge who heard the case. For example, we  estimated that the allowance rate for a typical claim heard by a judge with  low allowance rates would be 42 percent, compared to 88 percent for a  judge with high allowance rates. This estimated range fell from 50  percentage points in 2007 to 45 percentage points in 2015 (see fig. 5).  (Appendix I describes this statistical analysis in more detail.)

Allowance rates also varied across hearing offices during the same time  period, but this variation was considerably smaller than the variation  across judges in every year. The estimated range across the entire period  was 19 percentage points across hearing offices (see fig. 6), compared  to a 46 percentage-point estimated range across judges. Accounting for  differences in allowance rates across offices ensured that the variation  across judges did not reflect characteristics of their offices (such as the  types or severity of disability claims received by their offices).

SSA officials noted that the variation in allowance rates we observed  across judges was not surprising, nor was the modest narrowing in this  range over time. Administrative law judges usually hear complex appeals  that may not be clear-cut allowances or denials. As a result, according to  SSA officials, given judges’ decisional independence, different judges  could look at cases with similar fact patterns and circumstances and  come to different conclusions. At the same time, officials also pointed to  several factors potentially related to the modest narrowing in the range of  allowance rates. First, they noted that SSA started conducting quality  assurance reviews of a random sample of allowances in 2011— previously, such cases were not reviewed. In addition, they said that  Social Security’s disability programs and administrative law judges were  under increased public and Congressional scrutiny following a high-profile  fraud case in 2011 involving a judge and an attorney representative.  Further, officials said that the expanding use of electronic case files and  data analytics within SSA made it possible for the agency to enhance  monitoring of decision-making and share this information with judges.  Finally, while SSA cannot direct judges to decide cases in a particular  way, officials suggested that some judges may have “self-corrected” their  approach to decision-making, given all of these factors.

Our multivariate analyses had some limitations, but it provides more  information than simple comparisons in allowance rates across judges.  For example, the SSA data we used for this analysis do not include a  measure of the severity of a claimant’s impairment or their remaining  ability to work, which could help explain why one claim with a particular  impairment was allowed while another was denied. The data also do not  include a standardized measure for the nature of claimants’ prior work  (such as the skill level or extent of physical labor), which is also relevant  for the disability decision. Nevertheless, our multivariate analysis enabled  us to compare allowance rates across judges and hearing offices for  typical claims. In addition, SSA’s practice of assigning cases randomly  to judges makes it more likely that the remaining variation we found  across judges reflects the unique effect of having a particular judge hear  a case, rather than other factors. As a result, even though we could not  account for all factors that could explain differences in allowance rates,  random assignment increases the chances that such factors were similar  across all of the cases heard by individual judges.

Numerous Factors, Particularly Those Representing SSA’s Disability Criteria, Are Associated with Variation in Allowance Rates

Although variation in allowance rates persisted across judges, even after  controlling for certain factors, many of the factors we identified had  meaningful associations with the chance that a claimant was allowed  benefits. These factors represent criteria in SSA’s disability decision- making process, such as the claimant’s age, impairment, prior work, and  education. We also identified factors that did not have such  associations.

Certain claimant characteristics—such as older ages or certain  impairments—were associated with higher allowance rates.

Age: Claimants’ chances of being allowed benefits increased with  age, even holding constant other factors. For example, a 55-year-old  claimant was allowed benefits at a rate 4.3 times higher than a typical  35-year-old claimant. This association is consistent with Social  Security’s vocational guidelines, which are generally more lenient for  older claimants. As part of SSA’s five-step process to determine  eligibility for adult disability benefits, SSA uses a set of rules to  evaluate how a claimant’s age, education, and work experience affect  their remaining capacity for work. SSA’s criteria vary across four  primary age groups—45-49, 50-54, 55-59, and 60 and older. The  criteria are less stringent for claimants in older age groups than they  are for younger claimants, because the rules assume that individuals  at older ages may be less able to transition to other work.

Impairment: Certain impairments were also strongly associated with  the chance of being allowed benefits (see fig. 7). For example,  claimants with primary impairments recorded in SSA’s data of heart  failure or multiple sclerosis were allowed benefits at rates 4.2 and 5  times higher, respectively, than typical claimants with asthma. From  fiscal years 2007 through 2015, the allowance rates for claimants with  heart failure or multiple sclerosis were 78 and 80 percent,  respectively, compared to 44 percent for asthma.

Critical or terminal case: Claimants with critical or terminal cases  were allowed benefits at a rate 1.4 times higher than a typical  claimant without a critical or terminal case. Critical and terminal cases  are cases that require special processing, such as a terminal illness or  a veteran with a 100-percent permanent and total disability  compensation rating.

Prior work: Claimants reporting shorter work histories (4 years or  less in the last 15 years before applying for disability benefits) were  allowed at a rate 0.8 times as high as a typical claimant with 10 or  more years of work history. As expected, given the nature of the work  requirements for the DI program, the association with prior work  history was stronger for that program than for the SSI program.

College education: Claimants who reported having a college-level  education or higher were approved at a slightly higher rate (1.1 times  higher) than a typical claimant with a high-school education. SSA  officials suggested that this association could be an indirect measure  of the severity of a claimant’s impairment, a factor for which we did  not have data. They said that individuals with higher levels of  education often have higher incomes and, therefore, may be less  likely to forego their income to apply for disability benefits, were it not  for the severity of their disability.

Claim type: DI claimants were allowed at a rate 1.7 times higher than  a typical SSI claimant. Across judges, the average allowance rate for  DI claimants (67 percent) was higher than for SSI claimants (52  percent) from fiscal years 2007 through 2015, with the allowance rate  for claimants applying concurrently for DI and SSI benefits falling in  between (58 percent).

Other Participants in the Disability Appeals Process  Claimants who had appointed a representative to present their case, or  had a medical expert testify at their hearing, were associated with a  greater chance of being allowed benefits, but the presence of a vocational  expert had the opposite association.

Claimant representative: Similar to findings in our prior work,  claimants who had a representative—either an attorney or a  nonattorney representative—were allowed at a rate 2.9 times higher  than a typical claimant with no representative. SSA officials stated  that representatives may have a screening process for potential  clients, and under SSA’s fee structure, representatives are paid only if  the claimant is awarded benefits. As a result, representatives may  tend to take cases they believe will be successful. Officials also stated  that a representative can help the claimant by ensuring that the  medical evidence and other records are fully developed and help the  claimant present their case at a hearing. From fiscal years 2007  through 2015, most claimants (77 percent) had an attorney  representative, and 12 percent had a nonattorney representative.

Expert testimony: Claimants whose hearings involved testimony  from a medical expert were allowed at a rate 1.6 times higher than a  typical claimant without a medical expert present. Medical experts  include physicians, psychologists, and other types of medical  professionals who provide impartial, expert opinion evidence for an  ALJ to consider when making a decision about disability. SSA officials  said that the association of medical experts with an increased chance  of allowance is expected, given that judges are required to seek the  testimony of a medical expert in certain cases, for example, when the  judge is considering allowing benefits because the claimant’s  impairment may be medically equivalent to one in SSA’s Listing of  Impairments. In other cases, involving a medical expert is generally at  the judge’s discretion. From fiscal years 2007 through 2015, 12  percent of decisions involved a medical expert.

The presence of a vocational expert had the opposite effect— claimants with a vocational expert testifying were allowed at a rate 0.8  times as high as claimants without a vocational expert testifying.  Vocational experts provide objective, expert opinion evidence to the  ALJ, primarily at the last two steps of the disability decision-making  process where SSA considers whether claimants can do their prior  work or transition to other work available in the national economy.  Although involving a vocational expert is generally at a judge’s  discretion, SSA officials said that they were not surprised by this  result, because vocational experts are usually called upon at the final  two steps in the disability decision-making process. At that point,  claimants had already not been allowed benefits at an earlier step  because their impairment(s) did not meet or were not equivalent to an  impairment in SSA’s listings. From fiscal years 2007 through 2015,  most hearings (85 percent) involved a vocational expert.

Judges with certain characteristics, such as those appointed in earlier  years, were associated with a greater chance of allowing benefits.

Appointment cohort: A claimant whose claim was heard by a judge  appointed between 1995 and 1999 was allowed at a rate 1.5 times  higher than a typical claim heard by a judge appointed after 2010.  SSA officials said that, since 2010, they have changed the way they  train and mentor new judges, and introduced new tools to help  provide a standardized decision-making template. As a result, SSA  officials said, more recently hired ALJs may be more aware of agency  policies and procedures.

Certain characteristics of hearing offices and other factors also were  associated with higher chances of allowance. For example:

Hearing type: Claimants whose hearings were held in person were  allowed at a slightly higher rate (1.1 times higher) than a typical  claimant with a hearing conducted remotely using videoconference  technology. This is equivalent to a 2.8 percentage-point higher  probability of being allowed benefits for a claimant whose hearing was  held in person, compared to an otherwise typical claimant whose  hearing was conducted by videoconference. However, we did not  seek to estimate the causal impact of videoconferences on allowance  rates, and so did not design our analysis to account for all factors that  could affect this relationship. Rather, we accounted for the use of  videoconferences solely to further ensure that circumstances were  similar across the judges and offices we analyzed. Expanding video  service delivery is a key goal for SSA, including plans to partner with  other agencies, such as the Department of Veterans Affairs, to  increase the number of available video hearing sites beyond those  already available at hearing offices and the five National Hearing  Centers.

Year of decision: Claimants whose appeals were decided in earlier  years were associated with a greater chance of being allowed  benefits. While this trend is similar to the raw change over time shown  in figure 4, our multivariate analysis showed that this change held  even for claimants in similar circumstances. For example, claimants  who received decisions in 2007 were allowed at a rate 2.0 times  higher than a typical claim in 2015. This is consistent with other  studies that have found trends of lower allowance rates in recent  years.

Factors Not Associated with Differences in Allowance Rates  Some factors were not meaningfully associated with allowance rates  when holding other factors constant.

Workload measures: Workload and productivity measures at the  hearing office and judge level were not meaningfully associated with  allowance rates. This includes the annual percentage of cases that  were backlogged (that is, awaiting a judge’s decision for more than  270 days) at each hearing office, as well as the annual number of  dispositions (decisions plus dismissals) each judge issued. This may  suggest that judges’ decisions to allow or deny cases are not  significantly influenced by the number of cases before them, similar to  findings in prior research.

Hearing office type: We found no meaningful differences in  allowance rates between similar claims heard at one of SSA’s  National Hearing Centers or a traditional hearing office, after holding  constant other factors (including whether the hearing was held by  videoconference). SSA has five National Hearing Centers, which hear  cases from across the country by videoconference in order to reduce  backlogs in certain hearing offices.

Economic characteristics: The unemployment and poverty rates in  the claimant’s state at the time of the ALJ decision were not  associated with allowance rates. Higher unemployment rates can  result in increased applications for Social Security disability benefits  because workers with impairments that could qualify them for the  program who experience job loss may find it more difficult to become  re-employed during periods of high unemployment and apply for  benefits. However, the impact on allowance rates in the research we  reviewed is mixed.

SSA’s Efforts to Monitor Accuracy and Consistency of Hearings-Level Decisions Lack Performance Measures and Have Not Been Evaluated

SSA Has Timeliness Measures, but Lacks Public Performance Measures for Accuracy and Consistency

SSA has employed a range of efforts to monitor the accuracy and  consistency of hearings decisions, but it lacks performance measures to  report publicly on these efforts. SSA’s current strategic plan includes an  objective to “improve the quality, consistency, and timeliness” of its  disability decisions; however, all of the hearings-level measures  supporting this objective are related to timeliness. In a previous report, we  developed nine attributes of performance goals and measures based on  previously established GAO criteria, as well as relevant federal laws and  performance management literature. One key attribute states that an  agency’s suite of performance measures should be balanced to cover  various priorities. In addition, each measure should cover a priority such  as quality, timeliness, and cost of service. However, because SSA’s  performance measures do not fully reflect its goals, the overall success of  SSA’s efforts in this area may be limited.

SSA previously had performance measures related to hearings-level  accuracy, which used data from ALJ peer reviews. These measures were  discontinued in fiscal year 2009, when the ALJs conducting the reviews  were reassigned to hearing cases. By comparison, SSA continues to  have a measure for accuracy at the initial decision-making level (see  table 1).

SSA officials stated that they have no plans to add new performance  measures related to the accuracy and consistency of hearings decisions  to the strategic plan. They said that while they collect and monitor a wide  variety of workload and performance measures for day-to-day operations,  they have to select a few, representative measures that are meaningful to  stakeholders and represent agency-wide efforts to achieve its goals. They  stated that the current performance measures meet these requirements.

Although SSA officials said the agency does not publicly report  performance measures related to the accuracy and consistency of  hearings decisions, they said that SSA uses internal performance  measures related to hearings decisions. However, these internal  measures to monitor quality and consistency of hearings decisions have  limitations and are not shared with the public. Regional chief judges—who  oversee the hearing offices and judges within each of SSA’s 10 regions— and others told us that they use a measure known as the “agree rate” to  help monitor the quality of a judge’s decisions. This measure is based on  the number of cases that have been appealed to the Appeals Council by  the claimant or representative as a request for review. The agree rate  reflects the percentage of cases in which the Appeals Council—the final  level of appeals within SSA—concluded that the ALJ’s decisions were  supported by substantial evidence and contained no error of law or abuse  of discretion. However, the agree rate has some limitations. For example,  as noted earlier, it does not reflect the accuracy of ALJ decisions that the  claimant did not appeal. SSA’s Office of the Inspector General (OIG)  found that this measure provided information on less than one-quarter of  all ALJ dispositions and it is not representative of the ALJ’s entire  workload because it is based only on Appeals Council reviews of  appealed cases. In addition, a March 2017 SSA OIG report found that  SSA has not maintained historical data on agree rates, limiting the  agency’s ability to analyze agree rate trends.

SSA uses other internal measures to track consistency. For example,  SSA developed an internal early monitoring system that tracks 22 metrics  of ALJ performance to identify outliers. For example, three of these  metrics (average number of dispositions a judge issues per day, agree  rate, and allowance rate) have “alarm thresholds” to indicate when an  ALJ’s metrics fall outside of a given threshold. Based on these findings,  SSA may conduct a focused quality review (a type of quality assurance  review) to ensure the judge’s decisions complied with SSA policies, or  follow up with the regional chief judge to determine if additional policy  guidance or training is needed.

Although these internal measures are helpful for management to monitor  and improve accuracy and consistency, without sharing this or similar  information publicly, SSA lacks accountability for improving the quality of  hearings-level decisions. In addition, federal internal control standards  state that management should externally communicate the necessary  quality information to achieve objectives, including to external  stakeholders such as Congress and the public. Further, given the  persistent variation in allowance rates, SSA may be missing an  opportunity to provide the public with information on the results of its  efforts to improve the accuracy and consistency of disability decisions.

SSA Has Efforts to Monitor and Improve Accuracy and Consistency of Hearings- Level Decisions, but Quality Reviews May Overlap and Have Not Been Systematically Evaluated

SSA provides training and tools to all ALJs and initiates disciplinary  actions where needed, as part of its efforts to monitor and improve  accuracy and consistency. SSA also conducts multiple quality assurance  reviews, but some of these reviews may overlap and SSA has not  evaluated them.

Training, Tools, and Policy Guidance  ALJs receive ongoing training and guidance from several sources,  including through judicial trainings, mentoring, and policy memorandums.  In 2006, SSA implemented a three-phase training program for new ALJs,  which includes training on core competencies as well as a formal  mentoring program in which new ALJs are paired with experienced ALJs  for regular sessions over a nine-month period. Regional managers,  judges, and stakeholders we spoke with had positive feedback on the  training SSA provides to judges. For example, officials from one  stakeholder group told us that they believe training had created more  consistency in allowance rates. SSA’s chief judge also issues guidance  memorandums to clarify policies related to the hearings process. For  example, in July 2013, SSA issued a memorandum establishing  expectations for the instructions judges provide to decision writers, who  are SSA staff who prepare the draft decisions. SSA officials said that they  issued the memorandum in response to an ALJ who was providing low- quality instructions to decision writers and SSA realized it had not  provided formal guidance on the topic. In addition, ALJs also receive  quarterly continuing education training and have a library of reference  materials and on-demand video courses to use as needed.

SSA also uses internal metrics and provides electronic tools to judges to  monitor and improve accuracy and consistency. Regional chief judges  regularly review management information (MI) reports and develop  strategies, such as recommending training, to address identified issues.  Beginning in 2011, SSA established an electronic tool called “How MI  Doing?”, which allows ALJs to compare their productivity and timeliness  metrics to hearing office, regional, or national metrics. The tool also  provides data on the agree rate for each judge as well as the hearing  office, regional, and national agree rates. Using this tool, judges can also  learn the reasons any prior decisions have been remanded, and access  on-demand training pertaining to that reason. Regional chief judges we  spoke with generally found “How MI Doing?” to be a helpful tool, although  SSA does not track judges’ usage and has not formally evaluated its  effectiveness. In addition, SSA established the electronic Bench Book  (eBB), which is designed to assist users with documenting, analyzing,  and making consistent and accurate decisions on hearings-level adult  disability cases. However, the SSA OIG recently recommended that SSA  evaluate eBB and determine whether to continue it. Regional chief judges  we spoke with provided mixed feedback on the use of eBB and its  usefulness for ALJs. In fiscal year 2016, nearly 500 ALJs (about one- third) used eBB. In June 2017, SSA officials said that while no formal  evaluation of eBB was conducted, they recently received approval to  proceed with plans to replace eBB with a similar tool as part of updates to  SSA’s case management system.

SSA also addresses identified issues with the accuracy and consistency  of hearings decisions by taking disciplinary actions, as needed. SSA can  take non-disciplinary or disciplinary action to address performance  concerns. Non-disciplinary actions include training and counseling (known  as “collegial conversations”). Another non-disciplinary action is a written  directive, which SSA can issue to individual judges to improve  performance on workload, scheduling or policy compliance. From 2007  through 2016, SSA issued about 1,330 such directives. Nearly all (95  percent) were issued to improve timeliness, while about 2 percent were  issued to improve policy compliance. If an ALJ’s conduct or performance  does not change or becomes more egregious, SSA continues with  progressive discipline including reprimand or seeking disciplinary action  from the Merit Systems Protection Board, such as short- or long-term  suspension or removal. From 2007 through 2016, there were 98  reprimands, 34 proposed suspensions, and 16 proposed removals,  according to SSA.

SSA conducts various quality assurance reviews to improve accuracy and  consistency. SSA officials stated that the agency has been enhancing its  quality review efforts since 2009. Since then, it has added five types of  quality assurance reviews that are conducted by three additional offices  within SSA (see fig. 8).

SSA added quality assurance reviews for various reasons. For example,  in 2009, SSA’s regional staff under the Office of the Chief Administrative  Law Judge began conducting regional inline quality reviews, which  involve assessing the extent to which hearing office staff are processing  cases and preparing them for hearings in accordance with SSA policy, as  well as the policy compliance and legal sufficiency of the draft decision.  SSA added this review to enhance its reviews of decisions before they  are issued, in an effort to reduce remands. Also in 2009, SSA’s Office of  Quality Review began conducting disability case reviews to provide  feedback on decision-making accuracy to ALJs. In addition, in 2010, SSA  created the Division of Quality under the Appeals Council, a unit focused  on conducting reviews on a regular basis of decisions that claimants did  not appeal. Prior to 2010, SSA generally only reviewed decisions that  claimants appealed through the Appeals Council.

While these quality assurance reviews have somewhat different  focuses—for example, some assess aspects of how a case was  processed while others review the accuracy of the decisions—they  overlap in two key ways. According to prior GAO work, overlap occurs  when multiple agencies or programs have similar goals, engage in similar  activities or strategies to achieve them, or target similar beneficiaries.  Some of SSA’s quality review efforts fit the description of overlap in that  they have similar goals and review similar cases. For example:

Similar goals: Several of the reviews have similar goals (see table 2).  For example, two of the four entities conducting reviews—the Appeals  Council’s Division of Quality and staff in SSA’s 10 regional offices— both review decisions for policy compliance before those decisions go  into effect (known as pre-effectuation reviews). While one review  looks at the judge’s decision and the other looks at the draft decision  prior to the judge’s review and approval, according to officials and  documents we reviewed, these reviews share similar goals: to guide  training and provide feedback to judges. In addition, all the reviews  are designed to assess compliance with SSA policy.

Similar cases: SSA’s five quality assurance reviews look at similar  cases, and could potentially include the same cases (see table 3).  SSA takes some steps to prevent assessing the same claim in  multiple quality assurance reviews. Officials told us that, in conducting  focused quality reviews (conducted after the decision is final), they  exclude cases that were reviewed in a pre-effectuation review.  However, they said that the Division of Quality does not know whether  cases it has selected were also subject to a regional inline quality  review. They said that additional efforts to prevent multiple reviews of  a case are manual in nature, and thus there is still the potential for  claims to be reviewed more than once. Further, SSA officials said they  did not see a need to prevent multiple reviews of a case, in particular,  because some reviews are conducted before the decision is final and  others are conducted after the decision is final.

SSA officials stated that opportunities exist to improve coordination  across offices conducting quality assurance reviews. We found that  several offices coordinated their work in some cases. For example, SSA’s  Division of Quality and Office of Quality Review participate in a multi- office workgroup that addresses such issues as policy compliance across  the initial and hearings levels of the disability process. In addition, they  have also worked together on several studies, including a one-time  quality review of 454 claims that were denied at the initial determination  level, but were allowed as fully favorable at the hearings level. The Office  of Quality Review also reviews the content of selected training for judges.  In addition, the Division of Quality provided some initial input when the  regional inline review effort was being designed. Prior GAO work has  found that enhanced coordination can help to reduce overlap and improve  efficiency. Effective October 1, 2017, SSA created a new deputy  commissioner-level component, the Office of Analytics, Review and  Oversight. This agency reorganization moved six oversight offices into the  new component, including the Division of Quality and Office of Quality  Review. Officials said the new component will create opportunities for  improved coordination between these six offices. While this  reorganization creates the opportunity for SSA to assess many of its  quality assurance reviews, the regional quality review staff will not be  included in the new office, and it is too early to tell how this reorganization  will help manage the overlap between SSA’s various quality assurance  reviews.

In addition, SSA has struggled to sustain all of its quality assurance  reviews due to competing demands for the staff who perform them. For  example, SSA placed regional inline quality reviews on hold in September  2016 and again in December 2016, because officials said that the agency  needed staff to complete pending decisions before a change in the  medical listings for mental impairments took effect in January 2017.  Decisions not completed before the new listings took effect would have to  be redone. Also, the Office of Quality Review curtailed its Disability Case  Reviews in fiscal year 2016 to help prepare the oldest cases for hearings.  As a result, only the Appeals Council’s review of appealed ALJ decisions  (requests for review) and the Division of Quality’s quality assurance  reviews were active in 2016.

Even as SSA has added quality assurance reviews, it has not  systematically evaluated the efficiency and effectiveness of all the  reviews to determine the extent to which they may be overlapping or  complementary. We found that reviews conducted by the four entities  have resulted in similar findings, raising questions about the efficiency of  these reviews. For example, during the same 3-year period (fiscal years  2013 through 2015), quality reviews conducted by all four entities found  problems with judges’ assessment of a claimant’s ability to perform work- related tasks, known as a residual functional capacity assessment. In  addition, all four entities found problems with the evaluation of medical  opinion evidence.

Moreover, SSA has not conducted a cost-benefit analysis of the five  reviews. Officials said that there are no definite plans to do so, although  they may consider conducting such an analysis in the future. We found  that costs for the quality assurance reviews conducted in fiscal year 2015  were at least $23.7 million, and in fiscal year 2016 were at least $11.7  million (see table 4). By evaluating the quality assurance reviews to  determine the extent to which each is needed to monitor and improve  accuracy and consistency, SSA would be better positioned to meet its  goals within its resources.

In addition, SSA continues to develop and implement initiatives aimed at  improving hearing decisions, without evaluating the potential for overlap  with existing quality assurance reviews. For example, as part of its  backlog reduction plan known as the Compassionate And Responsive  Services (CARES) plan, SSA is using computer algorithms for natural  language processing to analyze the text of disability decisions and flag  potential errors. Although the agency is piloting this effort in the Appeals  Council before expanding it to hearing offices, it did not conduct a cost- benefit analysis. SSA officials said that natural language processing could  be used to identify cases for further review, similar to its current selective  reviews, and that decision writers could use the tool to conduct their own  reviews of their draft decisions. SSA officials said that they do not  anticipate much overlap between the use of natural language processing  and OAO’s pre-effectuation reviews. However, there could be potential for  overlap with regional inline reviews, which also review decisions drafted  by decision writers. Federal internal control standards state that  management should implement control activities through policies.  Periodically reviewing policies, procedures, and related control activities  for continued relevance and effectiveness in achieving objectives and  addressing related risks can help agencies meet this standard.

SSA’s disability programs provide more than $200 billion in benefits for  tens of millions of Americans annually, making it one of the largest  components of the nation’s social safety net. The hearings and appeals  level of the disability decision-making process is particularly important  because about one in three people receiving Social Security disability  benefits are granted benefits at this level. Given the number of people  and the dollars at stake, it is crucial that claimants are treated fairly and  their applications are evaluated accurately and consistently across the  country, at all levels of the program.

Some of the variation in allowance rates that we found across judges may  be expected, given the complexity of the cases and judges’ decisional  independence. However, the persistent variation we observed over time,  even after accounting for various factors that could otherwise explain  allowance rates, might warrant additional attention. SSA is rightly  focusing on oversight of judges, but our work suggests that the agency’s  emphasis on timeliness over accuracy in its public metrics and the  potential overlap in its quality assurance efforts may offer opportunities for  improving the accuracy and consistency of hearing decisions.

First, this amount of variation in allowance rates underscores the need for  SSA to measure and hold itself accountable for accuracy and  consistency. However, without sharing performance information on the  accuracy and consistency of its hearings-level decisions, such as the rate  at which the Appeals Council agrees with a judge’s decisions, SSA may  not be providing the public with adequate information on progress toward  its objective to improve the quality, consistency, and timeliness of its  disability decisions. Developing a set of performance measures that  includes the accuracy and consistency of hearings decisions will help  ensure the overall success of the program.

Second, SSA has not systematically considered how each of its quality  assurance reviews helps the agency meet its objective to improve the  quality of hearings-level decisions. Although the planned consolidation of  multiple oversight and quality review offices is a positive step, it will be  important for SSA to consider the usefulness of the information yielded by  each quality assurance effort, as well as the costs associated with  conducting the effort. Evaluating the efficiency and effectiveness of  quality assurance activities can help ensure that SSA is using its  resources for maximum benefit toward its objective to improve the quality,  consistency, and timeliness of its disability decisions.

We are making the following two recommendations to SSA:  The Commissioner of SSA should develop a set of public performance  measures, to include accuracy and consistency, as well as timeliness, of  administrative law judges’ (ALJ) disability decisions. SSA could consider  whether existing quality review or monitoring efforts could provide suitable  data for such measures. (Recommendation 1)

The Commissioner of SSA should systematically evaluate the efficiency  and effectiveness of its quality assurance reviews and take steps to  reduce or better manage any unnecessary overlap among them to ensure  strategic use of resources. Such steps could include enhancing  collaboration where reviews overlap or only conducting the reviews that  are most efficient and effective in achieving agency goals for improving  accuracy and consistency of ALJ disability decisions. (Recommendation  2)

In commenting on a draft of this report, SSA agreed with our two  recommendations to (1) establish public performance measures for the  accuracy and consistency of administrative law judges’ decisions, and (2)  systematically evaluate its various quality assurance reviews and take  steps to reduce or better manage any unnecessary overlap among them.  SSA stated that it would address both recommendations as part of a  comprehensive assessment and refinement of its oversight roles and  processes. SSA made several other comments about one of our  conclusions and our analysis of variation in administrative law judge  allowance rates, which we discuss below. SSA also provided technical  comments, which we incorporated into the report as appropriate.

In its comments, SSA described its evolving oversight activities at the  hearings level, including providing policy guidance and training for judges,  capturing and utilizing data to gain a better understanding of trends and  challenges, and implementing additional oversight review processes, all  of which we discussed in our report. SSA’s comments acknowledged that  our report describes the steps that the agency has taken to improve  oversight, but disagreed with our conclusion that SSA emphasizes  timeliness over accuracy. Our final report clarifies that we came to this  conclusion based on a review of the performance measures the agency  shares with the public in its annual strategic plan and performance  reports. As we state in the report, SSA has employed a range of efforts to  monitor the accuracy and consistency of hearings decisions, but it lacks  performance measures to report publicly on these efforts.

Regarding our analysis of variation in ALJ allowance rates, SSA raised a  concern about our finding (on page 26 of the final report) that claimants  whose hearings were held in person were slightly more likely (by about  2.8 percentage points) to be allowed benefits than a typical claimant with  a hearing held by videoconference. SSA cited its own internal analysis,  which found a small (0.6 percentage-point) difference in allowance rates  between in-person and videoconference hearings after controlling for a  number of factors. It is not surprising, however, that our estimates are  somewhat different, since SSA’s internal analysis differs from ours in  several ways. The primary purpose of our statistical analysis was to  isolate variation in allowance rates due to the unique judge or hearing  office assigned to each claim. To do this, we developed a multilevel  model using 9 years of data that controls for judge, hearing office, and  claimant-level factors associated with allowance rates. On the other hand,  SSA’s analysis was specifically designed to look at the difference in  allowance rates between in-person and video hearings. SSA’s analysis  also covered a shorter, more recent period of time (part of fiscal year  2015, fiscal year 2016, and part of fiscal year 2017), than our study (fiscal  years 2007 through 2015). Additionally, the version of the model SSA  cited in its comments included hearings held in person or by  videoconference only in regular hearing offices, whereas our analysis  included hearings held in National Hearing Centers as well as regular  hearing offices and controlled for the type of hearing office. These  differences notwithstanding, we agree with SSA that the estimated model- adjusted difference in allowance rates between in-person and  videoconference hearings in both GAO’s and SSA’s analyses could  potentially be explained by unmeasurable factors.

In addition, SSA noted that our measure of variation in judge decisions  focused on allowance rates at the extremes of the distribution. Given that  our charge was to explore the extent of variation in allowance rates  across judges, we believe it was appropriate and important to measure  the range of allowance rates between judges with high allowance rates  (at the 95th percentile) and those with low allowance rates (at the 5th   percentile). This would be more conservative than an approach that looks  at allowance rates across all judges, including potential extreme values;  and more nuanced than an approach that looks at the number of judges  whose allowance rates are higher or lower than a given threshold.  Further, our analysis shows that unadjusted allowance rates at the 95th  percentile declined over the period of our analysis, from a high of 96  percent in fiscal year 2008 to 82 percent in fiscal year 2015. We saw a  comparable decline in allowance rates after applying our multivariate  model. To provide additional context, our report figures also show the  middle of the distribution (the 25th and 75th percentiles), as well as the  average allowance rates. We have also added information to our report  further describing this middle range.

Finally, SSA noted that our analysis was not weighted by the number of  determinations a judge made, suggesting that judges who decided very  few claims, for example, could affect the range in allowance rates or the  trends. As we show in Appendix I, Table 7, only 2.3 percent of the judges  in our study population heard fewer than 250 claims per year. This group  of judges had an unadjusted allowance rate of 61.9 percent, very similar  to the allowance rate among judges who heard 500-699 claims per year  (61.6 percent). Furthermore, the statistical methods we used to estimate  the distributions of allowance rates (multilevel models) adjust the  estimates for judges with fewer claims by weighting them more heavily  toward the overall approval rate. This mitigates against judges with  smaller caseloads, and therefore higher sampling variation, from  contributing overestimated allowance rates that might have inflated our  estimated variation across judges.

As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Commissioner of Social Security, and  other interested parties. In addition, the report will be available at no  charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

Our objectives were to assess (1) the extent to which allowance rates  vary across administrative law judges, and any factors that are associated  with this variation, and (2) the extent to which the Social Security  Administration (SSA) has processes to monitor the accuracy and  consistency of hearings decisions.

To answer these objectives, we reviewed SSA policies and procedures  related to administrative law judge (ALJ) disability hearings and decisions;  manuals and documents describing SSA’s case processing systems for  each level of SSA’s disability decision-making process, and guidance  and training provided to judges for making disability decisions. We  interviewed SSA officials in several offices within the Office of Disability  Adjudication and Review (ODAR), including the Office of the Chief  Administrative Law Judge and the Office of Appellate Operations, as well  as conducted semi-structured interviews with Regional Chief  Administrative Law Judges in each of SSA’s 10 regions. We also  observed administrative law judge hearings in one of SSA’s five National  Hearing Centers, in Falls Church, Virginia, (hearings in these offices are  conducted by videoconference), as well as in two of SSA’s regular  hearings offices, in Washington, D.C., and Seattle, Washington. The  purpose of these observations was to gain a better understanding of the  hearings process in practice, and to inform our scope and methodology  for this study. We selected these sites, which are not generalizable to the  population of all hearing offices, for a number of reasons, primarily: (1) to  observe hearing offices in different geographic locations and observe  both in-person and video teleconference hearings, and (2) to select sites  at which a cross-section of cases with different types of disabilities and  impairments were available. We attended hearings involving both adult  and child claimants with a mix of physical and mental impairments.

This appendix is divided into three parts. The first describes our data  sources and analysis of allowance rates across judges and associated  factors, the second describes our multivariate statistical model, and the  third describes our work related to our second research objective on  SSA’s processes to monitor the accuracy and consistency of hearings  decisions.

Analysis of Variation in Allowance Rates across Judges and Associated Factors

For this objective, we analyzed data from two primary sources from fiscal  years 2007 through 2015: SSA’s administrative data systems for the initial  and hearings levels of the disability decision-making process, and the  agency’s personnel data system. We also obtained other SSA  administrative data on staffing levels and numbers of pending cases in  each hearing office. Finally, we obtained data on state poverty and  unemployment rates from the U.S. Census Bureau and the Bureau of  Labor Statistics, respectively.

To analyze information on all adult disability decisions made by  administrative law judges from fiscal years 2007 through 2015—the most  current data available at the time of our analysis—we compiled claims  data from several SSA administrative data systems. These data  contained information on the outcomes of the disability decisions and the  characteristics of claims associated with each decision. Specifically, the  information was drawn from the following systems:

831 File and Structured Data Repository: The 831 File pertains to  the initial and reconsideration level of the disability determination  process, within the state Disability Determination Services (DDS).  Data on claimant characteristics we used from this system include the  date of the claimant’s initial application for benefits and the claimant’s  self-reported years of education. We also received a limited set of  data captured from the claimant’s disability application in SSA’s  electronic case folder system (Structured Data Repository), including  the number of years a claimant reported being employed out of the 15  years before becoming disabled.

Case Processing and Management System (CPMS): This system  pertains to the hearings level and was our primary source of  information on hearing outcomes, claim, and claimant characteristics.  Specifically, this system provided information on claim type (i.e.,  Disability Insurance, DI; Supplemental Security Income, SSI; or  concurrent claim); the outcome of the claim (i.e., dismissed, allowed,  or denied) and the date the decision was made; the unique  identification number of the administrative law judge (ALJ) who made  the decision; whether a medical expert or vocational expert attended  the hearing; whether the claimant was represented; the hearing office  where the claim was decided and the type of hearing office (i.e.,  hearing office or National Hearing Center); the claimant’s date of birth;  the primary impairment at the time of the hearing level; the presence  of a secondary impairment; and whether the case was classified as  being a critical case—that is, a case requiring special processing,  such as a terminal illness.

We used case identifiers to link the information from each of these  databases that pertained to each disability decision we analyzed.

We obtained data from SSA’s Federal Personnel and Payroll System  (FPPS) database on all administrative law judges who were employed by  the agency at any time during the period from January 1, 2005, through  December 31, 2015. We obtained information on each judge, such as  their date of appointment as an ALJ and the type of appointment (regular  career appointment or non-permanent); service computation date; and  prior position titles within SSA, if any.

We obtained summary-level data, as of January 2017, from SSA on  staffing levels (numbers of ALJs, decision writers, and other support staff)  at each hearing office for each fiscal year in our study period (fiscal years  2007 through 2015) from SSA’s Payroll Operational Data Store system.  We also obtained data on the numbers of cases left pending at the end of  each fiscal year (including the number of cases pending for more than  270 days). SSA provided those data from a management information  report that uses CPMS data.

We used publically available estimates of state poverty rates for each  year in our analysis (calendar years 2007 through 2015) from the U.S.

Census Bureau’s American Community Survey (ACS). We considered  using estimates at the county level but that approach had limitations.  First, we would have been limited to using 3-year or 5-year estimates for  all counties, because 1-year ACS estimates are only available for areas  with populations of 65,000 or more. Second, the Census Bureau cautions  against using estimates for particular time periods that do not align with  the periods of its estimates. Although using state-level estimates reduced  the geographic precision of the estimates, we gained precision by having  annual estimates and the ability to measure potential variation in poverty  rates over narrower time intervals.

We also used publically available estimates of state unemployment rates  in calendar years 2007 through 2015 from the Bureau of Labor Statistics’  Local Area Unemployment Statistics data. This variable allowed us to  control for labor market conditions over time.

SSA constructed custom files for GAO from several SSA datasets in  response to our data requests. We assessed the reliability of the data  used in our analyses through electronic testing, analyzing related  database documentation, examining the SAS code used by SSA to  construct the custom files, and working with agency officials to reconcile  discrepancies between the data and documentation that we received. We  determined that the 831, Structured Data Repository, and CPMS data on  ALJ decisions and claimant characteristics and the FPPS data on ALJ  appointments were sufficiently reliable for the purposes of describing the  extent of variation in the outcomes of ALJ decisions. We also determined  that SSA’s data on pending caseloads and ALJ and decision writer  staffing, by year and hearing office, were sufficiently reliable for the  purpose of describing hearing office characteristics. Finally, we  determined that ACS data on state poverty rates and BLS data on state  unemployment rates were sufficiently reliable for the purposes of  describing these state economic characteristics.

Our analyses of ALJ decisions excluded various types of decisions from  the CPMS data because they were out of scope for our research  objectives (e.g., child cases, non-disability cases, or cases that were  decided by SSA staff who were not ALJs) or were not typically randomly  assigned to judges. We selected cases that should have been assigned  randomly to judges, according to SSA policy, because that random  assignment made it more likely that variation in allowance rates across  judges in our multivariate analysis reflects the unique causal effect of  having a particular judge hear a case, rather than other factors that also  vary across judges. Our exclusion criteria were similar to those used by  an internal SSA study of ALJ allowance rates, conducted in 2017. We  excluded cases that were:

Dismissed. Cases can be dismissed for reasons not related to the  merits of the case and that are usually beyond the ALJ’s control—for  example, the claimant’s failure to file a timely request or to appear at  the scheduled hearing (without good cause), or the claimant’s death  before the hearing. In addition, data on key factors for these cases,  such as the claimant’s impairment, were missing. From fiscal years  2007 through 2015, 1,007,526 claims (16 percent of all claims) were  dismissed.

Made “on the record” and not randomly assigned to judges.  While most appeals are decided after an ALJ hearing, ALJs and  senior attorney adjudicators (SAA) have the authority to issue on-the- record decisions. These are decisions where a hearing is not  necessary because the documentary evidence alone supported a fully  favorable decision. SSA has created screening criteria, such as the  claimant’s age (50 and older) and specific impairments, to help  identify possible on-the-record decisions earlier in the process. ALJs  and SAAs can also issue on-the-record decisions for cases involving  critical need, and claimants and their representatives can request that  the ALJ or SAA issue an on-the-record decision. These cases are not  assigned randomly to judges. From fiscal years 2007 through 2015,  716,574 claims (11 percent of all claims) were on-the-record  decisions, although SSA has issued fewer on-the-record decisions in  more recent years.

Issued for children. We excluded claimants younger than 18 at the  date of the initial application. We also excluded claimants with missing  or invalid age values. From fiscal years 2007 through 2015, 492,158  claims (8 percent of all claims) were for people under 18 or with  missing or invalid age values. We excluded child cases from our  analysis because they involve different evaluation criteria.

Remanded back to a judge from SSA’s Appeals Council (or  federal court). These cases represent decisions that were corrected  after an order from the Appeals Council or a federal court after the  original ALJ’s decision. In these cases, judges are often addressing a  narrow set of issues identified in the remand order. Remanded cases  are also not assigned randomly to judges, since the Appeals Council  generally sends them back to the judge who originally issued the  decision. However, SSA’s Office of the Inspector General (OIG) in  2017 found that about half of the remanded cases in its sample were  assigned to a different ALJ than the original ALJ. From fiscal years  2007 through 2015, 293,971 claims (less than 5 percent of all claims)  were remands.

Made by senior attorney adjudicators who were not  administrative law judges. We excluded decisions made by SAAs.  SSA implemented a program in 2007 whereby SAAs located in  hearing offices across the country could issue fully favorable on-the- record decisions. According to SSA, this allowed ALJs to focus on  cases that are more complex or require a hearing. From fiscal years  2007 through 2015, 227,133 claims (4 percent of all claims) were  decided by SAAs.

Appeals of continuing disability reviews (CDR). These cases  represent decisions about whether or not to continue benefits for  claimants who were previously found eligible for the program. As  such, they involve different evaluation criteria. From fiscal years 2007  through 2015, 245,862 claims (4 percent of all claims) were appeals  of CDRs.

Non-disability cases. These cases include Social Security retirement  and survivor benefit decisions. We excluded such cases because they  involve different evaluation criteria from disability claims and represent  a small minority of decisions at the hearings level. From fiscal years  2007 through 2015, 25,293 claims (less than 0.5 percent of all claims)  were for non-disability cases.

Decided by judges with limited experience. We excluded cases  decided by judges within the first year (365 days) after their  appointment as an ALJ, as calculated by the difference between their  date of appointment and the date of the decision on each claim. We  excluded these decisions to help ensure that variation we identified in  allowance rates was not due to the judges’ more limited experience  deciding Social Security disability claims. From fiscal years 2007  through 2015, 574,307 claims (approximately 9 percent of all claims)  were decided by judges with limited experience.

In total, our exclusion criteria reduced the number of records analyzed by  about half. Specifically, out of a universe of about 6.3 million records, our  study population included about 3.3 million decisions. Nevertheless, the  overall allowance rate for our study population over fiscal years 2007  through 2015 was 62 percent, very close to the overall allowance rate for  the entire population of claims during this period, which was 64 percent.

We calculated allowance rates by dividing the number of favorable  decisions by the total number of decisions (both unfavorable and  favorable). We calculated allowance rates for different units of analysis:

Overall, by program type (Disability Insurance, Supplemental Security  Income, and concurrent) and by year and for all years and program  types pooled together,

At the judge level, by year and for all years pooled together, and

At the hearing office level, by year and for all years pooled together.

When analyzing our data at the case level, we identified whether the case  was favorable or unfavorable to the claimant (that is, whether the claimant  was allowed benefits or not). We did not include cases that were  dismissed in our study population for two reasons. First, as discussed  above, these cases can be dismissed for reasons not related to the merits  of the case, and without a review of the medical evidence. Second, SSA’s  data on dismissed cases are limited, partially because cases are  dismissed without a review of medical evidence. For example, the  impairment code from the hearings-level decision was missing for virtually  100 percent of dismissed cases.

For concurrent claims—those in which an individual is applying for  Disability Insurance (DI) and Supplemental Security Income (SSI)  benefits—we considered a case an allowance if the claimant was  approved for either or both programs. Our classification of allowances for  concurrent cases differs from SSA’s usual practice (although a 2017  internal study of ALJ allowance rates used the same method as ours).  SSA officials said they usually allow the SSI decision to “control” the  overall outcome of the case. That is, SSA classifies a concurrent claim as  an allowance if the SSI decision is an allowance, regardless of the  outcome for the DI claim. Officials said that they chose this method  primarily for convenience. This results in a different classification of some  cases in which the SSI claim was denied but the DI claim was allowed. In  such cases, the claimant is receiving a benefit as a result of their  concurrent disability claim but would be classified in SSA’s data as a  denial. However, the resulting differences in the number of allowances is  very small—less than 4,000 claims over fiscal years 2007 through 2015— and the different definitions did not substantively affect allowance rates in  any year.

SSA policy states that cases are generally assigned on a “first in, first out”  basis, meaning that cases are assigned to judges in the order in which  they are received. Administrative law judges are assigned cases on a  rotational basis, with the oldest case in the backlog given to a judge who  most recently decided a case. Therefore, as noted in prior research, the  initial assignment of cases to judges is random (conditional on applying at  a given hearing office at a given time). Judges do not select their cases,  nor are claimants able to request another judge after one is assigned.  Claimants are generally assigned to hearing offices based on their ZIP  code, although some claimants in hearing offices with higher numbers of  pending claims may be transferred to one of SSA’s five National Hearing  Centers. In those cases, hearings are conducted by videoconference  rather than in person, as is traditionally done in SSA’s regular hearing  offices. However, the claimant may opt out of a video conference hearing  within 30 days of receiving a written notice acknowledging the request for  a hearing.

There are some exceptions to the “first in, first out” rule, such as cases  that are likely to be dismissed or decided on the record (without a  hearing) and critical cases (including terminal illness cases and veterans  who have a 100-percent permanent and total disability compensation  rating). However, as discussed previously, we have excluded all major  categories of exceptions but critical cases from our analyses, and  included a variable to identify critical cases in our analyses.

Assignment of Claims to Steps in SSA’s Sequential Disability Decision-Making Process

SSA’s disability decision-making process includes five sequential steps,  and one part of our analysis was to determine the step at which each  decision was made. In consultation with SSA officials, we used a code in  CPMS—called the regulation basis code—to assign each claim to a  particular step.

Each claim in CPMS has between one and four regulation basis codes,  depending on whether the claim was for a single program (DI or SSI), or a  concurrent claim for both. We assigned each claim to one of the five  steps in SSA’s disability decision-making process, based on its regulation  basis code. Each regulation basis code is associated with one of five  steps. Therefore, if a claim had just one regulation basis code, we  assigned it to the corresponding step. If a claim had more than one  regulation basis code, we used a series of decision rules to select the  most appropriate step. Specifically, claims for a single program have up  to two regulation basis codes listed, and we used the code that matched  the outcome of the case and/or the latest step. We used a similar  method for concurrent claims.

We found that approximately 19 percent of all allowances occur at step 3,  when SSA determines whether a claimant’s impairment meets or is  equivalent to an impairment listed in SSA’s Listing of Impairments. Most  (80 percent) of all allowances are made at the final step (step 5), when  SSA determines whether the claimant can do any work in the national  economy, given the limitations of their impairment and their age,  education, and work experience.

Nearly a third (28 percent) of denials are made at step 4, where SSA  determines whether the claimant can do their past work, and 62 percent  of denials are made at step 5. There are some differences between DI  claims and SSI claims in the distribution of allowances and denials over  the five steps. SSI allowances occur at step 3 to a greater extent than DI  allowances, while SSI denials occur at step 5 to a greater extent than DI  denials (see table 5 below).

Statistical Model of Variation in Allowance Rates across Judges and Associated Factors

We developed our multivariate statistical analyses in consultation with  GAO statisticians, economists, and social scientists and SSA officials and  experts. Our analysis was also informed by a comprehensive review of  the literature pertaining to judicial decision-making and, in particular,  adjudication for SSA’s disability programs. Specifically, we reviewed more  than 90 potentially relevant peer-reviewed academic journal articles,  government reports, and nonprofit association and think tank white  papers. We selected 39 of these studies or reports for a detailed review of  the scope and methodology, key factors or variables used in any  empirical analyses, and other relevant findings. We also reviewed  relevant SSA Office of the Inspector General (OIG) reports and consulted  with SSA and OIG officials, and reviewed prior GAO reports that modelled  judicial outcomes. Our statistical model included variables that are  either direct or approximate measures for: (1) claimant characteristics that  represent criteria used in the disability decision-making process, (2) judge  characteristics, (3) other participants in the decision-making process, (4)  SSA administrative characteristics, and (5) economic characteristics of  the claimant’s state.

Our analysis was purely statistical, in that we did not conduct the legal  analysis needed to reach conclusions about what legal factors might have  affected a judge’s decision or whether the decision that was reached in  any particular case was correct. Similarly, we are not making any  predictions about the likely or correct outcome of future individual  decisions. Each case is unique in both its facts and circumstances and  must be examined on its own merits.

We included factors that represent criteria used in decision-making  process, such as the type of claim (DI, SSI, or concurrent) and the  claimant’s age, years of education (grouped into equivalent levels:  less than high school, high school, some college, and college or  higher), and primary impairment.

We included factors related to the judge’s employment as an ALJ,  such as the year appointed as a judge, the type of appointment  (whether they had a career or temporary, non-permanent  assignment), and any prior work history at SSA (specifically, whether  they were an attorney or held another position prior to being  appointed as an ALJ).

Other participants in the decision-making process

We included factors that represent other participants in the decision- making process, such as the claimant’s use of an attorney or non- attorney representative, or the testimony of a medical or vocational  expert at the hearing. Our prior work has shown, for example, that  claimants who were represented by an attorney or a person who is  not an attorney (such as a relative or professional disability  representative) were more likely to be allowed disability benefits than  claimants who had no representative.

We included factors related to SSA’s administration of its disability  programs, such as the hearing office in which the claim was decided,  whether the claim was heard in one of 10 states that do not have a  reconsideration step between the initial state-level Disability  Determination Service decision and a hearing before an ALJ, and  the percentage of pending cases at the hearing office that were  pending for more than 270 days (SSA’s definition of a “backlogged”  case).

Finally, we assessed economic characteristics of the state in which  the claimant resided because some prior research suggests that such  factors may be associated with disability application and allowance  rates. Specifically, we analyzed:

The unemployment rate in the claimant’s state as of the year of  each decision in our analysis, from the Bureau of Labor Statistics’  Local Area Unemployment Statistics data. We selected this factor  in order to account for the labor market conditions where  claimants live.

The poverty rate in the claimant’s state as of the year of each  decision in our analysis, from the Census Bureau’s American  Community Survey (ACS).

The primary goal of our analysis was to isolate variation in allowance  rates due to the unique judge or hearing office assigned to each claim by  controlling for multiple factors that could otherwise affect this variation.  Some variation in allowance rates across judges and hearing offices  could reflect the distribution of other factors that are correlated with  allowances. For example, judges who hear disability cases in regions of  the country with higher obesity rates—a known risk factor for disability— may appear to have higher allowance rates than those in regions with low  obesity rates. Because judges’ decisions to allow benefits may be related  to this or other factors, simple univariate comparisons of allowance rates  across judges may reflect characteristics of the cases that judges hear.

To help isolate the potential unique effects of judges, we used multilevel,  multivariate statistical models that held constant various factors that could  have been associated with allowance rates. We held constant variables  available in SSA and other public data sources that were relevant to the  claim appeals process, in order to estimate the amount of potential  residual variation across judges.

The data we assembled had a multilevel structure, with applications for  disability benefits clustered within the same judges and hearing offices.  Judges were associated with multiple hearing offices, because judges  sometimes decided cases in multiple hearing offices during the period of  our analysis. For example, judges could travel to more remote sites to  hear cases on a part-time basis.

The data and outcome of interest suggested that a multilevel or mixed  logistic regression model would adequately reflect the data generation  process. We developed a mixed model that represented the grouping  variables—judge, hearing office, and primary diagnosis code—with  random intercepts, similar to prior research. We modeled group variation with random effects primarily for parsimony. Modeling group  variation with fixed effects would have required estimating several  thousand explicit parameters, one for each group level, which would have  consumed many degrees of freedom. Estimating the amount of variation  across groups then would have required interpreting many contrasts  between pairs of fixed effect estimates. In contrast, modeling group  variation using random effects allowed us to represent the variation with  probability distributions and a small number of summary (hyper)  parameters, such as the standard deviation of the judge random effect.

Substantively, random effects accurately represented the SSA policy of  randomly assigning judges to cases in our study population, using a “first  in, first out” method. Moreover, we modeled variation across judges and  hearing offices as random, which implies that we seek to make inferences  about a larger, hypothetical population of judges and hearing offices that  could exist if we replicated the study in the future. This seems  appropriate, because the application review process could be repeated  across many new judges and hearing offices in the future. We do not  seek to make inferences limited to the judges and hearing offices at the  particular time we assembled data.

We held constant case, judge, and hearing office characteristics using  covariates with fixed parameters. The smaller number of parameters  associated with these covariates made a fixed effects approach easier to  apply and interpret. We assumed that the covariate effects did not vary  across groups, so that only the model’s intercept varied randomly. We  had no prior expectation that specific covariate effects should have varied  across groups. Moreover, increasing the number of random effects would  have increased the complexity of the model and could have made it hard  to estimate computationally.

We viewed the covariates primarily as controls for isolating variation  across judges and offices. We did not attempt to build a comprehensive  model that correctly specified how all of the covariates were causally  ordered and related to each other and the probability of an allowance. As  a result, our estimates of these parameters may not be consistent with  those obtained from a more comprehensive modeling effort, or from  analyses designed to estimate the causal effects of particular variables,  such as the use of videoconferences. In the body of the report, we  present alternative explanations and provide context to avoid interpreting  the covariate effects as with a high degree of causal certainty. For  example, we note that claims with legal representation may have higher  approval rates if representatives accept claims with greater merit and,  therefore, a greater chance of compensation. Below, we test alternative  model specifications for covariates where the causal ordering may be  ambiguous, in order to avoid biasing estimates of the judge and office  parameters of primary interest.

Certain variables and parameters were applied across multiple versions  of the model (described below). Let Y denote the allowance or denial  decision for claimant i at any step of the appeals process, with Y = 1 if the  ALJ allowed the claim and 0 otherwise. Each model took a typical  hierarchical generalized linear form for a binary outcome:  The probability of allowance, π, was a function of covariate vectors  measuring characteristics of claims, Xijo, characteristics of the ALJs  assessing those claims, Xjo, and characteristics of the hearing offices  where the decision occurred, X. Claimants were clustered in j = {1, 2, … ,  J} judges, and judges were clustered in o = {1, 2, …, O} offices. g is the  inverse logistic link function.

We included normally distributed random effects, ε(.), for each judge,  office, and the claimant’s primary diagnosis, indexed by diagnosis codes  d = {1, … , D}. Random effects allowed the intercept for each group, α(.) =  α + ε(.), to vary around the population average intercept, α, as a function  of the group’s variance, σ(.):  To make interpretation and computation easier, we classified all  continuous covariates into substantively meaningful categories, and set  the omitted reference categories to the sample modes. This  transformation implied that the random effect variance, σ(.), described  variation across judges and offices for a claim that had the modal value of  all other covariates in the model and sample. The reference claim  remained constant across models fitted to different subsamples, in order  to make inferences about a claim that was typical for the study  population.

The center of the data at the modes, α, may not necessarily correspond  to an actual claim. For example, all judges do not practice at the modal  hearing office, and the modal age for the study population may not be  typical for claims made in the modal office. Nevertheless, rescaling  facilitates estimation and interpretation of the model, because all  inference can be done on α and α(.) directly, using the random effect  variance, σ(.), without transformation. This allowed us to concisely  describe variation in allowance rates for a hypothetical, typical claim in  the joint covariate distribution.

In the body of this report, we summarized variation across judges and  offices, holding constant other covariates at their sample means, using  the estimated distribution of group intercepts scaled in logits:  To describe variation across groups on the probability scale, we  estimated the quantiles bounding the middle 50 and 90 percent of the  group density on the logit scale and then transformed them using the  inverse logistic function, g. This reference case does not represent a  feasible claim, because the means of the categorical covariates are just  the sample proportions. However, this approach complements the  centering of the sample, which allows the group variance parameters,  σ(.), to represent variation across groups for a feasible reference case at  the sample modes.

We fit a sequence of models using different covariates and subsamples,  listed below. Fitting several models allowed us to assess how simplifying  assumptions, such as ignoring the step at which ALJs made allowance  decisions, affected our results. This approach also assessed the stability  of estimates across multiple runs of the computational model estimation  methods. We describe the substantive meaning of the covariates above,  and give their exact measurement when reporting results in table 7 below.

Model 2: Add Covariates  (Unemployment and poverty vary at the state level, not at the office level,  but we include them with the office covariates for simplicity.)

Model 3: Claims Decided at Steps 4 or 5  We estimated Model 2 for only those claims decided at steps 4 or 5,  according to each claim’s Regulation Basis Code. In these last two steps  of the sequential disability decision-making process, the judge determines  whether claimants retain the ability to perform their past work or other  work in the national economy, given the limitations of their impairment  and their age, education, and work experience. SSA officials provided  methods to map these codes to steps of the appeals process. Estimating  the model for decisions at steps 4 or 5 allowed all parameters to vary at  these steps versus all steps in the pooled sample. For example, diagnosis  may be less strongly associated with allowances at step 5 than at step 3,  while the claimant’s age may be more strongly associated.

Models 4-6: Stratify by Year of Decision and Claim Type  To assess how the amount of variation across judges and offices has  changed over time, we estimated Model 2 separately for each year of  decision, claim type, and the cross-classification of these variables.  Stratified models allowed all parameters to vary across claim types and  years.

Model 7: Exclude Potentially Endogenous Covariates  We excluded covariates from Model 2 that may not be exogenous to the  probability of approval. These include representation by an attorney or  other person and the presence of a medical or vocational expert. Claims  with legal representation may have higher approval rates if  representatives tend to accept claims with greater merit and, therefore, a  greater chance of compensation. (Representatives typically receive a  share of their client’s benefits as compensation.) According to SSA  officials, medical and vocational experts may be more likely to testify at a  hearing, depending on the judge’s expected ruling on the case. Although  judges generally have discretion about whether to involve medical and  vocational experts, judges are required to seek the opinion of a medical  expert in certain cases. For example, a judge must have a medical expert  provide an opinion if the judge is considering allowing benefits because  the claimant’s impairment may be medically equivalent to one in SSA’s  Listing of Impairments. Excluding these covariates avoids potentially  biasing estimates of the judge and office parameters of primary interest.

We provide the estimated distributions of allowance rates across judges,  hearing offices, and primary diagnoses, holding all other covariates at  their means, in table 6 below. Each row in the table lists results for one  specification of the model described above. We derived quantiles of the  distributions across groups with the data and estimated model  parameters, using the methods above. The standard deviations of the  allowance rates on the logit scale are explicit parameters in the model  and were directly estimated with the fixed coefficients. We used these  distributions to describe variation across judges, offices, and diagnoses in  the body of this report and in figures, where we interpret the results in  more detail.

Table 7 below provides estimated odds-ratios of allowances for the  factors other than judge, hearing office, and diagnosis in our primary  model of ALJ allowance rates (Model 2 above), along with sample  distributions and raw allowance rates. We used the primary model to  support our findings in the body of this report, where we interpret the  results below in more detail. Our model included variables that are  measures or approximate measures for (1) claimant characteristics that  represent criteria used in the disability decision-making process, (2) judge  characteristics, (3) other participants in the decision-making process, (4)  SSA administrative characteristics, and (5) economic characteristics of  the claimant’s state. The interpretation of the odds ratio for a particular  variable depends on whether the variable is a dummy variable or a  categorical variable. For dummy variables, a statistically significant odds  ratio that is greater/less than 1.00 indicates that claimants with that  characteristic are more/less likely to be allowed than claimants without it.  For categorical variables, a statistically significant odds ratio that is  greater/less than 1.00 indicates that claimants in that category are  more/less likely to be allowed than the claimants in the reference  category.

Evaluation of SSA’s Processes to Monitor Accuracy and Consistency in Hearings Decisions

For objective 2, we reviewed relevant federal laws, regulations, and  documentation, and collected testimonial evidence from SSA officials to  describe and evaluate the processes that SSA uses to monitor hearing  decisions, detect variation, and improve accuracy and consistency. We  interviewed SSA officials at different levels, including officials at  headquarters, regional, DDS, and field office levels. We reviewed  documents such as SSA’s Hearings, Appeals, and Litigation Law  (HALLEX) manual, policy memoranda issued by the Chief Administrative  Law Judge, monitoring and quality assurance reports, user manuals and  guides for electronic tools, SSA OIG reports, and descriptions of  processes that are under development. We assessed these monitoring  efforts against federal internal control standards and our management  and evaluation guide for assessing fragmentation, overlap, and  duplication in government programs. We also reviewed SSA’s annual performance plans from fiscal year 2006 through fiscal year 2017 to  identify performance measures the agency has established to improve  the accuracy and consistency of its hearings decisions. We evaluated the  current performance measures using key attributes of performance  measures used in prior GAO work and federal internal control  standards. In addition to interviews with agency officials, as described  above, we also interviewed officials from organizations representing  judges, disability claimants, and representatives to obtain their  perspectives on SSA’s efforts to monitor and improve accuracy and  consistency.

Appendix II: Comments from the Social Security Administration

In addition to the individual named above, Erin M. Godtland, Assistant  Director; Rachael Chamberlin, Analyst-in-Charge; Dana Hopings, LaToya  King, Stephen Komadina, Rhiannon Patterson, and Jeff Tessin made  significant contributions to the report. In addition, Daniel Bertoni, Deborah  Bland, David Chrisinger, Melinda Cordero, Holly Dye, Bill Egar, Alex  Galuten, Benjamin Licht, Serena Lo, Mimi Nguyen, Samuel Portnow,  Sheila McCoy, and Shana Wallace made valuable contributions.