Employment and Training Programs: Department of Labor Should Assess Efforts to Coordinate Services Across Programs

Federally funded employment and training (E&T) programs help job seekers enhance their job skills, identify job opportunities, and obtain employment. In 2011, GAO identified overlap and fragmentation among E&T programs administered by nine federal agencies. The Workforce Innovation and Opportunity Act (WIOA) was enacted in 2014, in part, to improve coordination and integration among these programs.

This report examines (1) how the number of and obligations for federal E&T programs have changed since GAO's 2011 review, (2) the extent to which E&T programs continue to provide similar services to similar populations and examples of potential effects, and (3) the extent to which agencies have taken actions to address previously identified fragmentation and overlap among E&T programs and what agencies have learned about the results. To address these objectives, GAO surveyed E&T program administrators, reviewed relevant reports and studies, and interviewed federal agency officials.

The number of federal employment and training (E&T) programs and program obligations have declined since GAO's 2011 report. In that review, GAO identified 47 E&T programs and found that 44 had overlap with at least one other program in that they provided similar services to a similar population. In fiscal year 2017, the most recent year data are available, GAO identified 43 E&T programs, or 4 fewer than in 2011 (see figure). From fiscal year 2009 to 2017, federal agencies' annual obligations for E&T programs decreased from about $20 billion to $14 billion. GAO analysis of survey data found the decrease in obligations was largely due to the expiration of funding from the American Recovery and Reinvestment Act of 2009, which had provided additional funding for selected E&T programs during and after the Great Recession.

Survey results from federal administrators of the 43 E&T programs show that the programs continue to span nine agencies and generally overlap by providing similar services, such as employment counseling and assessment services (39 of 43) and job readiness training (38 of 43). Further, programs targeting a specific population, such as Native Americans, veterans, or youth, also provided similar services. In some cases, such overlap may be appropriate or beneficial, but it may also suggest opportunities for greater efficiency.

Almost all (38 of 43) E&T programs reported at least one action to manage fragmentation or overlap, such as co-locating services and sharing information. However, the agencies were not able to consistently provide information on the results of these actions and few evaluations encompassed multiple programs. Among studies GAO identified, six examined more than one E&T program, but only one assessed how any coordinated activities benefited the population served. None of the six studies focused on Native Americans, youth, or refugees. The Workforce Innovation and Opportunity Act (WIOA) encourages agencies to conduct evaluations, and specifically requires the Department of Labor (DOL) to publish a 5-year plan describing certain E&T priorities, consistent with the purpose of aligning and coordinating certain programs. While DOL reported it took some steps, it continues to lack a strategic plan for E&T evaluations over a multi-year period. As a result, DOL does not know whether actions to manage overlap are successful.

GAO recommends that DOL, in consultation with other federal agencies, develop and publish a multi-year strategic plan for its evaluations of employment and training that includes assessing the completeness and results of efforts to coordinate among E&T programs. DOL agreed with our recommendation.

Federally funded employment and training (E&T) programs help job  seekers enhance their job skills, identify job opportunities, and obtain  employment. Since the 1990s, GAO has periodically reported on the  number of and funding for programs that are specifically designed to  provide E&T services and has identified areas of fragmentation and  overlap among them. The most recent report, issued in January 2011,  identified 47 programs administered by nine federal agencies. We found  then that these agencies obligated about $20 billion on E&T activities in  fiscal year 2009, adjusting for inflation, and reported serving more than  24 million individuals.

Since our 2011 review, the unemployment rate in the country has  improved and statutory changes have been made to improve the  structure of the workforce development system and coordination among  workforce development programs. For example, the Workforce Innovation  and Opportunity Act (WIOA) was enacted, which sought in part to  improve the quality and accessibility of services that job seekers and  employers receive at their local American Jobs Center, which are  intended as a “one-stop” location for receipt of employment services. In  addition, WIOA directed federal agencies to take certain actions to better  align and integrate service delivery across multiple E&T programs.

Given these recent changes, you asked us to update the status of  fragmentation and overlap among federal E&T programs. This report  examines (1) how the number of and obligations for federal employment  and training programs have changed since our 2011 report, (2) the extent  to which employment and training programs continue to provide similar  services to similar populations, and examples of potential effects, and (3)  the extent to which agencies have taken actions to address previously  identified fragmentation and overlap among the programs and what  agencies have learned about the results.

To address all of our objectives, we identified programs specifically  designed to provide E&T services based on the definition used in our  2011 review. This generally consists of programs for which objectives  cited in the Catalog of Federal Domestic Assistance (CFDA) included: enhancing the specific job skills of individuals in order to increase their  identifying job opportunities, and/or helping job seekers obtain employment.

To address all three research objectives, we administered a survey to  program officials that included questions to verify information listed in the  CFDA on program objectives, eligibility, and beneficiary requirements. To  answer our first research objective, we included questions about  budgetary information and participants served. To answer our second  research objective, we asked questions about services provided by each  program. To answer our third research objective, we also included  questions about agencies’ actions to manage previously identified overlap  and fragmentation. We reviewed survey responses for completeness and  apparent inconsistencies and clarified information with agency officials, as  needed. However, we did not review agencies’ financial reporting  systems or audit the figures provided to us. To address limitations this  may have created, we asked agencies to identify the data source of  reported budgetary information and to list any data limitations. We did not  conduct a legal analysis to confirm the various characterizations of the  programs in this report, such as services provided, target population,  eligibility criteria, or program goals. Instead, such program information in  this report is generally based on our survey results as confirmed by  agency officials.

To address the first objective, we compiled a list of E&T programs by  starting with the 47 programs administered by nine federal agencies that  were identified in our 2011 review. We updated that list by (1)  interviewing federal agency officials; (2) searching program objectives in  the CFDA to confirm that programs previously identified still met our  criteria for inclusion and to identify any additional programs that met our  criteria; and (3) reviewing other GAO reports published since 2011 that  provided a more in-depth review of specific types of E&T programs. In  any such analysis, the number of programs identified is likely to vary with  the definition used, and applying any definition can require subjective  judgment.

To address our second objective, to identify areas of overlap among E&T  programs, we reviewed beneficiary eligibility requirements listed in the  CFDA and confirmed by agencies to categorize programs by targeted  population served, and compared the categories of services the programs  reported providing in our survey. We had not identified duplication in prior  reports on E&T programs; we asked in our survey about programs’ efforts  to detect and prevent it.

To address our second and third objectives, we also reviewed GAO  reports and agency-funded studies published since 2011. We used these  sources, in part, to illustrate effects of overlap and fragmentation among  E&T programs and provide examples of actions agencies have taken to  address our prior findings or recommendations. In addition, to address  the third objective, we analyzed survey results regarding agencies’ efforts  to address previously identified overlap and fragmentation and to detect  and prevent duplication. For selected efforts, we contacted agency and  program officials regarding their knowledge of the results. We also  reviewed the impact or effectiveness studies agencies performed on the  programs identified for our review. For more details on our scope and  methodology, see appendix I.

We conducted this performance audit from September 2017 to March  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Program Fragmentation, Overlap, and Duplication

Over the years, we have issued several reports on fragmentation,  overlap, and potential for duplication among federally funded employment  and training (E&T) programs and identified areas where inefficiencies  might result. This report, like our prior work, uses the following definitions:

Fragmentation refers to circumstances in which more than one  federal agency (or more than one organization within an agency) is  involved in the same broad area of national need and opportunities  exist to improve service delivery.

Overlap occurs when multiple agencies or programs have similar  goals, engage in similar activities or strategies to achieve their goals,  or target similar beneficiaries.

Duplication occurs when two or more agencies or programs are  engaged in the same activities or provide the same services to the  same beneficiaries.

During the 1990s, we issued a series of reports that documented program  overlap among federally funded E&T programs and identified areas  where inefficiencies might result. For example, we found that program  overlap might hinder people from seeking assistance and frustrate  employers and program administrators. In 2000 and 2003, we identified  federally funded E&T programs for which a key program goal was  providing E&T assistance. In our most recent report in 2011, we  identified 47 E&T programs and found that 44 of them overlapped with at  least one other program in that they provided similar services to a similar  population. We also found that due to the American Recovery and  Reinvestment Act of 2009 (Recovery Act), both the number of—and  funding for—federal E&T programs had increased since our 2003 report,  but little was known about the effectiveness of most programs because  only five programs had conducted impact evaluations.

Our guide on identifying and reducing fragmentation, overlap, and  duplication notes that determining whether fragmentation and overlap  exist among programs is a key step in identifying opportunities to improve  efficiency and effectiveness of programs. In some cases, it may be  appropriate or beneficial for multiple agencies and programs to be  involved in the same programmatic or policy area due to the complex  nature or magnitude of the federal effort. However, our guide states that it  is also important to use the results of existing or new evaluations of  identified programs to assess options to reduce or better manage  negative effects of fragmentation, overlap, and duplication, such as  inefficient use of program funds.

Key Changes Since Our 2011 Review of Employment and Training Programs

Workforce Innovation and Opportunity Act (WIOA)

Enacted in July 2014, WIOA repealed and replaced the Workforce  Investment Act of 1998 (WIA). WIOA placed greater emphasis on  aligning and integrating workforce programs, which are administered  primarily by the Departments of Labor (DOL) and Education (Education),  with support from the Department of Health and Human Services (HHS)  and other agencies. For example, under WIOA, DOL and Education  review and approve 4-year strategic plans for states’ workforce  development systems. WIOA also requires certain programs and  encourages other programs to be available through centralized service  delivery points referred to as American Job Centers. In addition, WIOA  requires that DOL and Education collaborate to implement a common  performance accountability system for six core programs, which presents  agencies with an opportunity to align definitions, streamline performance  indicators, and integrate reporting across these programs.

Since our 2011 inventory of federal E&T programs, which focused on  fiscal year 2009, both the Great Recession and one-time funding made  available under the Recovery Act have ended. Recovery Act funds were  provided to help preserve and create jobs and promote economic  recovery, among other purposes.

With the end of the recession, the unemployment rate has substantially  declined. The rate increased from 4.6 in 2007 to a peak of 9.6 in 2010  before declining to 4.4 in 2017 (see fig.1).

WIOA encourages DOL, Education, HHS, and other relevant federal  agencies to conduct program research and evaluation. For example,  WIOA requires DOL to publish a plan every 2 years that describes the  research, studies, and multistate project priorities of DOL concerning  employment and training for the following 5-year period. This includes a  provision that the plan be consistent with certain purposes, including the  purpose of aligning and coordinating core programs with other partner  programs provided through American Job Centers.

In addition to WIOA requirements, we have also previously reported that  each federal agency should require its major program components to  prepare annual and multiyear evaluation plans and to update these plans  annually. The planning should take into account the need for evaluation  results to inform program budgeting, reauthorization, agency strategic  plans, program management, and responses to critical issues concerning  program effectiveness. These plans should include an appropriate mix of  short- and long-term studies to produce results for short- or long-term  policy or management decisions. To the extent practical, the plans should  be developed in consultation with program stakeholders. Furthermore,  leading organizations, including the American Evaluation Association and  the National Academy of Sciences, emphasize the need for research  programs to establish specific policies and procedures to guide research  activities. In addition to planning for formal evaluation, Standards for  Internal Control in the Federal Government emphasize the importance of  managers routinely assessing the results of their actions, for which  evaluation is a potential tool.

Since 2011, Employment and Training Services Are Delivered through Fewer Federal Programs and with Reduced Obligations

Number of Employment and Training Programs Declined, Due in Part to Eliminations

The number of federal E&T programs has decreased since our last report  on them in 2011. For fiscal year 2017, we identified 43 programs, four  fewer than we reported in 2011. The number decreased because more  programs were eliminated or defunded (6) than added (2). For example,  in 2014, the Workforce Innovation and Opportunity Act (WIOA) eliminated  at least four of our identified E&T programs. This included 1) DOL’s  Veterans’ Workforce Investment Program, 2) Education’s Grants to  States for Workplace and Community Transition Training for Incarcerated  Individuals, 3) Education’s Migrant and Seasonal Farmworkers Program,  and 4) Education’s Projects with Industry program. In addition, Congress  did not appropriate funds for Education’s Tech Prep Education State  Grants in fiscal year 2011 and DOL’s Community Based Job Training  Grants programs in fiscal year 2010, according to agencies’ budget  documents.

We also identified two additional E&T programs through interviews with  agency officials and a related GAO report: 1) Department of Veterans  Affairs’ (VA) Compensated Work Therapy, and 2) Department of  Defense’s (DOD) Job Training, Employment Skills Training,  Apprenticeships, and Internships.

For changes in the program list from our 2011 review to our current  review, see appendix II.

The 43 programs we identified in fiscal year 2017 are fragmented across  nine federal agencies, as programs were in 2011 (see fig. 2).

Federal Obligations for Employment and Training Programs Decreased, Due in Part to the End of Recovery Act Funding

Our survey results showed that the federal government obligated nearly  $14 billion to the E&T components of its programs in fiscal year 2017, a  decrease of about $5.4 billion or 30 percent, adjusting for inflation, from  the amount in our 2011 review (which reported fiscal year 2009  obligations). According to our analysis of survey data, much of the  decrease in E&T obligations can be explained by the expiration of  Recovery Act funding. For example, two-thirds of the Recovery Act  funding designated for E&T programs went to four DOL programs that  received a combined $3.8 billion in Recovery Act appropriations. From  fiscal year 2009 to fiscal year 2017, the combined E&T obligations for  these four programs decreased by $4.7 billion, or 58 percent.

Of the 31 E&T programs that reported E&T obligations in our survey,  eight programs were responsible for more than $11 billion, or 82 percent  of the total in fiscal year 2017. Their shares of 2017 E&T obligations  ranged from 5 percent for DOL’s Wagner-Peyser Act Employment  Service to 21 percent for Education’s State Vocational Rehabilitation  Services Program (see fig. 3). Among these eight programs responsible  for the vast majority of E&T obligations, all must be included in state plans  required under WIOA, except for DOL’s Job Corps, VA’s Vocational  Rehabilitation and Employment, and HHS’ Temporary Assistance for  Needy Families (TANF). In addition, all but DOL’s Job Corps and VA’s  Vocational Rehabilitation and Employment are state-administered.

For complete data on reported changes in E&T obligations between fiscal  years 2009 and 2017, for the 29 programs that provided estimates in both  years, see appendix III for numbers adjusted for inflation and appendix IV  for unadjusted numbers.

The number of people served by E&T programs also declined, from 24  million to 11 million individuals in the most recent year for which data  were available, or a 56 percent decrease from the number reported in the  2011 report. Two of DOL’s E&T programs—the Wagner-Peyser Act  Employment Service and the WIOA Adult Program—accounted for the  majority of this decrease, dropping by 8 million and 4 million, respectively.  Participation in certain programs, for example, Wagner-Peyser Act  Employment Service and WIOA Adult Program, changed markedly as the  economy improved, suggesting that enrollment is highly sensitive to  economic conditions. Since we last reviewed these programs in 2011, the  U.S. economy has improved and the unemployment rate dropped by 53  percent (see fig. 4).

DOL officials said these factors could have reduced the demand for  certain E&T services. Unemployment is an important driver of demand for  some, but not all, E&T programs. For example, demand for certain  employment and training services, such as vocational rehabilitation, may  be relatively insensitive to economic conditions. In addition, technology  has the potential to change workforce needs in certain industries, leading  to workers who need retraining. In addition, DOL officials told us that  under WIOA a new definition of program participant, effective in 2016,  that primarily impacted the number of participants reported for Wagner- Peyser Act Employment Service, WIOA Adult Program, and WIOA  National Dislocated Worker Grants.

Employment and Training Programs Administered by Various Agencies Generally Overlap, but Effects of Overlap May Vary

The 43 E&T programs generally overlap in that they provide similar  services to similar populations, according to our survey analysis (see  table 1). In our survey, almost all of the 43 programs reported providing  employment counseling and assessment services as well as job search  or job placement activities (39), job readiness training (38), and job  referrals (37). The least commonly provided service selected from our  list of service categories–high school completion or equivalency  assistance–was provided by over half (26) of the programs.

Through our survey, eight of the 43 programs reported serving the  general population (that is, a relatively broad target) and the remaining 35  reported serving a narrower target population, such as Native Americans  (8), veterans and transitioning servicemembers (7), or youth (5).,  Our survey analysis shows overlap in services exists among programs  serving the general population as well as among those serving each  specific target population. Specifically, a majority of programs targeting  the general population, Native Americans, and youth reported providing  many of the same services. For example, all of the five youth programs  reported providing similar E&T services, such as employment counseling  and assessment and job readiness training (see fig. 5). For more  information on services provided by programs serving selected target  populations, see appendix VI.

Many of the E&T programs targeting specific populations are fragmented  across multiple agencies. For example, four agencies administer the eight  Native American E&T programs and three administer the seven programs  for veterans (see table 2).

Other includes older workers, women, and unemployed and underemployed residents of solid and  hazardous waste-impacted neighborhoods.  According to VA officials, VA’s Vocational Rehabilitation and Employment program serves individuals  with a service connected disability and VA’s Compensated Work Therapy program serves individuals  enrolled in Veterans Health Care and does not require a service connected disability.

Overlap among program services may have benefits, but it may also  suggest opportunities for coordination or efficiencies in service delivery.  Overlap may be beneficial in 1) helping program participants with specific  needs better access E&T services, 2) providing more tailored or intensive  support services, or 3) achieving higher quality outcomes for specific  populations than would be achievable from their use of a more broadly  targeted program. For example:

A 2015 study funded by DOL on services provided to veterans  through the public workforce system in Texas found that veterans who  received intensive services from DOL’s Disabled Veterans’ Outreach  Program Specialist or Local Veterans’ Employment Representative  staff subsequently had higher earnings than veterans who did not,  although these same veterans may have been eligible for similar  services provided by other programs to the general population.

A 2017 study funded by the U.S. Department of Agriculture (USDA)  on its Supplemental Nutrition Assistance Program (SNAP) E&T— which helps participants who are eligible to receive nutrition  assistance from the federal government better access E&T services  —found that program participants also received support services,  such as child care vouchers and transportation assistance.  Participants said these services were important to their participation in  the E&T program and helped those with specific needs better access  E&T services.

However, when multiple programs overlap or are fragmented, there is  also a risk that program administrators may not make efficient use of  available resources if they do not coordinate their efforts. Without careful  coordination, programs may not fully leverage mutual benefits or  participants may find administrative requirements burdensome or  redundant. For example:

A 2018 GAO report on USDA’s SNAP E&T program found that 20  states’ SNAP E&T programs did not partner with workforce agencies  to provide E&T services. States that do not fully leverage resources  available through the workforce development system may miss  opportunities to serve a greater number of SNAP E&T participants  and provide a wider variety of services. GAO recommended the  administrator of the Food and Nutrition Service take additional steps  to assist states in leveraging available workforce development system  resources.

A 2017 study funded by DOL on American Job Centers found that  customers became frustrated filling out applications in what they  viewed as redundant paperwork requirements for multiple programs  with varying eligibility criteria.

Almost All Agencies Reported Actions to Address Program Fragmentation and Overlap, but Effectiveness of these Actions Remains Uncertain

Employment and Training Program Officials Reported Taking Actions to Address Fragmentation and Overlap

In response to our survey of agency officials for the 43 E&T programs,  almost all (38) reported taking at least one action to manage  fragmentation, overlap, and/or potential duplication. Common actions  included providing program guidance and technical assistance,  coordinating participant services (e.g., co-locating services or co-enrolling  participants), and effectively managing grants (see table 3).

Our survey analysis showed that of 43 E&T programs, 31 across eight  agencies reported taking at least one action to manage fragmentation. In  addition, 38 programs across all nine agencies reported taking at least  one action to manage overlap. For example, to address fragmentation  and overlap, officials representing seven programs within DOL and  Education reported in our survey that they participated in interagency  workgroups to share information and to facilitate cross-agency  communication to coordinate services. Likewise, VA reported that the  agency and DOL updated their interagency technical assistance guide to  better align the agencies’ veteran E&T programs. (See table 4.)

Program officials reported that their actions were motivated by a variety of  factors, including their own assessments, legal requirements such as  those in WIOA, and audit recommendations. They attributed some of their  actions to their assessment of the potential for duplicative services, or to  promote streamlined administration. For example:  In 2014, DOL released updated guidance to administrators of its  Disabled Veterans’ Outreach Program to encourage coordination with  its Wagner-Peyser Act Employment Service program to help ensure  that the two programs were not providing similar services to  veterans.

Education reported that its data collection and reporting system  integrates data from the State Vocational Rehabilitation Services  Program and State Supported Employment Services Programs.  Likewise, Education reported that its monitoring and technical  assistance guide addresses both the State Vocational Rehabilitation  Services Program and the State Supported Employment Services  Program.

In addition, DOL and other agencies reported taking actions that are  either required or encouraged by federal law in order to manage  fragmentation and overlap. For example:

DOL officials reported that since WIOA was enacted in 2014, DOL,  Education, and HHS have jointly issued directives and guidance to  help states implement and administer WIOA, such as guidance on  developing their required state strategic plans. Also under WIOA, DOL  and Education have issued joint regulations and established common  data definitions and joint data collection instruments to align  performance reporting for WIOA six core programs.

Agencies with E&T programs targeted toward Native Americans  reported that tribes’ use of authorized plans to integrate employment,  training, and related services programs can help manage  fragmentation and overlap. The potential scope of such plans  (referred to as 477 plans), which had been originally authorized in  1992, was increased via legislation in 2017 to include programs with  more purposes. With an authorized plan in place, tribes can integrate  certain federal funds received by the tribe and coordinate  employment, training, and related services across multiple programs  that serve the tribe. In December 2018, 12 agencies signed a  memorandum of agreement intended to set forth the basic functions  and relationships of those agencies in the funding and oversight of  tribal 477 plans and to facilitate coordination and collaboration  between the agencies.

Agencies have also taken actions to improve collaboration across multiple  E&T programs based on our recommendations or on internal audits. For  example:  In 2011, we recommended that the Secretaries of DOL and HHS work  together to develop and disseminate information that could facilitate  further progress by states and localities in increasing administrative  efficiencies in E&T programs, such as state initiatives to consolidate  program administrative structures and state and local efforts to co- locate E&T programs at one-stop centers. In response, DOL and  HHS took a number of steps, including issuing a January 2015 study  focused on identifying and documenting potentially promising  practices in coordinating Temporary Assistance for Needy Families  (TANF) and WIA services at the state and local levels.

In 2012, we found that the interagency handbook used by DOL and  VA to coordinate E&T services for veterans did not include, for  example, incorporating labor market information into rehabilitation  plans. In 2015, as GAO recommended, these agencies revised the  interagency handbook by outlining how VA and DOL staff should  coordinate efforts to provide veterans with labor market information  when developing employment and training objectives and assist them  in selecting training and credentialing opportunities as a part of their  rehabilitation plans.

In 2012, EPA’s Office of Inspector General conducted an audit of its  Environmental Workforce Development and Job Training Cooperative  Agreements program which concluded that, absent internal controls,  the program was at risk for duplication with other E&T programs. To  mitigate that risk, the lead program administrator now provides other  federal agencies a list of program applicants to ensure that no  applicant is receiving funds for the same purposes outlined in the  Environmental Workforce Development and Job Training program  application.

While most programs reported taking action to manage fragmentation or  overlap, officials from five programs reported in our survey that they had  taken no action. Officials from four of these programs reported that no  action was necessary because their program offered a unique service or  served a specialized population. While we did not further review the  need for coordination among these programs and others, they  nonetheless reported one or more services in common with others  serving the same population. In addition, while unique aspects may be  protective to some extent against the risk of duplication, unique features  may not necessarily reduce the risk of overlap or need for coordination.  For example, DOD officials stated that apart from its Job Training,  Employment Skills Training, Apprenticeships, and Internships program,  they were not aware of any other federal program that allows  servicemembers to participate in job training, including apprenticeships  and internships, beginning up to 6 months before their service obligation  is completed. DOL officials confirmed that its Transition Assistance Program does not offer job training to service members, but it does, like  the DOD program, offer pre-separation employment services and  counseling. VA also noted in its technical comments that  servicemembers who meet Vocational Rehabilitation and Employment  eligibility criteria may, with DOD permission, receive these job training  services as part of their rehabilitative program and that it partners with  DOD to train transitioning servicemembers as veterans’ services  representatives. We did not further review the need for coordination  among these or other programs that reported no action, but absent a  more complete evaluation, it is not possible to assess whether these  programs have taken sufficient steps to address overlap. Regarding  duplication, 14 programs reported no action either to detect it or to  prevent it.

Agencies Did Not Consistently Assess the Effectiveness of their Actions to Manage Overlap and Fragmentation

Agencies administering E&T programs did not consistently have  information on results to know how well their actions to manage program  fragmentation and overlap were working. DOL officials told us that they  generally had not assessed the actions they reported in our survey to  manage overlap, fragmentation, and potential for duplication, but noted  that the agency has begun an implementation study of WIOA that will  include examining state and local efforts to increase program coordination  and collaboration. DOL expects the final report will be completed in fall  2019, and agency officials said it is coordinating with other agency  partners.

Asked about efforts made by specific programs to manage overlap and  fragmentation, other agency officials said they had assessed results of  these efforts in some cases, but not others. For example, VA officials told  us that in 2016 they started tracking referrals between its Vocational  Rehabilitation and Employment Program and DOL’s programs targeted to  veterans to help ensure participants were obtaining labor market  information from DOL programs. In contrast, in the case of integrating  multiple E&T programs targeted toward Native Americans, HHS officials  reported that the agency has not made specific efforts to assess the  effectiveness of plans first provided for in 1992 which might reduce  administrative burden by allowing tribes more flexibility to combine E&T  services funded by multiple federal agencies.

GAO’s guide on fragmentation and overlap states it is important to use  the results of existing or new evaluations of identified programs to assess  options to reduce or better manage negative effects of fragmentation,  overlap, and duplication, such as inefficient use of program funds. For  example, evaluation and other periodic reviews could help identify ways  to address (1) gaps in information on how multiple programs are serving  the employment and training needs of specific populations, such as  Native Americans, youth, and refugees, or (2) the extent to which they  have implemented practices to manage unwanted effects of  fragmentation and overlap and improve coordination and efficiency.

Agencies reported completing additional impact studies since our 2011  review, but evaluations examining their programs’ effects have generally  been confined to a single program and/or specific target populations. Four  of the nine agencies in our review reported that they had completed at  least 13 impact studies since 2011 of individual programs that measured  effectiveness in terms of outputs and outcomes. (See appendix VII for a  list of these studies.) DOL officials told us that programs tend to be  evaluated individually for their effectiveness in achieving individual goals  and objectives rather than for collective effects or performance. DOL  officials said that they perform some research covering multiple programs  in preparation for conducting program impact or effectiveness studies, but  that the related findings tend to be more descriptive in nature. They also  cited plans to use common measures developed under WIOA to look at  outcomes across the core programs.

Some agencies have sponsored studies that focus on populations served  by multiple programs, including customer experience with receiving  services from multiple programs, and an early snapshot of the extent of  state-level coordination in implementing WIOA. Specific examples of  studies that reviewed issues related to implementing multiple programs  include:

A 2015 Mathematica study funded by HHS of WIOA-funded programs  that included numerous efforts state level administrators could  undertake to improve coordination among the programs, including  exchanging more information on strategies and methods used by  each program to address obstacles that impede coordination.

A 2015 Rand Corporation study funded by DOD that examined  employment support programs for reservists and recommended  assessing the costs and benefits of streamlining the current program  line-up to reduce any redundancies.

A 2017 study by IMPAQ International funded by DOL that identified  areas where customer service in WIOA job centers could be  improved, such as streamlining enrollment and registration  procedures and providing more information about the full array of  services at the centers.

However, of the six completed studies we identified that examined more  than one E&T program, only one study assessed how any coordinated or  integrated activities benefited the population served. We found no  similar studies conducted on the effects of multiple programs targeted  toward other populations, such as Native Americans, youth, or refugees.  VA officials told us that it is important that reviews of E&T programs for  specific population take into account the complex needs of that population  to understand when there is a need for involvement of multiple programs.  For example, officials said that special populations such as homeless  veterans require a breadth of unique services that may not available  through a single program or by programs serving the general population.  Further, as programs more commonly work together, learning about the  programs’ collective impact may be as important as studying the  programs’ individual results.

DOL officials told us that DOL, HHS, and Education tend to independently  create their evaluation plans for employment and training services. After  WIOA was enacted, these agencies formed the WIOA Evaluation  Workgroup with the intent of establishing greater collaboration among federal agencies on E&T program evaluation. DOL E&T programs make  up over a third of all federal E&T programs, and some of these programs  under WIOA coordinate or align their services with programs  administered by other agencies. DOL officials told us WIOA Evaluation  Workgroup members interacted with staff from other agencies, such as  USDA, who administered E&T programs to encourage their participation.  The workgroup met for the first time in September 2017. After the initial  meeting, according to DOL officials, the agencies dissolved the group  because they concluded that the topic of WIOA-related evaluation could  be covered through existing periodic interagency meetings. However,  DOL officials told us that these efforts do not focus on evaluation across  programs. In addition, the DOL agency-wide evaluation plan for fiscal  year 2018—issued in September 2018—does not list evaluations focused  primarily on cross-program coordination or collaboration, nor does it  address potential overlap and fragmentation among its E&T services.

Since 2013, DOL has not published a 5-year strategic research plan for  E&T programs. In our 2011 review of DOL’s research and evaluation  program for its E&T programs, we recommended that DOL develop a  mechanism to enhance the transparency and accountability of its E&T  research by consulting other key federal agencies and involving advisory  bodies or other entities outside DOL. In 2010, the Employment and  Training Administration (ETA), the division with lead responsibility for  DOL’s E&T programs, began a series of meetings with a panel of outside  experts to develop a 5-year research plan. This strategic research plan  set the research agenda for E&T programs by identifying and prioritizing  what research and evaluations would be initiated over the following 5  years. Before finalizing its research agenda, DOL obtained broad input  from federal officials at Education and HHS and a range of other key  stakeholders, such as officials in local and state government and  academics from the workforce community. In May 2013, DOL submitted  to Congress and posted on its website a 5-year strategic research plan  for its E&T programs which covered program years 2012 to 2017.

In contrast to the broad consultation and public exposure that  characterized past strategic planning for E&T research, in recent years  DOL has instead relied on an internal process to set its research and  evaluation priorities for its E&T programs and publishes only an agency- wide evaluation plan that is shorter-term and developed for a different  purpose. Specifically, ETA develops an annual learning agenda that  officials indicated highlights its research priorities, ideas, and proposed  studies. Officials stated that the E&T learning agenda is provided for  consideration with other agency-wide agendas in developing an annual  evaluation plan for all of DOL. While DOL’s annual evaluation plan and  the results of its evaluations are posted publicly through its website and  submitted to the relevant congressional committees, the learning  agendas, including those for E&T programs, are internal documents, and  DOL does not release them to the public.

The DOL-wide evaluation plan that is published presents neither a  strategy for E&T evaluation nor plans for any evaluation to be initiated  more than a year in the future. The fiscal year 2018 DOL-wide evaluation  plan discusses only research to be initiated during the next year (fiscal  year 2019) and lists studies that remain in progress from previous years.  Rather than project longer-term research needs, the plan’s main purpose,  according to DOL officials, is to comply with specific appropriations  language. DOL officials told us that the list of proposed studies in the  learning agendas may not ultimately appear in the annual evaluation plan because they are not near-term priorities for the agency-wide plan. DOL’s  fiscal year 2018 agency-wide plan describes initiation of four studies—two  on apprenticeship, one on strategies to prevent improper unemployment  insurance payments, and another on potential effects of application fees  for certain ETA programs.

WIOA requires that DOL publish a plan every 2 years that describes “the  research, studies, and multistate project priorities of the Department of  Labor concerning employment and training for the 5-year period following  the submission of the plan.” DOL officials told us that it is complying with  this requirement by providing ETA’s annual learning agendas to be  included in DOL’s overall evaluation plan. However, the resulting agency- wide plan falls short of meeting best practices for robust strategic  planning. As we have previously reported, these practices include:

Preparing annual and multiyear evaluation plans and updating these  plans annually to take into account the need for evaluation results to  inform program budgeting, reauthorization, agency strategic plans,  program management, and responses to critical issues concerning  program effectiveness.

Including an appropriate mix of short- and long-term studies to  produce results for short- or long-term policy or management  decisions.

Developing plans in consultation with program stakeholders to help  agencies ensure that their efforts and resources are targeted at the  highest priorities and to create a basic understanding among the  stakeholders of the competing demands that confront most  agencies.

A 2010 internal DOL memo stated that such a plan can guide the  development of research and evaluation projects and be a valuable tool  for the broader workforce research community. Furthermore, leading  organizations, including the American Evaluation Association and the  National Academy of Sciences, emphasize the need for research  programs to establish specific policies and procedures to guide research  activities. For example, a 2016 American Evaluation Association guide  stated that having annual and multi-year evaluation plans is useful in  guiding program decision-making in such areas as program management  and budgeting, and responding to issues concerning program  effectiveness. Finally, Standards for Internal Control in the Federal  Government state more broadly that program managers may need to  conduct periodic assessments to evaluate the effectiveness of their  actions. These may include but are not limited to formal evaluations.  However, without a long-term evaluation plan developed in consultation  with key stakeholders, DOL may not learn whether its actions to improve  E&T program coordination and integration are working, and thus may  continue undertaking activities that are not leading to desired results.

With the enactment of WIOA in 2014, steps were taken toward aligning  employment and training programs and ensuring greater cross-agency  coordination. Since then, agencies and programs have reported taking a  range of actions to increase coordination among E&T programs and  manage fragmentation and overlap. However, without knowing whether  these actions are working to improve program coordination and  integration, agencies may persist in activities that are ineffective, fail to  expand those that work, or ignore unintended consequences. Further, the  lack of evaluation focused on program coordination has resulted in a void  of information on programs’ collective impact. Without strategically  planning the use of evaluation resources, DOL and other agencies will not  learn efficiently about whether their efforts to coordinate the programs  have been successful and what impact the newly coordinated programs  are having, collectively, on their shared objectives.

We are making the following recommendation to DOL:  The Secretary of DOL should develop and publish a multi-year strategic  research plan for evaluation of its employment and training programs that  includes assessing the completeness and results of efforts to coordinate  among E&T programs to address overlap and fragmentation. In  developing this plan, DOL should also consult with other federal agencies  and key stakeholders on ways to address gaps in information on how  multiple programs are serving the employment and training needs of  specific populations, such as Native Americans, youth, and refugees.  (Recommendation 1)

We provided a draft of this report for review and comment to the  Departments of Agriculture, Defense, Education, Health and Human  Services, Interior, Justice, Labor, and Veterans Affairs, and to the  Environmental Protection Agency.

We received formal written comments from DOL and VA that are  reproduced in appendix VIII and IX. In addition, DOL, Education, HHS,  Interior, USDA and VA provided technical comments which we  incorporated into the report as appropriate. EPA, DOD, and DOJ did not  have any comments.

DOL agreed with our recommendation that it develop and publish a multi- year strategic research plan for evaluation of its E&T programs consistent  with the purpose of aligning and coordinating these programs. DOL stated  that it actively plans and makes public the research and evaluation topics  for these evaluations, but it did not identify a timeline or measures it  would take to augment these basic steps.

We recommended that DOL consult with other federal agencies and key  stakeholders in developing a strategic research plan that assesses the  completeness and results of efforts to coordinate among E&T programs  to address overlap and fragmentation. Consultation should include ways  to address gaps in information on how multiple programs are serving the  employment and training needs of specific populations, such as Native  Americans, youth, and refugees. DOL stated that it will consult with  stakeholders regarding the employment and training needs of specific  populations. VA commented that such reviews of E&T programs for  specific populations should take into account the complex needs of the  population being served and the breadth of needed services.  We agree  that any such reviews should address how the collection of programs is  serving each population’s needs.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff that made key contributions to this  report are listed in appendix X.

This appendix discusses our scope and methodology for our three  research objectives examining (1) how participation in and obligations for  federal employment and training programs have changed since our 2011  report, (2) the extent to which employment and training programs  continue to provide similar services to similar populations, and examples  of potential effects, and (3) the extent to which agencies have taken  actions to address previously identified fragmentation and overlap among  the programs and lessons learned.

The sections below discuss the methods we used to address each of the  three objectives. In addition to these methods, we reviewed relevant  federal guidance and other program documents; and interviewed federal  agency officials at headquarters offices. The focus of this review was how  employment and training services are coordinated among programs  specifically designed to deliver such services. As such, our scope  excluded some programs that offer or finance employment and training  services, but for which this is not a program objective (for example,  student loan programs, which focus primarily on enhancing access to  postsecondary education). Similarly, we focused on programs that deliver  direct service rather than tax expenditures, which may finance or  incentivize similar services through tax benefits.

To address all of our objectives, we compiled a list of employment and  training programs by starting with the 47 programs administered by nine  federal agencies that were identified in our prior work. We updated the  original list by asking federal agency officials to provide the current status  of previously identified programs and identify any new ones that might  meet our criteria. As in our 2011 review, we included programs for which  objectives cited in the Catalog of Federal Domestic Assistance (CFDA)  covered: enhancing the specific job skills of individuals in order to increase their  identifying job opportunities, and/or helping job seekers obtain employment.

We also searched the CFDA electronically in February 2018 to identify  any additional programs that met our inclusion criteria. To conduct an  electronic text search of the CFDA database, we used 12 search terms  used in GAO-11-92. These included:  We excluded any programs that met one or more of the following criteria:

Program objectives do not explicitly include helping job seekers  enhance their job skills, find job opportunities, or obtain employment.

Program does not provide employment and training services itself  (e.g., it provides financial support to other employment and training  programs, or subsidizes the cost of employment through tax credits).

Program is small or is a component of a larger employment and  training program, such as a pilot or demonstration program.

Programs that are economic development programs that aim to  increase job opportunities but do not provide services to individuals to  enhance their job skills, identify job opportunities, or find employment.

Programs that aim to achieve broad workforce-related goals, such as  increasing educational opportunities for minority individuals in  particular fields or improving the status of and working conditions for  wage-earning women, but do not provide employment or training  services themselves.

Education programs that fund student loans for educational expenses,  initiatives for student recruitment and retention, or other student  support services.

Programs that support training for training providers, such as  vocational rehabilitation specialists, or other programs that support  job-specific training for individuals who are already employed.

Two analysts independently reviewed the list of 211 programs identified in  the list generated from the 2018 CFDA search against the inclusion and  exclusion criteria described above. To reach concurrence on the  programs list, the analysts compared their lists and reached agreement  on which to include. If the analysts were undecided about including a  program, another analyst was consulted.

We also reviewed other GAO reports published since 2011 that provided  a more in-depth review of employment and training programs to identify  any additional programs that met our three inclusion criteria. As a result  of that process, we identified three programs that met our criteria and  added them to our list. It is important to note that the number of programs  identified will vary with the definition used, and applying any definition can  require subjective judgment.

After evaluating all identified potential programs, we determined that 46  employment and training (E&T) programs met all criteria to be included in  our audit. Once our determinations were made, we sent emails to agency  liaisons asking them to confirm the list of programs to be included in and  excluded from our review, and to provide the names and contact  information for the officials who would be responsible for completing our  planned survey. Agencies confirmed our final inclusion and exclusion  decisions. After administering our survey, we excluded DOD’s Troops to  Teachers Program because the program generally focused on teacher  quality rather than E&T services. We also excluded DOD’s Hiring Heroes  Program because DOD officials told us the program does not receive a  specific appropriation and is a small program that is part of DOD’s larger  effort to encourage the employment of servicemembers and veterans.  After we administered our survey, DOL officials clarified that the Women  in Apprenticeship and Nontraditional Occupations (WANTO) program was  not a sub-program under the Registered Apprenticeship Program, but  rather a discrete program. We sent a survey to WANTO program officials.  At the end of this process, we confirmed that 43 programs met our  definition and should be included in our review. We generally maintained  consistency with decisions made in our 2011 review.

To address all of our objectives, we administered a survey to program  officials that included questions about services provided, budgetary  information, and participants served. In addition, we included questions  asking agency officials to confirm or correct program objectives and  eligibility and beneficiary requirements listed in the CFDA. We also  included questions about agencies’ actions to manage overlap and  fragmentation. We conducted two pretests with VA to ensure (1) our  questions were clear and unambiguous, (2) terminology was used  correctly, (3) the survey did not place an undue burden on agency  officials, (4) the information could feasibly be obtained, and (5) the survey  was comprehensive and unbiased.

To assess the reliability of the data provided by agencies, we asked  officials to identify the databases and information sources they used to  respond to our survey questions and any limitations of the data they  provided. We then discussed with agency officials any identified data  limitations and, if unresolved issues remained, annotated the data, as  appropriate. We also identified responses that appeared to be  inconsistent or outliers, such as instances in which participants increased  as funds declined, and submitted them to agencies for verification.

From April to August 2018, we emailed the surveys to agency officials as  an attached Microsoft Excel form that they could return electronically. All  of the 45 surveys were completed and returned.

Because this was not a sample survey, it has no sampling errors.  However, the practical difficulties of conducting any survey may introduce  errors, commonly referred to as nonsampling errors. For example,  difficulties in interpreting a particular question, sources of information  available to respondents, or entering data into a database or analyzing  them can introduce unwanted variability into the survey results. We took  steps in developing the surveys, collecting the data, and analyzing them  to minimize such nonsampling error. For example, to minimize difficulties  interpreting a particular survey question, we incorporated the suggestions  from an independent reviewer to add explicit instructions for how to use  the pull-down menus and consistently phrased requests for information.

We reviewed the completed surveys and clarified information with agency  officials, as needed. We further reviewed the survey to ensure the  ordering of survey sections was appropriate and that the questions within  each section were clearly stated and easy to comprehend. To reduce  nonresponse, another source of nonsampling error, we sent out email  reminder messages to encourage officials to complete the survey. In  reviewing the survey data, we performed automated checks to identify  inappropriate answers. We further reviewed the data for missing or  ambiguous responses and followed up with agency officials when  necessary to clarify their responses. On the basis of our application of  recognized survey design practices and follow-up procedures, we  determined that the data were of sufficient quality for our purposes.

In terms of agency actions to manage overlap and fragmentation and to  detect/prevent duplication, we followed up with select agencies to better  understand what prompted the actions they took and the lessons they  learned from evaluating those efforts.

We did not conduct a legal analysis to confirm the various  characterizations of the programs in this report, such as information on  their budgetary obligations, services provided, target population, eligibility  criteria, or program goals. Instead, all such program information in this  report is based on our survey results, as confirmed by agency officials.  Further, we did not review agencies’ financial reporting systems or audit  the figures provided to us. We reviewed fiscal year 2019 budget  documents to determine if they could be used to verify data provided by  the agencies, but they did not consistently contain the program-level  details needed. Instead, to help mitigate reliability limitations that might  have accompanied agency reports, we asked agencies to identify the  data source of reported budgetary information and to list any data  limitations.

To address our second objective to identify areas of overlap among E&T  programs, we reviewed information reported by federal agency officials in  our survey. We used the definition of overlap established in GAO’s prior  work: overlap occurs when two or more programs provide at least one  similar service to a similar population. After reviewing survey responses  regarding the primary population groups served by the 43 programs and  the services they provided, we categorized programs according to the  primary population group served and identified programs within each  category that provided similar services. We did not focus on the effects of  potential duplication, which occurs when two or more agencies or  programs are engaged in the same activities or provide the same  services to the same beneficiaries. GAO has not previously identified  duplication in federal E&T programs, and our objectives in this  engagement focused on overlap and fragmentation previously identified  in these programs.

We categorized programs based on the type of program participant  served according to program objectives and program eligibility criteria  listed in the CFDA. Then, we verified these categorizations with agency  officials. In categorizing programs by target population, we used the  following categories: 1) general population, 2) dislocated workers or  trade-impacted workers, 3) migrant and seasonal farm workers, 4) Native  Americans (in this report, the term Native Americans refers to American  Indians and Native Hawaiians), 5) people with physical or mental  disabilities, 6) prisoners or ex-offenders, 7) refugees, 8) veterans or  transitioning servicemembers, 9) youth, and 10) older workers, women,  and unemployed and underemployed residents of solid and hazardous  waste-impacted neighborhoods (collectively, other). We also categorized  the VA’s Vocational Rehabilitation and Employment program with other  programs that target veterans, but noted that the program serves  veterans with a service-connected disability.

Review of Prior GAO Reports and Agency Funded Research

To address our second and third objectives, we also reviewed GAO  reports and agency funded research published since 2011. We used  these sources, in part, to illustrate effects of overlap and fragmentation  among E&T programs and provide examples of actions agencies have  taken to address our prior findings or recommendations. To address our  second research objective, we reviewed this literature to identify  examples of documented effects of overlap and fragmentation among  these programs, including positive effects (e.g., to fill a gap or  complement an existing program) and negative effects (e.g., inefficient  use of resources or confusion among individuals). To address our third  research objective, we conducted a literature search of agency- sponsored research on E&T programs and ultimately determined that six  of these studies were sufficiently rigorous and appropriately scoped to  include in our review. To identify studies on coordination and  collaboration of federally-funded programs, we conducted a literature  search through ProQuest. Our initial search terms included “federal  employment and training” and “coordination” or “collaboration,” “overlap,”  and “fragmentation”. We also reviewed these studies to assess the extent  to which agencies had evaluated actions to manage overlap and  fragmentation.

In addition, our survey asked program officials about whether an impact  study had been completed since 2011 to evaluate program performance  with regard to E&T activities and, if so, to provide a citation for at least  one of these studies. An impact study assesses the net effect of a  program by comparing program outcomes with an estimate of what would  have happened in the absence of the program. This type of study is  conducted when external factors are known to influence the program  outcomes, in order to isolate the program’s contribution to the  achievement of its objectives. Program officials provided 16 citations of  what they believed to be impact studies. Of the 16 cited studies, we  determined that 13 can accurately be described as impact studies. To  make this assessment, we reviewed the methodology section of each  study.

Appendix II: Update on the List of Federal Employment and Training Programs since GAO 2011 Review

The following table is a list of federal employment and training programs  using as a baseline programs identified in our most recent prior report  (GAO, Multiple Employment and Training Programs: Providing  Information on Colocating Services and Consolidating Administrative  Structures Could Promote Efficiencies, GAO-11-92 (Washington, D.C.:  Jan. 13, 2011)). We also reviewed the Catalog of Federal Domestic  Assistance (CFDA) to ensure that programs met our selection criteria and  to identify new programs. We did not conduct an independent legal  analysis to verify the information provided about the programs described  in this appendix, such as information on their status. For a description of  our methodology, see appendix I.

Appendix III: Change in Federal Employment and Training Program Obligations, Adjusted for Inflation in 2017 Dollars

Appendix IV: Change in Federal Employment and Training Obligations in Nominal Values

Appendix IV: Change in Federal Employment  and Training Obligations in Nominal Values *Program name from 2011 review was updated based on information confirmed by agency officials.

Appendix V: Estimated Number of Program Participants Who Received Federal Employment and Training Services

Appendix V: Estimated Number of Program  Participants Who Received Federal  Employment and Training Services *Program name from 2011 review was updated based on information confirmed by agency officials.

Appendix VI: Employment and Training Services Provided by Federal Programs Serving Selected Target Populations

Appendix VII: Agency-Funded Studies Examining Employment and Training Programs

Appendix VIII: Comments from the Department of Labor

Appendix IX: Comments from the Department of Veterans Affairs

Appendix X: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Betty Ward-Zukerman (Assistant  Director), Sheranda Campbell (Analyst-in-Charge), Camille Henley, Joel  Marus, David Perkins, and Jill Yost made key contributions to this report.  Also contributing to this report were Amy Anderson, Stephen Betsock,  Caitlin Croake, Alex Galuten, Kristen Jones, Benjamin Licht, Mimi  Nguyen, James Bennett, David Blanding, Elizabeth Mixon, Steven  Putansu, Monica Savoy, Paul Schearf, Ardith Spence, Almeta Spencer,  Kathleen van Gelder, and John Yee.

Federal agencies administer employment and training programs to help job seekers find and get work. While we identified 47 such programs in 2011, there are 43 now. Spending for these programs has decreased as well.

Multiple programs provide similar services, often to similar groups of people. While agencies have tried to manage this overlap by trying to coordinate their programs, they generally don't know whether their efforts are working.

We recommended that the Department of Labor develop a strategic plan for evaluating these programs that includes assessing what federal agencies are doing to coordinate them.