Job Corps: DOL Could Enhance Safety and Security at Centers with Consistent Monitoring and Comprehensive Planning

Deficiencies identified in multiple DOL Inspector General audits since 2009 and two student deaths in 2015 have raised concerns regarding the safety and security of Job Corps students. GAO was asked to review safety and security of students in the Job Corps program. GAO's June 2017 testimony summarized preliminary observations. This report further examines (1) the number and types of reported safety and security incidents involving Job Corps students; (2) student perceptions of their safety at Job Corps centers; and (3) the extent to which ETA has taken steps to address safety and security at Job Corps centers.

GAO analyzed ETA's reported incident data for Job Corps centers from July 1, 2016, through June 30, 2017. GAO also analyzed ETA's student survey data from the same period, reviewed relevant documentation, and interviewed ETA officials at its national office and all six regions. GAO also visited two Job Corps centers that had different operators and at least 100 recent incidents. These two centers are not generalizable to all centers.

Job Corps centers reported 13,673 safety and security incidents involving students from July 2016 to June 2017, according to GAO's analysis of the Department of Labor's (DOL) Employment and Training Administration's (ETA) data. Most reported incidents occurred onsite and involved recently enrolled male students under age 20. During that time, the program served about 79,000 students at 125 Job Corps centers, according to ETA officials. ETA's Office of Job Corps administers the program, which is the nation's largest residential, educational, and career and technical training program for low-income youth generally between the ages of 16 and 24. Drug-related incidents and assaults accounted for 48 percent of all reported incidents (see fig.).

Students generally felt safe at Job Corps centers, yet fewer felt safe in some situations, based on GAO's analysis of ETA's September 2016 and March 2017 Job Corps student satisfaction surveys. At least 70 percent of students reported that they felt safe on half of the 12 safety-related questions in the 49 question survey about their experiences in the Job Corps program; but fewer students reported feeling safe when asked if they were made to feel unimportant or if they heard students threaten each other. ETA plans to administer a new survey nationally by January 2019 that focuses solely on safety and security issues.

ETA has initiated several actions to improve safety and security at Job Corps centers, but insufficient guidance for its monitoring staff and absence of a comprehensive plan for safety and security may put the success of these actions at risk. Among its actions, ETA adopted a new risk-based monitoring strategy to identify emerging problems at the centers. Officials GAO spoke with in five of ETA's regional offices said that the new strategy has improved monitoring, but that more guidance on how to interpret and apply safety and security policies is needed to promote consistency across centers. Also, ETA lacks a comprehensive plan linking its new efforts to an overall safety and security framework. ETA officials told GAO that limited staff capacity and lack of expertise have hindered their efforts in developing such a plan. Without a comprehensive plan, ETA runs the risk that its new efforts will not be successful.

GAO is making three recommendations to DOL, including that ETA develop additional monitoring guidance and a comprehensive plan for safety and security. DOL agreed with GAO's three recommendations.

Job Corps is the nation’s largest residential, educational, and career and  technical training program for low-income youth generally between the  ages of 16 and 24. The Job Corps program is administered by the Office  of Job Corps in the Department of Labor’s (DOL) Employment and  Training Administration (ETA).The program enrolls approximately 50,000  new students each year at 123 Job Corps centers nationwide and for  fiscal year 2017 was appropriated about $1.7 billion.

For almost a decade, concerns have been raised regarding the safety  and security of Job Corps students. For example, DOL Office of Inspector  General (OIG) audits in 2009, 2010, 2015, and 2017 found that the Office  of Job Corps did not properly address serious incidents related to student  safety and security because of deficiencies in its oversight of program  disciplinary policies. As a result, the DOL OIG included providing a safe  learning environment at Job Corps centers among the department’s top  management challenges in November 2017. Additional concerns were  raised regarding the safety and security of students following the deaths  of two students at two separate Job Corps centers in 2015.

For a June 2017 hearing, you asked us to provide preliminary  observations on the safety and security of students in the Job Corps  program. Our preliminary results found that Job Corps centers reported  49,836 safety and security incidents of various types that occurred both  onsite and offsite between January 1, 2007, and June 30, 2016. During  this time period, approximately 539,000 students were enrolled in the  program, according to ETA officials. Beginning July 1, 2016, ETA  implemented policy changes that impacted the categorization and number  of reportable incidents. As such, incident data after July 1, 2016—the  focus of this report—are not comparable with the earlier incident data  presented in our June 2017 testimony. In addition, we reported in our  testimony that from March 2007 through March 2017, students generally  reported feeling safe at their Job Corps center, but reported feeling less  safe in certain situations such as when they witnessed physical fights and  heard threats between students.

This report examines (1) what is known about the number and types of  reported incidents involving the safety and security of Job Corps students  in program year 2016; (2) what is known about student perceptions of  safety and security at Job Corps centers, and what steps, if any, is ETA  taking to improve the survey used to collect this information; and (3) the  extent to which ETA has taken steps to address safety and security at  Job Corps centers.

To address our first objective, we analyzed ETA’s incident data for  program year 2016, the most recent year for which Job Corps data were  available. ETA captures these data in its Significant Incident Reporting  System (SIRS). We assessed the reliability of SIRS data by reviewing  relevant agency documentation about the data and the system that  produced them and interviewing knowledgeable ETA and DOL OIG  officials. We determined that the data were sufficiently reliable to report  the minimum number of incidents that occurred in program year 2016. It  is likely that the actual number of incidents was greater than the number reported in SIRS because the information is reported by Job Corps  centers, and the DOL OIG previously found instances of underreporting  by a non-generalizable sample of center operators. While ETA has  recently taken steps to improve center reporting of significant incidents,  according to DOL OIG officials, it is too early to determine if these steps  have resolved the OIG’s concerns regarding center underreporting. The  incident categories and definitions in this report are taken directly from  ETA documents and represent how ETA categorizes these incidents. We  did not assess these categories and definitions.

To address our second objective, we analyzed ETA’s national student  satisfaction survey data for program year 2016, the most recent year for  which data were available. The surveys were administered to students in  September 2016 and March 2017, and each had a response rate of about  90 percent. The semi-annual survey on various aspects of the Job Corps  program included 12 questions about students’ perceptions of safety at  their center. We assessed the reliability of the data by reviewing relevant  agency documentation about the data and the system that produced them  and interviewing knowledgeable ETA officials, among other steps. We  determined that the student survey data were sufficiently reliable for our  purposes.

To address our third objective, we reviewed documentation on ETA’s  recent actions to improve center safety and Job Corps policies for  monitoring center operators. We also used criteria to assess whether  ETA’s documentation of its recent and planned actions constituted a  comprehensive plan. These criteria included leading practices for  comprehensive planning and Standards for Internal Control in the Federal  Government. We selected these criteria because they included a  process for developing a comprehensive plan and specify the content of  such plans, which we determined to be most relevant, given our initial  understanding that ETA was early in its planning process.

To address all three objectives, we reviewed agency policies and  procedures and interviewed ETA national and regional officials. We also  conducted site visits to two Job Corps centers to interview center staff  and students about various safety and security issues. The two selected  centers were within geographical proximity to Washington, D.C., operated  by different contractors, and had over 100 reported incidents of various  types in program year 2016. While these two site visits are not  generalizable to all Job Corps centers, they provide examples of student  and staff experiences with safety and security. Additional details on our  methodology can be found in appendix I.

We conducted this performance audit from April 2017 to June 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Job Corps Eligibility Criteria and Program Services

To be eligible for the Job Corps program, an individual must generally be  16 to 24 years old at the time of enrollment; be low income; and have  an additional barrier to education and employment, such as being  homeless, a high school dropout, or in foster care. See table 1 for  characteristics of students served by Job Corps during program year  2016.

Once enrolled in the program, youth are assigned to a specific Job Corps  center, usually one located nearest their home and which offers a job  training program of interest. The vast majority of students live at Job  Corps centers in a residential setting, while the remaining students  commute daily from their homes to their respective centers. This  residential structure is unique among federal youth programs and enables  Job Corps to provide a comprehensive array of services, including  housing, meals, clothing, academic instruction, and job training. In  program year 2016, about 16,000 students received a high school  equivalency and about 28,000 students completed a career technical  training program, according to ETA officials.

ETA administers Job Corps’ 123 centers through its national Office of Job  Corps under the leadership of a national director and a field network of six  regional offices located in Atlanta, Boston, Chicago, Dallas, Philadelphia,  and San Francisco (see fig. 1). Job Corps is operated primarily through  contracts, which according to ETA officials, is unique among ETA’s  employment and training programs (other such programs are generally  operated through grants to states). Among the 123 centers, 98 are  operated under contracts with large and small businesses, nonprofit  organizations, and Native American tribes. The remaining 25 centers  (called Civilian Conservation Centers) are operated by the U.S.  Department of Agriculture’s (USDA) Forest Service through an  interagency agreement with DOL. Job Corps center contractors and the  USDA Forest Service employ center staff who provide program services  to students. The President’s fiscal year 2019 budget seeks to end  USDA’s role in the program, thereby unifying responsibility under DOL.  The Administration reported that it was proposing this action because  workforce development is not a core mission of USDA, and the 25  centers it operates are overrepresented in the lowest performing cohort of  centers. According to ETA officials, the Office of Job Corps has oversight  and monitoring responsibility to ensure that center operators follow Job  Corps’ Policy and Requirements Handbook, including the safety and  security provisions. Job Corps regional office staff are largely responsible  for these duties.

Requirements for Job Corps Centers Related to Incident Reporting

Job Corps’ Policy and Requirements Handbook requires centers to report  certain significant incidents to the national Office of Job Corps and to  regional offices using SIRS. Centers are required to report numerous  categories of incidents, including assaults, alcohol and drug-related  incidents, and serious illnesses and injuries (see appendix II for  definitions of these categories of incidents). Within the Policy and  Requirements Handbook, ETA establishes student standards of conduct  that specify actions centers must take in response to certain incidents.  In some cases, the incident categories in SIRS are related to the specific  infractions defined in the Policy and Requirements Handbook, which are  classified according to their level of severity. Level I infractions are the  most serious, and includes infractions such as arrest for a felony or  violent misdemeanor or possession of a weapon, and are required to be  reported in SIRS. Level II includes infractions such as possession of a  potentially dangerous item like a box cutter, or arrest for a non-violent  misdemeanor. The majority of these infractions are required to be  reported in SIRS. Minor infractions—the lowest level—include failure to  follow center rules, and are not required to be reported in SIRS.

Centers must report incidents involving both Job Corps students and  staff, and incidents that occur onsite at centers as well as those that occur  at offsite locations. According to ETA officials, the agency and its center  operators must take steps to protect the safety and security of Job Corps  students when students are under Job Corps supervision. Students are  under Job Corps supervision when they are onsite at Job Corps centers  and when they are offsite and engaged in center-sponsored activities,  such as work-based learning or community service. According to ETA  officials, the agency and its contractors are not responsible for protecting  the safety and security of Job Corps students when students are offsite  and not under Job Corps supervision, such as when students are at home  on leave. However, when offsite safety and security incidents of any type  occur, Job Corps center operators are responsible for enforcing the  student conduct policy. For example, if a student is arrested for a felony  offsite while not under Job Corps supervision, the arrest may result in a  Level I infraction and dismissal from the program.

Since 2002, ETA used its student satisfaction survey to periodically obtain  views from enrolled Job Corps students on various aspects of the  program, including career development services, interactions between  students and staff, access to alcohol and drugs, and overall satisfaction  with the program. The survey of 49 questions has remained the same  over time and included 12 questions on students’ perceptions of safety  and security at centers.

ETA used the responses to the 12 safety-related survey questions to  calculate a center safety rating, which represented the percentage of Job  Corps students who reported feeling safe at each center, as well as a  national safety rating, which represented the percentage of Job Corps  students who reported feeling safe nationwide. ETA officials said they  used these ratings to assess students’ perceptions of safety at individual  centers and nationwide, to monitor and evaluate center operators, and to  determine whether ETA needed to take action to better address students’  safety and security concerns. In 2018, ETA will pilot a stand-alone survey  for safety related topics and remove the safety questions from the student  satisfaction survey.

Job Corps Centers Reported Nearly 14,000 Incidents of Various Types during Program Year 2016, Which Mainly Occurred Onsite and Involved Recently Enrolled Males under Age 20

Almost Half of the Reported Onsite and Offsite Incidents Involved Drugs or Assaults

Our analysis of ETA’s data from the Significant Incident Reporting System  (SIRS) showed that Job Corps centers reported 13,673 safety and  security incidents involving students, including those that occurred both  onsite and offsite, in program year 2016. During this time period (July 1,  2016, through June 30, 2017), approximately 79,000 students were  served by the program, according to ETA officials. Drug-related  incidents (29 percent) and assaults (19 percent) accounted for 48 percent  of all reported incidents involving students. The remaining 52 percent of  reported incidents involving students included breaches of security and  safety (12 percent), alcohol-related incidents (6 percent), serious illness  and injury (6 percent), theft or damage to property (5 percent), danger to  self or others (5 percent), and all other types of incidents (18 percent)  (see fig. 2). According to ETA officials, about half of the 3,926 drug- related incidents are due to positive drug test results among students that  are administered drug tests about 40 days after entering the program.

We found that about 20 percent of reported onsite and offsite incidents in  program year 2016 were of a violent nature, which we define as  homicides, sexual assaults, and assaults. There were two reported  homicide incidents in program year 2016 and both occurred while  students were offsite and not under Job Corps supervision. Also,  centers reported 177 sexual assaults and 2,593 assaults involving  students during program year 2016. For each reported sexual assault and  assault, SIRS provides an additional description of the incident (see table  2).

In our June 2017 testimony, we stated that 49,836 onsite and offsite  safety and security incidents of various types were reported by Job Corps  centers between January 1, 2007, and June 30, 2016, based on our  preliminary analysis of ETA’s SIRS data. We cannot compare our  analysis of safety and security incidents in our June 2017 testimony to the  analysis contained in this report for program year 2016 due to a policy  change by ETA beginning July 1, 2016, which affected the categorization  and number of reportable incidents. Specifically, ETA changed the way  some incidents are defined, and required that some incidents be reported  in SIRS that previously had no such requirement. Anecdotally, officials  from one ETA regional office and two Job Corps centers that we visited  said that the number of reported incidents has increased since July 1,  2016, due to these changes. In its December 2017 report, the DOL OIG  compared the number of safety and security incidents reported to the OIG  for the same 8-month periods in 2016 and 2017 and found an increase of  134 percent. According to the DOL OIG, this increase is likely due to  more accurate incident reporting as a result of the recent policy change.  In addition, the DOL OIG said an actual increase in incidents is also  possible.

Most Reported Incidents Occurred Onsite, but Arrests and Deaths Most Frequently Occurred Offsite While Students Were Not Under Job Corps Supervision

Our analysis of SIRS data found that in program year 2016, 90 percent of  the 13,673 reported safety and security incidents involving students  occurred onsite at Job Corps centers, and 10 percent occurred at offsite  locations (see fig. 3). For example, 99 percent of drug-related incidents,  96 percent of assault incidents, and 84 percent of alcohol-related  incidents occurred onsite. While most reported incidents occurred onsite,  our analysis showed that the majority of reported arrests, deaths, and  motor vehicle accidents occurred offsite. For example, of the 21 student  deaths,18 occurred at offsite locations and 3 occurred onsite. In our  June 2017 testimony, we reported that from January 1, 2007, through  June 30, 2016, 76 percent of the reported safety and security incidents  occurred onsite at Job Corps centers, and 24 percent occurred at offsite  locations based on our preliminary analysis of ETA’s SIRS data.  However, as previously noted, that analysis is not comparable to the  analysis in this report for program year 2016 due to ETA’s July 1, 2016,  policy change that impacted the categorization and number of reportable  incidents.

We analyzed the 1,406 incidents of 13,673 total reported incidents that  were reported to have taken place offsite in program year 2016 to  determine if the students involved were on duty (i.e., under Job Corps  supervision) or off duty (i.e., not under Job Corps supervision). We  found that for offsite incidents, similar percentages of student victims and  perpetrators were on duty and off duty. Specifically, we found that 50  percent of student victims were on duty, 44 percent were off duty, and we  were unable to determine the duty status of 6 percent. For student  perpetrators, we found that 45 percent of students were on duty, 45  percent were off duty, and we were unable to determine the duty status of  10 percent. Some types of reported incidents occurred more frequently  when students were offsite and off duty. For example, of the reported  arrest incidents that occurred offsite, 76 percent of student perpetrators  were off duty. Of the reported death-related incidents that occurred  offsite, student duty status was reported as off duty for 16 of 18 incidents.

We were unable to determine the duty status for all students involved in  offsite incidents due to inconsistencies in ETA’s data. Of the 1,406 offsite  incidents reported in SIRS, there were 178 instances in which a student’s  duty status location conflicted with the incident location. For example, the  student’s duty status was listed as onsite and on duty, but the incident  location was listed as offsite. We asked ETA officials why these  inconsistencies existed and they were unable to explain all instances in  which these inconsistencies occurred. ETA officials did state, however,  that these inconsistences can sometimes occur when centers enter  information in SIRS based on the student’s duty status at the time the  incident report is completed instead of the student’s duty status at the  time the incident occurred. Due to this data limitation, we were unable to  determine if the 178 students involved in those incidents were on duty or  off duty.

Student Victims and Perpetrators Most Often Were Recently Enrolled Males under Age 20, Reflective of the Job Corps Population

We analyzed SIRS data to determine the characteristics of students  involved in reported safety and security incidents and found that about  17,000 students were reported as victims or perpetrators of all onsite and  offsite incidents in program year 2016. The total number of students  reported as victims or perpetrators is 22 percent of the students served in  program year 2016. The number of student victims and perpetrators  varied across incident types (see fig. 4).

In program year 2016, we found that about 5,000 students (6 percent of  students served) were reported as victims of various types of onsite and  offsite incidents. We separately examined the gender, age, and  enrollment time of reported student victims and found that for all reported  incidents the majority of student victims were male, under age 20, and  enrolled in Job Corps for less than 4 months (see fig. 5). These  characteristics are somewhat similar to the overall Job Corps student  population, which is primarily male and under age 20, as previously  noted. For example, 65 percent of reported assault victims and 73  percent of reported theft victims were male. However, the number of  female victims exceeded the number of male victims within some  reported incident categories, such as sexual assault, inappropriate sexual  behavior, and missing persons. Students under age 20 were victims of 67  percent of reported assault incidents and 63 percent of danger to self or  others incidents. According to ETA officials, 18 percent of students served  in program year 2016 were enrolled for less than 4 months; however,  across all reported incidents 56 percent of student victims were enrolled  for less than 4 months. For example, about 60 percent of student victims  of reported assault and danger to self or other incidents were enrolled in  Job Corps for less than 4 months.

Our analysis of SIRS data shows that about 13,000 students (17 percent  of students served) were reported as perpetrators of various types of  onsite and offsite incidents in program year 2016. The most commonly  reported incidents—drug-related and assaults—also had the highest  numbers of student perpetrators. We found that 6 percent and 5 percent  of students served in program year 2016 were perpetrators of reported  drug-related and assault incidents, respectively. Similar to our analysis of  student victims, we separately examined student characteristics and  found that the majority of reported student perpetrators of all reported  incidents were male, under age 20, and enrolled in Job Corps for less  than 4 months (see fig. 6).

Students Generally Reported Feeling Safe; ETA Plans to Create a New, Expanded Survey

Most Students Reported Feeling Safe, but Fewer Reported Feeling Safe on Selected Questions

Our analysis of ETA’s student satisfaction survey data from program year  2016 showed that while students generally reported feeling safe at Job  Corps centers, a smaller proportion reported feeling safe in certain  situations. ETA considers students to feel safe if they provide certain  responses to each of the 12 safety-related survey questions, some of  which are phrased as statements. For example, if a student provided a  response of “mostly false” or “very false” to the statement “I thought about  leaving Job Corps because of a personal safety concern,” that student  would be counted as feeling safe on that survey question. On 6 of the 12  safety-related survey questions in program year 2016, at least 70 percent  of responding students indicated that they felt safe (see table 3). For  example, 74 percent of students responded that they did not ever or in  the last month carry a weapon, and 83 percent of students responded  that it was very or mostly true that a student would be terminated from  Job Corps for having a weapon at the center. These are responses that  ETA considered to indicate feeling safe. At the two centers we visited,  students that we interviewed said that they felt safe onsite at their center.  For example, students at one center said that they felt safe because  absolutely no weapons, fighting, or drugs were allowed at the center.

A smaller number of students reported feeling safe on questions that  dealt with hearing threats or hearing things from other students that made  them feel unimportant. For example, 36 percent of students reported they  had not ever or in the last month heard a student threaten another  student at the center, which is considered safe according to ETA policy.  Meanwhile, 49 percent reported that they had heard a student threaten  another student at least once in the last month, and ETA considered  these responses to indicate that students felt unsafe. Another 15  percent chose “don’t know / does not apply.” On another question, 53  percent of students reported that other students had not ever or in the last  month said things that made them feel like they were not important, which  ETA considered as feeling safe. Yet 30 percent reported that others made  them feel unimportant at least once in the last month—which ETA  considered as feeling unsafe—and 17 percent chose “don’t know / does  not apply.”

In response to a question about the student conduct policy, 35 percent of  students indicated that the policy was not applied equally to all students.  At the two centers we visited, students that we interviewed had varying  views on applying the student conduct policy. Students from one center  said that staff have applied the policy in a fair way. Yet at another center,  students told us that they have occasionally perceived that staff have not  applied the student conduct policy fairly. They mentioned that they were  aware of favoritism in a few recent incidents when staff applied the  policy’s disciplinary consequences for certain students but not others. For  example, they said that a student they perceived as the perpetrator  remained in Job Corps while a student they perceived as innocent was  dismissed.

Our June 2017 testimony contained similar observations about students’  perceptions of their safety, with students generally reporting that they felt  safe at their Job Corps centers. For example, most students reported  feeling safe because a student found with a weapon at the center would  be terminated. In that testimony, we also noted that students reported  feeling less safe on such questions as hearing threats or applying the  student conduct policy.

In addition to the 12 safety-related questions, we examined data on the 2  questions about access to alcohol or drugs, and found that almost two- thirds of survey respondents said that it was mostly or very false that they  could access alcohol or drugs at their Job Corps center. Although a  large number of reported incidents in program year 2016 involved drugs  or alcohol, less than 15 percent of survey respondents said that it was  mostly or very true that they could access alcohol or drugs at their Job  Corps center.

National Measures of Safety and Security Have Been Developed

Based on students’ responses to the 12 safety-related questions, ETA  determined that 88 percent of students indicated that they felt safe in  program year 2016. ETA calculated its national measure of safety— referred to as a safety rating—to summarize and track students’  perceptions of their safety and to determine the need for additional action,  as noted previously. Similarly, it calculated a safety measure for each  center.

However, we calculated a national measure differently and found that an  average of 73 percent of students reported feeling safe in program year  2016. Our national measure reflected the average of how safe each  student felt on the 12 safety-related survey questions. We estimated  that one key difference accounted for about 11 of the 15 percentage  points between our and ETA’s measure. (See table 7 in appendix I.)  Specifically, we calculated our measure based on a numeric average for  each student without rounding. For example, if a student answered all 12  safety questions with 6 responses that he felt safe and another 6 that he  felt unsafe, we counted this student as half safe (0.5). Meanwhile, ETA  rounded the average to either safe or unsafe, so that ETA counted a  student with 6 safe responses and 6 unsafe responses as feeling safe.

In addition to differences in calculations, we developed our own national  measure of safety because it is important to assess and track students’  perceptions for the program as a whole, as ETA has noted. Also, a  national measure facilitates analysis of groups of students, such as male  or female students or younger or older students, as described below.

We examined whether our national measure differed by age, gender, time  in program, center size, or operator type and found statistically significant  and meaningful differences in our national measure by students’ length of  time in the program. In particular, an average of 78 percent of students in  the program for less than 4 months responded that they felt safe,  compared to an average of 71 percent for students in the program for at  least 4 months. According to ETA officials, differences in responses  based on length of time in the program may relate to new students being  less aware about life at the center because they begin the program with  other newly arrived students for up to 2 months. For example, ETA  officials said that new students may live in a dormitory specifically for new  students. Thus, they are not yet fully integrated into the larger student  body. Although differences were also statistically significant between age  groups, center size, and operator type, such differences were not  meaningful in a practical manner (i.e., around 3 percentage points or  less). Differences in our national measure by gender were not statistically  significant.

When we analyzed the survey’s separate question about overall  satisfaction with Job Corps, we found that students who reported they  were satisfied with the Job Corps program responded that they felt safer  than students who were not satisfied. In program year 2016, about two- thirds of students said it was very or mostly true that they would  recommend Job Corps to a friend, which ETA uses to gauge overall  satisfaction with the program. Of the 65 percent of students who would  recommend Job Corps to a friend, 79 percent said they felt safe. Of the  11 percent of students who would not recommend Job Corps to a friend,  52 percent felt safe.

ETA’s New Web-based Survey Is Designed to Be More Timely and Detailed

ETA officials said that the agency is creating a new expanded safety  survey to improve upon the prior survey. With Job Corps’ heightened  attention to safety and security, the new survey—the Student Safety  Assessment—is focused solely on safety and security issues and is  designed to provide more timely and more detailed information.

More timely information. ETA plans to administer the new safety  survey monthly to a random sample of students rather than twice per  year to all enrolled students. Also, it will be web-based, rather than the  current paper-based survey. As a result, ETA officials said that they  will receive more timely information from students because it will take  less time to administer the survey and analyze the responses.

More detailed information. The number of questions about center  safety will increase from 12 to about 50—pending finalization of the  survey—which is about the same number of questions on the current  student satisfaction survey. For example, the new questions will ask  about sexual assaults and harassment or the types of drugs bought or  used at the center, which were not topics covered by the prior survey.

ETA continues to work with its contractor with survey expertise to  develop, test, and administer the new survey in 2018, according to ETA  officials. To develop the new survey, ETA and its contractor have  considered, incorporated, and revised questions from other existing  surveys. For example, they have drawn from safety surveys of teenage  students and postsecondary students. ETA plans to continue developing  and refining the survey and its administration in 2018, including  conducting monthly pilots from January to June 2018, assessing  response rates, and developing a new way to calculate national and  center-level safety measures. Additionally, ETA officials said that, in  2018, they will seek to obtain comments and approval on the survey from  the Office of Management and Budget. ETA officials told us that they plan  to administer the new survey nationally by January 2019. As ETA refines  and administers this new survey, officials told us they plan to develop a  new way to measure student safety based on the more detailed survey.

ETA Initiated Multiple Actions to Improve Center Safety and Security, but the New Monitoring Strategy Was Implemented Inconsistently and ETA Lacks a Comprehensive Plan

ETA Initiated Multiple Actions to Improve Center Safety and Security

In 2014, ETA launched multiple actions to improve safety and security at  Job Corps centers in response to DOL OIG recommendations (see table  4). For example, in 2015 the DOL OIG found ETA’s oversight of Job  Corps centers ineffective, in part, because ETA’s student conduct policy  excluded some violent offenses. As a result, ETA revised its student  conduct policy by elevating several infractions previously classified as  Level II to Level I (the most severe) and by adding several new categories  of reportable incidents. Under the revised student conduct policy, assault,  a Level I infraction, now includes fighting, which was previously a Level II  infraction. In addition, the DOL OIG found that ETA did not monitor  centers regularly enough to ensure center consistency in administering  Job Corps disciplinary policies. In response, ETA implemented a risk- based monitoring strategy that identifies potential safety and security  issues before they occur.

Staff from five ETA regional offices and at one Job Corps center we  visited said that ETA’s actions overall helped to improve center safety and  security. For example, staff from five regional offices said that the  changes to the student conduct policy that were implemented in July  2016 clearly describe the penalties for infractions and eliminate grey  areas that previously allowed center staff to use their professional  judgement. Staff from four regional offices also said these changes  resulted in tradeoffs that reduced center staff discretion in imposing  penalties. In addition, at one center we visited, the Director of Safety and  Security told us he updated the center’s security-related standard  operating procedures in response to ETA’s guidance. ETA’s guidance  was part of the 2017 updates to the Policy and Requirements Handbook  in response to DOL OIG concerns about reporting potentially serious  criminal misconduct to law enforcement.

ETA Officials Reported That Some New Actions Improved Center Monitoring, but That Actions Were Inconsistently Implemented and May Create Reporting Overlaps

ETA national officials said that the new risk-based monitoring strategy  has improved center monitoring because it has allowed them to more  effectively direct resources to areas of greatest need. Officials in five ETA  regional offices agreed that the new strategy improved their ability to  monitor centers. The new monitoring strategy shifted the focus from  addressing problems after they have occurred to a data-driven strategy  that tracks center performance and identifies emerging problems. This  strategy provides ETA and center operators an opportunity to address  problems before they occur, according to ETA national officials. For  example, the new monitoring strategy features new tools, including the  Risk Management Dashboard. The dashboard is a summary analysis tool  that conducts trend analysis using center data and allows regional staff to  engage in targeted interventions at centers with potential safety and  security concerns. In addition, under the new monitoring strategy, instead  of only conducting scheduled monitoring visits to a center at set times,  regional staff conduct unannounced visits based on data indicating a  decline in center performance or other triggers. See appendix VI for  additional information on the new monitoring strategy.

Although the new risk-based monitoring strategy has improved center  monitoring, it is not consistently implemented across regional offices,  according to ETA national officials. They told us that similar problems  identified at centers may be treated with different levels of focus or  intensity from one region to another. In addition, national and regional  officials told us that regional office staff have relied on professional  judgment to determine the appropriate response to centers that may be at  risk of noncompliance with safety and security policies, which could lead  to inconsistencies. For example, when problems are identified at centers,  the type of assessment to conduct is left to regional office staff discretion.  As a result, staff in one region may decide that the most comprehensive  assessment, the Regional Office Center Assessment, is needed, while  another region’s staff would select a targeted assessment, which is more  limited in scope. ETA national officials said that although each  determination could be justified based on resource constraints and  competing priorities, they would like to increase implementation  consistency in this area.

To address regional inconsistencies, ETA national and regional office  staff said that guidance in the form of standard operating procedures  (SOP) would be helpful. These procedures would promote consistency in  how policies are interpreted and applied and would help ensure that  centers are held to the same standards, according to ETA national  officials. For example, SOPs could specify which type of assessment to  conduct in response to specific problems identified at centers. Internal  control standards state that managers should document in policies each  unit’s responsibility for an operational process.

Regional office staff said that they previously had a helpful tool, the  Program Assessment Guide, that linked policies in the Policy and  Requirements Handbook to the monitoring assessment process.  Regional office staff said they used the Program Assessment Guide to  prepare for center monitoring visits and it was a helpful training tool for  new staff. Our review of ETA documentation found that the Program  Assessment Guide included specific questions to ask center staff about  how they meet safety and security requirements and suggested where to  look for information to determine center compliance with policies.  However, the Program Assessment Guide, which has not been updated  since 2013, does not include recent changes to the Policy and  Requirements Handbook, such as the updated student conduct policy.  ETA national officials told us that limited staffing has made it difficult to  update the Program Assessment Guide as frequently as changes are  made to the Policy and Requirements Handbook.

In February 2018, ETA national officials told us they plan to issue a  variety of SOPs related to monitoring center safety and security issues  (see table 5). ETA officials initially said these SOPs would be completed  in August or November 2018 and later revised its plans with a goal of  completing all SOPs by August 2018. However, in August 2017, ETA  officials had told the DOL OIG that these SOPs would be completed in  the March to July 2018 timeframe. ETA officials said that a staffing  shortage in the Office of Job Corps’ Division of Regional Operations and  Program Integrity delayed development of the SOPs. This Division— established in 2015 to coordinate regional operations and strengthen  communications and quality assurance—includes eight staff positions;  however, as of January 2018, the Division has two staff members on  board. ETA officials said that they have not yet received departmental  approval to fill the six vacant positions in the Division.

Given this uncertainty, it is questionable whether ETA’s revised  timeframes will be met. Without SOPs or other relevant guidance, ETA  cannot ensure that monitoring for center safety and security will be  carried out uniformly across the program. As a result, centers may be  held to different standards, and the program may not achieve its center  safety and security goals.

In addition to inconsistencies in monitoring and a lack of sufficient  guidance, staff in all six regional offices told us that components of ETA’s  risk-based monitoring strategy created reporting overlaps. As part of the  new monitoring strategy, regional staff have additional reports that they  complete—such as the Risk Management Dashboard Action report and  Corrective Action Tracker—about potential safety and security problems  or actual violations found at centers. Some regional staff said the desk  monitoring report includes similar information to the Risk Management  Dashboard and Corrective Action Tracker reports, which regional offices  submit to the ETA national office. Staff in one regional office said that  they enter the same information about the status of center safety and  security violations multiple times on the Corrective Action Tracker  because the time between reporting periods is too short to allow for  meaningful action to be taken. Staff from four regional offices said  completing duplicative reports reduces time that could be used to conduct  additional center monitoring, such as onsite visits, or to perform other key  duties.

ETA national officials disagreed that overlap exists among monitoring  reports. They said that although reports may appear to overlap, the  reports are complementary and not duplicative, and are used at different  points in the monitoring process (see fig. 7 for an overview of ETA’s  monitoring process). For example, ETA national staff told us that desk  monitoring reports are primarily used by regional staff at the beginning of  the monitoring process to identify potential problems and are not  substantially reviewed by the national office. ETA national officials also  said that the Risk Management Dashboard report is used at the beginning  of the monitoring process to identify problems, whereas the Corrective  Action Tracker is used later in the process after violations have been  identified and corrective actions have been planned to bring the center  back into compliance. In addition, ETA national officials also noted that  regional staff are not asked to complete all reports every month. For  example, regional staff complete a Risk Management Dashboard Action  report only for those centers with potential safety and security concerns.

We compared the information included in five monitoring reports—the  Center Culture and Safety Assessment, Corrective Action Tracker, Desk  Audit, Regional Office Center Assessment, and Risk Management  Dashboard Action report—and found opportunities for streamlining. For  example, we found that the Center Culture and Safety Assessment,  Corrective Action Tracker, and Regional Office Center Assessment, all  include a narrative description of the violations identified by regional staff  categorized according to the corresponding requirement in the Policy and  Requirements Handbook. In addition, ETA regional office staff said the  Corrective Action Tracker, a Microsoft Excel spreadsheet, is cumbersome  to use and within the spreadsheet they attach and submit additional  documentation. ETA national officials agreed that streamlining or  automating monitoring tools would be helpful for its regional staff, along  with additional training to help staff understand the different reports and  how to write the required narratives. ETA national officials also told us  that they did not systematically review existing reports before creating  additional ones for the new risk-based monitoring process. Officials said  they have lacked the resources to make some improvements that could  reduce the time regional office staff spend on reporting.

Standards for internal control state that managers should identify the  organizational level at which the information is needed, the degree of  specificity needed, and state that managers should review information  needs as an on-going process. Streamlining or automating reporting  requirements can help centralize documentation relevant to monitoring  center safety and security, possibly eliminate seemingly duplicative  reporting requirements, and help regional staff manage their workloads.

ETA Lacks a Comprehensive Plan to Link Its Various Efforts to Improve Center Safety and Security

While ETA initiated multiple actions to address various safety and security  issues, the agency does not have a comprehensive plan to improve  center safety and security. A comprehensive plan describes the  organization’s long-term goals, its strategy and timelines for achieving  those goals, and the measures that will be used to assess its  performance in relationship to its goals. It can also guide decision-making  to achieve desired outcomes, including the priority with which to  implement these efforts. ETA officials told us that although they do not  have a single document that reflects a formal comprehensive plan, they  have employed a comprehensive approach to improve center safety and  security. However, in prior work, GAO established the importance of  comprehensive planning to ensure agencies effectively execute their  missions and are accountable for results.

GAO has also identified leading practices that help ensure organizations  achieve their objectives. These leading practices include developing  goals, strategies to achieve goals, plans to assess progress toward goals,  and leadership and stakeholder involvement in plan development (see  table 6).

ETA officials agreed that a comprehensive plan is needed, but told us that  limited staff capacity and lack of expertise have hindered their ability to  produce a comprehensive plan. In particular, the Division of Regional  Operations and Program Integrity would have a role in developing the  agency’s comprehensive plan. As previously mentioned, ETA officials told  us that they did not have approval to fill the six vacant positions in the  Division. With only two of the eight positions filled, ETA officials said that  they prioritized correcting the deficiencies identified by the DOL OIG and  responding to immediate safety and security concerns. ETA officials told  us they plan to produce a comprehensive plan when they have secured  the staff to do so. However, at this time, ETA does not have a specific  timeframe for producing such a plan.

When the agency begins developing a comprehensive plan, it could  consider using the leading practices outlined above and drawing on the  expertise of the government-wide Performance Improvement Council. In  the absence of a comprehensive plan for safety and security, ETA risks  the success of its new initiatives because they are not linked in an overall  framework that demonstrates how they are aligned or contribute to goals  for improving center safety and security.

It is important that Job Corps students be provided with a safe and secure  learning environment. For the last several years, however, numerous  incidents have threatened the safety and security of students. ETA has  taken steps to improve center safety and security, but its efforts could be  strengthened by ensuring regional office staff responsible for monitoring  Job Corps centers are better supported with additional guidance and  streamlined reporting requirements. Without providing regional staff with  this additional support, the full potential of the new monitoring strategy  may not be realized. While ETA has implemented several actions to  address safety and security concerns, it does not have a comprehensive  plan to guide all of its efforts. Without a comprehensive plan, ETA will not  be able to assess its overall effectiveness in addressing center safety and  security.

We are making the following three recommendations to ETA:  The Assistant Secretary of ETA should ensure the Office of Job Corps  expeditiously develops additional guidance, such as SOPs or updates to  the Program Assessment Guide, to ensure regional offices consistently  implement the risk-based monitoring strategy. (Recommendation 1)

The Assistant Secretary of ETA should ensure the Office of Job Corps  streamlines the monitoring reports completed by regional office staff. This  streamlining could include automating monitoring tools, consolidating  monitoring reports, or taking other appropriate action. (Recommendation  2)

The Assistant Secretary of ETA should ensure the Office of Job Corps  commits to a deadline for developing a comprehensive plan for Job Corps  center safety and security that aligns with leading planning practices,  such as including a mission statement with goals, timelines, and  performance measures. This could also include developing the planning  expertise within the Office of Job Corps, leveraging planning experts  within other agencies in DOL, or seeking out external experts, such as the  government-wide Performance Improvement Council. (Recommendation  3)

We provided a draft of this report to DOL for review and comment. We  received written comments from DOL, which are reprinted in appendix  VII. DOL concurred with our three recommendations. The department  stated that it will move forward to develop standard operating procedures  for its risk-based monitoring strategy, review and streamline existing  monitoring reports, and provide additional training for its regional office  staff. The department also plans to develop a formal written  comprehensive plan for Job Corps safety and security. DOL also provided  technical comments that we have incorporated in the report as  appropriate.

As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees and the Secretary of Labor. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VIII.

Appendix I: Additional Information about Our Methodology

The objectives of this review were to examine (1) what is known about the  number and types of reported incidents involving the safety and security  of Job Corps students in program year 2016; (2) what is known about  student perceptions of safety and security at Job Corps centers, and what  steps, if any, is the Employment and Training Administration (ETA) taking  to improve the survey used to collect this information; and (3) the extent  to which ETA has taken steps to address safety and security at Job Corps  centers.

To address all three objectives, we reviewed agency policies and  procedures, such as the Job Corps Policy and Requirements Handbook  and guidance issued to center operators and ETA staff. In addition, we  interviewed ETA officials, including Office of Job Corps national staff,  Office of Job Corps regional directors, and staff in all six regional offices.  We also conducted site visits at the Woodstock Job Corps Center in  Woodstock, Maryland, and the Potomac Job Corps Center in  Washington, D.C. We selected these two centers because they were  within geographical proximity to Washington, D.C., operated by different  contractors, and had over 100 reported safety and security incidents each  in program year 2016. At each center, we interviewed the Center Director,  Head of Safety and Security, a group of staff members, and a group of  students. The staff and students we spoke with were selected by the  centers. While these two site visits are not generalizable to all Job Corps  centers, they provide examples of student and staff experiences with  safety and security.

Analysis of Safety and Security Incidents at Job Corps Centers

To determine the number and types of safety and security incidents  reported by Job Corps centers, we analyzed ETA’s incident data for  program year 2016 (July 1, 2016 to June 30, 2017). This was the most  recent year of Job Corps data available at the time of our review. ETA  captures these data in its Significant Incident Reporting System (SIRS).  Centers must report incidents involving both Job Corps students and  staff, and incidents that occur at onsite and offsite locations. ETA has 20  categories of incidents in SIRS. See appendix II for incident category  definitions. The incident categories and definitions in this report are taken  directly from ETA documents and represent how ETA categorizes these  incidents. We did not assess these categories and definitions.

In this report, we present information on reported safety and security  incidents in program year 2016 involving at least one student victim or  perpetrator. There were 13,673 reported incidents involving students;  additional incidents are reported in SIRS that did not involve students.

When these additional incidents are included, a total of 14,704 safety and  security incidents were reported in program year 2016. See appendix III  for further information on the total number of incidents reported.

To calculate the number and types of reported incidents, we analyzed the  primary incident type that was assigned to each incident reported in  SIRS. To provide additional information on reported assaults and sexual  assaults, we also analyzed the secondary incident type that was assigned  to each reported assault and sexual assault in SIRS. To calculate the  total number and types of reported deaths, we analyzed both primary  incident types and secondary incident types. In SIRS, deaths can be  reported under three different primary incident types (“death”, “assault”,  and “danger to self or others”). When an incident is assigned to any of  these primary incident types, it may also be assigned a secondary  incident type of “homicide,” among other secondary incident types.

In addition, we analyzed the duty status for student victims and  perpetrators of offsite incidents. In SIRS, students are described as being  either (1) on duty, which means that they are onsite at a center or in a Job  Corps supervised offsite activity; or (2) off duty, which means they are  offsite and not under Job Corps supervision. For the 1,406 offsite  incidents, we were unable to determine student duty status in 178  instances due to inconsistencies in ETA’s data.

This report focuses on reported safety and security incidents in program  year 2016, which was from July 1, 2016, to June 30, 2017. On July 1,  2016, ETA implemented policy changes that impacted the categorization  and number of reportable safety and security incidents. Accordingly,  incident data after July 1, 2016, are not comparable with earlier incident  data, including incident data we reported in a June 2017 testimony.

We assessed the reliability of SIRS data by reviewing relevant agency  documentation about the data and the system that produced them and  interviewing ETA and Department of Labor Office of Inspector General  (DOL OIG) officials knowledgeable about the data. We determined the  data were sufficiently reliable to report the minimum number of incidents  that occurred in program year 2016. It is likely that the actual number of  incidents was greater than the number reported in SIRS because the  information is reported by Job Corps centers and the DOL OIG previously  found instances of underreporting by a non-generalizable sample of  center operators. In its March 2017 report, DOL OIG found that 12 of 125  Job Corps centers did not report 34 percent of significant incidents in  SIRS from January 1, 2014, through June 30, 2015. ETA has recently  taken steps to improve center reporting of significant incidents, such as  revising the student conduct policy to more clearly define behavior  infractions and conducting system-wide training to ensure uniform  understanding and enforcement of student conduct policies. However,  DOL OIG officials told us in January 2018 that it is too early to determine  if these steps have resolved the DOL OIG’s concerns regarding center  underreporting.

Analysis of Student Perceptions of Safety

To examine what is known about student perceptions of their safety and  security at Job Corps centers, we analyzed students’ responses to the  student satisfaction survey administered during program year 2016:  September 2016 and March 2017. We analyzed responses from both of  these surveys in program year 2016, which was the most recent year for  which data were available. ETA provided centers with the standardized  paper-based survey to administer to students in-person on designated  weeks. The survey of 49 close-ended questions contained 12 questions  that ETA used to assess students’ safety. In addition to questions on  student safety, the survey includes questions on other topics, including  student demographics, overall satisfaction with Job Corps, and access to  drugs and alcohol on center.

According to data from ETA, the response rate for each survey was  approximately 90 percent of all enrolled students. ETA calculated the  response rate by dividing the number of students who responded to the  survey by the number of enrolled students during the week of survey  administration. Students responded anonymously to the survey.

Because about 90 percent of students provided responses and about 10  percent did not, we analyzed the potential for non-response biases based  on several student characteristics. If the responses of those who did not  respond would have differed from the responses of those who did on  relevant safety questions, the results calculated solely from those who  responded may be biased from excluding parts of the population with  different characteristics or views. We compared age, time in program,  race, and gender—key characteristics available for the population of  enrollees and respondents—to determine areas for potential bias.

We determined that the potential for non-response biases existed for  particular groups of students: younger students and those enrolled in the  program for at least 6 months. For race, the potential for non-response  bias was unclear. We found no potential bias for gender. Specifically, we  found the following:

Age. Younger students were under-represented, and older students  were over-represented among survey respondents. Thus, to the  extent that non-responding younger students would have answered  safety questions differently than responding younger students, the  potential for bias existed in the survey results we analyzed. When we  asked ETA officials about such a potential bias, they responded that  they did not have evidence or documentation suggesting that age is a  predictor of students’ level of perceived safety in the program.

Length of time in the program. Students in the program less than 6  months were over-represented among survey respondents, and  students enrolled in the program over 6 months were under- represented in the survey. To the extent that non-responding students  would have answered safety questions differently based on length of  time enrolled, the potential for bias existed in the survey results we  analyzed. When we asked ETA officials about such a potential bias,  they noted that new students may be less aware about life at the  center because they begin the program with other newly arrived  students for up to 2 months. Thus, they are not yet fully integrated into  the larger student body. Otherwise, they did not have evidence or  documentation suggesting that length of time in the program  correlates with students’ level of perceived safety.

Race. It is unclear whether the distribution of race for respondents  differs from that in the population. Specifically, ignoring item non- response, about 7 percent of respondents selected “Other,” and if  those respondents were Black/African American, the distributions  between the respondents and sample would be similar since this  would result in the respondent race percentage being close to 50  percent, like the population of enrollees. If respondents who selected  “Other” were actually distributed across the race categories, this  would result in a difference between the respondent and population  race/ethnicity characteristics, and to the extent that students’  responses to safety questions differ by race, this could result in a  potential bias of respondent survey results we analyzed. We analyzed  race for purposes of potential non-response bias, and not as part of  statistical tests of survey results described below.

Gender. We found no potential non-response bias for gender because  the distribution of gender for respondents was similar to that in the  population of students enrolled in the program.

In addition to our non-response bias analysis, we assessed the reliability  of the survey data by reviewing relevant agency documentation about the  data and the system that produced them, testing data electronically, and  interviewing ETA officials knowledgeable about the data. We determined  that the student survey data were sufficiently reliable for our purposes.

Calculations of Safety for Individual Survey Questions and for National Measures

For the 12 safety-related survey questions, Job Corps policy specified  responses that the agency counted as safe or unsafe, which we followed.  As noted previously, ETA considers students to feel safe if they provided  certain responses to each of the 12 safety-related survey questions,  some of which are phrased as statements. For example, if a student  provided a response of “mostly false” or “very false” to the statement “I  thought about leaving Job Corps because of a personal safety concern,”  that student would be counted as feeling safe on that survey question  (see table 3). The percentages that we calculated are not comparable to  prior publications, including ETA reports, because, for example, ETA  revised (i.e., recoded) students’ responses in certain circumstances, as  explained below in table 7. Meanwhile, we used the original responses  that students provided and did not revise them. Also, ETA excluded  responses of “don’t know / does not apply” from its percentages. As a  result, our percentages are not comparable with those reported by ETA.

We also calculated national measures of safety for the program and for  particular demographic groups of students (e.g., male, female). Our  calculation was similar to ETA’s national safety rating in certain respects.  For example, as ETA did, we determined how safe each individual  student felt as the unit of analysis. Therefore, the national measures of  GAO and ETA may not equal the average of the 12 questions because,  for example, not all students answered every safety question.

However, in other respects, we produced our national measure differently  than ETA. Table 7 explains the three ways that our calculation differed  from ETA’s.

Although the student safety surveys were an attempt to survey a census  of the population of participants, we treated the survey as a sample in  certain respects due to the non-response of about 10 percent of students  as well as the ongoing nature of the regularly repeated survey. Therefore,  we considered these data as a random sample from a theoretical  population of students in this program and used statistical tests to assess  any differences.

Treating the data as a statistical sample, we carried out statistical tests of  differences in safety measures for student characteristics (e.g., age,  gender, length of time in the program). Because of the large sample size,  smaller differences may be detected as statistically significant. This is  because statistical significance is a function of the magnitude of the true  difference (statistical tests are more likely to detect differences when the  true values are very different) as well as the sample size (larger samples  can detect statistical significance of smaller magnitudes, when compared  to smaller sample sizes, when all else is equal). However, we used  statistical significance in conjunction with whether the detected  differences are meaningful or important, in a practical sense. In particular,  we used a series of f-tests to statistically test, at the alpha = 0.05 level, for  difference in average safety measure, across categories of age, gender,  time in program, center size, and operator type.

Appendix II: Categories of Incidents in the Significant Incident Reporting System (SIRS)

Appendix II: Categories of Incidents in the  Significant Incident Reporting System (SIRS)

Appendix III: All Significant Incidents Reported by Job Corps Centers in Program Year 2016

Our analysis of the Employment and Training Administration’s (ETA)  Significant Incident Reporting System (SIRS) data showed that there  were 14,704 reported safety and security incidents at Job Corps centers  in program year 2016, which include incidents involving students, staff,  and non-Job Corps individuals. See table 9.

Appendix IV: Reported Safety and Security Incidents Involving Students by Job Corps Center, Program Year 2016

Job Corps centers reported 13,673 safety and security incidents involving  students, including those that occurred both onsite and offsite, in program  year 2016. See table 10 for information on each Job Corps center,  including the number of incidents involving students reported in program  year 2016.

Appendix V: GAO Safety Measure for Job Corps Centers, March 2017

We calculated safety measures for each Job Corps center, based on  student responses to the safety-related questions on the student  satisfaction survey (see table 11). We used the methodology described in  appendix I to calculate safety measures for the centers. Results in table  11 are from the March 2017 survey, the most recent for program year  2016. The percentages in this table are not comparable and should not  be analyzed with the numbers of reported incidents at each center  because they are distinct measures that cover different periods of time.

Appendix VI: ETA’s Monitoring of Job Corps Centers

The Employment and Training Administration’s (ETA) risk-based  monitoring strategy is designed to identify emerging problems that place a  Job Corps center at-risk for safety and security problems. The strategy is  largely implemented by regional office staff, which work with the Office of  Job Corps’ newly formed Division of Regional Operations and Program  Integrity and use a variety of tools to assess, track, and report on center  performance (see table 12).

Appendix VII: Comments from the Department of Labor

Appendix VIII: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Mary Crenshaw (Assistant  Director), Andrea Dawson (Analyst-in-Charge), Sandra Baxter, and  Matthew Saradjian made key contributions to this report. Additional  assistance was provided by Alex Galuten, Gretta Goodwin, Benjamin  Licht, Grant Mallie, Mimi Nguyen, Nhi Nguyen, Monica Savoy, Almeta  Spencer, Manuel Valverde, Kathleen van Gelder, and Sonya Vartivarian.