Aviation Security: Basic Training Program for Transportation Security Officers Would Benefit from Performance Goals and Measures

TSA is responsible for ensuring that all airline passengers and their property are screened for items that could pose a threat to airplanes and passengers at 440 airports across the United States. Since 2016, TSO Basic Training—initial training for newly hired TSOs, including both TSA-employed and private screeners—has consisted of an intensive two-week course at the TSA Academy located at FLETC. TSA has obligated about $53 million for the program from its inception through March 2018. In 2015 and 2017, the Department of Homeland Security Inspector General raised questions about the effectiveness of checkpoint screening, which prompted concerns about training.

GAO was asked to review TSA's training of new TSOs. This report (1) describes the reasons why TSA established the TSO Basic Training program; (2) discusses factors OTD considers when updating TSO Basic Training curriculum; and (3) assesses the extent to which TSA evaluates its TSO Basic Training program. GAO reviewed documents on the development and modification of TSO Basic Training curriculum; visited FLETC; interviewed TSA officials; and compared TSA's program evaluation to leading practices.

The Transportation Security Administration (TSA) established the Transportation Security Officer (TSO) Basic Training program at the TSA Academy at the Federal Law Enforcement Training Centers (FLETC) in Glynco, Georgia to obtain benefits from centralized training. Prior to the Basic Training program, TSO training was conducted at individual airports, often by TSOs for whom instruction was a collateral duty. According to a business case developed by TSA for Congress in 2017 and TSA officials, TSA expected implementation of the TSO Basic Training program to provide efficiencies to the delivery of new-hire training for TSOs and to enhance the professionalism and morale of newly hired screeners. For example, GAO observed that TSO Basic Training facilities have airport checkpoint equipment and x-ray image simulators for students to practice skills, eliminating the challenge of finding available equipment and training times in a busy airport environment. According to program officials, centralized training also provides trainees with an increased focus on the TSA mission and instills a common culture among TSOs.

TSA's Office of Training and Development (OTD) updates and modifies the TSO Basic Training curriculum in response to evolving security threats and evaluations of effectiveness, among other factors. For example, OTD holds regular meetings with TSA's Office of Security Operations—the office responsible for managing TSO performance—to discuss issues such as imminent threats. The offices also discuss analyses of TSO effectiveness identified through covert tests, in which role players attempt to pass threat objects—such as knives, guns, or simulated improvised explosive devices—through the screening checkpoints. The two offices identify ways to address issues identified in covert testing, which are then incorporated into TSO Basic Training. OTD also gathers input from TSO Basic Training instructors and from participants to adjust training curriculum.

TSA has implemented a training evaluation model but has not yet established specific program goals and performance measures to assess TSO Basic Training. The Kirkpatrick model used by TSA is a commonly-accepted training evaluation model endorsed by the Office of Personnel Management and used throughout the federal government. While TSA reported expected benefits of TSO Basic Training in its business case and implemented the Kirkpatrick model to begin assessing training, it has not yet identified specific goals that the program is expected to achieve, nor has it developed applicable performance measures to evaluate progress toward goals, as called for by leading management practices for training evaluation. TSA officials told GAO that TSO Basic Training is a relatively new program and they planned to collect more data on TSO screening performance before further evaluating the potential impacts of the training program. However, TSO Basic Training serves as the foundation for TSOs to learn core skills and procedures, and it is important to establish goals and measures to better assess the effectiveness of the training they receive. This will help TSA determine the extent to which TSOs are able to fulfill their important role in ensuring passenger safety while also showing results for the funds spent on such training each year.

GAO recommends that TSA establish specific goals and performance measures for the TSO Basic Training program. TSA concurred with the recommendation.

The screening of airport passengers and their property is a critical  component in securing our nation’s commercial aviation system. Since  the terrorist attacks of September 11, 2001, the Transportation Security  Administration (TSA) has been tasked with screening airline passengers  and their carry-on and checked baggage for prohibited and other  potentially dangerous items that could pose a threat to aircraft and  passengers. According to TSA, in fiscal year 2017, more than 771 million  passengers were screened at 440 airports across the United States.

Since its inception, TSA has frequently adapted its standard operating  procedures (SOP) to address new and evolving threats to aviation  security. For instance, after the discovery of a plot to take the ingredients  for an improvised explosive device—including hydrogen peroxide  disguised as soda—onto multiple aircraft, TSA limited the amount of  liquids passengers could bring onto planes from outside the airport.  Transportation Security Officers (TSOs) at airports follow SOPs that guide  screening processes and utilize technology such as advanced imaging  technology (AIT) scanners or walk-through metal detectors to screen  passengers and their accessible property. TSOs have annual training  requirements, which are developed and implemented by TSA’s Office of  Training and Development (OTD). TSOs also receive basic training upon  being hired. Through 2015, basic training was provided at TSOs’ home  airports through TSA’s New Hire Training Program (NHTP). As of January  2016, NHTP was replaced by TSO Basic Training, an intensive two-week  training program at the TSA Academy (Academy) located at the Federal  Law Enforcement Training Centers (FLETC) campus in Glynco, Georgia.

In 2015 and 2017, the Department of Homeland Security (DHS) Inspector  General identified vulnerabilities in TSA checkpoint screening, after  agents carried metallic (inoperable handguns) and nonmetallic (simulated  explosive) test items concealed on their body through TSA passenger  security checkpoints and into the sterile areas of airports. The results of  these covert tests also sparked concerns about the training of TSOs. As a  result, you asked us to review TSA’s training of new TSOs through the  TSO Basic Training program. This report: (1) describes the reasons why  TSA established TSO Basic Training; (2) discusses factors OTD  considers when updating the TSO Basic Training curriculum; and (3)  assesses the extent to which TSA evaluates its TSO Basic Training  program.

To describe the reasons TSA established the TSO Basic Training  program at the TSA Academy, we reviewed documents related to the  establishment of the program, such as agency training plans, a management directive, and the business case for TSO Basic Training  submitted by TSA to Congress. We also visited the Academy to better  understand the benefits of establishing TSO Basic at FLETC. In addition,  we interviewed OTD officials at headquarters and at the Academy,  instructors at the Academy, and union officials representing TSOs  employed by TSA to better understand the benefits and challenges of  training all new TSOs at the Academy and how TSO Basic Training  differs from NHTP in place prior to TSO Basic Training.

To discuss what factors OTD considers when updating the TSO Basic  Training curriculum, we reviewed documentation from OTD regarding  changes made to TSO Basic Training since its inception, including a list  of changes to the curriculum and a contractor’s report that recommended  some of those changes. We also interviewed officials from the Office of  Security Operations (OSO), the office that manages TSOs at the nation’s  airports, the Office of Inspection (OOI), which conducts covert testing to  evaluate checkpoint effectiveness, and OTD to determine why  modifications were made and how the offices communicate regarding  updates to standard operating procedures and other processes that may  impact TSO Basic Training.

To assess the extent to which TSA evaluates TSO Basic Training, we  reviewed TSA documents used for evaluating training courses, including  end-of-course surveys administered to participants and surveys  distributed to participants and their supervisors approximately 3 months  after course completion. We also interviewed TSA headquarters officials  responsible for evaluating TSO Basic Training and for developing and  implementing TSO Basic Training as well as TSO Basic Training  instructors. We compared the training evaluation documentation to the  Kirkpatrick Model for training evaluation, which TSA uses as the model for  its evaluations of TSO training, and to leading practices in evaluating  training and development efforts.

We conducted this performance audit from November 2017 through July  2018, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

After the terrorist attacks of September 11, 2001, Congress passed and  the President signed the Aviation and Transportation Security Act into law  on November 19, 2001, with the primary goal of strengthening the  security of the nation’s civil aviation system. The act established TSA as  the agency with responsibility for securing all modes of transportation,  including civil aviation. As part of this responsibility, TSA performs or  oversees security operations at the nation’s nearly 440 commercial  airports, including managing passenger and checked baggage screening  operations.

TSOs inspect individuals and property to deter and prevent passengers  from bringing prohibited items on board an aircraft or into the airport  sterile area—in general, an area of an airport to which access is  controlled through the screening of persons and property. While working  at an airport checkpoint as shown in figure 1, TSOs perform a variety of  tasks, which include:

Travel document verification: a TSO checks passengers’ identification  against the boarding pass and the individual presenting the  identification.

Divestiture: a TSO assists passengers by informing them what items  need to be placed on the x-ray conveyor belt.

X-ray interpretation: TSOs screen passengers’ carry-on baggage and  personal property by interpreting x-ray images to identify any  prohibited items.

Advanced imaging technology operations: Passengers are screened  via advanced imaging technology (often referred to as body  scanners), which identifies areas where they may be concealing  prohibited items.

Walk-through metal detector operation: a TSO operates the walk- through metal detector.

Physical searches: Passengers can opt to be screened through a  physical search, or TSOs may perform a physical search to resolve an  alarm triggered by the AIT system or the walk-through metal detector,  among other reasons.

Explosive trace detection and manual searches of property: TSOs use  an explosives trace detection system by swabbing carry-on baggage  and testing the sample for explosive residue or vapors. This test is  usually performed in conjunction with a manual search of the carry-on  baggage.

Exit lane monitoring: a TSO watches the lane through which  passengers exit the sterile area to ensure that no one enters the  sterile area through that passage.

Within TSA, two offices work together to manage TSOs and ensure their  training is current and relevant. OSO is responsible for allocating TSO  staff to airports, scheduling TSO work hours and training availability, and  developing SOPs that govern how TSOs screen passengers and  baggage. OTD is responsible for developing initial and ongoing training  curricula for TSOs based in part on SOPs. Within OTD, a dedicated team  is located at the Academy to manage updates at TSO Basic Training.

In accordance with the Aviation and Transportation Security Act,  screeners must complete a minimum of 40 hours of classroom instruction,  60 hours of on-the-job training, and successfully complete an on-the-job  training examination. Until 2016, new TSOs completed these training  requirements at or near their home airports through the New Hire Training  Program (NHTP). Since TSA centralized the TSO Basic Training program  in January 2016, TSOs fulfill these training requirements through  classroom training at the Academy as well as training at their home  airports prior to the Academy and on-the-job training after completion of  TSO Basic Training. During the 2 weeks spent at the Academy, TSOs  receive 80 hours of training on standard operating procedures, threat  detection, and the use of screening equipment. Prior to attending TSO  Basic Training, new TSOs complete computer-based prerequisite training  and may shadow experienced TSOs at a checkpoint. TSO Basic Training  allows for participants to be trained at a dedicated facility with more  hands-on training than was possible for NHTP (see Appendix I for a  comparison of the two programs).

As shown in table 1, of the $53 million obligated from January 2016  through March 2018, TSA obligated $18.2 million for procurement and  development of the modular buildings on the FLETC campus used for  TSA training, as well as associated hardware and set-up obligations such  as audio/video equipment and fully operational simulated checkpoints.  TSA obligated an additional $12 million in fiscal year 2016 and $13.7  million in fiscal year 2017 for the delivery of TSO Basic Training, including  associated student travel and related equipment. TSA officials told us  that due to continuing budget resolutions that funded the government  between October 2017 and March 2018, TSA was not able to fully fund  the interagency contract between TSA and FLETC to support the TSO  Basic Training course in fiscal year 2018 at the beginning of the year. For  this reason, TSA does not yet have 2018 training obligations available for  reporting through its accounting system. However, based on the average  cost per student in fiscal year 2017 of about $2,300 to attend TSO Basic  Training, TSA estimates total training obligations of approximately $9.1  million in the first half of fiscal year 2018.

TSA Established the TSO Basic Training Program at the Academy to Obtain Benefits from Centralized Training

According to the business case for TSO Basic Training and TSA officials,  implementation of the TSO Basic Training program at the Academy was  anticipated to provide a number of potential benefits. The anticipated  benefits identified generally align under two distinct categories: (1)  efficiencies and improvements obtained through the centralized delivery  of training, and (2) enhanced professionalism and “esprit de corps”  obtained through bringing newly hired screeners together for centralized  training. Collectively, these benefits were also envisioned by TSA  headquarters officials to have a positive impact on screening  effectiveness and public perception of the TSA workforce.

Based on several analyses of training delivery options that TSA has  conducted since 2008, TSA determined that a centralized training  academy would have a number of potential benefits relative to the  decentralized training previously administered at field airports through  NHTP. Among the potential efficiencies and improvements cited by TSA  are:  Increased consistency and standardization. According to TSA  documents and OTD headquarters officials, centralized training  provides a standardized curriculum that serves as a foundation for the  skills, knowledge, and equipment used across an array of different  airport environments. The TSA business case and other supporting  analyses note that such an approach offers greater consistency of  training delivery and a better mechanism for developing, delivering,  and evaluating course content.

Equipment availability and expanded course content. TSO Basic  Training includes a full suite of dedicated checkpoint equipment and  x-ray image simulators for students to practice learned skills,  eliminating the challenge of finding available equipment and training  times in a busy airport environment (see figure 2). Officials told us that  being more familiar with the screening equipment increases TSOs’  readiness for on-the-job training when they return to their home  airports. Initial test results also indicate that participants trained at the  Academy receive higher pass rates on end-of course assessments of  x-ray image interpretation skills than those who received their initial  training at their home airports. Specifically, according to TSA data, of  the 5,877 test-takers who received training at TSO Basic Training in  2016, 91.5 percent passed the Image Interpretation Test on their first  attempt. In contrast, 83.2 percent of the 1,458 test-takers who  received training at local airports in 2016 passed the test on their first  attempt. In addition, the Academy curriculum incorporates new  learning opportunities, including a live demonstration of improvised  explosive devices and an active shooter drill, both of which would be  difficult to reproduce within the airport environment, according to TSA  officials.

Dedicated faculty and instructor development. TSO Basic Training  offers a dedicated faculty and support staff focused exclusively on  training TSOs. According to TSA officials, before TSO Basic Training,  training at individual airports was often conducted by TSOs for whom  instruction was a collateral duty, whereas instructors at the Academy  have full-time training responsibilities and enhanced opportunities to  learn from each other, increase their professional training skills, and  provide feedback on the delivery of course curriculum.

Centralized facility and shared logistics. By locating the TSA Academy  at FLETC, TSA is able to take advantage of the services and logistical  support that FLETC provides. Specifically, FLETC services and  logistics include accommodations, meals, and transportation, thereby  reducing the administrative demands on TSA personnel and allowing  students a focused and efficient training experience. Additional  efficiencies cited by TSA officials include lower overall costs for office  space, janitorial services, and other operational costs because such  costs are shared by the 96 agencies that use FLETC. According to  TSA officials, conducting training at FLETC can also help TSA  accommodate hiring surges and better augment future training, if  needed. For example, TSA officials reported that the facility has surge  capacity from its current capacity of 240 students up to 300 new  students if sufficient instructors are available.

According to TSA documents and training officials, another key benefit of  centralized training is the opportunity to enhance professionalism and  help foster camaraderie and esprit de corps. TSA anticipates that  centralized, standardized training will not only provide trainees with an  increased focus on the TSA mission and operational environment, but  can serve to instill a common culture and sense of belonging among the  broader community of TSOs nationwide. In its business case, TSA notes  that centralized training of new recruits is a common model employed by  the armed forces and other federal law enforcement agencies within  DHS, such as U.S. Customs and Border Protection and the U.S. Coast  Guard. According to the business case, by bringing together newly hired  TSOs from around the country, TSA also hopes to inspire in its trainees a  singular identity and unity of purpose, which previous analyses generally  found lacking as part of the decentralized training approach.

The business case also associates such increases in professionalism and  esprit de corps with greater employee satisfaction and the potential for  reduced attrition. Analysis conducted by TSA in 2017 provides some  initial support for positive trends in these areas. For example, results of a  2017 TSA employee engagement survey indicated that respondents who  attended TSO Basic Training reported higher scores in categories  including Organizational Commitment, Job Satisfaction, and Overall  Morale versus respondents who did not attend. TSA also reported a 19  percent reduction in the attrition rate during the first 180 days of being  hired for those attending TSO Basic Training at the Academy in 2016  versus those who received their initial training at field airports through the  New Hire Training Program.

Factors Considered in Updating the TSO Basic Training Curriculum Include Evolving Security Threats and Input from Course Participants

OTD Uses Information from OSO to Update the TSO Basic Training Curriculum to Address Evolving Security Threats

OTD updates and modifies the TSO Basic Training curriculum based, in  part, on regular communications from OSO, the office responsible for  developing SOPs for screening operations and managing TSO  performance. Officials from both offices told us that OSO provides  information to OTD on changes to SOPs as soon as changes are made  so they can update the TSO Basic Training curriculum. For instance, in  2017, when OSO began planning major changes to the SOPs, the office  gave OTD information about the planned SOP revisions, as well as the  airports where the new SOPs would be piloted. In response, OTD  modified its curriculum and was able to provide revised training for new  TSOs based at airports that were piloting the program, while providing  TSOs at all other airports the prior version of training. OTD officials noted  that in some cases TSA must quickly update SOPs to reflect imminent  threats. According to officials, a plan is in place to make changes to TSO  Basic Training curriculum in response to emerging or imminent threats,  although such threats have not been experienced since the establishment  of TSO Basic Training in 2016.

In addition to changes in SOPs, OSO officials indicated they may also  change the timing of when TSOs employed by TSA attend TSO Basic  Training. Specifically, officials told us that OSO plans to implement a new  model for TSO Basic Training in which TSOs will attend TSO Basic  Training 2 to 6 months after they are hired rather than as soon as is  practical. According to TSA, the agency is pursuing this change to,  among other things, implement a transparent career path for TSOs  employed by TSA and to encourage and reward skill development. During  the 2 to 6 months prior to attending TSO Basic Training, TSOs will  perform checkpoint tasks that require training that can be delivered at the  airport as soon as they are hired, such as checking passengers’ travel  documents and helping passengers move through the checkpoint. Once  TSOs are able to perform these initial tasks, they will attend TSO Basic  Training at the Academy, Officials told us they are preparing for the  change by modifying the TSO Basic Training curriculum to eliminate  subjects that will be covered at the airports and to emphasize skills that  more experienced TSOs will need, such as performing physical searches  of passengers. TSA plans to implement the revised model beginning in  August 2018.

Finally, OTD receives information on TSO performance and uses that  information to inform TSO Basic Training curriculum. For example, two  TSA offices—OSO and the Office of Inspections—perform regular  effectiveness testing of airport checkpoints through covert testing and  share the results with OTD. After each covert testing event, each office  conducts interviews with TSOs to determine the factors that contributed to  their effectiveness at identifying prohibited items. Officials told us that  OSO and OTD hold regular meetings to discuss the analyses of covert  testing failures and ways in which training curriculum can be modified to  address the reasons for the failures, which are then incorporated into the  TSO Basic Training curriculum. Office of Inspections officials noted that  they participated in the development of the TSO Basic Training curriculum  and provide regular reports to OTD on covert testing results.

When Making Updates to TSO Basic Training Curriculum, OTD Considers Feedback from Instructors, Course Participants, and Contractors

OTD gathers input from TSO Basic Training participants, instructors, and  contractors on ways to update the curriculum. For instance, TSO Basic  Training instructors told us they submit “white paper proposals” to TSO  Basic Training course managers detailing their suggested changes to the  course. They can also provide feedback and suggestions during “train the  trainer” sessions, in which all instructors participate when TSO Basic  Training is updated. Instructors told us that all sessions include an  opportunity for instructors to provide feedback after reviewing the new  curriculum. Officials told us that they take instructors’ feedback into  account when implementing new curriculum. For instance, officials told us  that at the suggestion of instructors, they added time for discussion at the  end of each checkpoint lab to help capture and share lessons learned.

OTD also collects feedback from TSOs who have participated in the  course, both at the end of their two weeks at the Academy and several  months after their completion of the course. At the end of TSO Basic  Training, OTD collects feedback from participants through a survey with  both multiple choice and open-ended questions. The survey includes  questions on the course curriculum and instructor performance. Officials  told us that they regularly review the results of the survey and consider  whether it is appropriate to address the feedback by modifying TSO Basic  Training. For instance, the most often provided feedback for altering the  curriculum was to increase the time in hands-on training using screening  equipment in the Academy’s simulated checkpoints. In response, OTD  officials told us they added nearly 5 hours of hands-on training to the 80- hour program in addition to the 6 hours that had previously been a part of  the curriculum.

In addition to collecting feedback from TSO Basic Training participants  and instructors, TSA officials told us that TSA regularly uses a contractor  to support the design and development of training courses and to assess  existing courses, including TSO Basic Training. In 2016, the contractor  conducted an evaluation of the instructional integrity of the TSO Basic  Training curriculum. The resulting report made a number of  recommendations to improve the curriculum and structure of TSO Basic  Training, many of which OTD has implemented. For instance, the  contractor recommended that TSO Basic Training include more  opportunities for review of the material to reinforce TSOs’ understanding.  In response, OTD implemented a review session at the end of the first  week of training so TSOs have an opportunity to clarify information  presented over the first week.

TSA Has Made Progress in Implementing a Training Evaluation Model but Has Not Established Specific Goals and Performance Measures to Assess TSO Basic Training

TSA has implemented three of the four levels of the Kirkpatrick Model, a  training evaluation model that, in part, helps TSA collect feedback from  course participants and evaluate the impact on individual development.  However, the agency has not developed goals for the program or related  performance metrics to demonstrate progress toward goals.

TSA Has Made Progress in Implementing the Kirkpatrick Model to Evaluate Its TSO Basic Training Program

To evaluate the TSO Basic Training program, TSA uses the Kirkpatrick  Model, which is a commonly accepted training evaluation model endorsed  by the Office of Personnel Management and used throughout the federal  government. The Kirkpatrick Model consists of a four-level approach for  soliciting feedback from training course participants and evaluating the  impact the training had on individual development, among other things.  To date, TSA has implemented the first three levels of the model by  administering (1) course surveys to participants at the end of the training  program; (2) an end-of-course written exam and an x-ray image  interpretation test to assess achievement of learning objectives; and (3)  course surveys to participants and their supervisors several months after  completing training to collect information regarding how the training  affected behavior or performance on the job. OTD officials told us that  they have not yet implemented Level 4 of the model because they do not  believe they have enough data. Table 2 provides a description of what  each level within the Kirkpatrick model is to accomplish and TSA’s  progress in implementing the levels.

TSA Has Not Yet Developed Goals and Performance Measures for TSO Basic Training

While TSA reported potential benefits of TSO Basic Training in its  business case and implemented the Kirkpatrick Model to assess training,  it has not yet identified specific goals that the TSO Basic Training  program is expected to achieve, nor has it developed performance  measures to evaluate progress toward goals. The business case and the  Kirkpatrick Model are positive steps and document certain benefits of  TSO Basic Training, but without a set of specific training goals and  associated performance measures for the program, TSA is not able to  fully evaluate the program’s effectiveness and ensure accountability  toward results. Such goals are important to help ensure alignment with  course objectives and the end-of course examinations administered as  part of Level 2 of the Kirkpatrick Model. In addition, without the  development of specific goals, it is not possible to determine what types  of performance measures should be used to help show progress toward  such goals. For example, in its business case, TSA identified improved  employee morale as one of the anticipated benefits of TSO Basic  Training. However, there are no goals or metrics specifically related to  this benefit. If TSA believes improved morale should be something for  which TSO Basic Training aims, goals and measures could help them  demonstrate the extent to which this benefit is being realized by the  training program.

Leading management practices related to training evaluation guidance  identifies the importance of agencies developing and using performance  measures regularly to ensure accountability and assess progress toward  achieving results that are aligned with the agency’s mission and goals.  In addition, these practices highlight the importance of agencies having  clear goals about what the training or development program is expected  to achieve as a precursor to developing such measures. When designed  effectively, performance measures help decision makers (1) determine  the contributions that training makes to improve results, (2) identify  potential gaps in performance, and (3) determine where to focus  resources to improve results. In particular, incorporating valid measures  of effectiveness into training programs can enable an organization to  better ensure that desired changes occur in trainees’ skills, knowledge,  and abilities.

According to OTD officials, the TSO Basic Training program was  established on an accelerated schedule in late 2015 as one of multiple  efforts to improve training delivery and help enhance screener  effectiveness. Officials stated that the program is still relatively new and  they plan to collect several additional years of data on system-wide  screening performance before conducting efforts to further evaluate the impact of the training. They reported that the lack of performance  measures is also due to the inherent difficulty of tying specific training  initiatives to broader organizational results. Officials told us that once  TSOs return to their home airports after TSO Basic Training, they are  exposed to additional on-the-job training and differing airport cultures,  which make it difficult to isolate the effects of TSO Basic Training.  However, senior training officials agreed that establishing applicable  goals and performance measures for the TSO Basic Training program  would be helpful to support ongoing efforts and better measure program  progress.

We recognize that developing metrics to assess the performance of  training programs on broad organizational results can be challenging.  However, there are additional opportunities to develop program goals and  performance measures as part of the training evaluation efforts at the  Academy to help ensure that participants can demonstrate proficiency in  performing core technical skills before returning to their home airports.  We believe that developing goals for a training program does not need to  wait for years of data. Goals reflect desired results, connected to an  agency’s mission, which a program plans to achieve. In the over 2 years  of using TSO Basic Training, TSA has not stated what results the training  program is to achieve.

TSOs provide a crucial function to help ensure passenger safety, and it is  important to have goals aligned with this mission, as well as associated  measures of effectiveness of the training they receive at TSO Basic  Training to determine the extent to which they are able to fulfill their  important role. As noted by leading management practices for training  evaluation, agencies need credible information to demonstrate a training  program is contributing to a goal and they can develop such data through  a mix of quantitative and qualitative indicators. We found that options for  assessing the effectiveness of TSO Basic Training could include  measuring TSO performance by leveraging data from end of course  examinations, such as the x-ray image interpretation test, and introducing  similar additional tests or mechanisms to further evaluate trainees’  knowledge and skills in effective screening procedures. Additional options  could include measuring employee morale as indicated by TSOs on their  Kirkpatrick Level 1 surveys at the completion of the training program, and  comparing these results against applicable program goals for employee  morale that TSA could establish related to TSO Basic Training. By  identifying annual goals and measures for TSO Basic Training, TSA will  also be better positioned to move forward with Level 4 of the Kirkpatrick  Model to evaluate the impact of training on broader organizational results.  Given that over $50 million has been obligated to set up and operate the  TSO Basic Training program to date, it is important that TSA incorporate  annual goals and measures into the training program to be better  informed when making training decisions and to help hold itself  accountable for training results on a regular basis.

TSOs perform a critical role in securing our nation’s commercial aviation  system and often represent the most visible face of TSA to the public. For  this reason, new hire training is an integral function to ensure that TSOs  are obtaining the foundational skills and knowledge to help prepare them  to perform their jobs effectively. In 2016, TSA initiated a major change to  its training approach for new hires to help ensure a consistent and  standardized training experience and promote enhanced camaraderie  and esprit de corps. Although TSA has implemented a framework to  assess participant reactions to the training and their knowledge of course  content, it has not yet established goals for the TSO Basic Training  program or measures to gauge effectiveness of the training TSOs receive  to determine the extent to which they can fulfill their crucial role in  ensuring passenger safety. By taking these steps, TSA will be better  positioned to determine if the program is improving trainees’ skills,  knowledge, and abilities and whether additional skill development, or  other training modifications, may be needed.

We are making one recommendation to the Administrator of TSA.  Specifically, the Administrator of TSA should establish specific goals for  the TSO Basic Training program and develop performance measures that  can be used to assess if the program is achieving desired outcomes and  help ensure accountability for training results on a regular basis.  (Recommendation 1)

We provided a draft of this report to DHS for review and comment. DHS  provided written comments, which are reprinted in appendix II, and  technical comments, which we incorporated as appropriate. DHS agreed  with our recommendation that TSA establish specific goals for the TSO  Basic Training program and develop performance measures that can be  used to assess if the program is achieving desired outcomes. In addition,  in its written comments DHS outlined steps to address this  recommendation.

With regard to performance goals, TSA plans to establish broad goals  that include successful screening and improved morale, among others.   The stated goals are an appropriate response to our recommendation  that TSA develop goals specifically for TSO Basic Training.  These  actions, if implemented effectively, should address the intent of our  recommendation.

With regard to developing performance measures that can be used to  assess program outcomes, TSA intends to leverage existing mechanisms  through its Kirkpatrick Model evaluations to measure program success.   As we noted in the report, implementing the first three levels of the  Kirkpatrick Model are positive steps that document certain benefits of  TSO Basic Training, but they do not address specific goals or  performance measures. Kirkpatrick Model Level 2 evaluations include  proficiency exams administered prior to TSOs’ departure from the  Academy. Data from these evaluations, in conjunction with specific goals,  may provide quantifiable metrics that could inform further refinement of  the TSO Basic Training curriculum. However, the surveys being used by  TSA for Level 3 of the Kirkpatrick Model do not include metrics that would  allow TSA to measure the program’s effectiveness and ensure  accountability toward results. Specifically, the surveys do not demonstrate  whether TSO Basic Training is reaching goals related to successful  screening or improved morale because survey results are influenced by  factors outside of the training program. We will continue to monitor TSA’s  efforts in this area.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Homeland Security, and other interested  parties. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (206) 287-4804 or AndersonN@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix III.

Appendix I: Comparison of New Hire Training Program and Transportation Security Officer (TSO) Basic Training

In 2016, the Transportation Security Administration (TSA) established the  TSO Basic Training program at the TSA Academy, located at the Federal  Law Enforcement Training Centers in Glynco, Georgia. TSO Basic  Training allows new TSOs to be trained at a dedicated facility with  simulated checkpoints. Previously, TSOs’ initial training was delivered  through the New Hire Training Program at or near their home airports, at  which they were able to practice using checkpoint equipment only when  the equipment was not being used, such as after hours. For further  comparison of the two programs, see Table 3.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Dawn Locke, Assistant Director;  Miriam Hill, Analyst in Charge; and Ryan Lambert made key contributions  to this report. Also contributing to the report were Elizabeth Dretsch, Eric  Hauswirth, Susan Hsu, Heidi Nielson, and Adam Vogt.