Aviation Security: Actions Needed to Systematically Evaluate Cost and Effectiveness Across Security Countermeasures

Since the attacks of September 11, 2001, TSA has spent billions of dollars on aviation security programs. However, recent attacks involving aircraft and airports in other countries underscore the continued threat to aviation and the need for an effective aviation security program.

GAO was asked to review TSA's passenger aviation security countermeasures. This report examines the extent to which TSA has (1) information on the effectiveness of selected passenger aviation security countermeasures and (2) systematically analyzed the cost and effectiveness tradeoffs among countermeasures.

GAO reviewed TSA documentation on the effectiveness of six passenger aviation security countermeasures in fiscal year 2015—the most recent year for which data were available. GAO selected these countermeasures because they involve direct interaction with passengers, their belongings, or their personal information, and are largely operated and funded by TSA. GAO also reviewed TSA documents and interviewed TSA officials regarding efforts to systematically analyze cost and effectiveness tradeoffs across countermeasures.

The Transportation Security Administration (TSA) has data on the effectiveness of some, but not all of its passenger aviation security countermeasures. Specifically, TSA has data on passenger prescreening, checkpoint and checked baggage screening, and explosives detection canines. Further, TSA is taking steps to improve the quality of this information. However, it does not have effectiveness data for its Behavior Detection and Analysis (BDA) program and the U.S. Federal Air Marshal Service (FAMS). For BDA—a program to identify potential threats by observing passengers for behaviors indicative of stress, fear, or deception—in July 2017, GAO reported that (1) TSA does not have valid evidence supporting most of its behavioral indicators, and (2) TSA should continue to limit future funding for its behavior detection activities until it can provide such evidence. For FAMS—a program that deploys armed law enforcement officers on certain flights at an annual cost of about $800 million for fiscal year 2015—officials reported that one of the primary security contributions is to deter attacks. However, TSA does not have information on its effectiveness in doing so, nor does it have data on the deterrent effect resulting from any of its other aviation security countermeasures. While officials stated that deterrence is difficult to measure, the Government Performance and Results Act of 1993, as updated, provides that agencies are to assess the effectiveness of their programs. Further, the Office of Management and Budget and GAO have suggested approaches for measuring deterrence. Developing such methods for TSA countermeasures, especially for an effort such as FAMS in which the primary goal is deterrence, would enable TSA to determine whether its substantial investment is yielding results.

TSA has a tool to compare the security effectiveness of some aviation security countermeasures, but has no efforts underway to systematically evaluate potential cost and effectiveness tradeoffs across all countermeasures. In 2014, the agency developed a tool to analyze the security effectiveness of alternate combinations of some countermeasures for the purpose of informing acquisition and deployment decisions, but does not have a tool to assess such tradeoffs across the entire system of countermeasures. TSA officials explained that the aviation security system is constantly evolving, and assessing a system in flux is challenging. However, DHS policy and TSA's strategic plan call for the systematic evaluation of costs and effectiveness of TSA's chosen mix of aviation security countermeasures. Without such an analysis, TSA is not well positioned to strike an appropriate balance of costs, effectiveness, and risk.

This is a public version of a classified report that GAO issued in August 2017. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted.

GAO recommends that TSA (1) explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures, with FAMS as a top priority to address, and (2) systematically evaluate the potential cost and effectiveness tradeoffs across aviation security countermeasures. DHS concurred with these recommendations.

It has been 16 years since the attacks of September 11, 2001, exposed  vulnerabilities in the U.S. aviation system. Since then, the Department of  Homeland Security’s (DHS) Transportation Security Administration  (TSA)—the primary federal agency with responsibility for securing the  nation’s civil aviation system—has spent billions of dollars on a wide  range of programs designed to enhance aviation security. However,  achieving TSA’s stated mission to protect the nation’s transportation  systems remains a daunting task. Senior DHS officials have stated that  terrorist organizations continue to regard civil aviation as an attractive  target for attacks. Further, recent attacks involving aircraft and airports in  Egypt, Somalia, Belgium, and Turkey have underscored the continued  threat to aviation and the need for an effective aviation security program.

Over the past 5 fiscal years, funding made available to TSA for aviation  security has remained relatively steady—ranging from a high of nearly  $7.7 billion in fiscal year 2012 to a low of nearly $7.0 billion in fiscal year  2013—but fiscal pressures facing the government continue. In this  environment of high threat and limited resources, it is essential that TSA  identify how to allocate its resources to obtain the greatest risk mitigation  value for each dollar spent.

You requested that we review what TSA knows about the costs and  effectiveness of its passenger aviation security countermeasures. In this  report, we examine  1.  the extent that TSA has information on the effectiveness of selected  passenger aviation security countermeasures, and what these data  indicate, and  2.  the extent that TSA has systematically analyzed the cost and  effectiveness tradeoffs of alternate combinations of countermeasures  within its aviation security system.

This report also presents fiscal year 2015 cost and effectiveness  information for selected aviation security countermeasures in appendixes  I and II, respectively.

This report is a public version of a classified report that we issued in  August 2017. TSA deemed some of the information in our August report  to be classified or sensitive security information, which must be protected  from loss, compromise, or inadvertent disclosure. Therefore, this report  omits classified and sensitive security information about the effectiveness  of certain aviation security countermeasures and some specifics about  TSA’s screening procedures. Although the information provided in this  report is more limited, the report addresses the same objectives as the  classified report and uses the same methodology.

To determine the extent of TSA’s information on the effectiveness of its  passenger aviation security countermeasures in fiscal year 2015, we  reviewed past reports and findings from GAO, TSA, and DHS’s Office of  Inspector General (OIG) related to the effectiveness of TSA aviation  security programs and evidence of steps TSA has taken to address the  issues identified. We also asked TSA to provide us with their evidence of  the security effectiveness of six selected aviation security  countermeasures—passenger prescreening (Secure Flight), checkpoint  screening, checked baggage screening, explosives detection canines, the  Behavior Detection and Analysis (BDA) program, and the U.S. Federal Air  Marshal Service (FAMS)—in detecting, disrupting, and deterring threats  to the nation’s aviation system. We selected these six passenger aviation  security countermeasures because they involve direct interaction with  passengers, their belongings, or their personal information and are largely  operated and funded by TSA. We analyzed TSA’s performance and  testing data to determine the extent of TSA’s effectiveness information for  these six selected countermeasures and what these data indicate about  their security effectiveness during fiscal year 2015—the most recent full  year for which data were available. We also reviewed TSA documents  related to countermeasure effectiveness such as the results of routine  TSA tests of screening technologies deployed at airports.

To obtain TSA perspectives on the extent and reliability of TSA’s  effectiveness information as well as the effect of any data limitations, we also interviewed senior TSA officials in (1) the Office of Requirements and  Capabilities Analysis (ORCA), which is responsible for assessing TSA’s  operational capability gaps and developing future requirements; (2) the  Chief Performance and Enterprise Risk Office (CPER), which is  responsible for the overall leadership, vision, and direction for risk  management across TSA; (3) the Office of Inspections (OOI), which  conducts covert testing of several aviation security countermeasures; and  (4) individual program offices associated with each of the six  countermeasures. We also interviewed officials with the National Center  for Risk and Economic Analysis of Terrorism Events (CREATE)—a DHS- funded research center—to obtain their perspective on the challenges  TSA faces in measuring the effectiveness of some countermeasures. We  compared TSA’s efforts to measure effectiveness to requirements in the  Government Performance and Results Act of 1993 (GPRA), as updated  by the GPRA Modernization Act of 2010, and leading practices  established in GAO’s prior work on using performance information to  inform management decision making. We assessed the reliability of  TSA’s effectiveness data by (1) reviewing TSA documentation of  procedures for recording effectiveness data, (2) manually testing for  obvious errors and discrepancies in select results, and (3) interviewing  knowledgeable TSA officials about procedures for collecting and  recording these data. We discuss our findings about the reliability of these  data later in this report.

To determine the extent that TSA officials have systematically analyzed  the cost and effectiveness tradeoffs of alternate combinations of  countermeasures, we reviewed TSA documentation of their efforts to do  so. Specifically, we reviewed technical papers on TSA’s Risk and Trade  Space Portfolio Analysis tool (RTSPA)—a tool the agency developed to  analyze security effectiveness tradeoffs among checkpoint screening and  checked baggage screening countermeasures. To learn more about  TSA’s efforts to analyze cost and effectiveness tradeoffs, we interviewed  senior ORCA officials about RTSPA, and observed a demonstration of  RTSPA’s analytical capabilities. We also met with senior TSA officials  including the prior Chief of Staff, and officials in CPER and the Office of  Finance and Administration to discuss the agency’s efforts to analyze the  cost and effectiveness tradeoffs of alternate combinations of  countermeasures. We then compared the extent of this analysis to DHS’s  Policy for Integrated Risk Management memorandum, which describes  how DHS components, including TSA, should use risk management to  inform strategies, processes and decisions to enhance security. We also  compared the extent of this analysis to TSA’s strategic plan.

The performance audit upon which this report is based was conducted  from February 2016 to August 2017 in accordance with generally  accepted government auditing standards. Those standards require that  we plan and perform the audit to obtain sufficient, appropriate, evidence  to provide a reasonable basis for our findings and conclusions based on  our audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives. We worked with DHS from July 2017 to September 2017 to  prepare this unclassified version of the original classified report for public  release. This public version was also prepared in accordance with these  standards.

Enacted in November 2001, the Aviation and Transportation Security Act  (ATSA) established TSA as the primary federal agency responsible for  implementing and overseeing the security of the nation’s civil aviation  system. In accordance with ATSA, TSA is to ensure that all passengers  and property transported by commercial passenger aircraft to, from, or  within the United States are adequately screened. Among other things,  TSA is responsible for ensuring that for all flights and flight segments  originating in the United States, such screening takes place before  boarding and is carried out by a federal government employee except as  otherwise permitted in statute. Pursuant to TSA-established policies and  procedures in effect at about 440 airports at which TSA performs, or  oversees the performance of screening operations (i.e., TSA-regulated  airports), all passengers, their accessible property, and their checked  baggage are to be screened prior to entering the sterile area of the airport  or boarding the aircraft. Among other things, these procedures generally  provide that passengers pass through security checkpoints where their  person, identification documents, and accessible property are screened  by Transportation Security Officers (TSO).

Overview of Selected Aviation Security Countermeasures

In this report, we examine six countermeasures specific to aviation  security—passenger prescreening (Secure Flight), checkpoint screening,  checked baggage screening, explosives detection canines, BDA, and  FAMS. An overview of these countermeasures is provided below and  figure 1 depicts an illustrative example of the process by which an  aviation passenger may encounter these selected countermeasures.

Passenger Prescreening (Secure Flight): TSA uses its Secure Flight  prescreening program to match passenger information against federal  government watch lists and other information to assign each passenger to  one of three risk categories—high risk, low risk, or unknown risk—that  either corresponds to the level of screening they will experience at the  checkpoint or may deny them an opportunity to board the aircraft. The  program requires U.S.- and foreign-flagged commercial aircraft operators  traveling to, from, within, or overflying the United States, as well as U.S.  commercial aircraft operators with international point-to-point flights, to  collect certain information from passengers—such as full name, gender,  and date of birth—and transmit that information electronically to TSA.  The Secure Flight program then identifies passengers’ risk levels by  matching them against federal government watch lists—for example, the  No Fly List, comprised of individuals who should be precluded from  boarding an aircraft, and the Selectee List, comprised of individuals who  should receive enhanced screening at the passenger security  checkpoint. Passengers identified as matching the No Fly List, for  example, are precluded from obtaining a boarding pass and proceeding  through the screening checkpoint. For passengers matching the Selectee  List, air carriers must mark their boarding passes accordingly so TSA can  identify them for enhanced screening.

In 2010, TSA began using risk-based criteria to create additional lists for  Secure Flight screening, which are composed of high-risk passengers  who may not be in the Terrorist Screening Database but whom TSA has  determined should be subject to enhanced screening procedures. TSA  also began conducting watch list matching against an Expanded Selectee  List in order to designate more passengers who are known or suspected  terrorists as selectees for enhanced screening. In addition, as part of  TSA Pre✓™—a 2011 initiative to preapprove passengers for expedited  screening—TSA uses Secure Flight to screen passengers against several  lists of preapproved low-risk travelers. Passengers determined to be  eligible for TSA Pre✓™ are identified as such on their boarding passes.

Checkpoint Screening: TSA screens individuals and property at airport  screening checkpoints to deter and prevent the carriage of any  unauthorized or prohibited items on board an aircraft or into the airport  sterile area. In general, passengers undergo one of three types of  checkpoint screening, based on the Secure Flight determinations shown  on boarding passes—standard screening, enhanced screening for  selectees, and expedited screening for low-risk passengers. Standard  screening typically includes passing through a walk-through metal  detector or advanced imaging technology (AIT) machine, which identifies  objects or anomalies on the outside of the body. Passengers may also  be subject to a pat down if they are screened by the AIT or walk-through  metal detector and the equipment alarms. Standard screening also  typically includes X-ray screening for the passenger’s accessible  property. During X-ray examination of the property, TSOs review the X- ray images, and if potential prohibited items are detected, the property will  be manually inspected and screened with an explosives trace detection  (ETD) machine to identify any traces of explosives material. Enhanced  screening generally includes, in addition to the procedures applied during  a typical standard screening experience, a pat-down and an explosives  trace detection or physical search of the interior of the passenger’s  accessible property, electronics, and footwear. Expedited screening  typically includes walk-through metal detector screening and X-ray  screening of the passenger’s accessible property, but unlike in standard  screening, travelers do not have to, among other things, remove their  belts, shoes, or light outerwear.

Checked Baggage Screening: TSA inspects passengers’ checked  baggage to deter, detect, and prevent the transport of any unauthorized  explosive, incendiary, or weapon onboard an aircraft. Checked baggage  screening is accomplished through the use of explosives detection  systems (EDS)—which use X-rays with computed tomography technology  to automatically measure the physical characteristics of objects in  baggage and trigger an alarm when objects that exhibit the physical  characteristics of explosives are detected—and ETD machines, which  use chemical analysis to manually detect traces of explosive materials’  vapors and residue. At airports with EDS, EDS machines are generally  employed for primary screening of checked baggage while ETD machines  are used for secondary screening to help resolve questions raised by  EDS screening. At airports without EDS machines, ETDs are used as the  primary method for screening checked baggage.

Explosives Detection Canines: TSA’s National Explosives Detection  Canine Team Program trains, deploys, and certifies explosives detection  canine teams in order to deter and detect the introduction of explosive  devices into U.S. transportation systems. Each canine team consists of a  handler paired with a canine trained in explosives detection. The canine  handlers are generally either a state or local law enforcement officer  (LEO) or a TSA employee. Two types of LEO teams and two types of  TSA-based teams were trained to operate in the aviation environment  during fiscal year 2015. First, TSA explosives detection canine teams  patrol terminals, curbside areas, and other airport environments while  TSA passenger screening canine teams primarily search for explosives  odor on passengers in airport terminals. Second, LEO aviation teams  patrol airport terminals, curbside areas, and sterile areas while LEO  multimodal teams operate in the airport environment and screen air cargo  but also operate in mass transit and maritime environments.

Behavior Detection and Analysis: TSA’s BDA program employs  behavior detection officers (BDO) at passenger screening checkpoints to  identify potential threats by observing individuals for certain behavioral  indicators—behaviors indicative of stress, fear, or deception.  These  behavioral indicators include, for example, assessing the way an  individual swallows or the degree to which an individual’s eyes are open.  According to TSA, these verbal and nonverbal cues and behaviors may  indicate mal-intent, such as the intent to carry out a terrorist attack, and  provide a means for TSA to identify passengers who may pose a risk to  aviation security and refer them for additional screening. During this  referral screening, if passengers exhibit additional such behaviors, or if  other events occur, such as the discovery of a suspected fraudulent  document, BDOs are to refer these passengers to a LEO for further  investigation. In fiscal year 2015, the program deployed BDOs primarily in  teams of two at passenger screening checkpoints. However, TSA officials  reported that in the summer of 2016, the agency began taking steps to  integrate BDOs into the TSO workforce by assigning BDOs to the travel  document checker position and other positions at passenger screening  checkpoints where they are able to observe and interact with passengers  in the performance of their screening duties.

U.S. Federal Air Marshal Service: FAMS deploys federal air marshals  on passenger flights to detect, deter, and defeat hostile acts targeting  U.S. air carriers, airports, passengers, and crews. In accordance with  ATSA, as amended, TSA is authorized to deploy federal air marshals on  every passenger flight of a U.S. air carrier and is required to deploy  federal air marshals on every such flight determined by the Secretary of  Homeland Security to present high security risks, with nonstop, long- distance flights, such as those targeted on September 11, 2001,  considered a priority. One of FAMS’s top priorities is to deploy air  marshals on flights that have a known or suspected terrorist on board.  When FAMS assigns air marshals to cover such flights, it refers to these  flights as special mission coverage assignments.

TSA’s System of Aviation Security Countermeasures

TSA uses a risk management strategy—referred to as “layers of  security”—whereby TSA simultaneously deploys a mix of screening and  other security countermeasures to deter and detect threats. TSA deploys  countermeasures in varying combinations at each airport based on  available resources, specific security concerns, and the airport’s risk  category, among other things. Since the terrorist attacks of September  11, 2001, TSA has implemented and added countermeasures, and  refined security procedures in response to specific attacks or threats— such as the liquid explosives plot in 2006. Figure 2 depicts examples of  this progression, illustrating the addition or enhancement of certain TSA  countermeasures over the years.

TSA Has Effectiveness Data on Some Countermeasures That Show Mixed Results, But Does Not Measure Deterrence

Data on the Effectiveness of Selected Countermeasures in Detecting and Disrupting Threats to Aviation Security Vary in Extent and Reliability

TSA collected fiscal year 2015 data on the effectiveness of four of the six  countermeasures we selected—passenger prescreening, checkpoint  screening, checked baggage screening, and explosives detection  canines—in detecting or disrupting threats to passenger aviation  security. TSA assesses this effectiveness differently for each of these  four countermeasures. For example, TSA assessed the effectiveness of  its passenger prescreening countermeasure in detecting passengers that  may pose a threat to aviation security by measuring the percentage of  airline passenger records vetted through its Secure Flight system and the  number of high-risk passengers identified. In contrast, TSA assessed the  effectiveness of its canine program in detecting and disrupting potential  security threats by measuring canine-handler team performance during  their annual certification tests as well as covert scenario-based tests  called short notice assessments (SNA).

Some of the effectiveness data TSA has for fiscal year 2015 are of limited  reliability and TSA is taking steps to improve this information. For  instance, we reported in September 2016 that checkpoint and checked  baggage screening effectiveness data from TSA’s Aviation Screening  Assessment Program (ASAP) Advantage covert tests conducted in fiscal  year 2015 were not reliable. Specifically, TSA found that TSOs  performed more poorly in ASAP tests conducted by an independent  contractor than in the same tests conducted by local TSA personnel at  the same airports. This raised questions about the validity of ASAP tests  conducted by local TSA personnel and indicated that TSA’s fiscal year  2015 ASAP pass rates likely showed a higher level of TSO performance  in screening for prohibited items than was actually the case. In response  to this issue, and to provide ongoing quality assurance for field-based  covert testing results, in April 2016, TSA began deploying headquarters- based covert testing teams in both the checkpoint and checked baggage  screening environments. TSA officials stated that comparing the results  of field- and headquarters-based tests provides TSA with a useful  indication of whether or not the field-based covert testing results are  valid.

In another example, we determined that fiscal year 2015 SNA data were  not reliable for the purpose of reporting explosives detection canine  teams’ covert testing pass rates. Specifically, in the course of our review  we found that these data included duplicate entries and errors, and TSA  officials stated that the results of an unknown number of SNAs may not  have been recorded. Further, we found that TSA’s data collection process  for SNA results that were recorded lacked procedures to ensure that  manually entered data were accurate and complete. To address these  data limitations, canine program officials stated that a new process was  implemented in October 2016 to incorporate SNA results directly into the  Canine Website System—a central electronic management database for  various canine program data. According to these officials, this new  process will better ensure that SNA data are complete, accurate, and  reliable for use by program officials and TSA leadership in evaluating the  effectiveness of the program. Appendix II presents specific fiscal year  2015 effectiveness data for the four selected countermeasures for which  TSA had effectiveness information.

During fiscal year 2015, TSA did not collect data on the effectiveness of  two of the six countermeasures we selected—FAMS and the BDA  program—in detecting and disrupting threats to aviation security. For  FAMS, TSA officials explained that it is very difficult to empirically  measure the effectiveness of federal air marshals and the program has  no efforts underway to collect such data. We discuss this issue later in  this report.

For the BDA Program, we reported in November 2013 that TSA had not  demonstrated that BDOs could consistently identify the behavioral  indicators and, further, that decades of peer-reviewed, published research  on the complexities associated with detecting deception through human  observation also called into question the scientific basis for TSA’s  behavior detection activities. As a result, we recommended that TSA  limit future funding for the agency’s behavior detection activities until TSA  can provide scientifically validated evidence that demonstrates that  behavioral indicators can be used to identify passengers who may pose a  threat to aviation security. DHS did not concur with the recommendation  but has since reduced funding for the BDA Program and taken steps to  begin to assess program effectiveness. For example, in 2014 TSA  revised its list of behavioral indicators and contracted for a literature  review to identify additional sources of evidence supporting these  indicators. However, in July 2017, we reported that in our review of all  178 sources TSA cited in support of its revised list, we found that 98  percent (175 of 178) did not provide valid evidence applicable to the  specific indicators TSA identified them as supporting. Based on our  findings, we continue to believe that TSA should limit future funding for  the agency’s behavior detection activities until TSA can provide valid  evidence that demonstrates that behavioral indicators can be used to  identify passengers who may pose a threat to aviation security, as we  recommended in our November 2013 report.

Table 1 identifies whether TSA has information on the effectiveness of the  six selected countermeasures in detecting and disrupting threats to  aviation security during fiscal year 2015, the data limitations we identified,  and steps TSA officials have taken to improve this effectiveness  information.

TSA Effectiveness Data on Selected Countermeasures Indicate Mixed Results

Some of TSA’s fiscal year 2015 data indicate countermeasure  effectiveness while other data highlight vulnerabilities in the agency’s  ability to detect and disrupt threats to aviation security. For example, for  the passenger prescreening countermeasure, TSA officials reported that  in fiscal year 2015, TSA’s Secure Flight program vetted 100 percent of  the more than 816 million records of passengers who flew into, out of,  over, or within the United States, and on U.S.-flagged aircraft operating  internationally point-to-point. In addition, for the checkpoint and checked  baggage countermeasures, TSA uses Annual Proficiency Reviews (APR)  to evaluate TSOs’ skill in performing various checkpoint and checked  baggage screening functions, such as pat downs of passengers, bag  searches, and use of explosives detection equipment. In 2015, the  average rate at which TSOs passed all APR component tests on the first  try was nearly 95 percent.

On the other hand, some fiscal year 2015 effectiveness data indicate  vulnerabilities. For example, results from covert testing conducted by  TSA’s OOI during fiscal year 2015 indicate vulnerabilities in the  checkpoint and checked baggage screening systems. Specific details  about OOI’s test results are omitted because the information is classified.

TSA Does Not Measure Deterrence for Any of Its Aviation Security Countermeasures

While TSA has methods to measure its effectiveness in detecting and  disrupting threats, the agency has no such methods to measure progress  toward its goal of deterring attacks on the U.S. aviation system. TSA  officials have cited the deterrent effect of various countermeasures— including FAMS, canine teams, BDOs, and AIT machines—but does not  have information on the deterrent effect of any of these countermeasures.  For example, TSA officials explained that canine teams that patrol  airports—searching unattended bags and unattended vehicles, among  other activities—provide a deterrent presence at airports, but officials  noted that they do not have any data on these canines’ deterrent effect.  Most notably, with regard to FAMS, TSA officials explained that one of the  primary security contributions and a key aspect of the FAMS’s mission is  to deter attacks. However, FAMS officials explained that they do not have  information on FAMS’s deterrent effect because it is difficult to model,  measure, and quantify. TSA officials in multiple offices explained that this  difficulty applies not just to FAMS, but also to other TSA countermeasures  with an intended deterrent effect.

OMB and GAO have acknowledged the difficulty in measuring the effect  of deterrence programs, but have identified options to overcome these  challenges. OMB guidance recognizes that programs with a deterrence or  prevention focus can be difficult to measure and suggests that proxy  measures that are closely tied to the outcome can be used to determine  how well a deterrence process is functioning. We have similarly  acknowledged such methodological challenges and identified alternate  evaluation methods that could be helpful to agencies, such as using  simulations. TSA could, for example, develop theoretical game  scenarios and have testers simulate would-be attackers’ decisions when  attempting to carry out an attack on the aviation system. Officials with  CREATE—a DHS-funded research center—told us that they have  conducted some conceptual research on the value of deterrence and  believe it would be possible to assess TSA’s deterrent effect by, for  example, allowing covert testers to choose their method of attack. Such  an assessment could provide TSA with insights regarding which  countermeasures a would-be attacker might choose to avoid in various  scenarios.

In a March 2016 report prepared for TSA, CREATE analyzed a  prospective risk-based security initiative TSA had begun developing and  highlighted the need for further research into deterrence including the  need to model the economic value of deterrence. CREATE officials  explained that they highlighted this issue because in a resource  constrained environment, optimizing TSA’s deterrent effect may be a  more cost effective solution to aviation security threats than focusing  solely on detection and interdiction. A senior official with CPER stated  that the office believes there is value in pursuing further research  regarding deterrence and noted that the office had included a request for  funding to study deterrence in its fiscal year 2017 expenditure plan, but  the request was on hold due to limited funding.

In accordance with GPRA, as updated by the GPRA Modernization Act,  agencies are to establish performance measures to assess progress  toward goals. Measuring performance allows organizations to track the  progress they are making toward their goals and gives managers critical  information on which to base decisions for improving their progress. For example, they can use performance information when developing  strategies, allocating resources, identifying problems, and taking  corrective action.

TSA officials told us that developing a means to assess TSA’s deterrent  effect would be difficult and require a multi-year effort but having such a  means would be helpful. For example, TSA’s prior Chief Risk Officer told  us that TSA’s countermeasures deter nefarious actors from attempting an  attack on an aircraft, but better understanding this concept will be critical  to TSA in its transition into a more holistic, system-wide approach to  aviation security. Additionally, a senior ORCA official explained that a  better understanding of the deterrent effect of TSA countermeasures  could help TSA optimize use of its resources. For example, this official  noted that there may be a point at which adding additional federal air  marshals has diminishing returns in terms of deterrence and better  understanding FAMS’s deterrent effect could help TSA identify that point.  This official further stated that developing a method to assess deterrence  for this purpose would be challenging but feasible.

In the absence of any systematic or methodological approach to  assessing TSA’s deterrent value, TSA officials have relied on theories of  causality and limited evidence available from U.S. intelligence sources.  For example, FAMS officials cited the fact that there has not been a  hijacking on a U.S. carrier since 2002 as evidence of FAMS’s deterrent  effect, but had no specific evidence to support FAMS’s contribution to this  outcome. In another example, ORCA officials noted that a 2014 article in  an online magazine published by al-Qaeda encouraging would-be- attackers to avoid airports with a certain countermeasure provided  evidence of its deterrent value. These observations may provide limited  insight into TSA’s deterrent effect, but developing a method to  systematically assess the deterrent effect of TSA’s security efforts would  better position TSA to improve progress toward its goal—deterring  attacks on the U.S. aviation system.

TSA Can Compare the Effectiveness of Certain Combinations of Aviation Security Countermeasures, but Does Not Systematically Analyze Cost and Effectiveness Tradeoffs Across All Countermeasures TSA Has a Tool to Assess the Security Effectiveness of Alternate Combinations of Some Countermeasures

In 2014, TSA’s ORCA began using a Risk and Trade Space Portfolio  Analysis Tool (RTSPA) to analyze the security effectiveness of alternate  combinations of some aviation security countermeasures for the purpose  of informing TSA acquisition and deployment decisions. RTSPA provides  a means for TSA to model its security effectiveness in different scenarios.  For example, the tool could be used to compare the security effectiveness  of a theoretical airport screening checkpoint with canines to that of a  checkpoint modeled without canines.

According to ORCA officials, they developed RTSPA to assess security  effectiveness tradeoffs among countermeasures that they believed would  most benefit from the detailed quantitative analyses that the tool provides,  rather than across TSA’s entire system of aviation security  countermeasures. Specifically, TSA officials explained that RTSPA is  designed to analyze tradeoffs among checkpoint screening  countermeasures—including canine teams and BDOs—and checked  baggage screening, but was not developed to analyze tradeoffs among  other countermeasures TSA deploys. For example, ORCA officials told us  that the tool was not developed to analyze crew vetting or FAMS because  understanding the security tradeoffs of these countermeasures, while  important, does not require the use of such a resource intensive tool like  RTSPA. In addition, RTSPA does not account for the full system of  aviation security countermeasures, including countermeasures such as  hardened cockpit doors and Federal Flight Deck Officers—flight crew  members authorized and trained to use firearms. ORCA officials further  explained that in 2014, when initially developing the tool, they also  developed comparable countermeasure cost data to allow for cost- effectiveness comparisons among countermeasures. However, ORCA  officials report that they subsequently stopped analyzing cost tradeoffs  because they believed other TSA offices could conduct such analysis.

In the last two years, TSA officials have used the results of RTSPA  analyses to inform some resource tradeoff decisions. For example, ORCA  officials told us that in 2015, TSA leadership used the results of a RTSPA  analysis when considering options for improving overall security  effectiveness at airports that did not have AIT machines. Specifically, TSA  used RTSPA to consider the level of risk and potential risk mitigation  value of alternative security measures at these airports. TSA officials  report that this RTSPA analysis contributed to TSA’s decision to deploy  146 additional AIT machines to such airports. In another example, ORCA  officials noted that in early 2017, they used RTSPA to analyze options for  resolving checked baggage alarms, taking into consideration the relative  risks of military-grade explosive materials and homemade explosive  devices.

TSA officials stated that their use of RTSPA has been limited to date  because it is still a relatively new tool. However, ORCA officials told us  that they expect use of the tool’s analysis to grow as the agency  increasingly seeks to use analytic tools to inform acquisition and  deployment decisions. As such, ORCA officials plan to update RTSPA  and expand its analytical capabilities.

TSA Has Not Systematically Analyzed Potential Cost and Effectiveness Tradeoffs Across the Entire System of Aviation Security Countermeasures

TSA does not have any efforts underway to systematically evaluate the  potential cost and effectiveness tradeoffs across the full aviation security  system. Although TSA’s use of RTSPA to identify effectiveness tradeoffs  among selected countermeasures provides some such information, the  tool’s analyses are limited and the tool is not designed to offer a system- wide view of effectiveness. When we asked TSA’s prior Chief of Staff  about any such efforts, he stated that TSA had not systematically  evaluated cost and effectiveness tradeoffs because TSA’s aviation  security system is constantly evolving to meet emerging threats, and  assessing a system in flux is challenging. However, he told us that such  an analysis would be helpful.

DHS policy and TSA’s strategic plan call for the systematic evaluation of  the costs and effectiveness of TSA’s chosen mix of aviation security  countermeasures. Specifically, DHS’s 2010 Policy for Integrated Risk  Management calls on components, including TSA, to evaluate the  performance of risk management strategies it decides to implement. In  the case of TSA, TSA’s chosen mix of aviation security countermeasures  represents TSA’s current risk management strategy. The policy further  establishes that components should develop and analyze alternative  strategies to manage risks by considering the projected costs, benefits,  and ramifications of each alternative. In addition, TSA’s current Strategic  Plan establishes the goal of increasing efficiency and operational  effectiveness through disciplined processes and dynamic resource  management. One of the stated outcomes associated with this goal is  the ability to effectively optimize resource allocation to strike a balance of  costs, benefits, and risk. In addition, it was the stated objective of ORCA’s  predecessor—the Office of Security Capabilities (OSC)—to develop and  implement a comprehensive tradeoff analysis across the security system  to inform investment decisions. OSC’s strategic plan further states that  such an analysis would include a full set of strategic choices TSA should  consider when determining how to respond to a threat or making an  investment decision, helping to determine which alternatives provide the  greatest risk mitigation value for each dollar spent.

A senior ORCA official explained that while there is a need for a system- wide tradeoff analyses, RTSPA alone may not be the right tool for this.  This official explained that TSA may not require detailed quantitative  analyses from a resource-intensive tool such as RTSPA to understand  the effectiveness tradeoffs among all aviation security countermeasures,  and a portfolio of tools of varying precision and depth could be used to  obtain a system-wide view. This official noted that developing TSA’s  capability for system-wide tradeoff analysis would be challenging and  require a multi-year effort. However, RTSPA could serve as a useful  starting place for a more comprehensive system-wide analysis. For  example, TSA could build upon ORCA’s past efforts to analyze the  comparative cost effectiveness of countermeasures and its experience  isolating the security effectiveness contributions of individual  countermeasures.

Without a systematic analysis of the cost and effectiveness tradeoffs  across aviation security countermeasures TSA is limited in its ability to  achieve its stated goal of optimizing resource allocation and striking a  balance of costs, effectiveness, and risk across the system. In an  environment of constrained resources and continuing threats to aviation  security, producing such analysis could assist TSA leadership in targeting  its limited resources to achieve the greatest system-wide risk mitigation  value for each dollar spent.

Since the terrorist attacks of September 11, 2001, TSA has spent billions  of dollars on a range of aviation security programs with the goal of  detecting, disrupting, and deterring threats. However, TSA does not have  a complete understanding of the contributions these programs are making  to this goal. Specifically, TSA has some information on how well it can  detect and disrupt threats and is taking steps to improve this information,  but does not have information on its ability to deter attacks—a key  component of TSA’s goal. For example, in fiscal year 2015, TSA spent  approximately $800 million on FAMS—a program with a focus on  deterring attacks on aircraft—yet the agency has no information on its  effectiveness in doing so. While we and OMB have acknowledged the  difficulty in measuring deterrence, we have also suggested options to  overcome these challenges. Further, in accordance with GPRA, as  updated by the GPRA Modernization Act, agencies are to assess the  effectiveness of their programs and leading practices established in  GAO’s prior work stress the importance of agencies tracking progress  toward goals. Developing a method to assess the deterrent effect of  aviation security countermeasures would better position TSA to improve  progress toward a key goal—deterring attacks on the U.S. aviation  system.

Since September 11, 2001, TSA has added countermeasures and refined  security procedures in response to specific attacks or threats, but has not  systematically evaluated its chosen combination of aviation security  countermeasures as called for in DHS policy and TSA’s strategic plan.  Specifically, TSA does not have any efforts underway to evaluate the  potential cost and effectiveness tradeoffs across the full aviation security  system because, according to a senior TSA official, the aviation security  system is constantly evolving in response to emerging threats, and  assessing a system in flux is challenging. However, it is using a model— known as RTSPA—that could serve as a useful starting place for a more  comprehensive system-wide analysis. Developing and implementing a  means to systematically evaluate the potential cost and effectiveness  tradeoffs across aviation security countermeasures would better position  TSA to achieve its stated goal of optimizing resource allocation and  striking a balance of costs, effectiveness, and risk. In an environment of  constrained resources and continuing threats to aviation security,  producing such an analysis could assist TSA leadership in targeting its  limited resources to achieve the greatest system-wide risk mitigation  value for each dollar spent.

We recognize that developing these analytical methods will be a difficult  undertaking that may take years to achieve. Nonetheless, as TSA  improves the reliability and extent of its countermeasure effectiveness  data, the agency will also improve its ability to perform system-wide cost  and effectiveness tradeoff analyses. In this high threat environment, it is  essential that TSA determine how to allocate its finite resources to best  position the agency to detect, disrupt and deter threats to aviation  security.

We are making the following two recommendations to TSA:   1.  The Administrator of TSA should explore and pursue methods to  assess the deterrent effect of TSA’s passenger aviation security  countermeasures; such an effort should identify FAMS—a  countermeasure with a focus on deterring threats—as a top priority to  address. (Recommendation 1)  2.  The Administrator of TSA should systematically evaluate the potential  cost and effectiveness tradeoffs across countermeasures, as TSA  improves the reliability and extent of its information on the  effectiveness of aviation security countermeasures.  (Recommendation 2)

We provided a draft of this report to DHS for review and comment. The  department’s letter is included in appendix III. In its comments, DHS  generally concurred. DHS also provided technical comments, which we  incorporated as appropriate.

With regard to our first recommendation that TSA explore and pursue  methods to assess the deterrent effect of its passenger aviation security  countermeasures, DHS concurred, noting that this may require proxy or  output measures and assumptions about potential adversary choices.  DHS also concurred with our second recommendation that TSA  systematically evaluate the potential cost and effectiveness tradeoffs  across countermeasures. In its comments, DHS stated that TSA will  continue efforts to improve both its analysis of information related to  security effectiveness and its cost information, leading to better informed  cost-benefit decisions for individual countermeasures. To address the  intent of our recommendation, TSA will need to evaluate the costs and  effectiveness of individual aviation security countermeasures and then  use this information to systematically evaluate the potential cost and  effectiveness tradeoffs across countermeasures. We will continue to  monitor TSA’s efforts in addressing these recommendations.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of the Department of Homeland Security, and  other interested parties. In addition, the report is available at no charge on  the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7141 or GroverJ@gao.gov. GAO staff who made key  contributions to this report are listed in appendix IV.

Appendix I: Transportation Security Administration (TSA) Information on the Direct Costs of Selected Countermeasures

As part of this review, we analyzed TSA’s fiscal year 2015 cost data for  six selected aviation security countermeasures—passenger prescreening  (Secure Flight), checkpoint screening, checked baggage screening,  explosives detection canines, the Behavior Detection and Analysis (BDA)  program, and the U.S. Federal Air Marshal Service (FAMS). We selected  these six passenger aviation security countermeasures because they  involved direct interaction with passengers, their belongings, or their  personal information and are largely operated and funded by TSA. We  determined that TSA can generally identify the fiscal year 2015 direct  costs to TSA of the six passenger aviation security countermeasures that  we reviewed, as shown in Table 2. TSA generally does not budget or  track costs by countermeasure, but is able to identify most direct costs  from their financial management system. For those passenger aviation  security countermeasures that align with TSA’s budget categories, such  as FAMS and passenger prescreening, TSA can run a single report to  obtain the direct cost information. However, for those countermeasures  that do not align with TSA’s budget categories, such as checkpoint  screening and checked baggage screening, TSA is able to run multiple  reports and use estimation based on their staffing model to estimate the  direct costs.

Appendix II: Fiscal Year 2015 Effectiveness Data for Selected Passenger Aviation Security Countermeasures

The Transportation Security Administration (TSA) collected fiscal year  2015 data on the effectiveness of four of the six countermeasures we  selected—passenger prescreening, checkpoint screening, checked  baggage screening, and explosives detection canines. These data show  mixed results with some data indicating TSA countermeasure  effectiveness and other data highlighting vulnerabilities. Below, we  describe what TSA knows about the fiscal year 2015 effectiveness of  these four countermeasures in detecting or disrupting threats to  passenger aviation security.

TSA uses its Secure Flight prescreening program to match passenger  information against federal government watch lists and other information  to assign each passenger to one of three risk categories—high risk, low  risk, or unknown risk—that either corresponds to the level of screening  they will experience at the checkpoint or may deny them an opportunity to  board the aircraft. Since TSA began implementing Secure Flight in 2009,  the passenger prescreening program has changed from a program that  identifies passengers as high risk solely by matching them against federal  government watch lists—for example, the No Fly List, comprised of  individuals who should be precluded from boarding an aircraft, and the  Selectee List, comprised of individuals who should receive enhanced  screening at the passenger security checkpoint—to one that uses  additional lists and risk-based criteria to assign passengers to a risk  category. Specifically, Secure Flight now identifies passengers as high  risk if they are matched to watch lists of known or suspected terrorists or  other lists developed using certain high-risk criteria and as low risk if they  are deemed eligible for expedited screening through TSA Pre✓™—a  2011 initiative to preapprove passengers for expedited screening—or  through the application of low-risk rules. Secure Flight identifies  passengers as unknown risk if they do not fall within the other two risk  categories.

To separate passengers into these risk categories, TSA utilizes lists in  addition to the No Fly and Selectee Lists, and TSA has adapted the  Secure Flight system to perform risk assessments, a system functionality  that is distinct from both watch list matching and matching against lists of  known travelers. At airport checkpoints, those passengers identified as  high risk receive enhanced screening, passengers identified as low risk  are eligible for expedited screening, and passengers identified as  unknown risk generally receive standard screening. Passengers matched  to the No Fly List or the Centers for Disease Control and Prevention’s Do  Not Board List—a list which includes individuals who pose a significant  health risk to other travelers and are not allowed to fly—are considered  highest risk, and thus are not to receive boarding passes, and should not  be allowed entry into the sterile area. Figure 3 illustrates this passenger  prescreening process.

The Effectiveness of Passenger Prescreening in Fiscal Year 2015

TSA officials reported that the percentage of passengers vetted and the  number of high-risk passengers identified by Secure Flight demonstrate  the effectiveness of this passenger prescreening program. Specifically,  TSA data indicate that in fiscal year 2015, Secure Flight vetted 100  percent of the over 816 million records submitted for passengers who flew  into, out of, over, or within the United States, and on U.S.-flagged aircraft  operating internationally point-to-point. Of these, TSA identified 15,383  (0.002 percent of passenger records vetted) as confirmed matches to  watch lists. Specifically, in fiscal year 2015, TSA identified 9,639  passengers as expanded selectees, 5,019 passengers on the Selectee  List, and 725 passengers on the No Fly List.

In September 2014, we reported that TSA collects and regularly reviews  data on the number of passengers identified by the Secure Flight system  as potential matches to the No Fly, Selectee, and Expanded Selectee  Lists. However, we found that TSA did not measure the extent to which  Secure Flight was missing passengers who were actual matches to these  lists—false negatives. We recommended that TSA establish such  measures. In response, in August 2016, TSA contracted with a third party  to conduct an independent assessment of the effectiveness of the Secure  Flight automated vetting system including whether Secure Flight identifies  the matches it should (i.e., how well the system minimizes false  negatives). TSA officials expect this assessment to be complete at the  end of calendar year 2017.

TSA ensures that all individuals and accessible property are screened as  part of its checkpoint screening process to deter and prevent the carriage  of any unauthorized explosive, incendiary, weapon, or other prohibited  items on board an aircraft or into the airport sterile area—in general, an  area of an airport that provides passengers access to boarding aircraft  and to which access is controlled through the screening of persons and  property. Ordinarily, screening of accessible property at the screening  checkpoint begins when an individual places accessible property on the X-ray conveyor belt or hands accessible property to a Transportation  Security Officer (TSO). As shown in figure 4, TSOs then review images  of the property running through the X-ray machine and look for signs of  prohibited items. If a TSO identifies a potential prohibited item, the  accessible property will be manually inspected and screened with an  explosives trace detection (ETD) machine to identify any traces of  explosives material. The passengers themselves are typically screened  via a walk-through metal detector or an advanced imaging technology  (AIT) machine—often referred to as a full-body scanner—and passengers  generally have the option to request screening by a pat down if they do  not wish to be screened by these technologies. Passengers will also be  subject to a pat down if they are screened by a walk through metal  detector or the AIT and the equipment alarms (in order to resolve the  alarm).

TSOs use several screening technologies in order to screen passengers  and carry-on bags for prohibited items. For more information on the  specific screening technologies deployed at the checkpoint in fiscal year  2015, see Table 3.

The Effectiveness of Checkpoint Screening in Fiscal Year 2015

In fiscal year 2015, TSA collected data on the effectiveness of checkpoint  screening by testing TSOs, screening technology (e.g., the AIT and X- ray), and the checkpoint screening system as a whole (i.e., the  combination of TSOs and technology).

TSA collected fiscal year 2015 data on the effectiveness of its TSO  workforce in detecting or disrupting threats to aviation security at the  checkpoint in three ways: (1) annual proficiency review (APR) of TSOs,  (2) threat-image projection (TIP) testing, and (3) Aviation Screening  Assessment Program (ASAP) Advantage covert tests.

Annual Proficiency Reviews. APRs evaluate TSOs’ skill in performing  the various checkpoint and checked baggage screening functions and all  TSOs must successfully complete the required APR component tests  related to their job function on an annual basis as a condition of  employment with TSA in their capacity as a screener. Components of  the APR focused on checkpoint screening specifically included tests that  evaluate TSOs’ ability to identify prohibited items on an X-ray machine  and tests that evaluate whether TSOs can perform various practical skills  such as pat downs, bag searches, and use of explosive trace detection  technology.

In calendar year 2015, TSA conducted roughly 150,000 APR component  tests focused on checkpoint screening. Table 4 provides descriptions of  these component tests.

Threat Image Projection (TIP) Testing. TSA’s TIP testing system  displays fictional threat items, such as guns or explosives, onto X-ray  images of actual passengers’ carry-on bags to test TSOs’ ability to  identify prohibited items in a live operational environment. TSOs  operating the X-ray machine at the checkpoint are monitored to see if  they positively identify the threat image and call for the bag to be  searched. TSA officials report that they use TIP images on a daily basis  to monitor TSOs’ ability to identify prohibited items, aid in keeping them  focused and attentive, and keep their skills sharp in identifying items they  do not routinely see. TSA requires airport personnel to conduct TIP  tests and upload monthly results data into TSA’s national database.

In September 2016, we reported that TSA’s TIP data from fiscal year  2009 through 2014 was incomplete as TSA could not provide TIP scores  for every airport during this period. Specifically, during fiscal year 2013,  nearly 14 percent of airports failed to report any TIP data. TSA officials  also acknowledged that, in addition to the airports that did not report any  TIP data for a year or more at a time, other airports may have reported  only partial TIP results data during this same time frame. We  recommended that TSA officials at individual airports submit complete  TIP results to the TSA national database as required and, further, that  TSA analyze national TIP data for trends that could inform training needs  and improve future training and TSO performance assessments. TSA  concurred with our recommendations and is taking steps to address  them. Specifically, a new TIP Operations Directive was implemented in  October 2016 to disseminate procedures for performance data collection  and submission to improve TIP data. According to agency officials, the  number of non-compliant airports decreased during fiscal year 2016.  However, since these improvements occurred during fiscal years 2016  and 2017, fiscal year 2015 TIP data remained incomplete and unreliable  for the purposes of assessing TSO’s effectiveness at identifying TIP  images. Therefore, we do not present fiscal year 2015 TIP test results in  this report.

Aviation Screening Assessment Program (ASAP) Advantage  Testing. To measure TSO performance nationwide in fiscal year 2015,  TSA used standardized ASAP covert tests conducted by local TSA  testers at each airport. ASAP tests focused on checkpoint screening were  designed to assess the operational effectiveness of TSOs in identifying  and preventing prohibited items, such as knives, guns, or simulated  improvised explosive devices, from being taken through the checkpoint by  testers. In fiscal year 2015, TSA conducted 5,213 ASAP covert tests on  checkpoint screening at 170 airports.

TSA hired a contractor in fiscal year 2015 to independently conduct ASAP  standard scenario tests at 40 airports to assess the validity of TSA testing  results at those airports. When comparing the contractor’s results to the  local TSA testers’ results, TSA found moderate to significant differences  in the two sets of test results for most of the 40 airports. According to TSA  officials, TSOs generally performed more poorly in the ASAP tests  conducted by the independent contractor personnel when compared to  the ASAP testing conducted by the local TSA personnel—indicating that  pass rates for tests conducted by local TSA personnel were likely  showing a higher level of TSO performance than was actually the case.  TSA officials reported that the differences in test results have led them to  question the extent to which the ASAP tests accurately measure TSO  performance. As a result, we do not present the fiscal year 2015 ASAP  test results in this report.

To address this validity issue, in April 2016, TSA officials reported that  they began using both headquarters-based covert testing teams  composed of headquarters-based TSA employees and field-based covert  testing teams composed of local testers in both the checkpoint and  checked baggage screening environments at all airports. Both  headquarters-based and field-based teams conduct the same scenario- based covert tests that were previously conducted as part of ASAP  testing. TSA officials stated that comparing the results of these separate  tests has provided TSA with a way to gauge the validity of its test  results.

TSA officials reported that the effectiveness of checkpoint screening  technology in fiscal year 2015 is best described by each type of  machine’s detection standard—the specified rate of detection each  technology is required to achieve in identifying explosives or prohibited  items. Specific details about TSA’s detection standards are omitted  because the information is classified. Prior to acquiring and deploying a  potential new screening technology, TSA conducts testing to evaluate  whether potential technologies can effectively achieve the detection  standards required by TSA, among other things.

Once technology is deployed in the airport environment, TSA policy  requires at least daily calibration testing of each individual piece of  technology deployed at the checkpoint—AIT machines, walk through  metal detectors, ETDs, and X-ray machines, among others—to ensure  the technology is functioning properly and able to achieve the required  detection standards. For example, each day when the screening  checkpoint opens, TSOs must ensure that AIT machines successfully  complete an image quality verification, a calibration test, and an  operational test process before they are cleared for screening  operations. TSA policy requires that TSOs record the results of these  tests in logbooks and, further, that any screening equipment that does not  pass daily testing be immediately taken out of service.

TSA’s Checkpoint Screening System as a Whole

In fiscal year 2015, TSA collected data on the effectiveness of its  checkpoint screening system as a whole—including both screening  technology and TSO performance—through Red Team covert testing  conducted by TSA’s Office of Inspection (OOI). In fiscal year 2015, TSA  conducted numerous Red Team covert tests on checkpoint screening at a  random sample of U.S. airports. During passenger checkpoint testing,  each team of inspectors carries threat items, such as simulated explosive  devices, through the passenger checkpoint. If the TSO identifies the  threat item during screening, the inspector identifies him or herself to the  TSO and the test is considered a pass. If the TSO does not identify the  threat item, the inspector proceeds to the sterile area of the airport and  the test is considered a failure. According to TSA, these tests are  designed to approximate techniques that terrorists may use in order to  identify vulnerabilities in the people, processes, and technologies that  comprise the aviation security system. In addition to OOI’s Red Team  testing, in fiscal year 2015 the Department of Homeland Security (DHS)  Office of Inspector General (OIG) also conducted covert tests of certain  TSA checkpoint operations at 8 U.S. airports that use AIT machines to  screen passengers. According to the DHS OIG, the objective of the tests  was to determine the effectiveness of TSA’s AIT, automated target  recognition software (which displays a box around anomalies on a  generic outline of a body), and checkpoint screener performance in  identifying and resolving anomalies and potential security threats at  airport checkpoints. The results of both the OOI Red Team and the DHS  OIG’s covert tests are omitted because the information is classified.

TSA inspects passengers’ checked baggage to deter, detect, and prevent  the transport of any unauthorized explosive, incendiary, or weapon  onboard an aircraft. Checked baggage screening is accomplished  through the use of explosives detection systems (EDS)—which use X- rays with computed tomography technology to automatically measure the  physical characteristics of objects in baggage and automatically trigger an  alarm when objects that exhibit the physical characteristics of explosives  are detected—and explosives trace detection (ETD) machines, in which  TSOs swab baggage and use chemical analysis to manually detect traces  of explosive materials’ vapors and residue.

Generally, a checked baggage screening system at airports with EDS  includes a three-level screening process. First, EDS machines perform  automated screening. If the EDS machine determines that a checked bag  requires additional screening, it sends an alarm to a TSO who performs a  secondary inspection known as On-Screen Resolution by reviewing an  image of the contents of the bag on a computer monitor. If the TSO  cannot resolve the alarm using on-screen resolution tools and determines  a physical bag search is necessary, the bag goes to the Checked  Baggage Resolution Area where a TSO performs a manual inspection of  the bag assisted by an ETD machine.

At the end of fiscal year 2015, TSA had 1,717 EDS machines deployed at  263 airports. At airports without EDS, which are typically smaller airports,  ETD machines are the primary method for manually screening checked  baggage. At the end of fiscal year 2015, TSA had 2,291 ETD machines  deployed at all 437 commercial (i.e., TSA-regulated) airports for primary  or secondary screening of checked baggage.

TSA officials estimate that 25 percent of total TSO time is spent on  checked baggage screening, and in fiscal year 2015, this would be the  full-time equivalent of approximately 11,000 of TSA’s roughly 45,000  TSOs conducting checked baggage screening.

The Effectiveness of Checked Baggage Screening in Fiscal Year 2015

In fiscal year 2015, TSA collected data on the effectiveness of its checked  baggage screening by testing screening personnel (i.e., TSOs), screening  technology (EDS and ETD machines), and the checked baggage  screening system as a whole (i.e., the combination of TSOs and  technology).

In fiscal year 2015, TSA collected data on the effectiveness of its TSO  workforce in detecting or disrupting threats to aviation security in the  checked baggage environment through its APR evaluations and ASAP  Advantage covert tests.

Annual Proficiency Reviews (APR). As discussed above, APRs  evaluate TSOs’ skill in performing the various checkpoint and checked  baggage screening functions. Components of the APR focused on  checked baggage screening include tests that evaluate TSOs’ ability to  resolve EDS machine alarms using the appropriate tools and practical  skills such as bag searches and the use of ETD technology.

In calendar year 2015, TSA conducted nearly 35,000 APR component  tests specific to the checked baggage screening environment. Table 5  provides descriptions of these component tests.

Aviation Screening Assessment Program (ASAP) Advantage. In fiscal  year 2015, TSA used standardized ASAP covert tests conducted by local  TSA testers at each airport to measure TSO performance in both the  checkpoint and checked baggage environments. Tests focused on  checked baggage screening were designed to assess the operational  effectiveness of TSOs in identifying and preventing a threat object  concealed in a checked bag from being cleared for loading onto a  passenger aircraft. In fiscal year 2015, TSA conducted 1,859 ASAP  covert tests on checked baggage screening at 225 airports.

TSA began deploying headquarters-based covert testing teams in fiscal  year 2016 to provide a means to validate the results of covert tests  conducted by local TSA testers for both checkpoint and checked baggage  screening. However, unlike in the checkpoint environment, the  contractor did not perform ASAP covert testing on checked baggage  screening during fiscal year 2015. When we compared fiscal year 2016  headquarters-based and field-based pass rates for covert testing of  checked baggage screening, we found discrepancies that indicate covert  tests conducted by local field-based TSA testers on checked baggage  may not be reliable in accurately portraying TSO performance.  Additionally, TSA officials stated that they cannot be certain these data  are reliable. As a result, we do not present ASAP Advantage data in this  report.

Checked Baggage Screening Technology and TSA’s Checked Baggage Screening System as a Whole

As with checkpoint screening technology discussed above, TSA officials  reported that in fiscal year 2015, technology deployed at airports for  checked baggage screening was calibrated and tested daily to ensure  that it was operating as intended. According to TSA officials, these daily  tests help to ensure that its screening technologies are meeting the  detection standards they were designed to achieve. TSA officials  reported that any equipment found not to meet required detection  standards was immediately taken out of service. As described above,  OOI also conducted Red Team covert testing on checked baggage  screening at airports with EDS machines in fiscal year 2015. Specific  details about TSA’s detection standards and the results of OOI’s covert  tests are omitted because the information is classified.

Through its National Explosives Detection Canine Team Program, TSA  trains, deploys, and certifies explosives detection canine teams in order to  deter and detect the introduction of explosive devices into U.S.  transportation systems. Each canine team consists of a handler— generally either a state or local law enforcement officer (LEO) or TSA  employee—paired with a canine trained in explosives detection.

As of September 2015, TSA had 692 canine teams deployed to 88  airports across the United States. These teams were composed of four  types of canine teams trained to operate in the airport environment: TSA  explosives detection canine (EDC) and Passenger Screening Canine  (PSC) teams as well as LEO aviation and multimodal teams. Table 6  shows the number of canine teams by type deployed in the airport  environment as of September 2015 and describes their roles and  responsibilities.

The Effectiveness of Explosives Detection Canines in Fiscal Year 2015

In fiscal year 2015, TSA collected data on the effectiveness of its canine  teams in detecting or disrupting threats to aviation security through its  annual certification evaluation process and short notice assessments  (SNA)—covert tests conducted to assess canine teams’ operational  effectiveness in detecting and responding to possible explosives.

Annual Certification Evaluations. TSA’s annual evaluations assess  whether canine teams meet the explosives detection certification  standards established by the program. Following initial training, new  canine teams must demonstrate certain critical skills in order to be  certified to work in their home operating environment. After initial  certification, all TSA canine teams are evaluated on an annual basis to  maintain certification. Canine teams that fail their annual evaluation are  decertified and limited to training and operating as a visible deterrent until  they successfully complete the annual evaluation and are recertified to  conduct screening.

To achieve EDC certification, canine teams must demonstrate their ability  to detect hidden explosive training aids across a specified number of  areas, a certain percent of the time. After passing this conventional  evaluation, PSC teams undergo further testing in different locations within  the sterile area of an airport. To achieve PSC certification, canine teams  must successfully identify an explosives-carrying target/decoy in a  specified number of search areas.

In fiscal year 2015, TSA conducted 673 EDC annual certification  evaluations and 116 PSC evaluations. The fiscal year 2015 first-time pass  rates for EDC and PSC canine teams has been designated as sensitive  security information and thus cannot be included in a public report.

Short Notice Assessments. TSA conducts covert testing of canine  teams to measure their effectiveness in detecting and responding to  explosives odor during normal operations. These covert tests, known as  SNAs, are conducted using one of four scenarios chosen to match a  canine team’s primary area of operations—an unattended bag,  unattended vehicle, cargo screening, and passenger screening. Field  Canine Coordinators—TSA officials that administer SNAs—are  responsible for debriefing participants after the assessment, determining  if corrective actions are necessary, and officially documenting outcomes.

We assessed the reliability of SNA results in fiscal year 2015 and  determined that the data were not reliable for the purpose of reporting  overall pass rates. Specifically, we found duplicate entries and errors in  the data. In addition, we found that fiscal year 2015 data on pass rates  may be incomplete since the results of some SNAs may not have been  subsequently recorded in TSA’s system. Further, TSA’s process of  manually recording SNA results in fiscal year 2015 lacked procedures to  ensure that data entered into TSA’s system were accurate and complete.

To address these data limitations, canine program officials stated that a  new process was implemented in October 2016 to incorporate SNA  results directly into the Canine Website System—a central electronic management database for various canine program data. According to  these officials, this new process will better ensure that SNA data are  complete, accurate, and reliable for use by program officials and TSA  leadership in evaluating the effectiveness of the program.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Maria Strudwick (Assistant  Director), Chuck Bausell, Claudia Becker, Bryan Bourgault, Bruce Crise,  Dominick Dale, Brianna Dieter, Michele Fejfar, Eric Hauswirth, Susan  Hsu, James Kernen, and Tom Lombardi made key contributions to this  report.