TSA Modernization: Use of Sound Program Management and Oversight Practices Is Needed to Avoid Repeating Past Problems

TSA conducts security threat assessment screening and credentialing activities for millions of workers and travelers in the maritime, surface, and aviation transportation industries that are seeking access to transportation systems. In 2008, TSA initiated the TIM program to enhance the sophistication of its security threat assessments and to improve the capacity of its supporting systems. However, the program experienced significant cost and schedule overruns, and performance issues, and was suspended in January 2015 while TSA established a new strategy. The program was rebaselined in September 2016 and is estimated to cost approximately $1.27 billion and be fully operational by 2021 (about $639 million more and 6 years later than originally planned).

GAO was asked to review the TIM program's new strategy. This report determined, among other things, the extent to which (1) TSA implemented selected key practices for transitioning to Agile software development for the program; and (2) TSA and DHS are effectively overseeing the program's cost, schedule, and performance. GAO compared program documentation to key practices identified by the Software Engineering Institute and the Office of Management and Budget, as being critical to transitioning to Agile and for overseeing and governing programs.

The Transportation Security Administration's (TSA) new strategy for the Technology Infrastructure Modernization (TIM) program includes using Agile software development, but the program only fully implemented two of six leading practices necessary to ensure successful Agile adoption. Specifically, the Department of Homeland Security (DHS) and TSA leadership fully committed to adopt Agile and TSA provided Agile training. Nonetheless, the program had not defined key roles and responsibilities, prioritized system requirements, or implemented automated capabilities that are essential to ensuring effective adoption of Agile. Until TSA adheres to all leading practices for Agile implementation, the program will be putting at risk its ability to deliver a quality system that strengthens and enhances the sophistication of TSA's security threat assessments and credentialing programs.

TSA and DHS fully implemented one of the key practices for overseeing the TIM program, by establishing a process for ensuring corrective actions are identified and tracked. However, TSA and DHS did not fully implement the remaining three key practices, which impede the effectiveness of their oversight. Specifically,

TSA and DHS documented selected policies and procedures for governance and oversight of the TIM program, but they did not develop or finalize other key oversight and governance documents. For example, TSA officials developed a risk management plan tailored for Agile; however, they did not update the TIM system life-cycle plan to reflect the Agile governance framework they were using.

The TIM program management office conducted frequent performance reviews, but did not establish thresholds or targets for oversight bodies to use to ensure that the program was meeting acceptable levels of performance. In addition, department-level oversight bodies have focused on reviewing selected program life-cycle metrics for the TIM program; however, they did not measure the program against the rebaselined cost, or important Agile release-level metrics.

TIM's reported performance data were not always complete and accurate. For example, program officials reported that they were testing every line of code, even though they were unable to confirm that they were actually doing so, thus calling into question the accuracy of the data reported.

These gaps in oversight and governance of the TIM program were due to, among other things, TSA officials not updating key program management documentation and DHS leadership not obtaining consensus on needed oversight and governance changes related to Agile programs. Given that TIM is a historically troubled program and is at least 6 months behind its rebaselined schedule, it is especially concerning that TSA and DHS have not fully implemented oversight and governance practices for this program. Until TSA and DHS fully implement these practices to ensure the TIM program meets its cost, schedule, and performance targets, the program is at risk of repeating past mistakes and not delivering the capabilities that were initiated 9 years ago to protect the nation's transportation infrastructure.

GAO is making 14 recommendations, including that DHS should prioritize requirements and obtain leadership consensus on oversight and governance changes. DHS concurred with all 14 recommendations.

The nation’s maritime, surface, and aviation transportation systems and  facilities are vulnerable and difficult to secure given their size, easy  accessibility, large number of potential targets, and proximity to urban  areas. In order to help reduce the threats to these critical transportation  systems and facilities, the Transportation Security Administration (TSA), a  component of the Department of Homeland Security (DHS), conducts  various security threat assessment screening and credentialing activities  for millions of workers and travelers seeking access to the maritime,  surface transportation, and aviation industries.

However, as we have previously reported, the effectiveness and the  efficiency of the agency’s threat assessments and credentialing programs  have been hindered by stove-piped information technology (IT) systems  and duplicative processes which cannot accommodate growing  enrollment demand. Moreover, the current stove-piped environment limits  TSA’s ability to effectively detect malicious individuals that apply for  multiple transportation credentials to try to gain access through at least  one of them.

In 2008, TSA initiated the Technology Infrastructure Modernization (TIM)  program to, among other things, enhance the sophistication of its security  threat assessments and improve the capacity of its systems. Specifically,  the program is intended to provide a modern and centralized end-to-end  credentialing system that includes registration and enrollment, individual  security threat assessments, adjudication, credential  issuance/management, and revocation for transportation workers and  travelers. The program’s initial baseline estimated that the program would  be deployed in 2015 and cost about $631 million.

However, the program experienced significant cost and schedule  overruns, a significant increase in its requirements, and critical system  performance and technical issues during its initial limited deployment in  2014. As a result, in January 2015, TSA suspended the program while it  established a new strategy for developing and deploying the TIM system.  The program was rebaselined in September 2016 and is now estimated  to cost about $1.27 billion ($639 million more than originally planned).  Further, full operational capability for the system is now planned for fiscal  year 2021 (6 years later than originally planned).

TSA officials decided to move away from implementing a commercial-off- the-shelf product for the program and instead planned to develop an open  source system. Moreover, officials decided to use an Agile software  development approach, rather than the traditional waterfall development  approach the program had been using.

Given the issues that the TIM program has faced in developing the  system, you asked us to review the agency’s current effort. Our objectives  were to (1) describe TSA’s past implementation efforts for the TIM  program and its new implementation strategy; (2) determine the extent to  which TSA’s new strategy for the program addresses the challenges  encountered during earlier implementation attempts; (3) determine the  extent to which TSA has implemented selected key practices for  transitioning to an Agile software development framework for the  program; and (4) determine the extent to which TSA and DHS are  effectively overseeing and governing the TIM program to ensure that it is  meeting cost, schedule, and performance requirements.

To address the first objective, we reviewed program documentation, such  as initial and current acquisition program baselines, initial and current life- cycle cost estimates, acquisition decision memorandums, and program  plans documenting a new strategy for implementing the TIM program. We  used the information in this documentation to summarize TSA’s earlier  attempts to implement TIM capabilities and the program’s new  implementation strategy, including estimated costs, schedule, and key  decisions made. In this report, we use the TIM program’s objective  estimated cost and schedule values (targets that reflect the most likely  cost and schedule), and not the threshold cost and schedule values  (ceilings which, if exceeded, initiate official replanning actions). We also  interviewed TSA officials, including the TIM Director and Deputy Director,  on the status of the program office’s efforts.

For the second objective, we reviewed program documentation on the  challenges the TIM program office faced when it experienced cost,  schedule, and system performance issues and synthesized the  information to identify a consolidated list of key challenges the program  had previously faced. We then reviewed documentation on the TIM  program’s new implementation strategy. We compared this new strategy  to selected prior challenges and assessed the strategy against leading  practices and guidance, such as DHS’s Systems Engineering Lifecycle  Guide and the Software Engineering Institute’s Capability Maturity  Model® Integration for Development. We also conducted a site visit at  the TSA Adjudication Center in Reston, Virginia, where we observed  demonstrations of the information systems that are currently used to  conduct security threat assessments.

To address the third objective, we first reviewed leading practices and  guidance from, among others, the Software Engineering Institute, the  Office of Management and Budget (OMB), and DHS, and identified those  practices that are critical to establish when transitioning to an Agile  software development framework. Then, in consultation with GAO’s  internal Agile expert, we selected six practices that were most applicable  to the status of the program. We then reviewed relevant program  documentation, such as Agile training records, program plans, and status  reports, and interviewed TSA and DHS officials to assess the extent that  the program met these practices. We also observed Agile software  development activities conducted at TSA facilities in Annapolis Junction,  Maryland, and at a contractor’s facilities in Beltsville, Maryland.

To address the fourth objective, we reviewed leading practices and  guidance from, among others, the Software Engineering Institute, OMB,  DHS, and TSA, and identified four key practices in oversight and  governance of programs using Agile software development. We then  reviewed relevant TIM program management and governance  documentation, such as program management plans, Agile contracts,  schedules, cost estimates, program status reports, and artifacts from  program oversight reviews. We also interviewed TSA and DHS officials to  determine the extent to which these officials were following the key  practices.

To assess the reliability of the data that we used to support the findings in  this report, we reviewed relevant program documentation to substantiate  evidence obtained through interviews with agency officials. We  determined that the data used in this report were sufficiently reliable for  the purposes of our reporting objectives. We made appropriate attribution  indicating the sources of the data. A full description of our objectives,  scope, and methodology can be found in appendix I.

We conducted this performance audit from September 2016 to October  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

As we have previously reported, transportation systems and facilities are  vulnerable and difficult to secure given their size, easy accessibility, large  number of potential targets, and proximity to urban areas. TSA’s mission  is to protect the nation’s transportation systems by providing effective and  efficient security to ensure freedom of movement for people and  commerce. Accordingly, TSA is responsible for managing vetting and  credentialing programs to ensure that individuals that transport hazardous  materials or have unescorted access to secure or restricted areas of  transportation facilities at maritime ports and TSA-regulated airports do  not pose a security threat. In order to carry out this responsibility, TSA  conducts background checks—known as security threat assessments— on individuals seeking an endorsement, credential, access, and/or  privilege (hereafter called a credential).

Specifically, TSA reviews applicant information and searches government  databases, such as criminal history records from federal, state, and local  sources in the Federal Bureau of Investigation’s National Crime  Information Center database and Terrorist Screening Database, which is  the federal government’s consolidated terrorist watchlist. This information  is used to determine whether the applicant has known ties to terrorism  and whether the applicant may be otherwise precluded from obtaining a  credential based on his or her immigration status and criminal history,  among other factors. If TSA determines that an applicant does not pose  a security threat, a credential may be supplied by an issuing entity. If it  determines an applicant should be denied, the agency issues a  preliminary determination of ineligibility letter to the applicant. The  applicant may seek redress by appealing the determination or requesting  a waiver.

TSA’s security threat assessments support over 30 credentialing  programs in the maritime, surface, and aviation transportation segments.  The largest programs include the Transportation Worker Identification  Credential program for maritime workers, Hazardous Materials  Endorsement program for commercially licensed drivers, the Aviation  Worker program, and TSA Pre® for travelers at airport checkpoints.  According to TIM program officials, these transportation programs are  collectively estimated to have processed about 12.8 million enrollments  by October 2017. Table 1 describes the largest transportation  credentialing programs, by segment, and purpose of each.

TSA Established the TIM Program to Address Shortcomings with Security Threat Assessments and Credentialing Systems

TSA’s legacy IT systems that are currently used to help conduct its  security threat assessment and credentialing functions are an  aggregation of stove-piped solutions that were developed over a period of  time to support individual transportation screening programs. These  systems are duplicative and lack needed sophistication to effectively  detect, for example, if an individual is attempting to gain access to  multiple facilities across different transportation programs in an effort to  find any successful entry point. Early detection of this type of threat is  difficult and time consuming because many aspects of the current  systems are not fully automated.

Additionally, we and the DHS Office of Inspector General (OIG) have  previously reported numerous shortfalls with TSA’s security threat  assessment and credentialing systems.

We reported in 2011 that the demand for security threat assessments  is expected to continue to grow and the existing credentialing systems  will not be able to accommodate this growing enrollment demand.

In July 2013, we reported on functional limitations and technical  problems with TSA’s legacy credentialing systems that were to be  addressed by the TIM system. These limitations included the  inability to run reports to measure TSA response times to applicants,  track adjudication of cases, and address case workload backlogs. We  also reported on delays in processing new cases. We made  recommendations to address these issues and DHS agreed with our  recommendations. DHS has taken several actions to implement the  recommendations, such as establishing a process for developing  accurate workload projections and hiring additional adjudicators.

In June 2015, DHS’s OIG reported on issues with TSA’s lack of  continuous vetting once a credential was issued, referred to as  recurrent vetting. For example, the OIG reported on the need for  recurrent vetting of aviation workers. Specifically, it found that TSA did  not have effective controls in place for ensuring that aviation workers  had not committed crimes that would disqualify them from having  unescorted access to secure areas of airports, and that they had  lawful immigration status and were authorized to work in the United  States. Instead, TSA depended on the commercial airports and air  carriers to verify criminal histories of workers who already hold  credentials, and on the credential holders themselves to report  disqualifying crimes to the airports where they worked. The DHS OIG  recommended that TSA pilot the Federal Bureau of Investigation’s  Rap Back program and take steps to institute recurrent vetting of  criminal histories at all commercial airports. TSA concurred with the  recommendation and stated that it planned to initiate a pilot Rap Back  program to help ensure full implementation across all eligible TSA- regulated populations in the future.

In September 2016, DHS’s OIG reported that, although TSA required  Transportation Worker Identification Credential cardholders to self- report to the administration and surrender their card when charged  with a disqualifying offense, this self-reporting occurred only once  between 2007 and 2016. The report also stated that TSA was  testing two methods to implement recurrent vetting into its  credentialing programs—the Federal Bureau of Investigation’s Rap  Back program to check for criminal violations and the use of DHS’s  Automated Biometric Identification System to check for both criminal  and immigration violations. However, TSA’s plans did not include a  method for determining the best approach, and the OIG reported that  this would impede TSA’s ability to implement recurrent vetting  successfully and efficiently. Accordingly, the OIG recommended that  TSA establish measurable and comparable criteria to use in  evaluating and selecting the best criminal and immigration recurrent  vetting option, and TSA concurred with this recommendation.

Also, in September 2016, the DHS OIG reported that the  background checks for the Transportation Worker Identification  Credential program were not as reliable as they could be. For  example, the OIG found that TSA did not have processes in place to  ensure the proper separation of duties for adjudicators, who had the  ability to assign, review, and perform quality assurance on the same  case. The OIG also found missing supervisory review controls in the  terrorism vetting process. Accordingly, the OIG recommended that  TSA identify and implement additional internal controls and quality  assurance procedures; TSA agreed with the recommendation. In  response, TSA planned to make improvements to the TIM system to  include an additional quality assurance component in which the  system would automatically select cases for senior adjudicators to  review and to incorporate into the overall reporting and monitoring  activities.

The TIM system is intended to address the shortfalls identified in these  prior reports by providing a modern and centralized end-to-end  credentialing system. The system is also intended to provide counter- terrorism and trend analytic capabilities to help identify unusual activities  (e.g., credential shopping and using multiple aliases) across the entire  credentialing process and all transportation populations supported by  TSA’s security threat assessments. In addition, the system is expected to  enable automated recurrent vetting of individuals against criminal and  immigration databases to ensure that a credential or endorsement is  revoked if an individual commits a disqualifying act.

The planned credentialing process that is to be supported by the TIM  system includes:

Registration and enrollment: Individuals seeking a credential or  endorsement under one of the transportation programs supported by  the system are expected to be able to apply for a security threat  assessment at a Universal Enrollment Center or via the system’s  online portal. The biographic and biometric information collected from  the applicant is to be received and processed by the system.

Eligibility vetting and risk assessment: The system is to conduct  automated vetting of the applicant’s information against criminal,  immigration, and terrorism watchlists to determine the security risk  associated with allowing access privileges based on the criteria for the  credential or endorsement that the individual is seeking to obtain. If  the results return a flag for a potentially disqualifying factor, the  applicant’s case is to be sent for adjudication. TSA adjudicators are to  use the system to review and adjudicate cases that did not pass  automated vetting by comparing the applicant’s information to the  criteria for the credential or endorsement that the individual is seeking  to obtain. The adjudicators are to determine the applicant’s eligibility  for the credential or endorsement, and approve or deny the  individual’s application.

Issuance: When an applicant is approved through eligibility vetting or  adjudication, the system is to notify the applicant of approval and  provide instructions on how to receive the credential, which is to be  activated by the system and supplied by the issuing entity. The  applicant also is to be able to login to the online portal to view the  status of the application.

Verification and use: Use of the credential in secured areas is to be  verified, including determining that the credential is authentic, that the  individual is the correct recipient of the credential, and that the  credential’s status is valid (not revoked or expired).

Revocation and expiration: The system is expected to conduct  subsequent automated recurrent vetting of individuals who previously  had been approved against criminal, immigration, and terrorist  databases on an ongoing basis. If, as a result of recurrent vetting or  self-reporting, there is new information indicating that an applicant’s  credential should be revoked, the system is to alert the adjudicators  who are then to determine if revocation is needed. The system is to  prompt credential expiration at the end of a specified period of time.

Redress or waiver: An applicant that is denied a credential is to be  able to apply to TSA to either appeal the decision, to include providing  documentation to prove that he/she is eligible, or request a waiver  from having to meet the eligibility criteria.

Trend analytics: The system is to allow TSA’s Office of Intelligence  and Analysis users to select from a standardized suite of analysis  tools that would allow them to identify unusual activities across  transportation populations. A key objective would be to identify  through analysis those adversaries and terrorists who may attempt to  hide behind multiple personas and aliases.

Figure 1 provides an overview of the intended future credentialing  process which the TIM system is expected to support.

TIM program officials decided to adopt an Agile software development  approach—a type of incremental development—which calls for the rapid  delivery of software in small, short increments rather than in the typically  long, sequential phases of a traditional waterfall software development  approach. This decision is consistent with OMB’s guidance as specified in  its IT Reform Plan, as well as the legislation commonly referred to as  the Federal Information Technology Acquisition Reform Act. Agile  emphasizes early and continuous software delivery, as well as using  collaborative teams and measuring progress with working software.  Figure 2 provides a depiction of software development using the Agile  approach compared to a waterfall approach.

The Agile approach significantly differs in several ways from traditional  waterfall software development. Table 2 highlights major differences  between the Agile and waterfall software development approaches.

Additionally, Agile practices integrate planning continuously throughout  the life-cycle. Although Agile requires some high-level, up front planning,  in general, planning in Agile focuses on the near term of the next few  software releases. Planning sessions are conducted to support each  release, iteration, and every work day. For example, development teams  have daily meetings, where the team members discuss what they did  yesterday and what they plan to do that day. Frequent planning is aimed  at ensuring the program is delivering the needed capabilities to the end  users.

As we have previously reported, numerous frameworks are available to  Agile practitioners to guide their Agile software development activities.  Scrum is one common framework, which is widely used in the public and  private sectors and its terminology is often used in Agile discussions. The  following are key Scrum terminology and concepts:

Product owners represent the end user community and have the  authority to set business priorities, make decisions, and accept  completed work.

Scrum iterations (also called sprints) are where development teams  build a piece of working software during a short, set period of time  (e.g., 2 weeks). A collective set of sprints is bundled into a software  release.

Sprint teams (or development teams) conduct the Agile software  development and testing work. These teams collaborate with minimal  management direction, often co-located in work rooms. They meet  daily and post their task status visibly, such as on wall charts.

Scrum masters, similar to project managers, are responsible for  removing impediments to the sprint teams’ ability to deliver the  product goals and deliverables.

User stories convey the customers’ requirements at the smallest and  most discrete unit of work that must be done to create working  software. Each user story is assigned a level of effort, called story  points, which is a relative unit of measure used to communicate  complexity and progress between the business and development  sides of the project.

To ensure that the product is usable at the end of every iteration,  teams adhere to an agreed-upon definition of what constitutes  acceptable, completed work.

Backlogs are lists of requirements, such as user stories, to be  addressed by working software. If new requirements or defects are  discovered, these can be stored in the backlog to be addressed in  future iterations.

Velocity is a metric which is used to track the rate of work completed  using the number of story points completed or expected to be  completed in an iteration (i.e., sprint), or release. For example, if a  team completed 100 story points during a 4-week iteration, the  velocity for the team would be 100 story points every 4 weeks.

Another framework, referred to as the Scaled Agile Framework (SAFe), is  a governance model for organizations to use to align and collaborate the  product delivery for modest to large numbers of Agile software  development teams. The framework is intended to be applied to several  organizational levels, including the development team level, the program  level, and the portfolio level. It is also intended to provide a scalable and  flexible governance framework that defines roles, artifacts, and processes  for Agile software development across all levels of an organization.

DHS has sought to establish Agile software development as the preferred  method for acquiring and delivering IT capabilities. Specifically, in  February 2016, the DHS Under Secretary for Management initiated an  Agile software development pilot to improve the execution and oversight  of the department’s IT acquisitions. The Under Secretary for Management  selected five DHS programs that were in various stages of the acquisition  life-cycle, including the TIM program, to be part of the pilot.

As part of this pilot initiative, DHS established integrated product teams  designed to support each of the five programs in their efforts to adopt  Agile practices. These teams were directed to focus on effectively  planning and executing the pilot programs, as well as developing  appropriate documentation to support program execution. According to  the Under Secretary for Management, the department plans to use  lessons learned from the pilots to develop and update policies and  procedures for executing the pilot programs and future IT acquisitions. As  of May 2017, department officials had not determined a completion date  for the pilot.

Additionally, DHS established a headquarters-level Agile team intended  to collaborate across the department on improvements to policy,  governance, and acquisition guidance. This group is intended to support  Agile delivery; codify and publicize process improvement artifacts  generated by the program-level integrated product teams; and eliminate  redundancies and conflicting guidance so that oversight groups speak  with one voice, reducing time through the acquisition process.

In addition to the use of Agile software development principles, the TIM  program is subject to the department’s oversight framework. Specifically,  the program is to adhere to DHS’s acquisition policy, including its systems  engineering life-cycle framework, which is intended to support efficient  and effective delivery of IT capabilities. The Under Secretary for  Management serves as the decision authority for the program, and is  responsible for overseeing adherence to DHS’s acquisition policies for the  department’s largest acquisition programs (i.e., those with life-cycle cost  estimates of $1 billion or more).

The Under Secretary for Management is supported by two offices within  the department. The first of these offices—the Office of Program  Accountability and Risk Management (PARM)—is responsible for DHS’s  overall acquisition governance process. PARM is responsible for, among  other things, periodically conducting program health assessments to  evaluate acquisition programs, in terms of a program’s management,  resources, planning and execution activities, requirements, cost and  schedule, and how these factors are impacting a program’s ability to  deliver a capability.

The other key supporting office—the DHS Chief Information Officer  (CIO)—is responsible for, among other things, setting departmental IT  policies, processes, and standards. The CIO is also responsible for  ensuring that acquisitions comply with the department’s IT management  processes, technical requirements, and the approved enterprise  architecture. Within the Office of the Chief Information Officer (OCIO), the  Enterprise Business Management Office is to ensure that the  department’s IT investments align with its missions and objectives. As  part of its responsibilities, this office periodically assesses investments to  gauge how well they are performing through a review of program risk,  human capital, cost and schedule, and requirements—referred to as the  CIO’s program health assessment.

According to the CIO, the Chief Technology Officer, which is responsible  for leading the development of IT and standards across the department,  and for management of the Agile pilot initiative, offers guidance and  assistance to programs to help improve their execution. In addition, the  Director of the Office of Test and Evaluation is to provide oversight of  components’ independent test and evaluation activities.

The DHS Acquisition Review Board is chaired by the Under Secretary for  Management and is made up of many executive level members including  the CIO, the Executive Director of the Office of PARM, and the Chief  Procurement Officer. The board is to meet periodically to oversee  programs’ business strategies, resources, management, accountability,  and alignment to strategic initiatives. Additionally, the department has  established executive steering committees, which generally are  comprised of component and DHS executive-level members, such as the  component CIO and Chief Financial Officer, as well as the DHS Chief  Technology Officer and the Executive Director of the Office of PARM. The  committees are to provide governance, oversight, and guidance to  programs and their related projects and initiatives to help ensure  successful development and operations.

Figure 3 shows the organizational structure of the key DHS organizations  with IT acquisition management responsibilities.

TSA Organizations Involved with the TIM Program

The TIM program office resides within the Mission Operations component  of TSA’s Office of Information Technology. The expected users of the TIM  system come from multiple offices under the Office of Intelligence and  Analysis, including the Security Threat Assessment Operations office,  which is responsible for conducting the security threat assessments, and  the Program Management office, which is responsible for managing  TSA’s maritime, surface, and aviation credentialing programs. The TIM  program’s Executive Steering Committee is chaired by the TSA CIO, who  is the head of the Office of Information Technology, and the TSA Deputy  Component Acquisition Executive, and meets quarterly. In addition, the  TSA Operational Test Agent is to perform operational testing and  evaluation of the TIM system’s operational effectiveness, interoperability,  cybersecurity, and suitability. As previously mentioned, the DHS Director  of the Office of Test and Evaluation is to provide oversight of these test  and evaluation activities.

Figure 4 shows the key TSA organizations involved with the TIM program.

After Experiencing Significant Cost, Schedule, and Performance Issues with the Initial TIM System Deployment, TSA Implemented a New Strategy

The TIM program experienced significant cost, schedule, and  performance issues during its initial implementation efforts. Specifically, in  May 2014, TSA launched an initial version of a commercial-off-the-shelf  (COTS) system for the maritime transportation segment of TIM that was  to support the Transportation Worker Identification Credential program.  However, as we previously reported, in September 2014, TSA reported  to DHS that the program had breached its baseline because it had  significant cost, schedule, and performance issues due to, among other  things, the addition of newly created credentialing programs that were  added to the program’s scope, such as TSA Pre® and Chemical Facility  Anti-Terrorism Standards.

TIM program officials also reported in the breach remediation plan other  issues that led to the breach, including different expectations between  TSA officials and the contractor regarding the extent of reuse of system  functionality among the different transportation segments. Specifically,  TSA expected that it would be able to reuse more of the maritime  functionality for the surface and aviation populations, while the contractor  expected there to be less reuse.

In January 2015, the Acting Under Secretary for Management directed  program officials to suspend all planning and development efforts related  to the other two segments of the program—surface and aviation—until  the issues with the maritime segment could be resolved. In August 2015,  program officials prepared a revised life-cycle cost estimate which  increased costs to approximately $1.34 billion (about $713 million more  than the original 2011 estimate), and delayed full deployment of the TIM  system (to include all three transportation segments) to fiscal year 2022  (7 years later than originally planned).

Also, in September 2015, the Director of the Office of Test and Evaluation  issued a letter of assessment which concluded that initial operational  testing of the COTS system for the maritime segment had determined  that the system was not operationally effective and not operationally  suitable. The Under Secretary for Management directed the DHS CIO to  conduct a thorough review of the proposed plans for moving forward with  the TIM program.

After conducting the review, the CIO did not support the program’s  proposal. As a result, in November 2015, the Under Secretary for  Management continued the suspension of all developmental efforts for  the surface and aviation transportation segments, but authorized the  program to continue resolving problems that were identified during initial  operational testing for the COTS system being used by the maritime  segment. The Under Secretary for Management also directed the CIO to  form and lead an integrated product team with senior TSA  representatives and the TIM program office to develop a new strategy for  the program.

In March 2016, DHS and TSA officials completed a new strategy for  delivering TIM capabilities. This strategy included the following changes:  replace proprietary COTS applications with custom-developed  applications using open source code;  transition traditional, large development teams using a waterfall  system development methodology to an Agile software development  framework to enable rapid, incremental development and deployment;  and migrate from a defined, fixed data center environment to a scalable  Federal Risk and Authorization Management Program (FedRAMP)  certified cloud computing environment.

Also, according to the new strategy, the move from the COTS product to  an open source solution is to include replacing the COTS product that  had already been deployed to the maritime segment with the open source  solution. It is also to include replacing the legacy systems that support the  credentialing programs from the other two transportation segments  (surface and aviation) with the open source solution. TSA plans to  incrementally transition the program from these legacy systems between  fiscal years 2018 and 2021.

Additionally, the system is expected to interface with at least 19 other  information systems, including the following key systems:

TSA’s Transportation Vetting System, which conducts initial and  recurrent name-based matching against defined terrorist related data  sets.

The Federal of Bureau of Investigation’s National Crime Information  Center, which is an electronic clearinghouse of crime data.

DHS’s Automated Biometric Identification System, also referred to as  IDENT, which is the central DHS-wide system for storage and  processing of biometric and associated biographic information for  national security, law enforcement, immigration and border  management, intelligence, and other background investigative  purposes.

TSA’s Secure Flight, which identifies individuals who may pose a  threat to aviation or national security and designates them for  enhanced screening or prohibition from boarding an aircraft, as  appropriate.

The U.S. Citizenship and Immigration Service’s Systematic Alien  Verification for Entitlements, which is the primary data source for  government agencies to verify legal entry and presence in the United  States of a non-U.S. citizen or naturalized U.S. citizen.

In April 2016, the Under Secretary for Management approved the TIM  program’s new strategy and, in September 2016—almost 2 years after  the program was initially suspended—the program was rebaselined to  reflect the new strategy. As we previously reported, the estimated cost  and schedule in the revised baseline was significantly different than the  initial baseline. The revised baseline estimate was for about $1.27  billion (a $74 million decrease from the previous 2015 cost estimate and  an overall increase of $639 million from the original 2011 estimate), with  full deployment planned for 2021 (a 1-year acceleration from the previous  2015 schedule and an overall delay of 6 years from the original 2011  schedule). Table 3 shows the estimated costs and schedules reflected in  the initial and revised estimates.

According to TIM officials, in the program’s first 8 years (between October  2008 and September 2016), TSA spent over $280 million to deploy the  initial COTS solution to the maritime segment and address critical fixes in  the solution (i.e., the solution that TSA determined it needs to replace).  Also during 2016, TSA began transitioning to an Agile software  development framework. In September 2016, TSA issued two task orders  to a contractor to provide Agile software development services. The  orders were issued to the same design and development contractor that  had assisted with the initial deployment of the TIM COTS solution.

From October 2016 to June 2017, the program deployed four software  releases using Agile software development practices. These releases  were focused on, for example, deploying new functionality to the COTS  system to enhance the criminal and immigration vetting data provided to  adjudicators.

In December 2016, between the first and second Agile releases, the  program suspended new development for 1 month while officials  reconsidered the order in which they would deliver functionality. Also  during this period, the program developed and deployed a smaller release  which program officials refer to as a “half release.” According to program  officials, this release did not produce any new capabilities and instead  addressed operations and maintenance-related fixes to the deployed  COTS system.

After development of the second software release, at the end of March  2017, the program was reviewed by DHS’s Acquisition Review Board.  The purpose was to review the results of follow-on operational testing that  was performed to determine whether the program had adequately  addressed the prior system and usability issues and implementation of  the program’s new strategy. The meeting was also intended to discuss  the status of several action items from a prior review board meeting that  occurred in September 2016, such as finalizing a test and evaluation  master plan, conducting a cybersecurity threat assessment, updating the  program’s mission needs statement and concept of operations, and  establishing software development cost metrics. Implementation of the  new strategy continues to be monitored by DHS and TSA oversight  bodies.

The New Strategy for the TIM Program Has Addressed Selected Prior Challenges, but Concerns Remain

The new strategy for the TIM program addressed a number of major  challenges that the program faced during earlier efforts to develop and  deploy the system; nevertheless, key challenges remain. Specifically, of  the seven major challenges that the program faced during its initial  implementation of a COTS solution for the maritime segment, four  challenges have been addressed related to (1) system performance and  usability issues, (2) data migration issues, (3) information security testing,  and (4) the inadequacy of the program’s previous hosting facility.  However, the remaining three challenges regarding constraints with  COTS product, significant addition of new transportation programs (e.g.,  TSA Pre®), and insufficient stakeholder coordination and  communication have not been fully addressed.

Fixes Have Been Implemented to Address Critical System Performance and Usability Issues

According to DHS guidance, among other things, an operational test and  evaluation examines systems for operational effectiveness. Specifically, it  tests for the ability of a system to accomplish a mission when used by  representative users in the expected environment.

The 2015 initial operational testing of the maritime segment (supporting  the Transportation Worker Identification Credential program) found that  the COTS system was extremely unreliable due to frequent critical  failures, and had several system performance and usability issues that  limited users’ ability to execute tasks in a timely and accurate manner.  These issues included lags, freezes, the need for excessive refreshes,  inadequate reporting and case management functionalities, as well as an  interface that was not user-friendly. For example, the system was unable  to produce accurate reports on case workload and status, so users  expended significant effort creating spreadsheets to manually assign  cases and manage their progress. The system was also unable to  perform certain waiver functions in a timely and complete manner, which  resulted in a significant backlog.

The program office has addressed the issues identified in the initial  operational test report by first identifying a list of over 900 action items.  According to TIM officials, they validated this list with the operational test  agent and prioritized the action items with the product owners (i.e., end  users) to identify which were the most critical to complete. For example,  critical items included addressing issues with the waiver functions,  assigning cases, and issuing credentials. The program implemented the  critical fixes by developing seven software releases from September 2015  to October 2016. In January 2017, the TSA operational test agent  reported that follow-on operational testing of the COTS system confirmed  that the program had adequately addressed the prior system and usability  issues. As a result, according to the test agent, the program’s previously  deployed maritime segment of the system performed as intended.

Actions Have Been Taken to Better Account for the TIM Program’s Future Data Migration Efforts

According to leading practices, IT programs should identify potential  problems before they occur. This allows programs to plan and execute  activities to mitigate the risk of such problems having adverse impacts on  the program.

When the TIM program transitioned maritime users from the legacy  system to the COTS system, according to TSA’s breach remediation plan,  program officials found that cleaning and properly migrating data was  very difficult and time consuming because the legacy systems were old  and the data mapping information was not readily evident. Program  officials stated that the data migration efforts were also difficult because  of the proprietary nature of the COTS product, which impacted the ability  to effectively migrate data from legacy systems. The additional time  needed for data migration resulted in higher than anticipated costs for the  maritime transportation segment.

Program officials have taken action to better account for the TIM  program’s future data migration efforts. Specifically, as part of the new  strategy, the officials plan to defer legacy data migration until after system  deployment efforts are complete to avoid disrupting deployment efforts.  The strategy focuses on the program migrating only closed case data  from the legacy systems to the new system. As such, adjudicators are to  continue to complete and close any security threat assessment cases  opened in the legacy system even after the new system is deployed, and  the new system is to only handle newly opened security threat  assessment cases. Once final disposition of the cases in the legacy  system is complete, those cases would then be included in the closed  case data migration effort, which is planned to occur at the end of  development, around fiscal years 2020 to 2021.

In addition, the new strategy includes streamlining the data migration by  using the open source solutions to help simplify the migration of data on  transportation populations from the legacy systems. As a result of the  new approach, the program should be better positioned to more  effectively migrate data during future transitions between the legacy  systems and new system.

Prior Information Security Weaknesses in the TIM System Have Been Addressed, and Deferred Cybersecurity Threat Testing Is Planned

According to DHS guidance, the operational test and evaluation also  should examine the department’s systems for operational suitability,  which is the degree to which a system is deployable and sustainable. The  evaluation is to take into account factors such as reliability,  maintainability, availability, and interoperability.

The 2015 initial operational testing of the COTS system found that it was  not suitable because the system had significant information security  weaknesses. Specifically, the system inappropriately provided users with  greater access than was necessary to do their jobs, which undermined  the security benefits of controlling what different users were able to do in  the system based on their role. The COTS system also contained critical  and high-risk system security vulnerabilities which could result in the  compromise of sensitive system information, such as passwords, and  could hinder TSA officials’ ability to effectively respond to incidents.

Program officials took actions to address the security weaknesses  previously identified. For example, in response to the findings from the  initial operational testing, between September 2015 and October 2016,  they developed and released fixes to the significant security weaknesses.  In April 2017, the results of the follow-on operational testing confirmed  that the COTS system was free of critical or high-risk system security  vulnerabilities and that it appropriately restricted access to the system by  only allowing users to access areas of the system needed to support their  specific business tasks.

In addition, critical steps to evaluate the system’s cybersecurity have  been planned, but not yet completed. Specifically, testing for realistic  cybersecurity threats which is used to help categorize the system’s risk- level in terms of confidentiality, integrity, and availability, was deferred  until March 2018. Program officials decided to defer this test until new  hosting environments for TIM are implemented, rather than testing TIM in  an environment that will soon be retired. These environments are  intended to enable the development, testing, and production of the  system. However, implementation of those environments has been  delayed until December 2017, and as a result, the cybersecurity  vulnerability assessment has been deferred to March 2018. The  identification of a time frame in which the program plans to conduct this  important cybersecurity test is a step in the right direction, and avoiding  additional delays will be important.

TSA Decided to Discontinue Use of a DHS-Provided Cloud, and Recently Took Actions to Address Delays in Implementing Interim Hosting Environments

According to OMB, a hosting facility or data center is to process or store  data and must meet stringent availability requirements. Additionally, cloud  computing can be used as a means for enabling on-demand access to  shared and scalable pools of computing resources.

During the initial implementation of TIM, the system was hosted in a cloud  that operated out of a DHS data center (referred to as DHS Data Center  1). However, the DHS cloud was higher in operations and maintenance  costs than the program originally planned, which presented a challenge  for the program.

To address this challenge, in 2016, TIM program officials decided to  move the COTS system that was previously deployed (the maritime  segment) out of the DHS cloud and set it up in a public cloud  environment. They also planned to use the public cloud environment to  develop, test, and operate the future TIM open-source based system. The  officials planned to use a phased migration that consisted of first  establishing hosting environments at two data centers—DHS Data Center  1 and TSA Colorado Springs Operations Center. The officials planned to  use the data centers for the development, testing, and production of the  future TIM open-source based system, and then eventually transition to a  public or hybrid cloud once the system reaches full operational capability  in fiscal year 2021. As part of this approach, officials planned to  establish 10 development, testing, and production environments at these  data centers from January to July 2017, so that TIM’s development teams  did not have to compete for the same environments during Agile software  development and testing efforts.

While the program experienced delays in setting up its production  environment, officials recently took actions to address these delays.  Specifically, the program was expected to have a new production  environment available at the TSA Colorado Springs Operations Center by  March 2017; however, it was delayed until May 2017. Additionally, while  migration of the TIM system to the new hosting environments was  planned to occur by September 2017, it has been delayed. These delays  have contributed, in part, to delays in other aspects of the program,  including the execution of the cybersecurity vulnerability assessment, as  well as delays in the implementation of automated testing and  deployment tools (discussed later in this report). In response to these  delays, program officials recently established a revised schedule in May  2017 for setting up the new environments by December 2017.

Effectively executing against this updated schedule should help to keep  the program on track with delivering these important environments and  fully addressing the related challenge that the program experienced  during its prior implementation efforts.

TSA Decided to Move the TIM System from COTS to Open Source, but Implementation Plans Continue to Significantly Change

According to leading practices and guidance, technology decisions should  seek to enable services to scale easily and cost-effectively and to avoid  vendor lock-in by, for example, using open source solutions. The benefits  of using open source solutions can include improved software reliability  and security through the identification and elimination of defects from  continuous and broad peer review of publicly available source code that  might otherwise go unrecognized by a more limited core development  team; unrestricted ability to modify software source code; no reliance on a  particular software vendor due to proprietary restrictions; reduced  software licensing costs; and the ability to “test drive” the software with  minimal costs and administrative delays in a rapid prototyping and  experimentation environment.

Also, according to leading practices, IT programs should ensure that their  plans include how they will transition from the current state to the final  state of system operations. Such planning provides a mutual  understanding to relevant stakeholders of how programs are to  accomplish the transition.

According to TSA’s breach remediation plan, the TIM program’s use of a  COTS solution led to several challenges. For example, program officials  reported that the COTS product restricted their ability to make changes to  the product to improve system usability and, as previously discussed,  impacted the ability to effectively migrate data from legacy systems  because of the proprietary COTS product. Program officials also reported  that they were highly dependent on the COTS vendor to remediate  compatibility issues and resolve problems, which required additional time.  The plan also stated that the COTS product required a complex system  architecture which prevented the program from implementing modern  software development and testing tools. Finally, use of the COTS product  resulted in higher software licensing costs.

The TIM program’s new strategy is intended to address these challenges  by moving away from using a COTS product to a custom-developed open  source solution. However, the program’s approach for developing and  delivering this new solution has been in a continual state of fluctuation  and implementation plans have not been defined. As such, this challenge  has yet to be fully addressed. Specifically,  In September 2016—after the 2-year pause in the program and  completion of its extensive rebaselining effort—DHS and TSA officials  decided that TSA would incrementally retire legacy systems as the  transportation programs that use those systems are migrated to the  open source solution; they also decided to eventually replace the  COTS system that was previously deployed to support the maritime  Transportation Worker Identification Credential program and migrate  to the open source solution. This was to be completed using a staged  approach between the migrations, and also by using two versions of  the COTS system as well as the open source system. However, the  program lacked a plan detailing how it was going to migrate from the  current legacy state, to the interim environment (with the two versions  of COTS plus an open source system), to the final state.

As previously mentioned, in December 2016, new development for  the TIM system was paused once again to, among other things,  further evaluate the transitioning approach that was agreed to 3  months prior. Four months later (in mid-March 2017), program officials  decided to continue pursuing the approach that was agreed to in  September. Subsequently, the high-level implementation schedule  was revised to adjust for delays that this most recent replanning effort  contributed to (other contributing factors for the delay are discussed  later in this report). The revised schedule delayed deployment of the  initial Pre® capabilities by 6 months and other key functionality up to  12 months.

Further adding to the fluctuation in the program, at the end of March  2017, the DHS Acquisition Review Board requested that the  program’s implementation approach be revised to accelerate the  delivery of the TIM program’s front-end interface for adjudication and  redress functions. However, it is unclear how the acceleration of the  development and implementation of these functions will impact the  delivery of the other planned functionality, and what tradeoffs the  program will need to make. Program officials were expected to  develop an overview of the acceleration efforts associated with cost,  schedule, risk, and impacts on the program and deliver it to PARM  and the Office of the Chief Technology Officer in August 2017.

As a result, while it has been 8 months since the TIM program was  rebaselined, the details of how the program will transition from its current  state, to an interim state, then to the final state of full open source, have  yet to be determined. This is contrary to leading practices that we have  previously identified, which state that when pursuing an IT modernization  effort, organizations should develop a plan for transitioning from the  current to the target environment.

In response to our concerns, program officials stated that after they  determine how they will adjust to incorporate the Acquisition Review  Board’s recent acceleration request, they will determine the details of how  the program will achieve the desired final state. However, until the  program establishes and implements specific time frames for determining  key implementation details, including how it will transition the program  from its current state to an interim state and to the final state, the TIM  program office, and TSA and DHS oversight bodies cannot be certain  about how the program will ultimately deliver its complete open source  solution.

New Transportation Programs Have Been Incorporated in TIM’s Rebaselined Schedule, but the Program Is Experiencing Significant Delays

According to leading practices, programs should manage changes to  requirements as they evolve during the project. Programs should also  ensure that planned schedules provide a realistic forecast for completion  of activities, including providing reasonable slack (i.e., flexibility in the  schedule).

After the TIM program was initiated in 2008, it experienced significant  increases in scope, such as the addition of TSA Pre® and Chemical  Facility Anti-Terrorism Standards populations in 2012, which required  more functionality and considerably more processing demands than  originally planned. The TIM program was challenged to accommodate the  additional work needed to incorporate these new transportation  populations and capabilities, and, in part, contributed to a significant  breach in its original cost and schedule estimates.

To address the challenge, the TIM program incorporated the additional  functionality and processing requirements into its cost and schedule  rebaseline that was approved in September 2016. In addition, the  program’s new strategy addressed the need to be adaptable to  accommodate any new transportation populations and capabilities that  could be added in the future by taking an enterprise-level approach to  providing capabilities.

Nevertheless, while the TIM program incorporated TSA Pre® into its  new plans, the implementation schedule for the program was very  compressed and program officials did not establish a schedule that  realistically forecasted when activities would be completed. Specifically,  program officials planned to deploy initial TSA Pre® capabilities by May  2017 without any slack in the schedule. According to program officials,  the reason for this approach, was because TSA Pre® was considered a  high priority for migrating from its legacy system in order to accommodate  an expected influx of applicants during the summer months. However,  slack was not incorporated in the implementation schedule; therefore,  when the program experienced schedule delays, it resulted in the  program missing the May 2017 implementation deadline and being  rescheduled to November 2017.

The 6-month delay in delivering initial Pre® capabilities was due to the  delays discussed in the prior section associated with replanning the  strategy for transitioning to the open source system, as well as delays in  onboarding additional development team members and setting up new  development and production environments. The delay in delivering  Pre® capabilities is especially problematic because program officials  have reported that the legacy system is at risk of exceeding its processing  capacity.

Additionally, as previously mentioned, the program’s revised schedule  shows the delivery dates for almost all (8 of 10) capabilities being  significantly pushed back—with 2 capabilities being delayed up to 12  months. Moreover, not only were the implementation dates delayed for  these efforts, the time to complete a number of these efforts was reduced  by about 1 to 12 months—thus further exacerbating our concerns about  unrealistic schedules. Without a schedule that realistically forecasts when  activities will be completed, TIM program officials cannot ensure that they  will meet the dates that they have committed to, such as when key  capabilities for TSA Pre® are to be deployed.

Efforts to Improve Stakeholder Coordination and Communication for the TIM Program Have Begun, but Key Actions Have Not Been Implemented

According to leading practices, programs should coordinate and  collaborate with relevant stakeholders (i.e., those that are affected by or  in some way accountable for the outcome of the program, such as  program or work group members, suppliers, and end users). Stakeholder  coordination includes, for example, involving stakeholders in reviewing  and committing to program plans, agreeing on revisions to the plans, and  identifying risks. Programs should also identify the needs and  expectations of stakeholders and translate them into end user  requirements.

However, during prior implementation efforts with the COTS solution, the  program experienced challenges with effectively coordinating and  communicating with end-users. For example, according to program  documentation, it had not adequately collaborated with end users in  developing and implementing business requirements and conducting  post-deployment user satisfaction assessments. This led to frustration  among end users who felt inadequately informed and prepared for the  new COTS system.

To address this challenge, the TIM program’s new strategy includes  establishing a product owner role, which, as previously mentioned, is  intended to represent the end user community and have the authority to  set business priorities, make decisions, and accept completed work. The  program’s adoption of the Agile software development approach has also  significantly increased the frequency of the program’s engagement with  stakeholders to define, test, and implement software releases.

In addition, program officials established an organizational change  management strategy in October 2016 that is intended to, among other  things, focus broadly on establishing overall communication processes for  program stakeholders. This strategy identifies key steps such as,  establishing a communication team and hiring a communication lead to  oversee the development and execution of the communication action  plans, establishing a communication working group, and serving as chair  of the communication working group. This group is to be responsible for  developing four communication action plans for key stakeholder groups  (e.g., new transportation populations, existing transportation populations,  and management). These particular steps were to be completed from  November 2016 through January 2017.

However, while as of May 2017, the TIM program had implemented  certain steps from the organizational change management strategy, such  as establishing a communication team, the program has been delayed in  implementing other steps. Specifically, the communication lead position  was to be filled in November 2016. However, in March 2017 TIM program  officials stated that the position had not yet been filled due to the federal  hiring freeze. Additionally, because of the vacancy in the communication  lead position, other key actions have been delayed, such as the  development and execution of the communication action plans.

Program officials have not established new time frames for completing  the remaining steps outlined in the organizational change management  strategy. Until these time frames are established and effectively executed,  program officials will have less assurance that there will be effective  communication with stakeholders and customers to ensure that the  program is meeting their needs.

The TIM Program Has Not Fully Implemented Leading Practices for Transitioning to Agile Software Development

As discussed previously, transitioning a program from waterfall  development to Agile software development is a significant effort, and  requires the implementation of fundamental practices to ensure that the  transition is successful. According to leading guidance, an organization  transitioning to Agile software development should establish critical  practices to help ensure successful adoption of the Agile approach, such  as obtaining full support from leadership to adopt Agile processes, enhancing Agile knowledge, ensuring product owners are engaged with the development teams  and have clearly defined roles, establishing a clear product vision, prioritizing backlogs of requirements, and  implementing automated tools to enable rapid system development  and deployment.

While the TIM program has fully implemented the first two of these  leading practices necessary to ensure the successful adoption of Agile,  the remaining four practices have not been fully implemented. The gaps  we have identified with the program’s implementation of Agile are  concerning given that it did not follow key IT acquisition best practices  when using its waterfall development approach during the program’s first  8 years and spent over $280 million on a system that TSA has  determined it needs to replace.

The TIM Program Has Received and Maintained Support from TSA and DHS Leadership to Adopt Agile Practices

According to leading practices and guidance, an organization transitioning  to Agile software development should get and maintain full support from  the organization’s leadership to adopt Agile processes. Leadership  support helps empower employees to continuously improve the use of  Agile software development practices.

DHS and TSA leadership have approved the TIM program’s adoption of  Agile software development, and continue to support the transition. For  example, the DHS OCIO worked closely with TSA officials in 2015 and  2016 to develop the new strategy for the program which included moving  away from a waterfall development approach to Agile software  development. As previously mentioned, the Under Secretary for  Management selected the TIM program to be part of the DHS Agile pilot  initiative in February 2016 and approved the program’s new strategy in  April 2016.

Moreover, the DHS Office of the Chief Technology Officer has continued  to provide guidance and resources to the program since it adopted Agile.  For example, TIM program officials stated that the DHS Chief Technology  Officer added two of the office’s full-time and one part-time staff members  to the TIM program. DHS and TSA officials stated that the Chief  Technology Officer also provided an Agile coach to assist the TIM Program Manager about 3 days per week with establishing an Agile  governance framework. Finally, DHS established an Agile Integrated  Product Team that is co-chaired by PARM and the TIM Program  Manager. The team meets bi-weekly to provide guidance on adopting  Agile processes. As a result of the sustained leadership commitment, the  program is better positioned to continuously improve its Agile practices.

Key TIM Program Staff Have Received Agile Training to Enhance Knowledge

According to leading practices and guidance, an organization transitioning  to Agile software development should ensure that the entire program  team receives Agile training. This allows organizations to achieve a faster  shift away from the previous culture and processes and toward a more  agile culture.

Toward this end, the TIM program requires its Agile contractor to ensure  that development teams are trained and skilled in Agile methods, as well  as in the specific Agile frameworks the program has adopted, which  include the Scrum and SAFe frameworks. Additionally, the program  provided initial Agile training for key program staff when it began  transitioning to Agile software development. Specifically, the program  provided a mandatory 2-day Agile workshop in October and December  2016 which covered basic Agile principles and the Scrum and SAFe  frameworks. This training was provided to many key staff members,  including contractor support staff, a contracting officer representative, and  product owners.

Further, in December 2016, the program began providing training on the  SAFe framework to its government employees. This training was tailored  based on different roles, such as Agile practitioner, program manager or  product owner, and scrum master. The training courses were provided to  key staff members, including TIM program leadership, team leads, branch  managers, and scrum masters. As a result of providing Agile training, the  program’s staff should be able to more effectively adopt and apply Agile  software development processes.

TIM Program Product Owners Frequently Engage with Development Teams, but Roles and Responsibilities Are Not Clearly Defined

According to leading practices and guidance, an organization transitioning  to Agile software development should designate a product owner who  represents the user community and establishes priorities based on  business needs, approves user stories and their acceptance criteria, and  decides whether completed work meets the acceptance criteria and can  be considered done. The product owner should also maintain close  collaboration with the development teams by, among other things,  providing daily support to help clarify requirements and attending key  Agile meetings, such as sprint- and release-level planning sessions and  system demonstrations. Additionally, roles and responsibilities among  relevant stakeholders, such as the product owner, should be clearly  defined and documented by the organization that is transitioning to Agile  software development, so that the stakeholders are aware of their  responsibilities and given the authority to perform their roles.

The TIM program has two different groups of individuals that collectively  share the responsibilities of product owner, and while these groups  frequently engage with the development teams, program officials have  not yet clearly defined the groups’ roles and responsibilities. Specifically,  according to program officials, the first group consists of five product  owners that represent end users and are collectively responsible for  supporting all development teams, attending all Agile meetings, and  prioritizing and approving planned and completed work. In addition,  according to program officials, these five individuals are also responsible  for approving user stories associated with new system functionality.

The other group is referred to as the solutions team, which includes, for  example, the TIM Chief Architect and Chief Engineer. According to  program officials, the technical work (which is to help enable the system  functionality, such as ensuring network connectivity and proper software  licenses) is approved by the solutions team.

Nevertheless, while program officials told us about these high-level roles  and responsibilities, the program’s documentation does not clearly define  them among the five product owners and the solutions team. Moreover,  program officials have not defined the rules of engagement for these  product owners, such as how competing priorities among different  product owners should be handled.

According to program officials, the lack of clearly defined roles and  responsibilities has not been a problem for the program because the  product owners and the solutions team regularly communicate and  coordinate with each other, and thus far, have been in agreement on the  priorities for the program. However, the program recently scaled up the  amount of work being conducted simultaneously, which adds to the  volume of the decisions that need to be made and the coordination that  has to occur among the five product owners and solutions team. Thus,  even if the program has not yet experienced issues with coordination,  without more clarity in the roles and responsibilities among the groups  that are responsible for prioritizing and accepting work, the program risks  facing challenges in establishing priorities, approving user stories, and  deciding whether completed work meets the acceptance criteria.

The TIM Program Established a Vision, but It Does Not Always Align to the Requirements; Recent Corrective Actions Should Yield Improvements

According to leading practices and guidance, a program transitioning to  Agile software development should have a clearly defined vision. This can  be in the form of a product roadmap, to guide the development of the  product and to help inform the planning and requirements development of  Agile software development releases.

Consistent with leading practices, TSA established a vision for the TIM  program. This vision is articulated in multiple documents—including the  Mission Needs Statement, Concept of Operations, and Operational  Requirements Document. Officials also use a strategic roadmap to  articulate the program’s vision, which specifies the high-level system  capabilities that are to be deployed over the life-cycle of the program  through 2021.

However, the program’s vision has not always informed the planning of  requirements for the software releases, as intended by leading practices.  Specifically, the capabilities outlined in the program vision documents,  such as the strategic roadmap, do not consistently map to program  requirements. While 5 of the 10 capabilities in the strategic roadmap align  to the high-level and large scope requirements, referred to as epics, the  other half of the capabilities do not clearly align to the epics. For example,  the adjudication and redress capabilities that are in the strategic roadmap  do not align to any epic. In addition, the capability for public-facing portals  does not clearly track to any epic.

TIM officials recognized the alignment issues, and in August 2017, stated  that they are in the process of establishing alignment from the program’s  vision down to the lowest level of requirements, by refining the program’s  vision and requirements. Officials also stated that they expected this effort  to be completed by 2018. Effective execution of this effort should help  ensure the program’s vision is informing requirements planning.

Requirements for the TIM System Have Not Been Fully Prioritized

According to leading practices and guidance, a program transitioning to  Agile software development should have a prioritized list of the  requirements that are to be delivered—referred to as the backlog. This  backlog should be maintained so that the program can ensure it is always  working on the highest priority requirements that will deliver the most  value to the users. In addition, according to TIM Agile management  documentation and program officials, the program’s backlog of features  (i.e., mid-sized requirements) is expected to represent the features that  are to be delivered over the next several software releases. These  features are to be assigned priority levels to help determine which should  be selected for development when planning the next release.

According to TIM Agile management documentation, the TIM program is  expected to manage a backlog for each software release, which is to  identify the features and their derived user stories (i.e., the smallest and  most detailed requirements) that are to be delivered in a specific release.  The documentation also indicates that each feature and user story is to  be assigned priority levels to determine which should be included in the  development of the next release and associated sprint. Figure 5  illustrates the intended prioritization in the features, releases, and user  stories backlogs.

However, as of July 2017, the program’s backlogs did not contain specific  prioritization levels for each of the features and user stories, as called for  in DHS guidance. According to program officials, instead of assigning  specific prioritization levels, they had more generally identified which  features should be developed within the near-term (e.g., in the next  several Agile releases). Program officials recognized that they still  needed to prioritize their backlogs by assigning priority levels to all  features and user stories, but they did not have a time frame for  completing this effort.

Without ensuring full prioritization of current and future features and user  stories, the program is at risk of delivering functionality that is not aligned  with the highest needs of those that are responsible for conducting  security threat assessments to protect the nation’s critical transportation  infrastructure.

The TIM Program Has Been Delayed in Implementing Many of the Planned Automated System Development and Deployment Tools

According to leading practices and guidance, automating system  development and deployment work and avoiding manual work is  especially important for Agile programs, as it enhances the ability for  rapid development and delivery of high quality software. Specifically, a  program transitioning to Agile software development should use an  automated tool for managing Agile activities, such as maintaining the  product backlog and tracking the status of completed work. The program  should also establish automated testing and deployment capabilities to  improve the quality of the system. For example, according a DHS’s Agile  development instruction manual, the vast majority of software defects are  discovered during system integration testing, and—if automated—this  testing can be run multiple times on a sprint or release in order to identify  more defects sooner. In addition, automated tools can enable more  efficient processes for frequently integrating computer code that is  developed by different team members (e.g., hourly or daily), in order to  quickly detect any code integration errors. Automation of testing can also  help decrease the risk of introducing security flaws due to human error.

However, program officials deferred implementation of an automated  Agile program management tool and many other testing and deployment  tools. Specifically, while the program had been using Agile software  development practices since October 2016, the program has not used an  automated management tool for tracking the status of completed work for  its first three Agile software releases. Instead the program has used  spreadsheets that require TIM program officials to manually populate and  track large amounts of program status information.

Program officials had planned to implement an automated management  tool by October 2016, but did not do so until the end of April 2017.  According to the officials, the delay occurred because they were in the  process of tailoring the SAFe governance framework and the  management tool needed to be customized to reflect the tailored  approach.

Regarding tools for testing and deployment, as of May 2017, the program  was only using 4 of the16 automated tools that program officials planned  to use. These included tools that enable the management of software  code development, defect tracking, and components of automated  functional testing. However, the remaining 12 testing and deployment  tools had not yet been implemented. These include, among others, tools  that enable the automated building of software code, frequent merging of  an individual piece of software code with the main code repository so that  new changes are tested continuously (referred to as continuous  integration), small automated tests to verify that each individual unit of  code written by the developer works as intended, and installation of  application patches to protect against known vulnerabilities. TIM program  officials stated that these testing and deployment tools are not expected  to be implemented until the new development, testing, and production  environments are set up. However, as previously mentioned, the program  has experienced challenges in implementing these environments.

As a result, the program’s use of manual processes have been time  consuming, impeded visibility into the process, and hindered software  testing. In addition, without automated tools, program performance  metrics were being manually calculated and this increases the risk for  incomplete and inaccurate data. While the automated Agile management  tool has just been implemented, until the remainder of the automated  Agile testing and deployment tools are implemented, the program is likely  to continue to operate at reduced efficiency levels, and be limited in its  ability to ensure product quality.

TSA and DHS Have Not Fully Implemented Most Key Practices for Overseeing the TIM Program’s Cost, Schedule, and Performance

According to leading practices, to ensure effective program oversight of  cost, schedule, and performance, organizations should: ensure that corrective actions are identified and tracked until the  desired outcomes are achieved, document relevant governance and oversight policies and monitor program performance and progress, and  rely on complete and accurate data to review performance against  expectations.

While TSA fully implemented the first practice, the remaining three  practices were not fully implemented by DHS and TSA. As a result, the  effectiveness with which the governance bodies oversee and monitor the  program has been limited.

TSA Established a Process for Ensuring Corrective Actions Are Identified and Tracked for the TIM Program

According to leading practices, effective program oversight includes  ensuring that corrective actions are identified and tracked until the desired  outcomes are achieved. In this regard, governance bodies should collect  and analyze data on program risks and issues and determine corrective  actions to address them and track them to completion.

TSA has established a process for ensuring that corrective actions are  identified and tracked. Specifically, the program has a process for  identifying corrective actions and monitoring the status of these actions in  its weekly program status reviews. The program also uses an automated  tool to track and maintain a complete list of all actions that have been  identified. As of February 2017, the list contained 89 actions and included  the status of the actions—83 of which had been tracked to completion. As  a result of the program having a process that can identify and track  corrective actions, it is better positioned to address significant deviations  in cost, schedule, and performance parameters.

TSA and DHS Have Documented Selected Oversight and Governance Processes for the TIM Program, but Other Key Processes Are Underdeveloped

According to leading practices, effective program oversight includes the  use of documented policies and procedures for program governance and  oversight, such as reporting and control processes. These processes may  include, among others, requiring programs to report on the status and  progress of activities; expected or incurred program resource  requirements; known risks, risk response plans, and escalation criteria;  and benefits realized. Oversight and governance documentation may also  include threshold criteria to use when analyzing performance, and the  conditions under which a program or project would be terminated.

TSA and DHS have documented selected policies and procedures for  governance and oversight of the TIM program. Specifically, DHS  documented procedures for its Acquisition Review Board and its  Executive Steering Committee for the TIM program on how these  governance bodies are to review the cost, schedule, and performance of  the program. For example, according to the Committee’s charter, it is  responsible for assessing the health of the program and identifying major  issues and risks, utilizing a standard reporting format at oversight  meetings.

TSA has also documented processes for the program’s Agile milestone  reviews, such as conducting workshops at the end of the release cycle to  perform a system demonstration, review qualitative metrics, and promote  continuous quality improvement. TSA also developed a risk management  plan tailored for the Agile approach to guide TIM staff members in  identifying, managing, and mitigating risks and issues impacting cost,  schedule, and performance of the program. The agency also developed a  test and evaluation master plan that outlines how it and DHS will conduct  and oversee testing and evaluation of the program’s capabilities under  the new Agile software development approach.

However, TSA and DHS have not developed or finalized other key  oversight and governance documents. Specifically, three oversight and  governance policies have not been finalized and/or appropriately  updated: the TIM program’s tailoring plan for SAFe, a DHS-level oversight  policy for Agile programs, and DHS Office of the Chief Technology  Officer’s guidance for Agile programs to use for collecting and reporting  on performance metrics.

The TIM program has not updated its Systems Engineering Life Cycle  Tailoring Plan (which outlines the Agile governance process and all  milestone reviews that are required for planning and deploying Agile  releases), to reflect changes in the way officials have reported using  the SAFe governance framework. As a result, there are  inconsistencies in the governance documentation. For example, the  Systems Engineering Life Cycle Tailoring Plan describes four levels of  governance—portfolio, value stream, program, and team—while  program officials have reported omitting the value stream level from  the governance framework. According to TSA officials in May 2017,  they planned to update the Systems Engineering Life Cycle Tailoring  Plan to reflect the revised governance framework, but they did not  have a specific time frame for completing the revision. Until the TIM  program fully updates its Systems Engineering Life Cycle Tailoring  Plan to reflect the revised governance framework, the program lacks a  clearly documented and repeatable governance process to effectively  oversee the program.

DHS officials stated that they plan to conduct biannual oversight  reviews of the five Agile pilot programs (including TIM), instead of the  annual reviews that are typically conducted for traditional waterfall  development programs. According to the officials, the purpose of  moving to biannual reviews is to better ensure cost, schedule, and  performance remain on track for these Agile programs. However,  officials in the Office of the Chief Technology Officer stated that DHS- level Agile governance and oversight policies and procedures have  not been revised to reflect this new oversight approach because  consensus among DHS leadership on related changes needs to be  established before this new oversight approach can be documented in  the department’s guidance. As of May 2017, officials had not specified  a time frame for reaching such consensus. Until DHS leadership  reaches consensus on needed oversight and governance changes,  and then documents and implements associated changes, the  program continues to plan as though it is undergoing annual oversight  reviews, versus biannual reviews.

As of early May 2017, officials in the Office of the Chief Technology  Officer were also in the process of drafting guidance for Agile  programs to use for collecting and reporting on performance metrics,  but did not know when this guidance will be finalized. According to  TSA officials, in the absence of complete Agile guidance, the TIM  program receives support from DHS’s Agile team supporting the pilot  initiative, which, as specified in the team’s charter, is intended to help  the program (as well as the other four pilot programs) facilitate Agile  software development. However, this team is not intended to perform  oversight functions to ensure that the program is meeting cost,  schedule, and performance targets. Thus, until the Office of the Chief  Technology Officer completes guidance for Agile programs to use for  collecting and reporting on performance metrics, TIM program officials  may not report the most informative Agile performance metrics to  oversight entities.

TSA and DHS Consistently Conduct Program Performance Reviews, but Lack Insights from Key Performance Metrics

According to leading practices, effective program oversight includes  monitoring program performance and progress by comparing actual cost,  schedule, and performance data with estimates in the plan and identifying  significant deviations from established targets or thresholds for  acceptable performance levels. Program reviews are to be conducted at  predetermined checkpoints or milestones in order to determine progress  by measuring programs against cost, schedule, and performance metrics.  In addition, Agile programs should be measured on, among other things,  velocity (i.e., number of story points completed per sprint or release),  development progression (e.g., the number of features and user stories  planned and accepted), product quality (e.g., number of defects and unit  test coverage), and user satisfaction.

The TIM program management office conducts frequent and regular  performance reviews and focuses on several important Agile release- level metrics. Specifically, program management officials monitor TIM’s  performance and progress during weekly program status review meetings  and in periodic Agile reviews that are conducted at the end of each  release. These reviews also include officials from the development teams  and program stakeholders. The reviews focus on, among other things,  velocity, progress, and product quality. They also include the status of key  activities and risks impacting cost, schedule, and performance.

Nevertheless, while the program management office uses performance  metrics, the program has not established thresholds or targets for  acceptable performance levels for these metrics. For example, program  status reports showed that about 47 percent of the work that was planned  to be completed in the first Agile release was accepted by the product  owners. While the program appears to have been improving in this  metric—74 percent was accepted in the second Agile release and 94  percent in the third Agile release—program officials have not established  the thresholds or targets to determine the acceptable level of  performance.

Program officials stated that they considered the performance in the first  Agile release to be low, but they have not yet established targets or  thresholds. According to program officials, they planned to establish  targets based on the capacity of work that development teams are  expected to complete in a release, which can be better predicted as the  teams spend more time together. However, the program has since  developed three releases and continues to lack performance thresholds  and targets. Until program officials establish performance thresholds or  targets, oversight bodies may lack important information to ensure the  program is meeting acceptable performance levels.

In addition, the program management office’s performance reviews have  included limited information on program cost. According to TIM officials,  the program manager holds weekly meetings with the contract, finance,  and budget groups to review costs associated with TIM’s contracts.  However, management does not review or produce reports on overall life- cycle cost performance for the program or Agile software development  cost performance. Program officials said they have not yet determined  how best to measure cost performance in an Agile software development  environment. In September 2016, the Under Secretary for Management  instructed the program to collaborate with DHS’s Cost Analysis Division  and the headquarters-level Agile integrated product team to establish  agreed-upon software development cost metrics as well as a method for  collecting and reporting on those metrics by the end of the March 2017.  However, as of May 2017, this effort was still in progress. Until the TIM  program begins collecting and reporting on Agile-related cost, oversight  bodies will have limited information by which to monitor TIM costs.

Department-level oversight bodies have focused on reviewing certain  program life-cycle metrics for the TIM program. Specifically, the DHS  Acquisition Review Board conducts periodic reviews of the program to  monitor the program’s performance and hold the program accountable.

Since the program was rebaselined in September 2016 and transitioned  to Agile software development, the Acquisition Review Board has  conducted one review. In addition, the Executive Steering Committee,  which is chaired by the TSA CIO and Deputy Component Acquisition  Executive, and includes representatives from the DHS Chief Technology  Officer and PARM, reviews the program quarterly. As of July 2017, the  Executive Steering Committee had conducted three reviews of the TIM  program since implementing its new development approach. These  oversight bodies reviewed, for example, performance information such as  comparisons of the dates that milestones were actually achieved, against  the planned schedule, and the burnup charts for the program (i.e.,  graphical representations of accumulated story points planned and  completed per release).

However, the Acquisition Review Board and the Executive Steering  Committee have not been measuring the program against the rebaselined  life-cycle costs, or important Agile release-level metrics, which are  essential for providing early indicators of issues with the program. For  example, these oversight bodies did not review the program’s velocity,  number of features/user stories planned and accepted, product quality, or  Agile software development cost metrics.

In addition, while we have previously reported that there was overlap in  the DHS OCIO’s and the PARM office’s assessments of certain IT  programs, neither of these offices assessed the TIM program’s progress  against key Agile performance metrics or cost performance. Specifically,  the DHS OCIO and the PARM office conducted periodic (monthly or  quarterly health assessments) of the program that included, among other  things, schedule and system performance indicators for the entire life- cycle of the program (similar to what is used to review traditional waterfall  programs). While these metrics are useful for understanding the  program’s progress against the full schedule (60 months to full  operational capability, or 30 Agile releases), they do not offer insight into  the progress of individual Agile releases, which are deploying high-priority  capabilities for the TIM program every 2 months. For example, as of April  2017, these two oversight bodies did not include Agile performance  metrics which would have offered important insights into the progress of  individual releases, such as velocity, progress metrics, quality metrics,  post-deployment user satisfaction, or Agile software development costs.  Thus, until DHS-level oversight bodies review key Agile performance and  cost metrics and use them to inform management oversight decisions, the  oversight bodies will be limited in their ability to obtain early indicators of  any issues with the program, and to call for course correction, if needed.

Recently, the TIM program also began measuring user satisfaction.  Specifically, in April 2017, the DHS Acting Under Secretary for  Management directed TSA’s Operational Test Agent to implement a  continuous evaluation dashboard based on the results from the program’s  third Agile release by the end of June 2017. This dashboard was to  measure, among other things, post-deployment user satisfaction. TSA  subsequently implemented the continuous evaluation dashboard in June  2017.

Table 4 summarizes the extent to which performance metrics are  reviewed by various oversight bodies.

TSA and DHS Do Not Always Rely on Complete and Accurate TIM Performance Data

According to leading practices, effective program oversight includes  relying on complete and accurate data to review program performance  against stated expectations. Complete and accurate data allow oversight  bodies to have transparency into the performance of programs and helps  them identify when course correction is needed.

However, TIM’s reported performance data were not always complete  and accurate. Specifically, when reporting on the velocity (i.e., total  number of story points completed per sprint and/or release across the  development teams) of TIM’s first release after it was deployed, program  officials inconsistently reported velocity among the program’s  performance reports, thus calling into question the accuracy and  completeness of the information. Since the data were being reported on a  completed release, the velocity should have been reported as one  consistent number that did not change. According to program officials, the  reason for inconsistent reporting was that, despite best practices, the  program’s methodology for measuring velocity was not consistent and  was calculated differently each time. For example, table 5 shows three  different numbers that were to represent the collective velocity across the  development teams, and that officials reported to program management  after the deployment of the first software release.

While there was less variation in the velocity data reported after the  second software release was deployed, discrepancies were still present.  For example, table 6 shows the different numbers that officials reported to  TIM program management after the deployment of the second software  release.

Program officials stated that the reason for the inconsistencies in reported  velocity data was that during the first release they were still in the process  of adapting Agile and were working to determine how best to calculate  velocity. However, as shown in table 6 inconsistent data continued to  occur beyond that first release.

These inconsistencies in reported data call into question the  completeness and accuracy of the velocity numbers reported, and the  potential impact on oversight bodies’ ability to hold the program  accountable. For example, velocity is most useful when tracked over time  to ensure consistent performance and for forecasting how quickly  development teams can work through the items in a backlog. However,  without a complete and accurate velocity number from each release, it is  difficult for oversight bodies to ensure the program is producing work at  an acceptable pace to enable the program to meet its cost, schedule, and  performance targets.

In addition, the program had been reporting inaccurate unit test coverage  data using a manual measurement approach. Specifically, from  December 2016 to March 2017, program officials were reporting that, for  each release, they tested every line of code, based on a manual estimate  (i.e., 100 percent). However, testing each line of code manually is  unrealistic because with manual tests, it is difficult to determine which  function, line of code, or logic decision is executed, and which is not. As  such, program officials were reporting that they were testing every line of  code, even though they were unable to confirm that they were actually  doing so, thus calling into question the reliability and accuracy of the data  reported.

In response to our concerns, program officials acknowledged that they  could not confirm whether they had tested every line of code. Accordingly,  program officials stopped estimating this metric manually and stated that  they planned to begin measuring unit test coverage again once lines of  code could be tracked using automated tools. As previously discussed,  program officials stated that the testing and deployment tools are not  expected to be implemented until the new development, testing, and  production environments are set up. However, until the program has  complete and accurate unit test code coverage data, program officials will  not know if portions of its code are going untested, which could lead to  undetected issues and impact the quality of the product.

TSA’s TIM program has taken notable steps to address several of the  major issues it faced during prior system development and deployment  efforts, such as implementing system fixes to address critical  performance and usability issues found in the maritime segment.  Nonetheless, a number of significant challenges have not been fully  addressed. In particular, until the TIM program establishes specific time  frames for determining key implementation details, ensures its schedule  provides planned completion dates based on realistic estimates, and  establishes new time frames for implementing the actions identified in the  strategy, it is at significant risk of repeating past mistakes and  experiencing the same pitfalls as it did during its initial implementation  attempts. An indication of concern is that the program is currently  experiencing a delay of at least 6 months in the rebaselined schedule for  delivering TSA Pre® capabilities.

While the program has also taken certain steps to successfully make the  transition from a waterfall development approach to Agile software  development—a substantial and complex effort—TIM has not defined key  roles and responsibilities, prioritized features and user stories, or  implemented automated capabilities that are essential to ensuring  effective adoption of Agile. The gaps we identified with the program’s  implementation of Agile are concerning given that it did not follow key IT  acquisition best practices when using its waterfall development approach,  in which the program spent approximately 8 years and over $280 million  on a system that TSA has determined it needs to replace. While selected  corrective actions have been taken, until the TIM program is implemented  in accordance with leading practices, the program will be putting at risk its  ability to deliver a quality system that strengthens and enhances the  sophistication of TSA’s security threat assessment and credentialing  programs.

In addition, while TSA and DHS have implemented certain practices for  overseeing and governing the program, the lack of other practices has  impeded their oversight effectiveness, including the lack of thresholds or  targets for acceptable performance levels, the lack of reporting on Agile- related cost metrics, and inconsistent measuring and reporting of program  velocity and unit test coverage for software releases. These gaps limit the  ability of DHS oversight bodies to obtain early indicators of any issues  with the program, and to call for course corrections, if needed.

Further, until DHS leadership reaches consensus on needed oversight  and governance changes related to Agile programs, and then documents  and implements associated changes to align oversight reviews with the  timing of Agile software releases, the department will not be well  positioned to hold the program accountable. Moreover, until the Office of  the Chief Technology Officer completes guidance for Agile programs to  use for collecting and reporting on performance metrics, and DHS-level  oversight bodies require the TIM program to report on key Agile  performance and cost metrics and use them to inform management  oversight decisions, the department will also be limited in its ability to hold  the TIM program accountable and ensure that it is meeting its cost,  schedule, and performance targets.

We are making the following 14 recommendations to DHS:  The TSA Administrator should ensure that the TIM program management  office establishes and implements specific time frames for determining  key strategic implementation details, including how the program will  transition from the current state to the final TIM state. (Recommendation  1)

The TSA Administrator should ensure that the TIM program management  office establishes a schedule that provides planned completion dates  based on realistic estimates of how long it will take to deliver capabilities.  (Recommendation 2)

The TSA Administrator should ensure that the TIM program management  office establishes new time frames for implementing the actions identified  in the organizational change management strategy and effectively  executes against these time frames. (Recommendation 3)

The TSA Administrator should ensure that the TIM program management  office defines and documents the roles and responsibilities among  product owners, the solution team, and any other relevant stakeholders  for prioritizing and approving Agile software development work.  (Recommendation 4)

The TSA Administrator should ensure that the TIM program management  office establishes specific prioritization levels for current and future  features and user stories. (Recommendation 5)

The TSA Administrator should ensure that the TIM program management  office implements automated Agile management testing and deployment  tools, as soon as possible. (Recommendation 6)

The TSA Administrator should ensure that the TIM program management  office updates the Systems Engineering Life Cycle Tailoring Plan to  reflect the current governance framework and milestone review  processes. (Recommendation 7)

The TSA Administrator should ensure that the TIM program management  office establishes thresholds or targets for acceptable performance-levels.  (Recommendation 8)

The TSA Administrator should ensure that the TIM program management  office begins collecting and reporting on Agile-related cost metrics.  (Recommendation 9)

The TSA Administrator should ensure that the TIM program management  office ensures that program velocity is measured and reported  consistently. (Recommendation 10)

The TSA Administrator should ensure that the TIM program management  office ensures that unit test coverage for software releases is measured  and reported accurately. (Recommendation 11)

The Secretary of Homeland Security should direct the Under Secretary  for Management to ensure that appropriate DHS leadership reaches  consensus on needed oversight and governance changes related to the  frequency of reviewing Agile programs, and then documents and  implements associated changes. (Recommendation 12)

The Secretary of Homeland Security should direct the Under Secretary  for Management to ensure that the Office of the Chief Technology Officer  completes guidance for Agile programs to use for collecting and reporting  on performance metrics. (Recommendation 13)

The Secretary of Homeland Security should direct the Under Secretary  for Management to ensure that DHS-level oversight bodies review key  Agile performance and cost metrics for the TIM program and use them to  inform management oversight decisions. (Recommendation 14)

DHS provided written comments on a draft of this report, which are  reprinted in appendix II. In its comments, the department concurred with  all 14 of our recommendations and described actions it has planned or  taken to address them. For example, with regard to recommendation 6,  which calls for DHS to implement automated Agile management testing  and deployment tools, the department stated that TSA plans to implement  such tools by June 30, 2018. Additionally, for recommendation 14, the  department stated that DHS intends to ensure that oversight bodies  review key Agile performance and cost metrics for the TIM program by  June 30, 2018. If implemented effectively, these actions should address  the weaknesses we identified.

The department also described recent actions that it and TSA had taken  to address three of the recommendations, and requested that we  consider these recommendations resolved. Specifically, in response to  recommendation 9, calling for TSA to ensure that the TIM program  management office begins collecting and reporting on Agile-related cost  metrics, the department stated that the program is now reporting these  metrics on a monthly basis. In response to recommendation 10, calling for  TSA to ensure that the program’s velocity is measured and reported  consistently, the department stated that velocity is now being reported  consistently and in accordance with DHS guidelines. Further, in response  to recommendation 13, which calls for DHS to complete guidance for  Agile programs to use for collecting and reporting on performance  metrics, the department stated that the guidance had recently been  published and provided to us. However, to date, we have received only  draft versions of the guidance. We will work with the department to obtain  finalized documentation related to the three recommendations, to  determine if the recent actions fully address the recommendations.

In addition to the aforementioned comments, we received technical  comments from DHS and TSA officials, which we incorporated, as  appropriate.

We are sending copies of this report to the Secretary of Homeland  Security and interested congressional committees. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe the Transportation Security  Administration’s (TSA) past implementation efforts for the Technology  Infrastructure Modernization (TIM) program and its new implementation  strategy; (2) determine the extent to which TSA’s new strategy for the  program addresses the challenges encountered during earlier  implementation attempts; (3) determine the extent to which TSA has  implemented selected key practices for transitioning to an Agile software  development framework for the program; and (4) determine the extent to  which the TSA and the Department of Homeland Security (DHS) are  effectively overseeing and governing the TIM program to ensure that it is  meeting cost, schedule, and performance requirements.

To address our first objective, we reviewed program documentation, such  as initial and current acquisition program baselines, initial and current life- cycle cost estimates, acquisition decision memorandums, and program  plans documenting a new strategy for implementing the program. We  used the information in this documentation to summarize the program’s  earlier attempts to implement TIM capabilities and its new implementation  strategy for delivering the program, including estimated costs, schedule,  and key decisions made. We also interviewed TSA officials, including the  TIM Director and Deputy Director, on the status of TIM program office  efforts.

To determine the extent to which the TIM program’s new strategy  addresses the challenges encountered during earlier implementation  attempts, we reviewed documentation on the challenges the TIM program  faced when it breached cost and schedule thresholds and experienced  system performance issues, such as those described in initial operational  test reports, the breach remediation plan, and the results of a technical  evaluation of program challenges. We synthesized the information in  these documents to identify a consolidated list of key challenges the  program had faced. We did not include challenges that were already  being evaluated as part of other objectives, such as the use of the  waterfall software development approach. We then reviewed  documentation on the program’s new strategy, such as plans  documenting the new strategy, follow-on operational test reports, program  schedules, program status reports, and identified risks. We assessed the  extent to which the new strategy outlined in these documents addressed  the prior challenges by comparing them against criteria identified in  leading practices and guidance, such as DHS’s Systems Engineering  Lifecycle Guide and the Software Engineering Institute’s Capability  Maturity Model® Integration for Development. In addition, we conducted  a site visit at the TSA Adjudication Center in Reston, Virginia. During this  site visit, we observed demonstrations of the current commercial-off-the- shelf system and legacy systems for TSA Pre® and Aviation Workers,  and we interviewed adjudicators and supervisors on current security  threat assessment processes and limitations. Further, we interviewed  TSA officials, including the TIM Director and Deputy Director, on the  program office’s efforts to address prior challenges.

To determine the extent to which the program has implemented selected  key practices for transitioning to an Agile software development  framework, we identified leading practices and guidance outlined in the  following sources:

GAO, Software Development: Effective Practices and Federal  Challenges in Applying Agile Methods

Software Engineering Institute, Agile Readiness and Fit

Software Engineering Institute, Agile Metrics  After reviewing the sources listed, in consultation with our internal expert,  we grouped practices that were identified as being critical to establish  when transitioning to an Agile software development framework, and  selected the practices that were most relevant based on the status of the  program’s transition and we discussed the practice areas with TSA  officials. The practices included:  full support from leadership to adopt Agile processes, enhancing Agile knowledge, ensuring product owners are engaged with the development teams  and have clearly defined roles, establishing a clear product vision, prioritized backlogs of requirements, and  implementing automated tools to enable rapid system development  and deployment.

We reviewed program management documentation against these  practices, such as Agile training records, Agile contracts, program  roadmaps, backlogs, test plans, Agile release artifacts, program status  reports, and identified risks. Additionally, we observed Agile release and  sprint development activities at TSA facilities in Annapolis Junction,  Maryland, and at a contractor’s facilities in Beltsville, Maryland, and we  observed a demonstration of how user stories map from high-level  capabilities and tracked through development and testing.

We also interviewed TSA officials, including the TIM Director and Deputy  Director and the five TIM product owners, on their efforts to transition the  program to an Agile software development framework. Further, we  interviewed DHS officials, including the Chief Technology Officer, on their  efforts to conduct an Agile pilot to assist programs like TIM in adopting  Agile software development processes. We assessed the evidence  against leading practices to determine the extent to which TSA met the  practices.

To determine the extent to which TSA and DHS are effectively overseeing  and governing the program to ensure that it is meeting cost, schedule,  and performance requirements, we identified leading practices and  guidance outlined in the following sources:

Software Engineering Institute, Agile Metrics  After reviewing the sources listed, we grouped practices related to  oversight and governance for programs using Agile software development  into four key practice areas and we discussed the practices with DHS and  TSA officials. These areas included:

Document relevant governance and oversight policies and  procedures.

Monitor program performance and progress.

Rely on complete and accurate data to review performance against  expectations.

Ensure that corrective actions are identified and tracked until the  desired outcomes are achieved.

To assess the extent that TSA and DHS had addressed these key  practices, we reviewed the most current program management and  governance documentation as of April 2017. Specifically, we analyzed  documentation on program management processes, such as TIM’s  Systems Engineering Life Cycle Tailoring Plan, TIM Agile and Technical  Strategy, TIM Agile software development contract, and draft DHS Agile  Acquisition Program Delivery Metrics Playbook; and artifacts from TIM’s  program execution and review, such as Agile release artifacts, program  status reports, contractor status reports, program schedules, life-cycle  cost estimates, risk registers, TSA Executive Steering Committee  reviews, DHS program health assessments, DHS Agile pilot integrated  product team meetings, DHS Office of the Chief Technology Officer Agile  pilot reviews, and DHS Acquisition Review Board reviews.

Additionally, we interviewed TSA officials, including the TIM Director and  Deputy Director, on their efforts to oversee TIM’s development. Further,  we interviewed DHS officials, including the Chief Technology Officer, on  their efforts to oversee the program’s Agile software development  activities. We compared this evidence against leading practices to  determine the extent to which TSA and DHS met the practices.

To assess the reliability of the data that we used to support the findings in  this report, we reviewed relevant program documentation to substantiate  evidence obtained through interviews with agency officials. We  determined that the data used in this report were sufficiently reliable for  the purposes of our reporting objectives. We made appropriate attribution  indicating the sources of the data.

We conducted this performance audit from September 2016 to October  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, the following staff made key  contributions to this report: Shannin G. O’Neill (Assistant Director),  Jeanne Sung (Analyst in Charge), Jennifer Beddor, Rebecca Eyler, Bruce  Rackliff, and Dwayne Staten.