DHS Financial Management: Better Use of Best Practices Could Help Manage System Modernization Project Risks

To help address long-standing financial management system deficiencies, DHS initiated its TRIO project, which has focused on migrating three of its components to a modernized financial management system provided by IBC, an OMB-designated, federal SSP. House Report Number 3128 included a provision for GAO to assess the risks of DHS using IBC in connection with its modernization efforts.

This report examines (1) the extent to which DHS and the TRIO components followed best practices in analyzing alternatives, and the key factors, metrics, and processes used in their choice of a modernized financial management system; (2) the extent to which DHS managed the risks of using IBC for its TRIO project consistent with risk management best practices; and (3) the key factors and challenges that have impacted the TRIO project and DHS's plans for completing remaining key priorities. GAO interviewed key officials, reviewed relevant documents, and determined whether DHS followed best practices identified by GAO as necessary characteristics of a reliable, high-quality AOA process and other risk management best practices.

The Department of Homeland Security's (DHS) TRIO project represents a key effort to address long-standing financial management system deficiencies. During 2012 and 2013, the TRIO components—U.S. Coast Guard (Coast Guard), Transportation Security Administration (TSA), and Domestic Nuclear Detection Office (DNDO)—each completed an alternatives analysis (AA) to determine a preferred alternative for modernizing its financial management system. GAO found that DNDO's AA substantially met the four characteristics—well-documented, comprehensive, unbiased, and credible—that GAO previously identified for a reliable, high-quality analysis of alternatives (AOA) process. However, Coast Guard's and TSA's AAs did not fully or substantially meet three of these characteristics, and DHS guidance for conducting AAs did not substantially incorporate certain best practices, such as identifying significant risks and mitigation strategies and performing an independent review to help validate the AOA process. Based on these analyses and other factors, the TRIO components determined that migrating to a federal shared service provider (SSP) represented the best alternative, and in 2014, DHS selected the Department of the Interior's Interior Business Center (IBC) as the federal SSP for the project. However, because Coast Guard's and TSA's AAs did not fully or substantially reflect all of the characteristics noted above, they are at increased risk that the alternative selected may not achieve mission needs.

DHS also did not fully follow best practices for managing project risks related to its use of IBC on the TRIO project. Specifically, DHS followed three of seven risk management best practices, such as determining risk sources and categories and establishing a risk management strategy. However, it did not fully follow four best practices for defining risk parameters, identifying risks, developing risk mitigation plans, and implementing these plans largely because its guidance did not sufficiently address these best practices. For example, although DHS created joint teams with IBC and provided additional resources to IBC to help address risk mitigation concerns, it did not always develop sufficiently detailed risk mitigation plans that also included contingency plans for selected critical risks. As a result, although IBC's capacity and experience for migrating large agencies the size of Coast Guard and TSA was identified as a risk in July 2014, a contingency plan working group to address this concern was not established until January 2017. By not fully following risk management best practices, DHS is at increased risk that potential problems may not be identified or properly mitigated.

DHS, IBC, Office of Management and Budget (OMB), and other federal oversight agencies identified various challenges that have impacted the TRIO project and contributed to a 2-year delay in the implementation of Coast Guard's and TSA's modernized solutions. These challenges include the lack of sufficient resources, aggressive schedule, complex requirements, increased costs, and project management and communication concerns. To help address these challenges, DHS and IBC established review teams and have taken other steps to assess potential mitigating steps. In May 2017, DHS determined that migrating the solution from IBC to a DHS data center represented the best option and initiated discovery efforts to further assess this as its path forward for the TRIO project.

GAO recommends that DHS more fully follow best practices for conducting an AOA process and managing risks. DHS concurred with GAO's recommendations and described actions it will take, or has taken, in response.

Since the Department of Homeland Security’s (DHS) creation in 2003,  significant internal control and financial management system deficiencies  have hampered its ability to manage operations and to reasonably assure  effective financial management. DHS received a clean audit opinion on  its financial statements for 4 consecutive years—fiscal years 2013, 2014,  2015 and 2016—in which it reported assets totaling $98 billion as of  September 30, 2016. However, according to DHS’s financial statement  auditors, these financial management deficiencies have persisted, and  accordingly, DHS faces long-term challenges in sustaining a clean audit  opinion on its financial statements and providing reliable, timely, and  useful financial data to support operational decision making. These  deficiencies contributed to our decision to designate DHS’s management  functions, including financial management, as high risk. As noted in our  2017 high-risk report, until remediated, these deficiencies will continue to  hamper DHS’s ability to establish effective internal controls over financial  reporting and comply with financial management system requirements.

In 2013, we reported on challenges related to DHS’s previous attempts to  implement a department-wide integrated financial management system.  DHS subsequently adopted a decentralized approach to upgrade or  replace legacy financial management systems. DHS has been evaluating  various options for modernizing its systems, including the use of shared  service providers (SSP). As part of this effort, DHS initiated three projects  for modernizing the systems of selected DHS components, including its  TRIO modernization project. The TRIO project has focused on migrating  the Domestic Nuclear Detection Office (DNDO), Transportation Security  Administration (TSA), and U.S. Coast Guard (Coast Guard), the TRIO  components, to a modernized financial management system solution  provided by the Department of the Interior’s Interior Business Center  (IBC). As of May 2017, significant challenges have also impacted the  TRIO project, including a 2-year delay in Coast Guard’s and TSA’s  expected migrations to a modernized financial management system.

In May 2014, the Office of Management and Budget (OMB) and the  Department of the Treasury (Treasury) designated IBC as one of four  federal SSPs for financial management to provide core accounting and  other services to federal agencies. In addition, Treasury’s Office of  Financial Innovation and Transformation’s (FIT) responsibilities related to  the governance and oversight of federal SSPs were subsequently  transferred to the Unified Shared Services Management office (USSM)  after USSM was established in October 2015 as an entity within the  General Services Administration.

Recognizing the cost and performance risks associated with DHS’s  efforts to transition to a SSP, the House report accompanying H.R. 3128,  a bill for DHS appropriations for fiscal year 2016, included a provision for  GAO to assess the risks of DHS using IBC in connection with DHS’s  financial management system modernization efforts. Our objectives were  to determine (1) the extent to which DHS and the TRIO components  followed best practices in analyzing alternatives, and the key factors,  metrics, and processes used in their choice of a modernized financial  management system; (2) the extent to which DHS managed the risks of  using IBC for its TRIO project consistent with risk management best  practices; and (3) the key factors and challenges that have impacted the  TRIO project according to DHS, IBC, OMB, FIT, and USSM and DHS’s  plans for completing the remaining key priorities.

To determine the extent to which DHS and TRIO components followed  best practices in analyzing alternatives, we reviewed documentation  developed by the TRIO components and evaluated the TRIO  components’ alternatives analyses (AA) processes against GAO’s  identified 22 best practices for conducting an analysis of alternatives  (AOA) process. Based on this assessment, we determined an overall  score for the TRIO components’ AA process as compared to the AOA  best practices in four summary characteristics—(1) well-documented,   (2) comprehensive, (3) unbiased, and (4) credible—for a reliable, high- quality AOA process. We provided our preliminary assessment to DHS  and the TRIO components for review and revised our assessment, as  needed, based on any additional information they provided. In addition,  we reviewed DHS’s guidance for conducting AOAs and AAs against  GAO’s identified AOA best practices.

To determine the key factors, metrics, and processes used by DHS’s  TRIO components in developing and evaluating DHS’s alternative  solutions and final choice for financial system modernization, we reviewed  each component’s AA, including a description of (1) the alternatives  considered, (2) the market research conducted, (3) the alternatives  evaluated, (4) the selection criteria used and how the criteria were  weighted, (5) how each alternative scored against the selection criteria,  and (6) the alternative that scored the best according to the components’  evaluations.

To determine the extent to which DHS managed the risks of using IBC  consistent with risk management best practices, we reviewed DHS’s and  the TRIO components’ risk management guidance and other  documentation supporting their risk management efforts, including risk  registers, mitigation plans, status reports, and risk management meeting  minutes. We also met with officials to gain an understanding of the key  processes and documents used for managing and reporting on TRIO  project risks. We compared the documentation and information we  obtained to best practices published by the Software Engineering Institute  (SEI) for preparing for risk management, identifying and analyzing risks,  and mitigating identified risks.

To identify the key factors and challenges that have impacted the TRIO  project and DHS’s plans for completing its remaining key priorities, we  met with the parties involved in the TRIO project consisting of DHS, IBC,  FIT, and USSM officials, as well as OMB staff to obtain their perspectives.  In addition, we reviewed documentation related to the TRIO project,  including status reports and memorandums, leadership briefings, and  other presentations.

We conducted this performance audit from March 2016 to September  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Appendix I provides  additional details on our scope and methodology.

Since DHS’s creation in 2003, significant internal control and financial  management system deficiencies have hampered its ability to reasonably  assure effective financial management and to manage operations. These  deficiencies contributed to our decision to designate DHS’s management  functions, including financial management, as high risk. To help address  these deficiencies, DHS initiated a decentralized approach to upgrade or  replace legacy financial management systems and has been evaluating  various options for modernizing them, including the use of SSPs. DHS  initiated three projects for modernizing the systems of selected DHS  components, including its TRIO modernization project. The TRIO project  has focused on migrating the financial management systems of Coast  Guard, DNDO, and TSA to a modernized solution provided by IBC. DHS’s  efforts to effectively assess and manage risks associated with this project  are essential to DHS’s realizing its modernization goals.

In 2013, OMB issued a memorandum directing agencies to consider  federal SSPs as part of their AAs. Also, in May 2014, Treasury and  OMB designated IBC as one of four federal SSPs for financial  management to provide core accounting and other services to federal  agencies. This designation was based on Treasury and OMB’s evaluation  of the four service providers’ ability to assist federal agencies in meeting  their accounting and financial management needs, including experience  with implementing financial management systems and providing other  financial management services to customers, cost of services provided,  compliance with financial management and internal control requirements,  commitment to shared services, capacity, and long-term growth strategy.  FIT’s responsibilities related to the governance and oversight of federal  SSPs were subsequently transferred to USSM after USSM was  established in October 2015.

Because of concerns that its Core Accounting System (CAS) Suite was  outdated, inefficient, and did not reliably meet requirements, Coast Guard  completed an AA in January 2012 to assist in developing a path forward  for modernizing its financial management system. In August 2012, Coast  Guard established its CAS Replacement project team to further evaluate  two of the alternatives considered in its AA and develop a recommended  course of action. In addition, Coast Guard determined that hosting,  owning, operating, and managing a financial management system were  not among its core competencies. Because TSA and DNDO also relied  on CAS as their primary accounting system, they also conducted AAs to  identify the best alternative for transitioning to a modernized financial  management system solution.

The AAs conducted by the TRIO components during 2012 and 2013  considered the use of federal and commercial SSPs and other options. In  addition, Coast Guard completed additional market research including  further analysis of commercial SSPs in June 2013. In July 2013, the TRIO  components determined that migrating to a federal SSP was the best  course of action and subsequently conducted discovery phase efforts with  IBC from November 2013 through May 2014 to further explore the  functional requirements for procurement, asset, and financial  management services. Based on these efforts, in July 2014, the TRIO  components recommended that they proceed with implementation of the  IBC shared services solution. In August 2014, FIT and OMB concurred  with this recommendation, and DHS entered into an interagency  agreement (IAA) with IBC for implementation. Figure 1 shows a timeline  of these key events.

The IAA for implementation and related performance work statement  included a description of the services that IBC is to provide and the roles  and responsibilities of DHS, the TRIO components, and IBC. The IAA  also required IBC to prepare a detailed project management plan  describing how the requirements would be managed and updated and an  integrated master schedule (IMS) for identifying tasks to be completed,  duration, percentage completed, dependencies, critical path, and  milestones.

According to the February 2015 project management plan, DNDO, TSA,  and Coast Guard were expected to go-live on the IBC solution in the first  quarter of fiscal years 2016, 2017, and 2018, respectively. However, in  May 2016, DHS and IBC determined that TSA’s and Coast Guard’s  planned implementation dates were not viable because of various  challenges impacting the TRIO project and recommended a 1-year delay  for their respective implementation dates. Figure 2 summarizes planned  and completed key implementation events for the TRIO project as of May  2016.

Best Practices for Conducting Analysis of Alternatives and Managing Risks

GAO, SEI, and other entities have developed and identified best practices  to help guide organizations in effectively planning and managing various  activities, including acquisitions of major information technology systems.  These include GAO’s identified best practices for the AOA process and  best practices identified by SEI for risk management.

GAO-identified best practices for AOA process. GAO identified 22  best practices for a reliable, high-quality AOA process that can be  applied to a wide range of activities in which an alternative must be  selected from a set of possible options, as well as to a broad range of  capability areas, projects, and programs. These practices can provide  a framework to help ensure that entities consistently and reliably  select the project alternative that best meets mission needs. Not  conforming to these best practices may lead to an unreliable process,  and the entity will lack assurance that the preferred alternative best  meets the mission needs. Appendix II provides additional details on  GAO’s identified AOA process best practices and how they can be  applied to a wide range of activities in which an alternative must be  selected from a set of possible options, as well as to a broad range of  capability areas, projects, and programs.

SEI’s risk management practices. SEI’s practices for the risk  management process area call for the identification of potential  problems before they occur so that risk-handling activities can be  planned throughout the life of a project to mitigate adverse impacts on  achieving objectives. These practices are determining risk sources and categories, defining parameters used to analyze and categorize risks and to  control the risk management effort, establishing and maintaining the strategy to be used for risk  identifying and documenting risks, evaluating and categorizing each identified risk using defined risk  categories and parameters and determining its relative priority, developing a risk mitigation plan in accordance with the risk monitoring the status of each risk periodically and implementing the  risk mitigation plan as appropriate.

DHS Did Not Always Follow Best Practices for Analyzing Alternatives for TRIO Components’ Choice of Modernized Financial Management System

Although the TRIO components conducted AAs to identify the preferred  alternative for modernizing their financial management systems, their  efforts did not always follow best practices. For example, Coast Guard’s  and TSA’s AAs supporting their selection of migrating to a federal SSP for  modernizing their financial management systems did not fully or  substantially meet all four characteristics of a reliable, high-quality AOA  process. In addition, we found that DHS guidance did not fully or  substantially incorporate five of GAO’s identified best practices for  conducting an AOA process. The TRIO components’ AAs included  descriptions of the key factors, such as scores for each alternative  against the selection criteria used to assess it. Based on these AAs, DHS  and the TRIO components selected the federal SSP alternative as their  preferred choice and subsequently selected IBC as their federal SSP.  However, because Coast Guard’s and TSA’s AAs did not fully or  substantially meet all four characteristics of a reliable, high-quality AOA  process, they are at increased risk regarding their decision on the solution  that represents the best alternative for meeting their mission needs.

DNDO Substantially, and Coast Guard and TSA Partially, Met Best Practices for Conducting AOAs

Based on the extent to which the DHS TRIO components followed the  GAO-identified 22 best practices for conducting an AOA process, we  found that DNDO’s AA process substantially met the four characteristics  of a reliable, high-quality AOA process while the Coast Guard and TSA  AA processes both substantially met one and partially met three of these  four characteristics. For example, we found that TSA’s AA partially met  the “well-documented” characteristic, in part, because risk mitigation  strategies, assumptions, and constraints associated with each alternative  were not discussed in its AA. In addition, we found that Coast Guard’s AA  partially met the “credible” characteristic, in part, because there was no  indication that it contained sensitivity analyses, an evaluation of the  impact of changing assumptions on its overall costs or benefits analyses.  Our overall assessment is summarized in table 1. Appendix III provides  additional details on our assessment of the TRIO components’ AAs for  each of the GAO-identified 22 AOA best practices.

Further, in comparing DHS AOA and AA guidance to the GAO-identified  22 AOA process best practices, we found that although DHS’s guidance  for conducting both AOAs and AAs fully or substantially incorporated 17  of the identified best practices, the guidance did not fully or substantially  incorporate 5 of these practices. For example, although the guidance  addressed risk management in general terms, it did not detail the need to  document risk mitigation strategies for each alternative. Not documenting  the risks and related mitigation strategies for each alternative prevents  decision makers from performing a meaningful trade-off analysis  necessary to choose a recommended alternative. In addition, while DHS  guidance describes the need for an AA or AOA review, it describes  reviews conducted within the organizational chain of command and does  not address the need for an independent review—one of the most reliable  means to validate an AOA process. Further, although the guidance noted  that weights for selection criteria may become more subjective when they  cannot be derived analytically, additional guidance on weighting selection  criteria was limited. Our overall assessment is summarized in table 2.

Because of these limitations in guidance, and because Coast Guard and  TSA did not fully adhere to the GAO-identified best practices, Coast  Guard’s and TSA’s AAs did not fully or substantially reflect all four  characteristics of a reliable, high-quality AOA process. As a result, Coast  Guard and TSA increased their risk of selecting a solution that may not  represent the best alternative for meeting their mission needs.

TRIO Components Used Key Factors, Metrics, and Processes to Analyze Alternatives and Related Results

Documentation supporting TRIO components’ AA efforts included  descriptions of the key factors, metrics, and processes involved in  conducting their analyses, including the (1) alternatives considered,   (2) market research conducted, (3) three alternatives evaluated,   (4) selection criteria used by each and how the criteria were weighted,   (5) scores for each alternative against the selection criteria, and   (6) alternatives that scored the best under the AOA evaluation.

The TRIO components conducted market research to develop reasonable  alternative solutions for consideration. For example, through its market  research, TSA identified OMB-designated federal SSPs and commercial  entities as potential alternatives for hosting and implementing a  modernized and integrated financial management system. According to  its AA, TSA was able to gain an understanding of the offerings,  capabilities, and related costs associated with these alternatives through  reviews of documentation and interviews.

After developing a diverse range of financial system modernization  alternatives for consideration, each of the TRIO components assessed  them for viability using various factors—such as measures of  effectiveness, cost, risk, and value—and identified the three top-rated  alternatives for further evaluation. For example, Coast Guard identified  nine alternatives for consideration and analyzed, scored, and ranked  them to determine its top three alternatives for further analysis:  incrementally improve the current CAS Suite and remove certain  outdated components, host the financial management system internally using software and  tools already owned, and use an SSP to host the financial management system.

Each component identified its three alternatives for further evaluation and  used defined selection criteria to rate them. For example, DNDO’s  selection criteria included four categories of operational effectiveness that  were weighted according to their level of importance. Based on their  evaluations, each component identified the best alternative for its  respective financial management system needs.

According to Coast Guard’s November 2012 decision memorandum,  Coast Guard further narrowed the alternatives it focused on to (1) using  an SSP to host its financial management system and (2) hosting the  system internally using already-owned software and tools, and it also  gathered rough order of magnitude cost estimates for both alternatives.  Based on its evaluation, Coast Guard determined that the two alternatives  were comparable. According to this memorandum, Coast Guard further  determined that owning, hosting, operating, and managing a financial  management system were not among its core competencies. Based on  this determination, OMB direction to agencies to use (with limited  exceptions) shared services, and other factors, Coast Guard decided that  migrating to an SSP was the best alternative. TSA found in its February  2013 analysis that the differences between federal and commercial SSP  alternatives were not significant and, as a result, recommended that a  competitive procurement be conducted to better evaluate each  alternative. However, DHS officials told us that TSA subsequently  determined that a competitive procurement was not warranted and chose  to migrate to a federal SSP. This determination was based on additional  OMB guidance issued in March 2013 directing agencies to consider  federal SSPs as part of their AAs and stating that commercial SSPs are  an appropriate solution and would be funded by OMB only in instances in  which the agency’s business case demonstrates that a commercial SSP  can provide a better value for the federal government. In addition,  DNDO determined that migrating to a federal SSP was its best alternative  in May 2013.

Because its preliminary research focused primarily on the federal SSP  marketplace, Coast Guard conducted additional market research to  include a more robust analysis of commercial SSPs. Coast Guard’s June  2013 market research report described the results of this effort, including  its evaluation of responses from 11 commercial SSPs. Coast Guard  reported that none of the commercial SSPs that responded could meet all  44 specific financial management system requirements and the extent to  which they could meet them varied significantly. Based on these results,  Coast Guard determined that there was a lack of maturity in the  commercial SSP market for federal financial management. According to  the report, this overall assessment was based on various considerations  of information provided by commercial SSP respondents, including  the wide variety of proposed configurations, solutions, prices, and  implementation schedules, the lack of federal experience and service  for agency-wide capabilities, and insufficient length of service to  establish positive trends in audit performance;  the lack of similar offerings that implied a lack of strong competition  between comparable products that would exert downward pressure  on cost; and  the lack of like product offerings, which increases the likelihood of  higher switching costs in the case of poor performance because of  increased difficulty in moving from one “turnkey” service to another.

In July 2013, the TRIO components and DHS selected the federal SSP  alternative as their preferred choice and subsequently selected IBC as  their federal SSP. DHS officials told us that IBC was selected based on  (1) DHS’s reliance on OMB and Treasury’s designation of IBC as a  federal SSP, (2) OMB guidance to consider the use of federal SSPs, and  (3) a review of the availability of the four federal SSPs indicating that IBC  was the only one available to meet the requirements and implementation  schedule at that time. In August 2013, DHS notified OMB that the TRIO  components had performed extensive market research and finalized their  respective AAs and independently concluded that migrating to a federal  SSP was in the best interests of the government. Also, in August 2013,  FIT notified OMB regarding the TRIO components’ AA efforts and that the  TRIO components would proceed to the discovery phase with IBC.  According to FIT’s notification memorandum to OMB, the TRIO  components’ AAs demonstrated that migrating to a federal SSP was the  best value to the federal government and that the components identified  IBC as a suitable partner based on the results of their market research  into federal SSPs.

DHS Met Three and Partially Met Four Best Practices for Managing the Risks of Using IBC for the TRIO Project

Risk management best practices call for the identification of potential  problems before they occur so that risk-handling activities can be planned  throughout the life of the project to mitigate adverse impacts on achieving  objectives. These best practices involve (1) preparing for risk  management, (2) identifying and analyzing risks, and (3) mitigating  identified risks. Preparing for risk management involves determining risk  sources and categories and developing risk mitigation techniques.  Identifying and analyzing risks includes determining those that are  associated with cost, schedule, and performance and evaluating identified  risks using defined risk parameters. Mitigating risks includes determining  the levels and thresholds at which a risk becomes unacceptable and  triggers the execution of a risk mitigation plan or contingency plan;  determining the costs and benefits of implementing the risk mitigation  plan for each risk; monitoring risk status; and providing a method for  tracking open risk-handling action items to closure.

Based on our evaluation, we found that DHS processes generally  reflected three of seven specific risk management best practices and  partially reflected the remaining four practices. Table 3 summarizes the  extent to which DHS followed these seven best practices for managing  TRIO project risks. Additional details on DHS and TRIO component  efforts to address these practices are summarized following this table.

Prepare for risk management. Key aspects of processes established by  DHS and TRIO components related to the three best practices associated  with preparing for risk management:

Determine risk sources and categories. This practice calls for a basis  for systematically examining circumstances that affect the ability of  the project to meet its objective and a mechanism for collecting and  organizing risks. DHS and the TRIO components established  processes that met this best practice. For example, DHS reviewed the  integrated master schedule that IBC prepared to identify sources of  risk and defined risk categories in TRIO project policies.

Define risk parameters. Risk parameters are used to provide common  and consistent criteria for comparing risks to be managed. The best  practice includes defining criteria for evaluating and quantifying risk  likelihood and severity levels and defining thresholds for each risk  category to determine whether risk is acceptable or unacceptable and  to trigger management action. DHS partially met this best practice.  DHS’s risk management program defined rating scales to provide  consistent criteria for evaluating and quantifying risk likelihood and  severity levels. However, DHS’s Risk Management Planning  Handbook and related template for developing risk management  plans for projects did not address the need for thresholds relevant to  each category of risk to facilitate review of performance metrics in  order to determine when risks become unacceptable or to invoke  selected risk-handling options when monitored risks exceed defined  thresholds.

Establish a risk management strategy. A risk management strategy  addresses specific actions and the management approach used to  apply and control the risk management program, including identifying  sources of risk, the scheme used to categorize risks, and parameters  used to evaluate and control risks for effective handling. DHS met this  best practice. DHS and IBC established risk management policies and  plans for the TRIO project based on DHS acquisition guidance, which  provided a framework for a risk management program. Collectively,  these policies and plans constitute a risk management strategy. DHS  and IBC have periodically updated these documents to maintain the  scope of the risk management effort; the methods and tools to be  used for risk identification, risk analysis, risk mitigation, risk  monitoring, and communication; the prioritization of risks; and the  allocation of resources for risk mitigation.

Identify and analyze risks. Key aspects of processes established by  DHS and the TRIO components related to the two best practices  associated with identifying and analyzing risks:  Identify risks. Risk identification should be an organized, thorough  process to seek out probable or realistic risks to achieving objectives.  This practice recognizes that risks should be identified and described  understandably before they can be analyzed and managed properly.  Using categories and parameters developed in the risk management  strategy and identified sources of risk guides the identification of risks  associated with cost, schedule, and performance. To identify risks,  best practice elements include reviewing the work breakdown  structure (WBS) and project plan to help ensure that all aspects of  the work have been considered. Best practices for documenting risks  include documenting the context, conditions, and potential  consequences of each risk and identifying the relevant stakeholders  associated with each risk.

DHS partially met this best practice. DHS’s July 2016 risk register  contained a wide range of risks associated with defined risk  categories. It also reflected DHS’s review of the TRIO project’s  integrated master schedule that IBC prepared based on the WBS and  work plans that IBC also developed. The risk register documented  the context, conditions, potential consequences, and relevant  stakeholders associated with each risk. However, DHS’s documented  risk management processes did not identify all significant risks or  reflect its efforts to revisit risks that had previously been closed. For  example, DHS officials told us that IBC was unable to provide  sufficient, reliable cost and schedule information for project  monitoring; however, a risk reflecting these concerns was not included  on its July 2016 risk register. Further, the risk register included certain  closed risks related to the need for a governance structure and  strategy for ensuring that IBC met performance, cost, and schedule  objectives. Although DHS had ongoing concerns about its ability to  ensure that IBC met these objectives, the risk register did not reflect  efforts to revisit these risks to determine whether their status needed  revision or if other risks should be included on the risk register to  address its accountability concerns. In addition, DHS did not always  take timely action to document its consideration of risks identified by  its independent verification and validation (IV&V) contractor for  potential inclusion on its risk register. For example, the IV&V  contractor identified a risk related to inefficiencies in DHS’s document  review process in June 2015 that was not included on DHS’s risk  register until February 2016. DHS officials indicated that a crosswalk  between the DHS risk register and IV&V contractor risk management  observations was performed weekly; however, results of these weekly  reviews were not documented.

Evaluate, categorize, and prioritize risks. Risk assessment uses  defined categories and parameters to determine the priority of each  risk to assist in determining when appropriate management attention  is required. Best practices for analyzing risks include categorizing  risks according to defined risk categories, evaluating identified risks  using defined risk parameters, and prioritizing risks for mitigation.  DHS’s processes met this practice. For example, the documented risk  management program included application of defined risk categories  and parameters for all identified risks, providing a means for reviewing  risks and determining the likelihood and severity of risks being  realized. The TRIO project’s Joint Risk Management Integrated  Project Team provided consistency to the application of parameters  by reviewing risk assessments when risks were first identified. By  determining exposure ratings for each identified risk, DHS prioritized  risks for monitoring and allocation of resources for risk mitigation.

Mitigate risks. Key aspects of processes established by DHS and the  TRIO components related to the two best practices associated with  mitigating risks:

Develop risk mitigation plans. Risk mitigation plans are developed in  accordance with the risk management strategy and include a  recommended course of action for each critical risk. The risk  mitigation plan for a given risk includes techniques and methods used  to avoid, reduce, and control the probability of risk occurrence; the  extent of damage incurred should the risk occur; or both. Elements of  this practice include determining the levels and thresholds that define  when a risk becomes unacceptable and triggers the execution of a  risk mitigation plan or contingency plan, identifying the person or  group responsible for addressing each risk, determining the costs and  benefits of implementing the risk mitigation plan for each risk,  developing an overall risk mitigation plan for the work to orchestrate  the implementation of individual risk mitigation plans, and developing  contingency plans for selected critical risks in the event impacts  associated with the risks are realized.

DHS partially met this best practice. DHS’s risk management program  documentation reflected the development of risk response plans for  most risks, including all those determined to be of medium and high  exposure level. DHS identified those responsible for addressing each  risk. However, DHS and IBC did not always develop sufficiently  detailed risk mitigation plans including specific risk-handling action  items, determination of the costs and benefits of implementing the risk  mitigation plan for each risk, and developing contingency plans for  selected critical risks in the event that their impacts are realized. For  example, a risk associated with IBC’s capacity and experience for  migrating large agencies the size of Coast Guard and TSA was  identified in July 2014. Although DHS developed plans to help  mitigate this risk, a contingency plan was not developed prior to  realizing the adverse impact of not implementing Coast Guard and  TSA on IBC’s modernized solution. Rather, a contingency plan  working group (CPWG) to address this and other concerns was  established in January 2017, over 2 years after the risk was initially  identified. Further, thresholds were not used within the risk  management program to define when a risk becomes unacceptable,  triggering the execution of a risk mitigation plan or contingency plan.

Implement risk mitigation plans. Risk mitigation plans are  implemented to facilitate a proactive program to regularly monitor  risks and the status and results of risk-handling actions to effectively  control and manage risks during the work effort. Best practice  elements include revisiting and reevaluating risk status at regular  intervals to support the discovery of new risks or new risk-handling  options that can require reassessment of risks and re-planning of risk  mitigation efforts. Elements also include providing a method for  tracking open risk-handling action items to closure, establishing a  schedule or period of performance for each risk-handling activity,  invoking selected risk-handling options when monitored risks exceed  defined thresholds, and providing a continued commitment of  resources for each risk mitigation plan.

DHS partially met this best practice. Risk monitoring of the TRIO  project consisted of reviews performed by DHS and TRIO component  officials responsible for risk management and oversight functions.  These reviews considered significant risks, risks approaching  realization events, and the effect of management intervention on the  resolution of risks. These reviews also relied, in part, on data  contained in DHS’s risk register, which represents the official  repository of TRIO project risks and information on the status of risks  and related risk mitigation efforts.

However, other aspects of DHS’s efforts to implement risk mitigation  plans did not fully adhere to certain elements associated with this best  practice. For example, we identified certain issues that raised  questions concerning the accuracy of data contained in the risk  register, such as (1) the lack of clear markings indicating when the  accuracy of data on each risk was last confirmed, including risk  records that had not been modified in the previous 3 months, and   (2) certain risks for which the estimated risk impact date had already  occurred but its status risk according to DHS’s risk register did not  reflect that it had been realized and become an issue. In addition,  DHS officials stated that IBC did not provide sufficiently detailed,  reliable cost and schedule information that could have been used to  monitor TRIO project risks more effectively. DHS’s ability to monitor  cost, schedule, and other performance metrics was also limited  because of the lack of thresholds for management involvement, as  noted above. DHS’s implementation of risk monitoring plans was  further limited by other issues, including (1) a period of performance  for each risk-handling activity, which includes a start date and  anticipated completion date to control and monitor risk mitigation  efforts, was not always established and (2) an inability to fully track  open risk-handling action items to closure existed because of the lack  of sufficient detail on specific risk-handling activities in the DHS risk  register.

According to DHS officials, DHS relied heavily on IBC to manage risks  associated with the TRIO project and, in particular, those for which IBC  was assigned as the risk owner. They also acknowledged DHS’s  responsibility for overseeing IBC’s TRIO project risk management efforts  and described various actions taken to address growing concerns  regarding IBC’s efforts. For example, DHS created the Joint Risk  Management Integrated Project Team, in part, to provide a forum in which  IBC could obtain assistance in developing risk responses and discuss  DHS’s risk mitigation concerns. Further, to help reduce exposure of  underlying risks, DHS offered assistance to IBC’s project management  functions, such as developing the integrated master schedule and  performing quality control checks on project deliverables. Despite these  efforts, DHS officials stated that challenges associated with the IAA  structure and terms of the performance work statement with IBC on the  TRIO project limited DHS’s visibility into IBC’s overall cost, schedule, and  performance controls and ability to oversee IBC’s risk management  efforts. For example, they stated that the performance work statement did  not specify the level of reporting to be provided by IBC on cost, schedule,  and performance in sufficient detail to effectively monitor progress on  achieving key project objectives.

Further, the limitations to managing risks related to the best practices we  assessed as partially met were largely attributable to limitations in DHS  and TRIO project guidance and policies. For example, DHS’s Risk  Management Planning Handbook and related template for developing risk  management plans for projects does not address the need to define  thresholds to facilitate review of performance metrics to determine when  risks become unacceptable. Also, TRIO project policies did not address  the need to periodically revisit consideration of risk sources other than  IMS-related milestones, specify periods of performance for specific risk- handling activities, or define an interval for updating and certifying risk  statuses. In addition, DHS guidance and TRIO project policies did not  describe the need to consider and document risks specifically related to  the lack of sufficient, reliable cost and schedule information to properly  manage and oversee the project or for timely disposition of risks that its  IV&V contractor identified. Further, TRIO project risk management  policies and management tools used to implement them address best  practice elements such as determination of the costs and benefits of  implementing risk mitigation plans, developing contingency plans, and  developing specific risk-handling action items. However, these policies do  not require, and the risk register was not designed to specifically capture,  these elements in documented risk mitigation plans. By not adopting  important elements of risk management best practices into project  guidance, DHS and the TRIO components increase the risk that potential  problems would not be identified before they occur and that activities to  mitigate adverse impacts would not be effectively planned and initiated.

Key Factors and Challenges Impacting the TRIO Project and DHS’s Path Forward

Although DHS has taken various actions to manage the risks of using IBC  for the TRIO project, including some that were consistent with best  practices, the TRIO project has experienced challenges raising concerns  regarding the extent to which its objectives will be achieved. In connection  with these challenges, the TRIO components notified DHS during April  2016 through January 2017 that certain baseline cost and schedule  objectives had not been, or were projected to not be, achieved as  planned. According to these notifications and DHS officials we  interviewed, several key factors and challenges significantly impacted  DHS’s and IBC’s ability to achieve TRIO project objectives as intended. In  addition, IBC, FIT, and USSM officials identified similar issues impacting  the TRIO project. In connection with these challenges, DHS and IBC  began contingency planning efforts in January 2017 to identify and  assess viable options for improving program performance and addressing  key TRIO project priorities. Plans for DHS’s path forward on the TRIO  project, as of May 2017, involve significant changes, such as transitioning  away from using IBC and a 2-year delay in completing Coast Guard and  TSA’s migration to a modernized solution.

Key Factors and Challenges Impacting the TRIO Project

We grouped the key factors and challenges impacting the TRIO project  that DHS, IBC, FIT, and USSM officials and OMB staff identified into five  broad categories: (1) project resources, (2) project schedule, (3) complex  requirements, (4) project costs, and (5) project management and  communications. The key factors and challenges related to each category  are summarized below.

Project resources: Concerns about IBC’s experience and its capacity to  handle a modernization project involving agencies the size of Coast  Guard and TSA were identified as significant risks in July 2014, resulting  from discovery phase efforts completed prior to DHS and IBC’s entering  the implementation phase in August 2014. According to DHS officials,  status reports, and other documentation, key TRIO project challenges  related to resources included concerns that (1) IBC encountered federal  employee hiring challenges and was unable to ramp up and deploy the  resources necessary to meet required deliverables, and (2) IBC  experienced significant turnover of key stakeholders which adversely  impacted its ability to achieve TRIO project objectives.

In connection with DHS’s decision to use IBC for the TRIO project, DHS  officials told us that DHS relied heavily on OMB and Treasury’s  designation of IBC as a federal SSP and their related assessment of  IBC’s capacity and experience. DHS officials also told us that DHS relied  on FIT’s federal agency migration evaluation model during discovery  phase efforts that focused on assessing the functionality of the software  rather than assessing IBC’s (1) capacity, experience, and capability;   (2) ability to address more complex software configurations and interfaces  associated with large agencies; and (3) cost, schedule, and performance  metrics. DHS officials stated that issues related to IBC’s capacity and  experience represented the most significant challenge impacting the  TRIO project.

IBC officials acknowledged that IBC was unable to ramp up its resources  until after the project had begun and that the IBC project team  experienced significant turnover in key leadership and TRIO project  positions over the course of the project. IBC officials also acknowledged  that during its early efforts on the TRIO project, assigned IBC staff lacked  the experience and expertise necessary for managing large-scale  projects and, as a result, many of the risks initially identified were not  effectively addressed. FIT and USSM officials and OMB staff also  acknowledged that resource challenges significantly impacted the TRIO  project. A FIT official acknowledged that assessing software functionality,  rather than implementation, was emphasized during the discovery  process. Although DHS relied on OMB and Treasury’s designation of IBC  as a federal SSP, this FIT official also told us that because agencies’  specific needs can vary significantly, agencies are responsible for  conducting sufficient due diligence to assess a federal SSP’s ability to  meet their requirements.

Project schedule: DHS, IBC, FIT, and USSM officials acknowledged that  migrating the TRIO components to IBC within original time frames was a  significant challenge given the overall magnitude and complexity of the  TRIO project. According to DHS officials and TRIO project  documentation, DHS identified delays in completing various tasks and  milestones including providing design phase technical documentation and design processing proposed change requests; meeting proposed baseline schedules for implementing Coast Guard  and TSA on the modernized IBC solution; and achieving initial operating capability requirements and stabilizing the  production environment after DNDO’s migration to IBC because of  various issues related to reporting, invoice payment processing,  contract management processes, and resolving help desk tickets in a  timely manner.

DHS officials also stated that IBC did not consistently update the IMS to  ensure that it accurately reflected all required tasks, the completion  status, and the resources required to complete them. Concerns related to  meeting milestones and updating the IMS were discussed during periodic  status update meetings that included DHS, IBC, OMB, FIT, and USSM  officials. IBC and DHS officials acknowledged that processes for  communicating and resolving issues were not always efficient and  contributed to schedule delays. In addition, in November 2016, USSM  noted several concerns based on its review of a draft IMS supporting  TSA’s re-planning efforts to go-live in October 2017. USSM’s concerns  included an incomplete project scope and schedule and need for additional  discovery to determine cost and level of effort, an extremely aggressive schedule with very limited contingencies for  the lack of interim checkpoints or oversight on tasks exceeding 30  days,  the need for a resource-loaded IMS that incorporates an appropriate  level of detail, and  the need for an expedited program governance strategy and  escalation path that DHS and IBC leadership could use to make  program decisions within the time allotted on the schedule.

Complex requirements: DHS, IBC, FIT, and USSM officials  acknowledged the overall complexity of the TRIO project and that the lack  of a detailed understanding of the components’ requirements earlier in the  project impacted IBC’s and DHS’s ability to satisfy the requirements as  planned. For example, USSM and FIT officials told us that under the  shared services model, the approach for onboarding new customers  usually involves migrating to a proven configuration of a solution that is  already being used by the provider’s existing customers. However, rather  than taking this approach, DHS and IBC agreed to implement a more  recent version of Oracle Federal Financial software (version 12.2) with  integrated contract life cycle and project modules. Under this approach,  IBC’s plans included migrating other existing customers to this upgraded  environment. USSM officials told us that migrating TRIO components to a  new solution that required configuring new software and related  applications and developing related interfaces introduced additional  complexities that contributed to issues on the TRIO project. According to  a FIT official, the functionality of this more recent version of software is  very different than that of the version IBC’s existing customers used. This  official stated that IBC did not have the needed government personnel  with knowledge and experience associated with this new software, a  condition that likely contributed to the challenges experienced on the  TRIO project. IBC officials acknowledged that IBC’s lack of familiarity with  Oracle 12.2 increased the complexity of the TRIO project.

In addition, DHS and IBC perspectives on the need for changes differed  because of the lack of clarity regarding TRIO project requirements. DHS  officials told us that many change requests on the TRIO project reflected  the need for required functionality based on previously stated  requirements. They also told us that they did not consider DNDO-related  requirements to be overly complex when compared to those associated  with IBC’s similarly sized customers. However, DHS officials stated that  as of June 2017, IBC has not yet met DNDO’s needs to deliver a  functioning travel system interface and other requirements. According to  IBC officials, TRIO project change requests to address components’  requirements were extensive and included significant customizations to  meet unique requirements that were not aligned with the federal shared  service model. IBC officials noted additional challenges in addressing  TRIO project requirements related to DHS’s efforts to address certain  organizational change management and business process reengineering  responsibilities. According to IBC officials, in some instances, the TRIO  components provided conflicting requirements related to the same  process that would have been more consistent had DHS completed more  of its business process reengineering efforts prior to providing them to  IBC.

Project costs: According to the July 2014 discovery report, proposed  implementation costs for the TRIO project totaled $89.9 million. However,  according to DHS officials and TRIO project documentation, estimated  costs significantly increased because of schedule delays, unanticipated  complexities, and other challenges. In January 2017, DHS prepared a  summary of estimated TRIO project implementation costs associated with  its IAA with IBC. According to this summary, estimated IBC-related TRIO  project implementation costs through fiscal year 2017 increased by  approximately $42.8 million (54 percent) from the $79.2 million provided  in the original August 2014 IAA with IBC as a result of modifications  required, in part, to address challenges impacting the project. DHS  officials also expressed concerns regarding increases in estimated  operations and maintenance costs for the IBC solution. For example,  according to a December 2016 memorandum to DHS on action items  associated with failing to meet the baseline schedule date for initial  operational capability, DNDO stated that IBC’s updated projected costs of  operations and maintenance of its system were unaffordable. In  connection with these costs, DHS officials also stated that IBC  determined that separate, rather than shared, help desk resources were  required to support the TRIO project because it was significantly different  from the solution that IBC’s existing customers used. As a result, the  officials indicated that these costs were more than originally expected.  However, IBC officials told us that a portion of the increase in help desk- related costs was also due to DNDO employees not using the system  properly because they were not sufficiently trained on it before it was  implemented. In addition, challenges impacting the TRIO project have  contributed to significant changes in the path forward on the project; as a  result, the extent to which overall TRIO project modernization costs will  be impacted going forward has not yet been determined.

Project management and communication: According to DHS officials,  various program management-related challenges impacted the TRIO  project. For example, they expressed concerns regarding the  effectiveness of IBC’s project management efforts including cost,  schedule, and change management as well as IBC’s allocation of  resources and slow decision-making process. They also stated that DHS  provided significant time and resources to make up for fundamental  project management activities that were under IBC’s control and not  performed. In addition, DHS officials identified limitations associated with  (1) poorly defined service level agreements and program performance  metrics, (2) poor quality control plan, and (3) the lack of mechanisms for  measuring delivery and addressing concerns regarding IBC’s  performance. DHS officials told us that although various mechanisms can  be used to hold commercial vendors accountable—such as cure notices,  quality assurance surveillance plans, and incentives or disincentives to  monitor performance—few mechanisms are available to hold federal  agency service providers accountable for performance concerns.

DHS officials also acknowledged challenges in their project management  and communication efforts and identified lessons learned to help improve  future efforts, including the need to establish a performance-based contract to determine objective and  enforceable activity level metrics; be more prepared for organizational changes;  improve vendor, project, and schedule management efforts; better understand SSP resource plans and monitor SSP efforts to  help ensure that sufficient resources are secured timely; and  centralize program management for financial system modernization  functions, rather than continuing with the structure used on the TRIO  project—for example, the TRIO project’s program management  structure consisted of program management offices at the component  level performing cost, schedule, and technical monitoring activities  with DHS headquarters’ involvement focused on governance and  oversight, resulting in duplicate efforts across components.

IBC officials acknowledged challenges concerning IBC’s lack of sufficient  resources and turnover, as described above. However, they told us that  DHS’s approach to project management often resulted in duplicative  meetings and a lengthy decision-making process involving several  officials and multiple review and approval processes. According to USSM  officials, the TRIO project team focused an unbalanced portion of its  efforts on the delivery of technology at the expense of organizational  change management, communication management, and other project  management areas. For example, the failure to incorporate lessons  learned from DNDO’s deployment adversely affected subsequent TRIO  project implementation efforts, as change management activities did not  address previously encountered risks. An OMB staff member concurred  with the lessons learned that DHS identified, including those indicating  the need for stronger project management. While the project is ongoing,  the OMB staff member noted the importance of DHS having well-defined  requirements for the project and better coordination to achieve the  desired outcomes.

Significant TRIO Project Changes Resulting from Challenges and Steps Implemented for the Path Forward

In connection with TRIO project challenges, DHS officials told us that IBC  notified DHS in April 2016 that it would not be able to meet the planned  October 2016 implementation date for TSA. In response, DHS and IBC  established the TSA Replan Tiger Team to perform a detailed  assessment of potential courses of action. According to DHS officials,  DHS and IBC subsequently took various actions to help address these  and other challenges impacting the TRIO project, as summarized below.

May 2016: IBC requested additional funding for fiscal year 2016 for 14  additional IBC and contractor personnel to strengthen program  coordination and management support. According to DHS officials, DHS  provided this requested funding along with additional funding to establish  a business integration office to help strengthen cross organizational  communication. DHS determined that plans for migrating TSA and Coast  Guard to IBC during the first quarter of fiscal years 2017 and 2018,  respectively, were not viable. As a result, their planned migrations were  each extended an additional year.

June 2016: DHS and IBC developed a comprehensive remediation plan  to track progress on efforts to resolve numerous issues associated with  DNDO’s production environment that continued to hamper its stability  since going live in November 2015. According to DHS officials, these  issues related to invoice payment and interest accruals, contract life cycle  management, reporting, and other activities and have required numerous  work-arounds to execute business processes.

August to October 2016: DHS, Coast Guard, and IBC determined that a  similar replanning effort was needed for Coast Guard’s successful  migration to IBC. According to DHS officials, IBC indicated that it was  unable to simultaneously provide DNDO production and TSA  implementation support while also addressing the complexities related to  Coast Guard. DHS officials told us that another Tiger Team established to  address Coast Guard issues failed to complete the scope of its charter,  and as a result, Coast Guard was forced to assume a minimum of a 2- year delay (rather than the 1-year delay previously determined in May  2016) and that this significantly increased program costs. They further  stated that some of the team’s deliverables have not been initiated or  remain outstanding as of June 2017.

December 2016: IBC communicated to DHS that it cannot support the  discovery phase with DHS’s CUBE modernization project. In addition,  DHS approved the establishment of a Joint Program Management Office  to serve as the overarching program management for DHS financial  systems modernization projects. According to DHS officials, using a  department-wide approach will enable DHS to more effectively leverage  the resources and expertise across all modernization projects.

January 2017: IBC communicated to DHS that it cannot support Coast  Guard implementation in October 2018, and DHS and IBC established a  joint CPWG to assess viable options for improving program performance  and addressing stakeholder concerns and key TRIO project priorities.

February 2017: DHS and IBC issued a joint memorandum to provide an  update on contingency planning discussions. DHS and IBC shared  commitments and determinations included (1) stabilizing the DNDO  production environment and executing TSA implementation activities,   (2) delivering the best value for the government and ensuring mutual  success to the greatest extent possible, (3) preserving and protecting the  current investment, and (4) making TSA implementation the first priority.  In addition, DHS and IBC presented two options as representing the best  opportunities for success in improving program performance and  addressing stakeholder concerns: (1) continue with the status quo plan  for Coast Guard implementation in October 2019, with significant  improvements to program management and overall support capability and  capacity, or (2) platform replacement. Platform replacement was  presented as the preferred path toward meeting the needs of both DHS  and IBC. Under this option, DHS and IBC would proceed with TSA  implementation and work toward an orderly transition of TRIO  components to an alternate service provider, hosting location, or both.

March 2017: According to DHS officials, DHS, IBC, and USSM officials  met to review certain critical success criteria for TSA’s implementation.  Based on these discussions, it was determined that TSA would not go live  with IBC in fiscal year 2018 given the high-risk schedule and critical  criteria involved and the Coast Guard implementation would also be  delayed accordingly. Further, TSA release 3.0 would be delivered in  October 2017 or as soon as possible thereafter. In addition, the CPWG  would continue working to identify an alternative path forward, and DHS  and IBC would identify and evaluate critical transition activities and  timelines.

April 2017: The CPWG recommended moving away from IBC to a  commercial service provider leveraging the cloud as the best course of  action to complete TRIO project implementation and as the most fiscally  responsible approach from a long-term sustainment and cost perspective.  The CPWG’s recommendation was based on its analysis of six options  and proposed a transition timeline, including key activities, as shown in  figure 3.

May 2017: During its May 3, 2017 briefing of the Financial Systems  Modernization Executive Steering Committee, DHS indicated that two of  the options that the CPWG considered were no longer viable, including  the CPWG’s recommendation to transition to a commercial cloud service  provider because the software was not yet cloud-ready. DHS ranked the  remaining four options using 13 OMB risk factors as selection criteria and  determined that migrating the solution to a DHS data center represented  the best option going forward. In addition, DHS decided to move forward  with discovery efforts related to this option. According to its briefing  presentation and DHS officials, the notional timeline of planned key  events for the TRIO project included various items, as shown in figure 4.

DHS officials indicated that DHS expects to present the findings and  recommendations resulting from discovery efforts associated with this  new path forward to USSM and OMB for concurrence. As of August 2017,  results of this effort were under review by DHS leadership.

The TRIO project represents a key element of DHS’s efforts to address  long-standing deficiencies in its financial management systems and  further improve financial management. Following best practices to  manage risks effectively can help provide increased assurance that large,  complex projects—such as the TRIO project—will achieve planned  objectives. DNDO’s AA process substantially met the four characteristics  of a reliable, high-quality AOA process. However, Coast Guard’s and  TSA’s AAs substantially met one and partially met three of these four  characteristics. Further, DHS did not always follow best practices for  managing the risks of using IBC for the TRIO project. As a result, TRIO  components faced an increased risk that the solution they chose would  not represent the best alternative for meeting their mission needs and that  the risks impacting the TRIO project would not be effectively managed to  mitigate adverse impacts. In addition, significant challenges have  impacted the TRIO project, raising concerns about the extent to which  objectives will be achieved as planned. Plans for DHS’s path forward on  the TRIO project, as of May 2017, involve significant changes, such as  transitioning away from IBC and a 2-year delay in completing Coast  Guard’s and TSA’s migration to a modernized solution. Without greater  adherence to best practices for analyzing alternatives and managing  project risks, DHS continues to face increased risk that its financial  management system modernization project will not provide reasonable  assurance of achieving its mission objectives.

We are making the following two recommendations to DHS:  The DHS Under Secretary for Management should develop and  implement effective processes and improve guidance to reasonably  assure that future AAs fully follow AOA process best practices and reflect  the four characteristics of a reliable, high-quality AOA process.  (Recommendation 1)

The DHS Under Secretary for Management should improve the Risk  Management Planning Handbook and other relevant guidance for  managing risks associated with financial management system  modernization projects to fully incorporate risk management best  practices, including defining thresholds to facilitate review of performance metrics to  determine when risks become unacceptable;  identifying and analyzing risks to include periodically reconsidering  risk sources, documenting risks specifically related to the lack of  sufficient, reliable cost and schedule information needed to help  properly manage and oversee the project, and timely disposition of  IV&V contractor-identified risks; developing risk mitigation plans with specific risk-handling activities,  the costs and benefits of implementing them, and contingency plans  for selected critical risks; and  implementing risk mitigation plans to include establishing periods of  performance for risk-handling activities and defining time intervals for  updating and certifying the accuracy and completeness of information  on risks in DHS’s risk register. (Recommendation 2)

We provided a draft of this product to DHS and the Department of the  Interior for comment. In its comments, reprinted in appendix IV, DHS  concurred with our recommendations and provided details on its  implementation of the recommendations as discussed below. In addition,  DHS provided technical comments, which we incorporated as  appropriate. The Department of the Interior only provided technical  comments, which we incorporated as appropriate.

DHS stated that it remains committed to its financial system  modernization program. Specifically, regarding our first recommendation  to develop and implement effective processes and improve guidance to  reasonably assure that future AAs fully follow AOA process best practices  and reflect the four characteristics of a reliable, high-quality AOA process,  DHS stated that it agrees that effective processes and guidance are  necessary to assure best practices. DHS also stated that it is important to  note that the GAO-identified best practices were published more than 2  years after the TRIO components’ AAs were completed. While this is the  case, as discussed in our report, these best practices are based on long- standing, fundamental tenets of sound decision making and economic  analysis and were identified by compiling and reviewing commonly  mentioned AOA policies and guidance that are known to and have been  used by government and private sector entities.

DHS also stated that it has already implemented this recommendation  through its issuance of guidance and instructions in 2016 and that a copy  of this additional guidance and instructions was provided to GAO.  However, the documentation provided by DHS does not fully address our  recommendation. As part of our recommendation follow-up process, we  will coordinate with DHS to obtain additional information on its efforts to  address our recommendation.

With regard to our second recommendation to improve the Risk  Management Planning Handbook and other relevant guidance, DHS  stated that it concurred and agreed that the Risk Management Planning  Handbook required updating to fully incorporate risk management best  practices. In addition, DHS described actions it will take, and has taken,  to revise and publish an updated handbook.

We are sending copies of this report to the appropriate congressional  committees, the Acting Secretary of Homeland Security, the DHS Under  Secretary for Management, the Acting DHS Chief Financial Officer, the  Secretary of the Interior, and the Director of the Interior Business Center.  In addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-9869 or khana@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix V.

To determine the extent to which the Department of Homeland Security  (DHS) followed best practices in analyzing the alternatives used in  choosing the preferred alternative for modernizing TRIO components’  financial management systems, we reviewed information that the TRIO  components provided as part of their alternatives analysis (AA) process,  referred to as the AA body of work, which includes the AA and other  supporting documentation that is not specifically included in the AA. In  addition, we discussed the DHS AA process with the TRIO components  and DHS officials. We evaluated each TRIO component’s AA body of  work and assessed this information against the GAO-identified 22  analysis of alternatives (AOA) process best practices. We then scored  each AA against those best practices. In appendix II, these GAO- identified best practices are described in detail.

Our evaluation comprised the following steps: (1) two GAO analysts  separately examined the AA information received for each component,  providing a score for each of 18 best practices; (2) a third GAO analyst  resolved any differences between the two analysts’ initial scoring; and   (3) a GAO specialist on AOA best practices, independent of the audit  team, reviewed the team’s AA documentation, scores, and analyses for  consistency. The GAO specialist also assessed the four best practices  related to cost estimating.

We used the average scores for each best practice to determine an  overall score for four summary characteristics—well-documented,  comprehensive, unbiased, and credible—of a reliable, high-quality AOA  process at each TRIO component. Next, we shared our preliminary  analysis with the TRIO components and DHS, and requested their  technical comments and any additional information for our further  consideration. For those characteristics of the AA process that received a  score of partially met or below, we met with TRIO component and DHS  officials to discuss potential reasons that an AA did not always conform to  best practices. Finally, using the same methodology and scoring process  explained above, we performed a final assessment based on our  preliminary analysis and the comments and additional information  received. The best practices were not used to determine whether DHS  made the correct decision in selecting Department of the Interior’s Interior  Business Center (IBC) to implement the financial management systems  modernization solution or whether the TRIO project would have arrived at  a different conclusion had it more fully conformed to these best practices.

We also reviewed DHS guidance for conducting AOAs and AAs against  the GAO-identified 22 AOA process best practices using the same  methodology described above for reviewing the TRIO components’ AAs.  In the course of applying these best practices to a TRIO component’s AA  and to DHS guidance for the AA process, we assessed the  reasonableness of the information we collected. We determined that the  information from the DHS AA process was sufficiently reliable to use in  assessing the TRIO components’ AAs and DHS guidance against these  22 best practices.

To determine the key factors, metrics, and processes used by the TRIO  components in developing and evaluating DHS’s alternative solutions and  final choice for financial system modernization, we reviewed each  component’s AA, including a description of (1) the alternatives  considered, (2) the market research conducted, (3) the three alternatives  evaluated, (4) the selection criteria used and how the criteria were  weighted, (5) how each alternative scored against the selection criteria,  and (6) the alternative that scored the best according to the component’s  evaluation.

To determine the extent to which DHS managed the risks of using IBC  consistent with risk management best practices, we reviewed DHS’s and  TRIO components’ risk management guidance and other documentation  supporting their risk management efforts, including risk registers,  mitigation plans, status reports, and risk management meeting minutes.  We also met with officials to gain an understanding of the key processes  and documents used for managing and reporting on TRIO project risks.  We assessed the processes against best practices that the Software  Engineering Institute (SEI) identified. The practices we selected are  fundamental to effective risk management activities. These practices are  identified in SEI’s Capability Maturity Model® Integration (CMMI®) for  Acquisition, Version 1.3.

In particular, the key best practices for preparing for risk management are determine risk sources and categories, define risk parameters, and establish a risk management strategy.

The key best practices for identifying and analyzing risks are evaluate, categorize, and prioritize risks.

The key best practices for mitigating identified risks are develop risk mitigation plans and  implement risk mitigation plans.

We applied the criteria from the CMMI risk management process area to  determine the extent to which the expected practices were implemented,  or future activities were planned for, by the program office. The rating  system we used is as follows: (1) meets, or generally satisfies all  elements of the specific practice; (2) partially meets, or generally satisfies  a portion of specific practice elements; and (3) does not meet, or does not  satisfy specific practice elements.

In the context of the best practices methodology, we assessed the  reliability of TRIO project risk data contained in DHS’s risk register. We  interviewed officials on how the risk register was developed and  maintained, including key control activities used to provide reasonable  assurance of the accuracy of the information reported in the register. We  reviewed DHS’s July 2016 risk register and minutes from risk  management committee meetings (one meeting per quarter, randomly  selected). Of 120 TRIO project risks on the July 2016 risk register, we  found 13 risks with missing data. Of 47 active risks identified, 28 risk  records had not been modified in the previous 3 months and the register  did not indicate when their accuracy was last confirmed and 35 risks were  beyond their indicated impact dates but had not been marked as issues.  We concluded that the pervasiveness of these data reliability problems  decreased the usefulness of the risk register in connection with managing  TRIO project risks.

To determine the key factors or challenges that have impacted the TRIO  project and DHS’s plans for completing remaining key priorities, we met  with DHS, IBC, Office of Financial Innovation and Transformation, and  Unified Shared Services Management office officials and Office of  Management and Budget staff to obtain their perspectives. In addition, we  reviewed documentation provided by these officials, including TRIO  project status reports and memorandums, leadership briefings, and other  presentations.

We conducted this performance audit from March 2016 to September  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix II: Best Practices for the Analysis of Alternatives Process

Many guides describe an approach to an analysis of alternatives (AOA);  however, there is no single set of practices for the AOA process that has  been broadly recognized by both the government and private sector  entities. GAO has previously identified 22 best practices for an AOA  process by (1) compiling and reviewing commonly mentioned AOA  policies and guidance used by different government and private sector  entities and (2) incorporating experts’ comments on a draft set of  practices to develop a final set of practices. These practices are based  on longstanding, fundamental tenets of sound decision making and  economic analysis.

In addition, these practices can be applied to a wide range of activities in  which an alternative must be selected from a set of possible options, as  well as to a broad range of capability areas, projects, and programs.  These practices can provide a framework to help ensure that entities  consistently and reliably select the project alternative that best meets  mission needs. The guidance below is an overview of the key principles  that lead to a successful AOA process and not as a “how to” guide with  detailed instructions for each best practice identified.

The 22 best practices that GAO identified are grouped into the following  five phases:  1.  Initialize the AOA process: Includes best practices that are applied  before starting the process of identifying, analyzing, and selecting  alternatives. This includes determining the mission need and  functional requirements, developing the study time frame, creating a  study plan, and determining who conducts the analysis.  2.  Identify alternatives: Includes best practices that help ensure that  the alternatives to be analyzed are sufficient, diverse, and viable.  3.  Analyze alternatives: Includes best practices that compare the  alternatives to be analyzed. The best practices in this category help  ensure that the team conducting the analysis uses a standard,  quantitative process to assess the alternatives.  4.  Document and review the AOA process: Includes best practices  that would be applied throughout the AOA process, such as  documenting all steps taken to initialize, identify, and analyze  alternatives and to select a preferred alternative in a single document.  5.  Select a preferred alternative: Includes a best practice that is  applied by the decision maker to compare alternatives and to select a  preferred alternative.

The five phases address different themes of analysis necessary to  complete the AOA process, and comprise the beginning of the AOA  process (defining the mission needs and functional requirements) through  the final step of the AOA process (selecting a preferred alternative).

We also identified four characteristics that relate to a reliable, high-quality  AOA process—that the AOA process is well-documented,  comprehensive, unbiased, and credible. Table 4 shows the four  characteristics and their relevant AOA best practices.

Conforming to the 22 best practices helps ensure that the preferred  alternative selected is the one that best meets the agency’s mission  needs. Not conforming to the best practices may lead to an unreliable  AOA process, and the agency will not have assurance that the preferred  alternative best meets mission needs.

Appendix III: GAO Assessment of TRIO Components’ Alternatives Analyses

The Department of Homeland Security’s TRIO components—the U.S.  Coast Guard (Coast Guard), Transportation Security Administration  (TSA), and Domestic Nuclear Detection Office (DNDO)—conducted  alternatives analyses (AA) during 2012 and 2013 to determine the best  alternative for transitioning to a modernized financial management system  solution. We evaluated the TRIO components’ AA processes against  analysis of alternatives (AOA) best practices GAO identified as necessary  characteristics of a reliable, high-quality AOA process (described in app.  II).

GAO’s assessment of the extent to which Coast Guard’s, TSA’s, and  DNDO’s AAs met each of the 22 best practices is detailed in tables 5, 6,  and 7.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the contact named above, James Kernen (Assistant  Director), William Brown, Courtney Cox, Eric Essig, Valerie Freeman,  Matthew Gardner, Jason Lee, Jennifer Leotta, and Madhav Panwar made  key contributions to this report.