DHS Acquisitions: Additional Practices Could Help Components Better Develop Operational Requirements

GAO has previously found that DHS's components had acquisition programs that did not meet requirements and that those requirements were, in some cases, poorly defined. Poorly defined requirements increase the risk that acquisitions will not meet the needs of users in the field—for example, border patrol agents or emergency responders.

GAO was asked to examine DHS components' practices for developing requirements. This report addresses the policies, organizations, and workforce that selected DHS components use to develop requirements for their acquisition programs.

GAO selected seven DHS components with significant acquisition programs and a non-generalizable sample of programs—based on cost, component, and acquisition phase—as case studies. GAO analyzed policies and program documentation; and interviewed DHS and component officials, as well as end users of DHS programs. GAO compared components' practices to industry best practices and federal internal control standards.

GAO has identified several best practices to ensure that operational requirements for acquisitions are well-defined and found some Department of Homeland Security's (DHS) components met them while others did not. These practices include a formal policy for developing requirements, an independent requirements organization, and an understanding of workforce needs and training. The table below shows GAO's assessment of seven of DHS's components against these practices.

Establishing a formal policy to guide the process is critical to developing well-defined requirements. However, only the Coast Guard has an approved policy for requirements development among the seven components reviewed. Without well-defined requirements, components are at risk of acquiring capabilities that will not meet mission needs. DHS officials told GAO that components have generally prioritized obtaining funding and starting programs over developing requirements.

Three components have a requirements development organization, separating requirements from acquisition in addressing capability gaps. Officials from components without such organizations told GAO that they have fewer major acquisitions and rely on DHS to assist in requirements development. DHS policy and best practices, however, maintain the importance of this separation regardless of the number of major acquisitions to guard against possible bias by acquisition officials toward a specific materiel solution.

Two components have assessed requirements development workforce needs, but both need to be updated; and one component has provided requirements development training and certification. Other component officials told GAO that they lack the resources necessary to take these steps. Best practices indicate that without an appropriately sized and trained workforce, components remain at risk of acquiring capabilities that fail to meet end user needs.

GAO is making 25 recommendations, including to individual components to establish policies and independent organizations for requirements development, assess workforce needs, and establish training and certifications. DHS concurred with all the recommendations.

The Department of Homeland Security (DHS) invests billions of dollars  each year in major acquisition programs to assist in executing its many  critical missions. In November 2014, in response to a GAO  recommendation, the department reestablished the Joint Requirements  Council (JRC) that the department had dissolved in 2006, to review  requirements submitted by DHS’s component agencies (e.g., the  Transportation Security Administration). The purpose of the council is to  validate and prioritize operational requirements—those capabilities that  are necessary to conduct DHS’s mission—for all major acquisitions and  to ensure that objective, analytical rigor supports these requirements. We  subsequently found that several programs did not meet their key  performance parameters—the most important requirements a capability  must meet—after initial approval by DHS and deployed capabilities prior  to meeting all of their key performance parameters, leaving their true  capabilities in doubt. One reason programs did not achieve their  specified key performance parameters was that programs poorly defined  them. Poorly defined key performance parameters can increase the risk  that end users—such as border patrol agents or first responders in a  disaster—receive capabilities that do not meet their missions.

You asked us to examine DHS’s components’ requirements development  practices. This report discusses (1) how often selected programs  changed requirements; and assesses the extent to which the selected  components have (2) developed policies for requirements development,  (3) established independent requirements organizations, and (4) taken  steps to assess and train a requirements workforce. Our focus for this  report was on the DHS components, as they are responsible for  developing the requirements to meet end user needs.

To conduct our work, we reviewed the DHS Master Acquisition Oversight  List as of April 2017 and selected seven DHS components with Level 1  and Level 2 major acquisition programs and covered a broad range of  missions. The seven components we selected are as follows:

National Protection and Programs Directorate

U.S. Citizenship and Immigration Services  From these seven components, we selected 14 major acquisition  programs with DHS-approved key performance parameters to serve as  case studies for our review. We selected this non-generalizable sample of  programs based on different factors, such as the acquisition phase and  component to have a mix of the types of programs that we reviewed. We  also reviewed two programs that do not have DHS-approved key  performance parameters to understand how requirements are determined  before DHS validation.

We focused on the presence of policies for requirements development,  independent requirements organizations, and requirements-specific  workforce and training in components as our past work on major  acquisitions has shown that these are the fundamental building blocks  required to develop well-informed operational requirements. This  selection was also informed by our standards for internal controls. To  inform each of our objectives, we interviewed officials at various levels  throughout DHS to understand both their relationship to and role in  components’ requirements development, including: (1) department-level,  (2) component-level, (3) program-level, and (4) program end users.  These discussions informed our understanding of the extent to which the  components have implemented requirements development policies,  organizations, and assessments and training for their components. In  addition, we furthered our understanding through reviewing component-  and program-level documentation such as guidance manuals, mission  needs statements, and operational requirements documents.

To determine the extent to which the selected programs changed  requirements, we examined key performance parameters after DHS  approval when key performance parameters should be stable. To  determine the extent to which DHS components developed requirements  development policies, as well as determine the extent to which those  components established independent organizations, we reviewed  component documentation pertaining to requirements development, such  as instruction manuals, mission statements, and capability analyses. To  determine assessment, training, and certification standards for DHS’s  requirements development workforce, we spoke with JRC and U.S. Coast  Guard officials regarding the training and certification programs they have  in place and reviewed available documentation.

We assessed the components’ requirements development practices  against GAO’s standards for internal control and additional supporting  criteria as stated in the findings. The standards identify key principles to  help entities achieve their objectives, such as delivering capabilities to  end users.

See appendix I for a detailed description of our objectives, scope, and  methodology. Appendix I also includes a detailed description of the major  acquisition programs we reviewed.

We conducted this performance audit from May 2017 to August 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

To help manage its multi-billion dollar acquisition investments across its  components, DHS has established policies and organizations for  requirements validation, acquisition management, and budgeting. The  department uses these to monitor and guide delivery of the acquisition  programs the components require to close critical capability needs,  enabling DHS to execute its missions and achieve its goals.

DHS has 14 components, which, as a part of their operational missions,  are responsible for assessing capability needs, developing the  requirements to fill these needs, and creating acquisition programs to  meet these requirements. The number and cost of acquisition programs  vary by component. DHS generally defines a capability as the means to  accomplish a mission or objective that may be achieved through materiel  and non-materiel solutions. Once the component has a JRC-validated  capability gap, and identifies and documents the need for a materiel  solution, it develops the operational requirements. Requirements can be  unique to an individual component, or they can be joint requirements that  apply to more than one component. Within the components, program  management offices are responsible for planning and executing individual  programs within cost, schedule, and performance parameters, and  preparing required acquisition documents.

Tracing Mission Needs to Program Requirements

The DHS requirements process generally starts with the identification of  mission needs and broad capability gaps from which components develop  a program’s operational requirements, key performance parameters, and  more definitive technical requirements. Figure 1 depicts this traceability  from mission needs to technical requirements.

Operational requirements are what the end users need to fill capability  gaps and conduct the mission. Operational requirements, in part, define  the purpose for the acquisition program and set boundaries for user  needs. Subject matter experts, such as system engineers, support  development of operational requirements to ensure that they are clearly  developed. Well-defined operational requirements trace to one or more of  the identified capability gaps.

After components define operational requirements, they identify some as  key performance parameters that denote the most important and non- negotiable requirements that the program has to meet to fulfill its  fundamental purpose. According to DHS policy, failure to meet any key  performance parameter results in a re-evaluation of a program that may  lead to requirements changes or program cancellation. See figure 2  below for an overview of the requirements process.

According to DHS policy on managing acquisition programs, components  further decompose operational requirements into technical requirements,  such as design or material specifications. For example, an operational requirement may be the ability to detect explosives at the airport. The  technical requirement may then be the ability to detect metal or explosive  material within certain parameters.

DHS’s Joint Requirements Council and Other Offices

Through the JRC, DHS provides oversight of operational requirements for  the acquisition programs developed by its components. The JRC consists  of a chair and 14 members, called principals, who are senior executives  or officers that represent key DHS headquarters offices and seven of the  department’s operational components. JRC principals represent the views  of both their components and DHS, and validate and prioritize capability  needs and operational requirements. Among other responsibilities, the  JRC is to provide requirements-related advice and validate key  acquisition documentation to prioritize requirements and inform DHS  investment decisions for all Level 1 and Level 2 major acquisitions, as  well as for programs that are joint interest, regardless of level.

Separate from the JRC, DHS’s Office of Program Accountability and Risk  Management, which reports directly to the Under Secretary for  Management, oversees major acquisitions and guides acquisition policy.  DHS also has a separate office for budget management and a planning,  programming, budgeting, and execution process to allocate resources,  such as funding, to acquisition programs. In addition, the Science and  Technology Directorate conducts systems engineering reviews and  technology assessments of the technical solutions for major acquisition  programs. The Directorate also provides department-level guidance on  requirements development in its Systems Engineering Life Cycle  Guidebook.

Multiple DHS directives and manuals establish the framework for the  department’s Joint Requirements Integration and Management System  (JRIMS)—a process by which the department reviews and validates  capability gaps—and requirements to mitigate those gaps. DHS further  clarified its directives in April 2016 through DHS Instruction Manual 107- 01-001-01, Department of Homeland Security Manual for the Operation of  the Joint Requirements Integration and Management System. The JRC  also instituted a series of training courses that provide an overview of  JRIMS and its core concepts. JRC validation of requirements confirms the  requirements are traceable, feasible, and cost-informed.

In addition to validation by the JRC, DHS’s Under Secretary for  Management approves the operational requirements that the components  developed and reviews them at a series of predetermined acquisition  decision events. Figure 3 depicts the acquisition life cycle established in  DHS acquisition policy. DHS initially established its acquisition process in  policy in November 2008.

An important aspect of acquisition decision event 2A, which begins the  “Obtain” phase and system development, is the decision authority’s  review and approval of key acquisition documents that establish the cost,  schedule, and requirements baselines for a program. The operational  requirements document and acquisition program baseline are key  acquisition documents requiring this approval and include a program’s  key performance parameters. DHS also revisits these baselines at subsequent acquisition decision events in order to determine whether the  requirements remain achievable.

Prior GAO Work on DHS Requirements Development

We have previously reported on the importance of stable requirements  and the costs of changing them.

In March 2016, we found that changes to key performance  parameters have been common and are likely to continue for several  reasons. While some changes may have a valid reason, such as a  response to emerging threats, we found that one of the most common  reasons programs changed key performance parameters was that the  originally approved key performance parameters had been poorly  defined. Key performance parameter changes on several programs  were associated with schedule slips and cost growth. DHS leadership  acknowledged that the department has had difficulty defining key  performance parameters, and said that the Office of Program  Accountability and Risk Management has improved its ability to help  programs define key performance parameters. We recommended,  among other things, that DHS should require the components to  submit program funding certification memos to aid affordability  discussions. DHS concurred and implemented our recommendation.

In October 2016, we found that the JRC’s structure and management  approach—informed by assessments of requirements processes,  guidance, and lessons learned from DHS components—are generally  consistent with key practices for mergers and organizational  transformations. However, we recommended that DHS’s Office of the  Chief Information Officer have a more formal and consistent role than  that of a non-voting advisor to the JRC, since 24 of 36 major  acquisitions were information technology programs, and we previously  identified poor requirements definition as a factor in failed information  technology programs. DHS concurred with our recommendation and  implemented it in November 2016.

In April 2017, we found that DHS’s acquisition policy was not  consistent with acquisition best practices in terms of when to enter the  “Obtain” phase depicted in figure 3. Specifically, best practices call for  ensuring that a program’s needs are matched with available  resources—such as technical and engineering knowledge, time, and  funding—prior to starting product development. We recommended,  among other things, that DHS require that major acquisition programs’  technical requirements be well-defined and conduct key technical  reviews prior to approving programs to initiate product development,  in accordance with acquisition best practices. DHS concurred with our  recommendation, but has not yet implemented it.

Over Half of the Selected Programs Changed Requirements

Our analysis found that 9 of 14 programs from the seven components that  we reviewed changed key performance parameters for various reasons  after program approval and entry into the “Obtain” phase. DHS had  initially approved most programs’ key performance parameters before  DHS reestablished the JRC in November 2014. Whether these  programs changed DHS-approved key performance parameters is shown  in table 1.

We found that the causes of these changes varied, but included  requirements did not accurately describe end user needs, were not  achievable given available technologies, or that programs pursued  greater capability than originally intended. Further details on the nine  programs that changed their requirements are in table 2.

To mitigate these types of requirements changes, we identified several  principles that are critical as the first steps to successful implementation  of programs and the remainder of this report presents examples of when  the principles have been implemented and when they have not.

One of the Seven Selected Components Has a Policy for Requirements Development

Among the seven DHS components we reviewed, each of which is  responsible for managing major acquisition programs, only the U.S. Coast  Guard has a formalized policy in place for developing requirements. Of  the other six components, some are developing such policies and others  rely on JRIMS guidance. In the absence of component-level policies,  some sub-organizations and programs within the components have  developed their own requirements policies.

U.S. Coast Guard Has an Approved Requirements Policy, While the Other Six DHS Components in Our Review Do Not

The U.S. Coast Guard, which has a long history of managing large  acquisition programs, established a requirements policy to assess needs  and fill capability gaps in 2009 and updated it in 2017. The most recent  version of this requirements policy, the Coast Guard Operational  Requirements Generation Manual, aligns its policies with DHS’s  acquisition and requirements policies. The manual contains guidance on  requirements development and the analytic efforts used to develop the  requirements documents. The manual also describes the personnel that  are to be included in requirements development and provides guidance  on drafting the necessary documentation, and includes templates to do  so. As part of the process, requirements development personnel work  with end users to generate requirements, which the U.S. Coast Guard  reviews and approves before going to the DHS JRC for validation.

The status of developing a requirements policy across the other six  components is as follows:  Immigration and Customs Enforcement, National Protection and  Programs Directorate, Transportation Security Administration, and  U.S. Citizenship and Immigration Services officials told us that they  are currently developing or considering developing policies. These  components have not yet set time frames for approving these policies.

A Federal Emergency Management Agency official stated that they  are planning to develop a formal requirements policy but are waiting  for the JRC to clarify JRIMS policy on information technology program  reviews and decision authorities before doing so. However, such  clarification does not prevent them from drafting an interim policy.

Customs and Border Protection has a draft requirements development  policy but did not provide a definitive timeline for completion.

Although Customs and Border Protection does not yet have a finalized  policy, the following sub-component operational organizations have  documented their requirements policies. For example:

Border Patrol finalized a requirements management process policy on  June 12, 2018 that defined roles and responsibilities throughout the  process. The requirements policy was preceded by an October, 18,  2016 policy on the process for identifying capability gaps. GAO  previously reported on the Border Patrol’s policy in February 2017 and  recommended clarifying the roles and responsibilities of the parties  involved.

The Office of Technology Innovation and Acquisition developed a  draft requirements handbook in 2011 that provided guidance for the  execution of activities within each stage of development, including  defining operational requirements.

The Passenger Systems Program Office also documented its  requirements management policy in 2010 that outlined requirements  development at a high level.

While these sub-components have taken the key step of documenting  their policies, without a single component policy, Customs and Border  Protection may not be efficiently and effectively meeting its mission.

Without Requirements Policies, Components Risk Failing to Meet Mission Needs

In the absence of component-level policies, we found that components  are less likely to establish the base of knowledge needed for  requirements development. Further, we found this contributes to an  inability to properly mitigate capability gaps and meet mission and end  user needs.

Outcomes for a number of our case study programs illustrate the potential  benefits of having component-level requirements development policies in  place.

National Flood Insurance Program PIVOT (not an acronym):  Federal Emergency Management Agency officials told us that the  current attempt is the third effort to modernize its information  technology systems after two failed attempts. Program officials said  that one of the previous program attempts failed to meet capability  gaps and end user needs because of a lack of clear policies for  developing requirements. The officials said that failure is less likely as  the program currently uses lessons learned from the previous  attempts. In addition, the JRC is encouraging the component to adopt  rigorous standards for developing requirements. However, without a  policy to capture these lessons learned, programs within the Federal  Emergency Management Agency are at risk for losing the knowledge.

National Security Cutter: The U.S. Coast Guard began requirements  development for the National Security Cutter in the late 1990s, before  it had established a documented requirements development policy in  2009. We found in 2010 that the lack of overarching, formalized policy  resulted in requirements that were vague, not testable, not prioritized,  and not supportable or defendable. In 2014, the National Security  Cutter completed initial operational testing but did not fully  demonstrate 7 of its 19 key performance parameters, including those  related to unmanned aircraft and cutter-boat deployment in rough  seas. To meet the cutter-boat deployment parameter, U.S. Coast  Guard officials said that the program had to overcome differing  interpretations of the parameter between the U.S. Coast Guard and its  independent test officials. One key practice for requirements  development is assigning roles and responsibilities, such as when and  in what capacity test officials should be involved in requirements  development, to avoid just such an outcome and the resulting effect  on cost and schedule. U.S. Coast Guard officials stated that end  users of the National Security Cutter have since demonstrated its key  performance parameters during U.S. Coast Guard operations.

Electronic Baggage Screening Program: Without a finalized  requirements development policy, the Transportation Security  Administration’s program developed requirements that focused on  how the system functioned as opposed to the capability that it would  provide. Program officials said that neither the Transportation Security  Administration nor the program office had a documented policy for  requirements development when the program began in 2004. In this  environment, the program adopted an informal approach to develop  operational requirements by collecting end user input. However,  officials noted that end users listed technical requirements rather than  broader operational requirements. Officials told us that the program  “backed into” operational requirements using these technical  requirements, resulting in a system more focused on function and less  on capability. Without a focus on the capability, the program risked not  meeting the capability gap and end user need.

We also found an example of where a component’s policy was beneficial  to a program developing requirements:

Offshore Patrol Cutter: The U.S. Coast Guard has matured its  requirements development policies since the National Security Cutter  program as described above. For the Offshore Patrol Cutter, the U.S.  Coast Guard has six DHS-approved key performance parameters,  such as operating range and duration. The U.S. Coast Guard plans to  use engineering reviews and developmental and operational tests  throughout the acquisition to refine and demonstrate requirements.  For example, to refine the requirements and ensure end user input,  the U.S. Coast Guard had an early operational assessment of the  cutter’s key performance parameters and associated lower level  technical requirements. According to officials, specific policies guided  the assessment to, in part, ensure that the program refined key  performance parameters before progressing through the remaining  acquisition phases.

DHS’s JRIMS directive and manual are not designed to provide the level  of specificity for component-level requirements development. JRIMS  encourages components to elicit end user needs and translate them into  requirements. It also authorizes the components to develop their own  policies consistent with the intent of and required capability  documentation in the JRIMS manual and DHS Instruction Manual.

Federal standards for internal control and key practices for requirements  development, such as those in Carnegie Mellon University’s Capability  Maturity Model Integration for Development, state that organizations  should establish responsibility and authority by having documentation that  communicates the “who, what, when, where, and why” of achieving their  missions. A policy also provides a means to retain organizational  knowledge and mitigate the risk of having that knowledge limited to a few  personnel. Such a policy should include a documented process for  developing and managing requirements which can help reduce the risk of  developing a system that does not meet end user needs, cannot be  adequately tested, and does not perform or function as intended. We  depict four key practices for requirements management in figure 4.

DHS officials indicated to us that one factor which contributes to a  component’s lack of finalized requirements policies is the prioritization of  starting an acquisition over developing requirements. This situation  reflects what we have found over many years at the Department of  Defense. Undesirable program outcomes share a common origin;  decisions are made to move forward with programs before the knowledge  needed to reduce risk and support those decisions is sufficient. There are  strong incentives within the acquisition culture to overpromise  performance while understating cost and schedule. A key enabler of  successful programs is firm, feasible requirements that are clearly  defined, affordable, and clearly informed. Once programs begin,  requirements should not change without assessing their potential  disruption to the program.

Of note, DHS established its formal acquisition process in 2008, and did  not have a similar emphasis on requirements development until 2016,  when the JRIMS process was set forth. DHS requirements officials said  that the renewed emphasis on requirements development at DHS  requires a significant culture change among the components, pushing the  components away from previous practices that undervalued well-defined  requirements. They said that the components generally completed the  necessary requirements documents to comply with department guidance  and formats rather than to ensure that the components identified the  needed capabilities and generated the correct requirements.

DHS officials said that in the past, some program offices would contract  out the capability assessment and requirements development, have them  approved by DHS, but not use the resulting documentation to guide the  acquisition. Two component requirements officials told us that their  components’ previous acquisition and requirements processes focused  on obtaining funding before developing requirements.

Most components indicated that they are planning on drafting a  requirements development policy. However, without specific timeframes  for completing their efforts, there is a risk that management attention will  not be sustained and planned actions will not be implemented. Without  component-level requirements policies that are aligned with the JRC and  JRIMS standards, DHS is missing an opportunity to help ensure that  components’ programs are set-up from the beginning to meet end user  needs and close capability gaps.

Utilization of an Independent Requirements Organization Inconsistent Across Selected Components

Three of the seven DHS components in our review have established  requirements development organizations, such as offices or directorates  independent of the acquisition function. Among the reasons cited by  these components’ officials was recognition of the importance of the  operational requirements development function for addressing capability  gaps. Those that do not have separate requirements organizations cited,  among other things, the smaller size of their components. However,  according to key principles, independent lines of authority should develop  operational requirements and manage acquisitions separately, regardless  of size.

Three Components Have Independent Requirements Development Organizations but Remaining Four Components Do Not

Three of the seven DHS components in our review have established  independent requirements development organizations that are separate  from acquisition offices, as shown in table 3.

The three components that established requirements organizations did so  at various times.

In 2009, the U.S. Coast Guard formally placed responsibility for its  requirements development policy in its capabilities directorate under  the Assistant Commandant for Capability, who reports to the Deputy  Commandant for Operations, one level below the Vice Commandant  of the Coast Guard. The capabilities directorate, which is separate  from the acquisitions directorate, provides oversight and management  of its requirements development process. This directorate provides  expertise as well as an independent quality review of the  requirements documents generated for approval.

Customs and Border Protection officials noted that they created a  requirements organization in 2010 in the Office of Technology  Innovation and Acquisition. In 2016, through an organizational  realignment, Customs and Border Protection separated the  requirements organization and established the Planning, Analysis,  and Requirements Evaluation Directorate. The officials stated that due  to concerns about independence from the acquisitions office,  Customs and Border Protection placed this Directorate in the  Operations Support office.

The Transportation Security Administration established the Office of  Requirements and Capabilities Analysis in 2017, in part, because  officials told us they recognized that prior requirements development  efforts were not being done the right way. This new office, which is  separate from the Office of Acquisition Management, reports directly  to the Executive Assistant Administrator of Operations Support.

The remaining four components that we reviewed did not have separate,  independent requirements development organizations. Officials from  Immigration and Customs Enforcement, National Protection and  Programs Directorate, and U.S. Citizenship and Immigration Services  noted that they are planning on developing such organizations but have  not provided specific time frames for doing so. An official from the  National Protection and Programs Directorate told us that although an  independent office has not been established, he serves as an  independent requirements official, separate from acquisitions. Among the  reasons cited by components’ officials for not having a requirements  organization at the time of our review was a primary focus on the  acquisition function, associated funding issues, and reliance on the JRC  to help refine their requirements.

Officials also noted the smaller size of their respective components and  the fewer number of major acquisitions as reasons for not having an  independent requirements organization. Regardless of size, components  need to ensure that requirements development is independent of  acquisitions in order to guard against possible bias by acquisition officials  toward a specific materiel solution.

A Separate, Independent Requirements Organization Is Critical to Addressing Capability Gaps

According to federal standards for internal controls, independent lines of  authority should develop requirements and manage acquisitions  separately. These standards state that management should design  control activities to achieve objectives and respond to risks. In addition,  authorities should segregate incompatible duties to prevent risks such as  management override. For example, if requirements developers were  part of the acquisition function, management could tailor operational  requirements to satisfy preferred acquisition outcomes, increasing the risk  that capability gaps will not be addressed. The absence of an  independent requirements organization hampers the components’ ability  to remove biases and identify crosscutting opportunities and investments.  See figure 5 for a notional example of organizations with separate  functions.

In accordance with these standards, DHS, at the department level, has  separate requirements, acquisitions, and resourcing organizations—each  with its own governance structure. In addition, U.S. Coast Guard policy  notes that requirements development, when separated from acquisition  organizations, results in an operational requirements document that  conveys the user’s true needs. The policy goes on to state that the  requirements development organization informs the acquisition process  by ensuring requirements are traceable to strategic objectives and  recommended courses of action to address capability gaps are cost  informed and assessed for feasibility. According to GAO’s best practices,  while these organizations should be separate, there should be consistent  collaboration and feedback throughout the process.

We found examples of programs in our review that would have benefited  from an independent organization at the component level.

Immigration and Customs Enforcement, TECS Modernization  (not an acronym): The acquisition program office set the  requirements without an understanding of the capability gaps it was  trying to close. Without a requirements development office to guide  development, program officials stated that they generated  approximately 25,000 requirements, which consisted of both technical  and operational requirements to address the capability gaps that they  were unable to prioritize. The program revised its operational  requirements a few times and went through a replanning initiative that  included a full review of all the requirements to ensure completeness  and accuracy to determine the program’s operational requirements.  Immigration and Customs Enforcement officials stated that they  recognize the importance of requirements development and are in the  process of establishing a requirements organization.

U.S. Citizenship and Immigration Services, Transformation: The  program began requirements development in 2006 in the absence of  an independent organization for requirements development and has  subsequently generated three operational requirements documents  over a six-year period. Our review showed that the key performance  parameters from the oldest document to the most recent one changed  significantly. For example, the operational requirements document  from 2009 had a key performance parameter called “account  hardening,” which involved gathering identity and biometric evidence.  The document from 2015 did not contain this parameter. In April 2015,  nine years after starting requirements development, DHS leadership  finalized a revised set of operational requirements after the program  struggled again to meet its previous requirements.

We also found an example of where a component’s requirements  organization was beneficial to a program developing requirements:

Customs and Border Protection, Cross Border Tunnel Threat:  This program is analyzing alternative capabilities as it moves toward  the JRC’s validation of its requirements. To aid in developing the  operational requirements, Border Patrol, a sub-component of  Customs and Border Protection, has its own Operational  Requirements Management Division. In addition, Customs and Border  Protection officials noted that its Planning, Analysis, and  Requirements Evaluation Directorate is coordinating, guiding, and  providing oversight to ensure the operational requirements address  the capability gaps. In doing so, these requirements organizations  facilitate input from subject matter experts on tunnel threats and from  end user agents who have to mitigate these threats.

Majority of Selected Components Have Not Assessed Workforce Needs or Established Training for Requirements Development

We found that two components have assessed requirements workforce  needs, and one has provided requirements specific training. Components  gave different reasons why they have not yet taken one or more of these  steps, including a lack of resources.

Two Components Have Assessed Requirements Workforce Needs, and One Has Provided Requirements Specific Training

Two of the seven components we reviewed, Federal Emergency  Management Agency and Customs and Border Protection, performed  assessments of workforce needs for requirements development. The  Federal Emergency Management Agency assessed its requirements  workforce needs in 2016 and found, among other things, that it does not  have the capacity to identify and analyze capability gaps or accurately  trace operational requirements to capability needs. As a result of the  assessment, the agency requested additional requirements personnel in  the fiscal year 2019–2023 budget cycle.

Customs and Border Protection requirements officials stated that they last  conducted an assessment in 2013. They stated that the assessment  identified the appropriate number and types of personnel necessary to  conduct requirements development through an analysis of historical  requirements workloads. In addition, Customs and Border Protection  officials said that they are currently performing an assessment as part of  their Acquisition Management Performance Improvement initiative. The  initiative assesses training needs and availability and is due at the end of  fiscal year 2018.

Requirements officials from Immigration and Customs Enforcement,  National Protection and Programs Directorate, Transportation Security  Administration, and U.S. Citizenship and Immigration Services told us that  they have not assessed their requirements workforce needs and have no  plans to do so. U.S. Coast Guard requirements officials told us that  although they have not conducted a formal assessment of their workforce  needs, they informally assess those needs and would like to increase the  personnel who have requirements training across the organization.

Although the U.S. Coast Guard has not conducted an assessment of its  workforce needs, it is the only component that has an established  requirements training process. Requirements officials told us that the U.S.  Coast Guard initially established training and training-related certification  standards in 2007 to emulate similar changes taking place at the  Department of Defense and address previous U.S. Coast Guard  acquisition challenges. Specifically, the U.S. Coast Guard requirements  development organization assigns end users for a two to three year  rotation and provides them training and certification on requirements  development. The requirements development certification program is two  levels and requires both classroom-based training and on-the-job  experience. The U.S. Coast Guard assigns those who complete a higher  level of certification to develop requirements for more complex and costly  programs. This helps to ensure that requirements personnel can give  timely, relevant end user input but also differentiate between operational  and technical requirements. U.S. Coast Guard requirements officials told  us that the training and certification standardizes the proficiency of the  requirements workforce across the component. In addition, Customs and  Border Protection officials told us that they are in the process of training  their personnel on operational requirements development as part of a  larger training program implemented through their Acquisition  Management Performance Improvement effort.

Components provided multiple reasons why they have not assessed their  requirements workforce development needs or implemented a  requirements training program. Specifically:

Federal Emergency Management Agency is waiting on resources to  build a requirements organization and provide component-specific  training.

Immigration and Customs Enforcement officials stated that they are  standing up a requirements development organization and have  requested additional personnel. However, they have not done a  comprehensive assessment of their workforce needs nor established  additional training as a result of resource constraints.

National Protection and Programs Directorate requirements officials  told us that they do not currently have plans to assess the sufficiency  of requirements development personnel and do not have component- specific requirements training.

Transportation Security Administration has recently established a  requirements development organization but has not yet assessed its  workforce needs or established component-specific training.

U.S. Citizenship and Immigration Services requirements officials told  us that they have not assessed their workforce and training needs, as  they are more focused on processes supporting information  technology programs rather than requirements overall.

Acquisition Programs Benefit from an Adequately Staffed and Trained Requirements Workforce

Assessment and training—according to GAO’s internal controls,  workforce development key principles, and DHS’s workforce guidance— are two key steps in workforce planning to ensure that the right numbers  of people with the right skills are available at the right time. Specifically,  an assessment should include an understanding of the goals and  objectives of the component, the workforce needed to achieve the goals,  and the capacity and capabilities needed to support workforce strategies.  With a better understanding of the needs and current capabilities of the  workforce, management can develop specific strategies to better educate  the workforce and standardize skill levels. Organizations can then  develop specific training to develop the workforce and fill areas of  identified need with involvement of management and employees.  Organizations can use a variety of instruction approaches for training—for  example, classroom based learning; distance learning; or structured on- the-job training. When warranted, organizations should consider blending  learning methods (such as web-based and instructor-led) within the same  training effort to leverage resources in the most efficient way possible.  See figure 6 for a notional workforce planning process that matches  workforce needs with the goals of the organization.

The JRC approved a DHS-wide Requirements Specialization as a part of  the Technology Manager Certification on June 21, 2018. In addition, JRC  officials stated that they are expanding requirements development  training and determining course content for the certification. We have  previously found the importance of having the appropriate workforce as a  factor in meeting an agency’s mission. Until the components assess  their needs and take appropriate action, acquisition programs may  continue to be at risk of not meeting end user needs, as they will not have  a trained workforce to develop requirements.

Selected case study acquisition programs further illustrate the effect of a  trained requirements development workforce.

Customs and Border Protection and Immigration and Customs  Enforcement, TECS Modernization (not an acronym) programs:  These programs illustrate the effect that knowledgeable requirements  officials can have. Customs and Border Protection’s TECS program  had an engineer with requirements development experience.  According to this official, TECS Modernization traced all program  requirements from the operational to the technical level in a matrix to  ensure that they were valid and understood. A trained workforce,  however, is one principle among many needed to provide a program  with a sound start. In this case, a trained requirements official took the  critical step of tracing the requirements to the gap, but his involvement  cannot address the requirements and program executing issues that  may arise throughout the life of a program. In fact, TECS  Modernization later experienced changes to requirements and  schedule.

In contrast, Immigration and Customs Enforcement’s TECS  Modernization program officials told us that the program initially  utilized contractors for requirements development. Rather than  develop operational requirements to close the capability gap,  development started with thousands of technical requirements. The  program could not trace these requirements back to the capability  gap, and could not implement the proposed solution. Immigration and  Customs Enforcement re-started the program by bringing in trained  requirements development personnel who worked with the end users  to determine the appropriate operational requirements. Current  Immigration and Customs Enforcement officials acknowledged the  problems of the past but indicated that with the operational  requirements now in place, they have a greater likelihood of success.

Transportation Security Administration, Electronic Baggage  Screening and Passenger Screening Programs: End users of the  screening units at an airport told us they are not aware of anyone,  such as a requirements development official, with whom they can  communicate emerging threats or problems with the screening units.  They also said that some of the key performance parameters, such as  the number of bags processed per hour, are not based on current  data. In their experience, the volume of travelers and bags has  increased significantly. Without a trained requirements development  official with whom end users can provide input, the program risks not  meeting end user needs.

U.S. Coast Guard, Offshore Patrol Cutter: Requirements officials  told us that they continue to mature their requirements workforce to  ensure the appropriate requirements for programs. The U.S. Coast  Guard’s requirements workforce, as stated previously, utilizes an end  user with requirements training as a subject matter expert on  requirements development. These end users with requirements  training work together with end users currently using the assets to  ensure that requirements are well-defined. For this program, the U.S.  Coast Guard recently held an assessment of the draft requirements  for the cutter that solicited input from users across the organization.  The trained requirements personnel facilitated the assessment and  gathered the input to refine the requirements. While it is too early to  determine how this acquisition program will perform against baselines,  this initial focus on requirements is positive.

As most components recognize the need for requirements development,  it is important that they assess their needs for a workforce and align those  needs with training to develop a workforce that can help ensure that  requirements match end user needs. DHS is taking steps to standardize  training and certification across its requirements workforce to ensure that  the workforce across all levels implements requirements development in  accordance with JRIMS. However, DHS remains at risk until such training  and certification are fully implemented throughout DHS and its  components.

While DHS now has the JRIMS in place, which authorizes the  components to create their own internal requirements development  organizations, the components lag in creating the means to develop  requirements and close identified capability gaps. While DHS  components generally are working toward developing their own  requirements policies, they have not yet established timeframes for  completing this effort. Without specific timeframes, there is the risk that  management attention will be lost. Further, some components do not  have in place independent requirements development organizations,  separate from their acquisition functions. The overlap in these  responsibilities does not comport with best practices and engenders a risk  that acquisition officials may override requirements developers to procure  a preferred solution as opposed to the one needed by the end user.

Further, most of the components in our review have not taken steps to  assess their requirements workforces and provide training. Compounding  this problem is a lack of training and certification standards for  requirements personnel at the agency level. Rather, components have  prioritized obtaining funding and starting acquisition programs over  requirements development. Not giving requirements development  adequate priority is likely to contribute to poorly defined requirements and  delays in achieving—or failure to achieve—the capabilities necessary to  perform components’ missions.

DHS, at a department level, has recognized the importance of having a  requirements policy, an independent requirements organization, and a  trained workforce by establishing JRIMS, the JRC, and associated  training. While the components vary in acquisition activity, it is incumbent  on them to recognize the importance of these critical elements. Past  acquisitions have demonstrated the need to do so.

We are making a total of 25 recommendations to the Secretary of DHS.  Specifically, that the Secretary of DHS ensures that:  The Commissioner of Customs and Border Protection through the  Executive Assistant Commissioner for Operations Support finalizes and  promulgates the Customs and Border Protection’s draft policy for  requirements development. (Recommendation 1)

The Commissioner of Customs and Border Protection through the  Executive Assistant Commissioner for Operations Support updates the  2013 workforce assessment to account for the independent requirements  organization’s current workforce needs. (Recommendation 2)

The Commissioner of Customs and Border Protection through the  Executive Assistant Commissioner for Operations Support establishes  component specific training for requirements development.  (Recommendation 3)

The Administrator of the Federal Emergency Management Agency  establishes a policy for requirements development. (Recommendation 4)

The Administrator of the Federal Emergency Management Agency  establishes an independent requirements development organization  within the Federal Emergency Management Agency. (Recommendation  5)

The Administrator of the Federal Emergency Management Agency  updates the 2016 workforce assessment to account for an independent  requirements organization’s workforce needs. (Recommendation 6)

The Administrator of the Federal Emergency Management Agency  establishes component specific training for requirements development.  (Recommendation 7)

The Director of Immigration and Customs Enforcement establishes a  policy for requirements development. (Recommendation 8)

The Director of Immigration and Customs Enforcement establishes the  planned independent requirements development organization within  Immigration and Customs Enforcement. (Recommendation 9)

The Director of Immigration and Customs Enforcement conducts a  workforce assessment to account for an independent requirements  organization’s workforce needs. (Recommendation 10)

The Director of Immigration and Customs Enforcement establishes  component specific training for requirements development.  (Recommendation 11)

The Under Secretary of Homeland Security for the National Protection  and Programs Directorate finalizes and promulgates the National  Protection and Programs Directorate’s draft policy for requirements  development. (Recommendation 12)

The Under Secretary of Homeland Security for the National Protection  and Programs Directorate establishes the planned independent  requirements development organization within the National Protection  and Programs Directorate. (Recommendation 13)

The Under Secretary of Homeland Security for the National Protection  and Programs Directorate conducts a workforce assessment to account  for an independent requirements organization’s workforce needs.  (Recommendation 14)

The Under Secretary of Homeland Security for the National Protection  and Programs Directorate establishes component specific training for  requirements development. (Recommendation 15)

The Administrator of the Transportation Security Administration through  the Executive Assistant Administrator of Operations Support finalizes and  promulgates the Transportation Security Administration’s draft policy for  requirements development. (Recommendation 16)

The Administrator of the Transportation Security Administration through  the Executive Assistant Administrator of Operations Support conducts a  workforce assessment to account for an independent requirements  organization’s workforce needs. (Recommendation 17)

The Administrator of the Transportation Security Administration through  the Executive Assistant Administrator of Operations Support establishes  component specific training for requirements development.  (Recommendation 18)

The Commandant of the U.S. Coast Guard through the Assistant  Commandant for Capability conducts a workforce assessment of the U.S.  Coast Guard’s capabilities directorate. (Recommendation 19)

The Director of the U.S. Citizenship and Immigration Services finalizes  and promulgates the U.S. Citizenship and Immigration Services’s draft  policy for requirements development. (Recommendation 20)

The Director of the U.S. Citizenship and Immigration Services establishes  the planned independent requirements development organization within  U.S. Citizenship and Immigration Services. (Recommendation 21)

The Director of the U.S. Citizenship and Immigration Services conducts a  workforce assessment to account for an independent requirements  organization’s workforce needs. (Recommendation 22)

The Director of the U.S. Citizenship and Immigration Services establishes  component specific training for requirements development.  (Recommendation 23)

The JRC collaborate with components on their requirements development  policies and, in partnership with the Under Secretary for Management,  provide oversight to promote consistency across the components.  (Recommendation 24)

In addition, the Secretary of DHS should ensure that training for  requirements development is consistent by establishing training and  certification standards for DHS and the components’ requirements  development workforces. (Recommendation 25)

We provided a draft of this report for review and comment to DHS.

DHS provided written comments, which are reproduced in appendix II. In  its comments, DHS concurred with all 25 of our recommendations and  identified actions it plans to take to address them. DHS also provided  technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees and the Secretary of Homeland Security. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report discusses (1) how often selected Department of Homeland  Security (DHS) programs changed requirements; and assesses the  extent to which the selected components have (2) developed policies for  requirements development, (3) established independent requirements  organizations, and (4) taken steps to assess and train a requirements  workforce. Our focus for this report was on the DHS components, as they  are responsible for developing the requirements to meet end user needs.

To conduct our work, we reviewed the DHS Master Acquisition Oversight  List as of April 2017 and selected seven DHS components with Level 1  and Level 2 major acquisition programs and cover a broad range of  missions. The seven components are as follows:

National Protection and Programs Directorate

U.S. Citizenship and Immigration Services  From these seven components, we selected 14 major acquisition  programs with DHS-approved key performance parameters to serve as  case studies for our review. We selected a non-generalizable sample of  programs based on different factors, including their acquisition phase,  component, acquisition level, and whether they were information  technology (IT) or non-IT. We selected the programs on these factors to  reflect the broad spectrum of DHS components’ operations. In addition,  we coordinated our program selection with the DHS Office of Inspector  General due to its ongoing audit on the implementation of Joint  Requirements Council (JRC) policies in DHS acquisition programs. See  table 4 below for a description of the programs.

We also reviewed two programs that did not have DHS-approved key  performance parameters at the time of our review to understand how  requirements are determined before DHS validation. The two programs  were Customs and Border Protection’s Cross Border Tunnel Threat and  Biometric Entry-Exit Program.

To determine the extent to which the selected programs changed  operational requirements, we examined key performance parameters,  which the programs document in requirements and acquisitions  documents, before and after DHS approval when key performance  parameters should be stable. Such program documents include the  operational requirements documents and acquisition program baselines.  In certain cases, programs had multiple iterations of these documents.  We then compared the extent to which key performance parameters  changed between documents. We selected operational requirements  documents and acquisition program baselines because these are the key  requirements documents validated by DHS management in order for  programs to begin development.

We focused on the presence of policies for requirements development,  independent requirements organizations, and requirements specific  workforce and training in components as our past work on major  acquisitions has shown that these are the fundamental building blocks  required to develop well-informed operational requirements. This  selection was also informed by our standards for internal controls.

To determine the extent to which DHS components’ requirements  development policies exist, as well as determine the extent to which those  components established independent organizations, we reviewed  component documentation pertaining to requirements development, such  as instruction manuals, mission statements, and capability analyses. We  also reviewed DHS documentation such as the Joint Requirements  Integration and Management System Instruction Manual and the  Acquisition Management Instruction to determine the requirements  development guidance provided to the components. We also reviewed  program-level documents such as mission need statements and  operational requirements documents to determine the capability gaps that  respective programs were intended to mitigate, and the programs’ key  performance parameters.

To help determine assessment, training, and certification standards for  DHS’s requirements development workforce, we spoke with officials from  Defense Acquisition University regarding comparable standards that  apply to the Department of Defense’s requirements workforce. We also  reviewed training standards materials provided by these officials. In  addition, we spoke with JRC and U.S. Coast Guard officials regarding  their requirements development training and certification standards and  reviewed available documentation.

To inform each of our objectives, we interviewed officials at various levels  throughout DHS to understand their relationship to requirements  development. We interviewed JRC officials to determine their interaction  with components for requirements development, policies, training, and  organizational standards. We also interviewed component-level officials  to understand the extent to which they have implemented requirements  development policies, organizations, and training for their components.  We then interviewed both program officials and program end users to  understand their roles in requirements development, the extent to which  their feedback is incorporated into the requirements development  process, and then the extent to which they receive requirements  development training. In addition, we furthered this understanding through  reviewing component- and program-level documentation including  guidance manuals, mission needs statements, and operational  requirements documents.

We assessed the components’ requirements development practices  against GAO’s standards for internal control and additional supporting  criteria. The standards identify key principles to help entities achieve their  objectives, such as delivering capabilities to end users. Specifically,  management should establish structure, responsibility, and authority  including developing an organizational structure and documentation. In  addition, management should have a commitment to competence by  developing individuals, such as through training.

Appendix II: Comments from the Department of Homeland Security

Marie A. Mak, (202) 512-4841, or makm@gao.gov.

In addition to the contact named above, J. Kristopher Keener, Assistant  Director; James Kim; Stephen V. Marchesani; Cody Knudsen; Claire  McGillem; Pete Anderson; Roxanna Sun; and Sylvia Schatz made key  contributions to this report.

The Department of Homeland Security invests billions of dollars each year in major acquisition programs to assist in executing its many critical missions. We’ve previously found that DHS agencies had acquisition programs that did not meet requirements. Sometimes operational requirements were poorly defined, increasing the risk of not meeting the needs of end users in the field, such as emergency responders.

We looked at seven DHS agencies and found that some meet GAO-identified best practices for defining operational requirements and others do not.

We made 25 recommendations to DHS to improve its agencies’ methods for developing requirements.