Information Technology Reform: Agencies Need to Improve Certification of Incremental Development

Investments in federal IT too often result in failed projects that incur cost overruns and schedule slippages. Recognizing the severity of issues related to government-wide IT management, Congress enacted federal IT acquisition reform legislation in December 2014. Among other things, the law states that OMB require in its annual IT capital planning guidance that CIOs certify that IT investments are adequately implementing incremental development.

GAO was asked to review agencies' use of incremental development. This report addresses the number of investments certified by agency CIOs as implementing adequate incremental development and any reported challenges, and whether agencies' CIO certification policies and processes were in accordance with FITARA. GAO analyzed data for major IT investments in development, as reported by 24 agencies, and identified their reported challenges and use of certification information. GAO also reviewed the 24 agencies' policies and processes for the CIO certification of incremental development and interviewed OMB staff.

Agencies reported that 62 percent of major information technology (IT) software development investments were certified by the agency Chief Information Officer (CIO) for implementing adequate incremental development in fiscal year 2017, as required by the Federal IT Acquisition Reform Act (FITARA) as of August 2016. However, a number of responses for the remaining investments were incorrectly reported due to agency error. Officials from 21 of the 24 agencies in GAO's review reported that challenges hindered their ability to implement incremental development, which included: (1) inefficient governance processes; (2) procurement delays; and (3) organizational changes associated with transitioning from a traditional software methodology that takes years to deliver a product, to incremental development, which delivers products in shorter time frames. Nevertheless, agencies reported that the certification process was beneficial because they used the information from the process to assist with identifying investments that could more effectively use an incremental approach, and using lessons learned to improve the agencies' incremental processes.

As of August 2017, only 4 of the 24 agencies had clearly defined CIO incremental development certification policies and processes that contained: descriptions of the role of the CIO in the process; how the CIO's certification will be documented; and included definitions of incremental development and time frames for delivering functionality consistent with Office of Management and Budget (OMB) guidance (see figure).

In addition, OMB's fiscal year 2018 capital planning guidance did not establish how agency CIOs are to make explicit statements to demonstrate compliance with FITARA's incremental provisions, while the 2017 guidance did. However, OMB's fiscal year 2019 guidance provides clear direction on reporting incremental certification and is a positive step in addressing this issue.

GAO is making 19 recommendations to 17 agencies, including 3 to improve reporting accuracy and 16 to update or establish certification policies. Eleven agencies agreed with GAO's recommendations, 1 partially agreed, and 5 did not state whether they agreed or disagreed. OMB disagreed with several of GAO's conclusions, which GAO continues to believe are valid, as discussed in the report.

Federal agencies rely on information technology (IT) systems to provide  essential services affecting the health, economy, and defense of the  nation. In fiscal year 2017, agencies were budgeted to spend more than  $89 billion on IT, including more than $43 billion on 752 major  investments. With many of these investments in a development phase, it  is important to ensure that agencies are making the most efficient use of  their financial resources through effective management practices.  However, as we have previously testified, IT projects often fail—that is,  even after exceeding their budgets by millions of dollars and delaying the  schedules by years—the results do not meet requirements.

Recognizing the severity of challenges related to the government-wide  management of IT, in December 2014, Federal Information Technology  Acquisition Reform Act provisions (commonly referred to as FITARA)  were enacted as a part of the Carl Levin and Howard P. ‘Buck’ McKeon  National Defense Authorization Act for Fiscal Year 2015. The law states  that the Office of Management and Budget (OMB) require in its annual IT  capital planning guidance that each covered agency’s chief information  officer (CIO) certify that IT investments are adequately implementing  incremental development, as defined in capital planning guidance issued  by OMB.

Further, in February 2015, we added improving the management of IT  acquisitions and operations to our high-risk list—a list of agencies and  program areas that have a higher potential for fraud, waste, abuse, and  mismanagement, or are in need of transformation. In introducing this risk  area, we specifically noted that agencies had used incremental  development less than 50 percent of the time for a selection of  investments that we reviewed and called on agencies to improve their  delivery of functionality to ensure that a minimum of 80 percent of the  government’s acquisitions deliver functionality every 12 months. We  recently issued an update to our high-risk report in February 2017 and  noted that, while progress has been made in addressing the IT  acquisitions and operations high-risk area, significant work remains to be  completed, including the need for demonstrated progress by agencies in  delivering functionality every 12 months on major acquisitions.

You asked us to review agencies’ use of incremental development in  managing major IT investments. Our objectives were to determine: (1) the  number of investments certified by agencies as implementing adequate  incremental development and any reported challenges that impact the  agencies’ incremental delivery of functionality, and (2) whether agencies  are establishing policies and processes for CIO certification of  incremental development in accordance with FITARA.

To address the first objective, we analyzed fiscal year 2017 data for the  major investments reported on the IT Dashboard as of August 31, 2016,  by the 24 agencies required to provide this information under FITARA.  These agencies are the Department of Commerce (Commerce),  Department of Defense (Defense), Department of Education (Education),  Department of Energy (Energy), Department of Health and Human  Services (HHS), Department of Homeland Security (DHS), Department of  Housing and Urban Development (HUD), Department of the Interior  (Interior), Department of Justice (Justice), Department of Labor (Labor),  Department of State (State), Department of Transportation  (Transportation), Department of the Treasury (Treasury), Department of  Veterans Affairs (VA), U.S. Department of Agriculture (USDA),  Environmental Protection Agency (EPA), General Services Administration  (GSA), National Aeronautics and Space Administration (NASA), National  Science Foundation (NSF), Office of Personnel Management (OPM),  Small Business Administration (SBA), Social Security Administration  (SSA), U.S. Agency for International Development (USAID), and U.S.  Nuclear Regulatory Commission (NRC).

Among the reported investments, we identified 166 investments  undertaking software development activities in which at least 50 percent  or more of funding was allocated to development, modernization, and  enhancement activities. For each of these investments, we assessed  the status of reported certifications by the CIOs of the respective  agencies. We presented the results of our data analysis to officials from  each agency’s Office of the CIO and incorporated any changes to the  data as appropriate. We determined that the data were sufficiently reliable  for the purpose of this report.

We also presented a list of challenges identified during our past work on  incremental development to officials in the Office of the CIO at each of the  24 agencies involved with investment management and software  development activities. We then asked the officials to identify from the  list their top three challenges to delivering incremental functionality;  explain the actions they took to addresses the challenges; and describe  the outcomes. In addition, we also asked agency officials to identify any  challenges that were not included in the list, but which were among their  top three challenges. Lastly, we also asked the agencies in our review to  identify how the CIO utilized the information obtained during the process  of certifying investments’ adequate incremental development to make  decisions regarding the agency’s major IT investments.

For the second objective, we analyzed the 24 agencies’ CIO certification  policies and processes to determine whether they were consistent with  the provision in FITARA and OMB guidance. In addition, we  interviewed staff from OMB’s Office of E-Government and Information  Technology regarding OMB’s guidance related to incremental  development and FITARA. A full description of our objectives, scope, and  methodology can be found in appendix I.

We conducted this performance audit from July 2016 to November 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Investments in federal IT have the potential to make agencies more  efficient in fulfilling their missions. However, as we have previously  reported, these investments too often result in failed projects that incur  cost overruns and schedule slippages, while contributing little to mission- related outcomes. For example:

The Farm Service Agency’s Modernize and Innovate the Delivery of  Agricultural Systems program, which was to replace aging hardware  and software applications that process benefits to farmers, was halted  in July 2014 after investing about 10 years and at least $423 million,  while only delivering about 20 percent of the functionality that was  originally planned.

Defense’s Expeditionary Combat Support System was canceled in  December 2012, after spending more than a billion dollars and failing  to deploy within 5 years of initially obligating funds.

VA’s Financial and Logistics Integrated Technology Enterprise  program was intended to be delivered by 2014 at a total estimated  cost of $609 million, but was terminated in October 2011 due to  challenges in managing the program.

OPM’s Retirement Systems Modernization program was canceled in  February 2011, after spending approximately $231 million on the  agency’s third attempt to automate the processing of federal  employee retirement claims.

DHS’s Secure Border Initiative Network program was ended in  January 2011, after the department obligated more than $1 billion to  the program, because the program did not meet cost-effectiveness  and viability standards.

The tri-agency (Defense, NASA, and the National Oceanic and  Atmospheric Administration) National Polar-orbiting Operational  Environmental Satellite System was a weather satellite program that  was disbanded by the White House Office of Science and Technology  Policy in February 2010 after the program spent 16 years and almost  $5 billion.

The VA Scheduling Replacement Project was terminated in  September 2009 after spending an estimated $127 million over 9  years.

One approach to reducing software development risks is to divide  investments into smaller parts, or increments. While a traditional waterfall  software development effort usually is broadly scoped, multiyear, and  produces a product at the end of a sequence of phases, an incremental  development approach delivers software products in smaller modules  with shorter time frames. This development technique has been  recognized in prior law since 1996 and in OMB guidance since 2000.

By following an incremental development approach, agencies have the  potential to: deliver capabilities to their users more rapidly, giving them more  flexibility to respond to changing agency priorities;  increase the likelihood that each project will achieve its cost,  schedule, and performance goals; obtain additional feedback from users, increasing the probability that  each successive increment and project will meet user needs; more easily incorporate emerging technologies; and  terminate a poorly performing investment, with fewer sunk costs.

Since 2000, OMB Circular A-130 has directed agencies to incorporate an  incremental development approach into their policies and ensure that  investments implement them. Further, since 2012, OMB has required  that functionality be delivered at least every 6 months.

In addition, FITARA states that OMB is to require in its annual IT capital  planning guidance that covered agency CIOs certify that IT investments  are adequately implementing incremental development, as defined in  capital planning guidance issued by OMB. Accordingly, in June 2015,  OMB released two related sets of guidance on the implementation of  FITARA that included instructions pertaining to CIO certification of  adequate incremental development. In particular, agencies were to,  among other things:

Develop policies and processes which ensure CIO certification.

OMB required agencies to define IT policies and processes which  ensure that the CIO certifies that IT resources are adequately  implementing incremental development. In the guidance, OMB  defined adequate incremental development as the planned and actual  delivery of new or modified technical functionality to users that occurs  at least every 6 months for development of software or services.

Report the status of CIO certification. OMB’s guidance required  agency CIOs to certify in each major IT investment’s business case  whether the investment’s plan for the current year adequately  implements incremental development.

OMB uses the major IT business cases to monitor major investments  once they are funded. Performance information on each major  investment, including the status of incremental delivery, is made publicly  available on the web-based IT Dashboard. In using the IT Dashboard,  OMB intends to provide transparency and oversight into these agencies’  investments. This public display of data is also intended to allow  Congress and government oversight bodies, as well as the general  public, to hold agencies accountable for the results and progress of the  investments.

Further, OMB issued its fiscal year 2018 and fiscal year 2019 capital  planning guidance in June 2016 and August 2017, respectively, which  required agency CIOs to provide the certifications needed to demonstrate  compliance with FITARA.

GAO Has Reported on Efforts to Improve IT Acquisitions Using Incremental Development

During the past several years, we have reported on a variety of  challenges related to improving federal IT acquisitions through the use of  incremental development. In 2011, we identified seven successful  investment acquisitions and nine common factors critical to their  success. Specifically, we reported that department officials had  identified seven successful investments that best achieved their  respective cost, schedule, scope, and performance goals. Notably, all of  these were smaller increments, phases, or releases of larger projects. For  example, the Defense investment in our sample was the seventh  increment of an ongoing investment; Energy’s system was the first of two  phases; the DHS investment was rolled out to two locations prior to  deployment to 37 additional locations; and Transportation’s investment  had been part of a prototype deployed to four airports.

Common factors critical to the success of three or more of the seven  investments were:  1.  Program officials were actively engaged with stakeholders.  2.  Program staff had the necessary knowledge and skills.  3.  Senior department and agency executives supported the programs.  4.  End users and stakeholders were involved in the development of  requirements.  5.  End users participated in testing system functionality prior to formal  end-user acceptance testing.  6.  Government and contractor staff were stable and consistent.  7.  Program staff prioritized requirements.  8.  Program officials maintained regular communication with the prime  contractor.  9.  Programs received sufficient funding.

These critical factors help support OMB’s objective of improving the  management of large-scale IT acquisitions across the federal  government.

In May 2014, we reported on the status of incremental development at  five agencies (Defense, DHS, HHS, Transportation, and VA). We noted  that these agencies planned to deliver functionality for fewer than half of  the investments in 12-month cycles and that only about one-fourth of  these investments would deliver in 6-month increments, as required by  OMB. Additionally, OMB staff reported to us that they did not expect that  many investments would meet the 6-month requirement. Therefore, we  questioned whether a 6-month delivery requirement was an appropriate  government-wide goal and whether OMB should instead consider a 12- month time frame, as called for in its IT Reform Plan. Accordingly, we  recommended that OMB require projects to deliver functionality at least  every 12 months. OMB disagreed with our recommendation, asserting  that changing the requirement from 6 to 12 months would reduce the  emphasis on incremental development that it had been advocating and  that 6 months was an appropriate goal. However, we noted in our report,  agencies’ plans to deliver functionality every 6 months was low and it  would not always be practical for certain types of investments to deliver  functionality every 6 months. We therefore continue to believe our  recommendation is appropriate.

We also recommended that OMB develop and issue clearer guidance on  incremental development to ensure that it has the necessary information  to oversee the extent to which projects and investments are implementing  its guidance. OMB took action to address this recommendation and  issued capital planning guidance in fiscal year 2016 that requires  agencies to report on whether each of their projects has delivered a  production release every 6 months and to provide a rationale if  functionality is not being delivered. In addition, we recommended that the  five selected agencies—Defense, DHS, HHS, Transportation, and VA— update and implement their associated policies. Most agencies agreed  with our recommendation or had no comment. As of September 2017,  Defense, DHS, Transportation, and VA have addressed our  recommendation.

In February 2015, we added improving the management of IT acquisitions  and operations to our high-risk list, citing a lack of disciplined and  effective management and inconsistent application of best practices to  the successful acquisition of IT projects throughout the federal  government. In particular, we noted the critical importance of  implementing incremental development in order to reduce investment risk  and called on federal agencies to ensure that a minimum of 80 percent of  the government’s major acquisitions deliver functionality at least every 12  months.

In August 2016, we reported on the status of incremental development  and noted that, for fiscal year 2016, 22 agencies had reported on the IT  Dashboard that 64 percent of their software development projects would  deliver useable functionality every 6 months, as required by OMB.  However, shortcomings in OMB’s guidance—the lack of clarity regarding  the types of projects where incremental development would not apply,  and how the status of these nonsoftware projects should be reported— affected the accuracy of the data on the IT Dashboard. We therefore  recommended in August 2016 that OMB clarify its existing guidance  regarding what IT investments were and were not subject to requirements  on the use of incremental development and how CIOs should report the  status of projects that were not subject to these requirements. OMB did  not specifically agree or disagree with our recommendation, but stated  that it generally agreed with our report. In April 2017, OMB staff reported  that the agency had taken action and included language to address our  recommendation in its fiscal year 2018 guidance; however, an analysis of  that guidance showed that it still lacked direction on how CIOs are to  report the status of nonsoftware projects.

In addition, for our August 2016 report, we reviewed seven departments’  guidance and found that only three departments (Commerce, DHS, and  Transportation) had policies and processes to ensure that the CIO would  certify that IT investments were adequately implementing incremental  development in accordance with FITARA. We therefore made  recommendations to the remaining four departments (Defense,  Education, HHS, and Treasury) to establish a policy and process for the  certification of major IT investments’ adequate use of incremental  development, in accordance with OMB’s guidance on the implementation  of FITARA. Two departments concurred with our recommendation, one  department disagreed, and one department did not comment. As of  August 2017, none of the four departments had taken action to address  the recommendation; as discussed later in the report.

We issued an update to our high-risk report in February 2017 and noted  that, while progress has been made in addressing this high-risk area,  significant work remains to be completed. For example, as of December  2016, OMB and agencies had implemented 366 (or about 46 percent) of  the 803 open recommendations that we had made from fiscal years 2010  through 2015 related to IT acquisitions and operations. We also noted  that agencies needed to make demonstrated progress in delivering  functionality every 12 months on major acquisitions.

Further, in April 2017, we reported on the results of a forum, convened by  the Comptroller General on September 14, 2016, to explore challenges  and opportunities for CIOs to improve federal IT acquisitions and  operations—with the goal of better informing policymakers and  government leadership. Thirteen current and former federal agency  CIOs, members of Congress, and private sector IT executives noted the  importance of federal agencies’ IT procurement offices and processes  evolving to align with new technologies, as agencies are not always set  up to take advantage of acquisitions using Agile development  processes.

Agencies Reported That Most of Their Major Software Development Investments Were Certified as Having Adequate Incremental Development, but Continue to Face Challenges and Identify Benefits

Agencies reported to OMB through the IT Dashboard that more than half  of their major software development investments were certified by the  CIO as implementing adequate incremental development as of August  2016. For the remaining investments, the agencies offered various  interpretations regarding what investments needed to be certified. For  example, officials of several agencies reported that they were not utilizing  incremental development for certain investments. In other instances,  agencies did not provide a response to OMB regarding the question in the  major IT business case about certification, or responded that they did not  consider certification to be applicable for their investments. However,  based on OMB’s guidance, a number of these “not applicable” responses  were incorrectly reported, as these agencies had investments that  included software development and were, therefore, required to report on  the certification of adequate incremental development.

In addition, officials from a majority of the agencies reported that multiple  challenges had impacted their ability to implement adequate incremental  development. These challenges related to inefficient governance  processes; procurement delays; the lack of stable, prioritized  requirements; and organizational and cultural changes associated with  the transition from a traditional software methodology to an incremental  methodology. Nevertheless, officials from 21 agencies reported that the  certification process was beneficial because they used the information  obtained during the process to assist with management oversight of major  IT investments, including identifying investments that could be using a  more effective incremental approach and using lessons learned to  improve the agency’s incremental processes.

CIOs Certified 62 Percent of Major IT Investments as Having Adequate Incremental Development

FITARA states that, in its annual IT capital planning guidance, OMB is to  require CIOs to certify that IT investments are adequately implementing  incremental development. In 2015, OMB defined adequate incremental  development as the planned and actual delivery of new or modified  technical functionality to users that occurs at least every 6 months for  development of software or services. Further, OMB’s IT capital planning  guidance for fiscal year 2017 required CIOs to certify whether their  agencies’ major IT investments had adequately implemented incremental  development for the current year. Specifically, agencies were to  respond to a question in the major IT business case regarding whether  the CIO certified adequate incremental development for each investment  with a response of either yes, no, or not applicable. Agencies’ responses  to this question are publicly reported by OMB on the IT Dashboard.

As of August 31, 2016, 21 of the 24 agencies in our review had reported  on the IT Dashboard a total of 166 major software development  investments that were planned to be primarily in development for fiscal  year 2017. Of these 166 investments, the agencies reported that 62  percent (103 investments) were certified by the CIO as using adequate  incremental development for fiscal year 2017, as shown in table 1 in  alphabetical order by department and agency. (For additional details on  the certification status of the 166 investments, see appendix II.)

For the remaining 63 investments, 8 agencies either reported in the major  IT business case that the investment was not certified as adequately  implementing incremental development or that certification was not  applicable. Three other agencies did not provide a response to the  question regarding certification in the major IT business case submitted to  OMB. Figure 1 shows the breakdown of responses by agency regarding  investments that were not certified as implementing adequate incremental  development, as reported on the IT Dashboard.

Officials in the Office of the CIO at each of the 3 agencies provided a  variety of reasons for why the 11 investments were not certified as  implementing adequate incremental development. For example, HHS  officials noted that certain investments are required to meet complex  statutory requirements and, thus, a 6-month release schedule is not  always appropriate for them. Interior officials stated that their investment  had just been categorized as a major investment and, at the time of the  submission of certification status, a baseline had not been approved. The  officials stated, however, that the baseline has since been approved and  the investment is expected to deliver functionality every 6 months.  Further, SSA officials reported that 3 investments were not software  development initiatives even though 2 of these investments had been  inaccurately reported as such on the IT Dashboard.

Regarding the 33 investments for which the 3 agencies did not provide a  response in the major IT business case for the investment, officials from  each agency’s Office of the CIO attributed the lack of a response to either  data entry errors or the agency not being required to publicly report this  information for the investments. In particular, USDA and Treasury officials  reported that the lack of certification data on the IT Dashboard was the  result of a data entry error. Treasury officials also stated that the agency’s  missing responses were due to a lack of administrative oversight in  reviewing the data for accuracy and consistency. The officials noted that  the Treasury CIO had certified all of the agency’s investments but some  investments failed to select the proper response in the business case.

Defense officials reported that 16 investments were categorized as  national security systems and, therefore, were exempt from public  reporting on the IT Dashboard (though not exempt from acquisition  policies regarding the use of incremental development). The officials  said that they did not provide a response on the remaining 7 investments  because 1 investment was not a software development effort and the  other 6 investments were designated as major automated information  systems and, therefore, the agency did not have to submit business  cases to OMB with this information.

Lastly, officials from the Office of the CIO at 7 agencies reported a variety  of reasons for why they had provided a response of “not applicable” for 19  investments. For example, Interior officials stated that, at the time of the  certification submission, the investment did not have any approved  development projects and, therefore, the agency had indicated not  applicable in its response for the one investment. However, the officials  stated that the investment’s projects have since been approved and the  CIO has reviewed the investment and certified adequate incremental  development.

For the remaining 18 investments at the other 6 agencies (Commerce,  DHS, Education, Energy, HHS, and Transportation), officials from each  agency’s Office of the CIO reported that the majority of the projects  associated with their investments were not primarily related to software  development, or that they were using either a non-incremental  development methodology or a mixed non-incremental/incremental  development methodology. As a result, the officials believed the  certification of adequate incremental development was not applicable,  even though at least one project within each of the investments involved  software development.

However, based on OMB’s guidance, these “not applicable” responses for  the 18 investments were incorrectly reported and the agencies should  have provided either a “yes” or “no” response to the certification question  because the investment included software development. Specifically,  OMB’s fiscal year 2017 capital planning guidance states that certification  of incremental development applies to any investment that is developing  software or services, as noted in its definition of adequate incremental  development. In addition, staff in OMB’s Office of E-Government and  Information Technology stated that a “not applicable” response to the  question was only acceptable in cases where software development was  not occurring, such as an investment related to infrastructure or  technology refreshment of equipment.

Staff in the Office of E-Government and Information Technology  acknowledged the need for more meaningful oversight of agencies’ use of  incremental development and stated that, beginning in fiscal year 2018,  OMB will no longer require agencies to report CIO certification information  in their investments’ major IT business cases or on the IT Dashboard.  Rather, OMB staff stated that agencies would be required to separately  provide the certifications needed to demonstrate compliance with  FITARA. OMB’s revised approach and agencies’ implementation of  OMB’s guidance are further discussed later in this report.

Regardless of the reporting requirements in place, it remains critical that  federal agencies report accurate incremental development information to  OMB because of OMB’s plans to use this information for investment  management and oversight. However, our September 2016 work has  highlighted the poor quality of data related to incremental development at  the project level, including whether a project is delivering a release every  6 months. Specifically, we reviewed seven agencies’ major IT software  projects and found inconsistencies that affected the accuracy of the  reported rates of delivery for all agencies—and at least a 10 percentage  point difference in the reported rate on the IT Dashboard for five of these  agencies. We therefore made recommendations to the seven agencies to  improve their reporting of incremental development data on the IT  Dashboard.

Having accurate data on agency investments’ use of incremental  development is critical for providing oversight and management of these  investments and to ensure that OMB and lawmakers can hold CIOs  accountable for the investments’ performance. We have previously made  recommendations to Commerce, Defense, DHS, Education, HHS,  Transportation, and Treasury to improve the accuracy of reporting on the  IT Dashboard and continue to believe these recommendations are  appropriate. In addition, until Energy, SSA, and USDA improve their  reporting of incremental development data on the IT Dashboard, their  efforts to improve the use of incremental development may not be  successful. As a result, the agencies increase the risk that the potential  impact of utilizing incremental development to more quickly deliver useful  functionality to users and improve the likelihood that these multimillion  dollar projects will meet their stated goals, may not be realized.

Multiple Challenges Were Commonly Identified by Agencies as Impacting the Delivery of Incremental Functionality

The majority of the 24 agencies in our review reported that multiple  challenges had impacted their ability to adequately implement  incremental development for their major IT software development  investments. In particular, when presented with a list of challenges  identified by our past work on incremental development, 21 of the  agencies selected seven common challenges to developing investments  incrementally. Each of these seven challenges was selected by 5 or more  agencies. For example:

14 agencies identified problems with program staff over-utilization and  the lack of skills and experience as their top challenge;

6 agencies reported that development work was slowed by inefficient  governance and oversight processes;

5 agencies reported that development schedules were impeded by  procurement delays; and

5 agencies identified the lack of stable, prioritized requirements as a  challenge.

In addition, 3 agencies identified a new challenge which had not been  described in our prior work. Specifically, they reported that organizational  and cultural changes associated with the transition from a traditional  waterfall software methodology to an incremental methodology required  more time and resources to implement than anticipated. Table 2  summarizes the common challenges identified by agencies and the  number of agencies that reported each challenge, ranked by number of  agencies reporting the challenge. Examples of the challenges—and  actions taken to overcome them—are discussed following the table.

Project staff were over-utilized or lacked the necessary skills and  experience. Officials from the Office of the CIO at 14 agencies (DHS,  Education, EPA, GSA, Justice, NASA, NRC, OPM, SBA, SSA, State,  Treasury, USAID, and VA) reported challenges in implementing  incremental development practices associated with project staff, such as  a lack of staff with the necessary skills and experience in utilizing  incremental approaches, inadequate training on these approaches,  overutilization of business or subject matter experts, and the lack of  engagement between product owners and subject matter experts. To  address these challenges, agency officials reported implementing new  approaches, such as training programs focused on incremental  development, coaching strategies to assist project managers in managing  acquisitions, and hiring practices. For example, among these agencies:

DHS officials reported that project staffs’ lack of necessary skills and  experience in understanding the requirements for managing major IT  acquisitions is an ongoing issue, not only related to incremental  development, but also to IT program and project management. The  officials stated that they had developed an acquisition coaching and  assistance strategy that was intended to establish an experienced  team of acquisition coaches who were up-to-date on the latest  acquisition, contracting, and development techniques to assist project  managers in managing the acquisitions. The officials stated that they  hoped to present lessons learned and recommendations on this  strategy to the agency’s Agile working group in summer 2017.

Treasury officials reported a significant need for specialized  engineers, architects, and developers with skills in older programming  languages to maintain its many legacy systems. For example, officials  noted that the agency is modernizing its core taxpayer account  processing applications, which utilized antiquated programming  languages, to more modernized platforms. Treasury officials noted  that they have been shifting staff to meet immediate needs;  augmenting teams with contractors, where possible; and hiring new  staff to fill critical open positions. Nevertheless, the officials said they  have had to slow work on four key projects and delay the launch of  other projects. In addition, the officials stated that they are relying on  contractors more to meet the agency’s staffing needs.

EPA officials noted that, as the agency transitions from using waterfall  software development approaches to Agile-based approaches, it  needs more skilled staff with experience in Agile development. These  officials stated that the agency’s CIO had taken several actions to  address this challenge, including creating an Office of Digital Services  and Technical Architecture to promote Agile and user-centered  design, establishing a fellowship program to bring outside Agile  experts into the project teams, and creating a blanket purchase  agreement to allow agency project teams to purchase Agile  programming and consulting services.

According to NRC officials, one of the greatest incremental delivery  challenges has been the difficulty of engaging sufficient business area  product owners and subject matter experts. For example, the officials  explained that, despite product owners’ enthusiasm for increased  engagement with developers, the demands of the agency’s core  mission work presents challenges for these owners in being available  for meetings related to Agile development activities. NRC officials  informed us that the agency had addressed the challenge by working  to establish a predictable, recurring schedule for product owner and  subject matter expert engagement on development projects, where  expectations are communicated to management about time  commitments.

Further, agency officials from a number of the 14 agencies that  experienced this challenge reported varying approaches to implementing  new incremental development training. For example, Treasury officials  stated that the agency has developed in-house training for existing  developers to meet the needs of its modernized programs. Education  officials noted that the agency identified a select team of IT professionals  within the agency to receive formal training in incremental development  practices. Further, VA officials told us that its Enterprise Program  Management Office is focused on training IT personnel on incremental  development principles. Finally, SSA officials reported that the agency  had launched a training program that had sent hundreds of developers  through a 6-week boot camp program, which included courses in  incremental development and modern coding languages.

Programs did not receive sufficient funding or received funding later  than needed. Officials from the Office of the CIO at nine agencies (GSA,  NASA, OPM, SSA, State, Treasury, USAID, USDA, and VA) reported  challenges associated with programs not receiving sufficient funding or  not receiving funding until late in the fiscal year. These challenges were a  result of changing funding priorities, budget cuts, and continuing  resolutions, which disrupted delivery schedules and required agencies to  delay, reprioritize, or discontinue the rollout of particular investments or  modernization activities. Agencies reported adopting various approaches  to overcome the challenges in this area, such as delaying project  schedules, developing alternate plans for delivering functionality, and  using flexible contracting strategies. For example:

USDA officials reported that funding for a number of projects was not  available until late in the fiscal year, which impacted project  schedules. The officials stated that one component agency addressed  the funding delay by adjusting schedule start dates for projects  relative to the current fiscal year, which helped to improve schedule  projections.

OPM officials told us that they had faced challenges in performing  work on incremental projects due to a lack of available resources  caused by delays in receiving funding. The officials stated that they  addressed this challenge by developing alternate plans to delivering  incremental functionality with a different scope or focus for the  system.

VA officials reported that they faced challenges with funding IT efforts  that span multiple years. The officials noted that administrative  priorities often change over time, impacting the level of funding  approved in subsequent years to undertake incremental development  projects. To address this, officials noted that they used flexible  contracting strategies—such as options that allow the government to  continue the contract only when funding is assured, adjusting a  contract’s time frames to match a delay, adjusting schedules,  designing contracts so that a vendor is paid based on completion of  measured functionality, and using the change request process to  contribute funding to other projects.

Treasury officials stated that the lack of a dedicated funding  commitment had led to difficulties in longer-term strategic planning for  IT improvements. The officials stated that resources assigned to  certain IT projects had to be leveraged for legislatively mandated  investments, causing project delays and pauses for these projects. As  a result, the officials reported that the agency had been reviewing  core initiatives and infrastructure programs, such as infrastructure,  hardware, and software refresh and process improvements, to  determine if they can scale back scope or lengthen schedules. The  officials said that at least one program has been formally paused.

Projects experienced management and organizational challenges  that introduced delays. Officials from the Office of the CIO at seven  agencies (Commerce, Interior, NASA, NRC, NSF, SBA, and  Transportation) reported that management and organizational challenges  had introduced delays in delivering functionality to users. These  challenges included delays in testing and meeting delivery schedules due  to dependencies on other systems or projects and a lack of approved  software or appropriate equipment. Agency officials reported  implementing various approaches to overcome these challenges, such as  addressing external dependencies, tailoring development processes, and  providing waivers for the acquisition of software and hardware. For  example:

Commerce officials reported that they faced organizational challenges  in meeting scheduled delivery time frames due to delays with another  project that was not ready for testing. In particular, the officials  reported that one of their systems was ready for testing but  experienced delays because the system had an interface with another  system that was not ready for testing. The officials said that the delay  in Commerce’s ability to test its system resulted in missed delivery  milestones. In order to continue development, the project team  separately tested its system without including the interface  functionality.

NRC officials reported that they had experienced delays in meeting  their incremental projects’ delivery schedules due to dependencies on  multiple complex projects. These officials told us that the agency  addressed these delays by improving existing processes and  implementing a change control board and an enterprise test  development environment.

SBA officials reported that delays were introduced when the agency  did not have necessary software and hardware available for  development activities. Officials noted that these challenges were a  result of the agency not maintaining an updated inventory of approved  software and developers not having access to laptops needed for  development activities. SBA officials stated that the agency addressed  the lack of approved software and equipment needed for incremental  development by processing a waiver to use software tools and  procuring laptops for the developers.

Incremental development work was slowed by inefficient  governance and oversight processes. Officials from the Office of the  CIO at six agencies (DHS, HUD, NRC, State, USAID, and USDA)  reported that they had experienced challenges in developing projects  incrementally because they were required to follow agency processes  that were lengthy, inefficient, or not easily adaptable to a more rapid  incremental delivery release schedule. Agency officials also noted that a  lack of understanding among project staff regarding the benefits of  incremental development was a challenge. The officials reported  implementing new guidance and management processes to overcome  these challenges. For example:

DHS officials reported that inefficient governance and oversight  processes had caused delays in obtaining necessary approvals for  moving projects forward. Specifically, these officials reported that the  agency’s acquisition lifecycle framework did not allow for tailoring any  of its processes to accommodate Agile development. The officials  noted that these challenges were addressed with the publication of  updated lifecycle documents that incorporated incremental  development guidance into the agency’s policies and procedures.

HUD officials reported that the agency’s internal approval process for  the Privacy Act System of Record Notice did not accommodate  incremental releases. Specifically, the agency’s incremental  development process called for the release of functionality every 60  days, but the agency’s Privacy Office required 90 to 180 days to  complete its approval process. HUD officials reported that the Office  of the CIO is collaborating with the Privacy Office to expedite the  existing approval process, and have proposed that a single system of  record notice be prepared for each incremental development project,  rather than one for each release.

USAID officials reported that the time needed for defining and  incorporating changes in response to IT security and privacy  standards, processes, and artifacts provided before the system is  granted an Authority to Operate is a challenge. These officials  stated that the Office of the CIO has acquired additional  knowledgeable staff to support projects in the incorporation and  execution of security and privacy requirements.

State officials reported that applying incremental development  principles to projects has been a challenge because agency  personnel have lacked a clear understanding of the benefits of  incremental development and how to apply incremental concepts to  unique project types. These officials reported that the agency was  updating its guidance and processes to place greater emphasis on the  importance of incremental development, and that the agency had  established a review process to ensure projects plan for implementing  incremental development.

Project characteristics made rapid delivery of functionality  infeasible or impracticable. Officials from the Office of the CIO at six  agencies (Interior, Justice, Labor, SSA, Transportation, and Treasury)  reported that they believed rapid delivery of functionality was infeasible or  impracticable for projects that addressed human health and safety  concerns, had legislative mandates that established immovable delivery  time frames, were primarily for infrastructure deployment, were updates to  existing systems to address legal or other regulatory changes, or were  updates to legacy systems that utilize old programming languages.  However, none of the agencies identified solutions for these challenges  that enabled them to deliver functionality in the 6-month time frames  required by OMB. For example:

Transportation officials noted that Federal Aviation Administration  projects, like those for its Next Generation Air Transportation  System, are unique and complex due to safety concerns that impact  the national airspace. As a result, these investments require years of  design, development, and testing, which officials believe precludes  using incremental approaches that must deliver usable functionality  every 6 months.

Labor officials reported that certain projects, which are initiated in  response to an executive order or other external mandate, come with  required delivery time frames. This results in relatively short  development schedules that do not lend themselves to using an  incremental approach.

Justice officials reported that several of the agency’s investments  primarily dealt with the deployment of secure telecommunications,  data centers, and other network infrastructure, making it difficult to  translate that delivery into meaningful increments. Justice officials  stated that they did not deploy incremental development because the  projects were infrastructure projects.

Treasury officials reported that the development and maintenance of  some major investments, such as the agency’s legacy tax systems,  are not conducive to a 6-month delivery schedule due to the number  of modifications that must be made based on changes to the tax laws,  legislative mandates, and other system updates. Treasury officials  stated that the agency has established a mature governance process  for rolling out changes to these tax systems so there is only one  annual update to the systems.

SSA officials stated that using an incremental software development  approach to modernize the agency’s legacy applications was  challenging because the code for these applications was  unstructured, overly complex, heavily interdependent, and utilized old  programming languages. The officials stated that, in order to  modernize these legacy applications, the project teams had to break  programming changes into useful segments, streamline imbedded  business process requirements, and rewrite the code using modern  programming languages. As a result, the officials stated that these  activities could not, at least initially, deliver functionality in smaller  increments.

Incremental development schedules were impeded by procurement  delays. Officials from the Office of the CIO at five agencies (Education,  HUD, Interior, OPM, and USDA) reported that they had experienced  challenges with meeting incremental development schedules due to  delays in getting contracts awarded or getting contract modifications  approved. To overcome this challenge, agency officials reported that they  negotiated with vendors and worked with the offices of procurement  within their agencies to reduce delays and ensure all paperwork was  completed in the time frames required. For example:

Education officials reported that the agency uses contractors to  perform most of its software development work. These officials stated  that modifying existing contracts to require the use of incremental  development approaches had caused delays in getting vendors to  deliver functionality in 6-month increments. Education officials  reported that they had negotiated with vendors to restructure delivery  schedules in order to meet incremental delivery time frames.

HUD officials reported that they had faced challenges in meeting  project schedules due to delays in getting paperwork approved by the  agency’s procurement office, which was busy with end-of-year  activities. To address this, HUD officials stated that they collaborated  with the Office of the Chief Procurement Officer to ensure the project  teams submitted the required documentation for approval in advance  of the procurement office’s end-of-year activities.

OPM officials noted that they had faced challenges with adapting their  procurement process to use incremental approaches. The officials  stated that they worked with their Office of Procurement to incorporate  incremental development procurement methodologies in order to  reduce the time from contract initiation to award, as well as to reduce  the amount of contract documentation and its complexity.

Programs did not have stable, prioritized requirements. Officials from  the Office of the CIO at five agencies (DHS, Justice, NSF, Transportation,  and VA) reported that maintaining stable requirements, including defining  a set of initial requirements, handling ongoing changes, and managing  stakeholder expectations regarding the scope of, and number of changes  to requirements, were challenges. To overcome these challenges,  agencies reported strengthening standards, implementing training and  coaching, and exercising better requirements and business practices. For  example:

DHS officials stated that managing stakeholder expectations related  to requirements was challenging because product owners and  business users expected project requirements not to change once  they were developed, while development teams had planned for  requirements to change and be reprioritized over the course of the  project since the team was using an incremental approach. These  officials reported that they issued new guidance and offered  assistance and coaching for programs and projects to better identify  and document needs and requirements, while encouraging business  users to plan for and prioritize the backlog of items to be deployed  incrementally.

Justice officials reported that it was a challenge to finalize the scope  of work for various projects because disparate stakeholders had  competing priorities which led to constant changes in the  requirements. The officials noted that, for one of the agency’s  projects, the project team is currently establishing a process to obtain  consensus on stakeholder priorities in advance. For other projects,  Justice teams have sought or received training from experienced,  certified Agile experts in developing customer requirements.

NSF officials reported that, when first establishing its incremental  development program, the agency had experienced challenges in  defining a stable set of priority requirements for the initial increments.  The officials told us that, to address this challenge, they elevated  customers to fill the leadership roles of the working groups that  provided the requirements to ensure the requirements of each  increment were well defined and clearly prioritized.

VA officials reported that, while the agency has transitioned to Agile  development methods over the past several months, it still works  through challenges in developing detailed user stories with its  business partners, and reported many instances when a project was  undertaken without knowing the full scope of requirements. VA  officials reported that they took several actions to help address this  challenge, including introducing a new development methodology to  promote incremental development principles, and establishing an  account management office that works with business partners to  ensure detailed business cases are prepared prior to approval. They  also integrated more rapid prototyping into the planning stages as a  way to gather requirements and test assumptions early and cheaply.

Organizational changes associated with the transition from a  traditional software methodology to an incremental development  methodology require time and resources. Officials from the Office of  the CIO at three agencies (EPA, GSA, and Labor) independently reported  challenges related to organizational changes, such as staff adapting to  the culture shift from being business customers to taking on a more active  role as product owners and project managers in the software  development process. For example:

EPA officials stated that the agency had experienced challenges as  staff transitioned from using waterfall development practices to Agile  practices because there had been skepticism within the agency on  whether an Agile approach could meet the requirements for agency  systems. The officials stated that the CIO had established an office to  provide support to project teams that needed assistance in adopting  Agile approaches, created a community of practice group, and  developed guides and other maturity models to provide guidance on  the adoption of Agile methodologies.

GSA officials explained that implementing incremental delivery has  required a culture shift for the agency’s business customers who were  accustomed to having a different set of roles and responsibilities in  the traditional software development process than what is used in the  incremental development process. The officials stated that they have  worked to train their customers to better capture the vision of what  needs to be built and to be more active product owners and managers  in communicating with the development team. As a result, the officials  in the GSA Office of the CIO stated that they are enabling the  business customers to serve as better product owners. The officials  further stated that, by implementing this change, project staffs have  (1) defined and prioritized clearer requirements; (2) selected the  proper technical tools to support business needs; (3) worked with the  contracting office to develop better-defined contracting documents  and make contract awards; (4) identified dependencies associated  with development efforts; and (5) provided transparency on what work  has been completed, what work is planned, and the challenges  associated with the investments.

Additionally, three agencies (Defense, Energy, and HHS) reported no  challenges with implementing incremental development. However,  officials from all three agencies discussed issues surrounding the use of  incremental development, both as part of this review and as part of our  prior work. In particular, Energy officials had told us that they had  projects that failed to adequately employ incremental development  practices, which required follow-up with program managers to identify  corrective actions. Also, both Defense and HHS officials have reported  facing management and organizational challenges, such as  dependencies on integrating changes with other systems, which impacted  the delivery of functionality every 6 months. Defense officials noted that  many of the agency’s investments were complex and could not adhere to  a 6-month delivery schedule.

Federal investments may continue to encounter increased cost and  schedule risks if agencies cannot adequately implement incremental  development approaches. The discussion of challenges identified in this  report—and the range of actions taken by the agencies to address  them— is a valuable resource that could have the potential to help  agencies that face similar concerns.

Agencies Reported Using Information from the Incremental Certification Process to Improve Investment Management Oversight

Although a number of agencies identified challenges in utilizing  incremental development, officials in the Office of the CIO at 21 of the 24  agencies also reported that the CIO certification process was beneficial  to their agencies because it had assisted them in overseeing the  management of agency investments. For example, officials from 13  agencies reported that they used the information derived from the  certification process to identify challenged development projects that  could be using a more effective incremental development approach and  officials from 2 agencies stated that the information helped them  determine whether an investment should undergo a TechStat review.  Table 3 lists the four benefits reported by federal agencies in utilizing the  CIO certification process and the number of agencies that reported each  activity, ranked by number of agencies reporting the challenge. Examples  of the benefits agency officials identified from these investment  management oversight activities are discussed following the table.

More effective use of incremental development approaches. Officials  from the Office of the CIO at 13 agencies (Defense, DHS, Education,  Energy, EPA, GSA, Interior, NASA, NRC, SBA, SSA, Transportation, and  USDA) stated that they review the information about the investment’s use  of incremental development to identify projects that could be  implementing a more effective incremental development approach. For  example, Energy, GSA, and SBA officials stated that they review projects  not using adequate incremental development in order to identify  necessary corrective actions, such as: (1) breaking out projects into  shorter duration activities; (2) implementing the use of investment  reviews, whereby funds are released incrementally upon completion of  clear success criteria; (3) developing major IT investment business cases  that outline project plans for incremental development; and (4) monitoring  new and existing investments to ensure delivery of capabilities within  schedule and cost thresholds.

In addition, DHS, NASA, NRC, and SSA officials reported that the CIO  uses the information to make corrections to projects that are not  adequately implementing incremental development through such actions  as the CIO’s office: (1) working with project team officials to convert  project activities to an incremental approach; (2) requiring any deviations  from approved releases of software development products to be  approved by the CIO; (3) requiring projects that deviate from the use of  adequate incremental development principles to be approved by the CIO;  and (4) determining which investments must use incremental  development, and requiring the projects to do so.

Provide oversight of IT investments. Officials from the Office of the  CIO at seven agencies (Commerce, Interior, Labor, OPM, NSF, State,  and VA) stated that they use the information to provide oversight of IT  investments. In particular, Interior and NSF officials reported that their  CIOs use the information obtained during the performance measurement  baseline approval process to make decisions regarding the agency’s  major IT investments. For Interior, officials stated that the types of  decisions the CIO may make include, but are not limited to, accelerating  delivery, reducing scope, or halting or terminating an IT project. For NSF,  officials stated that the decisions could result in changes to program  objectives or scope of individual projects under the program, redirection  of resources, changes to planned levels of expenditure, or  recommendations for corrective actions based on the evaluation.

In addition, Commerce officials stated that investment data are reviewed  by the CIO on a monthly basis and, based on the status, can undergo  further scrutiny at a review board meeting or other CIO review process.  Labor officials noted that its capital planning team updates the CIO’s  rating and explanation for each major IT investment in the agency’s  capital planning and investment control system, and submits the rating  information to the IT Dashboard each month.

Improve incremental development processes. Officials from the Office  of the CIO at five agencies (DHS, EPA, HUD, Justice, and USDA) stated  that they leveraged the information to improve their incremental  development processes. For instance, USDA officials reported that they  leveraged the results of the certification process to build an incremental  development community of practice. DHS officials stated that they  developed coaching and other assistance to help convert projects to an  incremental process. Lastly, Justice officials stated that they utilized the  results of the certification process to: (1) develop best practices and  lessons learned on using incremental development, (2) establish  additional training, and (3) establish mentoring programs or other  familiarization with incremental techniques to support business  improvement.

Determine if a TechStat is warranted. Officials from the Office of the  CIO at two agencies (Labor and SBA) stated that they use the results of  the certification process to determine whether an investment should  undergo a TechStat review. In particular, Labor officials stated that if an  investment is rated as high risk for 3 consecutive months during the  review process, then a TechStat is initiated. In addition, SBA officials  noted that, as part of their certification process, the Office of the CIO  portfolio management team meets with the CIO to determine if any IT  investments should have a Techstat review.

Given the significant size of the federal government’s annual investment  in IT and the often disappointing results from IT development efforts,  finding innovative ways to improve the quality and timeliness of agencies’  IT investments may help improve these development efforts. The  discussion of benefits identified with using the certification process—and  the range of management oversight activities taken by the agencies— may have the potential to help agencies improve their management and  oversight of IT acquisitions.

Most Agencies Lack Detailed CIO Certification Policies and OMB Has Improved Related Reporting Guidance

Of the 24 agencies in our review, only 4 had clearly defined processes  and policies to ensure that the CIO will certify that major IT investments  are adequately implementing incremental development. The remaining 20  agencies either did not include details such as the role of the CIO in the  certification process or how certification would be documented, or had not  yet finalized a policy.

OMB’s fiscal year 2018 guidance was not clear regarding what actions  agencies should take to demonstrate compliance with FITARA’s  certification requirement. However, OMB issued its new fiscal year 2019  guidance in August 2017, which addressed the weaknesses we identified.

Only 4 of 24 Agencies Have Clearly Defined a Policy for CIO Certification of Incremental Development

A provision in FITARA, enacted in December 2014, states that, in its  annual IT capital planning guidance, OMB is to require agency CIOs to  certify that IT investments are adequately implementing incremental  development. Subsequent OMB guidance on the law’s implementation,  issued in June 2015, directed agency CIOs to define processes and  policies for their agencies which ensure that they certify that IT resources  are adequately implementing incremental development. As part of the  guidance, OMB defined adequate incremental development as the  development of software or services, with planned or actual delivery of  new or modified technical functionality to users that occurs at least every  6 months.

OMB’s guidance allows agencies the flexibility to define the processes  that CIOs use for ensuring the certification of adequate incremental  development. For example, CIOs can rely on internal governance  processes, such as investment and capital planning processes, to  evaluate agency investments for adequate use of incremental  development. In addition, agency CIOs are to use OMB’s definition of  adequate incremental development when developing their certification  processes and determining whether to certify that their investments met  these criteria. While OMB’s guidance is not specific on what elements should be included in these certification policies and processes, GAO’s  Information Technology Investment Management framework notes that  policies and procedures should be clearly defined, including the role of  appropriate stakeholders, and have appropriate artifacts to document  decisions made.

Although OMB’s requirement has been in place since June 2015, only 4  of the 24 agencies we reviewed (Commerce, DHS, Energy, and  Transportation) have clearly defined processes and policies intended to  ensure that their CIOs certify that major IT investments are adequately  implementing incremental development. Specifically, all 4 agencies’  policies contained all the elements that we evaluated in the agency  guidance: descriptions of the role of the CIO in the process; how the  CIO’s certification will be documented; and definitions of incremental  development and time frames for delivering functionality consistent with  OMB guidance.

However, the remaining 20 agencies did not have clearly defined  processes and policies in place because their documentation either did  not describe the CIOs’ role in the certification process or how certification  would be documented, define incremental development and provide  delivery time frames consistent with OMB guidance; or the policy had not  yet been finalized. The results of our analysis of agencies’ policies is  shown in figure 2, while additional details regarding the status of the 24  agencies’ incremental policies are provided in appendix III.

The four agencies that had clearly defined policies for certification took a  variety of approaches to defining how the CIOs would conduct the review  and certification of major IT investments, determining how certification  would be documented, and ensuring OMB’s guidance regarding the  definition of adequate incremental development and delivery time frames  was followed. Specifically:

Commerce’s capital planning guidance requires bureau CIOs or other  accountable officials to review project documentation regarding  project deliverables and issue an e-mail or other time-stamped  document that certifies the adequate implementation of incremental  development. In addition, Commerce guidance adheres to OMB’s  guidance requiring delivery time frames every 6 months or less and  sets forth a definition of adequate incremental development that is  consistent with OMB guidance.

DHS’s technical investment review guidance states that the CIO is to  conduct a review of each investment using an investment review  checklist that includes information provided by project managers as to  whether the investments have used incremental development  adequately. The CIO is to certify whether the project is implementing  incremental delivery at least every 6 months and document this  certification in the checklist. DHS guidance also includes a definition  of adequate incremental development and time frames for delivering  functionality that are consistent with OMB guidance.

Energy’s capital planning guidance states that the CIO is to review  and certify each investment’s adequate use of incremental  development as part of monthly investment review board meetings  and during the monthly review of the IT Dashboard data. The status of  this certification is documented in the agency’s monthly investment  summary spreadsheet. In addition, Energy’s guidance adheres to  OMB’s definition of adequate incremental development and its  associated delivery time frames for its incremental development  activities.

Transportation’s investment management guidance states that the  CIO is to conduct a review of the investment as part of the investment  review board process; this board is co-chaired by the agency CIO.  The CIO is to certify adequate incremental development in the signed  investment decision review document. In addition, Transportation’s  guidance adheres to OMB’s definition of adequate incremental  development and delivery time frames.

However, the remaining 20 agencies did not have clearly defined policies  and processes in place to ensure CIOs are certifying each major IT  investment’s adequate incremental development. In particular, while  officials from the Office of the CIO at 11 agencies asserted that they had  a policy for CIO certification, these policies lacked details, such as a  description of the role of the CIO in the process, a description of how  certification would be documented, and definitions of incremental  development and delivery time frames consistent with OMB guidance.  Table 4 details our evaluation of the certification policies provided to us by  the 11 agencies.

Agency officials in the Office of the CIO at each of the 11 agencies  provided a variety of reasons for why their policies lacked details  regarding the role of the CIO in the process and how certification was  documented, or did not include definitions for incremental development  and delivery time frames. For example, State officials reported that  updating their policies to comply with FITARA was not seen as a priority  until Congress conducted its own evaluation of incremental development  in May 2016. They stated that their new policy is currently in the process  of being finalized but no time frames for finalization were provided.  However, we could not determine whether the guidance is expected to  address the issues we identified because State provided us excerpts of  its new draft policy and the new proposed guidance that did not include  any details in the areas we identified.

In addition, GSA officials stated that they had used existing governance  bodies and processes to determine whether the investment would be  certified. The officials stated that they did not see a reason to create a  separate policy for CIO certification, since the agency always looks at  using incremental development for new projects and the agency certifies  the investment in the major IT business case.

Further, OPM officials stated that their agency had been on a path to  address the FITARA requirements, but progress was slowed due to the  lack of a budget for fiscal year 2017. The officials stated that they intend  to update the agency’s policies, but had no firm plans for doing so  pending the availability of budgetary resources.

Lastly, NSF officials stated that they have not seen the need to have a  policy on CIO certification for a number of reasons. NSF reported that it is  a small agency with few large IT investments, and many of those are  legacy systems in operations and maintenance, rather than development.  Therefore, according to the officials, the agency has not had many  occasions for the CIO to need to certify adequate incremental  development for major IT investments. Second, the officials stated that  the NSF CIO is actively involved in the investment review process and did  not feel a policy was needed to describe these activities. Third, NSF  officials stated that it is their belief that policies are generally only required  to correct something which is not working. Lastly, NSF officials stated that  the agency’s definition of an Agile sprint was its definition of incremental  development. However, sprints are not released directly to users, and  therefore, the definition is not consistent with OMB guidance. However,  the officials said they might reconsider developing a policy, but did not  provide a time frame for doing so.

Finally, 9 agencies had not yet finalized a CIO certification policy. Office  of the CIO officials in each of these agencies reported that they had relied  on existing IT governance processes and budget mechanisms, or created  new targeted IT reviews to determine the CIO certification for fiscal year  2017 that was reported on the IT Dashboard. For example, HHS officials  reported that the agency used existing project and investment milestone  reviews as part of its enterprise performance lifecycle to determine  whether the investment would be certified as having adequate  incremental development.

SBA officials told us that the agency’s portfolio management team met  with investment managers during the monthly update process for the IT  Dashboard, while USAID officials noted that the agency’s CIO reviews the  incremental development status of all major investment software  development projects on a monthly basis. Further, Justice officials  reported that the IT Investment Oversight Manager’s staff reviewed the  major business cases and requested justification for software  development investments that were not: (1) using an iterative or Agile  methodology, (2) expected to have a production release containing  usable functionality every 6 months, or (3) showing an actual or planned  date for deployment production within a 6-month time frame.

In addition, while six of these agencies reported plans to finalize a policy  for CIO certification by December 2017, one agency reported its policy  would be finalized in 2018, and two agencies did not provide a time frame  for finalizing a policy. Figure 3 below shows the agencies’ reported time  frames for finalizing a policy on CIO certification of incremental  development.

Officials from each agency’s Office of the CIO provided a variety of  reasons for why they had not yet developed or finalized policies for CIO  certification of adequate incremental development. For example, EPA  officials stated that the agency has been focusing on standing up the  programs and structures needed to support incremental development  and, thus, had not prioritized developing a policy. In addition, EPA  officials stated that they had not developed a definition of functionality or  time frames, but that their guidance points to industry standards.

SBA officials stated that, since the majority of the agency’s investments  were in operations and maintenance, they did not see the need to have  policies or procedures for incremental development. In addition, HUD,  NASA, and USAID officials reported that their agencies were in the  process of finalizing policies, but had experienced delays due to the  number of stakeholder comments or limited staff resources.

Lastly, Defense officials stated that they had included information in their  fiscal year 2018 budget submission guidance for component CIOs to  certify adequate incremental development and were working to  incorporate this process into their Financial Management Regulations,  which were to be finalized in the first quarter of fiscal year 2018. However,  the officials stated that the agency’s process is driven by its efforts to  comply with whatever process OMB requires in the annual capital  planning guidance and, thus, they would not have a separate certification  policy from the budget guidance.

Additionally, Defense officials reported that, for their agency’s  investments, delivery every 12 to 18 months was more appropriate than  the 6 months that OMB requires. Nevertheless, while Defense officials  may believe that 12 to 18 month delivery cycles may be more appropriate  for their work, OMB’s guidance requires agencies to deliver functionality  at least every 6 months and does not allow for exceptions. We previously  recommended that Defense establish a policy on the CIO certification of  incremental development. Until this guidance is finalized, Defense may  not be able to ensure incremental development practices are adequately  implemented at the agency. We therefore continue to believe the  recommendation is appropriate.

Annual CIO certification of incremental development is critical to ensuring  that agency CIOs exercise the proper authority and oversight over their  agencies’ major IT investments. Having appropriate authority and  oversight helps to create IT systems that add value and are aligned with  agencies’ missions, while reducing the risks associated with low-value  and wasteful investments. In the absence of clearly defined policies,  agencies continue to run the risk of failing to deliver major investments in  a cost-effective and efficient manner.

We have previously made recommendations to Defense, Education,  HHS, and Treasury to establish CIO certification policies, but as noted in  this report, these agencies still have not yet finalized their guidance to  clearly detail their agencies’ processes for certification. Therefore, we  continue to believe these recommendations are appropriate.

Agencies that lacked finalized policies may not be able to meet their  reported time frames for finalizing their certification policies, since agency  officials have noted that their approval processes are quite lengthy, and in  some cases, the proposed dates for completion have changed several  times. In addition, several policies were still being developed. Therefore,  we cannot be assured that these documents will fully address the areas  we noted. Until the 20 agencies update or finalize processes and policies  for CIO certification, including defining the role of the CIO in the process,  describing how certification will be documented, and including definitions  of incremental development and delivery time frames consistent with  OMB guidance, they will not be able to fully ensure adequate  implementation of, or benefit from, incremental development practices. As  a result, the agencies increase the risk that federal government resources  will not be used in the most effective and efficient manner.

OMB Has Improved Its IT Capital Planning Guidance to Ensure CIO Certification Reporting Clearly Specifies Agency Responsibilities

FITARA states that OMB is to require in an agency’s annual IT capital  planning guidance that each covered agency CIO certify that IT  investments are adequately implementing incremental development, as  defined in capital planning guidance issued by OMB. However, since  the law was enacted in December 2014, OMB has taken three different  approaches to address this reporting requirement. Of the approaches,  one did not clearly and consistently provide agencies with the direction  needed to effectively implement this important provision and report the  status of certification.

As previously noted, OMB’s fiscal year 2017 IT capital planning  guidance (issued in June 2015) required each major IT investment to  respond to a question in the associated major IT business case regarding  whether the CIO certified the adequate implementation of incremental  development with either a yes, no, or not applicable. This reporting  approach required that agency CIOs provide an explicit statement  regarding the certification of adequate implementation of incremental  development for each major IT investment. Further, this approach allowed  for the status of CIO certification of each investment to be publicly  reported on the IT Dashboard via the investment’s major IT business  case.

However, OMB’s capital planning guidance for fiscal year 2018 (issued  in June 2016) lacked clarity regarding how agencies were to address the  requirement certifying adequate incremental development. While the  2018 guidance states that agency CIOs are to provide the certifications  needed to demonstrate compliance with FITARA, there is no specific  reference to the provision requiring CIO certification of adequate  incremental development. As a result of this change, OMB placed the  burden on agencies to know and understand how to demonstrate  compliance with FITARA’s incremental development provision. Further,  because of the lack of clarity in the guidance as to what agencies were to  provide, OMB could not demonstrate how the fiscal year 2018 guidance  ensured that agencies provided the certifications specifically called for in  the law.

OMB staff explained that the changes to the fiscal year 2018 capital  planning guidance were made with the intent to rely on agencies’ reported  responses on the IT Dashboard regarding the use of incremental  development by an investment’s projects, rather than relying on an  agency’s response to the yes, no, or not applicable question about the  status of an investment’s certification of incremental development.

Providing a clear and consistent approach for agencies to follow in  reporting the status of certification is critical to ensure that agencies are  able to comply with this key FITARA provision and to ensure that CIOs  are held accountable for the performance of their major IT investments.  OMB staff from the Office of E-Government and Information Technology  stated that the fiscal year 2019 guidance would be responsive to the  issues we raised.

Accordingly, in August 2017, OMB issued its fiscal year 2019 guidance,  which addressed the weaknesses we identified in the previous fiscal  year’s guidance. Specifically, the revised guidance requires agency CIOs  to make an explicit statement regarding the extent to which the CIO is  able to certify the use of incremental development, and to include a copy  of that statement in the agency’s public congressional budget justification  materials. As part of the statement, an agency CIO must also identify  which specific bureaus or offices are using incremental development on  all of their investments.

Agency CIO certification of the use of adequate incremental development  for major IT investments is critical to ensuring that agencies are making  the best effort possible to create IT systems that add value while reducing  the risks associated with low-value and wasteful investments. These  changes to OMB’s fiscal year 2019 guidance provide a key improvement  for ensuring that agency CIOs have a consistent approach to follow in  providing the certifications specifically called for in the law.

One of the aims of FITARA was to encourage the use of incremental  development throughout the federal government and, as of August 2016,  more than half of the 24 agencies’ IT investments had been certified as  adequately implementing incremental development, as required by  FITARA and defined in OMB guidance. However, a number of responses  for agency investments were incorrectly reported and it will be critical that  agencies continue to improve the accuracy of investment data reported  on the IT Dashboard. While we have previously made recommendations  to numerous agencies to improve the accuracy of reporting on the IT  Dashboard, issues with reporting remain, reinforcing the need for  agencies to ensure that accurate data are made available for the  oversight and management of their investments.

In addition, while OMB issued guidance in June 2015, requiring agency  CIOs to define policies and processes for CIO certification, as of August  2017, only 4 of 24 agencies had established policies that clearly define  these processes. At this point, over 2 years since the law’s enactment, it  is critical that agencies take action to put in place appropriate incremental  certification polices to ensure CIOs exercise the proper authority and  oversight over major IT investments, as required by law. Otherwise,  agencies run the risk of not realizing the benefits of incremental  development, as well as not implementing FITARA’s requirement for  incremental development. While we previously made recommendations to  Defense, Education, HHS, and Treasury to establish CIO certification  policies, these agencies have still not yet finalized their guidance, and  therefore, we continue to believe these recommendations are  appropriate.

Further, OMB has taken three different approaches to addressing  FITARA’s reporting requirement for CIO certification and one did not  clearly and consistently provide agencies with the direction needed to  effectively implement this important provision and report the status of  certification. OMB’s fiscal year 2017 capital planning guidance was  helpful to agencies, in that it clearly directed agencies on how to publicly  report their certifications. This also helped Congress in its oversight of  agencies’ FITARA compliance. In contrast, OMB’s fiscal year 2018 capital  planning guidance was a step backward, and OMB could not demonstrate  how the guidance ensured that agencies provided the certifications  specifically called for in the law. Going forward, the changes in guidance  that OMB has implemented for fiscal year 2019 recognize the importance  of providing clear direction to CIOs and how critical it is for agencies to  create IT systems that add value while reducing the risks associated with  low-value and wasteful investments.

We are making a total of 19 recommendations to 17 departments and  agencies in our review. Specifically:  The Secretary of Energy should ensure that the CIO of Energy reports  major IT investment information related to incremental development  accurately in accordance with OMB guidance. (Recommendation 1)

The Secretary of Agriculture should ensure that the CIO of USDA reports  major IT investment information related to incremental development  accurately in accordance with OMB guidance. (Recommendation 2)

The Commissioner of the Social Security Administration should ensure  that the CIO of SSA reports major IT investment information related to  incremental development accurately in accordance with OMB guidance.  (Recommendation 3)

The Secretary of Housing and Urban Development should ensure that the  CIO of HUD establishes an agency-wide policy and process for the CIO’s  certification of major IT investments’ adequate use of incremental  development, in accordance with OMB’s guidance on the implementation  of FITARA, and confirm that it includes: a description of the CIO’s role in  the certification process; a description of how CIO certification will be  documented; and a definition of incremental development and time  frames for delivering functionality, consistent with OMB guidance.  (Recommendation 4)

The Secretary of the Interior should ensure that the CIO of Interior  updates the agency’s policy and process for the CIO’s certification of  major IT investments’ adequate use of incremental development, in  accordance with OMB’s guidance on the implementation of FITARA, and  confirm that it includes: a description of the CIO’s role in the certification  process; a description of how CIO certification will be documented; and a  definition of incremental development, consistent with OMB guidance.  (Recommendation 5)

The Attorney General of the United States should ensure that the CIO of  Justice establishes an agency-wide policy and process for the CIO’s  certification of major IT investments’ adequate use of incremental  development, in accordance with OMB’s guidance on the implementation  of FITARA, and confirm that it includes: a description of the CIO’s role in  the certification process; a description of how CIO certification will be  documented; and a definition of incremental development and time  frames for delivering functionality, consistent with OMB guidance.  (Recommendation 6)

The Secretary of Labor should ensure that the CIO of Labor updates the  agency’s policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes a description of the CIO’s role in the certification process and a  description of how CIO certification will be documented.  (Recommendation 7)

The Secretary of State should ensure that the CIO of State updates the  agency’s policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes: a description of the CIO’s role in the certification process; a  description of how CIO certification will be documented; and a definition  of incremental development and time frames for delivering functionality,  consistent with OMB guidance. (Recommendation 8)

The Secretary of Agriculture should ensure that the CIO of USDA  establishes an agency-wide policy and process for the CIO’s certification  of major IT investments’ adequate use of incremental development, in  accordance with OMB’s guidance on the implementation of FITARA, and  confirm that it includes: a description of the CIO’s role in the certification  process; a description of how CIO certification will be documented; and a  definition of incremental development and time frames for delivering  functionality, consistent with OMB guidance. (Recommendation 9)

The Secretary of Veterans Affairs should ensure that the CIO of VA  updates the agency’s policy and process for the CIO’s certification of  major IT investments’ adequate use of incremental development, in  accordance with OMB’s guidance on the implementation of FITARA, and  confirm that it includes a description of the CIO’s role in the certification  process and a description of how CIO certification will be documented.  (Recommendation 10)

The Administrator of EPA should ensure that the CIO of EPA establishes  an agency-wide policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes: a description of the CIO’s role in the certification process; a  description of how CIO certification will be documented; and a definition  of incremental development and time frames for delivering functionality,  consistent with OMB guidance. (Recommendation 11)

The Administrator of GSA should ensure that the CIO of GSA updates the  agency’s policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes a description of the CIO’s role in the certification process and a  description of how CIO certification will be documented.  (Recommendation 12)

The Administrator of NASA should ensure that the CIO of NASA  establishes an agency-wide policy and process for the CIO’s certification  of major IT investments’ adequate use of incremental development, in  accordance with OMB’s guidance on the implementation of FITARA, and  confirm that it includes: a description of the CIO’s role in the certification  process; a description of how CIO certification will be documented; and a  definition of incremental development and time frames for delivering  functionality, consistent with OMB guidance. (Recommendation 13)

The Director of the NSF should ensure that the CIO of NSF updates the  agency’s policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes: a description of the CIO’s role in the certification process; a  description of how CIO certification will be documented; and a definition  of incremental development and time frames for delivering functionality,  consistent with OMB guidance. (Recommendation 14)

The Chairman of NRC should ensure that the CIO of NRC establishes an  agency-wide policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes a description of the CIO’s role in the certification process and a  description of how CIO certification will be documented.  (Recommendation 15)

The Director of OPM should ensure that the CIO of OPM updates the  agency’s policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes a description of the CIO’s role in the certification process and a  description of how CIO certification will be documented.  (Recommendation 16)

The Administrator of SBA should ensure that the CIO of SBA establishes  an agency-wide policy and process for the CIO’s certification of major IT  investments’ adequate use of incremental development, in accordance  with OMB’s guidance on the implementation of FITARA, and confirm that  it includes: a description of the CIO’s role in the certification process; a  description of how CIO certification will be documented; and a definition  of incremental development and time frames for delivering functionality,  consistent with OMB guidance. (Recommendation 17)

The Commissioner of the Social Security Administration should ensure  that the CIO of SSA updates the agency’s policy and process for the  CIO’s certification of major IT investments’ adequate use of incremental  development, in accordance with OMB’s guidance on the implementation  of FITARA, and confirm that it includes a description of the CIO’s role in  the certification process and a description of how CIO certification will be  documented. (Recommendation 18)

The Administrator of USAID should ensure that the CIO of USAID  establishes an agency-wide policy and process for the CIO’s certification  of major IT investments’ adequate use of incremental development, in  accordance with OMB’s guidance on the implementation of FITARA, and  confirm that it includes: a description of the CIO’s role in the certification  process; a description of how CIO certification will be documented; and a  definition of incremental development and time frames for delivering  functionality, consistent with OMB guidance. (Recommendation 19)

We received comments on a draft of this report from OMB and the 24  agencies that we reviewed. Of the 17 agencies to which we made  recommendations, 11 agencies agreed with our recommendations, 1  agency partially agreed, and 5 agencies did not state whether they  agreed or disagreed with the recommendations.

In addition, of the 7 agencies and OMB to which we did not make  recommendations, 2 agencies agreed with the report and 5 agencies  stated that they had no comments on the report. OMB did not agree with  certain findings in the report. In addition, OMB and multiple agencies  provided technical comments on the report, which we incorporated as  appropriate.

The following discusses the comments received from each agency to  which we made a recommendation.

In written comments, Energy concurred with our recommendation to  ensure that the CIO reports major IT investment information related to  incremental development accurately in accordance with OMB  guidance, and described actions it has taken to address the  recommendation. Specifically, the agency stated that its Office of the  CIO reviews the accuracy of Energy’s major IT investment project  reporting related to incremental development as part of monthly IT  Dashboard and Investment Review Board meetings. By taking these  actions, the agency considered the recommendation closed.

As noted earlier in our report, we identified issues with the accuracy of  Energy’s reported data related to the certification of incremental  development. If Energy consistently and effectively implements its  reviews of IT Dashboard data, as described, these actions should  help to improve the accuracy of reported incremental development  data on the IT Dashboard. We plan to continue to monitor the  agency’s reporting of its incremental data on the IT Dashboard and  accordingly, consider our recommendation to currently remain open.  Energy’s comments are reprinted in appendix IV.

In written comments, HUD concurred with our recommendation to  establish an agency-wide policy and process for CIO certification of  adequate incremental development and stated that it would provide  more definitive information and timelines on how it plans to address  the recommendation once our final report is issued. HUD’s comments  are reprinted in appendix V.

In written comments, Interior stated that the agency concurred with  our recommendation to update the agency’s policy and process for  CIO certification of adequate incremental development and described  planned actions to implement it. Specifically, the agency reported that  it is committed to updating its existing policy to include a description of  the CIO’s role in the incremental development certification process, a  description of how the CIO’s certification is documented, and a  definition of incremental development, consistent with OMB’s  guidance. Interior’s comments are reprinted in appendix VI.

In an e-mail received on September 15, 2017, an audit liaison  specialist in Justice’s Audit Liaison Group in the Internal Review and  Evaluation Office stated that the agency agreed with our  recommendation to establish an agency-wide policy and process for  CIO certification of adequate incremental development and described  planned actions to implement it. Specifically, the official stated that  Justice will amend existing policy and processes to implement this  recommendation. In addition, the official stated that Justice is fully  supportive of incremental development and has drafted  documentation, including guidance on an incremental system  development life cycle.

In an e-mail received on September 5, 2017, an administrative officer  in Labor’s Office of the Assistant Secretary for Administration and  Management stated that the agency had no comments on the report.

In written comments, State did not say whether the agency agreed or  disagreed with our recommendation to update the agency’s policy and  process for CIO certification of adequate incremental development,  but described ongoing actions to implement it. Specifically, the agency  reported that it has developed an incremental development policy that  addresses the recommendation we noted in our report. The agency  added that the policy is currently in the process of being approved.  State’s comments are reprinted in appendix VII.

In an e-mail received on September 1, 2017, a senior advisor in the  USDA Office of the CIO’s Enterprise Management office stated that  the agency concurred with our findings and recommendations to  report major IT investment incremental development information  accurately and to establish an agency-wide policy and process for  CIO certification of adequate incremental development, and had no  further comments.

In written comments, VA partially concurred with our recommendation  to update the agency’s policy and process for CIO certification of  adequate incremental development, stating that, while the agency  does not currently have a policy in place outlining the CIO certification  process, the agency CIO does direct that all investments utilize Agile  and incremental delivery. The agency stated that it would take action  to address our recommendation by drafting a policy that outlines the  CIO’s role in the certification process and describes how certification  will be documented. The agency added that the policy is targeted for  completion by November 2017.

If implemented as planned, these actions should address the intent of  our recommendation. VA’s comments are reprinted in appendix VIII.

In written comments, EPA stated that the agency generally agreed  with our recommendation to establish an agency-wide policy and  process for CIO certification of adequate incremental development,  and presentation of facts in the report. The agency also noted that the  policy developed in response to our recommendation is to address  FITARA issues above and beyond the certification of incremental  development.

In addition, the agency noted a technical correction to a sentence in  our report related to EPA’s use of information from certification. We  have incorporated changes to the draft, as appropriate, to address  this comment. EPA’s comments are reprinted in appendix IX.

In written comments, GSA agreed with our recommendation to update  the agency’s policy and process for CIO certification of adequate  incremental development and reported that it would develop and  implement a plan to fully address it. GSA’s comments are reprinted in  appendix X.

In written comments, NASA concurred with the recommendation to  establish an agency-wide policy and process for CIO certification of  adequate incremental development and described ongoing actions to  implement it. Specifically, the agency stated that it is currently  updating its policies to address the incremental development  requirement. In this regard, NASA Policy Directive 2800.1 is to include  a responsibility for the Office of the CIO to certify that IT resources are  adequately implementing incremental development. In addition, NASA  Policy Directive 7120.7 is being updated to include a definition of  incremental development and processes for ensuing that the CIO  certifies incremental development. According to the agency, these  policies are estimated to be completed by March 2018. NASA’s  comments are reprinted in XI.

In an e-mail received on September 14, 2017, a senior advisor in  NSF’s Office of the Director/Office of Integrative Activities stated that  the agency had no comments on our report.

In written comments, NRC stated that it was in general agreement  with the findings in our report. The agency did not state whether it  agreed or disagreed with our recommendation to establish an agency- wide policy and process for CIO certification of adequate incremental  development, but described the planned action to implement the  recommendation. Specifically, the agency reported that it plans to  establish agency-wide, formalized processes and procedures for the  CIO to approve the incremental development of major IT investments  by December 31, 2017. NRC’s comments are reprinted in appendix  XII.

In written comments, OPM concurred with the recommendation to  update the agency’s policy and process for CIO certification of  adequate incremental development and described planned actions to  implement it. Specifically, the agency reported that it intends to update  its policies and processes to include a description of the CIO’s role in  the certification process and a description of how certification will be  documented. OPM’s comments are reprinted in appendix XIII.

In an e-mail received on September 11, 2017, a program manager in  SBA’s Office of Congressional and Legislative Affairs stated that the  agency concurred with our recommendation to establish an agency- wide policy and process for CIO certification of adequate incremental  development, and had no further comments.

In written comments, SSA agreed with our two recommendations to  report major IT investment incremental development information  accurately and establish an agency-wide policy and process for CIO  certification of adequate incremental development, and described  planned actions being taken or planned to implement them.  Specifically, the agency reported that it had implemented two new  processes to support incremental development certification. According  to the agency, each IT investment program manager is to answer a  series of questions about the investment’s status and also certify  whether their investment adequately implements incremental  development. This information is to be used in the CIO’s ongoing  investment evaluation process for reporting investment information on  the IT Dashboard. SSA reported that these new processes are to be  defined in an upcoming revision to the agency’s Capital Planning and  Investment Control Guide. SSA’s comments are reprinted in appendix  XIV.

In written comments, USAID did not state whether it agreed or  disagreed with our recommendation to establish an agency-wide  policy and process for CIO certification of adequate incremental  development, but described ongoing actions to implement the  recommendation. Specifically, the agency reported that it is in the  process of establishing an agency-wide policy and process for the  CIO’s certification of adequate incremental development. It estimates  that this policy will be implemented by August 31, 2018. USAID’s  comments are reprinted in appendix XV.

In addition to the aforementioned comments, the seven agencies and  OMB to which we did not make recommendations provided the following  responses.

In written comments, Commerce stated that the agency concurred  with the report as written. Commerce’s comments are reprinted in  appendix XVI.

In an e-mail received on September 7, 2017, a GAO Affairs staff  member in Defense’s Executive Services Directorate stated that the  agency had no formal comments on the report.

In an e-mail received on September 8, 2017, a staff member in  Education’s Office of the Secretary/Executive Secretariat stated that  the agency had no comments on the report.

In an e-mail received on September 11, 2017, an audit liaison in  HHS’s Office of the Assistant Secretary for Legislation stated that the  agency had no comments on the report.

In an e-mail received on September 11, 2017, a program analyst in  DHS’s GAO-Office of Inspector General’s Liaison Office stated that  the agency would not be sending a management response letter.

In an e-mail received on September 8, 2017, the Director of Audit  Relations and Program Improvement in Transportation’s Office of the  Secretary stated that the agency would not be providing a written  management response.

In an e-mail received on September 15, 2017, a supervisory IT  specialist/GAO-Office of Inspector General liaison in Treasury’s Office  of the CIO stated that the agency generally agreed with the report.  The agency also provided comments related to various challenges  discussed in the report. Specifically, the official described Treasury’s  efforts to address challenges noted in the report related to project staff  lacking the necessary skills for implementing incremental  development practices and programs not receiving sufficient funding.  In this regard, the official stated that the agency continues to develop  knowledge, skills, and abilities for project managers and IT specialists  and continues to provide specialized programming training to its IT  staff in order to move to more modern programming languages and IT  tools as part of system modernization efforts. In addition, the official  stated that, to address challenges related to programs receiving  sufficient funding, Treasury continues to adjust planned and ongoing  projects to align with the availability of funds and external mandates.

In an e-mail received on September 19, 2017, an OMB Assistant  General Counsel stated that the agency generally disagreed with the  tone, tenor, and conclusions of law reflected in aspects of our report.  Among the concerns was that we had asserted that OMB’s prior  year’s guidance to agencies on CIO certification of incremental  development was not in compliance with OMB’s statutory obligations  under FITARA.

As our report states, FITARA mandates OMB to include in its annual  IT capital planning guidance, a requirement that CIOs certify that  investments are adequately implementing incremental development  as defined in the guidance. We reported that OMB had issued  guidance for fiscal years 2017, 2018, and 2019. However, we noted  that the fiscal year 2018 guidance differed from the guidance issued in  the other two fiscal years in that it did not clearly establish how  agency CIOs were to demonstrate compliance with FITARA’s  certification of adequate incremental development provision. Instead,  the fiscal year 2018 guidance placed the burden on agencies to know  and understand how to implement the FITARA requirement.

Thus, while we concluded that OMB’s fiscal year 2018 guidance was  not clear on how agencies were to certify adequate incremental  development, we did not assert that this guidance failed to comply  with FITARA. Accordingly, we did not make a conclusion of law  regarding OMB’s guidance, as the e-mail stated. We continue to  believe that our assessment of the fiscal year 2018 guidance is  correct.

OMB also stated that it disagreed with our conclusion that OMB could  not demonstrate compliance with FITARA.

However, our report did not make the conclusion that is stated in  OMB’s response. As noted above, our report pointed out that OMB’s  fiscal year 2018 guidance lacked clarity in terms of specifically stating  what information agencies were to provide OMB in order to be  compliant with FITARA’s requirement that agency CIOs certify  incremental development. Therefore, we concluded that OMB could  not demonstrate how the fiscal year 2018 guidance ensured that  agencies provided the certifications specifically called for in the law.  As such, we continue to believe that our conclusion is appropriate.

Further, OMB stated that our conclusion was predicated on OMB’s  reluctance to share agency pre-decisional budget information.

It is up to OMB to demonstrate that its fiscal year 2018 guidance  ensured agency compliance with FITARA. Though OMB asserted that  our conclusion was based on OMB’s reluctance to share agency pre- decisional budget information, our conclusion was instead based on  the fact that OMB provided no documentary evidence to establish how  agencies complied with the FITARA certification requirement for fiscal  year 2018. Consequently, we believe our assessment that OMB could  not demonstrate how the fiscal year 2018 guidance ensured that  agencies provided the certifications specifically called for in the law is  accurate.

In a subsequent e-mail to us on October 4, 2017, the OMB Assistant  General Counsel provided additional comments related to the  disagreements described above. Specifically, OMB stated that our  report’s “focus on the use of the term ‘certification’ was confusing in  that  appears to reference the term ‘certify’ [found in the  FITARA provision on the adequate use of incremental development],  and also seems to be a reference to the requirement that CIOs  ‘approve’ and define development processes.”

In our report, we discuss FITARA’s requirement that OMB annually  issue capital planning guidance requiring agency CIOs to certify that  IT investments are adequately implementing incremental  development. We analyzed the guidance that OMB has issued to  meet this requirement over the past 3 years, and we evaluated  agencies’ progress in implementing that guidance. In doing so, we  noted that OMB had also issued supplementary FITARA  implementation guidance in June 2015 that required agencies to  define policies and processes to ensure that the CIO certifies that IT  resources are adequately implementing incremental development.  Throughout our discussion, we clearly delineate between the  incremental development certification provided to OMB by an  agency’s CIO and the agency’s policies and processes that support  and inform that certification. As such, we believe we have used the  term “certification” appropriately and consistently throughout our  report.

We are sending copies of this report to interested congressional  committees, the Director of the Office of Management and Budget, the  Secretaries and agency heads of the departments and agencies in this  report, and other interested parties. This report will also be available at no  charge on our website at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix XVII.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this engagement were to determine (1) the number of  investments certified by agencies as implementing adequate incremental  development and any reported challenges that impact the agencies’  incremental delivery of functionality; and (2) whether agencies are  establishing policies and processes for chief information officer (CIO)  certification of incremental development in accordance with the Federal  Information Technology Acquisition Reform Act provisions (commonly  referred to as FITARA) enacted as a part of the Carl Levin and Howard P.  ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year 2015.

For our first objective, we obtained and analyzed major information  technology (IT) investment data reported by agencies on the IT  Dashboard as of August 31, 2016, for fiscal year 2017, which was the first  year that the Office of Management and Budget (OMB) required the 24  covered agencies to report the status of CIO certification of incremental  development for each investment. We chose this date because it was the  final day updated fiscal year 2017 data from the agencies would be  publicly available until the release of the President’s fiscal year 2018  budget submission.

Initially, we analyzed the fiscal year 2017 data of major IT software  development investments that were planning to allocate at least 50  percent of their funding to development, modernization, and  enhancement activities. We then reviewed agency responses to the  question regarding CIO certification of adequate incremental development and eliminated any investment where the agency’s rationale for choosing  “not applicable” was due to the investment not undertaking software  development activities. In doing so, we identified a total of 166  investments from 21 agencies. Three agencies (National Aeronautics and  Space Administration, National Science Foundation, and U.S. Nuclear  Regulatory Commission) out of the 24 in our review did not have any  investments that met these criteria for fiscal year 2017. For the 21  agencies with major IT investments to review, we then determined the  total number of investments that agencies reported were certified by the  CIO for adequate incremental development. We also reviewed and  summarized agency responses reported on the IT Dashboard for  investments that did not have CIO certification.

To help determine the reliability of the reported agency CIO certification  data on the IT Dashboard, we presented the results of our analysis of CIO  certification responses to officials from each agency’s Office of the CIO  that were involved in investment management and software development  activities and solicited their input and explanations for the results. Two  agencies each provided an update on one of their investments, which we  have incorporated as appropriate. We determined that the data were  sufficiently reliable for the purpose of this report.

In order to identify the challenges impacting the agencies’ incremental  delivery of functionality, we developed a list of common challenges based  on our prior work, in which eight agencies reported that the following eight  challenges inhibited their delivery of functionality:  1.  project staff were over-utilized or lacked the necessary skills and  2.  programs did not receive sufficient funding or received funding later  3.  projects experienced management and organizational challenges that  introduced delays;  4.  development work was slowed by inefficient governance and  5.  project characteristics made rapid delivery of functionality infeasible or  6.  development schedules were impeded by procurement delays;  7.  programs did not have stable, prioritized requirements; and  8.  incremental development was impeded by select technologies.

We sent the list of challenges to each of the 24 agencies and asked  officials from the Office of the CIO at each agency involved with  investment management and software development activities to identify  their top three challenges from this list that impacted their ability to deliver  incremental functionality for major IT investments. We also asked agency  officials to identify any challenges that were not included in the list, but  which were also among their top three challenges. Finally, we asked  agencies to explain what actions were taken to address the reported  challenges and describe the extent to which the challenges were  overcome.

Because of the open-ended nature of the agencies’ responses to our  questions, we conducted a content analysis of the information we  received in order to identify common challenges that impact agencies’  ability to deliver incremental functionality. In doing so, team members  individually reviewed the challenges reported by agencies and assigned  them to various categories. Team members then compared  categorization schemes, discussed the differences, and reached  agreement on the final list of challenges by totaling the number of times  each challenge was mentioned. For those challenges that were prompted  by the list we provided to agencies, we reported challenges that were  identified by five or more agencies. Three agencies also identified a new  challenge that was not on our list, which we reported due to the number  of agencies reporting it as a challenge. Three of the 24 agencies in our  review (Departments of Defense, Energy, and Health and Human  Services) reported that they had no challenges with implementing  incremental development.

We also asked the agencies in our review how the CIO utilized the  information obtained during the process of certifying investments’  adequate incremental development to make decisions regarding the  agency’s major IT investments. Because of the wide variety of responses  we received from agencies, we conducted a content analysis of the  information in order to identify ways the CIOs used the information. In  doing so, team members individually reviewed agencies’ responses and  assigned them to various categories. Team members then compared  their categorization schemes, discussed the differences, and reached  agreement on the final characterization of ways in which agencies  benefited from the certification process.

For our second objective, we analyzed the 24 agencies’ policies and  processes governing the CIO certification of adequate incremental  development to determine whether those policies and processes were  consistent with FITARA. The provision states that OMB is to require in its  annual IT capital planning guidance that agency CIOs covered by the law  certify that IT investments are adequately implementing incremental  development. To assess this, we reviewed guidance issued by OMB on  the implementation of FITARA, and assessed agencies’ documentation  of incremental development certification policies and processes against  GAO’s IT investment management framework. This framework states  that an organization’s policies and procedures should be clearly defined,  in that they provide details regarding the role of appropriate stakeholders  and the artifacts to document decisions made.

Because of the wide variety of responses and documents we received  from agencies related to their incremental development certification  processes, we conducted a content analysis of the information in order to  determine compliance with OMB’s guidance. In doing so, team members  individually reviewed agencies’ responses and documents and assigned  them to various categories and sub-categories. Team members then  compared their categorization schemes, discussed the differences, and  reached agreement on the final characterization of compliance with OMB  guidance. In cases where agencies provided multiple policies or  documents, we followed up to clarify which portions were considered by  the agency to support the CIO certification requirement.

In analyzing whether the agencies’ policies on CIO certification met  FITARA, OMB, and GAO criteria, we assessed whether the policies  clearly defined the role of the CIO in the certification of adequate  incremental development, and described how CIO certification was  documented. We also reviewed agencies’ incremental development  policies and processes to identify the agencies’ definitions of incremental  development and time frames for delivering functionality to determine  whether they were consistent with OMB guidance. Agencies found to not  have a policy where the CIO process was clearly defined were evaluated  as such for one of two reasons: either the agency’s formal policy did not  completely address our assessment criteria or the agency’s policy had  not yet been finalized. For agencies that told us they had not yet finalized  a policy for certification, we asked them to explain the process, if any,  used by the agency to certify major IT investments for fiscal year 2017. In  addition, we interviewed staff from OMB’s Office of E-Government and  Information Technology regarding its guidance to agencies related to  FITARA’s incremental development certification provision.

We conducted this performance audit from July 2016 to November 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Federal Agency Major IT Investments’ Reported Chief Information Officer Certification of Incremental Development on the IT Dashboard for Fiscal Year 2017

Table 5 lists the 166 major information technology (IT) software  development investments primarily in development, as reported on the IT  Dashboard as of August 31, 2016, and the agency’s reported response to  the question in the major IT business case regarding whether the  agency’s Chief Information Officer certified the adequate use of  incremental development for the investment for fiscal year 2017. All 166  investments reported in the major IT business case that the investment  included software development.

Appendix III: Analysis of Federal Agency Chief Information Officer Incremental Development Certification Policies

Table 6 shows our analysis regarding whether the agency had policies and processes that clearly defined the Chief Information Officer (CIO) certification process for the adequate use of incremental development, including: (1) describing the CIO’s role in the certification process; (2) describing how CIO certification is to be documented; (3) having a definition of incremental development in the policy consistent with Office of Management and Budget (OMB) guidance; and (4) having time frames for delivering functionality in the policy consistent OMB guidance.

Appendix V: Comments from the Department of Housing and Urban Development

Appendix VI: Comments from the Department of the Interior

Appendix VII: Comments from the Department of State

Appendix VIII: Comments from the Department of Veterans Affairs

Appendix IX: Comments from the Environmental Protection Agency

Appendix X: Comments from the General Services Administration

Appendix XI: Comments from the National Aeronautics and Space Administration

The report number GAO-17-556 has been changed to GAO-18-148.

Appendix XII: Comments from the U.S. Nuclear Regulatory Commission

Appendix XIII: Comments from the Office of Personnel Management

Appendix XIV: Comments from the Social Security Administration

Appendix XV: Comments from the U.S. Agency for International Development

Appendix XVI: Comments from the Department of Commerce

Appendix XVII: GAO Contact and Staff Acknowledgments

In addition to the individual named above, the following staff made key  contributions to this report: Dave Hinchman (Assistant Director), Chris  Businsky, Rebecca Eyler, Justin Fisher, Valerie Hopkins (Analyst in  Charge), Sandra Kerr, James MacAulay, Jamelyn Payan, Priscilla Smith,  and Andrew Stavisky.