Information Technology: Management Attention Is Needed to Successfully Modernize Tax Processing Systems

The IRS, a bureau of the Department of the Treasury, relies extensively on IT to annually collect more than $3 trillion in taxes, distribute more than $400 billion in refunds, and carry out its mission of providing service to America's taxpayers in meeting their tax obligations. For fiscal year 2016, IRS expended approximately $2.7 billion for IT investments, 70 percent of which was allocated for operational systems.

GAO has long reported that the effective and efficient management of IT acquisitions and operational investments has been a challenge in the federal government. Accordingly, in February 2015, GAO introduced a new government-wide high-risk area, Improving the Management of IT Acquisitions and Operations. GAO has also reported on challenges IRS has faced in managing its IT acquisitions and operations, and identified opportunities for IRS to improve the management of these investments.

In light of these challenges, GAO was asked to testify about IT management at IRS. To do so, GAO summarized its prior work regarding IRS's IT management, including the agency's management of operational, or legacy, IT systems.

GAO has issued a series of reports in recent years which have identified numerous opportunities for the Internal Revenue Service (IRS) to improve the management of its major acquisitions and operational, or legacy, information technology (IT) investments. For example,

In June 2016, GAO reported that IRS had developed a structured process for allocating funding to its operations activities, consistent with best practices; however, GAO found that IRS did not have a similarly structured process for prioritizing modernization activities to which the agency allocated hundreds of millions of dollars for fiscal year 2016. Instead, IRS officials stated that they held discussions to determine the modernization efforts that were of highest priority to meet IRS's future state vision and technology roadmap, and considered staffing resources and lifecycle stage. However, they did not use formal criteria for making final determinations. GAO concluded that establishing a structured process for prioritizing modernization activities would better assist Congress and other decision makers in ensuring that the right priorities are funded. Accordingly, GAO recommended that IRS establish, document, and implement policies and procedures for prioritizing modernization activities. IRS agreed with the recommendation and has efforts underway to address it.

In the same report, GAO noted that IRS could improve the accuracy of reported performance information for key development investments to provide Congress and other external parties with pertinent information about the delivery of these investments. This included investments such as Customer Account Data Engine 2, which IRS is developing to replace its 50-year old repository of individual tax account data, and the Return Review Program, IRS's system of record for fraud detection. GAO recommended that IRS take steps to improve reported investment performance information. IRS agreed with the recommendation, and has efforts underway to address it.

In a May 2016 report on legacy IT systems across the federal government, GAO noted that IRS used assembly language code to program key legacy systems. Assembly language code is a computer language initially used in the 1950s that is typically tied to the hardware for which it was developed; it has become difficult to code and maintain. One investment that used this language is IRS's Individual Master File which serves as the authoritative data source for individual taxpayer accounts. GAO noted that, although IRS has been working to replace the Individual Master File, the bureau did not have time frames for its modernization or replacement. Therefore, GAO recommended that the Department of Treasury identify and plan to modernize and replace this legacy system, consistent with applicable guidance from the Office of Management and Budget. The department had no comments on the recommendation.

GAO has made a number of recommendations to IRS to improve its management of IT acquisitions and operations. IRS has generally agreed with the recommendations and is in various stages of implementing them.

I am pleased to be here today to discuss our recent work related to the  Internal Revenue Service’s (IRS) management of information technology  (IT). IRS relies extensively on IT systems to annually collect more than $3  trillion in taxes, distribute more than $400 billion in refunds, and carry out  its mission of providing service to America’s taxpayers in meeting their tax  obligations. For fiscal year 2016, IRS expended approximately $2.7 billion  for IT investments, including $1.9 billion, or 70 percent, for operational  systems, and approximately $800 million, or 30 percent, for development  and modernization.

As you know, however, the effective and efficient acquisition and  management of IT investments has been a long-standing challenge in the  federal government. IRS, in particular, has faced challenges in managing  its acquisitions and operations, and we have reported on opportunities for  the agency to improve the management of its IT investments.

My statement today summarizes our prior reports that have addressed  IRS’s IT management, including the management of its operational, or  legacy, systems. A more detailed discussion of the objectives, scope,  and methodology for the work conducted is included in these reports.

We conducted the work on which this statement is based in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.

The mission of IRS, a bureau within the Department of the Treasury, is to  (1) provide America’s taxpayers top quality service by helping them  understand and meet their tax responsibilities and (2) enforce the law with  integrity and fairness to all. In carrying out its mission, IRS annually  collects over $3 trillion in taxes from millions of taxpayers, and manages  the distribution of over $400 billion in refunds. To guide its future  direction, the agency has two strategic goals: (1) deliver high quality and  timely service to reduce taxpayer burden and encourage voluntary  compliance; and (2) effectively enforce the law to ensure compliance with  tax responsibilities and combat fraud.

IRS Relies on Major IT Investments for Tax Processing

Effective management of IT is critical for agencies to achieve successful  outcomes. This is particularly true for IRS, given the role of IT in enabling  the agency to carry out its mission and responsibilities. For example, IRS  relies on information systems to process tax returns; account for tax  revenues collected; send bills for taxes owed; issue refunds; assist in the  selection of tax returns for audit; and provide telecommunications  services for all business activities, including the public’s toll-free access to  tax information.

For fiscal year 2016, IRS was pursuing 23 major and 114 non-major IT  investments to carry out its mission. According to the agency, it expended  approximately $2.7 billion on these investments during fiscal year 2016,  including $1.9 billion, or 70 percent, for operations and maintenance  activities, and approximately $800 million, or 30 percent, for development,  modernization, and enhancement. We have previously reported on a  number of the agency’s major investments, to include the following  investments in development, modernization, and enhancement:

The Affordable Care Act investment encompasses the planning,  development, and implementation of IT systems needed to support  tax administration responsibilities associated with key provisions of  the Patient Protection and Affordable Care Act. IRS expended $253  million on this investment in fiscal year 2016.

Customer Account Data Engine 2 is being developed to replace the  Individual Master File investment, IRS’s authoritative data source for  individual tax account data. A major component of the program is a  modernized database for all individual taxpayers that is intended to  provide the foundation for more efficient and effective tax  administration and help address financial material weaknesses for  individual taxpayer accounts. Customer Account Data Engine 2 data  is also expected to be made available for access by downstream  systems, such as the Integrated Data Retrieval System for online  transaction processing by IRS customer service representatives. IRS  expended $182.6 million on this investment in fiscal year 2016.

The Return Review Program is IRS’s system of record for fraud  detection. As such, it is intended to enhance the agency’s capabilities  to detect, resolve, and prevent criminal and civil tax noncompliance. In  addition, it is intended to allow analysis and support of complex case  processing requirements for compliance and criminal investigation  programs during prosecution, revenue protection, accounts  management, and taxpayer communications processes. According to  IRS, as of May 2017, the system has helped protect over $4.5 billion  in revenue. IRS expended $100.2 million on this investment in fiscal  year 2016.

We have also reported on the following investments in operations and  maintenance:

Mainframes and Servers Services and Support provides for the  design, development, and deployment of server; middleware; and  large systems and enterprise storage infrastructures, including  supporting systems software products, databases, and operating  systems. This investment has been operational since 1970. IRS  expended $499.4 million on this investment in fiscal year 2016.

Telecommunications Systems and Support provides for IRS’s network  infrastructure services such as network equipment, video conference  service, enterprise fax service, and voice service for over 85,000  employees at about 1,000 locations. According to IRS, the investment  supports the delivery of services and products to employees, which  translates into service to taxpayers. IRS expended $336.4 million on  this investment in fiscal year 2016.

Individual Master File is the authoritative data source for individual  taxpayer accounts. Using this system, accounts are updated, taxes  are assessed, and refunds are generated as required during each tax  filing period. Virtually all IRS information system applications and  processes depend on output, directly or indirectly, from this data  source. IRS expended $14.3 million on this investment in fiscal year  2016.

GAO, Congress, and the Administration Have Highlighted the Need for Government-wide Improvements for IT Acquisitions and Operations

In fiscal year 2017, the federal government planned to spend more than  $89 billion for IT that is critical to the health, economy, and security of the  nation. However, we have reported that prior IT expenditures have often  resulted in significant cost overruns, schedule delays, and questionable  mission-related achievements. In light of these ongoing challenges, in  February 2015, we added improving the management of IT acquisitions  and operations to our list of high-risk areas for the federal government.  This area highlights several critical IT initiatives in need of additional  congressional oversight, including (1) reviews of troubled projects; (2)  efforts to increase the use of incremental development; (3) efforts to  provide transparency relative to the cost, schedule, and risk levels for  major IT investments; (4) reviews of agencies’ operational investments;  (5) data center consolidation; and (6) efforts to streamline agencies’  portfolios of IT investments. We noted that implementation of these  initiatives has been inconsistent and more work remains to demonstrate  progress in achieving acquisitions and operations outcomes. Between  fiscal years 2010 and 2015, we made about 800 recommendations  related to this high-risk area to the Office of Management and Budget and  agencies. As of September, 2017, about 54 percent of these  recommendations had been implemented.

The Federal Information Technology Acquisition Reform provisions  (commonly referred to as FITARA), enacted as a part of the Carl Levin  and Howard P. ‘Buck’ McKeon National Defense Authorization Act for  Fiscal Year 2015, aimed to improve federal IT acquisitions and operations  and recognized the importance of the initiatives mentioned above by  incorporating certain requirements into the law. For example, among  other things, the act requires the Office of Management and Budget to  publicly display investment performance information and review federal  agencies’ IT investment portfolios.

The current administration has also initiated additional efforts aimed at  improving federal IT. Specifically, in March 2017, the administration  established the Office of American Innovation, which has a mission to,  among other things, make recommendations to the President on policies  and plans aimed at improving federal government operations and  services and modernizing federal IT. Further, in May 2017, the  administration established the American Technology Council, which has a  goal of helping to transform and modernize federal agency IT and how  the federal government uses and delivers digital services. Recently this  council worked with several agencies to develop a draft report on  modernizing IT in the federal government. The council released the draft  report for public comment in August 2017.

GAO Reviews Have Identified Weaknesses with IRS’s Management of Its Modernization Activities and Legacy Systems

In reviews that we have undertaken over the past several years, we have  identified various opportunities for the IRS to improve the management of  its IT investments. These reviews have identified a number of  weaknesses with the agency’s reporting on the performance of its  modernization investments to Congress and other stakeholders. In this  regard, we have pointed out that information on investments’ performance  in meeting cost, schedule, and scope goals is critical to determining the  agency’s progress in completing key IT investments. We have also  stressed the importance of the agency addressing weaknesses in its  process for prioritizing modernization activities. Accordingly, we have  made a number of related recommendations, which IRS is in various  stages of implementing.

In our June 2012 report on IRS’s performance in meeting cost,  schedule, and scope goals for selected investments, we noted that,  while IRS reported on the cost and schedule of its major IT  investments, the agency did not have a quantitative measure of  scope—a measure that shows whether these investments delivered  planned functionality. We stressed that having such a measure is a  good practice as it provides information about whether an investment  has delivered the functionality that was paid for. Accordingly, we  recommended that the agency develop a quantitative measure of  scope for its major IT investments, to have more complete information  on the performance of these investments. In response, IRS started  developing a quantitative measure of scope for selected investments  in December 2015 and has been working to gradually expand the  measure to other investments.

In April 2013, based on another review of IRS’s performance in meeting  cost, schedule, and scope goals, we reported that there were  weaknesses, to varying degrees, in the reliability of IRS’s investment  performance information. Specifically, we found that IRS had not  updated investment cost and schedule variance information with actual  amounts on a timely basis (i.e., within the 60-day time frame required by  the Department of Treasury) in about 25 percent of the activities  associated with the investments selected in our review. In addition, the  agency had not specified how project managers should estimate the cost  and schedule performance of ongoing projects.

As a result of these findings, we recommended that IRS ensure that its  projects consistently follow guidance for updating performance  information 60 days after completion of an activity and develop and  implement guidance that specifies best practices to consider when  estimating ongoing projects’ progress in meeting cost and schedule  goals. IRS agreed with, and subsequently addressed, the  recommendation related to updating performance information on a timely  basis. However, the agency partially disagreed with the recommendation  to develop guidance on estimating progress in meeting cost and schedule  goals for ongoing projects. In this regard, we had suggested the use of  earned value management data as a best practice to determine projected  cost and schedule amounts. IRS did not agree with the use of the  technique, stating that it was not part of the agency’s current program  management processes and that the cost and burden to use earned  value management would outweigh the value added.

We disagreed with the agency’s view of earned value management  because best practices have found that its value generally outweighs the  cost and burden of its implementation (although we suggested it as one of  several examples of practices that could be used to determine projected  amounts). We also stressed that implementing our recommendation  would help improve the reliability of reported cost and schedule variance  information, and that IRS had flexibility in determining which best  practices to use to calculate projected amounts. For those reasons, we  maintained that our recommendation was warranted. However, IRS has  yet to address the recommendation.

We reported in April 2014, that the cost and schedule performance  information that IRS reported for its major investments was for the  fiscal year only. We noted that this reporting would be more  meaningful if supplemented with cumulative cost and schedule  performance information in order to better indicate progress toward  meeting goals. In addition, we noted that the reported variances for  selected investments were not always reliable because the estimated  and actual cost and schedule amounts on which they depended had  not been consistently updated in accordance with Department of  Treasury reporting requirements as we had previously recommended.

We recommended that IRS report more comprehensive and reliable cost  and schedule information for its major investments. The agency agreed  with our recommendation and said it believed it had addressed the  recommendation in its quarterly reports to Congress. We disagreed with  IRS’s assertion, however, noting that, while the report includes cumulative  costs, they are cumulative for the fiscal year, not for the investment or  investment segment as we recommended and they therefore do not  account for cost variances from prior fiscal years. We therefore  maintained our recommendation.

In February 2015, after assessing the status and plans of the Return  Review Program and Customer Account Data Engine 2, we reported  that these investments had experienced significant variances from  initial cost, schedule, and scope plans; yet, IRS did not include these  variances in its reports to Congress because the agency had not  addressed our prior recommendations. Specifically, IRS had not  addressed our recommendation to report on how delivered scope  compared to what was planned, and it also did not address guidance  for determining projected cost and schedule amounts, or the reporting  of cumulative cost and schedule performance information. We  stressed that implementing these recommendations would improve  the transparency of congressional reporting so that Congress has the  appropriate information needed to make informed decisions. We  made additional recommendations for the agency to improve the  reliability and reporting of investment performance information and  management of selected major investments. IRS agreed with the  recommendations and has since addressed them.

In our most recent report in June 2016, we assessed IRS’s process  for determining its funding priorities for both modernization and  operations. We found that the agency had developed a structured  process for allocating funding to its operations activities consistent  with best practices, which specify that an organization should  document policies and procedures for selecting new and reselecting  ongoing IT investments, and include criteria for making selection and  prioritization decisions.

However, IRS did not have a similarly structured process for prioritizing its  modernization activities, to which the agency allocated hundreds of  millions of dollars for fiscal year 2016. Agency officials stated that  discussions were held to determine the modernization efforts that were of  highest priority to meet IRS’s future state vision and technology roadmap.  The officials reported that staffing resources and lifecycle stage were  considered, but there were no formal criteria for making final  determinations. Senior IRS officials said they did not have a structured  process for the selection and prioritization of business systems  modernization activities because the projects were established; and there  were fewer competing activities than for operations support.

Nevertheless, we stressed that, while there may have been fewer  competing activities, a structured, albeit simpler, process that is  documented and consistent with best practices would provide  transparency into the agency’s needs and priorities for appropriated  funds. We concluded that such a process would better assist Congress  and other decision makers in carrying out their oversight responsibilities.  Accordingly, we recommended that IRS develop and document its  processes for prioritizing IT funding. The agency agreed with the  recommendations and has taken steps to address them.

Further, we found that IRS had reported complete performance  information for two of the six selected investments in our review, to  include a measure of progress in delivering scope, which we have been  recommending since 2012. However, the agency did not always use best  practices for determining the amount of work completed by its own staff,  resulting in inaccurate reports of work performed. Consequently, we  recommended that IRS modify its processes for determining the work  performed by its staff. The agency disagreed with the recommendation,  stating that the costs involved would outweigh the value provided.  Specifically, IRS stated that modifying the use of the level of effort  measure would equate to a certified earned value management system,  which would add immense burden on IRS’s programs on various fronts  and would outweigh the value it provides. However, we did not specify the  use of an earned value management system in our report and believe  other methods could be used to more reliably measure work performed..  In addition, we believed that it is a reasonable expectation for IRS to  reliably determine the actual work completed, as opposed to assuming  that work is always completed as planned since, as noted in our report,  22 to 100 percent of the work for selected projects was performed by IRS  staff. Accordingly, we maintained that the recommendation was still  warranted.

IRS Faces Challenges with Managing Its Aging Legacy Systems

Our work has also emphasized the importance of IRS more effectively  managing its aging legacy systems. For example, in November 2013, we  reported on the extent to which 10 of the agency’s large investments had  undergone operational analyses—a key performance evaluation and  oversight mechanism required by the Office of Management and Budget  to ensure investments in operations and maintenance continue to meet  agency needs. We noted that IRS’s Mainframe and Servers Services  and Support had not had an operational analysis for fiscal year 2012. As  a result, we recommended that the Secretary of Treasury direct  appropriate officials to perform an operational analysis for the investment,  including ensuring that the analysis addressed the 17 key factors  identified in the Office of Management and Budget’s guidance for  performing operational analyses. The department did not comment on  our recommendation but subsequently implemented it.

In addition, we previously reported on legacy IT systems across the  federal government, noting that these systems were becoming  increasingly obsolete and that many of them used outdated software  languages and hardware parts that were unsupported. As part of that  work, we noted that the Department of the Treasury used assembly  language code—a computer language initially used in the 1950s and  typically tied to the hardware for which it was developed—and Common  Business Oriented Language (COBOL)—a programming language  developed in the late 1950s and early 1960s—to program its legacy  systems.

It is widely known that agencies need to move to more modern,  maintainable languages, as appropriate and feasible. For example, the  Gartner Group, a leading IT research and advisory company, has  reported that organizations using COBOL should consider replacing the  language and, in 2010, noted that there should be a shift in focus to using  more modern languages for new products. The use of COBOL presents  challenges for agencies such as IRS given that procurement and  operating costs associated with this language will steadily rise, and  because fewer people with the proper skill sets are available to support  the language.

Further, we reported that IRS’s Individual Master File was over 50 years  old and, although IRS was working to modernize it, the agency did not  have a time frame for completing the modernization or replacement.  Thus, we recommended that the Secretary of the Treasury direct the  Chief Information Officer to identify and plan to modernize and replace  legacy systems, as needed, and consistent with the Office of  Management and Budget’s draft guidance on IT modernization, including  time frames, activities to be performed, and functions to be replaced or  enhanced. The department had no comments on our recommendation.  We will continue to follow-up with the agency to determine the extent to  which this recommendation has been addressed. In addition, we have  ongoing work identifying risks associated with IRS’s legacy IT systems,  and the agency’s management of these risks.

In summary, IRS faces longstanding challenges in managing its IT  systems. While effective IT management has been a prevalent issue  throughout the federal government, it is especially concerning at IRS  given the agency’s extensive reliance on IT to carry out its mission of  providing service to America’s taxpayers in meeting their tax obligations.  Thus, it is important that the agency establish, document, and implement  policies and procedures for prioritizing its modernization efforts, as we  have recently recommended, and provide Congress with accurate  information on progress in delivering such modernization efforts. In  addition, we have emphasized the need for IRS to address the inherent  challenges associated with aging legacy systems so that it does not  continue to maintain investments that have outlived their effectiveness  and are consuming resources that outweigh their benefits. Continued  attention to implementing our recommendations will be vital to helping  IRS ensure the effective management of its efforts to modernize its aging  IT systems and ensure its multibillion dollar investment in IT is meeting  the needs of the agency.

Chairman Buchanan, Ranking Member Lewis, and Members of the  Subcommittee, this completes my prepared statement. I would be  pleased to respond to any questions that you may have at this time.

If you or your staffs have any questions about this testimony, please  contact me at (202) 512-9286 or at pownerd@gao.gov. Individuals who  made key contributions to this testimony are Sabine Paul (Assistant  Director), Rebecca Eyler, and Bradley Roach (Analyst in Charge).

IRS 2013 Budget: Continuing to Improve Information on Program Costs  and Results Could Aid in Resource Decision Making, GAO-12-603  (Washington, D.C.: June 8, 2012)

Information Technology: Consistently Applying Best Practices Could Help  IRS Improve the Reliability of Reported Cost and Schedule Information,  GAO-13-401 (Washington, D.C.: April 17, 2013)

Information Technology: Agencies Need to Strengthen Oversight of  Multibillion Dollar Investments in Operations and Maintenance,   GAO-14-66 (Washington, D.C.: Nov. 6, 2013)

Information Technology: IRS Needs to Improve the Reliability and  Transparency of Reported Investment Information, GAO-14-298  (Washington, D.C.: April 2, 2014)

Information Technology: Management Needs to Address Reporting of  IRS Investments’ Cost, Schedule, and Scope Information, GAO-15-297  (Washington, D.C.: February 25, 2015)

Information Technology: Federal Agencies Need to Address Aging  Legacy Systems, GAO-16-468 (Washington, D.C.: May 25, 2016)

This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.


2020 Census: Actions Needed to Improve In-Field Address Canvassing Operation

The success of the decennial census depends in large part on the Bureau's ability to locate every household in the United States. To accomplish this monumental task, the Bureau must maintain accurate address and map information for every location where a person could reside. For the 2018 End-to-End Test, census workers known as listers went door-to-door to verify and update address lists and associated maps in selected areas of three test sites—Bluefield-Beckley-Oak Hill, West Virginia; Pierce County, Washington; and Providence County, Rhode Island.

GAO was asked to review in-field address canvassing during the End-to-End Test. This report determines whether key address listing activities functioned as planned during the End-to-End Test and identifies any lessons learned that could inform pending decisions for the 2020 Census. To address these objectives, GAO reviewed key documents including test plans and training manuals, as well as workload, productivity and hiring data. At the three test sites, GAO observed listers conducting address canvassing.

The Census Bureau (Bureau) recently completed in-field address canvassing for the 2018 End-to-End Test. GAO found that field staff known as listers generally followed procedures when identifying and updating the address file; however, some address blocks were worked twice by different listers because the Bureau did not have procedures for reassigning work from one lister to another while listers work offline. Bureau officials told GAO that they plan to develop procedures to avoid duplication but these procedures have not been finalized. Duplicating work decreases efficiency and increases costs.

GAO also found differences between actual and projected data for workload, lister productivity, and hiring.

For the 2020 Census, the Bureau estimates it will have to verify 30 percent of addresses in the field. However, at the test sites, the actual workload ranged from 37 to 76 percent of addresses. Bureau officials told GAO the 30 percent was a nationwide average and not site specific; however, the Bureau could not provide documentation to support the 30 percent workload estimate.

At all three test sites listers were significantly more productive than expected possibly because a design change provided better quality address and map data in the field, according to the Bureau.

Hiring, however, lagged behind Bureau goals. For example, at the West Virginia site hiring was only at 60 percent of its goal. Bureau officials attributed the shortfall to a late start and low unemployment rates.

Workload and productivity affect the cost of address canvassing. The Bureau has taken some steps to evaluate factors affecting its estimates, but continuing to so would help the Bureau refine its assumptions to better manage the operation's cost and hiring.

Listers used laptops to connect to the Internet and download assignments. They worked offline and went door-to-door to update the address file, then reconnected to the Internet to transmit their completed assignments. Bureau officials told GAO that during the test 11 out of 330 laptops did not properly transmit address and map data collected for 25 blocks. Data were deleted on 7 laptops. Because the Bureau had known there was a problem with software used to transmit address data, it created an alert report to notify the Bureau staff if data were not properly transmitted. However, Bureau officials said that either responsible staff did not follow procedures to look at the alert reports or the reports were not triggered. The Bureau is working to fix the software problem and develop new alert reports, but has not yet determined and addressed why these procedures were not followed.

The Bureau's data management reporting system did not always provide accurate information because of a software issue. The system was supposed to pull data from several systems to create a set of real-time cost and progress reports for managers to use. Because the data were not accurate, Bureau staff had to rely on multiple systems to manage address canvassing. The Bureau agreed that not only is inaccurate data problematic, but that creating workarounds is inefficient. The Bureau is developing new requirements to ensure data are accurate but these requirements have not been finalized.

GAO is making seven recommendations to the Department of Commerce and Bureau including to: (1) finalize procedures for reassigning work, (2) continue to evaluate workload and productivity data, (3) fix software problem, or determine and address why procedures were not followed, and (4) finalize report requirements to ensure data are accurate. The Department of Commerce agreed with GAO's recommendations, and the Bureau provided technical comments that were incorporated, as appropriate.

The federal government is constitutionally mandated to undertake the  decennial census, a complex and costly activity—estimated at $15.6  billion (dollars inflated to the current 2020 Census time frame fiscal years  2012-2023) for the 2020 Census. The data that the census produces are  used to apportion the seats of the U.S. House of Representatives; realign  the boundaries of the legislative districts of each state; allocate hundreds  of billions of dollars in federal financial assistance; and provide a social,  demographic, and economic profile of the nation’s people to guide policy  decisions at each level of government.

The success of the census depends largely on the ability of the Census  Bureau (Bureau) to locate every person residing in the United States. To  accomplish this monumental task, the Bureau must maintain accurate  address and map information for every person’s residence. If the  Bureau’s address list and maps are inaccurate, people can be missed,  counted more than once, or included in the wrong location. In an effort to  help control costs, the Bureau is using new procedures to build its  address list for 2020. As these procedures have not been used in prior  decennials, the Bureau has conducted several tests in the last few years  to help ensure the new approach will function as planned and produce a  complete and accurate address database. The 2018 End-to-End Test is  the last opportunity to demonstrate census technology and procedures— including new methods for building the address list—across a range of  geographic locations, housing types, and demographic groups under  census-like conditions before the 2020 Census.

On August 28, 2017, the Bureau began what it calls the “in-field” address  canvassing operation for the End-to-End Test where temporary census  employees known as listers walked the streets of designated census  blocks. In three test sites—Bluefield-Beckley-Oak Hill, West Virginia;  Pierce County, Washington; and Providence County, Rhode Island— listers knocked on doors and, using laptops connected to the internet,  verified the address and geographic location of assigned housing units  and identified any additions, deletions, and any other changes that need  to be made to the address list. For example, they would add converted  basements, attics, and other “hidden” housing units to the list.

You asked us to review how the address canvassing operation performed  as part of the 2018 End-to-End Test. This report (1) determines the extent  to which key “in-field” address listing activities functioned as planned and  (2) identifies any lessons learned that could potentially affect pending  decisions for the 2020 Census.

To address these objectives, we reviewed key documents including the  2018 End-to-End Test plan that discussed the goals and objectives for  the test, as well as training manuals and other related documents for  address canvassing. We interviewed Bureau staff at the three 2018  Census test sites including census field supervisors (CFS), address  listers, and office personnel to discuss what went well and what  challenges they faced during address canvassing. At each test site, the  Bureau selected Census field staff for us to interview and observe from  among those working on the days of our visits. At all three test sites, we  observed listers conduct address canvassing. In addition, we used the  training manuals to determine whether listers collected address  information as prescribed by the Bureau. In total we conducted 18 in-field  observations of listers and used a data collection instrument to document  our observations. These observations are not generalizable. We also  interviewed Bureau headquarters officials to discuss the use of  management reports for monitoring and overseeing the operation.

We reviewed workload estimates, address lister productivity rates, and  hiring information for each test site in order to report how many housing  units were included at each test site, how many addresses the Bureau  expected to canvass per hour, and how many people they needed to hire.  To assess the reliability of these data, we reviewed available  documentation and interviewed knowledgeable officials. We found the  data to be sufficiently reliable for the purposes of our reporting objectives.  We also met periodically with Bureau headquarters staff to discuss  progress of the operation.

We conducted this performance audit from July 2017 to June 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The Bureau’s address canvassing operation updates its address list and  maps, which are the foundation of the decennial census. An accurate  address list both identifies all households that are to receive a notice by  mail requesting participation in the census (by Internet, phone, or mailed- in questionnaire) and serves as the control mechanism for following up  with households that fail to respond to the initial request. Precise maps  are critical for counting the population in the proper locations—the basis  of congressional apportionment and redistricting.

Our prior work has shown that developing an accurate address list is  challenging—in part because people can reside in unconventional  dwellings, such as converted garages, basements, and other forms of  hidden housing. For example, as shown in figure 1, what appears to be a  single-family house could contain an apartment, as suggested by its two  doorbells.

During address canvassing, the Bureau verifies that its master address  list and maps are accurate to ensure the tabulation for all housing units  and group quarters is correct. For the 2010 Census, the address  canvassing operation mobilized almost 150,000 field workers to canvass  almost every street in the United States and Puerto Rico to update the  Bureau’s address list and map data—and in 2012 reported the cost at  nearly $450 million. The cost of going door-to-door in 2010, along with the  emerging availability of imagery data, led the Bureau to explore an  approach for 2020 address canvassing that would allow for fewer boots  on the ground.

Traditionally, the Bureau went door-to-door to homes across the country  to verify addresses. This “in-field address canvassing” is a labor-intensive  and expensive operation. To achieve cost savings, in September 2014  the Bureau decided to use a reengineered approach for building its  address list for the 2020 Census and not go door-to-door (or “in-field”)  across the country, as it has in prior decennial censuses. Rather, some  areas (known as “blocks”) would only need a review of their address and  map information using computer imagery and third-party data sources— what the Bureau calls “in-office” address canvassing procedures.

According to the Bureau’s address canvassing operational plan, in-office  canvassing had two phases:

During the first phase, known as “Interactive Review,” Bureau  employees use current aerial imagery to determine if areas have  housing changes, such as new residential developments or  repurposed structures, or if the areas match what is in the Bureau’s  master address file. The Bureau assesses the extent to which the  number of housing units in the master address file is consistent with  the number of units visible in the current imagery. If the housing  shown in the imagery matches what is listed in the master address  file, then those areas are considered to be resolved or stable and  would not be canvassed in-field.

During the second phase, known as “Active Block Resolution,”  employees would try to resolve coverage concerns identified during  the first phase and verify every housing unit by virtually canvassing  the entire area. As part of this virtual canvass, the Bureau would  compare what is found in imagery to the master address file data and  other data sources in an attempt to resolve any discrepancies. If  Bureau employees still could not reconcile the discrepancies, such as  housing unit count or street locations with what is on the address list,  then they would refer these blocks to in-field address canvassing.

However, in March 2017, citing budget uncertainty the Bureau decided to  discontinue the second phase of in-office review for the 2020 Census.  According to the Bureau, in order to ensure that the operations  implemented in the 2018 End-to-End Test were consistent with  operations planned for the 2020 Census, the Bureau added the blocks  originally resolved during the second phase of in-office review back into  the in-field workload for the test. The cancellation of Active Block  Resolution is expected to increase the national workload of the in-field  canvassing workload by 5 percentage points (25 percent to 30 percent).

During in-field address canvassing, listers use laptop computers to  compare what they see on the ground to what is on the address list and  map. Listers confirm, add, delete, or move addresses to their correct map  positions. At each housing unit, listers are trained to speak with a  knowledgeable resident to confirm or update address data, ask about  hidden housing units, confirm the housing unit location on the map,  (known as the map spot) and collect a map spot using global positioning  systems (GPS). If no one is available, listers are to use house numbers  and street signs to verify the address data. The data are transmitted  electronically to the Bureau.

The Census Bureau expects that the End-to-End Test for address  canvassing will identify areas for improvement and changes that need to  be made for the 2020 Census. Our prior work has shown the importance  of robust testing. Rigorous testing is a critical risk mitigation strategy  because it provides information on the feasibility and performance of  individual census-taking activities, their potential for achieving desired  results, and the extent to which they are able to function together under  full operational conditions.

In February 2017, we added the 2020 Census to GAO’s High-Risk List  because operational and other issues are threatening the Bureau’s ability  to deliver a cost-effective enumeration. We reported on concerns about  the Bureau’s capacity to implement innovative census-taking methods,  uncertainties surrounding critical information technology systems, and the  quality of the Bureau’s cost-estimates. Underlying these issues are  challenges in such essential management functions as the Bureau’s  ability to:  collect and use real-time indicators of cost, performance, and  schedule;  follow leading practices for cost estimation; scheduling; risk  management; IT acquisition, development, testing, and security; and  cost-effectively deal with contingencies including, for example, fiscal  constraints, potential changes in design, and natural disasters.

The Listers Generally Followed Procedures, but the Bureau Experienced Some Issues Reassigning Work, Estimating Workload and Lister Productivity, and Managing to Staffing Goals

The Bureau completed in-field address canvassing as scheduled by  September 29, 2017, canvassing approximately 340,400 addresses. Most  of the listers we observed generally followed procedures. For example, 15  of 18 listers knocked on doors, and 16 of 18 looked for hidden housing  units, which is important for establishing that address lists and maps are  accurate and for identifying hard-to-count populations. Those procedures  include taking such steps as:  comparing the housing units they see on the “ground” to the housing  units on the address list,  knocking on all doors so they could speak with a resident to confirm  the address (even if the address is visible on the mailbox or house)  and to confirm that there are no other living quarters such as a  basement apartment,  looking for “hidden housing units”,  looking for group quarters such as group homes or dormitories, and  confirming the location of the housing unit on a map with GPS  coordinates collected on the doorstep.

To the extent procedures were not followed, it generally occurred when  listers did not go up to the door and speak with a resident or take a map  spot on the doorstep. Failure to follow procedures could adversely affect  a complete count, as addresses could be missed or a group quarter could  be misclassified as a residential address. After we alerted the Bureau to  our observations, the Bureau agreed moving forward, to emphasize the  importance of following procedures during training for in-field address  canvassing.

Some Listers Duplicated Each Other’s Work Due to a Lack of Operational Procedures for Reassigning Work

Address canvassing has tight time frames, so work needs to be assigned  efficiently. Sometimes this means the Bureau needs to reassign work  from one lister to another. During address canvassing, the Bureau  discovered that reassigned census blocks sometimes would appear in  both the new and the original listers’ work assignments. In some cases,  this led to blocks being worked more than once, which decreased  efficiency, increased costs, and could create confusion and credibility  issues when two different listers visit a house.

According to Bureau procedures, listers were instructed to connect to the  Bureau’s Mobile Case Management (MCM) system to download work  assignments (address blocks) and to transmit their completed work at the  beginning and end of the work day but not during the work day. Thus  during the work day, they were unaware when unworked blocks had been  reassigned to another lister. Bureau officials also told us that the Listing  and Mapping Application (LiMA) software used to update the address file  and maps was supposed to have the functionality to prevent blocks from  being worked more than once, but this functionality was not developed  because of budget cuts.

For 2020, Bureau officials told us they plan to create operational  procedures for reassigning work. According to Bureau officials, they plan  to require supervisors to contact the original lister when work is  reassigned. We have requested a copy of those procedures; however,  the Bureau has not finalized them. Standards for Internal Control in the  Federal Government (Standards for Internal Control) call for management  to design control activities, such as policies and procedures to achieve  objectives. Finalizing these procedures should help prevent blocks from  being canvassed more than once.

The Bureau Has Not Evaluated Workload, Productivity Rates, and Staffing Assumptions for Address Canvassing

The Bureau conducts tests under census-like conditions, in part, to verify  2020 Census planning assumptions, such as workload, how many  houses per hour a lister can verify (also known as a lister’s productivity  rate), and how many people the Bureau needs to hire for an operation.  Moreover, one of the objectives of the test is to validate that the  operations being tested are ready at the scale needed for the 2020  Census. For the 2018 End-to-End Test, the Bureau completed in-field  address canvassing on time at two sites and early at one site; despite  workload increases at all three test sites and hiring shortfalls at two sites.  The Bureau credits this success to better than expected productivity. As  the Bureau reviews the results of address canvassing, evaluating the  factors that affected workload, productivity rates, and staffing and making  adjustments to its estimates, if necessary, before the 2020 Census would  help the Bureau ensure that address canvassing has the appropriate  number of staff and equipment to complete the work in the required time  frame.

For the 2020 Census, the Bureau estimates it will have to send 30  percent of addresses to the field for listers to verify. However, at the three  test sites, the workload was higher than this estimate (see table 1). At one  test site, the percent of addresses verified through in-field address  canvassing was 76 percent or 46 percentage points more than the  Bureau’s expected 2020 Census in-field address canvassing workload  estimate of 30 percent.

Bureau officials told us that the 30 percent in-field workload estimate is a  national average and is not specific to any of the three test sites. Prior to  the test, officials said that the Bureau also knew that the West Virginia  site was assigning new addresses to some of the test site’s housing units  due to local government emergency 911 address conversion and that the  in-field workload would be greater in West Virginia when compared to the  other test sites.

We requested documentation for the Bureau’s original estimate that 30  percent of the 133.8 million expected addresses would be canvassed in- field for the 2020 Census. However, the Bureau was unable to provide us  with documentation to support how they arrived at the 30 percent  estimate. Instead, the Bureau provided us with a November 2017  methodology document that showed three in-field address canvassing  workload scenarios, whereby, between 41.9 and 45.1 percent of housing  units would need to go to the field for address canvassing. The three  scenarios consider a range of stability in the address file as well as  different workload estimates for in-field follow-up. At 30 percent the  Bureau would need to canvass about 40.2 million addresses; however, at  41.9 and 45.1 percent the Bureau would need to canvass between 56  million and 60.4 million addresses, respectively. According to Bureau  officials, they are continuing to assess whether changes to its in-office  address canvassing procedures would be able to reduce the in-field  address canvassing workload to 30 percent, while at the same time  maintaining address quality. However, Bureau officials did not provide us  with documentation to show how the in-field address canvassing  workload would be reduced because the proposed changes were still  being reviewed internally.

Workload for address canvassing directly affects cost – the greater the  workload the more people as well as laptop computers needed to carry  out the operation. We found that the 30 percent workload threshold is  what is reflected in the December 2017 updated 2020 Census cost  estimate that was used to support the fiscal year 2019 budget request.  Thus, if the 30 percent threshold is not achieved then the in-field  canvassing workload will likely increase for the 2020 Census and the  Bureau would be at risk of exceeding its proposed budget for the address  canvassing operation.

Standards for Internal Control call for organizations to use quality  information to achieve their objectives. Thus, continuing to evaluate and  finalize workload estimates for in-field address canvassing with the most  current information will help ensure the Bureau is well-positioned to  conduct addressing canvassing for the 2020 Census. For example,  according to Bureau officials, preliminary workload estimates will need to  be delivered by January 2019 for hiring purposes and the final in-field  workload numbers for address canvassing will need to be determined by  June 2019 for the start of address canvassing, which is set to begin in  August 2019. Moreover, by February 2019 the Bureau’s schedule calls for  it to determine how many laptops will be needed to conduct 2020 Census  address canvassing.

At the test sites, listers were substantially more productive than the  Bureau expected. The expected production rate is defined as the number  of addresses expected to be completed per hour, and it affects the cost of  the address canvassing operation. This rate includes time for actions  other than actually updating addresses, such as travel time. In the 2010  Census the rates reflected different geographic areas, and the country  was subdivided into three areas: urban/suburban, rural, and very rural.  According to Bureau officials, for the 2020 Census the Bureau will have  variable production rates based on geography, similar to the design used  in the 2010 Census. The Bureau told us they have not finalized the 2020  Census address canvassing production rates.

Table 2 shows the expected and actual productivity rates (addresses per  hour) for the in-field address canvassing operation at all three test sites.

To ensure address canvassing for the test was consistent with the 2020  Census, Bureau officials told us they included the blocks resolved during  the now discontinued second phase of in-office review, into the in-field  workload for the test. The Bureau attributed the greater productivity to this  discontinued second phase. Bureau officials told us that they believe that  listers spent less time updating those blocks because they had already  been resolved, and any necessary changes were already incorporated.  Moreover, while benefitting from the second phase of in-office address  canvassing may be one explanation for why listers were more productive.  Bureau officials told us that they are unable to evaluate the differences in  expected versus actual productivity for blocks added to the workload as a  result of the discontinued second phase because of limitations with the  data. However, there could be other reasons as well such as travel time  and geography. Standards for Internal Control require that organizations  use quality information to achieve their objectives. Therefore, continuing  to evaluate other factors from the 2018 End-to-End Test that may have  increased or could potentially decrease productivity will be important for  informing lister productivity rates for 2020, as productivity affects the  number of listers needed to carry out the operation, the number of staff  hours charged to the operation, and the number of laptops to be  procured.

For the 2018 End-to-End Test address canvassing operation, the Bureau  hired fewer listers than it assumed it needed at two sites and hired more  at the other site. In West Virginia, 60 percent of the required field staff  was hired and in Washington, 74.5 percent of the required field staff was  hired. Nevertheless, the operation finished on schedule at both these  sites. In contrast in Rhode Island the Bureau hired 112 percent of the  required field staff and finished early.

According to Bureau officials, both the West Virginia and Washington  state test sites started hiring field staff later than expected because of  uncertainty surrounding whether the Bureau would have sufficient funding  to open all three test sites for the 2018 End-to-End Test. When a decision  was made to open all three sites for the address canvassing operation  only, that decision came late, and Bureau officials told us that once they  were behind in hiring and were never able to catch up because of low  unemployment rates and the short duration of the operation. According  to Bureau officials, their approach to hiring for the 2018 End-to-End Test  was similar to that used for the 2010 and 2000 Censuses. In both  censuses the Bureau’s goal was to recruit and hire more workers than it  needed because of immutable deadlines and attrition.

After the 2010 Census we reported that the Bureau had over recruited;  conversely, for the 2000 Census the Bureau had recruited in the midst of  one of the tightest labor markets in three decades. Thus we  recommended, and the Bureau agreed to evaluate current economic  factors that are associated with and predictive of employee interest in  census work, such as national and regional unemployment levels, and  use these available data to determine the potential temporary workforce  pool and adjust its recruiting approach. The Bureau implemented this  recommendation, and used unemployment and 2010 Census data to  determine a base recruiting goal at both the Los Angeles, California and  Houston, Texas 2016 census test sites. Specifically, the recruiting goal for  Los Angeles was reduced by 30 percent.

Bureau officials told us that it continues to gather staffing data from the  2018 End-to-End Test that will be important to consider looking forward to  2020. Although address canvassing generally finished on schedule even  while short staffed, Bureau officials told us they are carefully monitoring  recruiting and hiring data to ensure they have sufficient staff for the test’s  next census field operation non-response follow-up, when census  workers go door-to-door to follow up with housing units that have not  responded. Non-response follow-up is set to begin in May 2018.  According to test data as of March 2018, the Bureau is short of its  recruiting goal for this operation which is being conducted in Providence  County, Rhode Island. The Bureau’s goal is to recruit 5,300 census  workers and as of March 2018, the Bureau had only recruited 2,732  qualified applicants to fill 1,166 spots for training and deploy 1,049 census  workers to conduct non-response follow-up. Bureau officials told us they  believe that low unemployment is making it difficult to meet its recruiting  goals in Providence County, Rhode Island, but they are confident they will  be able to hire sufficient staff without having to increase pay rates.

Recruiting and retaining sufficient staff to carry out operations as labor- intensive as address canvassing and nonresponse follow-up for the 2020  Census is a huge undertaking with implications for cost and accuracy.  Therefore, striking the right staffing balance for the 2020 Census is  important for ensuring deadlines are met and costs are controlled.

Resolving Challenges from the Address Canvassing Test Will Better Position the Bureau for the 2020 Census

The Bureau Does Not Have Procedures to Ensure All Collected Address Canvassing Data Are Retained

Bureau officials told us that during the test 11 out of 330 laptop computers  did not properly transmit address and map data collected for 25 blocks.  The lister-collected address file and map data are supposed to be  electronically transmitted from the listers’ laptops to the Bureau’s data  processing center in Jeffersonville, Indiana. The data are encrypted and  remain on the laptop until the laptops are returned to the Bureau where  the encrypted data are deleted. Prior to learning that not all data had  properly transmitted off the laptops, data on seven of the laptops was  deleted. Data on the remaining four laptops were still available. In  Providence, Rhode Island, where the full test will take place, the Bureau  recanvassed blocks where data were lost to ensure that the address and  map information for nonresponse follow-up was correct. Recanvassing  blocks increases costs and can lead to credibility problems for the Bureau  when listers visit a home twice.

Going into address canvassing for the End-to-End Test, Bureau officials  said they knew there was a problem with the LiMA software used to  update the Bureau’s address lists and maps. Specifically, address and  map updates would not always transfer when a lister transmitted their  completed work assignments from the laptop to headquarters. Other  census surveys using LiMA had also encountered the same software  problem. Moreover, listers were not aware that data had not transmitted  because there was no system-generated warning. Bureau officials are  working to fix the LiMA software problem, but told us that the software  problem has been persistent across other census surveys that use LiMA  and they are not certain it will be fixed.

Bureau officials told us that prior to the start of address canvassing they  created an alert report to notify Bureau staff managing the operation at  headquarters if data were not properly transmitted. When transmission  problems were reported, staff was supposed to remotely retrieve the data  that were not transmitted. This workaround was designed to safeguard  the data but according to officials was not used. Bureau officials told us  that they do not know whether this was because the alert reports were not  viewed by responsible staff or whether the alert report to notify the  Bureau staff managing the operation was not triggered. Bureau officials  told us they recognize the importance of following procedures to monitor  alert reports, and acknowledge that the loss of data on seven of the  laptops may have been avoided had the procedures that alert reports get  triggered and monitored been followed; however, officials did not know  why the procedures were not followed.

For 2020, if the software problem is not resolved, then officials said the  Bureau plans to create two new alert reports to monitor the transmission  of data. One report would be triggered when the problem occurs and a  second report would capture a one-to-one match between data on the  laptop and data transmitted to the data center so that discrepancies  would be immediately obvious. While these new reports should help  ensure that Bureau staff are alerted when data has not properly  transmitted, the Bureau has not determined and addressed why the  procedures that required an alert report get triggered and then reviewed  by Bureau staff did not work as intended. Standards for Internal Control  require that organizations safeguard data and follow policies and  procedures to achieve their objectives. Thus, either fixing the LiMA  software problem, or if the software problem cannot be fixed, then  determining and addressing why procedures that alert reports get  triggered and monitored were not followed would position the Bureau to  help prevent future data losses.

More Useful and Accurate Monitoring Data for Field Supervisors Would Strengthen Management of Operations

To effectively manage address canvassing, the Bureau needs to be able  to monitor the operation’s progress in near real time. Operational issues  such as listers not working assigned hours or falling behind schedule  need to be resolved quickly because of the tight time frames of the  address canvassing and subsequent operations. During the address  canvassing test, the Bureau encountered several challenges that  hindered its efforts to efficiently monitor lister activities as well as the  progress of the address canvassing operation.

System Alerts Were Not Consistently Used by Supervisors

The Bureau provides data-driven tools for the census field supervisors to  manage listers, including system alerts that identify issues that require the  supervisor to follow-up with a lister. For the address canvassing  operation, the system could generate 14 action codes that covered a  variety of operational issues such as unusually high or low productivity  (which may be a sign of fraud or failure to follow procedures) and  administrative issues such as compliance with overtime and completion of  expense reports and time cards.

During the operation, over 8,250 alerts were sent to CFSs or about 13  alerts were sent per day per CFS. Each alert requires the CFS to take  action and then record how the alert was resolved. CFSs told us and the  Bureau during debriefing sessions that they believed many of the  administrative alerts were erroneous and they dismissed them. For  example, during our site visit one CFS showed us an alert that incorrectly  identified that a timecard had not been completed. The CFS then showed  us that the lister’s timecard had indeed been properly completed and  submitted. CFSs we spoke to said that they often dismissed alerts related  to expense reports and timecards and did not pay attention to them or  manage them. Bureau officials reported that one CFS was fired for not  using the alerts to properly manage the operation.

To assist supervisors, these alerts need to be reliable and properly used.  Bureau officials said that they examined alerts for errors after we told  them about our observation. They reported that they did not find any  errors in the alerts. They believe that CFSs may not fully understand that  the alerts stay active until they are marked as resolved by the CFS. For  example, if a CFS gets an alert that a lister has not completed a timecard  the alert will remain active until the CFS resolves the alert by stating the  time card was completed. The Bureau’s current CFS manual does not  address that by the time a CFS sees the alert a lister may have already  taken action to resolve it. Because this was a reoccurring situation, CFSs  told us they had a difficult time managing the alerts.

Standards for Internal Control call for an agency to use quality information  to achieve objectives. Bureau officials acknowledge that it is a problem  that some CFSs view the alerts as erroneous and told us they plan to  address the importance of alerts in training. We spoke to Bureau officials  about making the alerts more useful to CFSs, such as by differentiating  between critical and noncritical alerts and streamlining alerts by perhaps  combining some of them. Bureau officials told us they would monitor the  alerts during the 2018 End-to-End Test’s nonresponse follow-up  operation and make adjustments if appropriate. However, while the  Bureau told us it will monitor alerts for the non-response follow-up  operation, the Bureau does not have a plan for how it will examine and  make alerts more useful.

Ensuring alerts are properly followed up on is critical to the oversight and  management of an operation. If the CFSs view the alerts as unreliable,  they could be likely to miss key indicators of fraud such as unusually high  or low productivity or an unusually high or low number of miles driven.  Moreover, monitoring overtime alerts and the submission of daily time  cards and expense reports is also important to ensure that overtime is  appropriately approved before worked and that listers get paid on time.

The Bureau’s Management Dashboard Did Not Always Display Accurate Information

Another tool the Bureau uses to monitor operations is its Unified Tracking  System (UTS), a management dashboard that combines data from a  variety of Census systems, bringing the data to one place where the  users can run or create reports. It was designed to track metrics such as  the number and percentage of blocks assigned and blocks completed as  well as the actual expenditures of an operation compared to the budgeted  expenditures. However, information in UTS was not always accurate  during address canvassing. For example UTS did not always report the  correct number of addresses assigned and completed by site. As a result,  Bureau managers reported they did not rely on UTS and instead used  data from the source systems that fed into it. Bureau officials agreed that  inaccurate data is a problem and that this workaround was inefficient as  users had to take extra time to go to multiple systems to get the correct  data.

Bureau officials reported problems importing information from the feeder  systems into UTS because of data mismatches. They said that address  canvassing event codes were not processed sequentially, as they should  have been, which led to inaccurate reporting. Bureau officials told us  that they did not specify that the codes needed to be processed in  chronological order as part of the requirements for UTS. Bureau officials  said UTS passed the requisite readiness reviews and tests. However,  Bureau officials also acknowledged that some of these problems could  have been caught by exception testing which was not done prior to  production.

To resolve this issue for 2020, Bureau officials stated they are developing  new requirements for UTS to automatically consider the chronological  order of event codes. The Bureau told us they are working on these UTS  requirements and will provide us with documentation when they are  complete. They also said the Bureau plans to implement a process which  compares field management reports with UTS reports to help ensure that  the reports have the same definitions and are reporting accurate  information. Standards for Internal Control call for an organization’s data  be complete and accurate and processed into quality information to  achieve their objectives. Thus, finalizing UTS requirements for the  address canvassing reporting should help increase efficiency for the 2020  Census by avoiding time consuming workarounds.

The Bureau Does Not Have Documented Procedures to Address Broadband Internet Service Coverage Gaps

The Bureau has taken significant steps to use technology to reduce  census costs. These steps include using electronic systems to transmit  listers’ assignments and address and map data. However, during the  address canvassing test, several listers and CFSs at the three test sites  experienced problems with Internet connections primarily during training.  The West Virginia site, which was more rural than the other sites,  experienced the most problems with Internet connectivity. All six West  Virginia CFSs reported Internet connectivity problems during the  operation. As a work around, CFSs told us that a couple of their listers  transmitted their work assignments from libraries where they could  access the Internet.

Bureau officials stated that the laptops in the 2018 End-to-End Test only  used two broadband Internet service providers, which may have  contributed to some of the Internet access issues. Bureau officials added  that despite the reported Internet connectivity issues, the 2018 End-to- End Test for address canvassing finished on schedule and without any  major problems. While this might be true for the test, we have previously  reported that minor problems can become big challenges when the  census scales up to the entire nation. Therefore, it is important that  these issues get resolved before August 2019 when in-field address  canvassing for the 2020 Census is set to begin.

The Bureau is analyzing the cellular network coverage across all 2020  Census areas using coverage maps and other methods to determine  which carrier is appropriate (including a backup carrier) for geographic  areas where network coverage is limited. According to Bureau officials,  they anticipate identifying the cellular carriers for each of its 248 area  census offices by the summer of 2018. The officials said they are  considering both national and regional carriers to provide service in some  geographic areas because the best service provider in a certain  geographic area may not be one of the national providers, but a regional  provider. In those cases, listers and other staff in those areas will receive  devices with the regional carrier. According to Bureau officials, for the  2020 Census, the ability to access multiple carriers should provide field  staff with better connectivity around the country.

We also found that there was no guidance for listers and CFSs on what to  do if they experienced Internet connectivity problems and were unable to  access the Internet. Bureau officials told us that staff in the field can use  different methods to access the Internet, such as using home wireless  networks or mobile hotspots located at libraries, or coffee shops to  transmit data. However, the Bureau did not provide such instructions to  listers. In addition, the Bureau also does not define what constitutes a  secure Internet public connection. Ensuring data are safeguarded is  important because census data are confidential. Bureau officials told us  that the Bureau plans to provide instructions to field staff on what to do if  they are unable to access census systems and what constitutes a secure  Internet connection for the next 2018 End-to-End Test field operation,  non-response follow-up. However, the Bureau has not finalized or  documented these instructions. Standards for Internal Control call for  management to design control activities, such as providing instructions to  employees to achieve objectives. Finalizing these instructions to field staff  will help ensure listers have complete information on how to handle  problems with Internet connectivity and that data are securely transmitted.

The Bureau Has Not Identified Alternative Sites for Listers to Take Online Training When Access to the Internet is Unavailable

Some listers had difficulty accessing the Internet to take online training for  address canvassing. This is the first decennial census that the Bureau is  using online training, in previous decennials training was instructor-led in  a class room. According to the Bureau, in addition to the Bureau provided  laptop, listers also needed a personal home computer or laptop and  Internet access at their home in order to complete the training. However,  while the Bureau reported that listers had access to a personal computer  to complete the training, we found some listers did not have access to the  Internet at their home and were forced to find workarounds to access the  training.

According to American Community Survey data from 2015, among all  households, 77 percent had a broadband Internet subscription. Bureau  officials told us they are aware that not all households have access to the  Internet and that the Bureau’s field division is working on back-up plans  for accessing online training. Specifically, Bureau officials told us for 2020  they plan to identify areas of the country that could potentially have  connectivity issues and plan to identify alternative locations such as  libraries or community centers where Internet connections are available to  ensure all staff has access to training. However, they have not finalized  those plans to identify locations for training sites. Standards for Internal  Control call for management to design control activities, such as having  plans in place to achieve objectives. Finalizing these plans to identify  alternative training locations will help ensure listers have a place to  access training.

The Bureau’s re-engineered approach for address canvassing shows  promise for controlling costs and maintaining accuracy. However, the  address canvassing operation in the 2018 End-to-End test identified the  need to reexamine assumptions and make some procedural and  technological improvements. For example, at a time when plans for in- field address canvassing should be almost finalized, the Bureau is in the  process of evaluating workload and productivity assumptions to ensure  sufficient staff are hired and that enough laptop computers are procured.  Moreover, Bureau officials have not finalized (1) procedures for  reassigning work from one lister to another to prevent the unnecessary  duplication of work assignments, (2) instructions for using the Internet  when connectivity is a problem to ensure listers have access to training  and the secure transmission of data to and from the laptops, and (3)  plans for alternate training locations. To ensure address and map data  are not lost during transmission, Bureau officials will also need to either  (1) fix the problem with the LiMA software used to update the address  and map files or (2) determine and address why procedures that alert  reports be triggered and monitored were not followed.

Finally, the Bureau has made progress in using data driven technology to  manage address canvassing operations. However, ensuring data used by  supervisors to oversee and monitor operations are both useful and  accurate will help field supervisors take appropriate action to address  supervisor alerts and will help managers monitor the real-time progress of  the address canvassing operation. With little time remaining it will be  important to resolve these issues. Making these improvements will better  ensure address canvassing for the actual enumeration, beginning in  August 2019, fully functions as planned and achieves desired results.

We are making the following seven recommendations to the Department  of Commerce and the Census Bureau:

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau continues to evaluate and finalize workload estimates  for in-field address canvassing as well as evaluates the factors that  impacted productivity rates during the 2018 End-to-End Test and, if  necessary, make changes to workload and productivity assumptions  before the 2020 Census in-field address canvassing operation to help  ensure that assumptions that impact staffing and the number of  laptops to be procured are accurate. (Recommendation 1)

Census Bureau finalizes procedures for reassigning blocks to prevent  the duplication of work. (Recommendation 2)

Census Bureau finalizes backup instructions for the secure  transmission of data when the Bureau’s contracted mobile carriers are  unavailable. (Recommendation 3)

Census Bureau finalizes plans for alternate training locations in areas  where Internet access is a barrier to completing training.  (Recommendation 4)

Census Bureau takes action to either fix the software problem that  prevented the successful transmission of data, or if that cannot be  fixed, then determine and address why procedures that alert reports  be triggered and monitored were not followed. (Recommendation 5)

Census Bureau develops a plan to examine how to make CFS alerts  more useful so that CFSs take appropriate action, including alerts a  CFS determines are no longer valid because of timing differences.  (Recommendation 6)

Census Bureau finalizes UTS requirements for address canvassing  reporting to ensure that the data used by census managers who are  responsible for monitoring real-time progress of address canvassing  are accurate before the 2020 Census. (Recommendation 7)

We provided a draft of this report to the Department of Commerce. In its  written comments, reproduced in appendix I the Department of  Commerce agreed with our recommendations. The Census Bureau also  provided technical comments that we incorporated, as appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we are sending copies of this report to the  Secretary of Commerce, the Under Secretary of Economic Affairs, the  Acting Director of the U.S. Census Bureau, and interested congressional  committees. The report also will be available at no charge on GAO’s  website at http://www.gao.gov.

If you have any questions about this report please contact me at (202)  512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff that made major contributions to this report are  listed in appendix II.

Appendix I: Comments from the Department of Commerce

Appendix II: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Lisa Pearson, Assistant Director;  Kate Wulff, Analyst-in-Charge; Mark Abraham; Devin Braun; Karen  Cassidy; Robert Gebhart; Richard Hung; Kirsten Lauber; Krista Loose; Ty  Mitchell; Kayla Robinson; Kate Sharkey; Stewart Small; Jon Ticehurst;  and Timothy Wexler made key contributions to this report.

The success of the census depends in large part on the Census Bureau's ability to record every location where a person resides. An ever-changing housing inventory and "hidden" housing—such as a converted attic or shed that may not show up in records—make this huge task even more difficult.

As part of a test run, census workers went door-to-door to try to record each housing unit in selected areas in 3 states. We reviewed this effort and found the workers generally followed procedures but software problems sometimes kept them from transmitting data they collected.

We made 7 recommendations for the 2020 Census.