2020 Census: Census Bureau Improved the Quality of Its Cost Estimation but Additional Steps Are Needed to Ensure Reliability

In October 2017, the Department of Commerce (Commerce) announced that the projected life-cycle cost of the 2020 Census had climbed to $15.6 billion, a more than $3 billion (27 percent) increase over its 2015 estimate. A high-quality, reliable cost estimate is a key tool for budgeting, planning, and managing the 2020 Census. Without this capability, the Bureau is at risk of experiencing program cost overruns, missed deadlines, and performance shortfalls.

GAO was asked to evaluate the reliability of the Bureau's life-cycle cost estimate. This report evaluates the reliability of the Bureau's revised life-cycle cost estimate for the 2020 Census and the extent to which the Bureau is using it as a management tool, and compares the 2015 and 2017 cost estimates to describe key drivers of cost growth. GAO reviewed documentary and testimonial evidence from Bureau officials responsible for developing the 2020 Census cost estimate and used its cost assessment guide ( GAO-09-3SP ) as criteria.

Since 2015, the Census Bureau (Bureau) has made significant progress in improving its ability to develop a reliable cost estimate. While improvements have been made, the Bureau's October 2017 cost estimate for the 2020 Census does not fully reflect all the characteristics of a reliable estimate. (See figure.) Specifically, for the characteristic of being well-documented, GAO found that some of the source data either did not support the information described in the cost estimate or was not in the files provided for two of its largest field operations. In GAO's assessment of the 2015 version of the 2020 Census cost estimate, GAO recommended that the Bureau take steps to ensure that each of the characteristics of a reliable cost estimate is met. The Bureau agreed and has taken steps, but has not fully implemented this recommendation.

A reliable cost estimate serves as a tool for program development and oversight, helping management make informed decisions. During this review, GAO found the Bureau used the cost estimate to inform decision making.

Factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include:

Changes in assumptions. Among other changes, a decrease in the assumed rate for self-response from 63.5 percent in 2015 to 60.5 percent in 2017 increased the cost of collecting responses from nonresponding housing units.

Improved ability to anticipate and quantify risk. In general, contingency allocations designed to address the effects of potential risks increased overall from $1.3 billion in 2015 to $2.6 billion in 2017.

An overall increase in information technology (IT) costs. IT cost increases, totaling $1.59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017.

GAO is not making any new recommendations but maintains its earlier recommendation—that the Secretary of Commerce direct the Bureau to take specific steps to ensure its cost estimate meets the characteristics of a high-quality estimate. In its response to this report, Commerce generally agreed with the findings related to cost estimation improvements, but disagreed that the cost estimate was not reliable. However, until GAO's recommendation is fully implemented the cost estimate cannot be considered reliable.

The U.S. Census Bureau (Bureau) estimates that the 2020 Census will be  the most expensive census in our nation’s history, at an estimated cost of  $15.6 billion after adjusting for inflation. The Bureau faces the challenge  of cost effectively counting a population that is growing steadily larger,  more diverse, and increasingly difficult to enumerate with a reengineered  design that relies in part on automation to locate housing units and count  the nation’s population. In an environment of constrained resources,  containing costs is imperative for the 2020 Census.

In February 2017, we designated the 2020 Census as a high-risk area, in  part because of uncertainty over costs and long-standing weaknesses in  the Bureau’s management of information technology intended to  automate the census. At the time the 2020 Census was designated high  risk, the life-cycle cost for the 2020 Census was $12.5 billion, a figure  since revised downward to $12.3 billion. However, in October 2017, the  Department of Commerce (Commerce) announced that it had updated its  October 2015 life-cycle cost estimate and now projects the life-cycle cost  of the 2020 Census will be $15.6 billion, a more than $3 billion (27  percent) increase over its earlier estimate. The higher estimated life-cycle  cost is due, in part, to the Bureau’s earlier failure to meet best practices  for a quality cost estimate, a concern we had reported in June 2016.

Reliable cost estimates that appropriately account for risks facing an  agency can help an agency manage large complex activities like the 2020  Census and can also help Congress provide oversight and make funding  decisions. Having a realistic estimate of projected costs makes for  effective resource allocation, and it increases the probability of a  program’s success.

You requested that we evaluate the reliability of the life-cycle cost  estimate the Bureau submitted to Congress in October 2017. This report  (1) evaluates the reliability of the Bureau’s revised life-cycle cost estimate  for the 2020 Census, and the extent to which the Bureau is using it as a  management tool; and (2) compares the 2015 and 2017 life-cycle cost  estimates to describe key drivers of cost growth.

For both objectives, we reviewed documentation related to the cost  estimate and interviewed Bureau officials responsible for developing the  2020 Census life-cycle cost estimate. For the first question, we  interviewed Bureau officials and evaluated whether the Bureau’s cost  estimate was generated according to best practices of our Cost  Estimating and Assessment Guide. We also assessed the extent to  which the Bureau is using the cost estimate as a management tool. For  the second question, we compared cost information included in the 2015  and 2017 cost estimates. We relied on our cost assessment guide as  criteria. More information on our scope and methodology can be found in  appendix I.

We conducted this performance audit from December 2017 to August  2018 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

A high-quality, reliable cost estimate is a key tool for budgeting, planning,  and managing the 2020 Census. According to OMB, programs must  maintain current and well-documented estimates of program costs, and  these estimates must encompass the full life-cycle of the program.  Among other things, OMB states that generating reliable program cost  estimates is a critical function necessary to support OMB’s capital  programming process. Without this capability, agencies are at risk of  experiencing program cost overruns, missed deadlines, and performance  shortfalls.

A reliable cost estimate is critical to the success of any federal  government program. With the information from reliable estimates,  managers can: make informed investment decisions, allocate program resources, measure program progress, proactively correct course when warranted, and ensure overall accountability for results.

To be considered reliable, a cost estimate must meet the criteria for each  of the four characteristics outlined in our Cost Estimating and Assessment  Guide. According to our analysis, a cost estimate is considered reliable if  the overall assessment ratings for each of the four characteristics are  substantially or fully met. If any of the characteristics are not met,  minimally met, or partially met, then the cost estimate does not fully  reflect the characteristics of a high-quality estimate and cannot be  considered reliable. Those characteristics are:

Well-documented: An estimate is thoroughly documented, including  source data and significance, clearly detailed calculations and results,  and explanations of why particular methods and references were  chosen. Data can be traced to their source documents.

Accurate: An estimate is unbiased, the work is not overly  conservative or overly optimistic, and is based on an assessment of  most likely costs. Few, if any, mathematical mistakes are present.

Credible: Any limitations of the analysis because of uncertainty or  bias surrounding data or assumptions are discussed. Major  assumptions are varied, and other outcomes are recomputed, to  determine how sensitive they are to changes in the assumptions. Risk  and uncertainty analysis is performed to determine the level of risk  associated with the estimate. The estimate’s results are cross- checked, and an independent cost estimate (ICE) is conducted to see  whether other estimation methods produce similar results.

Comprehensive: An estimate has enough detail to ensure that cost  elements are neither omitted nor double counted. All cost-influencing  ground rules and assumptions are detailed in the estimate’s  documentation.

Meeting best practices outlined in our Cost Estimating and Assessment  Guide for a reliable cost estimate has been a long-standing challenge for  the Bureau. In 2008 we reported that the 2010 Census cost estimate was  not reliable because it lacked documentation and was not  comprehensive, accurate, or credible. For example, in our 2008 report on  the Bureau’s cost estimation process, Bureau officials were unable to  provide documentation that supported the assumptions for the initial 2001  life-cycle cost estimate as well as the updates. Consequently, we  recommended that the Bureau establish guidance, policies, and  procedures for estimating costs that would meet best practices criteria.  The Bureau agreed with the recommendation and said at the time that it  already had efforts underway to improve its future cost estimation  methods and systems. Moreover, weaknesses in the life-cycle cost  estimate were one reason we designated the 2010 Census a GAO High- Risk Area in 2008.

In 2012 we reported that, while the Bureau was taking steps to strengthen  its life-cycle cost estimates, it had not yet established guidance for  developing cost estimates. We recommended that the Bureau finalize its  guidance, policies, and procedures for cost estimation in accordance with  best practices. The Bureau agreed with the overall theme of the report  but did not comment on the recommendation. During this review we found  that the Bureau took steps to address this recommendation, which is  discussed later in this report. Such guidance can help to institutionalize  best practices and ensure consistent processes and operations for  producing reliable estimates.

In a 2016 report we found that the October 2015 version of the Bureau’s  life-cycle cost estimate for the 2020 Census was not reliable. Overall,  we reported that the 2020 Census life-cycle cost estimate partially met  two of the characteristics of a reliable cost estimate (comprehensive and  accurate) and minimally met the other two (well-documented and  credible). We recommended that the Bureau take specific steps to ensure  its cost estimate meets the characteristics of a high-quality estimate. The  Bureau agreed with this recommendation, and took steps to improve the  reliability of its cost estimate, which we focus on later in this report.  Consequently, an unreliable life-cycle cost estimate is one of the reasons  we designated the 2020 Census a GAO High-Risk Area in 2017.

In October 2015, the Bureau estimated the cost of the 2020 Census to be  $12.3 billion. According to the Bureau, the October 2015 version was the  Bureau’s first attempt to model the life-cycle cost of its planned 2020  Census, in contrast to its earlier 2011 estimate, which the Bureau said  was intended to produce an approximation of potential savings and to  begin developing the methodology for producing decennial life-cycle cost  estimates covering all phases of the decennial life cycle. To help control  costs while maintaining accuracy, the Bureau introduced significant  change to how it conducts the decennial census in 2020. Its planned  innovations include reengineering how it builds its address list, improving  self-response by encouraging the use of the Internet and telephone, using  administrative records to reduce field work, and reengineering field  operations using technology to reduce manual effort and improve  productivity. In contrast to the estimated $12.3 billion in 2015, the 2020  Census would cost $17.8 billion in constant 2020 dollars if the Bureau  repeated the 2010 Census design and methods, according to the  Bureau’s estimates.

In October 2017, Commerce announced that it had updated the October  2015 life-cycle cost estimate, projecting the life-cycle cost of the 2020  Census to be $15.6 billion, an increase of over $3 billion (27 percent) over  its 2015 estimate. (See figure 1.) In developing the 2017 version of the  cost estimate, Bureau cost estimators identified cost inputs, their ranges  for possible outcomes, and overall cost estimating relationships (i.e.,  logical or mathematical formulas, or both). To identify cost inputs and the  ranges of potential outcomes, the Bureau worked with subject matter  experts and used historical data to support assumptions and generate  inputs. The Bureau’s cost estimation team used a software tool to  generate the cost estimate.

Because cost estimates predict future program costs, uncertainty is  always associated with them. For example, data from the past (such as  fuel prices) may not always be relevant in the future. Risk and uncertainty  refer to the fact that because a cost estimate is a forecast, there is always  a chance that the actual cost will differ from the estimate. One way to  determine whether a program is realistically budgeted is to perform an  uncertainty analysis, so that the probability associated with achieving its  point estimate can be determined, usually relying on simulations such as  those of Monte Carlo methods. This can be particularly useful in  portraying the uncertainty implications of various cost estimates.

Consistent with cost estimation practices outlined in our Cost Estimating  and Assessment Guide, the estimate was compared with two  independent cost estimates (ICE), developed by Commerce’s Office of  Acquisition Management (OAM) and the Bureau’s Office of Cost  Estimation, Analysis, and Assessment. The offices producing the ICEs  and the cost estimate team worked together to examine the process each  used, an effort known as the reconciliation process. Through this  reconciliation, the Bureau identified areas where discrepancies existed  and elements that could require additional review and possible  improvement.

According to Bureau documentation the estimate will be updated as the  program meets milestones and to reflect changes in technical or program  assumptions. Figure 2 details the Bureau’s cost estimation process. OAM  was involved extensively in the development of the 2017 estimate, an  increased involvement compared to 2015, according to Bureau officials.  OAM participated in regular review meetings throughout the development  of the estimate and also developed an independent cost estimate, as  shown in the figure below.

End-to-end system testing activities for the 2020 Census are currently  underway in Providence, Rhode Island. According to the Bureau,  information collected from the test, such as overall response rates and  the use of administrative records to inform census records, will inform  future versions of the life-cycle cost estimate. Some updates from the test  will be incorporated into the next cost estimate, which will be available in  the first quarter of the coming fiscal year.

Census Bureau Has Made Progress but Has Not Taken All the Steps Needed to Ensure the Reliability of 2020 Cost Estimate

Since our June 2016 report, in which we reviewed the Bureau’s 2015  version of the cost estimate, the Bureau has made significant progress.  For example, the Bureau has put into place a work breakdown structure  (WBS) that defines the work, products, activities, and resources  necessary to accomplish the 2020 Census and is standardized for use in  budget planning, operational planning, and cost estimation. However, the  Bureau’s October 2017 cost estimate for the 2020 Census does not fully  reflect characteristics of a high-quality estimate as described in our Cost  Estimating and Assessment Guide and cannot be considered reliable.

Our Cost Estimating and Assessment Guide describes best practices for  developing reliable cost estimates. For our reporting needs, we collapsed  these best practices into four characteristics for sound cost estimating— comprehensive, well-documented, accurate, and credible—and identified  specific best practices for each characteristic. To be considered reliable,  an organization must meet or substantially meet each characteristic. Our  review found the Bureau met or substantially met three out of the four  characteristics of a reliable cost estimate, while it partially met one  characteristic: well-documented. When compared to the October 2015  estimate, the 2017 estimate shows considerable improvement. (See  figure 3 below.)

Cost estimates are considered valid if they are well-documented to the  point they can be easily repeated or updated and can be traced to original  sources through auditing, according to best practices. The Bureau only  partially met the criteria for well-documented, as set forth in our Cost  Estimating and Assessment Guide. A cost estimate that does not fully  meet the criteria for well-documented cannot be used by management to  make informed and effective implementation decisions.

The well-documented characteristic comprises five best practices. The  Bureau substantially met two out of five best practices (as shown in figure  4). First, the estimate describes in sufficient detail the calculations  performed and the estimating methodology used to derive each element’s  cost, and the cost estimate had been reviewed by management. Since  cost estimates can inform key decisions and budget requests, it is vital  that management review and understand how the estimate was  developed, including risks associated with the underlying data and  methods.

The cost estimate only partially met three best practices for the  characteristic of being well-documented. In general, some documentation  was missing, inconsistent, or difficult to understand. First, we found that  source data did not always support the information described in the basis  of estimate document or could not be found in the files provided for two of  the Bureau’s largest field operations: Address Canvassing and Non- Response Follow-Up (NRFU). For example, the cost estimate  documentation referred to actual data from the 2010 Census and  information obtained from experts as sources for address canvassing  rework rates. However, the folder source documents provided as  support for the basis of estimate did not include this information. Next, in  several cases, we could not replicate calculations, such as for mileage  costs, using the description provided. Lastly, we found that some of the  cost elements did not trace clearly to supporting spreadsheets and  assumption documents.

Failure to document an estimate in enough detail makes it more difficult to  replicate calculations, or to detect possible errors in the estimate; reduces  transparency of the estimation process; and can undermine the ability to  use the information to improve future cost estimates or even to reconcile  the estimate with another independent cost estimate. The Bureau told us  it would continue to make improvements to ensure the estimate is well- documented. For the estimate to be considered well-documented, the  Bureau will need to address these issues.

An accurate cost estimate supports measurement of program progress by  providing unbiased and correct data, which can help management ensure  accountability for scheduled results. We found the Bureau’s cost estimate  substantially met the criteria for accuracy. As shown in figure 5, and in  line with best practices outlined in our Cost Estimating and Assessment  Guide, the estimate was not overly optimistic; appeared to be free of  errors; was based on historical data or input from subject matter experts;  and, according to Bureau officials, is updated regularly as information  becomes available.

The Bureau can enhance the accuracy of their estimate by increasing the  level of detail included in the documentation, such as detail on specific  inflation indices used, and by monitoring actual costs against estimates.  We identified areas for improvement, which, according to Bureau officials,  will be addressed as part of its ongoing efforts. For example, while the  basis of estimate document describes different inflation indexes, it was  not clear exactly which indexes were applied to the various cost elements  in the estimate. Also, evidence of how variances between estimated costs  and actual expenses would be tracked over time was not available at the  time of our analysis. Tools to track variance enable management to  measure progress against planned outcomes. Bureau officials stated that  they already have systems in place that can be adapted for tracking  estimated and actual costs.

All estimates include a certain amount of informed judgment about the  future. Assumptions made at the start of a program can turn out to be  inaccurate. Credible cost estimates identify limitations due to uncertainty  or bias surrounding data or assumptions, and control for these  uncertainties by identifying and quantifying cost elements that represent  the most risk. We found that the Bureau’s cost estimate substantially met  the criteria for credible, as shown in figure 6 below.

The Bureau’s cost estimate clearly identifies risks and uncertainties, and  describes approaches taken to mitigate them. In line with best practices  outlined in our Cost Estimating and Assessment Guide, the Bureau did  the following:

Sensitivity analysis. The Bureau conducted sensitivity analysis to  identify possible changes to estimated costs for the 2020 Census  based on varying major assumptions, parameters, and data inputs.  For example, the Bureau calculated the likely cost implications for a  range of possible response rates to identify a range of projected costs  and to calculate appropriate reserves for risk. Bureau officials stated  that they also identified the estimate input parameters that contributed  the most to estimate uncertainty.

Risk and uncertainty analysis. A cost estimate is a forecast, and as  such, there is always a chance that the actual cost will differ from the  estimate. Uncertainty is the indefiniteness about the outcome of a  situation. Uncertainty is assessed in cost estimate models to estimate  the risk (or probability) that a specific funding level will be exceeded.  We found the Bureau performed an uncertainty analysis on a portion  of the estimate to determine whether estimated costs were realistic  and to establish the probability of achieving projections outlined in the  estimate. The Bureau used a combination of modeling based on  Monte Carlo analysis and allocations of funding for risks. The Monte  Carlo simulation was performed on a portion of the estimate to  account for uncertainty around various operational parameters for  which a range of outcomes was possible, including Internet response  rates and the extent to which data collection issues might be resolved  using administrative records. To account for the inherent uncertainty  of assumptions included within the life-cycle cost estimate, the Bureau  added funding to the cost estimate totaling approximately $292 million  to account for risks based on the results of the Monte Carlo analysis.

For other risks, such as acquisition lead time and the possibility of  delays in information technology (IT) development, contingency  funding was added to the estimate to reflect the potential cost of  resolving these issues, through use of a backup system or an  alternative approach. These are described as “special risks” in the  Bureau’s basis of estimate, and total approximately $171 million.

Based on additional sensitivity analysis, the Bureau added  approximately $965 million to the cost estimate to reflect discrete risks  outlined in the risk register as well as those associated with (1)  variability in self-response rates, (2) the effect of fluctuations in the  size and wage rate of the temporary workforce on the cost of field  operations, and (3) the potential need to reduce the enumerator-to- manager staffing ratio in case expected efficiencies in field operations  are not realized.

In addition to these provisions, the Secretary of Commerce added a  contingency amount of about $1.2 billion to account for what the  Bureau refers to as unknown-unknowns. Bureau documentation  states that conducting a decennial census is an extremely complex,  high-risk operation. In order to mitigate some of the risk, contingency  funding must be available to initiate ad hoc activities necessary to  overcome unforeseen issues. According to Bureau documentation  these include such risks as natural disasters or cyber-attacks. The  Bureau provides a description of how the general risk contingency is  calculated. However, this description does not clearly link calculated  amounts to the risks themselves.

In our June 2016 report we reported the Bureau had not properly  accounted for risk and recommended the Bureau, in part; improve  control over how risk and uncertainty are accounted for. We continue  to believe the prior recommendation from our June 2016 report  remains valid and should be addressed: that the Bureau properly  account for risk in the 2020 Census cost estimate, among other  things. As such, risks need to be linked to the $1.2 billion general risk  contingency fund.

Independent cost estimate. According to best practices outlined in  our Cost Estimating and Assessment Guide, an independent cost  estimate should be performed to determine whether alternate  estimate approaches produce similar results. The Bureau compared  their estimate with two independent cost estimates, developed by  Commerce’s Office of Acquisition Management and the Bureau’s  Office of Cost Estimation and Assessment. As part of their process for  finalizing the cost estimate, Bureau officials reconciled differences  between the estimates in discussions with the two offices, resulting in  more conservative assumptions by the Bureau around risk and  uncertainty in both cases.

In addition to implementing our recommendation to properly account  for risk, going forward, while the Bureau substantially met the  credibility characteristic it will be important for them to also integrate  regular cross-checks of methodology into their cost estimation  process. In our analysis we observed that no specific cross-checks  of cost methodology were performed. According to the Bureau, cross- checks were not performed because the Bureau considered the  independent cost estimates as overall cross-checks on the reliability  of their methodology and did not conduct additional cross-checks. The  main purpose of cross-checking is to determine whether alternative  methods for specific cost elements within the cost estimate could  produce similar results. An independent cost estimate, though  important for the credibility of an estimate, does not fulfill the same  function as a targeted cross-check of individual elements.

Comprehensive estimates have enough detail to ensure that cost  elements are neither omitted nor double-counted, all cost-influencing  assumptions are detailed in the estimate’s documentation, and a work  breakdown structure is defined. Our analysis of the 2017 cost estimate  demonstrates improvement over the 2015 cost estimate when the  Bureau’s cost estimate only partially met the criteria for comprehensive.

We found the Bureau met or substantially met all four best practices for  the comprehensive characteristic, as shown in figure 7. For example, all  life-cycle costs are included in the estimate along with a complete  description of the 2020 Census program and current schedule. We also  found that the Bureau substantially met criteria for documenting cost  influencing ground rules and assumptions.

A standardized WBS (as detailed in table 1) with supporting dictionary  outlines the major work of the program and describes the activities and  deliverables at the project level where costs are tracked. In 2016, the  Bureau’s WBS did not contain sufficient detail and we found significant  differences in the presentation of the work between sources. In 2017,  based on our review of Bureau documentation and interviews with Bureau  officials, we found that the WBS is standardized and cost elements are  presented in detail. The WBS is a necessary program management tool  because it provides a basic framework for a variety of related tasks like  estimating costs, developing schedules, identifying resources,  determining where risks may occur, and providing the means for  measuring program status.

Although the Bureau’s updated life-cycle cost estimate reflects three of  the four characteristics of a reliable cost estimate, we are not making any  new recommendations to the Bureau in this report. We continue to  believe the prior recommendation, made in 2016, remains relevant: that  the Secretary of Commerce ensure that the Bureau finalizes the steps  needed to fully meet the characteristics of a high-quality estimate, most  notably in the well-documented area. The Bureau told us it has used our  best practices for cost estimation to develop their cost estimate, and will  focus on those best practices that require attention moving forward.  Without a reliable cost estimate, the Bureau is limited in its ability to make  informed decisions about program resources and to effectively measure  progress against operational objectives.

Life-Cycle Cost Estimate Is Used by Management to Inform Decisions

OMB, in its guidance for preparing and executing agency budgets, cites  that credible cost estimates are vital for sound management decision  making and for any program or capital project to succeed. A well- developed cost estimate serves as a tool for program development and  oversight, supporting management to make informed decisions.

According to the Bureau, the 2020 Census cost estimate is used as a  management tool to guide decision making. Bureau officials stated the  cost estimate is used to examine the cost impact of program changes.  For example, the cost estimate served as the basis for the fiscal year  2019 funding request developed by the Bureau. The Bureau also said it  used the 2020 Census life-cycle cost estimate to establish cost controls  during budget formulation activities and to monitor spending levels for  fiscal year 2019 activities. According to the Bureau, as detailed  operational and implementation plans are defined, the 2020 Census life- cycle cost estimate has been and will continue to be used to support  ongoing “what-if” analyses in determining the cost impacts of design  decisions. Specifically, using the cost estimate to model the impact of  changes on overall cost, the Bureau adjusted the scope of the Census  Enterprise Data Collection and Processing (CEDCaP) operation.

Census Bureau Guidance to Develop Cost Estimates Meets Best Practices

The processes for developing and updating estimates are designed to  inform management about program progress and the use of program  resources, supporting cost-driven planning efforts and well-informed  decision making. Our work has identified a number of best practices for  use in developing guidance related to cost estimation and analysis that  are the basis of effective program cost estimating and should result in  reliable and valid cost estimates that management can use for making  informed decisions.

In 2012 we reported that the Bureau had not yet established guidance for  developing cost estimates. We recommended that the Bureau establish  guidance, policies, and procedures for developing cost estimates that  would meet best practice criteria. The Bureau agreed with the theme of  the report but did not specifically agree with the recommendation.  Moreover, in June 2016, we also reported that the cost estimation team  did not record how and why it changed assumptions that were provided to  it and did not document the sources of all data it used. The  documentation of these changes to assumptions did not happen because  the Bureau lacked written guidance and procedures for the cost  estimation team to follow. During this review we found the Bureau has  since established reliable guidance, processes, and policies for  developing cost estimates and managing the cost estimation process.  The following documents, shown in table 2, establish roles and  responsibilities for oversight and approval of cost estimation processes,  provide a detailed description of the steps taken to produce a high-quality  cost estimate, and clarify the process for updating the cost estimate and  associated documents over the life of a project.

The Decennial Census Program’s Cost Estimate and Analysis Process,  which provides a detailed description of the steps taken to produce a  high-quality estimate, is reliable as it met the criteria for 8 steps and  substantially met the criteria for 4 steps of the 12 best steps outlined in  our Cost Estimating and Assessment Guide, as shown below in figure 8.  To avoid cost overruns and to support high performance, it will be  important for the Bureau to abide by their newly developed policies and  guidance and continue to use the life-cycle cost estimate as a  management tool.

Increased Costs Are Driven by an Assumed Decrease in Self-Response Rates and Increases in Contingency Funds and IT Cost Categories

The 2017 life-cycle cost estimate includes significantly higher costs than  those included in the 2015 estimate. In 2015, the Bureau estimated that  they could conduct the operation at a cost of $12.3 billion in constant  2020 dollars. The Bureau’s latest cost estimate, announced in October  2017, reflects the same design, but at an expected cost of $15.6 billion.  Figure 9 below shows the change in cost by WBS category for 2015 and  2017. The largest increases occurred in the Response, Managerial  Contingency, and Census/Survey Engineering categories. Increased  costs of $1.3 billion in the response category (costs related to collecting,  maintaining, and processing survey response data) were in part due to  reduced assumptions for self-response rates, leading to increases in the  amount of data collected in the field, which is more costly to the Bureau.  Contingency allocations increased overall from $1.35 billion in 2015 to  $2.6 billion in 2017, as the Bureau gained a greater understanding of  risks facing the 2020 Census. Increases of $838 million in the  Census/Survey Engineering category were due mainly to the cost of an IT  contract for integrating decennial survey systems that was not included in  the 2015 cost estimate. Bureau officials attribute a decrease of $551  million in estimated costs for Program Management to changes in the  categorization of costs associated with risks: In the 2017 version of the  estimate, estimated costs related to program risks were allocated to their  corresponding WBS element.

More generally, factors that contributed to cost fluctuations between the  2015 and 2017 cost estimates include:  changes in assumptions for census operations,  improved ability to anticipate and quantify risk, an overall increase in IT costs, and more defined contract requirements.

Several assumptions for the implementation of the 2020 Census have  changed since the 2015 cost estimate. Some assumptions contributing to  cost changes, mainly in the Response (related to collecting and  processing response data) and Frame (the mapping and collecting  addresses to frame enumeration activities) categories, include the  following:

Self-response rates. Changes in assumptions for expected self- response rates contributed to increases in the response category, as  the assumed rate decreased from 63.5 percent in 2015 to 60.5  percent in 2017, thereby increasing the anticipated percentage and  associated cost of nonresponse follow-up. When the Bureau does  not receive responses by mail, phone, or Internet, census  enumerators visit each nonresponding household to obtain data.  Thus, reduced self-response rates lead to increases in the amount of  data collected in the field, which is more costly to the Bureau. Bureau  officials attributed this decrease to a forecasted reduction in Internet  response due to added authentication steps at log in and the  elimination of the function allowing users to save their responses and  return later to complete the survey.

Productivity rates. The productivity of enumerators collecting data  for NRFU is another variable in the cost estimate that was updated,  contributing to cost increases in the response category. Expected  productivity rates for NRFU decreased from the 2015 estimate of 4  attempts per hour to 2.9. According to Bureau documentation, this  more conservative estimate is based on historical data, rather than  research and test data.

In-office address canvassing rates. The Bureau will not go door-to- door to conduct in-field address canvassing across the country to  update address and map information for every housing unit, as it has  in prior decennial censuses. Rather, some areas would only need a  review of their address and map information using computer imagery  and third-party data sources—what the Bureau calls “in-office”  address canvassing procedures. However, in March 2017, citing  budget uncertainty the Bureau decided to discontinue one of the  phases of in-office review address canvassing for the 2020 Census.  The cancellation of that phase of in-office review is expected to  increase the number of housing units canvassed in-field by 5 percent  (from 25 to 30 percent of all canvassed housing units). In-field  canvassing is more labor intensive compared to in office procedures.  The 2017 version of the cost estimate reflects this increase in  workload for in-field address canvassing, though overall changes in  estimated costs for the Frame category, of which Address Canvassing  is a part, were minimal.

Staffing. Updated analysis resulted in changes to several staffing  assumptions, which resulted in decreases across WBS categories.  Changes included reduced pay rates for field data collection staff  based on current labor market conditions and reductions in the length  of staff engagement.

In general, contingency allocations increased overall from $1.35 billion in  2015 to $2.6 billion in 2017. This increase in contingency can be  attributed, in part, to the Bureau gaining a clearer understanding of risk  and uncertainty in the 2020 Census as it approaches. The Bureau  developed some of its contingency based on proven risk management  techniques, including Monte Carlo analysis and allocated funding for  known risk scenarios. The 2017 estimate includes close to $1.4 billion in  estimated costs for these risks, almost three times the amount included in  the 2015 estimate. The basis of estimate contains detail on the various  risks and the process for calculating the associated contingency. The  2017 version also includes a contingency amount of $1.2 billion for  general risks, or unknown-unknowns, such as natural disasters and  cyber-attacks.

Contingency amounts were reallocated within the WBS to more closely  reflect the nature of the risk: Bureau officials attribute a decrease from the  2015 estimate of $551 million in estimated costs for program  management to changes in the categorization of costs associated with  risks. Officials stated that, in 2015, discrete program risks were previously  consolidated as program management costs. In 2017, these discrete  costs were reallocated to associate risks with the appropriate WBS  element. For example, contingency amounts related to the likelihood of  achieving a certain response rate previously included in the program  management work breakdown category are now a part of the “response”  work breakdown category.

Increases in IT costs, totaling $1.59 billion, represented almost 50 percent  of the total cost increase from 2015 to 2017. The total share of IT costs as  a percentage of total census costs increased from 28 percent in 2015 to  32 percent in 2017, or from $3.41 billion to approximately $5 billion.  Increases in IT costs are spread across seven cost categories. Figure 10  shows the IT and non-IT cost by WBS for the 2017 cost estimate. IT costs  in infrastructure, response data, and census/survey WBSs account for the  majority of the approximately $5 billion.

The Bureau’s October 2015 cost estimate included IT costs for, among  other things, system engineering, test and evaluation, and infrastructure,  as well as for a portion of the Census Enterprise Data Collection and  Processing (CEDCaP) program. The 2017 estimated IT cost increases  were due, in large part, to the Bureau (1) updating the cost estimate for  CEDCaP; (2) including an estimate for technical integration services that  contributed to increases in the Census and Survey Engineering category;  and (3) updating costs related to other major contracts (such as mobile  device as a service, field IT services, and payroll systems).

Bureau documents described an overall improvement in the Bureau’s  ability to define and specify contract requirements. This resulted in  updated estimates for several contracts, including for the Census  Questionnaire Assistance (CQA) contract. Assumptions regarding call  volume to the CQA were increased by 5 percent to account for expected  response by phone after the elimination of the option to save Internet  responses and return to complete the form later. The Bureau also cited  updated cost data and the results of reconciliation with independent cost  estimates as factors contributing to the increased costs of other major  contracts, including for the procurement of data collection devices.

The Secretary of Commerce provided comments on a draft of this report  on August 2, 2018. The comments are reprinted in appendix II.  The  Department of Commerce generally agreed with our findings regarding  the improvements the Census Bureau has made in its cost estimates.  However, Commerce did not agree with our assessment that the  Bureau’s 2017 lifecycle cost estimate is “not reliable.”  Commerce noted  that it had conducted two independent cost analyses and was satisfied  that the cost estimate was reliable.  The Bureau also provided technical  comments that we incorporated, as appropriate.

We maintain that, to be considered reliable, a cost estimate must meet or  substantially meet the criteria for each of the four characteristics outlined  in our Cost Estimating and Assessment Guide.  These characteristics are  derived from measures consistently applied by cost estimating  organizations throughout the federal government and industry and are  considered best practices for the development of reliable cost estimates.   Without a reliable cost estimate, the Bureau is limited in its ability to make  informed decisions about program resources and to effectively measure  progress against operational objectives. Thus, while the Bureau has  made considerable progress in all four of the characteristics, it has only  partially met the criteria for the characteristic of being well-documented.   Until the Bureau meets or substantially meets the criteria for this  characteristic, the cost estimate cannot be considered reliable.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date.  At that time, we will send copies of the report to the  appropriate congressional committees, the Secretary of Commerce, the  Under Secretary of Economic Affairs, the Acting Director of the U.S.  Census Bureau, and other interested parties. In addition, this report is  available at no charge on the GAO website at http://gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

The purpose of our review was to evaluate the reliability of the Census  Bureau’s (Bureau) life-cycle cost estimate using our Cost Estimating and  Assessment Guide. We reviewed (1) the extent to which the Bureau’s  life-cycle cost estimate and associated guidance met our best practices  for cost estimation using documentation and information obtained in  discussions with the Bureau related to the 2020 life-cycle cost estimate  and (2) compared the 2015 and 2017 life-cycle cost estimates to describe  key drivers of cost growth. For both objectives we reviewed  documentation from the Bureau on the 2020 life-cycle cost estimate and  interviewed Bureau and Department of Commerce officials.

For the first objective, we relied on our Cost Estimating and Assessment  Guide as criteria. Our cost specialists assessed measures consistently  applied by cost-estimating organizations throughout the federal  government and industry and considered best-practices for developing  reliable cost estimates. We analyzed the cost estimating practices used  by the Bureau against these best practices and evaluated them in four  categories: comprehensive, well-documented, accurate, and credible.

Comprehensive. The cost estimate should include both government  and contractor costs of the program over its full life-cycle, from  inception of the program through design, development, deployment,  and operation and maintenance to retirement of the program. It should  also completely define the program, reflect the current schedule, and  be technically reasonable. Comprehensive cost estimates should be  structured in sufficient detail to ensure that cost elements are neither  omitted nor double counted. Specifically, the cost estimate should be  based on a product-oriented work breakdown structure (WBS) that  allows a program to track cost and schedule by defined deliverables,  such as hardware or software components. Finally, where information  is limited and judgments are made, the cost estimate should  document all cost-influencing assumptions.

Well-documented. A good cost estimate—while taking the form of a  single number—is supported by detailed documentation that  describes how it was derived and how the expected funding will be  spent in order to achieve a given objective. Therefore, the  documentation should capture in writing such things as the source  data used, the calculations performed and their results, and the  estimating methodology used to derive each WBS element’s cost.

Moreover, this information should be captured in such a way that the  data used to derive the estimate can be traced back to, and verified  against, their sources so that the estimate can be easily replicated  and updated. The documentation should also discuss the technical  baseline description and how the data were normalized. Finally, the  documentation should include evidence that the cost estimate was  reviewed and accepted by management.

Accurate. The cost estimate should provide for results that are  unbiased, and it should not be overly conservative or optimistic. An  estimate is accurate when it is based on an assessment of most likely  costs; adjusted properly for inflation; and contains few, if any, minor  mistakes. In addition, a cost estimate should be updated regularly to  reflect significant changes in the program—such as when schedules  or other assumptions change—and actual costs, so that it is always  reflecting current status. During the update process, variances  between planned and actual costs should be documented, explained,  and reviewed. Among other things, the estimate should be grounded  in a historical record of cost estimating and actual experiences on  other comparable programs.

Credible. The cost estimate should discuss any limitations of the  analysis because of uncertainty or biases surrounding data or  assumptions. Major assumptions should be varied, and other  outcomes recomputed to determine how sensitive they are to changes  in the assumptions. Risk and uncertainty analysis should be  performed to determine the level of risk associated with the estimate.  Further, the estimate’s cost drivers should be cross-checked, and an  independent cost estimate conducted by a group outside the acquiring  organization should be developed to determine whether other  estimating methods produce similar results.

If any of the characteristics are not met, minimally met, or partially met,  then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable.

We also analyzed the Bureau’s cost estimation and analysis guidance  and evaluated them against a 12-step process outlined in our Cost  Estimation and Assessment Guide. A high-quality cost estimating process  integrates the following:  1.  Define estimate’s purpose.  2.  Develop estimating plan.  3.  Define program characteristics.  4.  Determine estimating structure.  5.  Identify ground rules and assumptions.  6.  Obtain data.  7.  Develop point estimate and compare it to an independent cost  estimate.  8.  Conduct sensitivity analysis.  9.  Conduct risk and uncertainty analysis.  10. Document the estimate.  11. Present estimate to management for approval.  12. Update the estimate to reflect actual costs and changes.

These 12 steps, when followed correctly, should result in reliable and  valid cost estimates that management can use for making informed  decisions. If any of the steps in the Bureau’s process do not meet,  minimally meet, or partially meet the 12 steps, then the cost estimate  guidance does not fully reflect best practices for developing a high-quality  estimate and cannot be considered reliable.

Lastly, to describe key drivers of cost growth, we compared cost  information included in the 2015 and 2017 cost estimates. We analyzed  both summary and detailed cost information to assess key changes in  totals overall, by WBS category, and by information technology (IT) vs.  Non-IT costs. We used this analysis in conjunction with information  received from the Bureau during interviews and through document  transfers to describe overall changes in the cost estimate from 2015 to  2017.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Lisa Pearson (Assistant Director),  Karen Cassidy (Analyst in Charge), Brian Bothwell, Jackie Chapin, Ann  Czapiewski, Jason Lee, Ty Mitchell, Kayla Robinson, and Tim Wexler  made significant contributions to this report.

In October 2017, the Census Bureau announced that the projected life-cycle cost of the 2020 Census had climbed to $15.6 billion, a 27-percent increase from its 2015 estimate.

A reliable cost estimate is essential to budgeting, planning, and managing the Census. We found that, while the Census Bureau has improved its cost-estimation process, the 2017 cost estimation still doesn’t meet all the necessary criteria of a reliable cost estimate. For example, the Bureau needs to improve documentation to support the cost estimate. We maintain our previous recommendation that the Census Bureau ensure that its cost estimate for the Census is reliable.