Defense Infrastructure: DOD Needs to Improve the Accuracy of Its Excess Capacity Estimates

DOD has used the Base Realignment and Closure (BRAC) process primarily to reduce excess infrastructure capacity, transform the force, and produce cost savings. DOD completed hundreds of base closures and realignments in previous BRAC rounds and intends to work with Congress to address remaining excess capacity. The NDAA for Fiscal Year 2016 required DOD to submit, among other things, a force structure plan and a categorical infrastructure inventory of worldwide military installations. In response, DOD submitted its infrastructure capacity report to Congress in October 2017.

The NDAA included a provision for GAO to evaluate DOD's report for accuracy and analytical sufficiency. In this report, GAO evaluates the extent to which (1) DOD's report included the required elements, and (2) DOD's methodology and analysis result in accurate and analytically sufficient information on excess capacity. To conduct this work, GAO reviewed DOD's 2017 report and compared it with the statutory requirements and generally accepted research standards. GAO also interviewed DOD and military service officials.

The Department of Defense's (DOD) 2017 infrastructure capacity report addressed four of five required elements from section 2815 of the National Defense Authorization Act (NDAA) for Fiscal Year 2016. Specifically, DOD's report addressed the elements requiring it to submit

a categorical inventory of worldwide military installations,

a discussion of categories of excess infrastructure, and

an assessment of the value of retaining certain excess infrastructure.

DOD's report partially addressed the element to include a description of the infrastructure capacity required to support the force structure. Specifically, DOD's report did not provide a complete picture of the infrastructure needed. For example, infrastructure at Air Force large aircraft installations was described by square yards of apron space, but did not include other infrastructure needs such as aircraft hangars and maintenance facilities.

DOD's excess capacity methodology and analysis has three key limitations that affect the accuracy and analytical sufficiency of the estimate. Specifically:

DOD used a 1989 baseline for excess capacity that may lead to inaccurate results. This 1989 baseline does not reflect updates in DOD facility standards and requirements or requirements associated with new weapon systems.

DOD's excess capacity methodology includes assumptions, such as not accounting for potential shortfalls—not having enough infrastructure to support the mission—that may not be reasonable. Specifically, when DOD's calculation identifies shortfall in capacity, DOD concludes that no excess capacity exists. As a result, DOD's analysis identifies no excess capacity in nearly half (14 of 32) mission categories. However, most installations support more than one mission and have more infrastructure present than the installation category metric measures. Thus, including potential capacity shortfalls could provide DOD and Congress with a more accurate estimate of excess capacity upon which to base decisions concerning the management of base infrastructure and excess capacity.

DOD's method for estimating excess capacity is not always sufficient because the installation selection process does not result in a generalizable sample. Furthermore, DOD's method is not always implemented effectively because the military departments did not follow a consistent approach.

According to DOD officials, specific department-wide guidance concerning DOD's methods for selecting installations in its analysis does not exist. Moreover, without developing guidance, the estimate of excess capacity may not be based on consistent methods across the department, resulting in inaccurate estimates. Furthermore, neither DOD nor Congress will have the necessary information to make decisions concerning the management of excess infrastructure capacity across the department.

GAO is making three recommendations to DOD to update the baseline; use reasonable assumptions; and develop guidance to improve its methods for estimating excess capacity. In comments on a draft of this report, DOD concurred with one recommendation, partially concurred with two recommendations, and plans to incorporate them in any future capacity analysis.

The Department of Defense (DOD) has used the Base Realignment and  Closure (BRAC) process primarily to reduce excess infrastructure,  readjust bases to accommodate changes in the size and structure of  DOD’s forces, and produce cost savings. Congress authorized five BRAC  rounds in 1988, 1991, 1993, 1995, and 2005, and DOD completed  hundreds of base closures and realignments as a result. To address  remaining excess capacity, between 2013 and 2017 DOD requested  additional BRAC rounds and, in February 2018, stated that it would work  with Congress to find common areas where reforms and changes could  be made. Congress has not authorized additional BRAC rounds to date.

Section 2815 of the National Defense Authorization Act (NDAA) for Fiscal  Year 2016 required the Secretary of Defense to submit: a force-structure plan for each military service—the Army, Navy, Air  Force, and Marine Corps; a categorical infrastructure inventory of worldwide military installations  for each military department; a description of the infrastructure necessary to support the force a discussion of categories of excess infrastructure and infrastructure an assessment of the value of retaining certain excess infrastructure  to accommodate contingency, mobilization, or surge requirements.

DOD provided its final report—Department of Defense Infrastructure  Capacity—to Congress on October 6, 2017.

The act also included a provision for us to evaluate the force-structure  plan and categorical infrastructure inventory for accuracy and analytical  sufficiency. In this report, we evaluate the extent to which (1) DOD’s 2017  infrastructure capacity report included the required elements and (2)  DOD’s methodology and analysis in its report results in accurate and  analytically sufficient information on excess capacity.

For objective one, we reviewed DOD’s 2017 infrastructure capacity report  and compared it with the required elements contained in section 2815 of  the NDAA for Fiscal Year 2016. Specifically, two analysts independently  reviewed DOD’s infrastructure capacity report using a scorecard to  determine whether the report included information that met the required  reporting elements. We considered an element to be “addressed” if  DOD’s 2017 infrastructure capacity report provided any evidence to  support all aspects of the requirement. We considered an element to be  “partially addressed” if the report provided evidence concerning some  aspects of the requirement and “not addressed” if the report did not  provide any evidence concerning any aspect of the requirement. We then  reconciled the individual reviews, and reached a consensus on our  assessment.

For objective two, we assessed the contents of DOD’s 2017 infrastructure  capacity report against a relevant subset of the generally accepted  research standards that we have described in a previous report.  Specifically, we focused on whether assumptions were reasonable and,  where appropriate, consistent; methods were sufficient and successfully  executed; and the baseline and other data used to support the analyses  were determined to be reliable and valid. All of the applied standards  were considered to have equal importance regarding the accuracy and  analytical sufficiency of the report. Some modification of the wording of  some of the standards was needed for the standard to be relevant and  appropriate in the context of DOD’s 2017 infrastructure capacity report. In  such instances, we kept the meaning of the standard, but modified the  language to align with the objectives and scope of DOD’s tasks. Appendix  I describes all the research standards, identifies the standards we chose  to use in evaluating the quality of the research results conveyed in DOD’s  report, and provides the rationale for the inclusion and exclusion of each  specific standard.

To conduct our analysis of DOD’s 2017 infrastructure capacity report  against the selected research standards, two analysts independently  reviewed the contents of the report for information indicating the extent to  which the presented analyses, results, and conclusions met these  standards. We considered the report to have “met” a research standard  when its contents explicitly addressed all aspects of the standard with  sufficient specificity and detail. We considered the report to have “partially  met” a standard when its contents addressed one or more attributes of  the standard, but not all of the standard’s attributes, or without sufficient  specificity and detail. We considered the report to have “not met” a  standard when its contents did not explicitly address any of the attributes  of the standard and any implicit references to the standard’s attributes  were too vague or general to be useful. We then reconciled the individual  assessments, and reached a consensus on the overall assessment. To  further corroborate our scorecard assessment, we had a second pair of  technical experts independently assess the validity and reliability of the  methodology used to generate the report’s results.

We conducted this performance audit from April 2016 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The Defense Base Closure and Realignment Act of 1990, as amended,  has governed the BRAC process since 1990. The law established the  procedures for making recommendations for base closures and  realignments and originally required DOD to submit a 6-year force- structure plan and base its closure and realignment decisions on that  plan. For the 1991, 1993, and 1995 BRAC rounds, DOD performed a  detailed capacity analysis based on extensive data-collection efforts to  identify specific bases capable of accommodating additional forces to  develop its proposed list of closures and realignments. In 1997, after  DOD requested another BRAC round, Congress required DOD to submit  a report on, among other things, the need for any additional BRAC rounds  and an estimate of the amount of DOD’s excess capacity at the time. In  2001, when Congress authorized a BRAC round to begin in 2005, it  required DOD to submit a force-structure plan to cover a 20-year period  and an infrastructure inventory with its budget-justification documents for  fiscal year 2005 before proceeding with the extensive data gathering  efforts and analysis associated with the BRAC process. The submission  was also to discuss categories of excess infrastructure and infrastructure  capacity.

Prior statutes included provisions for us to review DOD’s 1998 and 2004  excess capacity reports, which used a method to estimate excess  capacity that was very similar to the method used in its 2017 report. Our  1998 and 2004 reports reviewed DOD’s 1998 and 2004 excess capacity  reports, respectively. Our 2013 report assessed the estimating methods  used in both the 1998 and 2004 excess capacity reports. In these three  previous reports, we concluded that DOD’s methodology to estimate  excess capacity had a number of limitations, and thus gave a rough  indication that excess capacity existed. Specifically, we identified the  following four limitations with the method used in DOD’s 1998 and 2004  reports:  Installations were assigned to a single-mission category, yet most  installations perform more than one mission.

Military services used different metrics to evaluate installations in  similar mission categories.

DOD used a 1989 baseline that did not take into account any excess  capacity or capacity shortfall that may have existed at the time.

DOD’s analysis did not consider the possibility that a mission category  might have a capacity shortage; mission categories were determined  to have either an excess or no excess capacity.

DOD agreed that our 2013 report properly highlighted the limitations in  DOD’s methodology for estimating excess capacity. At that time, DOD  reiterated that the purpose of its methodology is to provide an indication  of whether sufficient excess exists to justify authorization of another  BRAC round. DOD concluded that only through the BRAC process is it  able to determine excess capacity by installation and mission or function  in a fair and thorough way. A list of related GAO products is included at  the end of this report.

DOD’s 2017 Infrastructure Capacity Report Addressed or Partially Addressed the Required Elements

DOD’s 2017 infrastructure capacity report addressed or partially  addressed the five required elements from section 2815 of the NDAA for  Fiscal Year 2016. As shown in table 1, DOD addressed four of the  required elements and partially addressed one element.

DOD’s report partially addressed the requirement to include a description  of the infrastructure capacity required to support the force structure  because the report describes only a small portion of the capacity needed.  For example, in the case of Air Force large aircraft installations, the  needed infrastructure was described in terms of the square yards of  apron space needed to support the assigned aircraft, but did not describe  other infrastructure needs such as aircraft hangars, maintenance  facilities, and administrative space used by squadrons assigned to the  installation. Similarly, in the case of Army maneuver installations, the  needed infrastructure was described in terms of maneuver acres needed,  but did not describe other infrastructure necessary to support assigned  units. Consequently, the description of infrastructure needed does not  provide DOD and Congress with a complete picture of the infrastructure  needed to support the force structure at these major installations.  However, as DOD points out in its report to Congress, the analysis  performed does not provide the detail necessary to identify specific  infrastructure for elimination; instead it provides an indicator of the  categories of excess. DOD also stated that this level of detail is only  provided through the formal BRAC process. Consequently, without a  formal BRAC round, DOD does not have the details necessary to identify  the total infrastructure necessary to support its current force structure.  Therefore, we are not making any recommendations concerning this  reporting requirement.

DOD’s Excess Capacity Methodology and Analysis Has Limitations That Affect the Accuracy and Analytical Sufficiency of the Estimate

DOD’s excess capacity methodology and analysis has limitations that  affect the accuracy and analytical sufficiency of the estimate. Specifically,  DOD’s use of a 1989 baseline for excess capacity results in inaccurate  estimates of excess capacity; DOD’s methodology included assumptions  that were not always reasonable; and DOD’s approach to estimating  excess capacity is not always sufficient or implemented consistently  across the military departments. DOD noted some of these same  limitations in its 2017 infrastructure capacity report.

DOD’s Use of 1989 Data as the Baseline for Its Excess Capacity Analysis Results in Inaccurate Estimates of Excess Capacity

DOD’s use of 1989 data as the baseline for its excess capacity analysis  resulted in inaccurate estimates of excess capacity. According to  generally accepted research standards, listed in appendix I, the baseline  and other data used to support the analysis should be determined to be  reliable and valid. Specifically, the baseline should be fully and completely  identified and used consistently, where appropriate. In addition, the data  limitations should be identified and the effect of these limitations should  be fully explained. DOD has also recognized that using 1989 as a  baseline did not account for excess capacity that existed in 1989.  However, DOD only partially explained the effect of this limitation on its  estimate of excess capacity.

First, using 1989 as the baseline assumes that the bases and facilities as  they existed in 1989 were appropriately sized to support their missions.  However, DOD’s 2017 infrastructure capacity report did not provide a  rationale for either why 1989 was an appropriate baseline or why the  bases and facilities were assumed to be appropriately sized at that time.  In fact, as discussed below, DOD has stated that excess capacity existed  in 1989, but does not attempt to quantify the amount. Further, in at least  one mission category, Marine Corps Bases, DOD acknowledges that it  overstated excess capacity because the baseline ratio was based on  infrastructure numbers that were not adjusted to recognize the  documented shortfalls that existed in 1989.

Second, the effects of DOD’s assumptions about the 1989 baseline have  not been consistently reported by DOD. DOD has used the same  baseline in its three analyses conducted over the past 20 years, yet DOD  draws different conclusions concerning how the baseline affects its  estimates of excess capacity. For example, DOD concluded  in 1998 that excess capacity existed in the 1989 baseline because the  majority of realignment and closures took place after 1989;  in 2004 that very significant excess capacity existed in the 1989  baseline; and  in 2017, in DOD’s infrastructure capacity report, that the 1989  baseline was both properly sized to support assigned missions and  forces and included significant excess capacity.

Nevertheless, DOD has consistently stated that its estimate of excess  capacity is likely conservative because significant excess existed in 1989.  DOD also stated that its analysis provides an indicator of the categories  where excess might exist and that only through a BRAC round can the  department undertake the detailed analysis necessary to make closure  and realignment recommendations. Since 1988, DOD has completed five  BRAC rounds that have closed a significant number of DOD facilities. In  addition, as discussed below, DOD facility standards and requirements  have been updated and new weapon systems have been introduced,  which can affect the amount and type of infrastructure needed.  Consequently, without a definitive measure of the excess that existed in  1989, as well as adjustments in the method to account for the effect of  updated facility standards and requirements, and new weapons systems,  there is no clear rationale for using 1989 as a baseline year in the  estimate of excess capacity provided by DOD’s analysis.

Third, during the last 29 years DOD facility standards and requirements  have been updated and new weapon systems with greater ranges and  capabilities have been developed that have changed the amount and type  of infrastructure needed to support DOD’s forces. For example, we  recently reported that only 11 of the Navy’s 18 drydocks are configured to  perform maintenance on the newer ship and submarine classes like the  Ford-class aircraft carrier and Virginia-class submarine. Using such an  old baseline, without making adjustments in the method to account for  these changes, leads us to conclude that DOD’s results are likely  inaccurate.

Because DOD continues to use its outdated 1989 baseline we found that  DOD’s 2017 excess capacity analysis results in estimates that are likely  inaccurate. Without updating the baseline that is used in the methodology  to calculate excess capacity across DOD, DOD will not have accurate  information for making critical decisions related to investments in  infrastructure. Furthermore, Congress will not have accurate information  to make fully informed decisions concerning whether and to what extent  another BRAC round is needed.

DOD’s Methodology for Estimating Excess Capacity Includes Assumptions That Are Not Always Reasonable

DOD’s excess capacity methodology includes assumptions that are not  always reasonable, such as assigning installations to only one mission  category. According to generally accepted research standards,  reasonable assumptions are characterized by being realistic, credible,  and accompanied by a statement of their rationale. In addition, these  standards also state that assumptions should support a sound analysis  (e.g., the assumptions should not skew the results of the analysis or  reduce the range of possible outcomes).

We previously reported limitations related to DOD’s assumptions when  we examined DOD’s excess capacity analyses in 1998, 2004, and  2013. DOD continues to use the same methodology in 2017 that it has  previously used to estimate excess capacity; thus, these limitations  continue to exist in its methodology in its 2017 report. First, DOD’s  approach of assigning an installation to only one mission category treats  an installation as if it has only one mission, yet most installations support  more than one mission. As a result, only a small portion of an  installation’s infrastructure may be considered by DOD’s analysis. For  example, in the case of Fort Bragg, North Carolina, which is included in  the maneuver base category by the Army, base acres are included in the  analysis, but more than 43.8 million square feet of infrastructure is not  considered. Similarly, in the case of Naval Base Kitsap, Washington,  which is included in the Naval Station category by the Navy, the pier  space is considered in the analysis, but the more than 7.5 million square  feet of facilities is not considered. In addition, as discussed later in this  report, there were instances where the military departments included  installations in more than one mission category. Finally, there are several  categories that measure capacity in terms of direct labor hours or work- years, but the analysis does not include the actual infrastructure, such as  buildings, structures, and linear structures. Consequently, the assumption  that each installation is included in one mission category may not be  reasonable because only a portion of the infrastructure at the installations  is being considered when identifying potential excess capacity.

Second, as implemented, DOD’s estimate of excess capacity may be  overstated because its methodology did not account for any potential  shortfalls in capacity—not having enough infrastructure to support the  mission—and did not provide a rationale for this approach in its  calculations. As illustrated in table 2, when DOD’s calculation identifies  that the proportional capacity is less than the infrastructure capacity for  the year being analyzed (i.e., DOD needs less infrastructure than it has),  DOD concludes that excess capacity exists and provides a percentage  amount of excess capacity. However, when the proportional capacity  exceeds the infrastructure capacity for the year being analyzed (i.e., DOD  may need more infrastructure), DOD concludes that no excess capacity  exists. Moreover, DOD’s calculation provides a zero percentage for  excess capacity, rather than a negative percentage that would account for  a potential capacity shortfall in its analysis. DOD’s 2017 infrastructure  capacity analysis identifies zero percent excess capacity in nearly half (14  of 32) of the installation categories that needed more capacity—included  in the analysis, including 8 or 12 Navy installation categories.

Because DOD’s methodology uses the excess capacity percentages from  the 32 installation categories to compute a weighted average for excess  capacity across the department, treating a negative percentage from a  mission category as 0.0 percent would increase DOD’s overall excess  capacity percentage. DOD officials believe that treating these 14  installation categories as if they have 0.0 percent excess capacity is  appropriate because the purpose of the analysis is to identify the  categories where excess capacity may exist. In addition, they asserted  that treating these categories as if they had a shortfall would assume that  infrastructure from 1 of the 18 other installation categories identified as  having excess capacity could be used to offset the shortfall when the  categories are likely to have different metrics. DOD officials also told us  that, from their perspective, no increase does not mean that there is large  deficit of infrastructure within a mission category; it just means that the  infrastructure to force-structure ratio indicates that the particular category  does not have excess. We found, however, 6 installation categories  where the force-structure measure exceeds the capacity measure, which  indicates that a shortfall exists. In addition, because most installations  support more than one mission and have more infrastructure present than  the mission category metric measures, including potential capacity  shortfall in its analysis could provide DOD and Congress with a more  accurate estimate of excess capacity.

DOD’s methodology to estimate excess capacity includes assumptions  that are not reasonable. Without using assumptions to estimate excess  capacity that are considered reasonable (i.e., realistic, credible, and  accompanied by a statement of their rationale), DOD’s methodology may  overstate its estimate of excess capacity.

DOD’s Method for Estimating Excess Capacity Is Not Always Sufficient or Implemented Consistently

DOD’s method for estimating excess capacity across the department is  not sufficient because it is based on a nongeneralizable sample and  therefore its reported estimates cannot be generalized to describe excess  capacity across the department. Furthermore, DOD’s sampling method  is not always implemented effectively because some of the military  departments adjusted the sampling approach. According to generally  accepted research standards, the methods used and the analysis should  be sufficient for accomplishing the objectives of the study. In addition, the  analysis should be executed consistently with the study plan or the  described methodology. We found that the calculations performed by  DOD in the analysis were generally accurate.

First, DOD and the military departments used a nongeneralizable sample  of different types of installations to develop an excess capacity estimate.  However, a nongeneralizable sample cannot be used to develop a  department-wide estimate of excess capacity because this technique is  not designed to yield a sound probable statistical estimate. Specifically,  when the analysis was first done in 1998, the military departments sorted  installations into categories and only included installations that were  considered by the departments to be “major installations.” The  departments were to assign each “major installation” to only one mission  category. The departments were to then calculate the estimated capacity  by mission category for both the baseline year, 1989, and the projected  force-structure year, 2003. The same approach was used for the 2017  analysis; however, neither the 1998 nor the 2017 analysis provided  guidance to the military department concerning what constitutes a “major  installation.” This approach for selecting and sorting samples of  installations relies on the judgment of each of the military departments,  yielding a nongeneralizable sample of installations that vary across the  military departments. Consequently, the results from the analysis cannot  be used to make inferences about the amount of excess capacity across  DOD.

Second, the military departments did not follow a consistent approach  when calculating excess capacity. Specifically, the DOD method bases its  excess capacity estimate on the number of installations in each mission  category. However, we found that, in the 2017 analysis, the military  departments did not consistently follow the practice of including  installations in only one category across the services when the analysis  was performed in 2017. For example, we found several installations that  were included in more than one category by some of the military  departments:  In the 2017 analysis, the Air Force included two subcategories under  the heading of “Education and Training”: “Flight Training” and  “Classroom.” The flight training subcategory included 13 installations  and the classroom subcategory included 14 installations. We found  that all 13 of the flight training installations were also included as  classroom installations. Yet, when the analysis was performed in both  1998 and 2004, the same 14 installations were used, but 8 of the  installations were then categorized as being flight training installations  and the other 6 installations were categorized as classroom  installations. If this previous categorization approach was used in the  2017 analysis, the Air Force estimate of excess capacity would have  been about 2 percent lower.

In two instances, the Navy included the same installations in both the  “Naval Station” and “Air Station” categories and, in one instance, the  Navy included a joint base in both the “Naval Station” and “Shipyards”  categories. According to a Navy official, these installations were  included in both categories because a major mission would have been  omitted from the analysis if the bases were included in only one  category. This treatment, however, is not consistent with DOD’s  methodology.

Including the same installation in multiple installation categories may have  resulted in double counting of capacity, and thereby affected the resulting  estimate of excess capacity for multiple installation categories.

Third, the military departments did not consistently account for the joint  bases in their excess capacity analysis. In some instances, we found  that only the lead military department included the joint base in its  analysis. For example, in the case of Joint Base Lewis-McChord,  Washington—an Army-led joint base comprising Fort Lewis and McChord  Air Force Base—the Army, consistent with its treatment of Fort Lewis in  previous excess capacity analyses, included the joint base in its  maneuver category. However, the Air Force did not include McChord Air  Force Base in its analysis in 2017 although it had in previous years. In  these instances where only the lead military department included the joint  base in its analysis, the infrastructure associated with the tenant military  department was usually left out of the analysis because the metric used  by the leading department does not incorporate the same measures of  infrastructure and force structure as the tenant department. In the Joint  Base Lewis-McChord example, the Army included the base in the  maneuver category, which is measured by the ratio of maneuver acres to  maneuver battalion equivalents while the Air Force had previously used  the ratio of parking apron space to number of aircraft to measure capacity  at McChord Air Force Base. Consequently, DOD’s analysis no longer  takes into account the infrastructure that supports the flying mission at  this joint base.

In other instances, we found that both the lead military department and  the tenant military department included their portion of the infrastructure  in their analyses. For example, for Joint Base Charleston, South  Carolina—an Air Force-led joint base comprised of Charleston Air Force  Base and Naval Support Activity Charleston—each of the military departments continued to include their portion of the infrastructure in their  individual analyses. Consequently, DOD’s analysis accounts for the  infrastructure that supports both missions at the joint base.

DOD’s method for estimating excess capacity is not always sufficient and  is not implemented consistently across the military departments because  DOD lacks specific department-wide guidance, according to DOD  officials. Specifically, explicit guidance does not exist that clearly defines  “major installations,” identifies whether and when it is appropriate to  include a facility in more than one category to take into account multiple  missions at the facilities, or provides protocols for assessing excess  capacity at joint bases. These topics were discussed in meetings with  military department officials, but, according to DOD officials, no specific  method was identified for department-wide use. Without developing  guidance for the military departments, the estimate of excess capacity  may not be based on consistent methods across the department,  resulting in inaccurate estimates.

DOD’s 2017 excess capacity analysis does not have the accuracy and  analytical sufficiency to provide Congress with a reasonable estimate of  the actual excess capacity within the department. DOD recognizes the  limitations of its analysis, specifically noting that the resulting percentages  of excess capacity are at best indicators to justify the more detailed  analysis of excess capacity provided by a full BRAC analysis. Specifically,  DOD used a baseline for the analysis that did not fully take into account  changes in infrastructure needs since 1989, used assumptions in its  analysis that are not reasonable, and used methods that were not  sufficient or implemented consistently. These limitations resulted in  excess capacity estimates that do not have the accuracy and analytical  sufficiency to support decision making on future BRAC rounds. Without  improvements to DOD’s method of estimating excess capacity, DOD is  not providing the information that Congress requires to make decisions  concerning the management of excess infrastructure capacity within the  department. Similarly, DOD does not have the information it needs to  appropriately manage its infrastructure capacity and therefore cannot  make informed decisions about what it needs to support its mission as  land and infrastructure requirements of newer weapon systems are  introduced. Moreover, the combined effect of neither DOD nor Congress  having the information means that DOD will continue to experience  challenges with funding related to its infrastructure and potential excess  costs.

We are making the following three recommendations to DOD:  The Secretary of Defense should ensure that the Assistant Secretary of  Defense for Energy, Installations, and Environment reliably updates the  baseline used for estimating excess infrastructure capacity.  (Recommendation 1)

The Secretary of Defense should ensure that the Assistant Secretary of  Defense for Energy, Installations, and Environment uses assumptions in  estimating excess capacity that are considered reasonable (i.e., realistic,  credible, and accompanied by a statement of their rationale).  (Recommendation 2)

The Secretary of Defense should ensure that the Assistant Secretary of  Defense for Energy, Installations, and Environment develops guidance to  improve the methods used in the analysis and ensure consistent  implementation of DOD’s methodology to produce reliable estimates of  excess capacity across the department. The guidance, at a minimum,  should clearly define “major installations,” identify whether and when it is  appropriate to include a facility in more than one category to take into  account multiple missions at the facilities, and provide protocols for  assessing excess capacity at joint bases. (Recommendation 3)

We provided a draft of this report to the Department of Defense (DOD) for  comment. DOD provided written comments, which are reproduced in  appendix II.

DOD concurred with one recommendation and partially concurred with  the other two recommendations.

DOD stated that it concurred with our first recommendation, which called  for it to reliably update the baseline used for estimate excess  infrastructure capacity. Specifically, the department stated that it would  review methods to update the baseline for future excess capacity analysis  that is undertaken.

The department partially concurred with our second recommendation,  which called for the department to use assumptions that were considered  reasonable (i.e. realistic, credible, and accompanied by a statement of the  rationale) in estimating excess capacity. Specifically, the department  agreed that its capacity report should lay out any assumptions made and  the rationale for each assumption and will ensure that any future capacity  report includes that information. The department did not concur, however,  that assumptions used in its 2017 infrastructure capacity report were  other than reasonable, realistic, or credible. While we are encouraged  that the department will lay out any assumptions and the rationale for  each assumption in future capacity reports, not all assumptions used in  the 2017 analysis were reasonable (i.e. realistic, credible, and  accompanied by a statement of the rationale) as outlined in this report.  For example, we found that assigning installations to only one mission  category was not realistic because most installations support more than  one mission.

The department partially concurred with our third recommendation that  DOD develop guidance to improve the methods used in the analysis and  ensure consistent implementation of DOD’s methodology to produce  reliable estimates of excess capacity across the department. This  guidance, at a minimum, should clearly define “major installations,”  identify whether and when it is appropriate to include a facility in more  than one category to take into account multiple missions at the facilities,  and provide protocols for assessing excess capacity at joint bases. DOD  concurred that guidance should precede any future infrastructure capacity  review and that such guidance should include definitions and  implementation instructions, but the three items identified would not  necessarily be applicable for a future analysis. Provided that future DOD  guidance addresses all appropriate characteristics for analysis, such  guidance would meet the intent of our recommendation.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Army, Navy,  and Air Force; and the Assistant Secretary of Defense for Energy,  Installations, and Environment. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please  contact me at (202) 512-4523 or leporeb@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on   the last page of this report. Key contributors to this report are listed in  appendix III.

Appendix I: Generally Accepted Research Standards Relevant to DOD’s Infrastructure Capacity Report

Table 3 describes the generally accepted research standards, identifies  the standards we used in evaluating the quality of the research results  conveyed in DOD’s 2017 infrastructure capacity report and provides the  rationale for the inclusion and exclusion of each specific standard.

Study plan, scope, and objectives follow   existing guidance?

Do the study scope and objectives fully   address the mandated elements?

Does the study plan address   specified guidance?

Team is not aware of any standard  guidance for the development of this  document.

Team is not aware of any study plan  that guided the development of this  report.

Were deviations from the study plan   explained and documented?

Was the study plan updated over the course   of the study and the updates explicitly identified   in the study and updated study plan?  Assumptions and limitations are reasonable   and, where appropriate, consistent   Are assumptions and limitations   explicitly identified?

Given the judgment required to   execute the analyses the   assumptions and constraints   are key to team’s determination   of the accuracy and analytical  sufficiency of the report.

Generally Accepted Research Standards II.a.1

Are the assumptions reasonable in that they   are realistic, credible, and accompanied   by a statement of their rationale?

Rationale for inclusion   in or exclusion from   GAO’s review  Given the judgment required to execute  the analyses, the assumptions and  constraints are key to the team’s  determination of the accuracy and  analytical sufficiency of the report.  Team felt that ‘reasonable’ was  sufficient and ‘necessary’ was not  readily apparent.

Do the assumptions support a   sound analysis?

Given the judgment required to execute  the analyses the assumptions and  constraints are key to team’s  determination of the accuracy and  analytical sufficiency of the report.

Are the assumptions used in analyses   common throughout the study and models?

This standard is not needed to answer  the objectives of our report. Other  standards for study assumptions are  more relevant and sufficient for our  purposes.

Do the assumptions contribute to an objective   and balanced research effort?

Scenarios and threats are reasonable   Did they synthesize the supporting analyses such   that it is traceable back to formal guidance?

Were the threat scenarios validated and   Joint Staff approved and documented?

Do scenarios represent a reasonably complete   range of conditions?

Were the threats varied to allow for the   conduct of sensitivity analysis?

Methods are sufficient and   successfully executed  Were the study methods executed consistent   with the study plan and schedule?

Were the methods and analyses sufficient   for accomplishing the objectives presented   in the study?

Given the judgment required to execute  the analyses the methodology is key to  determine if DOD accomplishes its  objectives.

Generally Accepted Research Standards IV.c

Were the models used to support the analyses   adequate for their intended purpose?  //Were the calculations used to support   the analyses accurate?  Baseline and other data used to support the  analyses were determined to be reliable and valid?   Is the baseline fully and completely identified   and used consistently, where appropriate,   throughout the various analyses?

Rationale for inclusion   in or exclusion from   GAO’s review  Important to ensure the model is  designed well in addition to accurate  arithmetic calculations. DOD conducted  analyses and calculations in the report.

DOD report includes the use of   baseline data in the underlying  analyses.

Were data limitations identified and the impact   of the limitations fully explained?

DOD report uses data obtained   from DOD components.

Were the data determined to be reliable   and valid?

Were the data reliability and validation   process documented?

Were the appropriate data gathered   to support the analyses?

OSD obtained data from other DOD  components and used it to generate   the report.

Analyses are reasonable   Was a verification, validation, and accreditation   report that addresses the models and data   certification signed by the study director   and included in the report?

In the context of our engagement,  redundant with section II above.

Were analytic limitations identified and explained?

Has each analysis in the study been described?

Were the analyses clearly explained,   documented?

Measures of effectiveness (MOEs) and essential elements of analysis (EEAs) are addressed Do MOEs adhere to the guidance in the study terms of reference?

The mandate language does not   require DOD to include measures of  effectiveness in its report. Furthermore,  DOD is not required to submit a  strategic plan so Government  Performance and Results Act  requirements are not applicable.

Are the MOEs fully addressed in the study?

Presentation of results support findings   Does the report address the objectives?

Does the report present an assessment that   is well documented and conclusions that   are supported by the analyses?

We will address conclusionary   language in the context of the   data used to support it above.

Are recommendations supported by analyses?

The mandate language does   not require DOD to include  recommendations and DOD   did not include recommendations.

Is a realistic range of options provided?

Not applicable. DOD’s report does   not include range of options for force- structureplans and categorical  infrastructure inventory.

Are the study results presented in the report   in a clear manner?

Are study participants/stakeholders   (i.e., services and Combatant Commands)   informed of the study results and recommendations?

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Gina Hoffman (Assistant  Director), Tracy Barnes, Ronald Bergman, Patricia Donahue, Kerstin  Hudon, Terrance Lam, Amie Lesser, Carol Petersen, Clarice Nassif  Ransom, Matt Spiers, Tristan To, and John Wren made key contributions  to this report.

High-Risk Series: Progress on Many High-Risk Areas, While Substantial  Efforts Needed on Others. GAO-17-317. Washington, D.C.: February 15,  2017.

Military Base Realignments and Closures: DOD Has Improved  Environmental Cleanup Reporting but Should Obtain and Share More  Information. GAO-17-151. Washington, D.C.: January 19, 2017.

Defense Infrastructure: DOD Efforts to Prevent and Mitigate  Encroachment at Its Installations. GAO-17-86. Washington, D.C.:  November 14, 2016.

Defense Facility Condition: Revised Guidance Needed to Improve  Oversight of Assessments and Ratings. GAO-16-662. Washington, D.C.:  June 23, 2016.

Defense Infrastructure: More Accurate Data Would Allow DOD to Improve  the Tracking, Management, and Security of Its Leased Facilities.  GAO-16-101. Washington, D.C.: March 15, 2016.

Underutilized Facilities: DOD and GSA Information Sharing May Enhance  Opportunities to Use Space at Military Installations. GAO-15-346.  Washington, D.C.: June 18, 2015.

Military Base Realignments and Closures: More Guidance and  Information Needed to Take Advantage of Opportunities to Consolidate  Training. GAO-16-45. Washington, D.C.: February 18, 2016.

Military Base Realignment and Closures: Process for Reusing Property  for Homeless Assistance Needs Improvements. GAO-15-274.  Washington, D.C.: March 16, 2015.

High-Risk Series: An Update. GAO-15-290. Washington, D.C.: February  11, 2015.

Federal Real Property: Strategic Focus Needed to Help Manage Vast and  Diverse Warehouse Portfolio. GAO-15-41. Washington, D.C.: November  12, 2014.

DOD Joint Bases: Implementation Challenges Demonstrate Need to  Reevaluate the Program. GAO-14-577. Washington, D.C.: September 19,  2014.

Defense Infrastructure: DOD Needs to Improve Its Efforts to Identify  Unutilized and Underutilized Facilities. GAO-14-538. Washington, D.C.:  September 8, 2014.

Defense Infrastructure: Army Brigade Combat Team Inactivations  Informed by Analyses, but Actions Needed to Improve Stationing  Process. GAO-14-76. Washington, D.C.: December 11, 2013.

Military Bases: DOD Has Processes to Comply with Statutory  Requirements for Closing or Realigning Installations. GAO-13-645. June  27, 2013.

Defense Infrastructure: DOD’s Excess Capacity Estimating Methods  Have Limitations. GAO-13-535. Washington, D.C.: June 20, 2013.

Military Bases: Opportunities Exist to Improve Future Base Realignment  and Closure Rounds. GAO-13-149. Washington, D.C.: March 7, 2013.

GAO’s 2013 High Risk Series: An Update. GAO-13-283. Washington,  D.C.: February 2013.

DOD Joint Bases: Management Improvements Needed to Achieve  Greater Efficiencies. GAO-13-134. Washington, D.C.: November 15,  2012.

Military Base Realignments and Closures: The National Geospatial- Intelligence Agency’s Technology Center Construction Project.  GAO-12-770R. Washington, D.C.: June 29, 2012.

Military Base Realignments and Closures: Updated Costs and Savings  Estimates from BRAC 2005. GAO-12-709R. Washington, D.C.: June 29,  2012.

Military Base Realignments and Closures: Key Factors Contributing to  BRAC 2005 Results. GAO-12-513T. Washington, D.C.: March 8, 2012.

Excess Facilities: DOD Needs More Complete Information and a Strategy  to Guide Its Future Disposal Efforts. GAO-11-814. Washington, D.C.:  September 19, 2011.

Military Base Realignments and Closures: Review of the Iowa and Milan  Army Ammunition Plants. GAO-11-488R. Washington, D.C.: April 1,  2011.

GAO’s 2011 High-Risk Series: An Update. GAO-11-394T. Washington,  D.C.: February 17, 2011.

Defense Infrastructure: High-Level Federal Interagency Coordination Is  Warranted to Address Transportation Needs beyond the Scope of the  Defense Access Roads Program. GAO-11-165. Washington, D.C.:  January 26, 2011.

Military Base Realignments and Closures: DOD Is Taking Steps to  Mitigate Challenges but Is Not Fully Reporting Some Additional Costs.  GAO-10-725R. Washington, D.C.: July 21, 2010.

Defense Infrastructure: Army Needs to Improve Its Facility Planning  Systems to Better Support Installations Experiencing Significant Growth.  GAO-10-602. Washington, D.C.: June 24, 2010.

Military Base Realignments and Closures: Estimated Costs Have  Increased While Savings Estimates Have Decreased Since Fiscal Year  2009. GAO-10-98R. Washington, D.C.: November 13, 2009.

Military Base Realignments and Closures: Transportation Impact of  Personnel Increases Will Be Significant, but Long-Term Costs Are  Uncertain and Direct Federal Support Is Limited. GAO-09-750.  Washington, D.C.: September 9, 2009.

Military Base Realignments and Closures: DOD Needs to Update Savings  Estimates and Continue to Address Challenges in Consolidating Supply- Related Functions at Depot Maintenance Locations. GAO-09-703.  Washington, D.C.: July 9, 2009.

Defense Infrastructure: DOD Needs to Periodically Review Support  Standards and Costs at Joint Bases and Better Inform Congress of  Facility Sustainment Funding Uses. GAO-09-336. Washington, D.C.:  March 30, 2009.

Military Base Realignments and Closures: DOD Faces Challenges in  Implementing Recommendations on Time and Is Not Consistently  Updating Savings Estimates. GAO-09-217. Washington, D.C.: January  30, 2009.

Military Base Realignments and Closures: Army Is Developing Plans to  Transfer Functions from Fort Monmouth, New Jersey, to Aberdeen  Proving Ground, Maryland, but Challenges Remain. GAO-08-1010R.  Washington, D.C.: August 13, 2008.

Defense Infrastructure: High-Level Leadership Needed to Help  Communities Address Challenges Caused by DOD-Related Growth.  GAO-08-665. Washington, D.C.: June 17, 2008.

Defense Infrastructure: DOD Funding for Infrastructure and Road  Improvements Surrounding Growth Installations. GAO-08-602R.  Washington, D.C.: April 1, 2008.

Military Base Realignments and Closures: Higher Costs and Lower  Savings Projected for Implementing Two Key Supply-Related BRAC  Recommendations. GAO-08-315. Washington, D.C.: March 5, 2008.

Defense Infrastructure: Realignment of Air Force Special Operations  Command Units to Cannon Air Force Base, New Mexico. GAO-08-244R.  Washington, D.C.: January 18, 2008.

Military Base Realignments and Closures: Estimated Costs Have  Increased and Estimated Savings Have Decreased. GAO-08-341T.  Washington, D.C.: December 12, 2007.

Military Base Realignments and Closures: Cost Estimates Have  Increased and Are Likely to Continue to Evolve. GAO-08-159.  Washington, D.C.: December 11, 2007.

Military Base Realignments and Closures: Impact of Terminating,  Relocating, or Outsourcing the Services of the Armed Forces Institute of  Pathology. GAO-08-20. Washington, D.C.: November 9, 2007.

Military Base Realignments and Closures: Transfer of Supply, Storage,  and Distribution Functions from Military Services to Defense Logistics  Agency. GAO-08-121R. Washington, D.C.: October 26, 2007.

Defense Infrastructure: Challenges Increase Risks for Providing Timely  Infrastructure Support for Army Installations Expecting Substantial  Personnel Growth. GAO-07-1007. Washington, D.C.: September 13,  2007.

Military Base Realignments and Closures: Plan Needed to Monitor  Challenges for Completing More Than 100 Armed Forces Reserve  Centers. GAO-07-1040. Washington, D.C.: September 13, 2007.

Military Base Realignments and Closures: Observations Related to the  2005 Round. GAO-07-1203R. Washington, D.C.: September 6, 2007.

Military Base Closures: Projected Savings from Fleet Readiness Centers  Likely Overstated and Actions Needed to Track Actual Savings and  Overcome Certain Challenges. GAO-07-304. Washington, D.C.: June 29,  2007.

Military Base Closures: Management Strategy Needed to Mitigate  Challenges and Improve Communication to Help Ensure Timely  Implementation of Air National Guard Recommendations. GAO-07-641.  Washington, D.C.: May 16, 2007.

Military Base Closures: Opportunities Exist to Improve Environmental  Cleanup Cost Reporting and to Expedite Transfer of Unneeded Property.  GAO-07-166. Washington, D.C.: January 30, 2007.

Military Bases: Observations on DOD’s 2005 Base Realignment and  Closure Selection Process and Recommendations. GAO-05-905.  Washington, D.C.: July 18, 2005.

Military Bases: Analysis of DOD’s 2005 Selection Process and  Recommendations for Base Closures and Realignments. GAO-05-785.  Washington, D.C.: July 1, 2005.

Military Base Closures: Observations on Prior and Current BRAC  Rounds. GAO-05-614. Washington, D.C.: May 3, 2005.

Military Base Closures: Assessment of DOD’s 2004 Report on the Need  for a Base Realignment and Closure Round. GAO-04-760. Washington,  D.C.: May 17, 2004.

Military Bases: Review of DOD’s 1998 Report on Base Realignment and  Closure. GAO/NSIAD-99-17. Washington, D.C.: November 13, 1998.