Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA's Risk-Based Grant Assessment Model

FEMA, a component of DHS, provides preparedness grants to state, local, tribal, and territorial governments to help prepare for, prevent, protect against, respond to, recover from and mitigate terrorist attacks or other disasters. SHSP grants fund the nation's 56 states and territories, while UASI grants fund eligible urban areas. Grant allocations have been based, in part, on FEMA's risk-based grant assessment model, with states and urban areas deemed to be at higher risk receiving more grant dollars than those deemed at lower risk. Since 2008, GAO and others have assessed the model and made recommendations to strengthen it.

This report 1) describes SHSP and UASI grant awards during fiscal years 2008 through 2018, and factors affecting grant distributions; and 2) examines the steps that FEMA has taken to strengthen its risk assessment model for allocating preparedness grants, and any additional opportunities to improve the model. GAO analyzed the information in FEMA's model, and data on SHSP and UASI grant awards for fiscal years 2008 through 2018. GAO also interviewed FEMA and DHS officials and collected documents.

GAO found that various factors affected Federal Emergency Management Agency (FEMA) State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grant awards from fiscal year 2008 through 2018. SHSP grant awards to states were based on two factors—(1) minimum amounts set in law each year, and (2) FEMA's risk model. For example, in fiscal year 2012, each state was to receive a minimum of approximately $2.74 million, with each state receiving additional funds based on its relative risk score. Conversely, UASI grant awards are made based on its FEMA's risk-based grant assessment model, which ranks each urban area relative to others in that year, and Department of Homeland Security (DHS) leadership decisions on how funding should be allocated. From fiscal year 2008 through 2018, the number of USAI grantees varied from year to year (see figure below).

Since 2008, FEMA has taken steps to strengthen its risk-based grant assessment model, but has not incorporated additional scientific practices into its model. For example, in 2011 FEMA included more information in its model on potential targets and their vulnerability in each state and urban area, addressing a prior GAO recommendation. More recently in 2018, FEMA added additional factors to better assess vulnerability in each state and urban area, such as the number of special events where large crowds gather and soft targets susceptible to lone wolf attacks, among other things. However, GAO found that FEMA does not fully utilize scientific practices recognized by the National Research Council and the Office of Management and Budget as best practices. Specifically, FEMA did not fully document its model's underlying assumptions, such as the weights in its model or the justification for changes to these weights. FEMA also did not perform the level of analysis needed to determine how changes to its model could affect the resulting risk scores. Finally, FEMA has not coordinated an independent external peer review of its model. Applying such scientific practices could assist FEMA in further strengthening its model.

GAO is making three recommendations to FEMA to further strengthen its risk-based grant assessment model by (1) fully documenting the model's assumptions and justifications, (2) performing additional in-depth analyses, and (3) coordinating an external peer review. FEMA concurred with our recommendations.

The Department of Homeland Security (DHS) through the Federal  Emergency Management Agency (FEMA) awarded over $50 billion in  preparedness grants from fiscal years 2002 through 2018. These grants  are designed to enhance the capabilities of state and local governments  to, prevent, prepare for, protect against, respond to, recover from, and  mitigate terrorist attacks and other disasters. The largest preparedness  grant programs are the State Homeland Security Program (SHSP), which  provides grants to the nation’s 56 states and territories, and the Urban  Area Security Initiative (UASI), which awards grants to high-risk urban  areas. In order to make grant award decisions for SHSP and UASI  grants, FEMA has developed and maintains a risk-based grant  assessment model with the assistance of other DHS components such as  the Office of Intelligence and Analysis. FEMA uses this model to  represent potential threats, vulnerabilities, and consequences of a  terrorist attack, among other factors. Specifically, this risk model is used  to determine the relative risk order of (a) 56 states and territories, and (b)  the nation’s 100 most populous urban areas for purposes of designating  high-risk urban areas that are eligible for funding, which serves to inform  DHS leadership in their final determinations of grant award amounts,  according to FEMA officials. According to FEMA officials, DHS  leadership considers multiple pieces of information when making grant  award decisions including historical funding, risk scores and ranking,  funding separation between jurisdictions, and any special circumstances.

In June 2008, we reported that DHS’s risk-based grant assessment  model for allocating grants was reasonable, but the way that this risk  model measured vulnerability across states and urban areas was limited.  We recommended that DHS and FEMA formulate a methodology to  measure variations in vulnerability across states and urban areas. DHS  concurred with our recommendation and we will discuss how they  implemented this recommendation later in this report.

You asked us to report on the grant awards to states and urban areas  and any changes to FEMA’s risk-based grant assessment model since  2008. Specifically, this report (1) describes SHSP and UASI total amounts  funded from fiscal years 2008 through 2018, and the factors that affect  the calculation of grant awards to states and territories, and urban areas;  and (2) examines the steps FEMA has taken to strengthen its risk model  for allocating SHSP and UASI grants, and what additional opportunities, if  any, exist to improve the model.

To address our first objective, we reviewed applicable laws governing the  establishment and distribution of these grants programs, as well as  FEMA’s annual grant announcements and guidance. We reviewed and  identified laws and regulations including the Homeland Security Act of  2002, the Implementing Recommendations of the 9/11 Commission Act of  2007, Department of Homeland Security appropriations acts and  accompanying congressional reports. We analyzed SHSP and UASI  annual grant awards to states, territories and urban areas, based on  publically-reported information contained in DHS and FEMA’s grant  funding notifications for fiscal years 2008 through 2018. We compared  data in these public documents to annual program funding and grant  award data provided by FEMA, and determined the data were reliable for  our purposes.

To address our second objective, we collected and reviewed policy,  guidance and annual grant funding notification documents from FEMA’s  Grant Programs Directorate. We reviewed the risk-based grant  assessment models for fiscal years 2008 through 2018, and we  interviewed officials from FEMA’s Grant Programs Directorate and DHS’s  Office of Intelligence & Analysis. We reviewed the elements used in  FEMA’s risk model, prior assessments of FEMA’s model, and FEMA’s  grant process to assess the status and continuing applicability of prior  recommendations. These included our prior reports and studies by  organizations such as the National Research Council (NRC) and DHS’s  Homeland Security Advisory Council. We also interviewed FEMA and  DHS officials to determine the extent to which prior recommendations  were implemented. Finally, we reviewed guidance and key practices from  OMB and our past work to compare them against FEMA’s processes and  updates to its risk model over fiscal years 2008 through 2018 to examine  what steps, if any FEMA undertook to improve its model.

We conducted this performance audit from September 2017 to  September 2018 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our finding and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.

The federal government has provided financial assistance to public and  private stakeholders for preparedness activities through various grant  programs administered by DHS through its component agency, FEMA.  Through these grant programs, DHS has sought to enhance the capacity  of states, localities, and other entities, such as ports or transit agencies,  to prevent, prepare for, protect against, respond to, and recover from, and  mitigate a natural or manmade disaster, including terrorist incidents. Two  of the largest preparedness grant programs are the SHSP and UASI  grant programs.

SHSP grants provide federal assistance to support states’  implementation of homeland security strategies to address the  identified planning, organization, equipment, training, and exercise  needs at the state and local levels to prevent, prepare for, protect  against, and respond to acts of terrorism. SHSP grants are annually  awarded to all the nation’s 56 states and territories. SHSP grant  awards are calculated in two parts. All states and territories are to  receive a minimum grant amount required by law, based on a  percentage of the total amount of SHSP and UASI appropriations in a  given fiscal year. The remaining award amounts are based on  FEMA’s risk-based grant assessment model.

UASI grants provide federal assistance to address the unique needs  of high-threat, high-density urban areas, and assists the areas in  building an enhanced and sustainable capacity to prevent, prepare  for, protect against, respond to acts of terrorism. Since 2015,  Congress has instructed through the Explanatory Statements  accompanying the annual DHS Appropriations Acts that the UASI  grants should be awarded to urban areas that reflect up to 85 percent  of nationwide risk. For the UASI program, FEMA uses the risk-based  grant assessment model each year to identify those urban areas that  will be eligible to receive funding.

Annual funding for the SHSP and UASI programs have generally declined  over the period of fiscal years 2008 through 2018, but have remained  consistent since fiscal year 2016. Figure 1 shows the changes to SHSP  and UASI programs’ annual funding during this period. For example,  annual funding for SHSP decreased from about $861 million in fiscal year  2008, to $402 million in fiscal year 2018. During this same period, annual  funding for UASI also declined, from about $782 million in fiscal year  2008 to $580 million in fiscal year 2018. However, annual funding for the  UASI program has been higher than the SHSP program since fiscal year  2010.

FEMA’s Risk-based Grant Assessment Model for Distributing Funding Awards

Risk=Threat x Vulnerability x Consequence  Threat–A natural or man-made occurrence,  individual, entity, or action that has or  indicates the potential to harm life,  information, operations, and/ or property  Vulnerability–Physical feature operational  attribute that renders an entity, asset, system,  network, or geographic area open to  exploitation or susceptible to a given hazard.  Consequence–Effect of an event, incident, or  occurrence, commonly measured in four  ways: human, economic, mission, and  psychological, but may also include other  factors such as impact on the environment.

FEMA’s risk-based grant assessment model uses three variables: Threat,  Vulnerability, and Consequence. The purpose of this model is to apply a  risk management process to provide a structured means of making  informed trade-offs and choices about how to use finite resources  effectively, and monitoring the effect of those choices. Specifically,  inherent “uncertainty” is associated with any effort to develop a risk model  such as assessing the risk of terrorist attacks, and thus, requires the  application of policy judgments and analytic assumptions. The effect that  uncertainty has on the results of the risk model can be especially  important if the model produces materially different results in response to  even small changes in assumptions, often referred to as the “sensitivity”  or “robustness” of a model’s assumptions and results.

As we reported in June 2008, FEMA’s risk-based grant methodology and  its continuous improvement efforts in estimating risk were part of a  reasonable process to assist in determining SHSP and UASI grant  allocations. For example, the risk-based grant assessment model used  from fiscal year 2001 through 2003 largely relied on measures of  population to determine the relative risk of potential grantees, and  evolved to measuring risk as the sum of threat, critical infrastructure and  population density calculations in fiscal years 2004 and 2005. Further, the  fiscal year 2006 process introduced a risk assessment model that  included measures of Threat, Vulnerability and Consequences.

In June 2008, we reported that the way the risk-based grant assessment  model measured vulnerability across states and urban areas was  limited. We found that the model considered all states and urban areas  equally vulnerable to a successful attack, and as a result, the final risk  scores were determined exclusively by the Threat and Consequence  scores. Specifically, the risk model did not measure vulnerability for each  state and urban area; rather it assigned a vulnerability score of 1.0 to  every state and urban area. We recommended that DHS and FEMA  formulate a methodology to measure variations in vulnerability across  states and urban areas. DHS components concurred with our  recommendation to measure vulnerability in a way that captures  variations across states and urban areas and apply this measure in future  iterations of FEMA’s model. In August 2011, FEMA reported that the  agency, in coordination with other DHS components, established a  Vulnerability Index for the fiscal year 2011 risk-based grant assessment  model to better capture the risk to states and urban areas, thereby  addressing our recommendation.

Other Reviews of FEMA’s Risk Methodology

DHS and the National Research Council (NRC) have also performed  reviews of FEMA’s risk assessment methodologies, providing their own  conclusions and recommendations, since our 2008 review. For example,  in 2010, the NRC reported that FEMA should strengthen its scientific  practices, such as documentation, analyses to determine how changes to  a model could affect its results, and peer review by technical experts  external to DHS, in order to further develop an understanding of the  uncertainties in its terrorism-related risk analyses.

Additionally, in 2016, the Homeland Security Advisory Council reported  that processes by which FEMA uses to assess risk should be made more  inclusive, comprehensive and effective. The Homeland Security  Advisory Council recommended the following actions to strengthen this  process:

FEMA should continue to send risk profiles to states and urban areas  to promote timely and meaningful feedback, and enable FEMA to  evaluate recommended adjustments.

Before each year’s budget submission, FEMA should discuss with  congressional appropriators the current grant allocation mechanism.

We discuss FEMA’s progress in implementing these recommendations  later in this report.

Various Factors Affected SHSP and UASI Grant Allocations to States and Urban Areas From Fiscal Years 2008 Through 2018

SHSP Allocations Reflect Both a State’s Relative Risk Score and the Minimum Allocation by Law

While all states and territories receive minimum SHSP program grant  allocations by law, the risk-based grant assessment model also informs  the grant allocation of the remaining funds to each state. However, for a  majority of states each year, their SHSP grant awards are primarily based  on a legal minimum amount. For example, in fiscal year 2012, 34 states,  like New Mexico, were awarded $2,801,000, which included $2,745,000  based on the minimum amount by law, and $56,000 was based on its risk  level. By contrast, New York was one of the high-risk states based on the  risk model. For that same fiscal year (2012) New York received a total of  $55,610,000, which included $2,745,000 based on the minimum amount  by law, plus $52,865,000 based on its risk level.

Over the period from fiscal years 2008 through 2018, the number of low- risk states whose SHSP grant awards were primarily based on the legal  minimum amount had varied from year to year, from 19 states in fiscal  year 2008, to 37 states in fiscal year 2018, as shown in table 1.

In addition, from fiscal year 2008 through fiscal year 2018, there was a  decrease in the percent of total SHSP funds awarded to states and  territories based on FEMA’s risk model. The percent of total SHSP  funding awarded to states and territories based on FEMA’s model ranged  from a high of 63 percent in fiscal year 2009 (about $536 million of the  $851 million of total SHSP funds), to 51 percent (about $149 million of  $294 million of total SHSP funds) for fiscal year 2012. For fiscal year  2018, the total SHSP funds awarded to states and territories based on the  risk-based grant assessment model was 55 percent—about $220 million  of $402 million.

For specific details on SHSP grant allocations for fiscal years 2008  through 2018 by states and territories, see appendix I, table 4.

UASI Grantee Eligibility and Allocations Reflect Results from FEMA’s Risk- Based Grant Assessment Model

The UASI program uses FEMA’s risk-based grant assessment model to  identify which of the 100 of the nation’s largest urban areas are eligible for  grant awards in a particular fiscal year. Then, FEMA’s risk model also  helps inform DHS leadership’s decisions on the final funding amounts for  each grantee, according to FEMA officials. Specifically, FEMA annually  assesses the risk of the 100 most populous metropolitan statistical  areas—a geographical region with a relatively high population density at  its core and close economic ties throughout the area—as defined by the  Office of Management and Budget, in determining the eligible urban  areas. From these 100 eligible urban areas, the risk-based grant  assessment model identifies those urban areas that reflect recent  congressional intent that up to eighty-five percent (85%) of nationwide  risk is funded each year. Those urban areas below this 85 percent  threshold are ineligible for UASI grant awards in that fiscal year,  according to FEMA officials.

From fiscal years 2008 through 2018, the number of UASI grantees has  remained relatively stable since fiscal year 2011. As figure 2 shows, the  annual number of grantees has fluctuated from fiscal years 2008 through  2018, ranging from 60 to 64 grantees during fiscal years 2008, 2009 and  2010. However, since fiscal year 2011 the number of UASI grantees has  averaged 31 urban areas, with a high of 39 urban areas in fiscal year  2014 and a low of 25 urban areas in fiscal year 2013. For fiscal year  2018, 32 urban areas were UASI grantees.

For additional details on UASI grant awards for fiscal years 2008 through  2018 by urban areas, see appendix I, table 5.

Because the UASI grant program is required by annual congressional  guidance to fund only those urban areas that comprise up to 85 percent  of risk nationally, this eligibility cut off can result in different urban areas  being eligible from one year to the next. Specifically, as we demonstrated  in June 2008, the variation of risk across urban areas takes on the  distribution curve illustrated in figure 3.

The few urban areas with the highest relative risk score are represented  along the steep part of the relative risk curve. For example, those urban  areas receiving the highest awards, informed by their risk scores and  ranks, are generally the same each fiscal year: New York City, Los  Angeles, and Chicago, as seen in table 2.

Those urban areas that have less relative risk are represented along the  flat section of the curve. There are urban areas with less risk that may not  fall within the 85 percent of risk nationally during a specific year and thus  would be ineligible to receive UASI funding during that year. Table 3 lists  the lowest-funded urban areas for the last 5 fiscal years, based on our  analysis of the funding amounts each received within each fiscal year. For  example, during the period of fiscal year 2008 through fiscal year 2018  San Antonio, Texas, and Hampton Roads, Virginia only received awards  in fiscal years 2008, 2009, 2014, 2017, and 2018.

In addition to changes to urban areas’ risk ranking from one year to the  next, the amount that an urban area received of the total amount of UASI  funds in a given year can change. FEMA has established a process for  developing grant award funding options based on the results of the risk- based grant assessment model. These funding options are provided to  the Secretary of Homeland Security for consideration and final approval.  According to FEMA officials, the options may vary each year based on  DHS leadership’s priorities and concerns at the time; however, all options  represent only those eligible grantees that represent up to 85 percent of  the nation’s risk, as determined by the risk-based grant assessment  model.  In fiscal year 2013, FEMA shifted its UASI grant funding to a  process referred to as “funding bands.” In fiscal year 2018, for example,  UASI grantees such as Orlando, Florida; Hampton Roads, Virginia; and  San Antonio, Texas each received a $1.5 million UASI grant, whereas a  grouping of UASI grantees that included Sacramento, California;  Pittsburgh, Pennsylvania; and Portland, Oregon each received $2.5  million.

According to FEMA officials, grouping jurisdictions with similar risk scores  into funding bands is an effort to stabilize and retain grantees’ funding  levels over multiple years, as annual UASI grants will fund projects that  are multiyear investments and carried out over a 24 to 36-month  performance period. For example, if one jurisdiction increased by four  ranks and another jurisdiction in the same group dropped six ranks, the  two jurisdictions would stay in the same funding band if the overall risk  scores remained close together. The purpose of the funding bands is to  ensure that some consistency in funding exists for jurisdictions, given  minor changes in the relative risk ranking. FEMA looks at the natural risk  breaks and historical grant allocation data for each year. For example,  each year FEMA presents for consideration by DHS leadership the  historical funding and the number of urban areas that have been placed in  specific funding bands in prior grant years, if any, and the differences  between the relative risk scores in the current fiscal year. According to  FEMA officials, the last few grant years had produced similar funding  bands, which are subject to change depending on DHS leadership’s final  decisions.

FEMA Has Improved Its Risk-based Grant Assessment Model, but Additional Steps Could Further Strengthen Its Model

FEMA Has Taken a Number of Steps to Improve the Risk-based Grant Assessment Model for Allocating SHSP and UASI Grants

Since 2008, FEMA has taken a number of steps to assess and improve  its risk-based grant assessment model for allocating grants based on past  reviews, our prior recommendations, and various changes related to  evolving terrorist threats and real-world scenarios. For example, FEMA  added a Vulnerability Index to its risk model in 2011 in response to our  2008 recommendation. Most recently, for fiscal year 2018, FEMA has  included a “soft target index.” According to FEMA officials, this index was  added to account for the current threat for areas where crowds  congregate. Figure 4 illustrates the timeline of FEMA changes to the risk- based assessment model and prior assessments.

Figure 5 depicts the risk-based grant assessment model used for fiscal  year 2018 SHSP and UASI grant awards.

Figure 6 depicts the changes in the Threat, Vulnerability, and  Consequence indexes used in the risk-based grant assessments model  for fiscal year 2008, compared to 2018. As we noted above, the 2008 risk  model did not measure Vulnerability for each state and urban area, and  risk scores were essentially determined by Threat and Consequences  indexes.

Changes to the Consequence Index can have the most impact on the  relative risk scores because of the weight of this index (50 percent),  relative to the weights for the Threat and Vulnerability indexes. Further,  the weight for population within the Consequence Index represented 30  percent of the total fiscal year 2018 risk model value. As a result, the  weight for the population index was greater than the weights of either the  Threat Index or Vulnerability Index, each 25 percent. FEMA has  decreased the weight for the population index over time, from 40 percent  in 2008 to 30 in 2011, where it has remained consistent through 2018.

For fiscal year 2018, FEMA modified how the population index was  calculated within the Consequence Index to better account for attacks  staged by individuals, so-called lone wolves. FEMA did so, in part, by  reducing the importance of population density within the population index.  In past risk models, the population index had favored high-density, high- rise urban areas, commensurate with building destruction scenarios —  the 9/11-style attack scenarios that focused on large building destruction  events, according to FEMA officials. The 2018 change to cap population  density in the population index reduces the impact those extremely-dense  population areas have in the methodology, according to FEMA officials.

The other measures used to make up the Consequence Index remain  relatively unchanged since our review in 2008, although FEMA has  renamed the indexes.

As explained earlier, FEMA added a Vulnerability Index to its risk-based  grant assessment model in 2011, in response to our 2008  recommendation. According to FEMA officials, the Vulnerability Index  helps support what DHS is trying to protect, primarily the protection of  citizens and critical infrastructure. For example, the Vulnerability Index  includes a measure designed to assess the extent that certain types of  national critical infrastructure assets may be considered for possible  attack. This Targeted Infrastructure Index measure uses actionable  intelligence on types of critical infrastructure targets, such as aviation,  mass transit and commuter rail. FEMA works with DHS’s National  Protection and Programs Directorate to match its critical infrastructure  dataset to actionable intelligence from DHS’s Office of Intelligence &  Analysis to compile this measure.

Vulnerability Index   Designed to measure the likelihood of a  successful attack in a state or urban area,  based on a) intelligence information of those  critical infrastructure assets identified by  foreign or domestic terrorists; b) the extent of  international borders entries (land, sea and  air) located in a state or urban area, and c)  special events where crowds congregate and  are susceptible to homegrown extremism and  lone wolf attacks.

For the fiscal year 2018 grant, FEMA has included a “soft target index.”  According to FEMA officials, this index was added to account for the  current threat for areas where crowds congregate. Based on previous  feedback received through this process, FEMA updated the fiscal year  2018 risk methodology to better account for the nation’s current threat  environment. The soft target index is composed of two new data  elements:

Visitors—domestic and international—using the same data used in  the calculation of the Population Index; and

Special events measure—uses Special Event Assessment Rating  data from DHS Office of Operations Coordination to identify large  events that are state and local events that may require federal  assistance. Examples of such events include the Super Bowl, the  Boston Marathon and New Year’s Eve in Times Square.

In fiscal year 2018, FEMA added a new “isolation” measure to account for  the challenges of response for those states, territories, and urban areas  outside the contiguous United States, who rely on prompt mutual aid from  neighboring jurisdictions. According to FEMA officials, the isolation data  element was included as a response to challenges the agency witnessed  as a result of the 2017 Hurricane season, specifically the unique  challenges of distant U.S. territories receiving timely mutual aid from other  states. For example, if Hawaii, Guam or American Samoa were attacked,  there would be little to no outside help for a number of days. As a result,  FEMA modified the fiscal year 2018 Border Crossings data element  weight, which was dropped from 6 percent to 4 percent, in order to  establish a 2 percent weight for the isolation measure.

The weight of the Threat Index was raised from 20 percent to 30 percent  in fiscal year 2011, and has been modified again for fiscal year 2018.  Specifically, according to FEMA and DHS officials, DHS leadership made  a policy decision to reduce the Threat Index’s weight from 30 percent in  2017, to 25 percent in 2018, due to the change in current threat  environment, since Congress directed FEMA in the Explanatory  Statement accompanying the FY 2017 DHS Appropriations Act to review  the risk model to account for this changing threat environment. FEMA  officials further stated that they assumed, as domestic terrorism and soft  targets are considered to be prevalent nationwide and pose more of a  challenge in identifying the source of actionable threats. FEMA officials  stated that this modification to the Threat Index better reflects real-world  scenarios.

Since fiscal year 2012, FEMA has included information on domestic  terrorism as well as international terrorism in its Threat Index. According  to DHS officials, home grown extremism is also a likely threat, often  through lone wolf attacks. DHS officials decided to assign all urban areas  a minimum threat score to reflect the fact that all areas have some level  of threat. According to DHS officials, the addition of a domestic terror  threat measure resulted in a decrease in the variation of threat scores  across states and urban areas. According to DHS officials, lone wolf  attacks are difficult to determine who the actors may be, or when and  where they will attack.

FEMA annually transmits risk profile information to states and urban  areas to promote timely and meaningful feedback. According to FEMA  officials, draft risk profiles are sent to all 56 states and territories and 100  eligible urban areas closely after the enactment of DHS’s annual  appropriations. States and urban areas are given a 2-week period prior to  the release of the Notices of Funding Opportunity to review their draft risk  profiles and provide FEMA any comments or data corrections that should  be considered. According to FEMA officials, it encourages and welcomes  stakeholders to make suggestions for new or different data sets for the  subsequent fiscal year's risk assessment at any time during the year  convenient to the stakeholder. FEMA also conducts webinars during this  period to can explain the risk profiles in detail, as well as discuss any  updates to data sets and/or any enhancements to the risk assessment.  This will often result in feedback on data elements and the methodology  of the risk-based grant assessment model, according to FEMA officials.  According to FEMA officials, this feedback process has been used to help  guide FEMA’s consideration of enhancements to the risk-based grant  assessment model. For example, FEMA officials noted that this process  helped them in their efforts to develop the soft targets index into the 2018  risk model.

FEMA Does Not Fully Make Use of Recognized Scientific Practices in Maintaining Its Risk Assessment Model

In 2010, the National Research Council (NRC) recommended that  incorporating scientific practices can provide decision makers a further  understanding of the effects of its policy judgments and assumptions—i.e.  addressing uncertainties—in its terrorism-related risk analyses. The NRC  identified “good scientific practice” for model-based work. Specifically, the  NRC recommended that detailed documentation for all risk models,  including rigorous mathematical formulations, be implemented  department-wide. Additionally, the NRC recommended that all risk  models undergo verification and validation—or a sensitivity analysis at the  least—of its risk-based grant assessment model. Finally, the NRC  recommended that FEMA should undertake an external peer review by  technical experts outside of DHS, and review its risk-informed formulas in  order to identify issues such as logic flaws, evaluate the ramifications of  the choices of weightings and parameters, and improve the risk model’s  transparency. However, FEMA has not fully adopted these scientific  practices for its risk-based grant assessment model.

Documentation: FEMA documentation on the sources of data used for  the model’s calculations does not include information that would enable a  reviewer to understand the underlying assumptions that form the basis for  its risk-based grant assessment model—such as the size of the weights  assigned to Threat, Vulnerability, and Consequence, or the justification  for changes to these weights from one year to the next. FEMA officials  stated that they focus their limited time and resources on developing the  executive summary-level materials that DHS leadership will use to  determine final grant eligibility and grant allocation amounts. Also, to a  lesser extent, FEMA officials said they rely on the expertise of the subject  matter experts from DHS’s Office of Intelligence and Analysis, and DHS’s  National Protection and Preparedness Division’s Office of Cyber and  Infrastructure Analysis, parts of DHS that contribute to the annual risk  assessment process.

In April 2018, we identified documentation as one of the key  methodological elements to the baseline structure of an economic  analysis. Specifically, the elements include that the analysis is clearly  written with a plain language summary, has clearly labeled tables that  describe the data used and results, and has a conclusion that is  consistent with these results. The analysis cites all sources used and  documents that it is based on the best available economic information.  The analysis documents that it complies with a robust quality assurance  process and, where applicable, the Information Quality Act, and should  disclose the use and contributions of contractors and outside consultants.  FEMA officials agreed with our analysis of FEMA’s supporting  documentation, and officials stated that maintaining additional  documentation could further assist reviewers. Documenting how subject  matter expert assumptions are made would help FEMA increase the  transparency of the model for key internal and external stakeholders.

In-Depth Analyses: Similarly, we could not determine whether FEMA  sufficiently performed all the analyses of the model’s sensitivity needed to  determine how changes to its risk-based grant assessment model could  affect the resulting risk scores. FEMA officials stated that they have only  analyzed the effect of a data element when it has been added to the  model (e.g.: the Soft Target Index in 2018). Further, FEMA officials were  unable to provide us with documentation on their sensitivity analyses  processes or their results.

DHS’s Risk Lexicon states that sensitivity analysis can be used to  examine how individual variables can affect the outputs of risk  assessment methodologies. In addition, OMB Circular A-94 recommends  that the outcomes from a risk model should be analyzed to determine  how sensitive such outcomes are to changes in the model’s  assumptions. The assumptions that deserve the most attention will  depend on the dominant elements and the areas of greatest uncertainty  of the program being analyzed. In addition, research in the actuarial  sciences also states that sensitivity analysis “is of fundamental  importance to risk analysts, especially in the presence of complex  computational models with uncertain inputs.”

As we stated earlier, understanding the extent that uncertainty has on the  results of the model can be especially important if the model produces  materially different results in response to even small changes in  assumptions—often referred to as the “sensitivity” or “robustness” of a  model’s assumptions and results. We have reported on FEMA’s risk- based grant assessment model in June 2008 and March 2013, where we  found grant years when the risk model was sensitive to even small  changes. For example, we noted that a potential increase or decrease  in a measure would have resulted in one urban area displacing the  eligibility of another, thereby potentially shifting funding as well. FEMA  officials stated that they focus their limited time and resources on  developing the executive summary-level materials that DHS leadership  will use to determine final grant eligibility and grant allocation amounts.  FEMA officials agreed that they could better document the steps used in  their analyses across all the model’s measures and weights so that a  complete understanding of potential impacts are documented and can be  made available to leadership when making decisions about changes.

FEMA’s implementation of sensitivity analyses could help the agency to  assess changes to the risk-based grant assessment model including the  introduction of new data elements into Threat, Vulnerability, and  Consequence indexes, the modifications to how existing data elements  are calculated, and the changing of the weights assigned to the Threat,  Vulnerability, and Consequence indexes. Further, FEMA’s  implementation of sensitivity analyses has the ability to show decision  makers the impact or predicted impact of adjustments to FEMA’s risk- based grant assessment model, including with potential shifts in funding  towards or away from certain grantees.

Use of External Peer Review: FEMA has not subjected its risk-based  grant assessment model to a peer review by independent, external  technical experts, as previously recommended in 2010 by the NRC.  According to FEMA officials, its risk assessment methodology has  undergone comprehensive internal reconsideration over time to better  reflect real-world scenarios, but such reviews have not included external  peer reviews. FEMA officials stated that its risk-based grant assessment  model has gone through past reviews including a review as part of DHS’s  quadrennial review in 2014, and the model is reviewed by internal subject  matter experts from DHS’s Office of Intelligence and Analysis, and DHS’s  National Protection and Preparedness Division’s Office of Cyber and  Infrastructure Analysis as part of the annual risk assessment process.  FEMA officials stated that the agency is exploring the possibility of  participating in a DHS collaborative group to internally review and provide  feedback on the model’s underlying assumptions and methods. Such a  group could review the underlying components of the current risk-based  grant assessment model and suggest improvements, as well as present  and evaluate other risk assessment theories and approaches. FEMA  officials told us they have encountered time and resources constraints on  establishing an external peer review process.

As we have previously reported, independent external peer reviews can  increase the probability of success by improving the technical quality of  projects and the credibility of the decision-making process, and provide  reasonable assurance that the agency’s approach is reproducible and  defensible. In addition, in December 2004, OMB issued the  memorandum “Final Information Quality Bulletin for Peer Review” which  established government-wide guidance aimed at enhancing the practice  of peer review of government science documents. OMB noted that peer  review can increase the quality and credibility of the scientific information  generated across the federal government, which was an effort to improve  the quality of the scientific information upon which policy decisions are  based. OMB also noted that, while peer review may take a variety of  forms, agencies will need to consider at least the following issues when  coordinating an external peer review: individual versus panel review;  timing; scope of the review; selection of reviewers; disclosure and  attribution; public participation; disposition of reviewer comments; and  adequacy of prior peer review.

These scientific processes are designed to help decision makers better  understand the impact or predicted impact of risk management  alternatives, and provide greater confidence in the reliability of the risk  assessment model’s results. Full implementation of these processes  better position FEMA to provide further assurances that their risk-based  grant assessment model and grant allocation approaches are reasonable,  of high-quality, and credible.

Given that risk management has been endorsed by the federal  government as a way to direct finite resources to states and those urban  areas that are most at risk of terrorist attack, it is important that FEMA’s  risk-based grant assessment model supports the application of policy  judgments and analytic assumptions in the model’s role of allocating  those limited resources. Decreased funding levels for SHSP and UASI  grant programs have increased the importance of using risk management  techniques to more effectively target finite federal dollars. DHS and FEMA  have strengthened its risk-based grant assessment model for allocating  grants, taking into account analysis and recommendations from a variety  of reviews. These improvements include the addition of a Vulnerability  Index and modifications to the Threat Index. We have identified  opportunities where FEMA could strengthen its scientific practices. First,  documenting the model’s underlying assumptions and the results of  sensitivity analysis can assist decision makers in better understanding the  predicted impact of risk management alternatives. Second, expanding the  use of sensitivity analysis could further enhance the model. Developing a  greater understanding of the how uncertainty affects its risk-based grant  assessment model’s results helps achieve the objectives of risk  management. Third, coordinating an independent external peer review of  the methodology of its risk-based grant assessment model would better  position the agency to provide reasonable assurance that FEMA’s risk  model and grant allocation approach that FEMA uses for its SHSP and  UASI programs are reasonable, of high-quality, and credible. Applying  such scientific practices could assist FEMA in further strengthening its  risk-based grant assessment model.

We are making the following three recommendations to FEMA.

The FEMA Administrator should fully document the underlying  assumptions and justifications that form the basis of the risk-based  grant assessment model, such as the size of the weights assigned to  Threat, Vulnerability, and Consequence, or the justification for  changes to these weights from one year to the next.

The FEMA Administrator should perform sensitivity analyses to verify  how changes to the risk-based grant assessment model could affect  the resulting risk scores, and document the results.

The FEMA Administrator should take steps to coordinate an  independent, external peer review of its risk-based grant assessment  model.

We provided a draft of this product to the FEMA and DHS for comment.   In its comments, reproduced in appendix II, FEMA generally concurred  with our findings and three recommendations.

In FEMA’s concurrence to our first recommendation that the agency fully  document the underlying assumptions and justifications that form the  basis of the risk-based grant assessment model, FEMA requested that  GAO consider this recommendation resolved and closed as implemented.  As part of FEMA’s response, they reiterate their process of providing draft  Risk Profiles to all 100 urban areas and 56 states and territories and their  annual communications to Congress on how FEMA calculated risk and  computed grant awards.  We recognized FEMA’s stakeholder feedback  efforts in this report. However, as we noted, FEMA’s documentation on  the sources of data used for the model’s calculations does not include  information that would enable a reviewer to understand the underlying  assumptions that form the basis for its risk-based grant assessment  model.   Further, as stated earlier, documentation is one of the key  methodological elements to the baseline structure of this type of analysis,  documenting that it complies with a robust quality assurance process and,  where applicable, the Information Quality Act, and should disclose the  use and contributions of contractors and outside consultants. In order to  fully implement this recommendation, documenting how subject matter  expert assumptions are made would help FEMA increase the  transparency of the model for key internal and external stakeholders, and  will further support the efforts of an independent external peer review of  FEMA’s risk-based assessment model.

Regarding the second recommendation, FEMA concurred, stating that the  agency will expand the use of sensitivity analysis to review the entire risk  methodology, and will also document these results for leadership review,  as appropriate.  Finally, regarding the third recommendation, FEMA  concurred, stating that they will coordinate an independent external peer  review and develop a detailed written response to leadership for further  appropriate action.

FEMA and DHS also provided technical comments, which we  incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Homeland Security, and other interested  parties. This report will also be available at no charge on our Web site at  http://www.gao.gov. Should you or your staff have any questions  concerning this report, please contact me at (202) 512-8777 or  CurrieC@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.

Appendix I: Grant Funding and Awards for State Homeland Security Grant Program (SHSP), and the Urban Area Security Initiative (UASI) Grant Program for Fiscal Years 2008 Through 2018

Fiscal Years Mississippi Total Award Amount above legal minimum

Fiscal Years West Virginia Total Award Amount above legal minimum

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

Chris P. Currie, at (202) 512-8777 or CurrieC@gao.gov.

In addition, key contributors to this report were Aditi Archer, Chris  Keisling, Assistant Director, John Vocino, Analyst-in-Charge, Chuck  Bausell, Dominick Dale, Dorian Dunbar, Eric Hauswirth, Serena Lo, Heidi  Nielson, and Hadley Nobles.

The Federal Emergency Management Agency distributes money to states, territories, and urban areas to help them get ready for potential terrorist attacks. The amounts provided are based in part on a model FEMA uses to rank the risk posed to each area. Locations deemed at higher risk of an attack with major consequences, for example, may receive more money.

We looked at FEMA's main grant programs and what it has done to improve its risk model. We recommended FEMA take additional steps to bolster the model's quality and credibility, including fully documenting the assumptions on which the model is based and subjecting it to external review.


Reserve Component Travel: DOD Should Assess the Effect of Reservists' Unreimbursed Out-of-Pocket Expenses on Retention

About 91 percent of DOD's 811,000 reservists are part-time, performing military service in addition to civilian employment and careers. These reservists may have to travel to perform required military training or other duties.

The National Defense Authorization Act for Fiscal Year 2017 contains a provision for GAO to review the cost of travel for members of the reserve components. This report (1) describes the conditions under which reservists may incur unreimbursed out-of-pocket travel expenses in connection with their service, and (2) addresses the extent to which DOD has assessed the effect of reservists' unreimbursed out-of-pocket travel expenses on retention.

GAO reviewed DOD's Joint Travel Regulations and interviewed officials to determine conditions under which reservists might incur unreimbursed travel expenses. It also compared DOD's efforts to analyze the effect of such expenses with federal internal control standards, which state that management requires quality information to make informed decisions and evaluate an entity's performance in achieving key objectives.

Reservists may incur unreimbursed out-of-pocket expenses under certain conditions in connection with their service. Although the Department of Defense's (DOD) six reserve components reported paying or reimbursing $925 million in travel costs for reservists in fiscal year 2015, the most recent year for which data were available, reservists may still incur various expenses that are not reimbursable under DOD's travel regulations.

Officials responsible for travel regulations told us that unreimbursed travel expenses for reservists generally arise because it is DOD's policy to: (1) not provide reimbursement, except in limited circumstances, for the cost of travel to attend Inactive Duty Training (i.e., the “1 weekend a month” training commitment for reservists) and (2) consider longer duration training or assignments as a Permanent Change of Station—a change in reservists' home of record—and not as temporary travel. The National Defense Authorization Act for 2008 established a reimbursement program for Inactive Duty Training travel costs, but reservists must meet certain eligibility criteria, such as serving in a critical occupation, and not all service Secretaries have chosen to participate. Under the program, reimbursement is limited to $300 for each roundtrip to the training location. Further, DOD's policy to consider longer duration training or assignments as a Permanent Change of Station may also result in unreimbursed expenses. Specifically, according to DOD officials, reservists may have to maintain two households if, because of their part-time status, they decide not to move themselves and their families to the location of Active Duty Training for 140 days or longer, or of other active duty assignments for 181 days or longer.

DOD and the services have conducted a few limited assessments of the potential effect of reservists' unreimbursed travel expenses on the retention of reservists. However, several DOD reports and studies and officials whom GAO interviewed have expressed concern that such unreimbursed expenses may, among other factors, be a challenge for reservists and may therefore negatively affect retention. For example, a 2012 survey commissioned by the Army Reserve of a small sample of reservist officers potentially eligible for battalion command positions reported that unreimbursed travel costs were among several factors that could influence their decision to apply for these positions. DOD and the reserve components are considering changes to reserve travel policy to mitigate the effect of unreimbursed expenses on reservists, by, for example, increasing the $300 limit for Inactive Duty Training reimbursement. However, without the benefit of quality information, DOD risks not managing the potential influence of these policies on reservists' retention or agency expenditures.

GAO is recommending that DOD collect quality information and conduct an analysis of the potential effects of reservists' unreimbursed travel expenses on retention, and respond to these risks by considering the costs and benefits of any possible actions to address the identified issues. DOD concurred with this recommendation.

Of the Department of Defense’s (DOD) 2.1 million servicemembers in  2016, more than 811,000 serve in the department’s six reserve  components–the Army National Guard, the Army Reserve, the Air Force  Reserve, the Air National Guard, the Marine Corps Reserve, and the  Navy Reserve. In 2011, DOD stated that it had transformed its reliance  on the reserve component, from considering it to be a strategic force of  last resort to using the component as an operational reserve to be  deployed on a periodic basis. When not deployed, reservists travel to  perform training and other reserve duties, ranging from monthly weekend  drills to long-duration training for career fields such as cyber defense or  for maintaining skills such as proficiency in foreign languages. About 91  percent of reservists are part-time, performing military service in addition  to civilian employment, and may have to travel to perform training or other  duties if they do not live close to the location where they train or are  needed for other duties. DOD’s 2012 Report of the Eleventh Quadrennial  Review of Military Compensation noted concerns over reservists’  unreimbursed expenses to perform military service, and a 2008 report by  the Commission on the National Guard and Reserves cited similar  concerns as having possible effects on retention.

The Under Secretary of Defense for Personnel and Readiness has overall  responsibility for commercial travel management and policy. The Defense  Travel Management Office oversees commercial travel management and  serves as the focal point for commercial travel within DOD. This office is  also responsible for any updates and revisions to the Joint Travel  Regulations, DOD’s central source of travel policy for all  servicemembers. DOD is currently updating the Joint Travel Regulations,  which is expected to be published in fall 2017.

The National Defense Authorization Act for Fiscal Year 2017 contains a  provision for us to review the cost of travel for members of the reserve  components. This report (1) describes the conditions under which  reservists may incur unreimbursed out-of-pocket travel expenses in  connection with their service, and (2) addresses the extent to which DOD  has assessed the potential effect of reservists’ unreimbursed out-of- pocket travel expenses on the retention of reservists. Appendix I provides  DOD’s reported fiscal year 2015 travel costs for the reserve components.  In addition, in June 2017, we provided a briefing to congressional  committees on these costs.

To address our first objective, we reviewed DOD’s Joint Travel  Regulations and interviewed officials from the Defense Travel  Management Office, each service’s reserve policy office, and officials  from each component responsible for travel issues to understand and  characterize scenarios under which reservists would not be reimbursed  for travel expenses. To address our second objective, we collected and  reviewed selected DOD studies and reports on the subject of reservists’  out-of-pocket travel expenses, and compared DOD’s efforts to analyze  the potential effects of these expenses with federal internal control  standards. These standards state that management requires quality  information to make informed decisions and evaluate the entity’s  performance in achieving key objectives and addressing risk. These  standards further require that management identify, analyze, and respond  to risks related to achieving the defined objectives. To report on DOD’s  reported fiscal year 2015 travel costs for the reserve components we  obtained and reviewed fiscal year 2015 execution cost data, the most  recent complete data available. We interviewed budget officials from each  reserve component and from the Office of the Under Secretary of  Defense (Comptroller) concerning their efforts to assess the  completeness and accuracy of these cost data and determined that they  were sufficiently reliable for the purpose of summarizing the cost of  reserve component travel for fiscal year 2015.

We conducted this performance audit from January 2017 to October 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The Selected Reserve comprises over 811,000 full- and part-time  members from the military services’ respective National Guard and  reserve components, whom DOD can call to active duty to augment  military forces in time of war or national emergency. DOD requires these  reservists to maintain readiness by participating regularly in training to  maintain the military skills needed to perform their mission. About 91  percent of the members of the Selected Reserve, or 735,876 reservists,  are part-time, performing military service in addition to their civilian  employment and careers. Reservists typically train for about 1 weekend  a month and 2 weeks a year. Reservists may also be required to  participate in longer duration training to develop and maintain specialized  skills related to their military occupation, such as cyber specialists, or to  perform other activities such as backfilling positions in other reserve or  active units. The following are descriptions of reservists’ required  training and other duties:

Annual Training: All six reserve components require an annual  training period, typically 2 weeks, to acquire and maintain required  military skills.

Inactive Duty Training: This training is commonly referred to as the  “1 weekend a month” commitment, and reservists fulfill this  commitment in connection with prescribed training or maintenance  activities of the units to which they are assigned.

Active Duty for Training: So that reservists acquire and maintain  required military skills, individuals serving as reservists participate in  training programs such as initial basic training and advanced  individual training, and may attend full time specialized schools. The  duration of Active Duty for Training varies considerably, from days to  several months.

Active Duty Other than Training: All six reserve components may  require that reservists perform other support activities, such as  backfilling a position in a reserve or active unit.

For a variety of reasons, reservists may not live in the same location  where they train. For example, reservists may relocate for their civilian  occupation, and officials told us that as reservists are promoted,  command opportunities are more geographically dispersed. As a result,  travel may be necessary to facilitate their service. DOD’s six reserve  components reported paying or reimbursing over $925 million in travel  costs for reservists in fiscal year 2015, representing about 4.3 percent of  the total obligations identified in the Reserve Personnel accounts. With  an actual part-time endstrength of 742,683 reservists in fiscal year 2015,  DOD spent an average cost of about $1,246 per reservist. Officials told us that DOD does not specifically collect and track data on reservists’  unreimbursed travel expenses, which are therefore unknown. Officials  told us that reservists process their travel claims through DOD-wide or  military-service-based electronic data systems, such as the Defense  Travel Service or the Air Force’s Reserve Travel System, or sometimes  using hard-copy forms, depending on the type of duty performed, the  reserve component, and other factors. DOD’s Joint Travel Regulations  govern the extent to which reservists are eligible to be reimbursed for  travel expenses to participate in required training or in other duties. The  regulations authorize the reimbursement of different types of expenses  depending on the nature and duration of the assignment. Eligible  reimbursements include:

Per diem, which includes reimbursement for food, temporary lodging,

Transportation expenses, ranging from reimbursement for mileage  traveled in reservists’ private vehicles to reimbursement for  commercial flights;

Permanent Change of Station reimbursements related to reservists  changing their home of record to the location of the assignment, such  as reimbursement for the movement of household goods; and

Basic Allowance for Housing, which is based on the costs of adequate  rental properties for civilians with comparable income levels in the  same location as the permanent duty station, which in the case of  reservists is generally the location of their home; is received when  reservists are in an active duty status, which includes Active Duty for  Training and Active Duty Other than Training; and is determined  based on the duration of reservists’ active duty assignments.

Reservists also receive cash compensation for the various types of  training and other duties they perform; non-cash compensation, such as  access to TRICARE Reserve Select and education benefits; and deferred  compensation, such as participation in the military retirement system. In  addition, reservists may be able to take advantage of a federal tax  deduction for out-of-pocket travel expenses associated with their  service.

Reservists May Incur Out-of-Pocket Travel Expenses under Certain Conditions

Reservists may incur expenses under certain conditions in connection  with their service that are not reimbursable under DOD’s travel  regulations. Officials responsible for travel regulations and reserve policy  issues told us that this can occur because: (1) the cost to attend Inactive  Duty Training is a reservist’s responsibility, except in limited  circumstances; and (2) DOD designates longer duration training or  assignments as a Permanent Change of Station—a change in a  reservist’s home of record—and not as temporary travel.

Travel Expenses to Attend Inactive Duty Training Are the Responsibility of Reservists Except for in Limited Circumstances

Under most circumstances, travel expenses to and from the 1 weekend a  month training commitment are reservists’ responsibility with no  reimbursement provided, and as a result reservists may incur  unreimbursed travel expenses to attend this training. Specifically, the  Joint Travel Regulations states that a reserve component member  performing Inactive Duty Training ordinarily receives no travel or  transportation allowances, particularly when the training duty is performed  at the reservist’s assigned unit location. This principle is reflected in  travel policy such as the Navy Reserve’s requirement that reservists who  live more than 100 miles from their Inactive Duty Training site sign a  waiver acknowledging that they will not be reimbursed for travel  expenses. Navy travel policy, citing a previous version of the Joint  Travel Regulations, states that as part of the requirement to perform  Inactive Duty Training, “inherent to this obligation is the travel between  the member’s home and the location at which the member normally  performs drills” with no reimbursement provided.

To mitigate expenses incurred by reservists traveling long distances, the  National Defense Authorization Act for Fiscal Year 2008 established a  reimbursement program for Inactive Duty Training whereby each  component may, at the discretion of the service Secretary and under  certain circumstances, provide reimbursement of up to $300 in expenses  for each roundtrip to the training location. The Joint Travel Regulations  further specifies that reservists must travel no fewer than 150 miles or  greater one way from their primary residence to their normal drilling site to  be eligible. DOD spent nearly $33.5 million on Inactive Duty Training  travel costs in fiscal year 2015. While each service Secretary decides  whether an individual component can participate in the program, the Joint  Travel Regulations requires such programs to make servicemembers  eligible for reimbursement when they meet one of the following criteria:  They are qualified in a skill designated as critically short by the Secretary assigned to a unit of the Selected Reserve with a critical staffing  shortage, or in a pay grade in the reservists’ component with a critical  staffing shortage; or assigned to a unit or position that is disestablished or relocated as a  result of Base Realignment and Closure or other force structure  reallocation.

See table 1 for scenarios illustrating reimbursement eligibility for Inactive  Duty Training expenses in the Army Reserve.

Three of the six reserve components have established policies to allow  for reimbursement of expenses of travel related to Inactive Duty Training,  according to component-specific criteria. The Marine Corps Reserve and  the Air Force Reserve authorize Inactive Duty Training reimbursement for  several occupations, and in the case of the Marine Corps Reserve, entire  rank levels. The Army Reserve authorizes reimbursement, but according  to its policy targets reimbursements to soldiers and units with the highest  payoff in achieving readiness. Specifically, Army Reserve commanders  establish Inactive Duty Training reimbursement policy that designates and  prioritizes positions, units, and occupational specialties eligible to  participate. Both Air National Guard and Army National Guard officials  told us that their respective components do not authorize Inactive Duty  Training reimbursement. Similarly, Navy officials told us that the Navy  does not participate in the reimbursement program, primarily because  under its training construct Navy reservists conduct most Inactive Duty  Training at a Navy Operational Support Center close to their homes,  thereby limiting the training that may occur at a further distance from their  homes to a minority of sessions.

Travel distances for reservists to their drilling site may have increased  over time. For example, the 2012 Report of the Eleventh Quadrennial  Review of Military Compensation noted that reservists traditionally lived  near a reserve site or drilling location, but reported that at the time of its  review more than 100,000 reservists lived more than 100 miles from their  drilling locations. Further, according to a 2008 report by the Commission  on the National Guard and Reserves, after Base Realignment and  Closure actions some reservists may have fewer locations available to  them to perform such training. As a result, reservists may be travelling  greater distances to attend such training. Officials also told us that the  travel distances required to attend Inactive Duty Training can be further  increased as reservists progress in their careers in certain occupational  specialties or ranks. For example, officials from the Marine Corps  Reserve told us that as reservists are promoted to higher ranks, there are  fewer positions, which can result in long-distance travel by reservists,  while Army Reserve officials told us that some reservists may turn down  command positions to avoid long-distance travel.

DOD’s Joint Travel Regulations Treats Long- Duration Training and Other Assignments of Long-Duration as a Permanent Change of Station

DOD’s Joint Travel Regulations treats Active Duty for Training and other  assignments of long-duration as a Permanent Change of Station, or a  change in a reservist’s home of record, generally his or her civilian home,  and not as Temporary Duty. The treatment of long-duration training or  other assignments as a Permanent Change of Station applies equally to  reservists and active component members, as DOD travel regulations  require all military personnel at a given training or assignment to be in the  same status. However, officials told us that due to the interim nature of  such assignments reservists are unlikely to move their families, and  reservists may incur unreimbursed expenses due to the cost of  maintaining two homes. For example, according to a reserve policy  official, based on an internal analysis, about two-thirds of Air Reserve  members on long-duration training do not move from their civilian homes.  Further, the 2012 Report of the Eleventh Quadrennial Review of Military  Compensation concluded that reservists would likely return to their civilian  homes and employers at the conclusion of their assignments. In addition,  officials stated that long-duration training is becoming more common. For  example, Army language or medical training can routinely last longer than  140 days and require a Permanent Change of Station.

The treatment of long-duration training and other assignments as a  Permanent Change of Station and not as Temporary Duty affects the type  of expenses that will be reimbursed and the Basic Allowance for Housing  rate received by reservists. A Permanent Change of Station is triggered  when Active Duty for Training assignments last 140 days or longer and  Active Duty for Other than Training assignments last 181 days or longer.  The changes in eligibility for reimbursement discussed below can affect  the amounts of reservists’ unreimbursed expenses:

Per diem: Reservists on training or other assignments that are  treated as a Permanent Change of Station are not eligible for  reimbursement of per diem expenses, including for temporary lodging  and meals. Reservists are unlikely to relocate their civilian homes for  such long-duration, though interim, training and assignments. They  may therefore incur expenses typically associated with a Temporary  Duty assignment, such as temporary lodging expenses, but for which  they cannot be reimbursed.

Basic Allowance for Housing: Reservists on training or other  assignments that are treated as a Permanent Change of Station  receive an adjusted Basic Allowance for Housing based on the  location of their new duty station. This adjusted housing allowance  applies regardless of whether a reservist actually moves his or her  civilian home and family to the new duty station. Depending on the  new duty location, a reservist may receive Basic Allowance for  Housing at a higher or lower rate than the allowance amount based  on the location of their civilian home. If a reservist were in Temporary  Duty status—training for 139 days or fewer, or an assignment for 180  days or fewer—he or she would continue to receive Basic Allowance  for Housing based on the cost of maintaining his or her civilian home.

If reservists decide not to relocate themselves and their families to the  location of the long-duration training or assignment, reservists may face  unreimbursed costs for maintaining two homes. Once a Permanent  Change of Station has been triggered, a reservist is no longer in a  Temporary Duty status and may no longer receive per diem for temporary  lodging. Reservists must either (1) move to government lodging and  forego any Basic Allowance for Housing, or (2) receive Basic Allowance  for Housing based on the location of the assignment, which may be  higher or lower than the allowance based on the location of their home of  record, generally their civilian home. In the first situation, reservists must  maintain their civilian home without payment of a Basic Allowance for  Housing, and thus may face unreimbursed costs associated with the  home’s maintenance. In the second situation, reservists must maintain  both their civilian home and a new home with a Basic Allowance for  Housing adjusted for the location of the home at the new duty station.  Unreimbursed costs may result if the Basic Allowance for Housing  adjusted for the location of the new duty station is significantly lower than  the housing costs in the area of the reservist’s civilian home. As shown in  the 2017 illustrative example in figure 1, reservists receive different levels  of payment for the temporary lodging allowance and the Basic Allowance  for Housing based on the duration of their Active Duty for Training  assignments.

A service Secretary may grant a waiver for individuals attending a training  course to maintain Temporary Duty status beyond the 140-day time limit,  which normally would require a Permanent Change of Station. However,  such waivers apply to all course attendees, whether they are members of  the active or reserve components. DOD maintains data on the number of  these waivers, but not for the discrete number of waivers for reserve  component training. Individual reservists can also apply for a waiver for  the rate of their Basic Allowance for Housing payment to be based on the  location of their dependents, effectively allowing payment at the  geographic rate of a reservist’s civilian home. However, this option is not  available to reservists without dependents.

DOD Has Not Fully Assessed the Potential Effect of Unreimbursed Out-of- Pocket Travel Expenses on the Retention of Reservists

Within the last decade, DOD and the services have conducted a few  limited assessments of the potential effect of unreimbursed out-of-pocket  travel expenses incurred by reservists to perform required training and  other reserve activities on retention of reservists. Although various entities  have raised concerns regarding reservists’ out-of-pocket travel expenses,  the available information is either anecdotal or applicable to only one  reserve component or one aspect of travel policy. DOD reports have  noted that such unreimbursed travel expenses, among other factors, may  be a challenge for reservists and may therefore affect retention. For  example, in 2008, the Commission on the National Guard and Reserves  reported that travel requirements and associated costs had a negative  effect on DOD’s ability to recruit and retain qualified personnel,  particularly for leadership positions. In addition, in minutes of its  meetings, the Air Reserve Forces Policy Committee has called for  changes to the Permanent Change of Station requirement for long- duration training, noting in 2015 that it, “frequently creates financial  hardship for RC  Airmen who typically maintain a  residence near their assigned unit or civilian employer.”

Three DOD studies have explored potential links between reservists’  unreimbursed travel expenses and retention:

A 2012 survey commissioned by the Army Reserve of a small sample  of reservist officers potentially eligible for battalion command positions  reported that unreimbursed travel costs were among several factors  that could influence their decision to apply for these positions.

A 2014 study commissioned by the Marine Corps found that, based  on a statistical model of a sample of Marines eligible to participate in  its Inactive Duty Training travel reimbursement program between May  2012 and September 2013, the program had increased the Marine  Corps’ ability to fill critical positions. The study also included an  assessment of the cost of increasing the level of reimbursement for  Inactive Duty Training and its possible effect on staffing.

A 2016 survey commissioned by the Army Reserve, drawn from a  non-generalizable sample of a few thousand reservists, reported that  a significant majority of respondents in 2015 viewed the Inactive Duty  Training travel reimbursement program as an incentive for soldier  retention.

In addition, during our review, officials from most of the reserve  components told us that despite the establishment of the reimbursement  program for travel costs associated with Inactive Duty Training, such  expenses continue to be a challenge for some reservists. In particular,  officials noted that this especially affects personnel who do not qualify for  reimbursement. One official noted that, in extreme cases, reservists may  find that the cost to attend Inactive Duty Training may exceed drill pay,  effectively requiring them to pay out-of-pocket to perform military service.

While these reports and studies have alerted DOD to a potential problem,  DOD has not yet assessed the effect of unreimbursed travel expenses on  retention of reservists in a comprehensive manner and the related overall  cost to the federal government. For example, DOD has not yet  systematically collected data and assessed the potential effect of current  travel reimbursement policy on retention across all services, as measured  by outcomes such as fill rates for critical positions and other metrics, or  collected more basic information such as the number of reservists who do  not move their home during long-duration training, the distances traveled  for Inactive Duty Training, and the amount of unreimbursed expenses  incurred by reservists. The 2014 Marine Corps’ study on Inactive Duty  Training reimbursement did explore fill rates for its potential effect on  critical positions. However, its findings are not necessarily applicable to  the other reserve components. In addition, DOD has not conducted an assessment on the issue of Permanent Change of Station rules for long- duration training or other assignments. While travel policy officials noted  that there is no requirement for such an assessment, some agreed that  more robust information would allow for a better understanding of the  situation as well as any potential changes that are necessary in DOD’s  travel policy. One travel official stated that until a direct connection  between unreimbursed travel expenses and retention or related areas is  observed within their component, change is unnecessary.

As of July 2017, DOD and the reserve components were considering  changes to reserve travel policy to mitigate the effect of out-of-pocket  expenses on reservists. Specifically, these changes include (1)  requesting that Congress increase reimbursement for Inactive Duty  Training travel expenses from $300 to $500 and (2) increasing the length  of time of Temporary Duty travel for training courses or other assignments  before such travel is considered a Permanent Change of Station. The  Marine Corps Reserve has developed a draft proposal for congressional  consideration for an increase in Inactive Duty Training reimbursement,  which an official stated was necessary to address the challenge of filling  critical occupations. The Military Advisory Panel, which advises on  defense travel issues, has considered an increase in the length of time of  Temporary Duty travel for training courses or other assignments before a  Permanent Change of Station would be required, but no specific  proposals have been developed.

Federal internal control standards state that management requires quality  information to make informed decisions and evaluate an entity’s  performance in achieving key objectives and addressing risk. They  further require that management identify, analyze, and respond to risks  related to achieving the defined objectives. However, without collecting  more comprehensive information on the potential effect of the current  travel policy on the retention of reservists, DOD would be considering  alternative proposals with only the limited data and analysis available to  date. Further, the lack of comprehensive data and analysis on the  influence of current travel policies will limit DOD’s ability to reach an  analytically based decision which weighs the costs and benefits of any  potential changes. In deciding to continue or change current travel  policies relating to travel reimbursement without the benefit of quality  information, DOD risks not managing the potential influence of these  policies on reservists’ retention or agency expenditures.

Reservists often maintain civilian careers and homes that in some cases  can require them to travel long distances to perform their part-time  military service. In some instances, such as when performing Inactive  Duty Training and long-duration Active Duty for Training or other active  duty assignments, such service can result in expenses that cannot be  reimbursed to the reservist under DOD’s travel policy. Despite long- standing concerns that out-of-pocket travel expenses reservists incur to  perform their service may be increasing, DOD does not have sufficient  data and analysis on how reservists’ incurring these expenses could  negatively affect DOD’s ability to achieve its mission, the overall costs  and benefits of DOD’s travel policy, and how various proposed changes  to the travel policy could potentially mitigate any of its possible negative  effects. As a result, DOD is not well positioned to move forward with  possible changes to travel policy absent further analysis.

We recommend that the Under Secretary of Defense for Personnel and  Readiness collect quality information and conduct an analysis of the  potential effects of unreimbursed travel expenses incurred by reservists to  perform military service on DOD’s ability to retain reservists in the force,  and respond to these risks by considering the costs and benefits of any  possible actions to address the identified issues.

We provided a draft of this report to DOD for review and comment. In its  written comments, reproduced in appendix II, DOD concurred with our  recommendation.

We are sending copies of this report to appropriate congressional  committees, the Secretary of Defense, the Under Secretary of Defense  for Personnel and Readiness, the Secretaries of the military departments,  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (213) 830-1011 or vonaha@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.

Appendix I: Reserve Components’ Reported Travel Costs for Fiscal Year 2015

To determine DOD’s travel costs for the reserve components, we  obtained and reviewed fiscal year 2015 execution cost data, the most  recent complete data available, that each reserve component reported on  travel costs for training and other activities. These costs are reported in a  travel cost exhibit included in each component’s annual Reserve  Personnel budget justification document. We did not include any travel  costs not included in the reserve components’ Reserve Personnel  accounts, such as any travel costs in the components’ respective  Operations and Maintenance accounts. We did not include travel costs for  the Active Guard and Reserve because individuals serving these  components are responsible for the full-time administration of the reserve  components and differ significantly from part-time drilling reservists in  their responsibilities and associated travel. We also did not include costs  for the Individual Ready Reserve because these reservists have different  training patterns than other reservists. In table 2, we summarize the costs  reported by DOD’s six reserve components for Annual Training, Inactive  Duty Training, and all other travel costs for fiscal year 2015 by  component.

Appendix II: Comments from the Department of Defense

In addition to the contact named above, Margaret Best (Assistant  Director), Patricia Donahue, Mae Jones, Linda Keefer, Felicia Lopez,  Carol Petersen, and Adam Smith made major contributions to this report.