Analyzing Regulatory Burden: Policies and Analyses under the Regulatory Flexibility Act and Retrospective Reviews Could Be Improved

Federal financial regulators must comply with various rulemaking and review requirements, including those in RFA and EGRPRA. These statutes require analyses relating to regulatory burden, small entities, or both. RFA requires analyses of a rule's impact on small entities and alternatives that may minimize any significant economic impact. It also requires agencies to review rules (within 10 years) to determine if the rules should be amended or rescinded. EGRPRA directs specified regulators to review regulations at least every 10 years and identify areas that are outdated, unnecessary, or unduly burdensome on insured depository institutions.

This statement is based on findings from GAO's January 2018 report on RFA implementation ( GAO-18-256 ) and February 2018 report on regulatory burden on community banks and credit unions ( GAO-18-213 ). GAO discusses regulatory burdens and how financial regulators address regulatory burdens through the rulemaking process and retrospective reviews. For those reports, GAO's work included reviewing Federal Register notices; regulators' workpapers, policies and procedures; and reports to Congress on EGRPRA reviews. GAO also interviewed more than 60 community banks and credit unions.

More than 60 smaller depository institutions told GAO that regulations for reporting mortgage characteristics; reviewing transactions for potentially illicit activity; and disclosing fees, conditions, and mortgage terms to consumers were the most burdensome. Institution representatives said these regulations were time-consuming and costly because the requirements were complex and required reporting that had to be reviewed for accuracy. Financial regulators and others noted these regulations provide various benefits as well, such as preventing lending discrimination or use of the banking system for illicit activity.

The Regulatory Flexibility Act (RFA) requires federal agencies to analyze the impact of their regulations on small entities. GAO found several weaknesses with the analyses of six financial regulators—Board of Governors of the Federal Reserve System (Federal Reserve), Office of the Comptroller of the Currency (OCC), Federal Deposit Insurance Corporation (FDIC), Securities and Exchange Commission, Commodity Futures Trading Commission, and Consumer Financial Protection Bureau (CFPB)—that could undermine the goal of RFA and limit transparency and public accountability. For example, some analyses lacked important information, such as data sources, methodologies, and consideration of broad economic impacts. Evaluations of potential economic effects and alternative regulatory approaches also were limited. Finally, regulators generally lacked comprehensive policies and procedures for RFA implementation. By not developing such policies and procedures, regulators' ability to consistently and effectively meet RFA objectives may be limited.

The Economic Growth and Regulatory Paperwork Reduction Act of 1996 (EGRPRA) and RFA require regulators to conduct retrospective reviews, and GAO found weaknesses.

EGRPRA. GAO found limitations in activities regulators undertook for retrospective reviews under EGRPRA. CFPB, which has regulatory authority for a number of consumer financial laws, was not included in the most recent review process. Moreover, as part of their EGRPRA reviews, the Federal Reserve, OCC, FDIC, and the National Credit Union Administration had not conducted and reported analyses of quantitative data nor had these regulators assessed the cumulative effect of regulations. Addressing these limitations in the EGRPRA processes likely would make the analyses they perform more transparent, and potentially result in additional burden reduction.

RFA. The issues GAO identified with RFA retrospective reviews (section 610 reviews) included some regulators using the EGRPRA process to fulfill RFA requirements and gaps or weaknesses in analysis and documentation. But EGRPRA requirements do not fully align with RFA's, and it is not clear if the EGRPRA process satisfies the requirements of section 610. Also, regulators generally have not developed policies and procedures for section 610 reviews. By meeting section 610 review requirements, regulators will be in a better position to minimize any significant economic impact of a rule on a substantial number of small entities, as the statute seeks to ensure.

GAO made a total of 20 recommendations to the financial regulators in the two reports to improve their policies and procedures and analysis under RFA and in retrospective reviews. The regulators generally agreed with the recommendations.

I am pleased to be here today to discuss our recent work on regulatory  burden for small entities (such as community banks and credit unions)  and efforts by financial regulators to reduce such burden when  developing and retrospectively assessing regulations. Federal financial  regulators normally must comply with various rulemaking and review  requirements, including those in the Regulatory Flexibility Act (RFA) and  the Economic Growth and Regulatory Paperwork Reduction Act of 1996  (EGRPRA). These statutes require analyses relating to regulatory burden,  small entities, or both:

Analyses during rulemaking. RFA requires federal agencies,  including financial regulators, to provide an assessmentknown as a  regulatory flexibility analysisof a rule’s potential impact on small  entities and consider alternatives that may minimize any significant  economic impact on small entities. Alternatively, agencies may certify  that a rule would not have a significant economic impact on a  substantial number of small entities instead of performing a regulatory  flexibility analysis.

Retrospective reviews. (1) EGRPRA directs three depository  institution regulators—the Board of Governors of the Federal Reserve  System (Federal Reserve), the Federal Deposit Insurance  Corporation (FDIC), and the Office of the Comptroller of the Currency  (OCC)—to review regulations at least every 10 years and identify  areas that are outdated, unnecessary, or unduly burdensome on  insured depository institutions. (2) RFA requires agencies to review  within 10 years of publication those rules assessed as having a  significant economic impact on a substantial number of small entities.  Agencies are to determine if rules should be continued without  change, amended, or rescinded to minimize such impacts.

My remarks today are based on our January 2018 report on RFA  implementation and our February 2018 report that includes an analysis of  the EGRPRA review process. My statement will focus on (1) regulations  community banks and credit unions regarded as most burdensome and  why; (2) the extent to which financial regulators performed analyses  required by RFA and established policies and procedures for complying  with RFA requirements; (3) efforts to reduce regulatory burden on  community banks and credit unions during EGRPRA reviews; and (4)  retrospective reviews required by RFA.

For the January 2018 report, our work included a review of the RFA  section of the Federal Register notices and financial regulators’ internal  workpapers for all RFA certifications made in the final rule (66) and all  rules for which agencies performed an initial regulatory flexibility analysis  in the proposed rule and a final regulatory flexibility analysis in the final  rule (39) in calendar years 2015 and 2016. We also reviewed internal  agency policies, procedures, and guidance for RFA analyses and  certifications and documentation of retrospective reviews required by RFA  (section 610 reviews) performed from calendar years 2006 through 2016.  For the February 2018 report, our work included reviewing the EGRPRA  report the Federal Reserve, FDIC, OCC, and the National Credit Union  Administration (NCUA) issued in 2017. To identify regulations that  community banks and credit unions viewed as most burdensome, we  obtained opinions from a non-probability selection of more than 60  community banks and credit unions. We also reviewed comment letters  received and transcripts of public meetings held as part of the review. We  compared the requirements of Executive Orders 12866, 13563, and  13610 with actions regulators took implementing reviews. Detailed  information on our scope and methodology can be found in our January  and February 2018 reports.

We conducted the work on which this statement is based in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.

RFA requires that federal agencies, including financial regulators,  engaged in substantive rulemaking analyze the impact of proposed and  final regulations on small entities. If a rule might have a significant  economic impact on a substantial number of small entities, regulators are  to consider any significant regulatory alternatives that will achieve  statutory objectives while minimizing any significant economic impact on  small entities. RFA defines “small entity” to include small businesses,  small governmental jurisdictions, and certain small not-for-profit  organizations. RFA does not seek preferential treatment for small entities.  Rather, it requires agencies to use an analytical process that includes  identifying barriers to small business competitiveness and seeks a level  playing field for small entities.

For each draft rule that requires a notice of proposed rulemaking, RFA  requires regulators to prepare an initial regulatory flexibility analysis that  contains an assessment of the rule’s potential impact on small entities and describes any significant alternatives to reduce the rule’s significant  economic impact on small entities while achieving statutory objectives.  Following a public comment period, RFA requires regulators to conduct a  similar analysis when they promulgate the final rule. If the head of the  agency certifies in the Federal Register that the rule would not have a  significant economic impact on a substantial number of small entities,  agencies do not have to conduct the initial or final analysis. Certifications  must include a statement providing a factual basis for the certification.

Section 610 of RFA requires agencies to review, within 10 years of a final  rule’s publication, those rules assessed as having a significant economic  impact on a substantial number of small entities to determine if they  should be continued without change, amended, or rescinded (consistent  with statutory objectives) to minimize any significant economic impact on  small entities.

RFA designates certain responsibilities to the Small Business  Administration’s Chief Counsel for Advocacy, including monitoring agency  compliance with RFA and reviewing federal rules for their impact on small  businesses. Executive Order 13272 requires the Small Business  Administration’s Office of Advocacy (Office of Advocacy) to provide  notifications and training about RFA requirements. The Office of  Advocacy published guidance on RFA compliance in 2003 (updated in  2012 and August 2017). For example, the guidance details components  regulators should include in their certifications to obtain meaningful public  comments, such as a description and estimate of the economic impact.

Economic Growth and Regulatory Paperwork Reduction Act of 1996

Under EGRPRA, the Federal Reserve, FDIC, and OCC are to categorize  their regulations by type and provide notice and solicit public comment on  all regulations for which they have regulatory authority to identify areas of  the regulations that are outdated, unnecessary, or unduly burdensome.  The act also includes requirements on how the regulators should conduct  the reviews, including reporting results to Congress.

The first EGRPRA review was completed in 2007. The second began in  2014, and the report summarizing its results was submitted to Congress  in March 2017. While NCUA is not required to participate in the  EGRPRA review, NCUA has been participating voluntarily. NCUA’s  assessment of its regulations appears in separate sections of the 2007  and 2017 reports to Congress.

Community Banks and Credit Unions Saw Regulations on Mortgage Reporting and Disclosures and Anti-Money Laundering as Most Burdensome

Community bank and credit union representatives we interviewed  identified three areas of regulations as most burdensome to their  institutions:  1.  Data reporting requirements related to loan applicants and loan terms  under the Home Mortgage Disclosure Act of 1975 (HMDA).  2.  Transaction reporting and customer due diligence requirements as  part of the Bank Secrecy Act and related anti-money laundering  regulations (collectively, BSA/AML).  3.  Disclosures of mortgage loan fees and terms to consumers under the  Truth in Lending Act and the Real Estate Settlement Procedures Act  of 1974 Integrated Disclosure (TRID) regulation.

Institution representatives told us they found these regulations were time- consuming and costly to comply with because the requirements were  complex, required individual reports that had to be reviewed for accuracy,  or mandated actions within specific timeframes. For example, among the  28 community banks and credit unions whose representatives  commented on HMDA-required reporting in our focus groups, 61 percent  noted having to conduct additional HMDA-related training.  Representatives in most of our focus groups said that they had to  purchase or upgrade software systems to comply with BSA/AML  requirements, which can be expensive, and some representatives said  they have to hire third parties to comply with BSA/AML regulations.  Representatives in all of our focus groups and many of our interviews  said that the TRID regulations have increased the time their staff spend  on compliance, increased the cost of providing mortgage lending  services, and delayed the completion of mortgages for customers.

However, federal regulators and consumer advocacy groups’  representatives said that benefits from these regulations were significant,  such as collecting HMDA data that has helped address discriminatory  practices. Staff from Financial Crimes Enforcement Network (FinCEN),  which has delegated authority from the Secretary of the Treasury to  implement anti-money laundering regulations, told us that the transaction  reporting required and due-diligence programs required in BSA/AML rules  are critical to safeguarding the U.S. financial sector from illicit activity,  including illegal narcotic trafficking proceeds and terrorist financing  activities.

The Consumer Financial Protection Bureau (CFPB) has taken steps to  reduce the burdens for community banks and credit unions associated  with the HMDA and TRID regulations. Also, FinCEN has developed  several efforts in reducing the reporting requirements from BSA/AML  regulations to reduce regulatory burden, such as a continuous evaluation  process to look for ways to reduce burden associated with BSA reporting  requirements, soliciting feedback through an interagency working group  about potential burden, and expanding the ability of institutions to seek a  Currency Transaction Report filing exemption when possible.

To reduce institutions’ misunderstanding of the TRID regulation, CFPB  has published a Small Entity Compliance Guide and a Guide to the Loan  Estimate and Closing Disclosure Forms. However, CFPB officials  acknowledged that some community banks and credit unions may be  misinterpreting the regulation’s requirements. We found that CFPB had  not directly assessed the effectiveness of the guidance it provided to  community banks and credit unions. Until the guidance is assessed for  effectiveness, CFPB may not be able to respond to the risk that small  institutions have implemented TRID incorrectly. We recommended that  CFPB should assess the effectiveness of TRID guidance to determine the  extent to which TRID’s requirements are accurately understood and take  steps to address any issues as necessary. CFPB agreed with the  recommendations and intends to solicit public input on how it can improve  its regulatory guidance and implementation support.

Financial Regulators Consider Burden When Developing Regulations, but Their Reviews under RFA Need to Be Enhanced

One of the ways that financial regulators attempt to address the burden of  regulations is during the rulemaking process. For example, staff from the  Federal Reserve, FDIC, and OCC all noted that when promulgating rules,  their staff seek input from institutions and others throughout the process  to design requirements that achieve the goals of the regulation at the  most reasonable cost and effort for regulated entities. Once a rule has  been drafted, the regulators publish it in the Federal Register for public  comment. The staff noted that regulators often make revisions in  response to the comments received to try to reduce compliance burdens  in the final regulation. Under RFA, financial regulators conduct analyses  during the rulemaking process that are intended to minimize economic  impact on small entities. However, we found several weaknesses with the  RFA analyses, policies, and procedures of six financial regulators— Federal Reserve, OCC, FDIC, Securities and Exchange Commission  (SEC), Commodity Futures Trading Commission (CFTC), and CFPB— that could undermine the goal of RFA and limit transparency and public  accountability.

Certifications Were Not Always Consistent with Office of Advocacy Guidance and Other Best Practices

In reviewing 66 certifications by the six regulators, we found that in most  (43 of 66) the regulators provided a factual basis and concluded the rule  would not apply to small entities or have any economic impact.  According to the regulators, these rules included activities in which small  entities do not engage, pertained to the regulator’s internal processes, did  not create new regulatory requirements, or eliminated duplicative rules.  Additionally, regulators concluded in 5 of 66 certifications that the rule  would have a beneficial impact on small entities.

Other certifications lacked information that would help explain the  determination. Specifically, in 18 of 66 certifications, the regulators found  the rule would have some economic impact on small entities, but  concluded the impact would not be significant for a substantial number of  small entities. But the factual basis provided for most of the 18  certifications (across all six regulators) lacked key components the Office  of Advocacy and the Office of Management and Budget (OMB)  recommended for understanding the analyses regulators used to support  their conclusion. Examples include the following:

Data sources or methodologies. In 15 of 18 certifications regulators  did not describe or did not fully describe their methodology or data  sources for their conclusions.

Broader economic impacts. The certifications generally did not  address broader economic impacts such as cumulative effects,  competitive disadvantage, or disproportionality of effects and focused  most of the analysis on specific compliance costs.

Defining key criteria. Regulators generally did not define the criteria  they used for “substantial number” and “significant economic impact”  in their certifications.

Limited information. Three certifications included none of the Office  of Advocacy’s suggested components, such as the number of affected  entities, the size of the economic impacts, or the justification for the  certification.

While many of the regulators’ certification determinations incorporated  key components, the weaknesses and inconsistencies we found could undermine the act’s goal. For example, incomplete disclosure of  methodology and data sources could limit the public and affected entities’  ability to offer informed comments in response to regulators’ certification  assessments in proposed rules.

Many RFA-Required Analyses Had Weaknesses

Our review of recent rules in which the agency performed an initial and  final regulatory flexibility analysis found that the evaluation of key  components—potential economic effects and alternative regulatory  approaches—was limited in many cases, although the extent varied by  regulator. RFA requires initial and final analyses to include information to  assist the regulator, regulated entities, and the public in evaluating the  potential impact of rules on small entities. The most important  components include the assessment of a rule’s potential economic effects  on small entities—such as compliance costs—and the identification and  evaluation of alternative regulatory approaches that may minimize  significant economic effects while achieving statutory objectives.

The evaluations for some rules of economic impact on small entities  did not describe or estimate compliance costs. Analyses we  reviewed also generally did not evaluate differences in estimated  compliance costs for identified alternatives.

Five of six regulators did not consistently disclose the data sources or  methodologies used for estimating the number of subject small  entities or compliance costs.

By not fully assessing potential economic effects or alternatives,  regulators may not be fully realizing the opportunity to minimize  unnecessary burdens on small entities, which is the primary goal of RFA.

Regulators Lacked Comprehensive Policies and Procedures for RFA Analyses

Five of six regulators have written guidelines that restate statutory  requirements for certifications and preparing regulatory flexibility analyses  and provide some additional guidance for staff. However, the regulators  generally have not developed comprehensive policies and procedures to  assist staff in complying with RFA, which may contribute to the  weaknesses we identified in some certifications and regulatory flexibility  analyses. Federal internal control standards state the importance for  agency management to establish through policies and procedures the  actions needed to achieve objectives.

The extent to which regulators’ guidance included policies and  procedures varied. But the guidance generally did not include procedures  for evaluating a rule’s potential economic impact on small entities;  identifying and assessing regulatory alternatives that could minimize  economic impact on small entities; disclosing methodology and data  sources; and creating and maintaining documentation that supports  findings.

By developing policies and procedures that provide specific direction to  rulemaking staff, the regulators could better ensure consistent and  complete implementation of RFA requirements and more fully realize the  RFA goal of appropriately considering and minimizing impacts on small  entities during and after agency rulemakings.

In our January 2018 report, we recommended that each of the regulators  develop and implement specific policies and procedures for consistently  complying with RFA requirements and related guidance for conducting  RFA analyses. Five agencies generally agreed with this recommendation  and one did not provide written comments.

EGRPRA Reviews Resulted in Some Reduction in Burden, but the Reviews Have Limitations

Regulators took some actions to reduce burden as part of EGRPRA  reviews, but we also identified opportunities to improve analyses and  reporting.

Results of 2017 EGRPRA Review Included Some Actions to Reduce Regulatory Burden

To conduct the most recent EGRPRA review, the Federal Reserve, FDIC,  and OCC sought comments from banks and others and held public  meetings to obtain views on the regulations they administer. In the report  they issued in March 2017, the regulators identified six significant areas in  which commenters raised concerns: (1) capital rules, (2) Call Reports, (3)  appraisal requirements, (4) examination frequency, (5) Community  Reinvestment Act, and (6) BSA/AML regulations. In the report, these  regulators described various actions that could address some of the  concerns that commenters raised including:

On September 27, 2017, the regulators proposed several revisions to  capital requirements that would apply to banks with less than $250  billion in assets and less than $10 billion in total foreign exposure. For  example, the revisions simplify capital treatment for certain  commercial real estate loans and would change the treatment of  mortgage servicing assets.

The regulators developed a new Call Report form for banks with  assets of less than $1 billion and domestic offices only. In June 2017  and November 2017, the regulators issued additional proposed  revisions, effective June 2018, to the three Call Report forms that  banks are required to complete. For example, community banks  would report certain assets (nonperforming loans not generating their  stated interest rate) less frequently—semi-annually instead of  quarterly.

The regulators proposed raising the threshold for commercial real  estate loans requiring an appraisal from $250,000 to $400,000. They  also recently issued guidance on how institutions could obtain waivers  or otherwise expand the pool of persons eligible to prepare appraisals  if suitable appraisers are unavailable.

The three regulators also issued a final rule in 2016 making qualifying  depository institutions with less than $1 billion in total assets eligible  for an 18-month examination cycle rather than a 12-month cycle.

Although NCUA is not required to participate in the EGRPRA process, the  2017 EGRPRA report also includes a section in which NCUA describes  actions it has taken to address regulatory burdens on credit unions. In the  report, NCUA identified five significant areas raised by commenters  relating to credit union regulation, including: (1) field of membership and  chartering; (2) member business lending; (3) federal credit union  ownership of fixed assets; (4) expansion of national credit union share  insurance coverage; and (5) expanded powers for credit unions.

In response, NCUA took various actions. For example, NCUA modified  and updated its field of credit union membership by revising the definition  of a local community, rural district, and underserved area, which provided  greater flexibility to federal credit unions seeking to add a rural district to  their field of membership. NCUA also lessened some restrictions on  member lending to small business and raised some asset thresholds for  what would be defined as a small credit union so that fewer requirements  would apply to these credit unions.

CFPB Was Not Included in 2017 Review and Significant Mortgage Regulations Were Not Assessed

One of the limitations in the EGRPRA process is that the statute  mandating the process does not include CFPB and thus the significant  mortgage-related regulations and other regulations that it administers— regulations that banks and credit unions generally must follow—were not  included in the most recent EGRPRA review. The depository institution  regulators cannot address these mortgage regulation-related burdens  because they no longer have rulemaking authority for certain consumer  financial statutes.

However, CFPB does have its own processes to assess the burden of  regulations it has implemented. For example, section 1022(d) of the  Dodd-Frank Wall Street Reform and Consumer Protection Act requires  CFPB to conduct a one-time assessment of each significant rule it adopts  under federal consumer financial law within 5 years of the rule’s effective  date. But CFPB staff told us that they have not yet determined whether  certain other regulations that apply to banks and credit unions, such as  the revisions to requirements, will be designated as significant and thus  subjected to the one-time assessments.

During 2017, CFPB launched an internal task force to coordinate and  bolster its continuing efforts to identify and relieve regulatory burdens for  small businesses, such as community banks, that potentially will address  any regulation the agency has under its jurisdiction. However, CFPB has  not provided public information on the extent to which it intends to review  regulations applicable to community banks and credit unions or provided  information on the timing and frequency of the reviews. In addition, it has  not indicated the extent to which it will coordinate the reviews with  depository institution regulators as part of EGRPRA reviews.

Until CFPB publicly provides additional information indicating its  commitment to periodically review the burden of all its regulations,  community banks, credit unions, and other depository institutions may  face diminished opportunities for regulatory relief.

In our February 2018 report, we recommended that CFPB issue public  information on its plans for reviewing regulations, including information on  the scope of regulations, timing and frequency of reviews, and the extent  to which the reviews will be coordinated with the other regulators as part  of the EGRPRA reviews. CFPB agreed with the recommendation and  committed to developing additional plans for reviews of key regulations  and publicly releasing such information. In the interim, CFPB stated it  intends to solicit public input on how it should approach reviewing  regulations.

Regulators Have Not Conducted or Reported Quantitative Analyses

Another limitation in the EGRPRA process conducted by the Federal  Reserve, FDIC, OCC, and NCUA was that these regulators did not  conduct or report on quantitative analyses during the EGRPRA process to  help them determine if changes to regulations would be warranted. Our  analysis of the 2017 EGRPRA report indicated that in responses to  comments in which the regulators did not take any action, the regulators  generally provided only their arguments against taking actions and did not  cite analysis or data to support their narrative.

EGRPRA does not require the regulators to collect and report on any  quantitative data they collected or analyzed as part of assessing the  potential burden of regulations. In contrast, executive branch agencies  tasked under executive orders to conduct retrospective reviews of  regulations generally must collect and analyze quantitative data as part of  assessing the costs and benefits of changing existing regulations.  Conducting quantitative analysis for retrospective reviews could serve as  a best practice for the depository institution regulators.

By not performing and reporting quantitative analyses where appropriate  in the EGRPRA review, the regulators may be missing opportunities to  better assess regulatory impacts, (including identifying the need for any  changes or identifying benefits) and making their analyses more  transparent to stakeholders.

In our February 2018 report, we recommended that the four depository  institution regulators develop plans for their regulatory analyses  describing how they will conduct and report on quantitative analysis  whenever feasible to strengthen the rigor and transparency of the  EGRPRA process. The regulators agreed with the recommendation. For  example, the Federal Reserve plans to coordinate with FDIC and OCC to  identify opportunities to conduct quantitative analyses where feasible  during future EGRPRA reviews. NCUA also said it should improve its  quantitative analysis.

Regulators Have Not Considered the Cumulative Effects of Regulations

An additional limitation in the EGRPRA process we identified was that the  depository institution regulators had not assessed the ways in which the  cumulative burden of the regulations they administer may have created  overlapping or duplicative requirements. Under the current process, the  regulators have responded to issues raised about individual regulations  based on comments they have received, not on bodies of regulations.

However, congressional intent in tasking regulators with EGRPRA  reviews was to ensure they considered the cumulative effect of financial  regulations. A 1995 Senate Committee on Banking, Housing, and Urban  Affairs report stated while no one regulation can be singled out as being  the most burdensome, and most have meritorious goals, the aggregate  burden of banking regulations ultimately affects a bank’s operations, its  profitability, and the cost of credit to customers.

In our February 2018 report, we recommended to the Federal Reserve,  FDIC, NCUA, and OCC that as part of their EGRPRA review they develop  plans for conducting evaluations that would identify opportunities to  streamline bodies of regulation. The regulators generally agreed with the  recommendation and said they would work together to identify ways and  opportunities to decrease the regulatory burden created by bodies of  regulation. In addition, FDIC stated it would continue to monitor the  cumulative effects of regulation; for example, through a review of  community and quarterly banking studies and community bank Call  Report data.

Regulators’ Approach to RFA-Required Retrospective Reviews Varied, Including the Extent to Which They Developed Policies

Financial regulators took varying approaches to performing retrospective  reviews for RFA; additionally, some regulators had not yet developed  policies and procedures for conducting and reporting reviews.

Federal Banking Regulators Relied on Other Retrospective Reviews to Meet RFA Section 610 Requirements

We assessed section 610 reviews and found that the Federal Reserve,  FDIC, and OCC conducted retrospective reviews that did not fully align  with RFA’s requirements. Officials at each of the agencies stated that  they satisfy the requirements to perform section 610 reviews through the  EGRPRA review process.

But the requirements of the EGRPRA reviews differ from those of the  RFA-required section 610 reviews. For example, the EGRPRA review  process relies on public comments to identify rules that may be outdated,  unnecessary, or unduly burdensome, while public comments are only one  component of section 610 reviews. The Office of Advocacy stated that  agencies may satisfy section 610 requirements through other  retrospective reviews if these other reviews meet the criteria of section  610. According to an official from the Office of Advocacy, the office has  not yet made a determination on whether the EGRPRA review process  satisfies those requirements.

Although the agencies stated that they fulfill RFA requirements through  EGRPRA, without confirming this with the Office of Advocacy, it is  possible that they are not meeting RFA section 610 requirements and  therefore may not be achieving the small-entity burden reduction that the  statute seeks to ensure.

In our January 2018 report, we recommended that the Federal Reserve,  FDIC, and OCC coordinate with the Office of Advocacy to determine  whether the EGRPRA review process satisfies the requirements of  section 610 and, if not, what steps should be taken to align the process  with section 610 requirements. The Federal Reserve and FDIC generally  agreed with this recommendation, and OCC did not provide written  comments.

SEC Reviews Were Late and Not Fully Consistent with RFA Requirements or Office of Advocacy Guidance

Our review of 46 SEC section 610 reviews found that they were  conducted late and were not fully consistent with RFA requirements or the  Office of Advocacy’s guidance for such reviews. RFA requires rules to be  reviewed within 10 years of their publication as final rules, but SEC  conducted all but one of its reviews 12 years after the rules were  published. The reviews generally lacked substantive analysis, and no  rules were amended as a direct result of their section 610 review. The  reviews generally provided no evidence of empirical analysis and no data  to support the conclusions of the reviews, as recommended by the Office  of Advocacy and OMB. In most cases, the reviews lacked a description of  whether, or to what extent, the rule was affecting small entities.

SEC does not have written policies or procedures for completing rule  reviews pursuant to RFA section 610, potentially contributing to the  weaknesses we identified (timing and lack of data and analysis to support  findings). Therefore, in our January 2018 report, we recommended that  SEC develop and implement specific policies and procedures for  performing section 610 reviews. SEC generally agreed with the  recommendation.

SEC also does not publicly disclose the findings or conclusions of its  section 610 reviews. Although RFA does not require that agencies  publish the results of 610 reviews, the Office of Advocacy recommends  that to enhance transparency, agencies should communicate with  interested entities about the reviews. Executive orders also highlight  public disclosure of retrospective reviews. Lack of public disclosure  limits the transparency of the reviews, hindering the public’s ability to hold  agencies accountable for the quality and conclusions of their reviews. In  our January 2018 report, we recommended that SEC publicly disclose its  section 610 reviews, or summaries, with the basis for any conclusions.  SEC generally agreed with the recommendation.

CFTC and CFPB Plan to Develop Policies and Procedures for Future Retrospective Reviews

CFTC and CFPB plan to put procedures in place for section 610 reviews.  According to CFTC officials, the agency has not conducted any section  610 reviews in at least the last 10 years. CFPB has not yet been required  to conduct any section 610 reviews. Section 610 reviews are required  within 10 years of a rule’s publication as a final rule; to date, none of the  rules issued by CFPB, which was created in 2010, have met this  deadline.

In our January 2018 report, we recommended that CFTC and CFPB  develop policies and procedures for section 610 reviews that would  include documenting analyses and public reporting of results. CFTC and  CFPB generally agreed with the recommendation.

Chairman Chabot, Ranking Member Velázquez, and members of the  Committee, this concludes my statement. I would be pleased to respond  to any questions you may have.

If you or your staff have any questions about this testimony, please  contact Michael E. Clements, Director, Financial Markets and Community  Investment, at (202) 512-8678 or clementsm@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this statement. GAO staff who made key contributions  to this testimony are Cody Goebel (Assistant Director), Stefanie Jonkman  (Assistant Director), Katherine Carter (Analyst in Charge), Kevin Averyt,  Bethany Benitez, Jeremy A. Conley, Pamela R. Davidson, Nancy Eibeck,  Andrew Emmons, Courtney L. LaFountain, William V. Lamping, Marc  Molino, Lauren Mosteller, Barbara Roesmann, and Jena Y. Sinkfield.  Other assistance was provided by Farrah Graham and Tim Bober.

This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.


Warfighter Support: An Assessment of DOD Documents Used in Previous Efforts to Rebalance to the Pacific

In 2011, President Obama announced that the United States would turn its attention to the Asia-Pacific region and make the U.S. presence there a top priority. Rebalancing to the Pacific became strategic guidance that informed military planning. By the end of 2015, DOD published strategy documents that included references to the rebalance to the Pacific or related concepts. In February 2018, the Assistant Secretary of Defense for Asian and Pacific Security Affairs stated that while DOD continues to prioritize the Asia-Pacific region, the rebalance to the Pacific is no longer U.S. policy. DOD has published the 2018 National Defense Strategy, which establishes an objective of maintaining a favorable regional balance in the Pacific region, among other regions.

Prior to the change in policy, House Report 114-102 included a provision for GAO to review matters related to the U.S. rebalance to the Asia-Pacific region. GAO evaluated the extent to which DOD developed strategy documents to guide the rebalance to the Pacific that included desired elements of an effective national strategy.

GAO analyzed six DOD strategy documents that officials identified as providing guidance for the rebalance to the Pacific to determine whether, as a set, they included desired elements associated with an effective national strategy. DOD had no comments on this report.

Department of Defense (DOD) strategy documents that collectively guided the rebalance to the Pacific included most of the desired elements of an effective national strategy. The U.S. Pacific Command (PACOM), which is responsible for the Asia-Pacific region, used DOD strategy documents to implement the President's direction to rebalance to the Pacific, which generally refocused U.S. efforts to that region. PACOM officials told GAO that there was no single rebalance-specific strategy document. Instead, officials identified a number of strategy documents published since 2012 that guided activities associated with the rebalance to the Pacific, including: Sustaining U.S. Global Leadership: Priorities for 21st Century Defense ; Quadrennial Defense Review ; National Military Strategy ; Guidance for the Employment of the Force ; Joint Strategic Capabilities Plan ; and the PACOM 2015 Theater Campaign Plan (DRAFT) .

Based on GAO's analysis, DOD's six strategy documents that guided the rebalance to the Pacific included 24 of the 31 desired elements of an effective national strategy. However, two key elements were missing from the group of strategy documents: (1) a definition of the rebalance to the Pacific, and (2) the identification of the overall results desired, or end state, for the rebalance. DOD officials also could not identify a definition for the rebalance to the Pacific in the strategy documents or provide a definition that was used consistently across the department.

According to a DOD official with performance management responsibilities, defining the rebalance to the Pacific and identifying the initiative's strategic objectives, or end state, were important for establishing accountability and measuring progress. For instance, a clear definition of rebalance could have helped those charged with implementation to distinguish activities essential to operationalizing the strategic guidance from activities that were peripheral to that effort. Similarly, knowing the end state could have helped management make the best use of resources, enable the assessment of progress, and facilitate the development of strategic and military objectives. In moving forward in the Asia-Pacific region, considering the identification of strategic end states as well as other missing elements could help position DOD to achieve its objectives in the region.

In the fall of 2011, President Obama announced that the United States  would turn its attention to the Asia-Pacific region and make the U.S.  presence there a top priority. According to the President, the United  States was going to play a larger and long-term role in shaping the Asia- Pacific region after spending a decade at war in Iraq and Afghanistan.  The President then directed his national security team to conduct a  global, strategic assessment of U.S. priorities. The team made key  determinations including that a pivot toward the Asia-Pacific region would  help to rebalance the projection and focus of U.S. power. Rebalancing  efforts to the Pacific subsequently became defense strategic guidance  that began to inform military planning. Consequently, in 2012 the  commander of U.S. Pacific Command (PACOM) described the military  focus of the rebalance as strengthening relationships with allies and  partners; adjusting military posture and presence; and employing new  concepts, capabilities and capacities. In a 2015 address, the Secretary of  Defense outlined other aspects of the rebalance such as reinforcing  partnerships and enhancing posture in the Asia-Pacific region. By the end  of 2015, the Department of Defense (DOD) had published strategy  documents that included references to the rebalance to the Pacific or  related concepts. PACOM used these documents as guidance for  developing the command’s plans to implement the rebalance to the  Pacific.

In January 2018, DOD published the 2018 National Defense Strategy  which establishes an objective of maintaining a favorable regional  balance in the Pacific region. In February 2018, the Assistant Secretary  of Defense for Asian and Pacific Security Affairs stated that while DOD  continues to prioritize the Asia-Pacific region, the rebalance to the Pacific  is no longer U.S. policy.

Prior to the change in policy, House Report 114-102 included a provision  for GAO to review matters related to the U.S. rebalance to the Asia- Pacific region. We evaluated the extent to which six DOD-developed  strategy documents used to guide the previous rebalance to the Pacific  effort included the desired elements of an effective national strategy.  Based on interviews with DOD officials from multiple offices, we identified  six key DOD strategy documents that were used for the rebalance and  contained information about the rebalance, such as purpose and scope.  We systematically reviewed these strategy documents to determine  whether this set of strategy documents included desired elements  associated with an effective national strategy. Additionally, we  interviewed DOD officials from numerous organizations responsible for  planning, providing guidance or implementing the rebalance to the  Pacific. See appendix I for more details on our scope and methodology.

We conducted this performance audit from July 2015 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

DOD Strategies Inform Combatant Command Plans

DOD, through the Secretary of Defense and the Chairman of the Joint  Chiefs of Staff, develops department-wide strategic guidance based on  direction from the President and issues this guidance through strategy  documents. According to joint doctrine and Chairman of the Joint Chiefs  of Staff guidance, combatant commanders use strategy documents as  guidance for planning operations. Specifically, combatant commanders  translate this guidance into their commands’ campaign and contingency  plans. The military services organize, train, equip, and provide forces to  the combatant commanders to execute command plans. The combatant  commander must make certain the combatant command can execute  these plans.

PACOM is one of six geographic Unified Combatant Commands of the  U.S. Armed Forces. With an area of responsibility extending from the  waters off the west coast of the United States to the western border of  India, and from Antarctica to the North Pole, PACOM is the primary U.S.  military authority in the Pacific. In 2016, PACOM reported that  approximately 380,000 U.S. military and civilian personnel were assigned  to this area. PACOM describes the 36 nations that comprise the Asia- Pacific region as home to more than 50 percent of the world’s population  and 3,000 different languages, several of the world’s larger militaries, and  five nations allied with the United States through mutual defense treaties  or agreements. PACOM’s commander reports to the President and the  Secretary of Defense through the Chairman of the Joint Chiefs of Staff,  and is supported by four service component commands: U.S. Pacific  Fleet, U.S. Pacific Air Forces, U.S. Army Pacific, and U.S. Marine Forces,  Pacific.

DOD’s Previous Rebalance to the Pacific Strategy and Current Policy

In President Obama’s speech to the Australian Parliament in November  2011, he stated that after a decade of fighting two wars, the United States  was turning its attention to the vast potential of the Asia-Pacific region.  The President described the U.S. as a historic Pacific power whose  interests are inextricably linked with Asia’s economic, security, and  political order. According to a senior administration official, the United  States planned to implement a comprehensive, multidimensional strategy  in the Asia-Pacific region.

PACOM used military strategy documents to implement presidential  strategic direction to rebalance efforts to the Pacific. However, according  to officials from the Office of the Under Secretary of Defense for Policy,  the Joint Staff, and the U.S. Pacific Command there was no single  rebalance-specific strategy document. Instead, these officials identified a  number of strategy documents published since 2012 that guided activities  associated with the rebalance to the Pacific effort. Based on our  interviews with U.S. Pacific Command (PACOM) and DOD officials, we  focused our review on six strategy documents, issued between 2012 and  2015, that these officials considered relevant and representative of DOD’s  previous strategy to implement the rebalance to the Pacific through 2016.  The six documents that we reviewed are:

Sustaining U.S. Global Leadership: Priorities for 21st Century  Defense. DOD issued this document in January 2012. This  publication reflected presidential strategic direction to DOD and  described the key military missions for which the department would  prepare. In describing the security environment, this strategic  guidance stated that the United States would, of necessity, rebalance  toward the Asia-Pacific region.

Quadrennial Defense Review (QDR). According to DOD guidance,  the QDR articulates a national defense strategy consistent with the  broader government-wide National Security Strategy by defining force  structure, modernization plans, and a budget plan allowing the military  to successfully execute the full range of missions within that  strategy. The 2014 QDR referred to the rebalance to the Pacific as a  part of sustaining U.S. presence and posture abroad to better protect  U.S. national security interests.

National Military Strategy (NMS). The 2015 NMS described how  DOD would employ military forces to protect and advance U.S.  national interests. The NMS provided focus for military activities by  defining a set of military objectives and concepts used by the  combatant commanders and others. The 2015 NMS referenced the  rebalance to the Pacific as part of a national military objective. The  NMS was informed by the QDR.

Guidance for the Employment of the Force (GEF). According to  joint doctrine, the GEF provides direction to combatant commands for  operational planning, force management, security cooperation, and  posture planning. The GEF is the method through which the  Secretary of Defense translates strategic priorities in the QDR and  other strategy documents into direction for operational activities. The  GEF is described in joint doctrine as an essential document for  combatant command planners as it provides the strategic end states  for the deliberate planning of campaign and contingency plans.

Joint Strategic Capabilities Plan (JSCP). The JSCP is the primary  vehicle through which the Chairman of the Joint Chiefs of Staff directs the preparation of joint plans. The JSCP provides military strategic  and operational guidance to combatant commanders for the  preparation of plans based on current military capabilities. The JSCP  tasks combatant commanders to develop campaign, contingency, and  posture plans and translates requirements from the GEF and other  guidance into prioritized military missions, tasks, and plans. The JSCP  is informed by the GEF and the NMS.

PACOM 2015 Theater Campaign Plan (DRAFT) (TCP). Campaign  plans, such as PACOM’s TCP, focus on the combatant command’s  steady-state or daily activities and operationalize combatant  command theater strategies. According to joint doctrine, joint planning  draws from tasks identified in the GEF and JSCP and campaign plans  should focus on the combatant command’s steady-state activities.  These include ongoing operations, military engagement, security  cooperation, deterrence, and other shaping or preventive activities.  Campaign plans provide the vehicle for linking steady-state shaping  activities to the attainment of strategic and military end states.

In January 2018, DOD announced its new 2018 National Defense  Strategy that cited as the department’s principal priorities the long-term  strategic competition with China and Russia. The strategy also stated that  concurrently the department would sustain its efforts to deter and counter  rogue regimes such as North Korea and Iran, defeat terrorist threats to  the United States, and consolidate gains in Iraq and Afghanistan while  moving to a more resource-sustainable approach. In February 2018, the  Assistant Secretary of Defense for Asian and Pacific Security Affairs  notified GAO that although DOD continues to prioritize the Asia-Pacific  region, the rebalance to the Pacific is no longer U.S. policy.

DOD Strategy Documents Associated with Rebalancing to the Pacific Collectively Included Most of the Desired Elements of an Effective National Strategy

Six DOD strategy documents that helped guide the rebalance to the  Pacific collectively included most of the desired elements of an effective  national strategy.

We have previously reported that effective national strategies incorporate  six characteristics, and their associated desired elements. Table 1 lists  desired elements that we adapted from our prior work and tailored toward  our review of the six DOD strategy documents.

We found these six DOD strategy documents that collectively guided the  rebalance to the Pacific included, to varying degrees, 24 of the 31 desired  elements we determined as being the most relevant to an effective  strategy for the rebalance. For example, as a set, the six strategy  documents contained a detailed description of the operating environment  in which activities for the rebalance were to take place and included  references that described the relationship of the rebalance to the Pacific  to other strategies, goals, and objectives. The strategy documents  referenced their purposes and, in unclassified and general descriptions,  the threats that the strategies were to address including long-range  missile threats and weapons of mass destruction. Collectively, the  strategy documents referred to selected types of resources needed, such  as the deployment of ships and aviation assets, and who would be  implementing the strategies.

We were, however, unable to find any reference to 7 of the 31 elements  in any of the six strategy documents. For example, 2 of the 7 missing  elements were:  Lack of a documented, consistent definition of the rebalance to the  Pacific. Based on our systematic review, we found that none of DOD’s  six strategy documents issued from 2012 to 2015 included a definition of  the rebalance to the Pacific that described the rebalance’s key terms,  major functions, mission areas or activities. Further, DOD officials from  the Office of the Under Secretary of Defense for Policy, the Joint Staff,  and the U.S. Pacific Command involved in planning and implementing the  rebalance to the Pacific were unable to identify a definition for the  rebalance to the Pacific in the strategy documents, and consequently  could not provide a definition that was in use consistently across the  department. During discussions about the absence of a definition, these  PACOM officials told us that all PACOM activities were rebalance  activities, even activities that were underway before the President’s  announcement to rebalance. Senior DOD policy officials referred us to the  speeches of senior administration officials given since the President’s  2011 address to derive the definition of the rebalance. However, as noted  earlier, after the President’s speech in 2011, there were a number of  pronouncements from senior administration officials that varied over time.  The lack of consistent attributes to a strategy can make it difficult for  policy makers to assess its effectiveness and accountability.

Lack of a documented end state for the rebalance to the Pacific.  Based on our systematic review, we found that none of DOD’s six  strategy documents from 2012 to 2015 identified an end state for the  rebalance to the Pacific. Identifying the end state is a desired element  associated with establishing goals and objectives for effective strategies  and plans. Joint doctrine also states that military planners must know  where to look for the guidance to ensure that plans are consistent with  national priorities and are directed toward achieving national security  goals and objectives.

A national strategy that identified the end state of the rebalance could  distinguish new efforts from the longstanding U.S. military presence in the  region, and the associated increase in resources to support the post-2011  rebalancing. For example, we found a lack of clarity concerning the end  state for the rebalance. DOD officials from the Office of the Under  Secretary of Defense for Policy, Joint Staff, and PACOM—whom we  interviewed because they were involved in planning and implementing the  rebalance to the Pacific—said that they were unaware of an end state for  DOD’s efforts to rebalance. The same officials told us that there was no  foreseeable end state because, as long as the Asia-Pacific region was  important to the U.S., the focus would remain on the region. However,  officials from different military service components told us that their  individual services had an end state for their service-specific activities to  support the rebalance. For example, officials from U.S. Army Pacific told  us that they had completed their service’s rebalance. They stated that  they achieved the end state with the completion of force posture changes  and that some efforts supporting rebalancing had begun before  rebalancing was inaugurated. In contrast, a Marine Corps official in the  Pacific reported there was no end state for rebalancing. According to the  official, Marine Corps activities such as posture realignments supported  rebalancing, but these longstanding activities were ongoing prior to the  President’s announcement to rebalance.

Moreover, we found a lack of an awareness of a command-wide end  state for rebalancing and coordination among the various military service  activities in support of rebalancing. It was unclear how service-defined  end states could have been fully integrated or prioritized for funding  without a consistent overall end state for DOD’s overall effort. In such  instances, a department-wide defined end state could have helped with  the allocation of resources because the most important priorities would be  known.

A clear and consistent definition for rebalance and the identification of an  end state, as well as the inclusion of the other 5 missing elements, could  have better positioned decision makers to effectively plan, manage, and  assess DOD’s progress toward rebalancing efforts to the Pacific.  According to DOD officials from the Office of the Under Secretary of  Defense for Policy responsible for policy for the rebalance to the Pacific,  the speeches by senior administration officials between 2012 and 2015  supplanted the need to identify and document a definition of the  rebalance or an end state in a strategy document. However, as noted  earlier, these statements included varying descriptions of the strategy and  objectives over time. According to a DOD official from an office with  department-wide performance management responsibilities, defining the  rebalance to the Pacific and identifying the initiative’s strategic objectives,  or end state, were both important for establishing accountability and  measuring progress. For instance, a definition could have helped those  charged with implementation to distinguish activities essential to  operationalizing the strategic guidance to rebalance from those activities  that were routine or peripheral to that effort.

Further, knowing the end state could have helped management make the  best use of resources, enable the assessment of progress toward a  particular goal, and as described in joint doctrine, facilitate the  development of strategic and military objectives. In moving forward in the  Asia-Pacific region, considering the identification of strategic end states  (one of the desired elements of an effective national strategy that is also  discussed in joint doctrine) —as well as the other missing elements—  could help position DOD to achieve its objectives in the region.

We provided a draft of this report to DOD for review. DOD had no  comments.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; the Under Secretary of Defense  for Policy; the commander of the U.S. Pacific Command; the Chairman of  the Joint Chiefs of Staff; and the Secretaries of the military departments.  In addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you have any questions about this report or need additional information,  please contact me at (202) 512-5431 or RussellC@gao.gov. Contact  points for our Office of Congressional Relations and Public Affairs may be  found on the last page of this report. Key contributors to this report are  listed in appendix II.

To determine the extent to which the Department of Defense (DOD) has  developed strategy documents to guide the rebalance to the Pacific that  include desired elements of an effective national strategy, we conducted  a search of the literature, from January 2010 to July 2015, to identify  official statements on, guidance for, and studies of DOD’s implementation  of the rebalance to the Pacific. We reviewed department guidance, such  as Chairman of the Joint Chiefs of Staff instructions and joint publications,  to understand DOD’s processes and procedures for developing and  disseminating guidance and strategic plans. We also interviewed DOD  officials from numerous organizations listed below who were involved with  planning, providing guidance or implementing the rebalance to the Pacific  to identify DOD’s rebalance efforts and whether a strategy or strategies  existed that focused on or included the rebalance. The organizations  contacted included:

Under Secretary of Defense (Comptroller) and Chief Financial Officer

Office of the Deputy Chief Management Officer, Deputy’s

Assistant Secretary of Defense for Logistics and Materiel Readiness

Assistant Secretary of Defense for Strategy, Plans and Capabilities

Director of the Office of the Secretary of Defense Cost Assessment

U.S. Transportation Command  Based on these interviews and written responses to questions we  submitted to the officials associated with these organizations, officials  identified documentation and speeches that they indicated informed DOD  organizations about implementing the rebalance. Also, based on this  information, we found that there was not a single strategy or plan that  provided guidance for or outlined DOD’s implementation of the rebalance  to the Pacific. Instead, DOD officials from multiple offices identified a  number of strategy documents that guided activities associated with the  rebalance to the Pacific, including government-wide documents. Based  on our interviews with U.S. Pacific Command (PACOM) and DOD  officials, we focused our review on the six selected strategy documents,  issued between 2012 and 2015, that these officials considered relevant  and representative of DOD’s previous strategy to implement the  rebalance to the Pacific. Those six strategy documents are described  earlier in the main report.

We reviewed and analyzed these six strategy documents to determine  whether, as a set, they included the 31 desired elements of the  associated key characteristics of an effective national strategy. Our prior  work on effective national strategies included examples of desired  elements that we adapted and tailored toward our review of DOD strategy  documents. We selected 31 desired elements as most relevant to DOD’s  rebalance effort and for systematically reviewing DOD’s strategy  documents associated with the rebalance. These elements and  associated key characteristics are described in table 2 below.

To determine whether as a set these strategy documents included the  desired elements of an effective national strategy, we reviewed each  strategy document using a scorecard method, using the following steps:

First, we developed scorecards with a two-level scale of “address”  and “did not address.” We used a binary scale of “address” or “did not  address” and scored a passage as “address” if it included any part of  an element description in order to provide the widest latitude in  determining whether the selected passage included the specific  element. Also, we used 31 desired elements from the six  characteristics to make the comparison because these elements  provided more specificity than the broad six characteristics.

Second, analysts reviewed all of the selected passages from each  strategy document and determined whether they were relevant to  understanding the rebalance to the Pacific in order to reach  agreement on which passages they would consider in the comparison  to the desired elements. The readers agreed upon the inclusion and  exclusion of passages before assessing whether these passages  included the desired elements.

Third, two analysts reviewed the relevant passages in each strategy  document related to the rebalance and determined whether or not the  passages included the element. The analysts used the scorecards to  score each passage.

Fourth, upon completion of the independent scoring process for each  strategy document, the analysts compared their respective scores and  reconciled any differences, thereby reaching a consensus on the final  score. As needed, a third analyst facilitated reconciliations where  there was a difference in the assessment reached by the individual  analysts and documented the consensus results.

Lastly, upon completion of scoring, the team compiled and  summarized the results.

To further corroborate our systematic review of the six strategy  documents, we asked officials from DOD organizations responsible for  the Asia-Pacific region a standard set of related questions. We asked  officials these questions in order to obtain DOD’s perspective regarding  the applicability of using the selected desired elements and associated  key characteristics in reviewing these specific DOD strategy documents.

Appendix II: GAO Contact and Staff Acknowledgments

In addition to the contact named above, key contributors to this report  were Guy LoFaro, (Assistant Director), Pedro Almoguera, Patricia  Donahue, Richard Powelson, Paulina Reaves, Michael Shaughnessy,  and Stephen Woods.

Combating Terrorism: Strategy to Counter Iran in the Western  Hemisphere Has Gaps That State Department Should Address.  GAO-14-834. Washington, D.C.: September 29, 2014.

U.S. Public Diplomacy: Key Issues for Congressional Oversight.  GAO-09-679SP. Washington, D.C.: May 27, 2009.

Influenza Pandemic: Further Efforts Are Needed to Ensure Clearer  Federal Leadership Roles and an Effective National Strategy.  GAO-07-781. Washington, D.C.: August 14, 2007.

Financial Literacy and Education Commission: Further Progress Needed  to Ensure an Effective National Strategy. GAO-07-100. Washington, D.C.:  December 4, 2006.

Rebuilding Iraq: More Comprehensive National Strategy Needed to Help  Achieve U.S. Goals. GAO-06-788. Washington, D.C.: July 11, 2006.

Combating Terrorism: Evaluation of Selected Characteristics in National  Strategies Related to Terrorism. GAO-04-408T. Washington, D.C.:  February 3, 2004.

Combating Terrorism: Observations on National Strategies Related to  Terrorism. GAO-03-519T. Washington, D.C.: March 3, 2003.