Small Business Research Programs: Agencies Need to Take Steps to Assess Progress Toward Commercializing Technologies

Through the SBIR and STTR programs, federal agencies have awarded about 162,000 contracts and grants totaling $46 billion to small businesses to help them develop and commercialize new technologies. Eleven federal agencies participate in the SBIR program, and 5 agencies also participate in the STTR program. Each program has three phases, which take projects from initial feasibility studies through commercialization activities. SBA oversees both programs.

In response to the 2011 reauthorization of the programs, SBA and the participating agencies developed benchmarks to measure small businesses' progress in developing and commercializing technologies. GAO was asked to review SBA's and the agencies' efforts related to these benchmarks. This report examines the extent to which SBA and the participating agencies have implemented these benchmarks, including assessing businesses against them and establishing the consequence of not meeting them. GAO analyzed award data and interviewed officials from SBA and the 11 participating agencies.

Data challenges have limited the Small Business Administration's (SBA) and the 11 participating federal agencies' efforts to assess businesses against two benchmarks—the Transition Rate Benchmark and the Commercialization Benchmark—of the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs.

Transition Rate Benchmark. Small businesses that received more than 20 awards for the first phase of the programs in the past 5 fiscal years—excluding the most recent fiscal year—must have received an average of 1 award for the second phase of the programs for every 4 first phase awards. Since 2014, SBA and the agencies participating in the programs have regularly assessed small businesses against this benchmark. From 2014 through 2017, SBA determined that 4 to 7 businesses did not meet the benchmark each year. SBA officials provided GAO guidance on how to enter data into the programs' awards database they said is available to agencies, but GAO found evidence that suggests agencies are not fully utilizing it. For example, GAO found that the database used to perform the assessments contained inaccurate and incomplete data, such as about 2,700 businesses with multiple records with different spellings of their names and more than 1,400 instances in which a unique identification number had errors, such as an incorrect number of digits, all zeros, or hyphens. Thus, it could be difficult to determine which small businesses should be subject to the benchmark.

Commercialization Benchmark. Small businesses that received more than 15 awards for the second phase of the programs in the past 10 fiscal years—excluding the most recent 2 fiscal years—must have received a certain amount of sales, investments, or patents resulting from their efforts. SBA and participating agencies have assessed small businesses against this benchmark only once, in 2014, and identified 12 businesses that did not meet the benchmark. This is, in part, due to challenges in collecting and verifying the accuracy of the data that small businesses report and that are needed to implement the benchmark, according to officials from SBA and several agencies. For example, agency officials told GAO that some needed data, such as for reported sales, are not consistently applicable across agencies or projects. The Small Business Act and policy directives provide flexibility in how the agencies can implement the benchmark. Working together to implement it as designed or revise it so that it can be implemented could allow the agencies to fulfill statutory requirements.

SBA and the participating agencies have provided inconsistent information to small businesses about the consequence of not meeting the benchmarks. SBA and the agencies agreed to change how the consequence of not meeting the benchmarks was to be implemented, starting in 2017, from ineligibility to receive certain awards to ineligibility to submit certain proposals. However, as of November 2017, some agencies had not updated this information in their project solicitations. Furthermore, SBA has not updated this information in its policy directives. Without consistent information, businesses may be confused about their eligibility to submit proposals or receive awards and could invest time developing and submitting proposals when they are not eligible to do so.

GAO is making 11 recommendations to SBA and other agencies to take actions to improve implementation of the benchmarks, including improving the reliability of award data; implementing or revising the Commercialization Benchmark; and updating information about the consequence of not meeting the benchmarks. SBA and these agencies agreed with GAO's recommendations.

Federal agencies support research or research and development (R&D)  projects at small businesses through the Small Business Innovation  Research (SBIR) and Small Business Technology Transfer (STTR)  programs. Federal agencies have awarded about 162,000 contracts and  grants, totaling about $46 billion, since the programs’ inceptions in 1982  and 1992, respectively. Through the SBIR and STTR programs, federal  agencies have sought to help small businesses develop and  commercialize innovative technologies. For example, an SBIR award  from the Department of Health and Human Services helped a small  business develop glasses that allow people with color vision deficiency to  see the full color palette, and this business has made more than $20  million in annual sales, according to information on the SBIR website.

Federal agencies with obligations of $100 million or more for extramural  R&D are required to participate in the SBIR program, and federal  agencies with obligations of $1 billion or more for extramural R&D are  required to also participate in the STTR program. Currently, 11 agencies  participate in the SBIR program, and 5 of these agencies also participate  in the STTR program, as shown in table 1.

The Small Business Act authorizes the SBIR and STTR programs, and  the Small Business Administration’s (SBA) Office of Investment and  Innovation is responsible for overseeing and coordinating the participating  agencies’ efforts for the programs. As part of SBA’s oversight and  coordination role, the agency issued SBIR and STTR policy directives to  explain and outline requirements for agencies’ implementation of these  programs.

The SBIR and STTR programs each include three phases:  In phase I, agencies make awards to small businesses to determine  the scientific and technical merit and feasibility of ideas that appear to  have commercial potential. Phase I awards normally do not exceed  $150,000. For SBIR, phase I awards generally last up to 6 months.  For STTR, these awards generally last 1 year.

In phase II, small businesses with phase I projects that demonstrate  scientific and technical merit and feasibility, in addition to commercial  potential, may compete for awards of up to $1 million to continue the  R&D for an additional period, normally not to exceed 2 years.

In phase III, small businesses pursue commercialization of technology  developed in prior phases. Phase III work derives from, extends, or  completes an effort made under prior phases, but it is funded by  sources other than the SBIR and STTR programs. In this phase, small  businesses are expected to raise funds from private investors, the  capital markets, or funding sources within the agency that made the  initial award other than the agency’s SBIR or STTR program.  Agencies can also participate in phase III by, for example, purchasing  the technology developed in prior phases.

The 2011 reauthorization of the SBIR and STTR programs amended the  Small Business Act to require participating agencies to establish, and  SBA to approve, minimum standards for businesses’ progress in  developing and commercializing technologies. In response to this  requirement, SBA and the participating agencies worked together to  develop two “benchmarks” to serve as those minimum standards across  all the participating agencies:

Transition Rate Benchmark: Businesses that received more than 20  phase I awards over the prior 5 fiscal years—excluding the most  recent fiscal year—must have received an average of 1 phase II  award for every 4 phase I awards during this period. That is, the ratio  of phase II to phase I awards must be at least 0.25.

Commercialization Benchmark: Businesses that received more  than 15 phase II awards over the prior 10 fiscal years—excluding the  most recent 2 fiscal years—must have received an average of at least  $100,000 in sales, investments, or both per phase II award received,  or have received a number of patents resulting from the work equal to  or greater than 15 percent of the number of phase II awards received  during this period.

The reauthorization act states that, generally, small businesses that do  not meet the minimum standards may not participate in phase I of the  SBIR or STTR program. The policy directives state that small businesses  that do not meet the Transition Rate or Commercialization Benchmarks  are ineligible to receive phase I awards starting on June 1, when the  consequence period takes effect, through May 31 of the following year.

You asked us to review SBA’s and the participating agencies’ efforts  related to the benchmarks. This report examines the extent to which SBA  and the participating agencies have implemented the Transition Rate  Benchmark and the Commercialization Benchmark, including assessing  businesses against them and establishing the consequence of not  meeting them.

To accomplish this objective, we reviewed relevant legislation and the  SBIR and STTR policy directives to determine the requirements of SBA  and the participating agencies for implementing the benchmarks. We  reviewed project solicitations and agencies’ websites, and we interviewed  officials from SBA and each of the 11 participating agencies to obtain  information about the agencies’ efforts to implement the benchmarks. We  reviewed the applicable lists of small businesses that SBA assessed as  not having met a benchmark from 2014 through 2017 and analyzed SBIR  and STTR program award data. Further, we reviewed Standards for  Internal Control in the Federal Government and SBA’s data policy and  guidelines. We also performed electronic testing of the data to search for  missing or incorrect information, and interviewed officials from SBA about  the data. We found instances in which the award data were unreliable,  which we discuss later in this report.

We conducted this performance audit from April 2017 to January 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The SBIR program was initiated in 1982 and has four main purposes: (1)  use small businesses to meet federal R&D needs, (2) stimulate  technological innovation, (3) increase commercialization of innovations  derived from federal R&D efforts, and (4) encourage participation in  technological innovation by small businesses owned by women and  disadvantaged individuals. The STTR program was initiated a decade  later, in 1992, and has three main purposes: (1) stimulate technological  innovation, (2) foster technology transfer through cooperative R&D  between small businesses and research institutions, and (3) increase  private-sector commercialization of innovations derived from federal R&D.

The SBIR and STTR programs are similar in that participating agencies  identify topics for R&D projects and support small businesses, but the  STTR program requires the small business to partner with a nonprofit  research institution, such as a college or university or a federally funded  research and development center.

Each participating agency must manage its SBIR and STTR programs in  accordance with program laws and regulations and the policy directives  issued by SBA. In general, the programs are similar across participating  agencies. All of the participating agencies follow the same general  process to obtain proposals from and make awards to small businesses  for both the SBIR and STTR programs. However, each participating  agency has considerable flexibility in designing and managing specific  aspects of these programs, such as determining research topics,  selecting award recipients, and administering funding agreements. At  least once a year, each participating agency issues a solicitation  requesting proposals for projects in topic areas determined by the  agency. Each participating agency uses its own process to review  proposals and determine which proposals should receive awards. The  agencies that participate in both SBIR and STTR programs usually use  the same process for both programs. Also, each participating agency  determines whether to provide the funding for awards as grants or  contracts.

According to the policy directives, SBA maintains a system that records  SBIR and STTR award information—using data submitted by the  agencies—as well as commercialization information, such as information  about patents, sales, and investments reported by small businesses that  received these awards. SBA is to use these data to assess small  businesses that received awards against the benchmarks and identify any  small businesses that did not meet the benchmarks. SBA is to initially  assess the small businesses against the benchmarks and then in April of  each year notify those that do not meet the benchmarks so that the  businesses can review their award data and work with participating  agencies to correct the database if necessary. SBA then is to analyze the  award data again to identify, on June 1, those small businesses that still  do not meet the benchmarks. These small businesses are then ineligible  for certain awards from that date through May 31 of the following year.

Data Challenges Have Limited the Implementation of the Benchmarks, and SBA and Participating Agencies Have Provided Inconsistent Information about the Consequence SBA and Participating Agencies Assessed Small Businesses against the Transition Rate Benchmark, but the Assessments Have Been Based on Inaccurate or Incomplete Data

Data challenges have limited SBA’s and the 11 participating agencies’  efforts to fully implement the benchmarks. Since 2014, SBA and the  participating agencies have regularly assessed small businesses against  the Transition Rate Benchmark, but the assessments have been based  on inaccurate or incomplete data. SBA and the participating agencies  have assessed small businesses against the Commercialization  Benchmark only once, in 2014, because of challenges in collecting and  verifying the accuracy of data. In addition, SBA and the participating  agencies have provided inconsistent information to small businesses  about the consequence of not meeting the benchmarks.

Since 2014, SBA and the participating agencies have regularly assessed  small businesses against the Transition Rate Benchmark, which, in  general, measures the rate at which businesses move projects from  phase I to phase II. From 2014 through 2017, SBA determined that 4 to  7 small businesses did not meet the benchmark each year and placed  those businesses on a list of those ineligible to receive certain additional  awards.

However, we found instances in which the data used to generate the list  were inaccurate or incomplete. For example, we identified an instance in  which the data in the awards database changed considerably after SBA’s  initial assessment, indicating that the data used for that assessment were  inaccurate. SBA’s list of small businesses subject to the benchmark in  2015 showed that a small business received 297 phase I awards during  the assessment period. However, data received from SBA officials in  August 2017 showed that this small business received only 1 phase I  award. Agencies can update their data in the awards database at any  time to, for example, submit additional award data or correct previously  submitted award data, which is what an SBA official stated may have  caused this change. Because the small business received only 1 award, it  would not have been subject to the Transition Rate Benchmark. In this  case, the change meant that SBA did not miss identifying a small business that should have been ineligible for an award; however, in other  instances, changes to the data may lead SBA to miss identifying a small  business that should have been ineligible for awards.

In addition, we identified instances in which the publicly available data on  awards were incomplete, including data that were missing or otherwise  unusable. For example, based on our review of the award data from 2007  through 2016, we identified more than 2,700 small businesses that had  multiple records with different spellings of the same business’s name.  Furthermore, we identified more than 1,400 instances in which a unique  identification number had errors, such as having an incorrect number of  digits, all zeros, or hyphens. SBA officials told us that the quality of the  award information in the database has been an issue, and that accurate  information is important because small businesses may avoid being  identified as subject to the benchmark if their business names and  identification numbers are different across multiple records. For example,  if the database contains 18 phase I awards made within the assessment  period to a small business with a certain unique identification number but  also contains 3 other phase I awards within that period with a different or  missing unique identification number, the small business may avoid being  identified as subject to the benchmark because the data would suggest it  did not meet the threshold of receiving more than 20 phase I awards,  even if it did. As a result, it could be difficult to determine which small  businesses actually received more than 20 awards and should be subject  to the benchmark.

Standards for Internal Control in the Federal Government state that  management should use quality information to achieve the entity’s  objectives, and SBA’s Information Quality Guidelines state that SBA  seeks to ensure the quality, utility, and integrity of the information it  shares with the public, among other things. SBA’s policy directives for  the SBIR and STTR programs state that SBA maintains a system that  records SBIR and STTR award information, which is publicly available,  and uses this information to calculate small businesses’ performance  against the benchmark. SBA officials told us they depend on the accuracy  of the data received from the participating agencies to perform SBA’s  assessment. These officials also acknowledged that confirming the  accuracy of SBA’s annual assessments against the benchmarks has  been challenging because agencies can update their data over time. SBA  officials stated that they have sought to improve the quality of the data  after the data are entered into the database, such as fixing instances in  which small businesses’ names were spelled differently across multiple  records; however, the officials said that correcting the data already  entered in the awards database is an ongoing and time-consuming  process. SBA officials told us that there are errors in the database, in part  because SBA has not worked with participating agencies to ensure that  agencies enter high-quality, accurate data into the database. SBA officials  provided us guidance on how to enter data that they said is available to  agencies, but the errors we found suggest that agencies are not fully  utilizing this guidance. As a result, SBA cannot reasonably ensure the  quality and reliability of its award data and therefore cannot reasonably  ensure that it has correctly assessed small businesses against the  Transition Rate Benchmark.

SBA and the Participating Agencies Assessed Small Businesses against the Commercialization Benchmark Only in 2014

The Small Business Act requires agencies to evaluate whether small  businesses have met a minimum performance standard for  commercializing their technology. SBA and participating agencies do not  know the extent to which small businesses are meeting the  Commercialization Benchmark because SBA and the agencies have  assessed businesses against the benchmark only once, in 2014, when  SBA determined that 12 businesses did not meet the benchmark. This is  in part because, according to officials from SBA and several agencies,  they cannot collect and verify the accuracy of the data needed to  implement the benchmark as written.

For SBA and participating agencies to assess whether small businesses  meet the Commercialization Benchmark, these small businesses must  provide data on sales, investments, or patents resulting from the awards.  However, agency officials told us about challenges related to obtaining  the data they need to implement this benchmark. For example, agency  officials told us that the needed data are not consistently applicable  across agencies or projects. Specifically, these officials said that an  agency may purchase the technology developed as a result of the SBIR  or STTR award, while another agency may focus on funding technologies  that will be sold on the commercial market, leading to different kinds of  data on “sales.” Additionally, officials from SBA and several of the  participating agencies told us they have been unable to collect and verify  the accuracy of the information from small businesses to assess them  against the Commercialization Benchmark. In addition, officials from 2  agencies told us that small businesses can easily circumvent the  benchmark by submitting incorrect data.

The Small Business Act and the policy directives provide agencies  flexibility in how they can implement the Commercialization Benchmark.  Officials from participating agencies said that they thought the  Commercialization Benchmark should be revised, but they provided  differing views on how to do it. Officials from SBA and 2 agencies told us  that they would consider having individual agencies develop a benchmark  or metric tailored to their agency, in part because the definition of  successful commercialization could vary across the agencies. However,  officials acknowledged that collecting and verifying the accuracy of the  data would still be a concern with this approach. Officials from 2  participating agencies told us that collecting and verifying the accuracy of  the data is a significant amount of work, and officials from a third agency  added that implementing the benchmark independently is impractical  because they do not have the capability to track small businesses’  commercialization efforts. Officials from 1 agency said they preferred to  keep a uniform benchmark across the agencies, in part because having  varying benchmarks could lead to a small business being eligible to  participate in the programs with one agency but not with another.  Although views differed across agencies, working together to find a way  to implement the benchmark as designed or revising it so that it can be  implemented could allow the agencies to fulfill the requirement in the  Small Business Act.

Officials from 3 agencies told us they would prefer to consider  businesses’ prior commercialization experience as part of their overall  evaluation of businesses’ proposals, rather than implement the current  Commercialization Benchmark. The SBIR and STTR policy directives  currently allow agencies to define the benchmark in terms other than  revenue or investment, such as using a commercialization scoring system  that rates awardees on their past commercialization success. Defining the  benchmark in these terms could help agencies to implement the statutory  requirement. Officials from SBA said they see the value of allowing  reviewers to use professional judgment in determining the  commercialization success of applicants, rather than assessing small  businesses against standard criteria. Officials from 1 agency said that  such a change could help achieve the goal of the benchmark without the  challenges of collecting data from all small businesses participating in the  programs.

Nine of the 11 participating agencies currently consider prior  commercialization experience as part of their evaluation when making  award selections (see table 2), which shows that evaluating  commercialization experience at individual agencies can be feasible. For  example, project solicitations from the Department of Agriculture, the  Department of Defense, and the National Science Foundation state that  these agencies require applicants to provide sales or revenue information  for products resulting from SBIR or STTR awards, and the Department of  Homeland Security’s solicitation requires applicants to provide a history of  previous federal and nonfederal funding and subsequent  commercialization of their products. All agencies consider  commercialization potential when selecting these awards.

SBA and Participating Agencies Have Provided Inconsistent Information to Small Businesses on the Consequence of Not Meeting the Benchmarks

The consequence for small businesses not meeting the benchmarks is  ineligibility to participate in phase I of the SBIR or STTR program for a  year, according to the Small Business Act. SBA officials stated that they  and the agencies initially interpreted this to mean that small businesses  could not receive awards during the ineligibility period of June 1 through  May 31 of the following year, and this is how the consequence is  described in the SBIR and STTR policy directives. SBA officials told us  that they and the participating agencies sought to change how to  implement the consequence of businesses not meeting the benchmarks  because of SBA’s and agencies’ difficulties in implementing the  benchmarks. Officials from 4 agencies said that they generally evaluate  and select awards shortly before SBA releases the list of ineligible  companies, leading them to potentially select projects from small  businesses that will be on the ineligible list by the time the award period  begins. Based on our review of award data from October 2014 to May  2017, we identified 13 phase I awards across 5 small businesses with  award start dates during the period that the business was ineligible to  receive such awards. According to agency officials, each of these awards  was selected before the small business became ineligible to receive the  award. SBA and the participating agencies agreed to change how the  consequence would be implemented, starting in 2017, so that small  businesses that do not meet the benchmarks are ineligible to submit  proposals, according to SBA officials.

As of November 2017, however, the information available about this new  way to implement the consequence was inconsistent because some of  the agencies had not updated their project solicitations. Specifically,  information in the most recent project solicitations available at that time  for 2 agencies and one subunit of an agency stated that businesses that  do not meet the benchmarks are ineligible to submit certain proposals,  consistent with the revised approach for how to implement the  consequence. However, the most recent project solicitations available at  that time for 7 other agencies and the other subunit of the agency  mentioned above instead stated that those businesses that do not meet  the benchmarks are ineligible to receive certain awards, consistent with  the prior approach for how to implement the consequence. One other  agency directed users to SBA’s website in its solicitation. Table 3 shows  the information about the consequence of not meeting the benchmarks  that each agency included in its most recent project solicitations as of  November 2017.

As of November 2017, the SBIR and STTR policy directives stated that  the consequence for not meeting these benchmarks is ineligibility to  receive certain awards. SBA officials told us they are in the process of  updating the policy directives to reflect this change in how the  consequence is implemented, but these officials said that it is a long  process and they could not provide a timeframe for when the update  would be complete. As mentioned earlier in this report, SBA’s Information  Quality Guidelines state that SBA seeks to ensure the quality, utility, and  integrity of the information it shares with the public, among other things.  Until participating agencies update their project solicitations and SBA  updates its policy directives to accurately reflect agreed-upon practices  about the consequence for small businesses that do not meet the  benchmarks, small businesses may be confused about their eligibility to  submit proposals and could invest time developing and submitting  proposals when they are not eligible to do so.

Under the SBIR and STTR programs, federal agencies have awarded  billions of dollars to small businesses to help these businesses develop  and commercialize innovative technologies. SBA and the participating  agencies have assessed these small businesses against the Transition  Rate Benchmark, but those assessments have been based on inaccurate  or incomplete data. Without ensuring the reliability of its data, SBA cannot  reasonably ensure that it has correctly assessed small businesses  against the Transition Rate Benchmark.

SBA and the participating agencies developed a Commercialization  Benchmark across all the participating agencies but have not fully  implemented it, in part because they have been unable to collect  information from the small businesses and verify the accuracy of that  information. Working together to implement the benchmark as written or  revise it so that it can be implemented could allow the agencies to fulfill  the requirement in the Small Business Act to evaluate whether small  businesses have met a minimum performance standard for  commercializing their technology.

Lastly, SBA and the participating agencies have provided inconsistent  information to small businesses about the consequence of not meeting  the benchmarks. Officials from SBA and the participating agencies had  agreed to change how the consequence would be implemented, starting  in 2017, because of difficulties implementing the benchmarks. However,  as of November 2017, seven agencies, and a subunit of one agency, had  not updated their project solicitations and SBA had not updated its policy  directives. Without consistent information on the benchmarks, small  businesses may be confused about their eligibility to submit proposals  and could invest time developing proposals that they are not eligible to  submit.

We are making a total of 11 recommendations, including 3 to SBA and 1  each to the Department of Commerce’s National Oceanic and  Atmospheric Administration; the Departments of Defense, Education,  Energy, Health and Human Services, and Homeland Security; the  Environmental Protection Agency; and the National Science Foundation.  Specifically:  The Director of the Office of Investment and Innovation within SBA should  work with participating agencies to improve the reliability of its SBIR and  STTR award data (Recommendation 1).

The Director of the Office of Investment and Innovation within SBA should  work with participating agencies to implement the Commercialization  Benchmark or, if that is not feasible, revise the benchmark so that it can  be implemented (Recommendation 2).

The Director of the Office of Investment and Innovation within SBA should  update the SBIR and STTR policy directives to accurately reflect how the  consequence of the benchmarks is to be implemented (Recommendation  3).

The SBIR Program Manager of the Department of Commerce’s National  Oceanic and Atmospheric Administration should update the agency’s  SBIR project solicitation to accurately reflect how the consequence of not  meeting the benchmarks is to be implemented (Recommendation 4).

The SBIR Program Administrator within the Department of Defense  should update the agency’s SBIR and STTR project solicitations to  accurately reflect how the consequence of not meeting the benchmarks is  to be implemented (Recommendation 5).

The SBIR Program Manager within the Department of Education should  update the agency’s SBIR project solicitation to accurately reflect how the  consequence of not meeting the benchmarks is to be implemented  (Recommendation 6).

The SBIR Program Manager within the Department of Energy should  update the agency’s combined SBIR and STTR project solicitation to  accurately reflect how the consequence of not meeting the benchmarks is  to be implemented (Recommendation 7).

The SBIR/STTR Program Coordinator within the Department of Health  and Human Services should update the agency’s SBIR and STTR project  solicitations to accurately reflect how the consequence of not meeting the  benchmarks is to be implemented (Recommendation 8).

The SBIR Program Director within the Department of Homeland Security  should update the agency’s SBIR project solicitation to accurately reflect  how the consequence of not meeting the benchmarks is to be  implemented (Recommendation 9).

The SBIR Program Manager within the Environmental Protection Agency  should update the agency’s SBIR project solicitation to accurately reflect  how the consequence of not meeting the benchmarks is to be  implemented (Recommendation 10).

The SBIR and STTR Program Manager within the National Science  Foundation should update the agency’s SBIR and STTR project  solicitations to accurately reflect how the consequence of not meeting the  benchmarks is to be implemented (Recommendation 11).

We provided a draft of this report to SBA and the 11 participating  agencies for review and comment. In written comments, the Department  of Commerce’s National Oceanic and Atmospheric Administration; the  Departments of Defense, Education, Energy, Health and Human  Services, and Homeland Security; the Environmental Protection Agency;  and SBA agreed with the respective recommendations directed to their  agencies. Agencies’ written comments are reproduced in appendixes I  through VIII. An official from one agency—the National Science  Foundation—stated in an email that the agency concurred with the  recommendation and did not have any further comments. Two  agencies—the Department of Homeland Security and SBA—also  provided technical comments, which we incorporated as appropriate.  Three agencies—the Departments of Agriculture and Transportation, and  the National Aeronautics and Space Administration—as well as the  Department of Commerce’s National Institute of Standards and  Technology stated via email that they had no technical or written  comments.

In its comments, SBA stated that it disagreed with a statement in our draft  report that SBA had not worked with agencies to enter high-quality and  accurate data into the database and provided us documentation of an  instruction guide on entering data that SBA officials said was available to  agencies. Based on our review of this information, we clarified the text of  the report and modified the draft report’s recommendation by removing  the suggested example that SBA provide guidance to the agencies to  improve SBIR and STTR award data reliability. SBA agreed with the  revised recommendation.

After we provided a draft of the report to the agencies for comment, the  Departments of Education and Homeland Security took action on their  respective recommendations. Specifically, in December 2017, the  agencies issued new project solicitations that reflected the updated  consequence of not meeting the benchmarks. We agree that these  agencies fully implemented the recommendations we made to them in  this report.

We are sending copies of this report to the appropriate congressional  committees; the Secretaries of Agriculture, Commerce, Defense,  Education, Energy, Health and Human Services, Homeland Security, and  Transportation; the Administrators of the Small Business Administration,  the Environmental Protection Agency, and the National Aeronautics and  Space Administration; the Director of the National Science Foundation;  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or neumannj@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix IX.

Appendix I: Comments from the Small Business Administration

Appendix II: Comments from the Department of Commerce

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Education

Error! No text of specified style in document.

Appendix V: Comments from the Department of Energy

Appendix VI: Comments from the Department of Health and Human Services

Appendix VII: Comments from the Department of Homeland Security

Appendix VII: Comments from the Department of Homeland  Security Error! No text of specified style in document.

Appendix VIII: Comments from the Environmental Protection Agency

Appendix IX: GAO Contact and Staff Acknowledgments

Appendix IX: GAO Contact and Staff  Acknowledgments Error! No text of specified style in document.

In addition to the contact named above, Hilary Benedict (Assistant  Director), John Barrett, Natalie Block, Antoinette Capaccio, Tanya Doriss,  Justin Fisher, Ellen Fried, Juan Garay, Cindy Gilbert, Perry Lusk, William  Shear, and Elaine Vaurio made key contributions to this report.

Under two federal programs, agencies have awarded $46 billion to small businesses to develop and commercialize innovative technologies. The Small Business Administration and participating agencies set two benchmarks to assess the recipient's progress and eligibility for future awards. For example, small businesses must meet a certain amount of sales, investments, or patents resulting from their efforts.

However, we found data problems that prevented the agencies from accurately measuring businesses against these benchmarks.

We made several recommendations to SBA and agencies, including the need to improve data reliability.


Unmanned Aerial Systems: Air Force Pilot Promotion Rates Have Increased but Oversight Process of Some Positions Could Be Enhanced

An increasing number of Air Force missions use unmanned aerial systems, or RPAs, to provide their specialized capabilities in support of combat operations. The demand for crew members for these systems has grown rapidly. For example, RPA pilot requirements increased by 76 percent since fiscal year 2013 while those for fighter pilots stayed about the same. These requirements include pilots who serve in non-operational staff positions, such as trainers.

Senate Report 115-125 included a provision that GAO review career advancement for Air Force RPA pilots compared to other pilots. This report, among other things, describes (1) the rates that RPA and other pilots were promoted; (2) the rates that non-operational staff positions requiring RPA pilot expertise were assigned to various organizations, and (3) the extent to which the Air Force has reviewed its oversight process to effectively manage non-operational staff positions requiring aviator expertise.

Among other things, GAO analyzed Air Force pilot promotion data from 2006-2017. GAO also analyzed non-operational staff position data from fiscal years 2013-2018 and interviewed officials regarding the management and oversight of these positions.

The promotion rates for Air Force Remotely Piloted Aircraft (RPA) pilots have been generally similar to those of other pilots since 2013 and have increased over time. See figure below for promotion rates from major to lieutenant colonel. Air Force officials stated that RPA pilot promotion rates increased because the creation of a dedicated career field resulted in more competitive candidates.

Since 2013, over 75 percent of non-operational staff positions requiring RPA pilot expertise were assigned to various organizations within the Air Force, according to GAO's analysis. These positions carry out support and other noncombat-related activities as well as training functions and are essential to the development of officers. However, the overall number of these positions that require a RPA pilot is about one-tenth of the combined number of those requiring other pilots. For example, in fiscal year 2018, 83 non-operational staff positions required RPA pilots compared to 330 requiring fighter pilots. Air Force officials stated that the small number of RPA positions is because the career field is new.

The Air Force has not reviewed its oversight process to ensure that it is efficiently managing its non-operational staff positions that require aviator expertise. Air Force officials explained that over the last 10 years, the Air Force reduced the number of squadrons but had not reviewed the number of non-operational staff positions. Similarly, the Air Force has had no widely accessible oversight process to monitor whether it had established an accurate number of non-operational staff positions required to support the new RPA career field. In August 2018, the Air Force identified 513 non-operational staff positions (out of 2,783) as needing further review because they lacked adequate justification of the need for aviator expertise. Officials described the process for managing these positions as time and labor intensive, which can cause delays in obtaining reliable information needed to inform decision-making. By reviewing this process, the Air Force may be able to identify opportunities to create efficiencies and more effectively manage its non-operational staff positions requiring aviator expertise.

GAO recommends that the Air Force review its oversight process for managing the non-operational staff positions, including those for RPA pilots, to identify opportunities to increase efficiencies. DOD concurred with this recommendation.

An increasing number of Air Force missions call for the use of Remotely  Piloted Aircraft (RPAs) to provide their specialized capabilities in support  of combat operations. The Air Force uses RPAs to gather intelligence,  conduct surveillance and reconnaissance, and launch attacks against a  variety of targets. RPA aircrews consist of two people—a pilot and a  sensor operator. The pilot—in most cases a rated officer, i.e., an officer  possessing aviation expertise—flies the aircraft. The Air Force relied  solely on manned aircraft pilots assigned to fly RPAs until 2010 when it  established an RPA pilot career field for officers trained to fly only RPAs.  The sensor operator—an enlisted servicemember—controls the aircraft’s  sensors that record video and other intelligence information. The demand  for these skilled pilots and sensor operators has grown rapidly. For  example, from fiscal years 2013 to 2018, the Air Force experienced about  a 76 percent increase in its requirements of RPA pilots (1,366 to 2,404)  while its requirements for fighter pilots remained about the same (3,976 to  3,951).

Our prior work has identified challenges that the Air Force has  experienced with the growth of the RPA pilot career field. For example, in  April 2014, we found shortages of RPA pilots and that the Air Force faced  challenges recruiting, developing, and retaining pilots and building their  morale. Additionally, we found that RPA pilot promotion rates were lower  than those for other career fields. The Air Force generally concurred with  our seven recommendations in that report to address these issues and  subsequently has fully implemented all but one recommendation to  analyze the career field effect of being an RPA pilot to determine whether  and how being an RPA pilot is related to promotions. Further, in January  2017, we found, among other things, that the Air Force could improve its  strategic human capital planning, and we made three recommendations  to the Air Force and two to the Office of the Under Secretary of Defense  for Personnel and Readiness to which they generally concurred. As of  July 2018, the Air Force had taken some action, but has not fully  implemented these recommendations. Appendix I contains more details  related to the recommendations that we have made regarding unmanned  aerial systems pilot issues along with DOD’s and the Air Force’s actions  taken to address them.

In a report accompanying a bill for the National Defense Authorization Act  for Fiscal Year 2018, the Senate Armed Services Committee included a  provision for us to provide, among other things, the promotion rates for Air  Force RPA pilots since our 2014 report as well as for RPA sensor  operators. This report describes (1) the rates at which RPA pilots were  promoted and nominated to attend developmental education opportunities  as compared to the rates for pilots in other career fields; (2) the rates at  which enlisted RPA sensor operators were promoted as compared to the  rates for other enlisted servicemembers; (3) the rates at which non- operational staff positions requiring RPA pilot expertise were assigned  among Air Force organizations; and (4) reviews the extent to which the  Air Force has evaluated its oversight process used to manage non- operational staff positions requiring aviator expertise.

To determine the rates at which RPA pilots have been promoted, we  obtained from the Air Force Personnel Center data on the number of  promoted officers and the number of officers eligible from 2006 through  2017 for pilots from four career fields—bombers, fighters, mobility, and  RPA—who qualified as “In-the-Promotion-Zone” to the ranks of major  (grade O-4), lieutenant colonel (grade O-5), and colonel (grade O-6).  The promotion rates from 2006 through 2012 were initially reported in our  2014 report on Air Force RPA workforce issues and included in this  report for comparison purposes with the promotion rates from 2013  through 2017 that we calculated. From these data from 2013 through  2017, we calculated promotion rates to each rank for pilots from four  career fields—bombers, fighters, mobility, and RPAs—by dividing the  number of promoted officers by the number of eligible officers. We then  compared the annual RPA pilot promotion rates we calculated to (1)  those that we calculated for the other types of pilots to determine the  extent to which the rates were similar and (2) the promotion rates that we  reported in our 2014 report covering 2006 through 2012 to determine the  extent to which the rates from 2013 through 2017 had changed from  these previously reported promotion rates. Further, we reviewed Air Force  documents governing the officer promotion processes and interviewed  Headquarters Air Force Operations and Air Force Personnel Center  officials to obtain their perspectives on trends in RPA pilot promotion  rates.

To determine the rates at which RPA pilots have been nominated to  attend developmental education—e.g. professional military education— opportunities as compared to the rates for pilots in other career fields, we  analyzed intermediate and senior level developmental education  nomination data for all eligible officers for academic years 2014 through  2018. In order to analyze data consistent with the promotion analyses we  performed, we requested developmental education nomination data from  the same time period. However, officials told us that no academic year  2013 data was available but that academic year 2018 data were  available. Therefore, we obtained the most recent 5-year developmental  education nomination data available, which covered academic years 2014  through 2018. From these data, we calculated nomination rates by  dividing the number of nominated officers by the number of eligible pilots  from four career fields—bombers, fighters, mobility, and RPAs—that  competed for nominations. We then compared the annual RPA pilot  nomination rates we calculated to those of the other types of pilots to  determine the extent to which the rates were similar. We also reviewed  Air Force documents governing the developmental education nomination  process and interviewed Headquarters Air Force Personnel and Air Force  Personnel Center officials about such processes.

To determine the rates at which enlisted sensor operators have been  promoted as compared to the rates for other enlisted servicemembers,  we obtained from the Air Force Personnel Center data on the number  promoted and the number eligible for promotion to the ranks of Staff  Sergeant (grade E-5) through Chief Master Sergeant (grade E-9) for RPA  sensor operators and the entire population of enlisted personnel. In order  to analyze data consistent with promotion analyses we performed, we  obtained this enlisted promotion data from the same period of 2013  through 2017. From these data, we calculated promotion rates for each  year by dividing the number of promoted enlisted servicemembers by the  number of eligible enlisted servicemembers. For each year, we compared  the annual enlisted RPA sensor operator promotion rates to those of the  entire population of enlisted servicemembers that were eligible for  promotions to determine the extent to which the rates were similar. We  also reviewed Air Force documents governing the enlisted promotion  processes and interviewed Headquarters Air Force Personnel and Air  Force Personnel Center officials to obtain their perspectives on trends in  enlisted sensor operator promotion rates.

To determine the rates at which non-operational staff positions that  require RPA pilot expertise are assigned among Air Force organizations,  we obtained the number of non-operational staff positions required and  assigned within Air Force organizations for rated officers (i.e., have  aviator expertise) from four selected careers fields—bombers, fighters,  mobility and RPA pilots. These data were obtained from Headquarters  Air Force Operations, Headquarters Air Force Personnel, and Air Force  Personnel Center. In order to analyze data consistent with the promotion  analyses we performed, we obtained this non-operational staff position  assignment data from the same fiscal years 2013 through 2017. Because  fiscal year 2018 data became available during the time of our review, we  also included it in our analysis. Therefore, we obtained the most recent 6- year non-operational staff position assignment data available, which  covered fiscal years 2013 through 2018. From these data, we calculated  annual non-operational staff position assignment rates by dividing the  number of positions assigned by the number of positions required by Air  Force organizations for the four types of pilots. We then compared the  annual RPA non-operational staff position assignment rates to those  calculated for the other types of pilots to determine the extent to which the  rates were similar.

Additionally, to determine the extent to which the Air Force has reviewed  its oversight process to effectively manage its non-operational staff  positions that require aviator expertise, we reviewed Air Force instructions  related to the requirements and responsibilities for managing the process  for reviewing and justifying the need for aviator expertise in non- operational staff positions. We evaluated Air Force practices for  obtaining and using information for various Air Force organizations  regarding their justification for aviator expertise in their non-operational  staff positions against requirements from both the applicable Air Force  instructions and the Standards for Internal Control in the Federal  Government. This included the importance of designing control activities  to achieve objectives and respond to risks and using quality information  by identifying information requirements, obtaining relevant data from  reliable sources in a timely manner, and processing the obtained data into  quality information. Further, we also interviewed operations officials from  both Headquarters Air Force and the Air Force Air Combat Command to  obtain their perspectives of the process used to review and justify the  need for aviator expertise in staff positions. Further, we also interviewed  Headquarters Air Force officials regarding the status of their efforts to  respond to a House of Representatives requirement for the Secretary of  Air Force to report the results of a review of every staff position requiring  aviator expertise within the Air Force.

To assess the reliability of the data used for each of the objectives, we  reviewed technical documentation for each data source to understand the  methods used to collect, store, and maintain these data; assessed the  data for errors, omissions, and inconsistencies; and interviewed officials  from Headquarters Air Force operations directorate, Headquarters Air  Force personnel directorate, and the Air Force Personnel Center who  were familiar with the systems from which the data were extracted. We  also considered the use of the data in prior related GAO reports. We  determined that the data were sufficiently reliable for our purposes of  reporting historical promotion, developmental education selection, and  rated staff position allotment trends, respectively.

We conducted this performance audit from January 2018 to February  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

RPA aircrews consist of a pilot and a sensor operator. The Air Force in  most cases assigns officers to fly its RPAs. The Air Force relied solely  on manned aircraft pilots to fly remotely piloted aircraft until 2010 when it  established a RPA pilot career field (designated as Air Force Specialty  Code 18X) for officers trained to fly only RPAs. As of December 2013,  approximately 42 percent of the RPA pilots were temporarily assigned,  manned aircraft pilots and manned aircraft pilot training graduates. Both  of those groups of RPA pilots are temporarily assigned to fly RPAs with  the assumption that after their tour they will return to flying their manned  aircraft. By comparison, as of September 2018, manned aircraft pilots and  manned aircraft pilot training graduates comprised only 17 percent of the  RPA pilots. Further, the number of permanent RPA pilots has increased  from 58 percent of all RPA pilots in December 2013, to 83 percent as of  September 2018, as shown in figure 1.

Additionally, Air Force enlisted personnel operate the RPAs’ sensors,  which provide intelligence, surveillance, and reconnaissance capabilities.  As a crewmember, the RPA sensor operators provide assistance to the  RPA pilot with all aspects of aircraft use, such as tracking and monitoring  airborne, maritime and ground objects and continuously monitoring the  aircraft and weapons systems status.

The Defense Officer Personnel Management Act, as amended, created a  standardized system for managing the promotions for the officer corps of  each of the military services. Pursuant to the established promotion  system, the secretaries of the military departments must establish the  maximum number of officers in each competitive category that may be  recommended for promotion by competitive promotion boards. Within the  Air Force, there are groups of officers with similar education, training, or  experience, and these officers compete among themselves for promotion  opportunities. There are several competitive categories including one that  contains the bulk of Air Force officers called the Line of the Air Force,  which includes RPA pilots, as well as pilots of manned aircraft and other  operations-oriented careers.

To determine the best-qualified officers for promotion to positions of  increased responsibility and authority, the Air Force appoints senior  officers to serve as members of a promotion selection board for each  competitive category of officer in the Air Force. Promotion selection  boards consist of at least five active-duty officers who are senior in grade  to the eligible officers and who reflect the eligible population with respect  to minorities and women, as well as career field, aviation skills, and  command in an attempt to provide a balanced perspective. Promotion  boards convene at the Air Force Personnel Center headquarters to  perform a subjective assessment of each officer’s relative potential to  serve in the next higher grade by reviewing the officer’s entire selection  folder. This “whole-person concept” involves the assessment of such  factors as job performance, professional qualities, leadership, job  responsibility, depth and breadth of experience, specific achievements,  and academic and professional military education.

Developmental Education Program Selection Process

The Air Force developmental education programs expand expertise and  knowledge as well as a path that helps to ensure that personnel receive  the appropriate level of education throughout their careers. Officers  have three opportunities to compete for intermediate developmental  education programs, which focus on warfighting within the context of  operations and leader development, such as at the Air Command and Staff College. Officers have four opportunities to compete for senior  developmental education programs, such as at the Air War College,  which are designed to educate senior officers to lead at the strategic level  in support of national security, and in joint interagency, intergovernmental  and multinational environments.

A subset of developmental education is Professional Military Education,  which includes resident and non-resident attendance options open to  officers in both the intermediate and senior developmental education  programs. Nonresident programs exist to provide individuals who have  not completed resident programs an opportunity to complete them via  correspondence, seminar, or other approved methods. Prior to 2017,  officers who were identified by their promotion board as a developmental  education candidate or “selectee” were assured of the opportunity to  attend some form of developmental education in-resident program.  However, in March 2017, the Air Force announced changes to its  nomination process for officer developmental education by separating in- residence school selection status from promotion decisions. Since that  time, commanders nominate candidates for in-residence, developmental  education programs based on individual performance.

Various Career Assignments for Officers with Aviation Expertise

Officers with aviation expertise, including RPA pilots, at various points in  their careers, may rotate through both flying and nonflying positions to  broaden their career experiences. Operational positions, whether flying or  nonflying, include those positions that exist primarily for conducting a  military action or carrying out a strategic, tactical, service, training or  administrative military mission. Operational positions include a range of  flying positions, such as for RPA pilots, operating aircraft to gather  intelligence or conduct surveillance, reconnaissance or air strikes against  a variety of targets. Operational positions that are non-flying positions  could include assignments as a close-air-support duty officer in an Air  Operations Center.

Non-operational staff positions are generally non-flying positions and  include assignments to headquarters or combatant command positions.  Certain non-operational staff positions can be filled only by qualified  pilots. Other non-operational positions are more general in nature and are  divided among officer communities to help carry out support activities,  training functions, and other noncombat related activities in a military  service. These positions could include positions such as a recruiter,  working as an accident investigator, advisor to foreign militaries, or a  policy position at an Air Force major command. The Air Force views  nonoperational staff positions as a means to develop leaders with the  breadth and depth of experience required at the most senior levels inside  and outside the Air Force.

Roles and Responsibilities Related to Aircrew Management

Various offices within the Air Force have roles and responsibilities for the  management of aircrew positions and personnel.

The Deputy Chief of Staff for Operations is to establish and  oversee policy to organize, train and equip forces for the Department  of the Air Force. This specifically includes the responsibility for all  matters pertaining to aircrew management.

The Directorate of Operations is responsible for developing and  overseeing the implementation of policy and guidance governing  aircrew training, readiness, and aircrew requirements. The  directorate is the approval authority for aircrew distribution plans,  rated allocation oversight and any other areas that have significant  aircrew management implications.

The Operational Training Division produces the official Air  Force aircrew personnel requirements projections, and in  conjunction with the Military Force Policy Division, develops  and publishes the Rated Management Directive, formerly  known as the Rated Staff Allocation Plan, as approved by the  Chief of Staff of the Air Force as designed to meet near-term  operational as well as long-term leadership development  requirements.

The Office of the Deputy Chief of Staff for Manpower, Personnel,  and Services has responsibilities that include developing personnel  policies, guidance, programs, and other initiatives to meet the Air  Force’s strategic objectives to include accessions, assignments,  retention, and career development.

The Directorate of Force Management Policy, the Force  Management Division analyzes officer, enlisted and civilian  personnel issues. The division also maintains a variety of  computer models and databases to analyze promotion, retention,  accession, compensation and separation policy alternatives.

Additionally, it is responsible for providing official aircrew  personnel projections for use in various management analyses.

The Air Force Personnel Center, one of three field-operating  agencies reporting to the Deputy Chief of Staff of the Air Force,  Manpower, Personnel and Services, conducts military and civilian  personnel operations such as overseeing performance  evaluations, promotions, retirements, separations, awards,  decorations and education. The Center also directs the overall  management and distribution of both military and civilian  personnel.

Since 2013 RPA Pilots Have Been Promoted and Nominated for Education Opportunities at Rates Generally Similar to Pilots in Other Fields

RPA Pilots Have Been Promoted at Rates Generally Similar to Those of Pilots in Other Career Fields

Based on our analysis of Air Force promotion data, the percentage of  RPA pilots promoted were generally similar in comparison to the  promotion rates of pilots in other career fields since 2013. However, it is  important to note that since the population of eligible RPA pilots to be  considered for promotion was smaller than other pilot populations, the  promotion of one or two RPA pilots could have a large effect on their  promotion rate. For example, the RPA pilot promotion rates were within  10 percentage points of the promotion rates for the other types of pilots in  each year of those years in 8 out of 10 promotion boards to major and to  lieutenant colonel held during that time frame.

RPA pilot promotion rates from captain to major were generally similar as  the promotion rates for other pilots from 2014 through 2017, as shown in  figure 2. For example, in 2014, 94 percent of eligible RPA pilots (29 of  31), bomber pilots (47 of 50), fighter pilots (189 of 201) and 91 percent of  eligible mobility pilots (355 of 388) were promoted from captain to major.  This is an improvement in promotion rates for RPA pilots compared to  2006 through 2012, where RPA pilot promotion rates fell below those for  all other pilots in 5 of the 7 promotion boards held.

Additionally, the promotion rates for RPA pilots from major to lieutenant  colonel relative to other types of pilots in 2013 through 2017 showed a  similar improvement compared to 2006 through 2012, as shown in figure  3. For example, in 2017, 75 percent of eligible RPA pilots (15 of 20) were  promoted, which is generally similar to the promotion rates for the other  pilots—78 percent for bomber pilots (18 of 23), 83 percent for fighter  pilots (75 of 90), and 72 percent for mobility pilots (143 of 199). However,  in 7 of the 8 promotion boards held from 2006 through 2012, RPA pilot  promotion rates from major to lieutenant colonel fell below the promotion  rates for all other pilots.

The one exception to the promotion rates being generally similar was the  rate at which RPA pilots were promoted from lieutenant colonel to  colonel. In this case, the rates for RPA pilots diverged notably from the  promotion rates of bomber, fighter, and mobility pilots from 2013 to 2017.  For example, in 2016, 1 out of the 5 (20 percent) eligible RPA pilots was  promoted to colonel. In contrast, 13 of 21 (62 percent), bomber pilots, 32  of 51 (63 percent) fighter pilots, and 34 of 65 (52 percent) mobility pilots  were promoted from lieutenant colonel to colonel. However, the  promotion rates of RPA pilots from lieutenant colonel to colonel that we  calculated should be considered cautiously as fewer than 10 RPA pilots  were eligible for promotion boards each year through this time period.  The promotion of one or two officers could have a large effect on the  promotion rate due to the small number of eligible RPA pilots.

In April 2014, we reported that Air Force officials attributed the low RPA  pilot promotion rates from 2006 through 2012 generally to the process  that it used to staff RPA pilot positions at that time. Specifically, they  stated that commanders generally transferred less competitive pilots from  other pilot career fields to RPA squadrons to address the increased  demand. Air Force officials also stated that these officers generally had in  their records fewer of the factors that the Air Force Personnel Center  identified that positively influence promotions than their peers. They said  that because the bulk of RPA pilots who competed for promotion during  the time of our previous review was transferred using this process, these  were the reasons that RPA pilots had been promoted at lower rates than  their peers.

Air Force officials stated that they believed the trend of increased  promotion rates for RPA pilots from 2013 through 2017 mostly reflected  the change in the population of eligible pilots who were recruited and  specialized as an RPA pilot (i.e., the 18X career field). According to Air  Force officials, the creation and establishment of this career field resulted  in an increase in the number of skilled and more competitive promotion  candidates. Specifically, as of September 2018, the number of permanent  RPA pilots outnumbered all other types of pilots serving as RPA pilots  combined.

RPA Pilots Have Been Nominated to Developmental Education Programs at Rates Similar to Pilots in Other Career Fields

RPA pilots were nominated to attend developmental education programs,  such as professional military education, at rates similar to the rates for  other pilots from academic years 2014 through 2018, according to our  analysis of Air Force data. An officer’s attendance at developmental  education programs can be a factor that is taken into consideration when  being assessed for promotion. Our analysis showed that, for the  academic years 2014 through 2018, nomination rates for RPA pilots to  Intermediate and Senior Developmental Education programs combined  ranged from a low of 25 percent for academic year 2016 to a high of 31  percent for academic year 2015. In comparison, nomination rates across  the same time period for pilots in other career fields ranged from a low of  21 percent for mobility pilots for academic year 2016 to a high of 35  percent for fighter pilots for academic year 2014. Table 1 provides the  various nomination rates for each of the different types of pilots that we  analyzed.

RPA Sensor Operators Have Been Promoted at Rates Similar to Other Enlisted Servicemembers

The Air Force promoted enlisted RPA sensor operators at a rate similar to  the rates of all enlisted servicemembers, according to our analysis of Air  Force promotion data. Specifically, the Air Force promoted an average  of 100 RPA sensor operators (or an average of 26 percent) annually for  the period from 2013 through 2017. Similarly, the Air Force annually  promoted an average of approximately 27,000 enlisted personnel (or an  average of 25 percent) for the same period. Our analysis showed that in  2013 through 2017, promotion rates for RPA sensor operators ranged  from a low of 18 percent in 2014 to a high of almost 35 percent in 2017.  The promotion rates across the same time period for all other enlisted  servicemembers ranged from a low of approximately 19 percent in 2014  to a high of 32 percent in 2017. Table 2 provides the various promotion  rates that we analyzed.

Air Force enlisted servicemembers in the lowest four levels (grades E1- E4) are selected for promotion based on time in grade and time in  service. Selection for promotion to the next two levels, known as the non- commissioned officer levels (grades E5 and E6), is based on the  Weighted Airman Promotion System to fill the requirement. This system  provides weighted points for an individual’s performance record and  service decorations received, and the results of tests to assess an  individual’s promotion fitness and job skills and knowledge. Selection for  promotion to the senior non-commissioned officer level (grades E7-E9) is  based on the same Weighted Airman Promotion System plus the results  from a central board evaluation. Servicemembers eligible for promotions  to the non-commissioned ranks are assessed and then listed from the  highest to lowest scores and offered promotion if they fall above a specific  cutoff score established to meet quotas within each career field and for  each rank.

While enlisted servicemembers must pass knowledge and skills tests to  qualify for promotions, officials explained that the resulting promotion  rates essentially reflect requirements and are not indicative of  competitiveness across career fields as with officer promotion rates.  Officials stated that enlisted servicemember promotions are based on the  service’s numeric personnel requirements for each enlisted grade. To  consider an enlisted servicemember for promotion from among those who  are eligible, a vacancy must first be required at the next higher grade  within that servicemember’s occupational area, known as their Air Force  Specialty Code that needs to be filled. For example, in 2017, the Air  Force required promotions for 128 RPA sensor operators, and officials  promoted that many enlisted servicemembers from the cohort of 370  eligible servicemembers.

Air Force Assigned Non-operational Staff Positions Requiring RPA Pilots at High Rates Since 2013

For each year since 2013, the Air Force has assigned over 75 percent of  the non-operational staff positions that require an RPA pilot to the  organizations that had requested those positions, according to our  analysis of service headquarters data. However, the overall number of  non-operational staff positions that require an RPA pilot is about one- tenth of the number of those requiring pilots in other career fields. For  example, in fiscal year 2018 the Air Force had 83 non-operational staff  positions that required an RPA pilot compared to 330 positions requiring  fighter pilots. Air Force officials stated that the number of RPA positions  was smaller than for other pilots because the career field is relatively new  and still growing.

Non-operational staff positions are generally non-flying positions and  include assignments to headquarters or combatant command positions.  Certain non-operational staff positions can be filled only by qualified  pilots. Other non-operational positions are more general in nature and are  divided among officer communities in a military service. Officers with  aviation expertise, including RPA pilots, at various points in their careers  may rotate through both flying and nonflying positions to broaden their  career experiences and Air Force officials stated that staff assignments  are essential to the development of officers who will assume greater  leadership responsibilities.

Headquarters Air Force prepares allocation or “assignment” plans to  provide positions requiring aviator expertise to various Air Force  commands and other entities. Under this process, these organizations  identify the number of non-operational staff positions requiring aviator  expertise (e.g., pilots) they require as well as indicate the type of aviator  expertise that is needed to fill those positions, (e.g., fighter, bomber,  RPA). Headquarters Air Force then determines the extent to which the  staff position requirements can be met in accordance with senior  leadership priorities designed to equitably manage the shortage of  officers with aviation expertise. The results of this process are outlined in  the Air Force’s annual Rated Management Directive which reinforces  each organization’s flexibility for using their entitlements in non- operational staff and other positions.

In some instances, the Air Force is able to assign enough positions to an  organization to meet nearly all of its non-operational staff position  requirements. For the purposes of our analyses, the assignment rate is  determined by the number of positions assigned compared to the number  of positions the organization required. For example, in fiscal year 2018  the Air Force assigned 99 percent of the non-operational staff positions  that require an RPA pilot to the requesting entities. In other instances, the  Air Force assignment rate of non-operational staff positions may be much  lower because of competing management priorities or shortages of  personnel in a career field. As a result, the Air Force’s assignment of staff  positions can vary across the different career fields. For example, the Air  Force fighter pilot career field has had fewer fighter pilots than its  authorization number since 2013. Therefore, the Air Force assignment  rate for staff positions requiring fighter pilots is significantly lower than the  rate for staff positions requiring other types of pilots. For example, in fiscal  year 2017, the Air Force assignment rate for staff positions requiring a  fighter pilot was 18 percent, which was less than a quarter of the rate for  staff positions requiring an RPA pilot, as shown in table 3.

The Air Force Has Not Reviewed Its Oversight Process to Manage Its Non- operational Staff Positions That Require Aviator Expertise

The Air Force has not reviewed its oversight process to ensure that it is  effectively and efficiently managing its review of non-operational staff  positions that require aviator expertise, such as RPA pilots. Air Force  officials explained that its oversight process for managing these positions  requiring pilot expertise consists of a time-consuming, labor-intensive  process of exchanging emails and spreadsheets with 57 organizations,  such as various Air Force Major commands like the Air Combat  Command, the Air Force Special Operations Command, and the National  Guard Bureau. According to these officials, this process consists of the  maintenance and exchange of spreadsheets and briefing slides with  information about every position found throughout the Air Force and in  various other entities that are required to be reviewed and validated  annually. Additionally, this process is maintained by one official within the  Headquarters Air Force who must exchange the spreadsheets via email  approximately twice a year with officials from each of the organizations  that are responsible for annually justifying their continued need for non- operational staff positions requiring aviator expertise. Air Force officials  stated that this process does not always produce complete and accurate  information in a timely manner as in some instances the information  produced is not relevant by the time a complete review of the positions is  accomplished.

Headquarters Air Force officials familiar with its oversight responsibilities  stated that using a different system would more efficiently and effectively  support their ability to manipulate, analyze and share information among  the applicable organizations and make informed decisions. For example,  these officials explained that over the last 10 years, the Air Force drew  down the number of squadrons, but did not do a good job of cross  checking that reduced number of squadrons with a revised number of  staff positions required for support. Therefore, the number of non- operational staff positions was not adjusted and are now artificially high in  some career fields and others may have fewer non-operational staff  positions than needed. These officials added that as the new RPA pilot  career field has developed, there has been no timely and widely  accessible system of checks and balances to establish an accurate  number of non-operational staff positions required to support the career  field. Further, they said that using a different system that allows them to  have more timely and quality information would enhance their ability to  manage and make decisions regarding the appropriate mix of expensive  pilots and others with aviator expertise between operational line positions  and non-operational staff position needs. They said this would better  ensure that there is a reasonable range of non-operational staff positions  required for each career field, such as for the growing RPA pilot career  field.

An October 2017 memorandum from the Air Force Chief of Staff stated  that the number of non-operational staff positions which require aviation  expertise must be brought into balance with the Air Force’s ability to  produce the appropriate number of officers with aviator expertise. The  memorandum also stated that organizations were strongly encouraged to  change their current requirements to meet the available current force  levels including converting chronically unfilled non-operational staff  positions requiring aviator expertise to positions specifically designated  for RPA pilots. As a result of two separate reviews, Air Force officials  identified hundreds of these positions that lacked adequate justification or  qualifications to support the positions’ requirement to be filled by officers  with aviator expertise. For example, in August 2018, out of 2,783 non- operational staff positions, the Air Force found that 513 of these positions  were evaluated as lacking adequate justification or mission qualifications  to support the need for aviator expertise and 61 positions were eliminated  after further review.

Prior to 2010, according to officials, the Headquarters Air Force  maintained a web-based management oversight system to review and  approve the justifications for its non-operational staff positions requiring  aviator expertise that allowed for wide access to and manipulation and  timely analyses of information. Additionally, this former system provided  multilevel coordination among Headquarters Air Force and its major  commands for reviewing the justifications of all of the positions. According  to Headquarters Air Force officials, the use of this management oversight  system was discontinued in 2010 due to a decision to no longer fund the  contractor maintaining the system. In October 2018, officials from one of  the Air Force’s Major Commands confirmed that the current oversight  system in use is time-consuming, does not readily support information  analysis and that plans to integrate it with another existing management  system had not happened.

The Headquarters Air Force official in charge of managing this process  told us that he had submitted multiple requests over the last 3 years to  integrate the information being managed with spreadsheets and emails  into an existing personnel management system to improve the efficiency  of the process. However, according to this official, higher priorities and funding issues have precluded the information from being integrated into  another existing system. In September, 2018, another Air Force official  told us that the Program Management Office that manages a system into  which the information could be integrated was behind schedule in  implementing several other system updates. Because of these delays, the  official acknowledged that no review has yet been done of what is needed  to provide the most efficient management oversight process of the  information currently being managed via the spreadsheet process. The  official said that before any actions could take place, a review of  requirements and priorities would be needed in order to make a  determination as to what changes could be made. Therefore, he said that  there are no decisions or timelines available for reviewing a process that  would provide the validation information for non-operational staff positions  in a timelier and widely accessible manner.

Air Force instructions state that major commands are required to perform  annual aircrew requirements reviews including review and revalidation of  all aircrew positions, except for rank of colonel or higher, to ensure aviator  expertise is required, and report the results to the Headquarters Air Force  Operations Training Division. Further, the Headquarters Air Force  Operations Training Division has the responsibility to ensure a  management process is in place to provide efficient and effective  oversight of the major commands’ annual review and revalidation of the  aircrew position requirements process. Additionally, Standards for Internal  Control in the Federal Government states that management should  identify needed information, obtain the relevant information from reliable  sources in a timely manner, and process the information into quality data  to make informed decisions and evaluate its performance in achieving  key objectives and addressing risks.

By reviewing its oversight process, the Air Force may be able to identify a  more efficient manner to manage its non-operational staff positions that  require aviator expertise. A management oversight process that provides  timely and widely accessible position justification information may help  ensure that the proper type of aviator expertise needed in these positions  is up to date. In turn, this could result in a more efficient use of the Air  Force’s short supply of expensive pilot resources, particularly fighter  pilots, and could potentially improve its ability to assign and develop  effective leaders, such as those within the growing RPA career field.

The Air Force continues to expand the use of RPAs in its varied missions  of intelligence gathering, surveillance and reconnaissance, and combat  operations. While the overall number of eligible RPA pilots is much  smaller compared to other pilots, over the last 5 years RPA pilots have  achieved promotions and nominations to attend developmental education  programs at rates that were generally similar in comparison to pilots in  other career fields. Additionally, non-operational staff positions requiring  RPA pilots have been assigned to entities at high rates since 2013, but  the number of positions available to them is smaller than the number that  require fighter, bomber, and mobility pilots because the career field is still  growing.

Air Force officials have noted problems with the current oversight process  which may be hindering its ability to efficiently and effectively manage  these non-operational staff positions as required by Air Force policy. For  example, the Air Force has recently identified that a large number of  these positions designated as requiring officers with aviator expertise  lacked adequate justification for that requirement. By reviewing the  efficiency and effectiveness of its management oversight process that  provides information in a timelier and more widely accessible manner, the  Air Force could better ensure that it makes informed decisions regarding  the need for pilots in certain non-operational staff positions and is in  compliance with policy. It also could help ensure that the Air Force more  efficiently uses its short supply of expensive pilot resources. Ultimately,  this may positively affect its ability to assign and develop effective  leaders, such as those within the growing RPA career field.

The Secretary of the Air Force should review its management oversight  process that provides information and documents the justifications of the  Air Force’s non-operational staff positions requiring aviator expertise,  including RPA positions, to identify opportunities for increased efficiency  and effectiveness and take any necessary actions. (Recommendation 1)

In written comments reproduced in appendix II, DOD concurred with  comments to the recommendation, and provided separate technical  comments, which we incorporated as appropriate.

DOD concurred with the recommendation to review the management  oversight process that provides information and documents the  justifications of the Air Force’s non-operational staff positions requiring  aviator expertise, including RPA positions, to identify opportunities for  increased efficiency and effectiveness and to take any necessary actions.  In its comments, DOD stated that it agrees the current oversight process  is time-consuming and could be more efficient. However, it believes this  process is effective because the Air Force was able to validate the need  for having pilots fill a majority of its non-operational staff positions during  a recent congressionally-mandated review of these positions. As we  reported, this review of all staff positions requiring aviator expertise  across the Air Force and other defense entities discovered more than 500  of approximately 2,800 positions that were initially found to be lacking  adequate justifications, and 61 positions eventually were eliminated. We  believe the Air Force’s results from this one-time review is an example of  how the current process is not consistently yielding up-to-date validations  of positions. Further, DOD also stated that while a move to automating  the process again has been considered, current funding shortfalls prevent  the Air Force from establishing an automated system to increase the  process’s efficiency. We continue to believe that the Air Force should  review its current process in order to identify any viable means to  increase its efficiency and effectiveness. Such a review may provide the  Air Force with opportunities to more consistently provide the proper type  of aviator expertise needed to fill its staff positions as well as potentially  provide more leadership opportunities to those within growing career  fields, such as RPA pilots.

We provided a draft of this report to DOD for review and comment.

We are sending copies of this report to the appropriate congressional  committees, the Acting Secretary of Defense, and the Secretary of the Air  Force. In addition, this report will be available at no charge on the GAO  website at http://www.gao.gov.

If you or your staff have any questions regarding this report, please  contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.

Appendix I: Steps Taken by the Department of Defense and the Air Force to Address Prior GAO Report Recommendations

Since 2014, we have issued three reports assessing the Air Force’s  remotely piloted aircraft (RPA) workforce management.

In April 2014, we found that the Air Force had shortages of pilots of  remotely piloted aircraft (RPA) and faced challenges to recruit,  develop, and retain pilots and build their morale. We also found that  Air Force RPA pilots experienced potentially challenging working  conditions and were promoted at lower rates than other career fields.  We made seven recommendations, and the Air Force generally  concurred with our recommendations. It has fully implemented all but  one recommendation to analyze the career field effect of being an  RPA pilot to determine whether and how being an RPA pilot is related  to promotions.

In May 2015, we found that the Air Force faced challenges ensuring  that their RPA pilots completed their required training and that the  Office of the Deputy Assistant Secretary of Defense for Readiness  had not issued a training strategy that addresses if and how the  services should coordinate with one another to share information on  training pilots who operate unmanned aerial systems. We made one  recommendation related to these findings with which DOD concurred.  However, in September 2018, an official from the Office of Secretary  of Defense for Readiness stated that there are compelling reasons  why a training strategy is no longer necessary and that no action is  planned to implement the recommendation.

In January 2017, we found, among other things, that the Air Force had  not fully tailored a strategy to address the UAS pilot shortage and  evaluated their workforce mix of military, federal civilian, and private- sector contractor personnel to determine the extent to which these  personnel sources could be used to fly UAS. We made five  recommendations related to these findings with which the Air Force  and DOD generally concurred. As of July 2018, the Air Force has  taken some action to address the first three recommendations and  officials from the Office of the Under Secretary of Defense for

Personnel and Readiness have fully implemented the other two  recommendations.

In table 4, we present the recommendations that we made to the Air  Force and the Under Secretary of Defense for Personnel and Readiness  and summarize the actions taken to address those recommendations as  of September 2018.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Lori Atkinson (Assistant Director),  Rebecca Beale, Amie Lesser, Felicia Lopez, Grant Mallie, Ricardo  Marquez, Richard Powelson, Amber Sinclair, and John Van Schaik made  key contributions to this report.

Many Air Force unmanned aerial systems (UAS) are larger and more complex than commercial drones, requiring special training and investment in their pilots.

Since 2013, promotion rates for UAS pilots have increased to match those of other pilots. However, there are relatively fewer non-flying positions—which are essential for career development—requiring UAS pilots' expertise.

We recommended improving how the Air Force manages and justifies its non-flying pilot positions to help it better use its resources and develop leaders among UAS pilots.

This text was updated due to an incorrect date.