Small Business Administration: Actions Needed to Improve Confidence in Small Business Procurement Scorecard

Each year SBA produces a scorecard measuring federal contract spending allocated to small businesses. The 2016 NDAA included a provision for SBA to revise the scorecard's methodology and for GAO to evaluate the effects of those revisions for fiscal year 2017. This report discusses, among other things, (1) SBA's changes to the scorecard methodology and plans, if any, to evaluate the effects of these changes, (2) the extent to which SBA has processes to disseminate reliable information, and (3) views of selected stakeholders on the scorecard's effects on small business procurement opportunities.

GAO analyzed SBA's prior and revised scorecard methodology and results and interviewed officials from SBA, four other federal agencies selected based on small business procurement volume and other attributes, and three groups representing the interests of small businesses.

For fiscal year 2017, the Small Business Administration (SBA) revised the methodology for its Small Business Procurement Scorecard, which is used to assess federal agencies' progress toward small business procurement goals. SBA made revisions to address requirements specified in the National Defense Authorization Act for Fiscal Year 2016 (2016 NDAA). SBA (1) reduced the share of the total scorecard grade devoted to prime contracting achievement, which is the dollar amount of contracts awarded directly to small businesses, and (2) added an element calculating changes in the number of small businesses receiving prime contracts. SBA made two additional revisions—with input from other agencies' representatives—to increase the share of subcontracting achievement results and peer review of required activities designed to facilitate small business procurement (see figure). In July 2018, officials said they had begun developing a plan to evaluate the effects of the revised scorecard methodology but did not provide a draft plan. Conducting a well-designed and comprehensive evaluation could aid SBA in determining whether the scorecard is an effective tool for helping to achieve the agency's strategic goals.

(Scorecard elements are expressed as a percentage of total scorecard grade.)

The published fiscal year 2017 scorecards originally contained errors, including an incorrect grade and numeric score for one agency, and SBA does not have a process to ensure that scorecard results are published accurately. Although SBA later corrected the errors, the agency did not initially document that scorecards had been changed, which is inconsistent with SBA's policy on information quality. SBA officials said that errors occurred in the process of formatting scorecards for publication. Errors in the published scorecards—and the initial lack of disclosure about corrections—weaken data reliability and may undermine confidence in scorecard data.

Agency officials and representatives of small business groups that GAO interviewed generally expected the scorecard revisions to have little impact on small business procurement opportunities. However, one agency's officials said they would focus more on tracking subcontracting activity as a result of changes to the scorecard.

GAO is recommending that SBA (1) design and implement a comprehensive evaluation to assess scorecard revisions and (2) institute a process for reviewing scorecards for accuracy prior to publication and a mechanism for disclosing corrected information. SBA generally agreed with GAO's recommendations.

The federal government, through contracts, purchased more than $440  billion worth of goods and services in fiscal year 2017. To help small  businesses access these federal contracting opportunities, Congress set  a requirement that the federal government allocate at least 23 percent of  its contracted spending to small businesses. In turn, according to the  Small Business Administration (SBA), small businesses provide the  federal government with quality, performance, innovation, agility, and  competitive pricing and are a key source of job creation. Each year, SBA  produces a Small Business Procurement Scorecard (scorecard) to  measure how much contracted spending federal agencies allocate to  small businesses and whether the federal government is meeting its  goals for awarding contracts to small businesses.

As part of the National Defense Authorization Act for Fiscal Year 2016  (2016 NDAA), Congress directed SBA to take steps to revise the  scorecard methodology for measuring small business procurement. The  2016 NDAA also included a provision for us to evaluate how well the  scorecard methodology accurately and effectively measures federal  agencies’ compliance with small business contracting goals and how well  it encourages federal agencies to expand small businesses’ procurement  opportunities.

This report discusses (1) revisions to the Small Business Procurement  Scorecard methodology and results of the revised fiscal year 2017  scorecard, as well as the extent to which SBA plans to evaluate the  effects of revisions; (2) the extent to which SBA’s revised scorecard  methodology uses relevant and reliable information and SBA publishes  accurate scorecards; and (3) views of selected federal agencies and  industry stakeholders on the extent to which SBA’s revised scorecard  methodology may encourage agencies to expand small business  procurement opportunities.

To examine changes SBA made to the Small Business Procurement  Scorecard, we reviewed SBA’s documentation describing the revised  scorecard methodology and interviewed SBA officials about their process  for implementing a revised scorecard methodology. We also interviewed  SBA officials about their plans, if any, to evaluate the revised scorecard.  We reviewed and analyzed scorecard data from fiscal years 2014 through  2017. We assessed the reliability of these data by analyzing them for  obvious errors of accuracy. We determined that SBA’s corrected data  were sufficiently reliable for the purpose of our analyzing scorecard  results for fiscal year 2017. We evaluated SBA’s process for revising the  scorecard against federal internal control standards. We also used GAO  guidance on evaluation design to identify examples of key attributes of  effective evaluation planning. We also interviewed representatives from  a judgmental, nongeneralizable sample of four agencies (the  Departments of Agriculture, Defense, Energy, and Homeland Security) to  obtain their views about the process of providing input on scorecard  revisions and the revised scorecard methodology. We selected the four  departments based on a variety of attributes, including small business  procurement volume, recent improvement in scorecard results, and level  of participation in discussions with SBA and other agencies about  potential changes to the scorecard. To determine the extent to which  SBA’s revised scorecard methodology uses relevant and reliable  information, we examined SBA documentation about the revised  scorecard methodology for fiscal year 2017, as well as prior GAO work.  We also interviewed officials from SBA and the four departments listed  above. Finally, to obtain stakeholder views on the extent to which SBA’s  revised scorecard methodology might encourage agencies to expand  small business procurement opportunities, we interviewed  representatives from the selected departments, as well as officials from  three groups representing the interests of small businesses. Appendix I  describes our objectives, scope, and methodology in greater detail.

We conducted this performance audit from January 2018 to September  2018 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

According to SBA, the purposes of the scorecard program are to monitor  government-wide performance in meeting small business contracting  goals and to provide accurate and transparent information through the  public reporting of small business procurement data for individual  agencies and government-wide. SBA uses its scorecard methodology to  calculate a numeric score for each agency annually. SBA then converts  those numeric scores to letter grades on an A+ through F scale. Each  year, SBA negotiates small business prime contracting goals with each  federal agency with procurement authority such that, in the aggregate, the  federal government meets its overall 23-percent goal for the percentage  of prime contract dollars awarded to small businesses. In setting annual  agency goals, SBA considers prior-year achievement and other factors.

In addition to an overall prime contracting goal, Congress also  established statutory contracting goals for various socioeconomic  subcategories of small businesses. These small business subcategories  are small disadvantaged businesses, women-owned small businesses,  service-disabled veteran-owned small businesses, and businesses  located in Historically Underutilized Business Zones (HUBZone). SBA  does not negotiate agency-specific goals for prime contracting and  subcontracting achievement within each small business socioeconomic  subcategory. Instead, each agency’s goal is the same as the government- wide goals. Prime contracting and subcontracting achievement goals for  each subcategory are shown in table 1 below.

SBA uses two government-wide data systems maintained by the General  Services Administration (GSA) to measure agencies’ small business  contracting activity. SBA uses the Federal Procurement Data System- Next Generation (FPDS-NG) to calculate agencies’ prime contracting  awards to small businesses. Federal agencies are required to report to  FPDS-NG all contracts whose estimated value is $3,500 or more, and  FPDS-NG also records whether the contract has gone to a small  business. GSA requires that agencies annually certify the accuracy of  data submitted. To measure subcontracting, SBA uses the Electronic  Subcontracting Reporting System (eSRS), which captures data on  spending on first-tier subcontracts, including spending directed to small  businesses. Prime contractors that hold one or more government  contracts totaling more than $700,000 are required to report their small  business subcontracting activity in eSRS.

In 1978 Congress amended the Small Business Act to require that all  federal agencies with procurement powers establish an Office of Small  and Disadvantaged Business Utilization (OSDBU). These offices are  intended to advocate for small businesses in procurement and contracting  processes, and thus work with agencies to achieve contracting goals.  OSDBUs have multiple functions and duties that are codified in section  15(k) of the Small Business Act, as amended. In addition to their agency  responsibilities, OSDBU directors serve with the SBA administrator or a  designee on the Small Business Procurement Advisory Council, which  was established in 1994. The council’s duties include identifying best  practices for maximizing small business utilization in federal contracting  and conducting peer reviews of each OSDBU to determine compliance  with section 15(k). SBA has included the results of this peer review as  part of its scorecard calculations for several years.

SBA Made Several Revisions to the Scorecard for Fiscal Year 2017 but Has Not Completed a Plan to Evaluate Those Changes

Scorecard Revisions Focused Largely on Mandated Changes

SBA revised the scorecard methodology prior to fiscal year 2017 to make  it consistent with changes required by the 2016 NDAA. Specifically, SBA  reduced the proportion of the total scorecard results related to prime  contracting performance from 80 percent to 50 percent and added an  element to calculate changes in the number of small business prime  contractors compared to the prior year. SBA officials said they  considered, but did not add, a scorecard element that calculated changes  in the number of small business subcontractors, which the 2016 NDAA  required to be included if data were available. Officials said that unlike  prime contracting data, which are validated by agencies, subcontracting  data are recorded by the prime contractor and are based on contracting  plans and not obligated federal funds. As a result, SBA officials said they  determined that data were not available to implement this change.

SBA also made other changes to the scorecard methodology, as the  agency was permitted to do under the 2016 NDAA. SBA adjusted the  weights of other scorecard elements, increasing subcontracting  performance from 10 percent to 20 percent of the total scorecard result  and increasing the peer review evaluation element from 10 percent to 20  percent. SBA also established that the new statutorily required element to  assess changes in the number of prime contractors would be weighted at  10 percent. (See fig. 1 for a summary of revisions to the scorecard  methodology.) Officials said they increased the subcontracting weight  because it was an increasingly important area of small business  procurement activity.

In addition, SBA officials and other Small Business Procurement Advisory  Council members revised the peer review evaluation methodology in an  effort to facilitate a more in-depth review of agencies’ compliance with  section 15(k) requirements. SBA included the results from this new peer  review process in its revised scorecard methodology. Specifically, the  council changed the peer review process in an effort to have peer  reviewers make compliance determinations for categories that directly  corresponded to the individual subparts of section 15(k). The prior peer  review process asked reviewers to assign scores in seven areas, which  the process termed “success factors.” For the fiscal year 2017  scorecard, SBA asked peer reviewers to assess and provide scores for  18 of the 21 individual subparts. Categories for the three remaining  15(k) subparts were incorporated starting with the fiscal year 2018  scorecard methodology.

SBA officials said members of the Small Business Procurement Advisory  Council were active participants in determining the revisions to the  scorecard methodology. For example, SBA officials said the council  members gave input on proposed revisions and recommended changes  prior to the adoption of the new scorecard methodology. OSDBU directors  also discussed potential methodological revisions in meetings of the  Federal OSDBU Directors Interagency Council. SBA officials said the  OSDBU directors’ input was incorporated into SBA’s revised scorecard  guidance and, as a result, the criteria within the scorecard were more  robust. Officials we interviewed from SBA and other agencies said the  adopted scorecard revisions were the result of a consensus among Small  Business Procurement Advisory Council members, although no formal  votes were taken. Revisions to the scorecard methodology were outlined  in a memorandum circulated to agencies in August 2016, about 8 weeks  before the start of fiscal year 2017. SBA officials said that many agencies  were tracking their progress toward goals using the revised methodology  before results were issued. Agencies also had an opportunity to review  preliminary scorecard results for fiscal year 2017 before the official  scorecard results were published in May 2018.

Fiscal Year 2017 Scorecard Outcomes Were Similar to Those of Prior Years

Scorecard results under the revised methodology were similar to those of  prior years. For example, in fiscal year 2017, the distribution of agencies’  letter grade results was similar to those of fiscal years 2014 through 2016,  with between 19 and 21 of the 24 scored agencies achieving at least an A  grade each year (see table 2).

Prime contracting achievement. Agencies’ performance in small  business prime contracting was similar in fiscal year 2017 and fiscal year  2016 (see table 3). In both years, 18 of 24 agencies met their overall  prime contracting goals. In fiscal year 2017, 15 of 24 agencies met at  least three of the four small business subcategory goals—one fewer than  in fiscal year 2016.

Subcontracting achievement. In fiscal year 2017, 15 of 24 agencies  met their subcontracting goals compared to 16 of 24 in the prior year.  However, among the small business subcategories, more agencies met  at least three subcategory goals in 2017 (14 agencies) than in fiscal year  2016 (10 agencies) (see table 4).

Peer review evaluations element. The fiscal year 2017 government- wide score for the peer review of section 15(k) compliance (a score of  19.25 out of a maximum 20.00) was nearly identical to the government- wide score for fiscal year 2016, once we adjusted for changes in the  scoring scale between the 2 years. The government-wide score in fiscal  year 2016 was 9.60 out of 10, which equates to 19.20 on a 20-point  scale.

Number of small business prime contractors. The overall number of  small business prime contractors declined between fiscal years 2016 and  2017. The number of prime contractors overall decreased from 120,009 in  fiscal year 2016 to 117,480 in fiscal year 2017, a decrease of  approximately 2 percent. However, the 24 agencies, in aggregate, had  more small business prime contractors in three of the four small business  subcategories in fiscal year 2017 than in the prior year (see table 5).

Comparison with prior scorecard weighting formula. We found that  agencies’ numerical scores for fiscal year 2017 were generally lower  under the revised scorecard methodology than they would have been  under the fiscal year 2016 methodology’s weighting of scorecard  elements. Twenty-two of 24 agencies had a lower score than they would  have had under the prior methodology’s weighting. The revised  methodology adjusted the weight of multiple scorecard elements, and  there are a variety of reasons why an agency might have received a lower  score than under the fiscal year 2016 methodology’s weighting. However,  reducing the weight for prime contracting achievement under the revised  methodology could explain at least part of the lower score for 21 of the 22  agencies. The overall median score for fiscal year 2017 was about 7  points lower than it would have been under the weighting formula used in  fiscal year 2016. (The median score for fiscal year 2017 scorecards was  111 and would have been 118 under the prior methodology’s weighting  formula.)

SBA Said It Was Preparing but Had Not Completed a Plan to Evaluate the Effects of Scorecard Revisions

In June 2018, SBA officials told us they were not preparing a plan for  evaluating the effects of scorecard revisions because they thought such a  plan would be premature. At that time, SBA officials said they had  identified some aspects of the revised methodology for further review,  including two issues related to the peer review evaluations—the peer  review scoring scale and whether agencies believed SBA’s requests for  supporting information were reasonable. In July 2018, however, SBA  officials said that, in response to our preliminary findings, they had begun  to develop a plan for evaluating the revised scorecard methodology’s  effects, if any, on meeting the government-wide procurement goals. The  officials did not provide us a draft plan or details about the plan. They said  they expected to complete the evaluation plan by October 2018 and to  complete the evaluation itself by the end of December 2018.

Federal internal control standards state that management should use  quality information to achieve the entity’s objectives, such as those in an  agency’s strategic plan. These standards also call for management to  design control activities to achieve goals and respond to risks—for  example, activities to monitor performance measures and indicators.  SBA’s strategic plan includes an objective to ensure federal contract and  innovation set-aside goals are met or exceeded. The agency uses  scorecard results to measure progress toward meeting or exceeding the  statutory goal of 23 percent for overall small business prime contracting.  Scorecard results are also used to measure progress toward other goals  for the small business socioeconomic subcategories.

We have previously identified key attributes of effective program  evaluation design, which include the following:  clear criteria for making comparisons that would lead to strong,  defensible evaluation conclusions; an established evaluation scope that would ensure that the evaluation  is tied to its research questions, effectively defines the subject matter  to be assessed, and can be completed in a timely fashion to meet  reporting deadlines;  clear and specific research evaluation questions that use terms that  can be readily defined and measured; and  carefully thought-out data and analysis choices, which can enhance  the quality, credibility, and usefulness of the evaluation.

A comprehensive evaluation of revisions to the scorecard that includes  the key attributes outlined above could aid SBA officials in determining  whether the revised scorecard provides better information and whether  the scorecard revisions are designed and implemented appropriately.  Such an evaluation also could assist SBA in understanding whether the  scorecard revisions may contribute to maximizing contract dollars  awarded to small businesses, which is one of the goals in SBA’s strategic  plan. In addition, the 2016 NDAA requires that SBA report to Congress by  March 31, 2019, about changes stemming from the revised methodology  and recommend whether the scorecard program should continue or be  further modified. Such an evaluation could also be used by SBA to inform  its report to Congress and future decisions about the scorecard  methodology and program.

SBA Uses Available Procurement Data to Calculate Scorecard Outcomes, but the Process for Producing Scorecards Has Weaknesses

Subcontracting Data Have Known Limitations That May Affect the Reliability of Scorecard Calculations

The two data systems SBA uses to measure agencies’ small business  contracting activity—FPDS-NG and eSRS—are the best available  sources of procurement data for calculating scorecard results, according  to SBA. However, eSRS has limitations that agency officials cited and  that we have previously identified that could hinder the reliability of  scorecard results on subcontracting. Federal law prohibits SBA from  requiring agencies to use alternative data collection methods for the  purposes of the scorecard calculations. GSA intends to replace both  systems as part of an initiative to consolidate the functions of several  existing data systems, according to GSA documents. As we reported in  2014, this new system is intended to better link prime contracting and  subcontracting data.

Agency officials we interviewed said eSRS has limitations that make it  challenging to verify the accuracy of reported subcontracting activity, and  we also have identified eSRS limitations in our prior work. Prime  contractors are responsible for reporting their subcontracting activity to  the federal government, and the self-reported nature of these data is a  limitation that could hamper the accuracy of eSRS data, agency officials  said. Although prime contractors generally are required to submit a plan  describing planned subcontracting activity, officials explained that eSRS  did not provide a method to allow agency officials to verify that actual  subcontracting activity matched the levels described in prime contractors’  plans. In addition, not all prime contractors are required to file  subcontracting plans. Exceptions to the requirement include, for example,  when the prime contract is for goods or services worth $700,000 or less  or if the prime contractor is exempt. Small business prime contractors  are one example of an exempt group that is not required to prepare  subcontracting plans.

SBA officials added that measuring subcontracting activity also is  challenging because there are no federal funds obligated for  subcontracts. Therefore, the federal government does not have a verified  record of who performed subcontracting work and the amount paid. In  addition, our previous work has found that eSRS was not designed to  provide a list of subcontractors associated with a particular prime contract  and that linking small business subcontractors to prime contracts when  there is a subcontracting plan that pertains to multiple contracts is  especially difficult.

In addition, our previous work has identified some limitations with FPDS- NG focused on specific agencies and small business programs, although  we have not more broadly assessed the reliability of the FPDS-NG data  fields that SBA uses to compile scorecard results. For example, we found  mismatches between certain accounting records from the Department of  Veterans Affairs and data captured in FPDS-NG, and we identified  challenges in using FPDS-NG data to monitor the eligibility of Alaska  Native Corporations for certain small business contracts available to small  disadvantaged businesses. However, officials from SBA and two  departments we interviewed for this work said prime contracting data in  FPDS-NG generally do not have the same weaknesses they identified  with subcontracting data in eSRS.

Errors in Published Scorecard Results Weaken Reliability and Perceived Integrity of Scorecard Program

Scorecard results originally published by SBA on May 22, 2018,  contained errors, including one agency scorecard published with an  incorrect letter grade. SBA officials said they discovered the publication  errors within approximately 2 days of publication and published corrected  versions. However, these corrections occurred after SBA issued a public  announcement highlighting the new results, and interested parties may  have downloaded erroneous results prior to the corrected versions being  posted on SBA’s website.

We identified errors from SBA’s originally published scorecards  independent of SBA’s determination that the agency had published  scorecards containing errors. The errors we and SBA identified were  concentrated in the scorecard for the Department of Education and the  government-wide scorecard:

The scorecard for the Department of Education showed an incorrect  letter grade of A+, rather than the correct grade of A. The published  scorecard also showed an incorrect overall numeric score.

The Department of Education’s score for the peer review component  of the scorecard was incorrect.

The government-wide scorecard showed incorrect scores for changes  in the number of women-owned small business contractors and the  number of service-disabled veteran-owned small business  contractors.

SBA did not initially document on the corrected scorecards how they had  been changed from the original scorecards. However, SBA later added  documentation that the scorecards for the Department of Education and  government-wide results had been corrected. SBA took this step after we  inquired about the absence of documentation about revisions that had  been made to the fiscal year 2017 scorecards.

SBA officials said they performed accurate calculations for determining  agencies’ performance and that inaccuracies in the published scorecards  were the result of transcription errors associated with formatting the  results for publication. Officials said SBA used new software to publish  the fiscal year 2017 scorecards so that they could be accessible to  visually impaired readers. Making the scorecards more accessible  required some additional steps and at times required manual data entry  due to limitations in SBA’s software. These additional steps resulted in  errors, officials said. One set of errors—the inaccurate government-wide  scores for changes in the number of women-owned small business  contractors and the number of service-disabled veteran-owned small  business contractors—canceled each other out and did not lead to  erroneous overall scorecard results. SBA officials said they review the  scorecard data and calculations before they are prepared for publication.  However, the agency does not have a process to review formatted  scorecards prior to publication to confirm that the version for publication  matches actual calculations. Agency officials said they believed that such  a process was not necessary. Additionally, agency officials said SBA has  instituted a process to update previously issued scorecards to make them  accessible for the visually impaired. SBA officials said they intend to  review the accuracy of these updated scorecards for characteristics such  as accurate letter grades as agency resources permit.

Both the Office of Management and Budget and SBA have issued policies  related to transparency and integrity of government data. The Office of  Management and Budget has issued government-wide guidance on  transparency in sharing government data and instructed federal agencies  to develop their own policies. SBA’s policy on information quality says  the policy is intended, in part, to ensure the integrity of information SBA  disseminates. SBA’s policy also says the agency should have full,  accurate, transparent documentation and should identify and disclose to  users any error sources affecting data quality. In addition, federal internal  control standards cite the need for management to design controls— including controls over information processing—to achieve objectives.

Errors in the published scorecards may impair the other agencies’ or  Congress’s access to quality information to make informed decisions and  evaluate an agency’s performance in meeting small business goals. The  scorecard errors that we and SBA identified after publication—and the  lack of any indicator that scorecards had been corrected—also may  undermine confidence in the integrity and transparency of the scorecard  data.

Agency Officials and Other Stakeholders Expected the Revised Scorecard to Have Little Impact on Small Business Opportunities

Agency officials and representatives of small business groups we spoke  with generally expected the revised scorecard methodology for fiscal year  2017 to have little impact on small business procurement opportunities.  OSDBU officials in the four agencies we interviewed said their offices, in  general, are not altering existing efforts at advocating for small business  opportunities as a result of scorecard revisions. Some agency officials  also said they would need additional years of scorecard data before  making any changes to their efforts or reassessing how their priorities  align with the revised scorecard’s formula. However, officials from one  agency said they updated their agency’s internal monitoring of  subcontracting activity as a result of the revised scorecard methodology’s  increased emphasis on subcontracting measures. Officials said they  updated the monitoring process so the agency would place more  emphasis on small business subcontracting activity. Officials said the  change to this agency’s internal monitoring process took effect for fiscal  year 2018.

Officials from three of the four federal departments and representatives  from the three small business groups we interviewed said they had not  seen any changes in opportunities for small business prime contracting  as a result of the scorecard’s methodological changes. Instead,  representatives from three small business groups and officials from two  departments said any changes in prime contracting opportunities that  might have occurred would be influenced by other government-wide  procurement initiatives. Specifically, representatives from the three small  business groups said the federal government’s emphasis on “category  management” was resulting in fewer prime contracts available to all  government contractors, including small business contractors. Under the  category management initiative, the federal government groups  commonly purchased goods and services into categories to streamline  procurement processes with the goal of eliminating redundancies and  reducing costs. However, representatives of small business groups said  these policies result in fewer contract awards and opportunities for small  businesses. Representatives from the three small business groups said  that the new scorecard element that calculates the annual changes in the  number of small business contractors could help highlight the effects of  these prime contracting trends on procurement opportunities.

According to agency officials and small business representatives,  subcontracting opportunities are also unlikely to be impacted by the  revised scorecard methodology, which increased the weight of  subcontracting performance. Officials from two of the four departments  we interviewed told us that their agencies have stable purchasing  patterns and that subcontracting activity is not likely to change as a result  of scorecard revisions. Representatives from two of the three small  business groups said the influence of the scorecard revisions in  incentivizing agencies to focus on subcontracting opportunities is limited  by the reliability of available subcontracting data, discussed previously.  For example, one agency told us that the shift from prime to  subcontracting performance reduces the agency’s ability to influence  scorecard outcomes because the agency has no means of validating the  subcontracting data that are recorded. Similarly, representatives from two  of the three small business groups said that because the data on  subcontracting are entered by the prime contractors at the time of  proposed contracting rather than confirmed contracting, the data do not  include verification of subcontracting activity and therefore might not be  an accurate measure of subcontracting activity.

Representatives from agencies and small business groups said the  scorecard program has generally played a role in drawing attention to  agencies’ performance in identifying small business procurement  opportunities. For example, SBA officials said the scorecard results  provide public information about how well the government performed  overall in providing small business procurement opportunities and help to  ensure that all agencies are contributing toward those goals. Officials at  one agency told us that the scorecard was an important factor in driving  internal goals and opportunities for small businesses. Another agency  said that while it had been reaching its overall prime contracting goal, its  performance in certain small business subcategories was falling short of  goals. As a result, the agency has directed additional outreach efforts to  those types of small businesses. In addition, representatives of all three  small business groups said because results are public, the scorecard has  created additional pressure on agencies to meet procurement goals.

SBA uses its scorecard program to monitor federal agencies’ compliance  with goals set by Congress to promote small business participation in  federal contracting, and SBA has identified having agencies meet or  exceed those participation benchmarks as one of its agency-wide goals in  its strategic plan. The effects of recent changes to the scorecard and their  potential benefits for improving federal contracting opportunities for small  businesses are uncertain. SBA recently began to develop a plan for  evaluating whether or how changes to the scorecard might facilitate  SBA’s ability to meet government-wide procurement goals. Completing  such an evaluation and making sure the evaluation plan is aligned with  key attributes for effective evaluations could help SBA management: determine whether the revised scorecard provides quality  information—consistent with federal internal control standards—and  whether it helps meet the agency’s strategic goals;  fully address whether the revisions are effective in measuring and  creating small business procurement opportunities; and make a well-supported recommendation about whether to continue or  modify the scorecard program. Congress required that SBA  recommend by March 31, 2019, whether to continue or modify the  scorecard program.

In addition, the scorecard appears to have played a role in drawing  attention to agencies’ performance in identifying small business  procurement opportunities. However, there were errors in the initial fiscal  year 2017 scorecards published on SBA’s website, and SBA did not  initially take steps to notify the public after it made corrections. SBA  officials said that SBA does not have a process to ensure that published  scorecard results are accurate. Errors in the published scorecards and a  lack of timely disclosure about corrections may impair other agencies’ or  Congress’s access to quality information to make informed decisions.

We are making the following two recommendations to SBA:  The SBA Administrator or her designee should complete the design and  implementation of a comprehensive evaluation of the Small Business  Procurement Scorecard aligned with key attributes of effective program  evaluations to assess the effectiveness of the revised scorecard in  measuring agency performance and promoting small business  procurement opportunities. (Recommendation 1)

The SBA Administrator or her designee should institute a process to  review Small Business Procurement Scorecards for accuracy prior to  publication and a mechanism for publicly identifying when issued  scorecards have been revised. (Recommendation 2)

We provided a draft of this report to SBA for review and comment. In  written comments, reproduced in appendix II, SBA generally agreed with  both of our recommendations.

Regarding our recommendation that SBA design and implement an  evaluation of the revised scorecard methodology, SBA said it planned to  evaluate the changes to the scorecard methodology mandated by the  2016 NDAA. As discussed in our report, in revising the scorecard, SBA  also made other changes not specifically mandated by the 2016 NDAA,  such as increasing the emphasis on small business subcontracting  activity and incorporating a revised peer review process to facilitate a  more in-depth review of agencies’ compliance with section 15(k)  requirements. As stated in our report, we recommend that SBA plan and  implement an evaluation of all aspects of the revised scorecard  methodology. SBA also indicated that it will not complete the evaluation  until after it has validated data for the fiscal year 2018 procurement  scorecard. We note that SBA can prepare an evaluation plan and begin to  consider potential evaluation findings using available scorecard data from  fiscal year 2017. We also note that our recommendation states that SBA’s  evaluation plan should be aligned with the key attributes of effective  evaluation design.

Regarding our recommendation that SBA institute a process to review  scorecards for accuracy prior to publication and a mechanism for publicly  identifying when issued scorecards have been revised, SBA said it had  taken several steps to revise the processes for publishing accurate  scorecard results, including adding steps to compare the prepared  scorecard documents to source documents prior to publication and to  annotate any score corrections that are made to published scorecards.  While we have not yet had the opportunity to assess SBA’s actions, the  steps SBA describes in response to our recommendation could improve  other agencies’ or Congress’s access to quality information.

We will send copies to the Administrator of SBA and other interested  parties. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  III.

Appendix I: Objectives, Scope, and Methodology

This report describes (1) revisions to the Small Business Procurement  Scorecard (scorecard) methodology for fiscal year 2017 and results of the  fiscal year 2017 scorecard, as well as the extent to which the Small  Business Administration (SBA) plans to evaluate the effects of revisions;  (2) the extent to which SBA’s revised scorecard methodology uses  relevant and reliable information and SBA publishes accurate scorecards;  and (3) views of selected federal agencies and industry stakeholders on  the extent to which SBA’s revised scorecard methodology may encourage  agencies to expand small business procurement opportunities.

To examine the changes SBA made to the Small Business Procurement  Scorecard and the rationale for these changes, we reviewed relevant  documents, including the National Defense Authorization Act for Fiscal  Year 2016, SBA’s descriptions of the prior and revised scorecard  methodology, and revised peer review guidance used for the scorecard  element that assesses compliance with section 15(k) of the Small  Business Act. We also interviewed officials from SBA and four other  agencies about the revisions to the scorecard calculation methodology,  the peer review guidance, the process for providing input on scorecard  revisions, and how revisions were implemented. The four agencies (the  Departments of Agriculture, Defense, Energy, and Homeland Security)  represented a judgmental, nongeneralizable sample of federal agencies  with procurement powers, selected based on small business procurement  volume, recent improvement in scorecard results, and level of  participation in discussions with SBA and other agencies about potential  changes to the scorecard. We also interviewed SBA officials about their  plans to evaluate the effects of scorecard revisions on small business  procurement opportunities and about their plans, if any, to evaluate the  revised scorecard. In addition, we reviewed federal internal control  standards and GAO’s key attributes for designing effective evaluations.

We analyzed the distribution of agencies’ letter grade results (A+, A, B, C,  D, and F) from the fiscal year 2017 scorecard and compared this  distribution to fiscal years 2014 through 2016, which used a different  scorecard methodology. We also reviewed the distribution of results of  fiscal year 2017 individual scorecard elements—specifically, results of  prime contracting achievement, subcontracting achievement, and peer  reviews—and compared this distribution to results for fiscal year 2016.

We compared agencies’ prime contracting and subcontracting  performance against their small business procurement goals for fiscal  years 2016 and 2017. To compare peer review results across years, we  made adjustments to account for changes in the value of peer review  results (raised from 10 points to 20 points from fiscal years 2016 to 2017).  To adjust for this difference, we doubled the value of fiscal year 2016  scores to put both years’ scores on a 20-point scale. Finally, we  compared actual fiscal year 2017 scorecard results to the results if SBA  had used the 2016 scorecard weighting. To do this, we increased the  weighting of fiscal year 2017 prime contracting results from 50 percent to  80 percent of each agency’s total scorecard grade, decreased the weight  of subcontracting results from 20 percent to 10 percent, and decreased  the weight of peer review results from 20 percent to 10 percent. We also  excluded results from the new scorecard element calculating changes in  the number of small business contractors, which was not part of the 2016  methodology.

To examine the extent to which SBA’s revised scorecard methodology  considers relevant and reliable information, we interviewed officials from  SBA and the Departments of Agriculture, Defense, Energy, and  Homeland Security. We reviewed documents describing the prior and  revised scorecard methodology. We discussed limitations, if any, in the  electronic data systems that capture government-wide data on prime  contracting and subcontracting (which SBA uses to calculate those  respective scorecard elements). We also reviewed our prior work that  assessed these data systems. To assess the data reliability of the  published scorecards, we reviewed them for obvious errors and  interviewed SBA officials about the cause of errors we identified. We  found the scorecards to be reliable for analyzing scorecard results for  fiscal year 2017. We also compared SBA’s revised scorecard  methodology against the agency’s policies on information quality and  against GAO’s standards for internal control in the federal government.

To collect views on the extent to which SBA’s revised scorecard  methodology may encourage agencies to expand small business  procurement opportunities, we interviewed officials from SBA and the four  selected departments cited above, as well as representatives from three  organizations representing the interests of small businesses. These three  organizations were selected to represent a mix of small business types:  one (The American Small Business Chamber of Commerce) represented  all types of small businesses; one (Women Impacting Public Policy)  represented a small business socioeconomic subcategory with a 5  percent goal for prime contracting and subcontracting (as a percentage of  total prime contracting and subcontracting); and one (The Task Force for  Veterans’ Entrepreneurship, also known as Vet-Force) represented a  small business subcategory with a 3 percent goal for prime contracting  and subcontracting.

Appendix II: Comments from the Small Business Administration

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Andy Pauline (Assistant  Director), Steve Robblee (Analyst in Charge), William Chatlos, Holly  Hobbs, Marc Molino, Jessica Sandler, and Jennifer Schwartz made key  contributions to this report.

The federal government is required to allocate at least 23% of its direct contract spending to small businesses. The Small Business Administration's annual scorecards track progress toward that goal.

SBA changed how it calculates scores for fiscal year 2017 but didn't initially plan to evaluate the effect of the changes.

We recommended SBA evaluate its new scoring methodology to assess how effectively the scorecards measure agency performance and promote contracting opportunities for small businesses. SBA could use the evaluation results to inform Congress in its upcoming required report.