Science, Technology, Engineering, and Mathematics Education: Actions Needed to Better Assess the Federal Investment

Education programs in STEM fields are intended to enhance the nation's global competitiveness. GAO reported in 2012 that there were more than 200 federal STEM education programs in fiscal year 2010. Since then, this portfolio of programs has changed. GAO was asked to review the landscape of federal STEM education programs.

This report examines (1) how the federal investment in STEM education programs changed from 2010 to 2016, and (2) the extent to which the STEM education portfolio has been assessed.To answer these questions, GAO administered a web-based questionnaire to all federal STEM education programs funded in fiscal year 2016 and analyzed the results. GAO also reviewed relevant federal laws and agency documents, examined the implementation of relevant assessment requirements, and interviewed officials from relevant federal agencies.

The federal investment in science, technology, engineering, and mathematics (STEM) education programs remained relatively stable from fiscal years 2010 to 2016, although the number of programs declined from 209 to 163 (see figure). While agencies reported that many of the same STEM education programs existed during this time period, the portfolio underwent various changes, including program consolidations, creations, and terminations. Nearly all STEM education programs in fiscal year 2016 overlapped to some degree with at least one other program in that they offered similar services to similar groups in similar STEM fields to achieve similar objectives. The Committee on STEM Education, an interagency body responsible for implementing the federal STEM education strategic plan, reported it managed this overlap through coordination with agencies administering these programs.

The Committee on STEM Education has not fully met its responsibilities to assess the federal STEM education portfolio. Specifically, the Committee has not reviewed programs' performance assessments, as required by its authorizing charter, nor has it documented those assessments in its inventory, as required by law. Such efforts could encourage the use of evidenced-based practices across the portfolio—a key national goal of the STEM education strategic plan. These efforts could also enhance public awareness of the administering agencies' efforts to assess programs' performance. In addition, the Committee has not reported the participation rates of underrepresented groups in federal STEM education programs, as required by law. By reporting this information, the Committee could better assess whether programs are broadening access to groups historically underrepresented in STEM fields—another key goal of the strategic plan.

GAO is making four recommendations, including three to the Committee on STEM Education to review performance assessments of STEM education programs, document those assessments, and report programs' participation rates of underrepresented groups. The Committee on STEM Education agreed with GAO's recommendations.

Education programs in science, technology, engineering, and  mathematics (STEM) play an important role by preparing students for  careers in STEM fields and are intended to enhance the nation’s global  competitiveness. Various levels of government foster STEM education, as  do other stakeholders, including institutions of higher education,  professional and scientific societies, and private industries. In 2012, we  reported that more than 200 federal STEM education programs across 13  different agencies spent more than $3 billion in fiscal year 2010. We also  found a high degree of overlap among those programs and made several  recommendations to improve their coordination and management. You  asked us to review the current landscape of federal STEM education  programs. This report examines: 1) how the federal investment in STEM  education programs changed from 2010 to 2016, and 2) to what extent  the STEM education portfolio has been assessed.

To gather information on how the federal investment in STEM education  changed from 2010 to 2016, we administered a web-based questionnaire  to STEM education programs funded in fiscal year 2016. We identified  these programs based on input from the 13 agencies that administered  the STEM education programs we previously reported for fiscal year  2010. Program officials provided information on all 163 programs funded  in fiscal year 2016 that we determined met our definition of a STEM  education program. For more information on our definition, program  identification, and data collection, see appendix I. The questionnaire  solicited information on the programs’ objectives, groups served, services  provided, and spending, among other things, and we analyzed program  officials’ responses. We used this information to gauge the level of  overlap among programs funded in fiscal year 2016. We also compared  the results to similar data previously collected for programs funded in  fiscal year 2010.

To understand the extent to which the federal programs within the STEM  education portfolio had been assessed, we reviewed the requirements in  the America COMPETES Reauthorization Act of 2010 (COMPETES Act)  regarding federal STEM education program assessment. Subsequently,  we examined the information produced to fulfill these requirements,  including a strategic plan and annual progress reports. We sent our initial  determination of whether these COMPETES Act requirements had been  fulfilled to the entities responsible for implementing them and solicited  their views. We obtained a response from the Office of Science and  Technology Policy as to whether the requirements had been fulfilled and  considered any additional information the agencies provided in finalizing  our analysis. In addition, we reviewed the assessment responsibilities of  the Committee on STEM Education as specified in its authorizing charter  and examined its efforts to meet those responsibilities. Further, we  conducted a limited review to test whether program assessment  information was routinely contained within agency budget documents. In  response to our questionnaire, program officials reported that 73 of the  163 STEM education programs had been evaluated since fiscal year  2010. From those 73, we randomly selected a nongeneralizable sample  of 10 STEM education programs—one from each agency that reported it  had evaluated at least one of its STEM education programs recently. We  then searched for supporting information within the agencies’ budget  documents.

In addition, we interviewed officials responsible for coordinating federal  STEM education efforts from the National Science Foundation and the  Office of Science and Technology Policy. We also interviewed officials  from the Office of Management and Budget and other federal agencies  leading strategic interagency efforts on STEM education, including the  National Aeronautics and Space Administration, the Departments of  Education, Energy, Health and Human Services, and the Smithsonian  Institution. Further, to inform both of our objectives, we reviewed relevant  federal laws and regulations as well as reports on STEM education.

We conducted this performance audit from October 2016 to March 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The term “STEM education” includes educational activities across all  grade levels—from preschool to graduate school. STEM education  programs have a variety of primary objectives, which include preparing  students for STEM coursework, providing postsecondary students with  grants or fellowships in STEM fields, and improving STEM teacher  training (see appendix I for our definition of STEM education programs).

Federal STEM education programs have been created in two ways— either by law or by federal agencies under their statutory authorities.

We previously reported that most federal STEM education programs  overlapped to some degree with at least one other program, in that they  offered similar services to similar groups in similar STEM fields to achieve  similar objectives (see sidebar for definition of overlap).

Duplication occurs when two or more  agencies or programs are engaged in  the same activities or provide the same  services to the same beneficiaries.

Overlap occurs when multiple agencies  or programs have similar goals, engage  in similar activities or strategies to  achieve their goals, or aim to serve  similar beneficiaries.  Fragmentation refers to those  circumstances in which more than  one federal agency (or more than one  organization within an agency) is  involved in the same broad area of  national need and opportunities exist  to improve service delivery.

Although those programs may not be duplicative, we reported that they  were similar enough that they needed to be well coordinated and guided  by a robust strategic plan. And, through its strategic planning and other  coordination efforts, the Office of Science and Technology Policy and the  National Science and Technology Council implemented our  recommendations to work with agencies to better align their activities with  a government-wide strategy; develop a plan for sustained coordination;  identify programs for potential consolidation or elimination; and assist  agencies in determining how to better evaluate their programs.

America COMPETES Reauthorization Act of 2010

Enacted in 2007, the America COMPETES Act authorized several  programs to promote STEM education. The America COMPETES  Reauthorization Act of 2010 (COMPETES Act) reauthorized the America  COMPETES Act and addresses coordination and oversight issues,  including those associated with the coordination and potential duplication  of federal STEM education efforts. The COMPETES Act required the  Director of the Office of Science and Technology Policy to establish,  under the National Science and Technology Council, the Committee on  STEM Education to serve as the interagency coordination body for STEM  education in the federal government (see fig. 1).

In May 2013, the Committee on STEM Education issued a 5-year  Strategic Plan for federal STEM education efforts, as required by the  COMPETES Act. To improve collaboration across the portfolio, the  Strategic Plan identified five priority investment areas and two  coordination objectives, specifying national goals for each (see fig. 2).

The COMPETES Act also requires that the Committee create, and  periodically update, an inventory of federal STEM education programs  that includes documentation of program assessments and the  participation rates of women, underrepresented minorities, and persons in  rural areas.

In addition, the COMPETES Act requires that the Office of Science and  Technology Policy publish annual reports on coordinating federal STEM  education efforts. The law mandates that these reports include specific  information, such as: a description of each federal agency’s STEM education programs  funded in the previous and current fiscal years, as well as those  proposed under the President’s budget request;  the levels of funding for each participating federal agency’s programs  described above; an evaluation of the levels of duplication and fragmentation of the  programs described above; and a description of the progress made implementing the Strategic Plan,  including a description of the outcome of any program assessments  completed in the previous year, and any changes made to the  Strategic Plan since the previous annual report.

In January 2017, the President signed into law the American Innovation  and Competitiveness Act, which, among other things, amended certain  provisions of the COMPETES Act. The Act added some requirements  for both the Office of Science and Technology Policy and the Committee  on STEM Education. For example, it created new mandates for the  Committee to:  review the measures federal agencies use to evaluate their STEM  education programs, and make recommendations for reforming, terminating, or consolidating  the federal STEM portfolio.

Any such recommendations for an upcoming fiscal year are to be  included in the Office of Science and Technology Policy’s annual report.

Cross-agency Priority Goal on STEM Education

In 2014, the Office of Management and Budget, in consultation with the  federal agencies that administer STEM education programs, established  STEM education as a cross-agency priority goal. The Office of Science  and Technology Policy and the National Science Foundation led the  oversight and management of this goal, and as part of this work, goal  leaders from these agencies identified milestones that aligned with the  Strategic Plan’s priority investment areas and coordination objectives  (see fig. 2). For example, goal leaders reported progress toward meeting  key milestones associated with improving STEM instruction. In 2017, to  ensure alignment with the current administration’s priorities, the Office of  Management and Budget removed the priority status of all cross-agency  priority goals, including STEM education; this ended the required public  issuance of quarterly priority goal reports. The STEM Education goal’s  final quarterly progress report was issued at the end of fiscal year 2016.

Other Data and Transparency Requirements

Other government-wide efforts are underway to improve the transparency  around federal programs in general. These efforts are not directed at the  STEM education programs specifically, but may assist in managing the  STEM education portfolio. The GPRA Modernization Act of 2010 requires  the Office of Management and Budget to present a coherent inventory of  all federal programs by making information about each federal program  available on a website. However, we previously reported that, because  agencies used different approaches to define their programs,  comparability of programs within and across agencies on this inventory  was limited. We recently identified a potential framework for the  development of a useful federal program inventory. The Office of  Management and Budget decided to postpone further development of the  inventory in order to coordinate with the implementation of related  requirements of the Digital Accountability and Transparency Act of  2014. Once fully implemented, this act is expected to expand the types  and transparency of public information on federal spending to make it  easier to track it to specific federal programs. The act requires  government-wide reporting on a greater variety of data related to federal  spending, such as budget and financial information, as well as tracking of  these data at multiple points in the federal spending lifecycle.

From 2010 to 2016, the Number of STEM Education Programs Decreased While Spending Remained Stable, and Most Programs Continued to Overlap

Agencies Reported Fewer STEM Education Programs and Relatively Stable Levels of Spending in Fiscal Year 2016 Compared to Fiscal Year 2010

Program officials from the 13 federal agencies that administer STEM  education programs reported a total of 163 STEM education programs in  fiscal year 2016, compared to 209 programs in fiscal year 2010. Three  agencies—the Department of Energy, the Department of Health and  Human Services, and the National Science Foundation—administered  more than half of all STEM education programs in fiscal years 2010 and  2016.

Despite collectively reporting fewer STEM education programs, program  officials responding to our questionnaire reported spending about the  same amount in fiscal year 2016 as they did in fiscal year 2010. In fiscal  year 2016, program officials reported spending about $2.9 billion on the  163 programs. Spending by individual programs ranged from about  $14,000 annually to hundreds of millions of dollars. The National  Science Foundation and the Department of Health and Human Services  programs account for about 60 percent of this spending. Figure 3  provides an agency-level summary of the number of programs and their  reported spending. Appendix II contains a complete list of the 163 STEM  education programs and their reported spending for fiscal year 2016.

While agencies reported many of the same STEM education programs in  fiscal years 2010 and 2016, the federal portfolio evolved in various ways.  About half of the 209 programs previously reported for fiscal year 2010  were reported again for fiscal year 2016—accounting for about two-thirds  (109 programs) of the fiscal year 2016 portfolio. The remaining third (54  programs) were newly reported for fiscal year 2016. (See appendix I for  more information on changes to the STEM portfolio between fiscal years  2010 and 2016.)

The portfolio underwent various changes from fiscal years 2010 to 2016,  including program consolidations, creations, and terminations.  According to leadership of the Committee on STEM Education, these  changes were due to many factors. One key factor is the STEM  Education Strategic Plan, which, among other things, calls for greater  efficiency and cohesion across federal STEM education programs. Other  factors include agencies’ individual priorities, including their mission and  budget, and congressional interest in specific programs. For example,  agencies reported:

Consolidations. Starting in 2014, for greater efficiency and cohesion,  the National Science Foundation consolidated a number of related  undergraduate STEM education programs, including STEM Talent  Expansion Programs, Transforming Undergrad Education in STEM,  and Nanotechnology Undergraduate Education in Engineering.

Creations. Department of Health and Human Services officials  reported administering 28 new STEM education programs. These  programs are housed in the Department’s National Institutes of  Health, which generally bases its funding decisions on scientific  opportunities and its own peer review process. One new program is  the Building Infrastructure Leading to Diversity Initiative. This program  supports undergraduate institutions in implementing and studying  approaches to engaging and retaining students from diverse  backgrounds in biomedical research.

Terminations. Department of Education officials reported that four  STEM education programs funded in fiscal year 2010 were terminated  before fiscal year 2016. One such program was the Women’s  Educational Equity program. Congress last funded this program in  fiscal year 2010.

Significant Overlap Continued to Exist Among STEM Education Programs, Although Programs May Differ in Meaningful Ways

Based on our analysis of questionnaire responses, nearly all STEM  education programs in fiscal year 2016 overlapped with at least one other  STEM education program, in that they offered at least one similar service  to at least one similar group in at least one similar STEM field to achieve  at least one similar objective (see text box). Similar levels of overlap  occurred among programs funded in fiscal year 2010.

Similarities Among Overlapping Federal Science, Technology, Engineering, and Mathematics (STEM) Education Programs  Similar Services  Many of the 163 STEM education programs provided similar services. To support students, most programs (143) provided research  opportunities, internships, mentorships, or career guidance. In addition, 110 programs supported short-term experiential learning  activities, and 99 programs supported long-term experiential learning activities. Short-term experiential learning activities include field  trips, guest speakers, workshops, and summer camps. Long-term experiential learning activities last throughout a semester in length  or longer. To support teachers, 77 programs provided curriculum development and 45 programs supported teacher in-service training,  professional development, or retention activities.

Similar Groups Intended to be Served  Many programs also provided services to similar groups, such as K-12 students, postsecondary students, K-12 teachers, and college  faculty. A majority of STEM programs reported primarily benefiting postsecondary students; specifically, 103 programs intended to  serve 4-year undergraduate students, 76 intended to serve Master’s degree students, and 83 intended to serve doctoral students.  Most programs also intended to serve multiple groups; 137 of the 163 programs served two or more groups.

Similar STEM Fields  More than 75 percent of programs focused on specific STEM academic fields of study. The most common fields were biology (85  programs), technology (75 programs), engineering (72 programs), and computer science (71 programs). Of those programs that  focused on specific STEM fields of study, about 55 percent (68 programs) focused on 5 or more different fields.

Similar Objectives  Many STEM education programs had similar objectives. An objective of a majority of programs (115) was to provide training  opportunities for undergraduate or graduate students in STEM fields. Most programs (139) also reported having multiple primary  STEM objectives.

Despite these similarities, overlapping programs may differ in meaningful  ways, such as their specific field of focus and those programs’ stated  goals. For example, a primary objective of the Department of Health and  Human Services’ Cancer Education Grants program and the National  Aeronautics and Space Administration’s National Space Grant College  and Fellowship Project is to provide training opportunities for  undergraduate or graduate students in biological sciences, among other  fields. However, these programs have different program goals:

The Cancer Education Grants program aims to develop innovative  cancer education programs and cancer research dissemination  projects.

The National Space Grant College and Fellowship Project encourages  interdisciplinary education, research, and public service programs  related to aerospace.

Although many STEM education programs are designed to provide  similar services to similar groups, some programs serve distinct  populations within those broader groups, such as minority,  disadvantaged, or underrepresented groups. Within the broad group— middle and high school students, an individual program may focus on  serving only minority, disadvantaged, or underrepresented students. For  example, the Department of Transportation’s Garrett A. Morgan  Technology and Transportation Education program focuses services on  students who are girls and minorities, whereas the Department of  Education’s Upward Bound Math-Science program aims to serve  students who are economically disadvantaged.

The Committee on STEM Education and the Office of Science and  Technology Policy reported managing overlap in the portfolio by  coordinating with other agencies through a:

Cross-agency priority goal. Project management and oversight of  this goal provided an additional mechanism to facilitate coordination.  Goal leaders published quarterly progress reports describing their  efforts to achieve each of the five priority investment areas and two  coordination objectives.

Federal coordination subcommittee. Creating a federal  coordination subcommittee and various interagency working groups  helped to advance goals identified in the Strategic Plan. Committee  leadership structured working groups to connect agencies with similar  programs (see fig. 4).

Efforts to Assess Programs’ Performance and Participation Rates of Underrepresented Minorities Are Limited

Performance Assessments of STEM Education Programs Are Not Reviewed or Documented

The Committee on STEM Education and Office of Science and  Technology Policy have not fully met their responsibilities to assess the  STEM education portfolio. Specifically, the Committee on STEM  Education has not reviewed performance assessments of STEM  education programs to ensure effectiveness—a primary function of its  authorizing charter. Committee leadership acknowledged that they have  not conducted such reviews. Overall, the Committee made limited  progress advancing its strategic goal of increasing the use of evidence- based approaches because, according to Committee leadership, they  focused on achieving other strategic goals. By reviewing programs’  performance assessments, the Committee could leverage existing  performance information to identify and share promising practices that  agencies could use in designing or revising their programs. Moreover, in  doing so, the Committee could further its strategic goal of increasing the  use of evidence-based approaches across the portfolio of STEM  education programs. We previously have reported that managers can use  performance information to identify and increase the use of program  approaches that are working well. Additionally, such a review could help  the Committee meet its new responsibilities under the 2017 American  Innovation and Competitiveness Act, including reviewing the measures  federal agencies use to evaluate their STEM education programs and  making recommendations for terminating, consolidating, and reforming  programs in the federal STEM education portfolio.

Further, the Committee on STEM Education has not met the COMPETES  Act requirement to document the performance assessments of STEM  education programs in its federal STEM inventory (see sidebar).

In 2011, the Committee on STEM Education reported summary  information on programs’ performance assessments, including the total  number of programs funded in fiscal year 2010 that had been evaluated  since 2005. However, the information provided was not program- specific; therefore, it is unclear which programs were assessed for  effectiveness. Further, that information is outdated, as the STEM  education portfolio has changed considerably since 2010, as we have  discussed in this report. Committee leadership said they do not have  plans to update the summary information provided in 2011, noting that  agency budget justifications include program performance assessments.  However, we reviewed the budget justifications for 10 STEM education  programs that program officials reported had been recently evaluated and  found that 8 had no information on performance assessments. By  periodically documenting in its federal STEM education inventory whether  programs have been assessed for effectiveness, the Committee can  enhance communication of performance information among agency  officials and stakeholders. This could facilitate the use of performance  information by agency managers and lead to greater public awareness  regarding the effectiveness of many of the nation’s STEM education  programs.

The Office of Science and Technology Policy has not done everything  required of it either. It has not described the outcomes of programs’  performance assessments completed in the previous year in its annual  reports, as required by the COMPETES Act (see sidebar).

Office of Science and Technology Policy officials said that they have not  reported on recent program assessments, and added that many STEM  education programs were not mature enough to provide sufficient data for  a definitive assessment. However, many of the 2016 programs that we  identified were at least 7 years old and had been assessed. Specifically,  67 percent (109) of the programs reported by program officials for fiscal  year 2016 had also been reported for fiscal year 2010. Of the programs in  existence since 2010, 49 percent (53) have been assessed, according to  program officials’ questionnaire responses. By reporting information on  the outcomes of performance assessments completed in the previous  year, the Office of Science and Technology Policy could enhance  awareness of promising practices in federal STEM education programs.

Program Participation Rates of Underrepresented Groups Are Not Reported

The Committee on STEM Education has not reported STEM education  programs’ participation rates of groups historically underrepresented in  STEM fields, although broadening participation of those groups is one of  the Committee’s strategic goals. Moreover, the COMPETES Act  requires that the Committee report the participation rates of women,  underrepresented minorities, and persons in rural areas in its inventory of  federal programs (see sidebar).

Committee leadership acknowledged they have not reported these data,  and added that such participation data are not fully available across all  STEM education programs. However, we found that such participation  data were generally available. an inventory of federal STEM education  programs that includes documentation of  participation rates of women,  underrepresented minorities, and persons  in rural areas.

In response to our questionnaire, nearly three-quarters of STEM  education programs (120 of 163) reported tracking participants in fiscal  year 2016. Of those programs, many also tracked specific participant  characteristics. For example, 61 percent (73) of programs that tracked  participants also captured whether their participants were women and 54  percent (65) documented those who were African American. Programs  primarily intended to serve minority, disadvantaged, or underrepresented  groups tracked participant characteristics at higher rates than programs  that intended to serve broader groups of beneficiaries (see fig. 5).

In addition, 7 of the 13 administering agencies, such as the Department of  Health and Human Services, reported that they tracked participation in  fiscal year 2016 for at least two-thirds of their STEM education  programs. Officials from the Department of Health and Human Services said that the department maintains data for many of its STEM education  programs in a database that captures individual participants’ demographic  data, including race and gender, and aggregates such information for  internal reporting. Officials also said they use this information to evaluate  whether individual programs are meeting their goals of serving particular  groups. Although we found that many agencies reported collecting data  on participants in their STEM education programs, the Committee on  STEM Education has not reported such information in its inventory, as  required.

Reporting information on the participation rates of women,  underrepresented minorities, and persons in rural areas could help the  Committee assess whether STEM education programs have broadened  participation to groups historically underrepresented in STEM fields—a  key goal of the Strategic Plan. Committee leadership said they measured  progress toward this goal with general performance indicators, such as  the number of women who earned STEM degrees, regardless of  participation in federal programs, because such data were readily  available. However, those performance indicators are influenced by  various factors, including some external to federal STEM education  efforts. For example, the number of women earning STEM degrees could  be affected by broader economic factors or college enrollment trends,  rather than the activities of the agencies.

The federal government continues to invest billions of dollars annually in  STEM education programs to enhance the nation’s economic and  educational competitiveness. Since 2010, the federal portfolio of STEM  education programs has evolved considerably. The Committee on STEM  Education reported that, through its leadership and strategic planning  efforts, it fostered coordination among agencies administering STEM  education programs, which helped them implement the STEM Education  Strategic Plan. Such efforts to encourage interagency coordination can  help ensure efficient use of resources, particularly given the overlap of  programs in the STEM education portfolio.

The Committee on STEM Education and the Office of Science and  Technology Policy have not fulfilled their responsibilities to review,  document, and report performance information on STEM education  programs. Reviewing performance assessments of the many programs in  the federal STEM education portfolio is a vital management responsibility  that could, for example, improve the Committee’s ability to disseminate  information on promising practices or make recommendations that  agencies can use to make well-informed decisions about designing or  revising their programs. Further, documenting programs’ performance  assessments in the Committee’s federal STEM education inventory and  reporting the outcomes of recent assessments in the Office of Science  and Technology Policy’s annual reports could enhance the availability of  performance information.

In addition, the Committee falls short in reporting required information on  programs’ participation rates of women, underrepresented minorities, and  persons from rural areas. Without such information, it is unclear whether  the federal investment in STEM education is ultimately supporting its  strategic goal of broadening participation to groups historically  underrepresented in STEM fields.

Moreover, as the Committee on STEM Education begins to implement its  new responsibilities prescribed by the American Innovation and  Competitiveness Act, its efforts to review programs’ performance  assessments could improve its capacity to make well-informed  recommendations to further enhance the portfolio of STEM education  programs.

We are making a total of four recommendations, including three to the  Committee on STEM Education and one to the Office of Science and  Technology Policy. Specifically:  The leadership of the Committee on STEM Education should  review performance assessments of federal STEM education  programs and then take appropriate steps to enhance  effectiveness of the portfolio, such as by sharing promising  practices that agencies could use in designing or revising their  programs. (Recommendation 1)

The leadership of the Committee on STEM Education should  improve public awareness of information on programs’  performance assessments by documenting program-level  information on performance assessments in its federal STEM  education inventory. (Recommendation 2)

The leadership of the Committee on STEM Education should  report required information on the participation rates of women,  underrepresented minorities, and persons from rural areas in  federal STEM education programs that collect this information.  (Recommendation 3)

The Director of the Office of Science and Technology Policy  should report the outcomes of programs’ performance  assessments completed in the previous year in its annual report.  (Recommendation 4)

We provided a draft of this report to the National Science and Technology  Council’s Committee on STEM Education and the Office of Science and  Technology Policy for review and comment. These entities jointly  provided written comments, which are reproduced in appendix IV, and  technical comments, which we incorporated, as appropriate. They agreed  with all four of our recommendations and noted initial strategies for how  they would implement three of them. Regarding implementation of the  fourth recommendation to report on participation rates of  underrepresented groups in federal STEM education programs, they  noted plans to examine confounding factors inhibiting the reporting of the  information required under the COMPETES Act. Gaining insight on the  challenges agencies face collecting this information is an important first  step. However, to comply with the requirement of the COMPETES Act  and help ensure programs reach populations historically  underrepresented in STEM fields, we continue to believe that the  Committee should report  the participation rates of women,  underrepresented minorities, and persons from rural areas in federal  STEM education programs that collect this information. To do so, the  Committee may also need to develop strategies to help agencies  overcome some of these confounding factors.

We are sending copies of this report to leadership of the Committee on  STEM Education, and the Assistant Director of STEM Education at the  Office of Science and Technology Policy, and the appropriate  congressional committees. In addition, the report will be available at no  charge on GAO’s website at http://www.gao.gov.

If you or your staff should have any questions about this report, please  contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix V.

Appendix I: GAO’s Methodology for Program Identification and Data Collection

How GAO Identified Federal Education Programs on Science, Technology, Engineering, and Mathematics

To identify the programs that should receive our questionnaire, we sought  input from the 13 agencies that administer federal science, technology,  engineering, and mathematics (STEM) education programs. We provided  each of the agencies with our definition of a STEM education program  and asked agency officials to identify programs funded in fiscal year 2016  that met this definition (see text box). We also asked agency officials to  provide information on the status of the 209 STEM education programs  we included in our previous report on STEM education programs.  Specifically, we asked whether the programs were funded in fiscal year  2016 and, if not, whether they were consolidated or terminated.

Definition of Science, Technology, Engineering, and Mathematics (STEM) Education Program  GAO defined “STEM education program” as a program funded by allocation or congressional appropriation. An organized set of  activities was considered a single program even when its funds were also allocated to other programs. A STEM education program  that met the definition had one or more of the following as a primary objective:  attract or prepare students to pursue classes or coursework in STEM areas through formal or informal education activities  (informal education programs provide support for activities that offer students learning opportunities outside of formal schooling  through contests, science fairs, summer programs, and other means; outreach programs aimed at the general public were not  included);  attract students to pursue degrees (2-year, 4-year, graduate, or doctoral degrees) in STEM fields through formal or informal  education activities;  provide training opportunities for undergraduate or graduate students in STEM fields (this can include grants, fellowships,  internships, and traineeships that are intended for students; general research grants that involve hiring a student for  lab work  were not considered a STEM education program);  attract graduates to pursue careers in STEM fields;  improve teacher (preservice or in-service) education in STEM fields;  improve or expand the capacity of K-12 schools or postsecondary institutions to promote or foster education in STEM fields; and   conduct research to enhance the quality of STEM education programs provided to students.

Programs designed to retain current employees in STEM fields were not included. Programs that fund retraining of workers to pursue a  degree in a STEM field were included because these programs help increase the number of students and professionals in STEM fields  by helping retrain non-STEM workers to work in STEM fields. Also included were health care programs that train students for careers  that are primarily in scientific research, but not those that train students for careers that are primarily in patient care (e.g. those that  trained nurses, doctors, dentists, psychologists, or veterinarians).

Lastly, GAO considered STEM fields to include any of the following broad disciplines: agricultural sciences; astronomy; biological  sciences; chemistry; computer science; earth, atmospheric, and ocean sciences; engineering; material science; mathematical  sciences; physics; social sciences (e.g., psychology, sociology, anthropology, cognitive science, economics, behavioral sciences); and  technology.

GAO used this same definition of STEM education program in its 2012 report. However, in the current report, GAO explicitly specified  astronomy and material science as STEM fields and also revised “mathematics” to be “mathematical sciences” based on feedback  from agency officials.

We reviewed the information agencies submitted and took steps to  corroborate it, such as by reviewing program descriptions and budget  documents. Based on our analysis of this information, we sent a web- based questionnaire to 198 programs (see table 1).

To develop the questionnaire and collect the data, we used recognized  survey design practices to enhance data quality. For instance, we ordered  the questionnaire appropriately and ensured the questions were clearly  stated and easy to understand. The questionnaire solicited information on  federal STEM education programs, including programs’ objectives,  intended groups served, services provided, STEM fields, and obligations.  We did not conduct pretests because most of the questions were included  in our prior questionnaire and had already been pretested.

On May 8, 2017, we sent an email announcing the online questionnaire to  the officials responsible for programs identified as STEM education and  also notifying them that the questionnaire would be activated that week.  On May 10, 2017, we sent a second message to officials informing them  that the questionnaire was activated and providing them with unique  usernames and passwords. As necessary, we followed-up with program  officials by telephone and email. We collected responses through August  31, 2017.

Based on our analysis of the questionnaire responses and other  information we received from program officials, we excluded 35 programs  from our inventory. (See table 2 for a summary of those 35 programs and  the reasons we excluded them.)

Nine of the 35 excluded programs had been reported by agency officials  as STEM education programs in our previous report. In most cases (8 of  9), we excluded these programs in this report because the programs did  not include STEM education as a primary objective in fiscal year 2016. In  the remaining case, we excluded the program because it was a  component of another fiscal year 2016 STEM education program, and  thus would be duplicative. We confirmed this information and the  programs’ exclusion with the administering agencies.

After we completed our analysis, we identified 163 programs as STEM  education for fiscal year 2016. Programs officials responsible for all 163  of these programs completed our questionnaire. We used standard  descriptive statistics to analyze responses to these completed  questionnaires. We also used recognized survey design practices to  process and analyze data collected via the questionnaire. For instance,  we performed automated checks to review the data and identify  inappropriate answers. We also reviewed the data for missing or  ambiguous responses and followed up with program officials when  necessary to clarify their responses. We did not verify all responses since  we had applied recognized survey design practices and follow-up  procedures, and had determined that the data used in this report were of  sufficient quality for the purposes of our reporting objectives.

Appendix II: Federal Science, Technology, Engineering, and Mathematics (STEM) Education Programs and Reported Fiscal Year 2016 Obligations

Appendix III: Current Implementation Status of Selected COMPETES Act Provisions to Coordinate Federal Science, Technology, Engineering, and Mathematics Education

Appendix IV: Comments from the Office of Science and Technology Policy and the Committee on Science, Technology, Engineering, and Mathematics Education

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Bill J. Keller (Assistant Director),  Kathryn O’Dea Lamas (Analyst-in-Charge), Morgan Jones, and Karissa  Robie made significant contributions. Also contributing to this report were  James Bennett, Deborah Bland, Charles Culverwell, Jill Lacey, Sheila  McCoy, James Rebbe, Kathleen van Gelder, and Sarah Veale.