NASA Commercial Crew Program: Plan Needed to Ensure Uninterrupted Access to the International Space Station

In 2014, NASA awarded two firm-fixed-price contracts to Boeing and SpaceX, worth a combined total of up to $6.8 billion, to develop crew transportation systems and conduct initial missions to the ISS. In February 2017, GAO found that both contractors had made progress, but their schedules were under mounting pressure. The contractors were originally required to provide NASA all the evidence it needed to certify that their systems met its requirements by 2017.

A House report accompanying H.R. 5393 included a provision for GAO to review the progress of NASA's human exploration programs. This report examines the Commercial Crew Program, including (1) the extent to which the contractors have made progress towards certification and (2) how NASA's certification process addresses safety of the contractors' crew transportation systems. GAO analyzed contracts, schedules, and other documentation and spoke with officials from NASA, the Commercial Crew Program, Boeing, SpaceX, and two of NASA's independent review bodies that provide oversight.

Both of the Commercial Crew Program's contractors, Boeing and Space Exploration Technologies Corporation (SpaceX), are making progress finalizing designs and building hardware for their crew transportation systems, but both contractors continue to delay their certification milestone (see figure). Certification is the process that the National Aeronautics and Space Administration (NASA) will use to ensure that each contractor's system meets its requirements for human spaceflight for the Commercial Crew Program.

Further delays are likely as the Commercial Crew Program's schedule risk analysis shows that the certification milestone is likely to slip. The analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average. The average certification date was December 2019 for Boeing and January 2020 for SpaceX, according to the program's April 2018 analysis. Since the Space Shuttle was retired in 2011, the United States has been relying on Russia to carry astronauts to and from the International Space Station (ISS). Additional delays could result in a gap in U.S. access to the space station as NASA has contracted for seats on the Russian Soyuz spacecraft only through November 2019. NASA is considering potential options, but it does not have a contingency plan for ensuring uninterrupted U.S. access.

NASA's certification process addresses the safety of the contractors' crew transportation systems through several mechanisms, but there are factors that complicate the process. One of these factors is the loss of crew metric that was put in place to capture the probability of death or permanent disability to an astronaut. NASA has not identified a consistent approach for how to assess loss of crew. As a result, officials across NASA have multiple ways of assessing the metric that may yield different results. Consequently, the risk tolerance level that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its assessment. Federal internal controls state that management should define risk tolerances so they are clear and measurable. Without a consistent approach for assessing the metric, the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew.

GAO is making five recommendations, including that NASA develop a contingency plan for ensuring a U.S. presence on the ISS and clarify how it will determine its risk tolerance for loss of crew. NASA concurred with three recommendations; partially concurred on the recommendation related to loss of crew; and non-concurred with a recommendation to report its schedule analysis to Congress. GAO believes these recommendations remain valid, as discussed in the report.

Following the retirement of the Space Shuttle in 2011, the United States  was left with no domestic ability to provide crew access to the  International Space Station (ISS). Since then, the National Aeronautics  and Space Administration (NASA) has relied on obtaining seats on the  Russian Soyuz spacecraft to maintain a U.S. presence on the station.  NASA’s Commercial Crew Program is facilitating the commercial  development of a crew transportation system that can provide safe,  reliable, and cost-effective transportation to and from the ISS and that  would end this dependency. NASA’s goal is to have one or more  contractors that can provide crew transportation services to the ISS.  NASA is planning for the ISS to be operational until at least 2024.

NASA’s acquisition strategy for the Commercial Crew Program is similar  to the one it used on the Commercial Cargo Program, but different from  other spacecraft it has built for humans, from Mercury to Gemini and  Apollo to the Space Shuttle. Each contractor is to design, develop, build,  own, and operate its own spaceflight system and infrastructure. The  contractors will have access to NASA’s expertise and resources  throughout the development process, but NASA engineers are not the  ones making design decisions and NASA personnel will be less involved  in processing, testing, launching, and operating the crew transportation  system. In the end, NASA will buy a crew transportation service—a ride  for its astronauts to and from the ISS—much like it does for ISS cargo.

In the most recent phase of the Commercial Crew Program, NASA  awarded firm-fixed-price contracts in 2014 to Boeing and Space  Exploration Technologies Corporation (SpaceX), valued at up to $4.2  billion and $2.6 billion, respectively, for the development of crew  transportation systems that meet NASA requirements and for the initial  service missions to the ISS. According to the contracts, the companies  were originally supposed to complete certification—the process by which  each contractor provides NASA the evidence it needs to certify that its  systems meet performance and safety requirements—by 2017.

The House Report accompanying H.R. 5393, Commerce, Justice,  Science, and Related Agencies Appropriations Bill, 2017 includes a  provision for GAO to review the acquisition progress of NASA’s human  exploration programs, including the Commercial Crew Program. For this  review, we assessed (1) the extent to which the contractors have made  progress towards meeting NASA’s certification requirements and NASA’s  plans to ensure continued access to the ISS; and (2) how NASA’s  certification process addresses safety of the contractors’ crew  transportation systems. To assess the contractors’ progress toward  certification, we reviewed Commercial Crew Program and contract  documents, including quarterly progress updates, monthly risk charts, and  monthly schedule summaries. We also interviewed program and contract  officials to identify steps being taken to mitigate risks and assess the  extent of cost or schedule effects if the risk is realized. For each  contractor, we compared the original contract schedule to the most  current contract schedule and the contractors’ internal development  schedules to determine upcoming events and expected delays. We also  spoke with ISS program officials to determine how NASA plans to  mitigate the effects of these delays on its access to the ISS.

To assess how NASA’s certification process addresses safety of the  contractors’ crew transportation systems, we reviewed agency safety  policies, program plans, and contract documents to establish when  certification approval would be granted, what safety assessments were  required of the program, and which safety factors would be considered in  certification reviews. For each contractor, we compared agency  requirements to contract requirements for a key safety metric known as  “loss of crew,” and interviewed program and agency officials to determine  how loss of crew would be assessed and considered throughout the  certification process. We reviewed program and agency documentation  and interviewed program and agency officials to determine the role of the  safety and mission assurance technical authority in the program. We also  met with two organizations that provide NASA with independent  assessments of the program, the program’s standing review board and  the Aerospace Safety Advisory Panel, to gain their perspectives on the  contractor’s progress and how NASA addresses safety in its certification  process. Appendix I contains detailed information about our scope and  methodology.

We conducted this performance audit from April 2017 to July 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

NASA’s Commercial Crew Program is a multi-phased effort that began in  2010. Across the phases, NASA has engaged several companies, using  both agreements and contract vehicles to develop and demonstrate crew  transportation capabilities. As the program has passed through these  phases, NASA has generally narrowed down the number of participants.  The early phases of the program were under Space Act agreements,  which is what NASA calls the agreements entered into pursuant to its  other transaction authority. These types of agreements are generally not  subject to the Federal Acquisition Regulation (FAR) and allow the  government and its contractors greater flexibility in many areas. Under  these Space Act agreements, NASA relied on the commercial companies  to propose specifics related to their crew transportation systems,  including their design, the capabilities they would provide, and the level of  private investment. In these phases, NASA provided technical support  and determined whether the contractors met certain technical milestones.  In most cases, NASA also provided funding.

For the final two phases of the program, NASA awarded FAR-based  contracts. By using FAR-based contracts, NASA gained the ability to  procure missions to the ISS, while continuing to provide technical  expertise and funding to the contractors. NASA levied two sets of  requirements on the contractors:  the ISS program requirements, which must be met by all spacecraft  visiting the ISS whether they carry cargo or crew; and  the Commercial Crew Program requirements, which have a focus on  system capabilities and safety rather than design.

The program also established a verification closure notice process, in  which the contractors submit data to NASA to verify they have met all the  requirements to be certified. This certification must occur before  contractors are allowed to fly initial crewed missions to the ISS.

In September 2014, NASA awarded firm-fixed-price contracts to Boeing  and SpaceX, valued at up to $4.2 billion and $2.6 billion, respectively, for  the Commercial Crew Transportation Capability phase. Under a firm- fixed-price contract, the contractor must perform a specified amount of  work for the price negotiated by the contractor and government. This is in  contrast to a cost-reimbursement contract, in which the government  generally agrees to pay the contractor’s allowable costs regardless of  whether work is completed. During this phase, the contractors will  complete development of crew transportation systems.

Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module  and a service module.

The crew module will carry the crew and cargo. It also includes  communication systems, docking mechanisms, and return systems for  Earth landing.

The service module provides propulsion on-orbit and in abort  scenarios as well as radiators for thermal control.

SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we  refer to as the crew module, and a trunk, which we refer to as the support  module.

The crew module is composed of a pressure section and a service  section. This module will carry the crew and cargo. It also includes  avionics, docking mechanisms, and return systems for a water  landing.

The support module includes solar arrays for on-orbit power and  guidance fins for escape abort scenarios.

Figure 1 shows the spacecraft and launch vehicles for Boeing and  SpaceX’s crew transportation systems.

The Commercial Crew Transportation Capability phase contracts include  three types of services:

Contract Line Item 001 encompasses the firm-fixed-price design,  development, test, and evaluation work needed to support NASA’s  certification of the contractor’s spacecraft, launch vehicle, and ground  support systems.

Contract Line Item 002 covers any service missions that NASA  orders to transport astronauts to and from the ISS. Under this  indefinite-delivery, indefinite-quantity line item, NASA has ordered six  post-certification missions from each contractor. Each service  mission is its own firm-fixed-price task order. NASA must certify the  contractors’ systems before they can fly these missions.

Contract Line Item 003 is an indefinite-delivery, indefinite-quantity  line item for any special studies, tests, and analyses that NASA may  request. These tasks do not include any work necessary to  accomplish the requirements under contract line item 001 and 002. As  of April 2018, NASA had funded studies worth approximately $30  million to Boeing, including approximately $27 million for additional  testing of the parachute system. NASA had funded studies worth  approximately $44 million to SpaceX, including approximately $34  million for additional testing of the parachute system. For each  contractor, the maximum value of this contract line item is $150  million.

NASA has made changes to the contracts that have increased their  value. While the contracts are fixed-price, their values can increase if  NASA adds work or otherwise changes requirements, among other  means. As of April 2018, NASA requirement changes had increased the  value of contract line item 001 for Boeing by approximately $191 million  and for SpaceX by approximately $91 million.

NASA divided the certification work under contract line item 001 into two  acceptance events: the design certification review and the certification  milestone. An acceptance event occurs when NASA approves a  contractor’s designs and acknowledges that the contractor’s work is  complete and meets the requirements of the contract.

The first acceptance event—the design certification review—verifies the  contractor’s crew transportation system’s capability to safely approach,  dock, mate, and depart from the ISS, among other requirements. After the  contractor has successfully completed all of its flight tests, as well as  various other activities, the second acceptance event—the certification  milestone—determines whether the crew transportation system meets the  Commercial Crew Program’s requirements. Following this contract  milestone is an agency certification review, which authorizes the use of a  contractor’s system to transport NASA crew to and from the ISS. Figure 2  shows a notional path leading up to the agency certification review.

The Commercial Crew Program’s certification plan outlines how the  program will incrementally review required deliverables leading up to, and  supporting, the agency certification review. For each review, the plan  describes the information that the contractor and the program will present.  At the agency certification review, which is chaired by the Associate  Administrator of the Human Exploration and Operations Mission  Directorate, the agency will review the program’s formal recommendation  to certify the contractor’s crew transportation system. Program officials  said that their goal is to develop and review certification evidence  incrementally in order to reduce the risk that issues will be identified  during the agency certification review.

In our February 2017 report, we evaluated the progress made by the two  contractors on the Commercial Crew Program and found the following:

Both of the Commercial Crew Program’s contractors had made  progress developing their crew transportation systems, but both also  had aggressive development schedules that were increasingly under  pressure. We reported that both Boeing and SpaceX had determined  that they would not be able to meet their original 2017 certification  dates, and both expected certification to be delayed until 2018. We  found that the schedule pressures were amplified by NASA’s need to  provide a viable crew transportation option because its contract with  Russia’s space agency was to provide crew transportation to the ISS  for six astronauts through 2018 with rescue and return through late  spring 2019. Purchasing additional seats from Russia involves a  contracting process that typically takes 3 years. Without a viable  contingency option for ensuring uninterrupted access to the ISS in the  event of further Commercial Crew delays, we concluded that NASA  was at risk of not being able to maximize the return on its multibillion  dollar investment in the space station.

The Commercial Crew Program was using mechanisms laid out in its  contracts to gain a high level of visibility into the contractors’ crew  transportation systems, but maintaining that level of visibility through  certification could add schedule pressures. We noted that, for  example, due to NASA’s acquisition strategy for this program, its  personnel were less involved in the testing, launching, and operation  of the crew transportation system. While the program developed  productive working relationships with both contractors, obtaining the  level of visibility that the program required had also taken more time  than the program or contractors had anticipated. Ultimately, we noted  that the program had the responsibility for ensuring the safety of U.S.  astronauts, and its contracts gave it deference to determine the level  of visibility required to do so. We concluded that the program office  could face difficult choices moving forward about how to maintain the  level of visibility it feels it needs without adding to the program’s  schedule pressures.

In order to ensure that the United States had continued access to the ISS  if the Commercial Crew Program’s contractors experienced additional  schedule delays, we recommended in our February 2017 report that the  NASA Administrator develop a contingency plan for maintaining a  presence on the ISS beyond 2018, including options to purchase  additional Russian Soyuz seats, and report to Congress on the results.  NASA concurred with this recommendation, and in February 2017, NASA  executed a contract modification that purchased two seats and included  an option to purchase three additional crewmember seats from Boeing on  the Russian Soyuz vehicle. These seats represent a contingency plan for  U.S. access to the ISS through 2019. In April 2017, NASA informed the  Congress of this action.

Contractors Have Made Progress, but NASA Has Not Finalized Plans to Ensure ISS Access Given Persistent Delays

Boeing and SpaceX continue to make progress developing their crew  transportation systems, but both contractors have further delayed the  certification milestone to early 2019. These changes have occurred as the  contractors continue to work to aggressive schedules, and they have had  to delay key events regularly. Further delays are likely as the Commercial  Crew Program’s schedule risk analysis shows that the certification  milestone is likely to further slip. In addition, as of mid-June 2018, NASA  officials told us that these dates may change soon but that both  contractors have not yet provided official updates to their schedules to  NASA. NASA has not fully shared information with Congress regarding  the risks of future schedule delays for the contractors and, as a result,  Congress lacks insight into when the contractors will be certified. Also,  there may be a gap in access to the ISS if the Commercial Crew Program  experiences additional delays. While NASA has begun to discuss  potential options, it currently does not have a contingency plan for how to  ensure an uninterrupted presence on the ISS beyond 2019.

Contractors Continue to Make Progress, but Risks Remain

Boeing and SpaceX have continued to make progress finalizing their  designs and building hardware as they work toward their certification  milestones. The contractors are manufacturing test articles to  demonstrate system performance and flight spacecraft to support the  uncrewed and crewed flight tests, which are expected to demonstrate the  ability to meet contract requirements. As table 1 shows, these test articles  and spacecraft vary in levels of completion. Some are built and  undergoing testing while others are starting the manufacturing phase.  Should any issues arise during integration and test or the flight tests, the  contractors may have to complete rework on the spacecraft already under  construction.

While both contractors are making progress, the Commercial Crew  Program is tracking risks that each contractor has to address through  testing and other means as they work towards the certification milestone.

As we have previously reported, these types of risks are inherent in  NASA’s major acquisitions, which are highly complex, specialized, and  often pushing the state of the art in space technology, but they could also  delay the contractors’ progress if issues arise during testing.

The Commercial Crew Program’s top programmatic risks identified for  Boeing include challenges related to its abort system performance,  parachutes, and launch vehicle.

Abort System: Boeing is addressing a risk that its abort system, which  it needs for human spaceflight certification, may not meet the  program’s requirement to have sufficient control of the vehicle through  an abort. In some abort scenarios, Boeing has found that the  spacecraft may tumble, which could pose a threat to the crew’s safety.  To validate the effectiveness of its abort system, Boeing has  conducted extensive wind tunnel testing and plans to complete a pad  abort test in July 2018.

Parachute System: Boeing is also addressing a risk that during  descent, a portion of the spacecraft’s forward heat shield may re- contact the spacecraft after it is jettisoned and damage the parachute  system. Boeing’s analysis indicates the risk exists only if one of two  parachutes that pull the forward heat shield away from the spacecraft  does not deploy as expected, and that potential re-contact is non- detrimental. However, NASA’s independent analysis indicates that  this may occur even if both parachutes deploy as expected. If the  program determines this risk is unacceptable, Boeing would need to  redesign the parachute system, which the program estimates could  result in at least a 6-month delay.

Launch Vehicle Data: One of the program’s top programmatic and  safety concerns is that it may not have enough information from  Boeing’s launch vehicle provider, United Launch Alliance, to assess  whether the Atlas V launch vehicle prevents or controls cracking that  could lead to catastrophic failures. NASA estimates that unfinished  work in this area could take Boeing and the United Launch Alliance  until the fourth quarter of 2018 to complete. Additionally, the first stage  of the Atlas V is powered by the Russian built RD-180 engine, and,  according to program and Boeing officials, access to its data is highly  restricted by agreements between the U.S. and Russian  governments. Since our last report, the Commercial Crew Program  has lowered the risk that certification of the launch vehicle might not  occur by negotiating steps to access necessary data, but work is still  ongoing.

The Commercial Crew Program’s top programmatic risks identified for  SpaceX are in part related to ongoing design and development efforts  related to its launch vehicle design, the Falcon 9 Block 5.

Composite Overwrap Pressure Vessel: This Block 5 design includes  SpaceX’s redesign of the composite overwrap pressure vessel, which  is intended to contain a gas under high pressure. SpaceX officials  stated the newly designed vessel aims to eliminate risks identified in  the older design, which was involved in an anomaly that caused a  mishap in September 2016. SpaceX plans to qualify the updated  design for flight prior to the uncrewed flight test design certification  review.

Engine Turbine Cracking: The Block 5 design also includes design  changes to address cracks in the turbine of its engine identified during  development testing. NASA program officials told us that they had  informed SpaceX that the cracks were an unacceptable risk for  human spaceflight. SpaceX officials told us that they have made  design changes to this Block 5 upgrade that did not result in any  cracking during initial testing. However, this risk will not be closed until  SpaceX successfully completes qualification testing in accordance  with NASA’s standards without any cracks. As of March 2018,  SpaceX had not yet completed this testing.

Propellant Loading Procedures: Both the program and a NASA  advisory group have raised SpaceX’s plan to fuel the launch vehicle  after the astronauts are on board the spacecraft to be a potential  safety risk. In the May 2018 meeting minutes, however, the  Aerospace Safety Advisory Panel stated that with appropriate controls  in place, this approach could be a viable option for the program to  consider. SpaceX’s perspective is that this operation may be a lower  risk to the crew because it reduces the crew exposure time while the  launch vehicle is being loaded with propellant. To better understand  the propellant loading procedures, the program and SpaceX agreed to  demonstrate the loading process five times from the launch site in the  final crew configuration prior to the crewed flight test. The five events  include the uncrewed flight test and the in-flight abort test. Therefore,  delays to those events would lead to delays to the agreed upon  demonstrations, which could in turn delay the crewed flight test and  certification milestone.

Program’s Schedule Risk Analysis Indicates More Delays Likely to Certification Milestone

Both contractors have notified NASA that their certification milestones  have slipped to January 2019 for Boeing and February 2019 for SpaceX,  but the Commercial Crew Program’s schedule risk analysis indicates  more delays are likely. This analysis identifies a range for each  contractor, with an earliest and latest possible completion date, as well as  an average. In April 2018, the program’s schedule risk analysis found  there was zero percent chance that either contractor would achieve its  current proposed certification milestone. The analysis’s average  certification date was December 2019 for Boeing and January 2020 for  SpaceX. Figure 3 shows the original Boeing and SpaceX contract  schedules and the current proposed schedule for five key events in each  contract, as well as NASA’s schedule risk analysis for the certification  milestone.

Each month, the program updates its schedule risk analysis based on the  contractors’ internal schedules as well as program officials’ perspectives  and insight into specific technical risks. The Commercial Crew Program  manager told us that differences between the contractors’ proposed  schedules and the program’s schedule risk analysis include:

The contractors are aggressive and use their schedule dates to  motivate their teams, while NASA adds additional schedule margin for  testing.

Both contractors assume an efficiency factor in getting to the crewed  flight test that NASA does not factor into its analysis.

The program manager also told us that the program meets with each  contractor monthly to discuss schedules and everyone agrees to the  relationships between events in the schedule even if they disagree on the  length of time required to complete events. The program manager added,  however, that she relies on her prior experience to estimate schedule  time frames as opposed to relying on the contractors’ schedules, which  are often optimistic.

Our analysis also shows that the contractors often delay their schedules.  Both contractors have repeatedly stated that their schedules are  aggressive and have set ambitious—rather than realistic—dates, only to  frequently delay them. Since the current contracts were awarded in 2014,  the Commercial Crew Program has held 13 quarterly reviews for each  contractor. For the five key events identified above, Boeing has reported  a delay at 7 of those quarterly reviews and SpaceX has reported a delay  at 9 of them.

In mid-June 2018, NASA officials told us that the dates for these key  events may change soon. The information presented in Figure 3 above is  based on first quarter calendar year 2018 data. NASA officials stated both  contractors have not yet officially communicated new schedule dates to  NASA as of the second quarter calendar year 2018. We found that both  contractors have updated schedules that indicate delays are forthcoming  for at least one key event, but NASA officials told us they lack confidence  in those dates until they are officially communicated to NASA by the  contractors. As a result, NASA is managing a multibillion dollar program  without confidence in its schedule information as it approaches several  big events, including uncrewed and crewed flight tests.

NASA Has Neither Shared Complete Information on Delay Risk with Congress nor Developed a Contingency Plan

The risk of future delays in the contractors’ schedules is critical  information that NASA has not fully shared with Congress. Moreover,  NASA has not yet developed a contingency plan to address the potential  gaps that these delays could have on U.S. access to the ISS after 2019.  Specifically, in the Explanatory Statement accompanying the fiscal year  2018 Consolidated and Further Continuing Appropriations Act, the House  Appropriations Committee stated its expectation that NASA report  quarterly to the Senate and House Committees on Appropriations on the  status of the Commercial Crew Program contracts. Previously,  members of Congress had asked for this information in order to ensure  that Congress had adequate insight into this program. While NASA  includes both contractors’ proposed schedules in its quarterly report to  Congress, NASA does not include the results of its own schedule risk  analysis. Given the frequency with which the contractors delay key events  in their schedules, the program’s schedule risk assessment provides  valuable insight into potential delays that NASA currently is not providing  to Congress.

In addition, as previously mentioned, NASA executed a contract  modification that purchased two seats and included an option to purchase  three additional crew member seats through Boeing for an undisclosed  value and reported this action to Congress in April 2017. Ultimately, the  option was exercised, and NASA purchased a total of five seats on four  different Soyuz flights. Boeing obtained these seats through a separate  settlement with the Russian firm RSC Energia, which manufactures the  Soyuz.

These seats were intended to serve as a contingency plan based on  schedule information available at that time. However, subsequent delays,  as well as the risk of future delays as discussed above, indicate that this  contingency plan will likely no longer be sufficient. The earliest and latest  possible completion dates for certification in NASA’s April 2018 schedule  risk analysis indicate it is possible that neither contractor would be ready  before August 2020, leaving a potential gap in access of at least 9  months. We calculated the potential gap based on the contractor  certification milestone dates, but there could be some additional time  required between that review and the first post-certification service  mission to the ISS.

As seen in figure 4, if the contractors can maintain their current proposed  schedules for their respective certification milestones, a gap in access to  the ISS is not expected. However, there would be a gap in access to the  ISS if neither contractor has its certification milestone before November  2019, which is when NASA expects the final Russian Soyuz seat for a  U.S. astronaut to return.

Senior NASA officials told us that sustaining a U.S. presence on the ISS  is essential to maintain and operate integral systems, without which the  ISS cannot function. Given the importance of maintaining a U.S. presence  on the ISS, NASA officials have stated they are working on options to  address the potential gap in access. However, officials told us that  planning for contingencies is difficult given the extensive international  negotiations required for some options. Obtaining additional Soyuz seats  seems unlikely, as the process for manufacturing the spacecraft and  contracting for those seats typically takes 3 years—meaning additional  seats would not be available before 2021. As a result, according to  NASA’s Associate Administrator for Human Exploration and Operations,  the options NASA is considering include:

Refine the remaining Soyuz launch schedule to allow for a return in  January 2020, as opposed to November 2019. This would provide 2  additional months of access to the ISS before the commercial crew  flights need to start.

Use the crewed flight tests as operational flights to transport U.S.  astronauts to and from the ISS. In March 2018, NASA modified  Boeing’s contract to allow NASA to add a third crew member and  extend the length of the flight test, if NASA chooses to do so. This  would have limited usefulness, however, in filling a potential gap in  access to the ISS if the schedule for Boeing’s crewed flight test slips  past the return date for the last Soyuz flight and SpaceX also  continues to experience delays.

NASA’s Associate Administrator for Human Exploration and Operations  stated that he is “brainstorming” other options to ensure access to the ISS  but does not have a formal plan. While options are not unlimited and  decisions have to be made within the context of the current geopolitical  environment, Congress stated in the NASA Authorization Act of 2005 that  it is U.S. policy to possess the capability for human access to space on a  continuous basis. In 2010, Congress further stated that one of the key  objectives of the United States’ human spaceflight policy is to sustain the  capability for long-duration presence in low-Earth orbit through full  utilization of the ISS. If NASA does not develop options for ensuring  access to the ISS in the event of further Commercial Crew delays, it will  not be able to ensure that the U.S. policy goal and objective for the ISS  will be met.

Agency Certification Process Includes Mechanisms to Assess Safety, but Is Complicated by Assessment of Key Safety Metric and Oversight Structure

The Commercial Crew Program relies on several contractual mechanisms  to assess safety throughout the certification process, and those  mechanisms are in varying stages of completion. The program itself, its  contractors, and two of NASA’s independent review organizations have  raised concerns about the program’s ability to assess and evaluate all of  the deliverables in a timely manner. In addition, one of the key safety  requirements levied by the program is loss of crew, which captures the  probability of death or disability to a crew member. NASA does not have a  consistent approach for how to incorporate key inputs to assess this  metric, which means the agency as a whole may not clearly capture or  document its risk tolerance with respect to loss of crew. Further, the  program’s chief safety and mission assurance officer is dual hatted to  serve simultaneously in a programmatic position as well as the program’s  safety technical authority. This approach creates an environment of  competing interests because it relies on the same individual to manage  technical and safety aspects on behalf of the program while also serving  as the independent oversight of those same areas.

Program Has Several Contractual Mechanisms to Assess Safety

The contractors are required to provide several key deliverables to the  Commercial Crew Program, which inform the agency certification review  and help NASA determine the level of risk it is accepting with respect to  safety of each spacecraft. As described below, these deliverables are in  varying stages of completion and the program itself, its contractors, and  two of NASA’s independent review organizations have raised concerns  about the program’s ability to assess and evaluate all of the deliverables  in a timely manner.

Certification Data Package. Among other things, the certification data  package includes a list of seven system safety assessments. For  example, the certification data package includes a fault tolerance  assessment, which describes the system’s ability to sustain a certain  number of undesired events, such as software or operational anomalies.  A human error analysis—one of the seven assessments in the data  package—evaluates human errors to minimize their negative effects on  the system.

Boeing held its uncrewed flight test design certification review in  December 2017 and submitted its certification data package for NASA  approval. Boeing plans three more updates to this data package prior  to the final certification milestone.

SpaceX has begun to submit data and plans to submit its final  certification data package as part of its crewed flight test design  certification review, which is scheduled for September 2018.

According to the program’s certification review plan, program officials will  review and approve the contractors’ certification data packages, which  will be used to inform the agency certification review.

Phased Safety Review Process. A three-phased safety review process  informs the program’s quality assurance activities, and it is intended to  ensure that the contractors have identified all safety-critical hazards and  implemented associated controls prior to the first crewed flight test.

In phase one, the contractors identified risks in their designs and  developed reports on potential hazards, the controls they put in place  to mitigate them, and explanations for how the controls will mitigate  the hazards.

In phase two, which is nearing completion, the program reviews and  approves the contractors’ hazard reports and develops strategies to  verify and validate that the controls are effective. For example, if a  control requires that an item be waterproofed, verification and  validation strategies could include inspections and tests to confirm  that the item is waterproof. As of April 2018, the program had yet to  complete this phase, having approved 97 percent of Boeing’s phase  two reports and 72 percent of SpaceX’s phase two reports.

In phase three, the contractors will conduct the verification activities  and submit the hazard reports to the program for approval. The  program has begun phase three, including approving 19 percent of  Boeing’s phase three reports.

Program Requirements. While the program manager told us that all of  the requirements contribute to the safety of the commercial systems,  safety officials are required to approve a subset of these requirements.  Examples of requirements approved by safety officials include the ability  to leave the spacecraft in an emergency or to abort a launch. When a  contractor is ready for NASA to verify that it has met a requirement, the  contractor submits data for NASA to review through a verification closure  notice. We define “safety-specific notices” as those requiring safety  officials’ approval. As shown in table 2, as of March 2018, the program  had approved 2 percent of Boeing’s safety-specific notices and 0 percent  of SpaceX’s safety-specific notices.

Testing. The program also requires testing to verify and validate the crew  transportation system. Agency officials emphasized the importance of  testing to safety, stating that testing reduces uncertainty about a system’s  performance and can uncover unknown problems. As noted above, both  contractors will be conducting an uncrewed and a crewed flight test prior  to being certified. While a certain level of risk needs to be accepted to  conduct human spaceflight, these flight tests help to mitigate this risk by  validating the integrated performance of the hardware and software.  Agency and program officials stated that the contractors’ flight tests are  critical evidence to support certification of a safe and reliable system.

As evidenced by the data above, the program still has a significant  amount of work ahead with respect to approving certification packages  and closing hazard reports and verification closure notices. We have  previously found that the program’s workload was an emerging schedule  risk, and the contractors have continued to express concern about  program officials’ ability to process and approve certification paperwork in  a timely manner. Workload has also been a concern for two of NASA’s  independent review organizations. For example, the Aerospace Safety  Advisory Panel noted in its January 2018 annual report that the sheer  volume of work that remains for the program in terms of closing hazard  reports and verification closure notices is significant. In addition, the  program’s safety and mission assurance office identified the upcoming  bow wave of work in a shrinking time period as a top risk to achieving  certification.

NASA Lacks a Consistent Approach to Assess Key Safety Metric

One mechanism the program put in place to assess the overall safety of  each spacecraft—loss of crew—has been a focus of the Aerospace  Safety Advisory Panel, Members of Congress, our prior work, and the  program itself. Loss of crew captures the probability of death or  permanent disability to one or more crew members. It has received a lot  of attention, in part, because it has been a top risk for the program since  2015. Specifically, the program has been concerned that neither  contractor would be able to meet the contract requirement of a 1 in 270  probability of incurring loss of crew. We identified two key concerns with  how NASA is using the loss of crew metric: (1) inconsistent approaches to  assess the loss of crew metric and (2) no identified plan to share lessons  learned about using the loss of crew metric as a safety threshold.

A loss of crew value is generated through a probabilistic safety analysis,  which models scenarios that could result in the loss of crew using various  inputs. According to the program’s analysis, the probability of on-orbit  debris damaging the vehicle has the greatest effect on a loss of crew  value. This probability is informed by an orbital debris (debris) model,  which was updated in 2014, after the loss of crew requirement was  established. The updated debris model makes it harder to meet a loss of  crew value, in part, because the modeling environment where the  contractors’ systems will operate has changed. For example, the updated  model includes a larger span of orbit, greater range of debris sizes, and  the addition of material density classifications, which were not included in  the former model. Further, the probabilistic safety analysis may include  operational mitigations, such as on-orbit inspections that would include  using cameras on the ISS to visually survey the spacecraft for damage,  which, according to officials, makes it easier to meet a loss of crew value.

NASA describes the probabilistic safety analysis as a powerful tool that  should be used as part of the overall risk management process to ensure  the risk associated with development and operation of a system is  understood, evaluated, managed, and mitigated. However, we found  differences in the approaches that officials plan to use to assess loss of  crew as well as in the loss of crew value being measured that could limit  the usefulness of this tool.

Agency Certification. The agency certification review for each  contractor will include an assessment of whether its crew  transportation system meets a loss of crew threshold of 1 in 150 for  missions to the ISS, which is based on a May 2011 safety memo from  the Office of Safety and Mission Assurance. A loss of crew value with  a higher denominator, such as 1 in 270, is harder to meet than with a  lower denominator, such as 1 in 150. According to the Chief of the  Office of Safety and Mission Assurance, he will assess the 1 in 150  threshold using a probabilistic safety analysis that includes the  updated debris model and operational mitigations, such as the on- orbit inspections cited above.

Program Office. According to program officials, they will assess  whether either contractor meets a 1 in 270 loss of crew value based  on a probabilistic safety analysis using the former debris model (not  the updated model) and not including operational mitigations.

Contracting Officer. According to the contracting officer, each  contractor’s loss of crew requirement is 1 in 270 without including  operational mitigations. The contracting officer stated that SpaceX’s  contract requirement uses the updated debris model in the  probabilistic safety analysis, whereas Boeing’s contract requirement  uses the former debris model in the probabilistic safety analysis.

Program’s Chief Safety and Mission Assurance Officer. According  to the program’s chief safety and mission assurance officer, he will  conduct a probabilistic safety analysis using the updated debris model  and will not include operational mitigations to assess whether each  contractor meets a 1 in 200 loss of crew value. This loss of crew value  stems from a program update that occurred after the initial contracts  were signed.

These different approaches are summarized in table 3 below.

Agency policy requires human spaceflight programs to set a safety  threshold, which NASA did for the Commercial Crew Program when it  identified the 1 in 150 loss of crew threshold in the May 2011 safety  memo. Subsequently, the program set more rigorous loss of crew values  in contract and program documents. NASA also updated the debris  model, which we previously noted makes it more difficult to meet a loss of  crew value. As a result, NASA does not have a consistent approach for  how to incorporate key inputs to the probabilistic safety analysis, including  changes to the debris model. Instead, the risk tolerance that NASA is  accepting with loss of crew varies based upon which entity is presenting  the results of its probabilistic safety analysis. For example, it is possible  that the program’s assessment will determine that neither contractor will  meet the 1 in 270 contract requirement, but that the agency’s assessment  will determine that the contractors meet the 1 in 150 agency certification  value because that analysis will include operational mitigations. Or the  program’s assessment could determine that Boeing meets the 1 in 270  contractual loss of crew requirement, but the agency’s assessment may  determine that Boeing does not meet the 1 in 150 agency certification  value because that analysis will use the updated debris model.

Federal internal controls state that agency management should define  objectives clearly to enable the identification of risks and define risk  tolerances. Specifically, management should define risk tolerances in  specific and measurable terms, so they are clearly stated and can be  measured. In this case, because there will be multiple analyses  conducted using different inputs, NASA risks not clearly capturing or  documenting, in a coherent manner, its overall risk tolerance with respect  to loss of crew before a final decision must be made on whether to certify  either crew transportation system.

Moreover, capturing the challenges and lessons learned from using the  loss of crew metric is critical, particularly because agency officials told us  that this is the first time this metric has been used as a safety threshold.  Also, there are different viewpoints about the utility of the metric as a  safety threshold across the agency. The program manager repeatedly  told us that loss of crew is best used as a design tool. For example,  program officials told us that both contractors incorporated additional  orbital debris shielding into their designs to mitigate the orbital debris risk  and improve their loss of crew values. In addition, the Aerospace Safety  Advisory Panel reported in 2018 that loss of crew should not be viewed  as an absolute measure of actual risk during operations. However, the  May 2011 agency safety memo states that a breach of the loss of crew  threshold would initiate a termination review of the Commercial Crew  Program, which is a more strict application of the loss of crew metric.

Both program and safety officials told us that, after the agency  certification is complete and lessons learned are available to be compiled,  sharing those lessons learned across NASA would be a good idea given  the complexities associated with assessing the loss of crew metric. As of  April 2018, however, agency officials said they did not have a plan for  loss of crew knowledge-sharing. We have previously found that lessons  learned provide a powerful method of sharing good ideas for improving  work processes, facility or equipment design and operation, quality,  safety, and cost-effectiveness. Further, according to NASA’s Knowledge  Policy on Program and Projects, which is managed through the Office of  the Chief Engineer, a principle of each center and mission directorate’s  knowledge strategy is that knowledge is the cornerstone of NASA’s ability  to achieve mission success. The policy acknowledges that NASA faces  continuous challenges in using what it knows effectively. These  challenges include, but are not limited to, enabling the identification and  flow of knowledge across organizational boundaries; preserving  knowledge at risk of being lost; and providing means for individuals,  teams, and the organization to learn from experiences. If NASA does not  capture lessons learned from the Commercial Crew Program on using the  loss of crew requirement to set a program’s safety threshold and whether  it met the agency’s intended goal, future programs will not be able to  benefit from the knowledge gained from this multibillion dollar investment.

The Commercial Crew Program’s Organizational Structure Impairs Independence of Safety Technical Oversight Supporting the Certification Process

NASA’s governance model prescribes a management structure that  employs checks and balances among key organizations to ensure that  decisions have the benefit of different points of view and are not made in  isolation. As part of this structure, NASA established the technical  authority process as a system of checks and balances to provide  independent oversight of programs and projects in support of safety and  mission success through the selection of specific individuals with  delegated levels of authority. The technical authority process has been  used in other parts of the government for acquisitions, including the  Department of Defense and Department of Homeland Security.

The Commercial Crew Program is organizationally connected to three  technical authorities within NASA: the Office of the Chief Engineer  technical authority, the Office of Chief Health and Medical technical  authority, and the Office of Safety and Mission Assurance (safety)  technical authority. The safety technical authority is responsible for  ensuring from an independent standpoint that the program’s products and  processes satisfy NASA’s safety, reliability, and mission assurance  policies. The NASA safety technical authority has delegated authority  through the Kennedy Space Center Director to the Chief Safety and  Mission Assurance Officer for the Commercial Crew Program.

We have previously reviewed how NASA has organized its technical  authorities for its Exploration Systems Development organization—an  organization that oversees the development of the Space Launch  System, Orion crew capsule, and associated ground systems that have  the goal of extending human presence beyond low-Earth orbit. In October  2017, we found that the Exploration Systems Development organization  had established an organizational structure in which the technical  authorities for engineering and safety and mission assurance were dual  hatted simultaneously in programmatic positions. We found that having  the same individual simultaneously fill both a technical authority role and  a program role created an environment of competing interests, where the  technical authority’s ability to impartially and objectively assess the  programs while at the same time acting on behalf of the Exploration  Systems Development organization in programmatic capacities may be  subject to impairments.

We found that this was in contrast to a recommendation from the  Columbia Accident Investigation Board report—the result of an in-depth  assessment of the technical and organizational causes of the 2003 Space  Shuttle Columbia accident—for NASA to establish a technical authority to  serve independently of the Space Shuttle program, so that employees  would not feel hampered to bring forward safety concerns or  disagreements with programmatic decisions. The board’s findings that  led to this recommendation included a broken safety culture in which it  was difficult for minority and dissenting opinions to percolate up through  the hierarchy; dual center and programmatic roles vested in one person  that had confused lines of authority, responsibility, and accountability and  made the oversight process susceptible to conflicts of interest; and  oversight personnel in positions within the program, increasing the risk  that these staffs’ perspectives would be hindered by too much familiarity  with the programs they were overseeing. In October 2017, we  recommended that the division no longer dual hat two individuals who  had both programmatic and technical authority responsibilities. As of  April 2018, NASA had taken steps to separate the engineering technical  authority position from the programmatic position, and NASA’s Chief of  Safety and Mission Assurance said he planned to separate the safety  position but had not yet completed that action.

The Commercial Crew Program employs a similar structure to the  Exploration Systems Development organization in that the safety  technical authority is dual hatted simultaneously in a programmatic  position as the Commercial Crew Program’s Safety and Mission  Assurance Manager. According to the program’s safety technical  authority, in his programmatic role for the program, he helps set priorities  for safety issues, including how staff will be utilized to meet those  priorities. In the technical authority role, he provides independent  oversight in support of safety and mission success. In his dual-hatted  role, this official will be responsible for endorsing the program’s  certification recommendations in two different capacities: as the technical  authority and as a program authority.

As a result, this structure relies on the same individual to completely  separate two roles—one to manage the Commercial Crew Program’s  safety issues within programmatic cost and schedule constraints, and the  other to assess the same issues in an independent oversight role. While  the Commercial Crew Program may have an additional level of separation  between the safety technical authority and the program’s involvement in  the design of commercial systems due to its shared assurance model with  the commercial providers, the Commercial Crew Program still maintains a  structure where one individual simultaneously serves in both technical  authority and programmatic roles. Figure 5 describes some of the  conflicting roles and responsibilities of this official in his two different  positions.

During our review, officials cited several factors in support of a dual- hatted approach:

The safety technical authority retains independence because his  technical authority reporting path and performance reviews are not  under the purview of the Commercial Crew Program chain of  command.

Due to the Commercial Crew Program’s shared assurance model with  commercial providers, the program is operating in a quality assurance  role that provides an additional level of separation between the safety  technical authority and the program’s involvement in the design of  commercial systems.

For safety decisions involving cost and schedule where the individual  who is dual hatted with both technical authority and programmatic  responsibilities may feel conflicted, he stated that he would discuss  these matters with his management to validate the logic behind his  decision.

There are cost and knowledge efficiencies gained from one individual  serving in both programmatic and safety technical authority  capacities.

NASA’s Chief of Safety and Mission Assurance stated that he has great  confidence in the individual currently serving in the dual hatted role for the  Commercial Crew Program, but acknowledged there is inherent conflict  even with the program’s shared assurance model. In December 2017, he  stated that, based on our previous work and current discussions, he  intends to decouple the programmatic and technical authority  responsibilities for the Commercial Crew Program but had not done so as  of April 2018. Federal internal control standards state that an agency  should design control activities to achieve objectives and respond to risks,  which includes segregation of key duties and responsibilities to reduce  the risk of error, misuse, or fraud. By overlapping technical authority and  programmatic responsibilities, NASA will continue to run the risk of  creating an environment of competing interests for the Commercial Crew  Program’s safety technical authority.

NASA’s Commercial Crew Program is a multibillion dollar effort to  facilitate the commercial development of a crew transportation system  that can end the United States’ reliance on Russia to maintain an  uninterrupted presence on the ISS. Boeing and SpaceX continue to make  progress developing a capability to fly to the ISS, but both have continued  to experience delays. Program analysis indicates risks of further delays in  each contractor’s current schedule, but NASA has not provided that  information to Congress in its routine briefings. Without this information,  Congress does not know the full extent of potential delays to inform  decision making. Additional delays could also disrupt U.S. access to the  ISS. While NASA is working on potential solutions, there is no  contingency plan in place to address this potential gap. Without a viable  contingency plan, NASA puts at risk achievement of the U.S. goal and  objective for the ISS.

NASA must balance safety with acceptable risk for human spaceflight. As  part of the certification process for each contractor’s spacecraft, NASA  has developed one key safety metric, loss of crew. However, the  complicated nature of this metric is further muddled by the inconsistent  approaches being used across NASA about what inputs to be considered.  As a result, there is no clear articulation of what level of risk NASA will  accept with respect to this program. In addition, NASA does not have  plans to capture lessons learned from how the Commercial Crew  Program has used this metric to assess safety and is missing an  opportunity to capture this knowledge for future human spaceflight  programs.

Finally, a space program’s management and oversight approach is an  integral part of ensuring that human spaceflight is as safe and successful  as possible. Independence of the program management and oversight  functions is key to achieving the balance between safety and success.  The Commercial Crew Program’s approach, however, burdens the safety  technical authority with both programmatic and independent technical  authority responsibilities. As a result, NASA has limited assurance that  independence can be maintained as part of its institutional process to  ensure safety and success.

We are making the following five recommendations to NASA:  The NASA Associate Administrator for Human Exploration and  Operations should direct the Commercial Crew Program to include the  results of its schedule risk analysis in its mandatory quarterly reports to  Congress. (Recommendation 1)

The NASA Administrator should develop and maintain a contingency plan  for ensuring a presence on the ISS until a Commercial Crew Program  contractor is certified. (Recommendation 2)

The NASA Administrator should direct the Chief of Safety and Mission  Assurance, the NASA Associate Administrator for Human Exploration and  Operations, the Commercial Crew Program Manager, and the  Commercial Crew Program Contracting Officer to collectively determine  and document before the agency certification review how the agency will  determine its risk tolerance level with respect to loss of crew.  (Recommendation 3)

After completing the agency certification review, NASA’s Chief Engineer  and Chief of Safety and Mission Assurance, with support from the NASA  Associate Administrator for Human Exploration and Operations and the  Commercial Crew Program Manager, should document lessons learned  related to loss of crew as a safety threshold for future crewed spaceflight  missions, given the complexity of the metric. (Recommendation 4)

The NASA Chief of Safety and Mission Assurance should restructure the  technical authority within the Commercial Crew Program to ensure that  the technical authority for the Office of Safety and Mission Assurance is  no longer dual hatted with programmatic and independent technical  authority responsibilities. (Recommendation 5)

We provided a draft of this report to NASA for review and comment.  NASA provided written comments that are reprinted in appendix II.

In its response, NASA concurred with three of our recommendations, did  not concur with one, and partially concurred with another.

NASA concurred with our recommendation to develop and  maintain a contingency plan to ensure a U.S. presence on the ISS  and expects to take action to close this recommendation by the  end of December 2018.

NASA concurred with our recommendation to document lessons  learned related to the loss of crew requirement and expects to  take action to close this recommendation by the end of May 2019.

NASA concurred with our recommendation to restructure the  safety technical authority so that it is no longer dual hatted with  programmatic and independent technical authority responsibilities.  NASA expects to take action to close this recommendation by the  end of August 2018.

NASA did not concur with our recommendation that the Commercial Crew  Program should include the results of its schedule risk analysis in its  quarterly reports to Congress. NASA stated that it uses the contractors’  schedules as a baseline to provide qualitative statements in the NASA  summary that accompanies each contractor’s quarterly reports to  Congress. NASA believes that this approach is appropriate and is in  accordance with the explanatory statement accompanying the  Consolidated and Further Continuing Appropriations Act, 2015. NASA  also stated that it will be working to ensure that the contractors’ schedules  and the program’s internal assessments sync up as the program gets  closer to launch. As a result, NASA explained that there will not be a  requirement for a detailed NASA assessment, because the contractors’  schedule will either match NASA’s analysis or NASA will discuss its  position as it has done in previous reports to Congress.

We continue to believe the recommendation is valid because the  program’s schedule risk analysis would provide Congress with valuable  insight into potential delays, which are likely. Both contractors have  repeatedly stated that their schedules are aggressive and that the dates  are ambitious. As a result, we found that the contractors frequently delay  dates for key events. For example, Boeing has delayed its certification  milestone by 17 months and SpaceX by 22 months since the original  schedules were established. The program’s recent schedule risk analysis  indicates that more delays to certification are likely, but that information is  not presented to Congress in NASA’s quarterly reports. Without this  information, Congress does not know the full extent of potential delays to  inform decision making.

NASA partially concurred with our recommendation that the Chief of  Safety and Mission Assurance, the NASA Associate Administrator for  Human Exploration and Operations, the Commercial Crew Program  Manager, and the Commercial Crew Program Contracting Officer should  collectively determine and document how the agency will determine its  risk tolerance level with respect to loss of crew before the agency  certification review. In its response, NASA stated that it documented the  agency’s risk tolerance level with respect to loss of crew for the program  in its May 2011 safety memo. Further, NASA stated that it documented  the requirement to limit risks to the loss of crew in a certification  requirements document. NASA stated that ultimately the Commercial  Crew Program is accountable for ensuring that the contractors’ systems  meet the loss of crew value in this certification requirements document,  which is a loss of crew value of 1 in 270. If a contractor’s system cannot  meet that loss of crew value, or any other requirement, the program will  request a waiver as part of the human rating certification process to  ensure transparency.

NASA acknowledged in its response that the existence of multiple  documents defining residual risk requirements and an agency threshold  for loss of crew can be confusing. NASA’s response, however, does not  address our finding that it does not have a consistent approach for how to  incorporate key inputs, including which debris model should be used or  whether to include operational mitigations. NASA stated that it had taken  action to address this recommendation; however, NASA did not outline  any steps it took to resolve the concern that the risk tolerance for the loss  of crew requirement depends on which entity is presenting the results of  its analysis. We continue to believe that, before the agency certification  review, the key parties must collectively determine how the agency will  determine its risk tolerance with respect to loss of crew. We believe this  approach will reduce confusion and increase transparency.

We are sending copies of this report to NASA Administrator and  interested congressional committees. In addition, the report is available at  no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202)512-4841 or chaplainc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.

The objectives of our review were to assess (1) the extent to which the  contractors have made progress towards meeting the National  Aeronautics and Space Administration’s (NASA) certification  requirements and NASA’s plans to help ensure continued access to the  International Space Station (ISS); and (2) how NASA’s certification  process addresses safety of the contractors’ crew transportation systems.

To assess the contractors’ progress towards certification, we obtained  and reviewed program and contractor documents, including monthly and  quarterly updates from April 2017 through May 2018. We interviewed  program and contractor officials to discuss the contractors’ recent  progress, including upcoming events and any expected delays, and to  understand technical risks, potential consequences, and planned  mitigation activities. To identify total delays to date, we compared original  contract schedules to Boeing and SpaceX’s calendar year 2018 first  quarter proposed schedules, which are the most recent. Based on our  review of program and contractor documents, we defined the contractors’  key events as: the uncrewed and crewed flight tests, the design  certification reviews for each of those flights, and the certification  milestone. We selected the two flight tests for each contractor as key  events because they are intended to test key system capabilities,  including the ability to launch, dock with the ISS, and return safely to  Earth. We selected the design certification reviews because they verify  the contractors’ crew transportation systems’ capability to safely  approach, dock, mate, and depart from the ISS, among other  requirements. We selected the certification milestone because it  determines whether the crew transportation system meets the  Commercial Crew Program’s requirements.

To determine the extent to which contractors have delayed these key  events over time, we analyzed the contractors’ schedule data from the 13  quarterly progress reports to date, from first quarter 2015 through first  quarter calendar year 2018. We also obtained the results of the program’s  April 2018 schedule risk analysis. We presented the schedule analysis  range from the end of the month of the earliest possible completion date  to the end of the month of the latest possible completion date. We  reviewed the program’s Congressional requirements to report on cost,  schedule, and technical status. Finally, to assess the potential effects of  any certification delays on NASA’s access to ISS, we reviewed NASA’s  contracts with Boeing and the Russian Federal Space Agency for  transportation on the Soyuz vehicle. We interviewed officials from the ISS  program and NASA’s Human Exploration and Operations Mission  Directorate to determine if the agency had developed contingency plans  to mitigate the effects of any certification delays on its access to the ISS.

To assess how NASA’s certification process addresses safety of the  contractors’ crew transportation systems, we reviewed agency safety  policies, program plans, and contract documents to identify what safety  assessments were required of the program, which safety factors would be  considered in certification reviews, and when certification approval would  be granted. We also interviewed program officials and the contractors  about their safety policies and procedures as well as about the  certification process. We identified the loss of crew requirements for the  program and the contractors, and interviewed program and agency  officials to determine how loss of crew would be assessed and  considered throughout the certification process. To gain a broader  understanding of the relative importance of loss of crew, we reviewed  NASA safety policies, prior GAO reports, and annual reports from the  Aerospace Safety Advisory Panel. To determine how the Orbital Debris  Engineering Model (ORDEM) was updated and to assess differences  between the former and updated models, we reviewed NASA  documentation about the ORDEM 2000 and ORDEM 3.0 models and  obtained information about the models from NASA’s Orbital Debris  Program Office.

To assess the program’s progress in closing requirements, including  those requirements specifically related to safety, we reviewed program  data for each contractor on contract requirements and closure status. We  limited this analysis to requirements included in the CCT-REQ-1130 ISS  Crew Transportation and Services Requirements Document because  these requirements are verified by the Commercial Crew Program,  whereas requirements contained within SSP 50808 ISS to Commercial  Orbital Transportation Services Interface Requirements Document are  managed by the ISS Transportation and Integration Office. We classified  requirements as “safety-specific” when the program’s Office of Safety and  Mission Assurance was listed as a verification closure notice signatory  (i.e., approver). We then analyzed the program’s data to determine how  many verification closure notices had been approved for all requirements  and for the subset of safety-specific requirements. We reviewed program  and agency documentation, such as organizational charts, program  plans, and safety policies as well as interviewed program and agency  officials, to determine the role of the safety and mission assurance  technical authority in the program. We also reviewed the 2003 Columbia  Accident Investigation Board’s Report’s findings and recommendations  related to culture and organizational management of human spaceflight  programs. We reviewed annual briefings and reports and met with  representatives from two organizations that provide NASA with  independent assessments of the program, the program’s standing review  board and the Aerospace Safety Advisory Panel, to gain their  perspectives on the contractor’s progress and how NASA addresses  safety in its certification process. We also met with representatives from  the National Transportation Safety Board and three experts with  background on safety in human spaceflight in order to increase our  contextual understanding of the role of safety in human spaceflight  missions.

We conducted this performance audit from April 2017 to July 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Comments from the National Aeronautics and Space Administration

Appendix III: GAO Contact and Staff Acknowledgments

Cristina T. Chaplain, (202) 512-4841 or chaplainc@gao.gov.

In addition to the contact named above, Molly Traci, Assistant Director;  Kazue Chinen; Lorraine Ettaro; Lisa Fisher; Laura Greifner; Kurt Gurka;  Miranda Riemer; Juli Steinhouse; Roxanna T. Sun; Hai Tran; Kristin Van  Wychen; and Alyssa Weir made significant contributions to this report.

NASA contracted with two companies, Boeing and SpaceX, to develop vehicles to transport astronauts to the International Space Station. Neither is expected to be ready until 2019.

Before any missions happen, NASA will have to certify that both contractors' vehicles are safe for human spaceflight. One way that NASA will assess safety is the loss of crew metric, which captures the probability of a crew member's death or disability. However, NASA doesn't have a consistent approach for calculating this metric, so results can vary based on who within NASA is conducting the analysis.

We recommended that NASA clarify how it will assess loss of crew.