James Webb Space Telescope: Opportunity Nears to Provide Additional Assurance That Project Can Meet New Cost and Schedule Commitments

JWST, a large, deployable telescope, is one of NASA's most complex projects and top priorities. The project has delayed its planned launch three times since September 2017 due to problems discovered in testing. In June 2018, NASA approved new cost and schedule estimates for JWST. Since the project established its cost and schedule baselines in 2009, the project's costs have increased by 95 percent and the launch date has been moved back by 81 months.

Conference Report No. 112-284, accompanying the Consolidated and Further Continuing Appropriations Act, 2012, included a provision for GAO to assess the project annually and report on its progress. This is the seventh report. This report assesses (1) the considerations NASA took into account when updating the project's cost and schedule commitments and (2) the extent to which NASA has taken steps to improve oversight and performance of JWST, among other issues. GAO reviewed relevant NASA policies, analyzed NASA and contractor data, and interviewed NASA and contractor officials.

In June 2018, the National Aeronautics and Space Administration (NASA) revised the cost and schedule commitments for the James Webb Space Telescope (JWST) to reflect known technical challenges, as well as provide additional time to address unanticipated challenges. For example, the revised launch readiness date of March 2021 included 5.5 months to address a design issue for the cover of the sunshield (see image). The purpose of the sunshield is to protect the telescope's mirrors and instruments from the sun's heat. NASA found that hardware on the cover came loose during testing in April 2018. The new cost estimate of $9.7 billion is driven by the schedule extension, which requires keeping the contractor's workforce on board longer than expected.

Before the project enters its final phase of integration and test, it must conduct a review to determine if it can launch within its cost and schedule commitments. As part of this review, the project is not required to update its joint cost and schedule confidence level analysis—an analysis that provides the probability the project can meet its cost and schedule commitments—but government and industry cost and schedule experts have found it is a best practice to do so. Such analysis would provide NASA officials with better information to support decisions on allocating resources, especially in light of the project's recent cost and schedule growth.

NASA has taken steps to improve oversight and performance of JWST, and identified the JWST project manager as responsible for monitoring the continued implementation of these changes. Examples of recent changes include increasing on-site presence at the contractor facility and conducting comprehensive audits of design processes. Sustaining focus on these changes through launch will be important if schedule pressures arise later and because of past challenges with communications. GAO will follow up on the project's monitoring of these improvements in future reviews.

GAO recommends NASA update the project's joint cost and schedule confidence level analysis. NASA concurred with the recommendation made in this report.

The James Webb Space Telescope (JWST) is one of the National  Aeronautics and Space Administration’s (NASA) most complex projects  and top priorities. It is intended to revolutionize our understanding of star  and planet formation and advance the search for the origins of our  universe. The innovative technologies within the telescope, as well as the  sheer size of some of its components—such as the tennis-court-sized  sunshield—illustrate some of the immense development challenges. The  project has experienced significant cost and schedule growth in recent  years due to problems discovered during integration and testing. In June  2018, NASA approved new cost and schedule commitments for JWST  with a cost of $9.7 billion and a launch readiness date of March 2021.  This represents a total of 95 percent of cost growth and 81 months of  schedule delays since the project’s cost and schedule baselines were first  established in 2009.

Conference Report No. 112-284 included a provision for GAO to assess  the JWST program annually and to report to the Committees on  Appropriations on key issues relating to program and risk management,  achievement of cost and schedule goals, program technical status, and  oversight mechanisms. This report is our seventh in response to that  provision. For this report, we assessed (1) the considerations that NASA  took into account when updating the JWST project’s cost and schedule  commitments, (2) the technical and integration challenges, if any, that are  affecting the execution of the JWST project, and (3) the extent to which  NASA has taken steps to improve oversight and performance of the  JWST project.

To address these objectives, we examined the schedule, technical, and  cost performance of the project since our last report in February 2018.  To determine the considerations that NASA took into account when  updating the JWST project’s cost and schedule commitments, we  analyzed the JWST project’s proposed changes to its 2011 rebaseline.  We compared the project’s prior cost and schedule baseline to the  project’s new cost and schedule estimates to determine what changes  were made, including changes to workforce projections. In addition, we  compared the process the project used to update cost and schedule  estimates with NASA guidance and policies on updating cost and  schedule estimates and best practices identified in GAO’s Cost  Estimating and Assessment Guide. We discussed the project’s cost and  schedule estimates from the 2018 replan with NASA, Northrop Grumman  Aerospace Systems (Northrop Grumman), and the James Webb Space  Telescope Independent Review Board (IRB) and interviewed officials  within the Office of the Chief Financial Officer on relevant NASA  guidance.

To assess the extent to which technical challenges and integration risks  are affecting the execution of the JWST project, we reviewed project and  contractor schedule documentation to identify the progress made and any  challenges faced since our last report in February 2018. We examined  monthly project status reports to management to monitor schedule  reserve levels and usage, identify potential risks and technical challenges  that may affect the project’s schedule, and gain insights on the project’s  progress. We interviewed project officials at Goddard Space Flight Center  and contractor officials from Northrop Grumman and the Association of  Universities for Research in Astronomy’s Space Telescope Science  Institute concerning the challenges that have affected schedule, and the  project’s and contractor’s plans to address these challenges. We also  discussed what benefits exist, if any, of conducting cost and schedule risk  analyses during integration and test with government and industry cost  and schedule experts. Further, we attended two project reviews, where  project and program officials briefed NASA headquarters officials on the  current status of the project. We examined project risks from monthly risk  registers to understand the likelihood of occurrence, the potential effect  on cost and schedule, and the steps the project plans to take to mitigate  the risks. We also requested updates on the status of project risks and  top issues that affected technical challenges the project was working to  mitigate during the course of our review. We also obtained information on  the status of project software and challenges from independent NASA  reviewers with NASA’s Independent Verification and Validation program.

Finally, to assess the extent to which NASA has taken steps to improve  oversight and performance of the JWST project, we reviewed  documentation on changes that NASA made to contractor oversight and  project oversight activities between April and November 2018. We also  reviewed the May 2018 IRB final report. We discussed these changes  and the implementation of related IRB recommendations with Goddard  Space Flight Center and NASA headquarters project officials; Northrop  Grumman and Space Telescope Science Institute contractor officials; IRB  members; and Defense Contract Management Agency officials who are  responsible for providing oversight of software and quality assurance  processes at the Northrop Grumman facility on behalf of NASA. In  addition, we reviewed project and Northrop Grumman documentation that  described or tracked changes to oversight and implementation of IRB  recommendations. We also reviewed award fee documentation to  examine NASA’s determination of the percentage of the available award  fee Northrop Grumman earned in two award fee periods—April 1, 2017 to  September 30, 2017 and October 1, 2017 to March 31, 2018. These  award fee periods are inclusive of the time period in which NASA first  started identifying that additional schedule slips would be likely. We  compared the findings and recommendations of a 2010 independent  review report to the 2018 independent review report to determine whether  there were similar findings or recommendations. Further, we discussed  challenges of workmanship errors and steps projects and companies can  take to avoid them with an expert on quality in NASA’s Office of Safety  and Mission Assurance and experts at the Aerospace Corporation.

Our work was performed primarily at NASA headquarters in Washington,  D.C.; Goddard Space Flight Center in Greenbelt, Maryland; and Northrop  Grumman Corporation in Redondo Beach, California. We met with  officials from the Space Telescope Science Institute in Baltimore,  Maryland, which will be responsible for operating the observatory on  behalf of NASA, among other things. We also obtained information from  officials at the Independent Verification and Validation facility in Fairmont,  West Virginia. This facility is responsible for providing independent review  of project safety and mission critical software.

We conducted this performance audit from April 2018 to March 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

JWST is envisioned to be a large deployable space telescope, optimized  for infrared observations, and the scientific successor to the aging Hubble  Space Telescope. JWST is being designed for a 5-year mission to find  the first stars, study planets in other solar systems to search for the  building blocks of life elsewhere in the universe, and trace the evolution of  galaxies from their beginning to their current formation. JWST is intended  to operate in an orbit approximately 1.5 million kilometers—or 1 million  miles—from the Earth. With a 6.5-meter primary mirror, JWST is  expected to operate at about 100 times the sensitivity of the Hubble  Space Telescope. JWST’s science instruments are designed to observe  very faint infrared sources and therefore are required to operate at  extremely cold temperatures. To help keep these instruments cold, a  multi-layered tennis court-sized sunshield is being developed to protect  the mirrors and instruments from the sun’s heat.

The JWST project is divided into three major segments: the observatory  segment, the ground segment, and the launch segment. When complete,  the observatory segment of JWST is to include several elements (Optical  Telescope Element (OTE), Integrated Science Instrument Module (ISIM),  and spacecraft) and major subsystems (sunshield and cryocooler). The  hardware configuration referred to as OTIS was created when the Optical  Telescope Element and the Integrated Science Instrument Module were  integrated. Additionally, JWST is dependent on software to deploy and  control various components of the telescope, and to collect and transmit  data back to Earth. The elements, major subsystems, and software are  being developed through a mixture of NASA, contractor, and international  partner efforts. See figure 1 for the elements and major subsystems of  JWST and appendix 1 for more details, including a description of the  elements, major subsystems, and JWST’s instruments.

For the majority of work remaining, the JWST project is relying on two  contractors: Northrop Grumman and the Association of Universities for  Research in Astronomy’s Space Telescope Science Institute. Northrop  Grumman plays the largest role, developing the sunshield, the Optical  Telescope Element, the spacecraft, and the Mid-Infrared Instrument’s  cryocooler, in addition to integrating and testing the observatory. Space  Telescope Science Institute’s role includes soliciting and evaluating  research proposals from the scientific community, and receiving and  storing the scientific data collected, both of which are services that it  currently provides for the Hubble Space Telescope. Additionally, the  Institute is developing the ground system that manages and controls the  telescope’s observations and will operate the observatory on behalf of  NASA. JWST will be launched on an Ariane 5 rocket, provided by the  European Space Agency.

JWST depends on 22 deployment events—more than a typical science  mission—to prepare the observatory for normal operations on orbit. For  example, the sunshield and primary mirror are designed to fold and stow  for launch and deploy once in space. Due to its large size, it is nearly  impossible to perform deployment tests of the fully assembled  observatory, so the verification of deployment elements is accomplished  by a combination of lower level component tests in flight-simulated  environments; ambient deployment tests for assembly, element, and  observatory levels; and detailed analysis and simulations at various levels  of assembly.

Schedule and Cost Reserves for NASA Projects

We have previously found that complex development efforts like JWST  face numerous risks and unforeseen technical challenges, which can  often become apparent during integration and testing. To accommodate  unanticipated challenges and manage risk, projects reserve extra time in  their schedules, which is referred to as schedule reserve, and extra funds  in their budgets, which is referred to as cost reserve.

Schedule reserve is allocated to specific activities, elements, and major  subsystems in the event of delays or to address unforeseen risks. Each  JWST element and major subsystem has been allocated schedule  reserve. When an element or major subsystem exhausts schedule  reserve, it may begin to affect schedule reserve on other elements or  major subsystems whose progress is dependent on prior work being  finished for its activities to proceed.

Cost reserves are additional funds within the project manager’s budget  that can be used to address unanticipated issues for any element or  major subsystem, and are used to mitigate issues during the development  of a project. For example, cost reserves can be used to buy additional  materials to replace a component or, if a project needs to preserve  schedule reserve, reserves can be used to accelerate work by adding  shifts to expedite manufacturing. NASA’s Goddard Space Flight Center— the NASA center with responsibility for managing JWST—has issued  procedures that establish the requirements for cost and schedule  reserves. In addition to cost reserves held by the project manager,  management reserves are funds held by the contractors that allow them  to manage program risks and to address unanticipated cost increases  throughout development. We have previously found that management  reserves should contain 10 percent or more of the cost to complete a  project and are generally used to address various issues tied to the  contract’s scope.

NASA’s cost-plus-award-fee contract with Northrop Grumman has  spanned almost two decades, during which there have been significant  variances in contractor performance. Cost-reimbursement contracts are  suitable when uncertainties in the scope of work or cost of services  prevent the use of contract types in which prices are fixed, known as  fixed-price contracts. Award fee contracts provide contractors the  opportunity to obtain monetary incentives for performance in designated  areas identified in the award fee plan. Award fees may be used when key  elements of performance cannot be defined objectively, and, as such,  require the project officials’ judgment to assess contractor performance.  For JWST’s contract with Northrop Grumman, these areas include cost,  schedule, technical, and business management and are established in  the contracts’ performance evaluation plans.

In December 2013, the JWST program and the contractor agreed to  replace a $56 million on-orbit incentive—incentives based on successful  performance in space—with award fees. The award fees are to  incentivize cost and schedule performance during development. This  shift increased the available award fee for the entire contract to almost a  quarter of a billion dollars. According to officials, restructuring the  incentives gave NASA more flexibility to incentivize the contractor to  prioritize the cost and schedule performance over exceeding technical  requirements. In December 2014, we found that NASA award fee letters  of award fee periods from February 2013 to March 2014 indicated that the  contractor had been responsive to interim award fee period criteria  provided by NASA and that contractor officials confirmed that they pay  close attention to this guidance in prioritizing their work. For example,  Northrop Grumman officials reported that they had made specific  changes to improve communications in direct response to this guidance,  which was validated by award fee letters from NASA.

History of Cost Growth and Schedule Delays

The JWST program has a history of significant schedule delays and  increases to project costs, which resulted in replans in 2011 and 2018.  Before 2011, early technical and management challenges, contractor  performance issues, low levels of cost reserves, and poorly phased  funding caused the JWST program to delay work. As a result, the  program experienced schedule overruns, including launch delays, and  cost growth. The JWST program underwent a replan in September  2011, and a rebaseline in November of that same year, and Congress  placed an $8 billion cap on the formulation and development costs for the  project. On the basis of the replan, NASA rebaselined JWST with a life- cycle cost estimate of $8.835 billion, which included additional money for  operations and a planned launch in October 2018.

Congress also required that NASA treat any cost increase above the cap  according to procedures established for projects that exceed their  development cost estimates by at least 30 percent. This process is known  as a rebaseline. Congress must authorize continuation of the JWST  program if formulation and development costs increase over the $8 billion  cost cap.

In June 2018, after a series of launch delay announcements due to  technical and workmanship issues identified during spacecraft element  integration, NASA notified Congress that it had again revised the JWST  program’s cost and schedule estimates. NASA estimated that it now  required $828 million in additional resources and 29 more months to  complete beyond those estimates agreed to in the 2011 rebaseline. As of  November 2018, NASA had funding to continue to execute the program  and was waiting to see if Congress would authorize the program’s  continuation and appropriate funds for the program in fiscal year 2019.  Figure 2 shows the project’s history of changes to its cost or schedule  and key findings from two external independent review teams and our  prior work.

As discussed above, various technical and workmanship errors drove  some of the more recent delays. Examples of some of the workmanship  issues we found in the past include:  In October 2015, the project reported that a piece of flight hardware  for the sunshield’s mid-boom assembly was irreparably damaged  during vacuum sealing in preparation for shipping. The damaged  piece had to be remanufactured, which consumed 3 weeks of  schedule reserve.

In April 2017, a contractor technician applied too much voltage and  irreparably damaged the spacecraft’s pressure transducers,  components of the propulsion system that help monitor spacecraft fuel  levels. The transducers had to be replaced and reattached in a  complicated welding process. At the same time, Northrop Grumman  also addressed several challenges with integrating sunshield  hardware. These issues combined took up another 1.25 months of  schedule reserve.

In May 2017, some of the valves in the spacecraft propulsion system’s  thruster modules were leaking beyond permissible levels. Northrop  Grumman determined that the most likely cause was the use of an  improper cleaning solution, and the thruster modules were returned to  the vendor for investigation and refurbishment. Reattaching the  refurbished modules was expected to be complete by February 2018,  but was delayed by one month when a technician applied too much  voltage to one of the components in a recently refurbished thruster  module. NASA and Northrop Grumman reported that resolving the  thruster module issue resulted in a 2-month delay to the project’s  overall schedule.

In October 2017, when conducting folding and deployment exercises  on the sunshield, Northrop Grumman discovered several tears in the  sunshield membrane layers. According to program officials, a  workmanship error contributed to the tears. The tears resulted in  another 2-month delay to the project’s overall schedule.

In addition, some first-time efforts took longer than planned. For example,  in fall 2017, the project determined that it would need to use up to 3  months of schedule reserve based upon lessons learned from the  contractor’s initial sunshield folding operation. This first deployment, or  unfolding, took 30 days longer than planned. The sunshield has since  undergone another deployment, and will be deployed twice more before  launch.

The IRB took into account these technical and workmanship errors, as  well as other considerations, when it analyzed the project’s organizational  and technical issues. The board’s final report, issued in May 2018,  included 31 recommendations that addressed a range of factors. For  example, the IRB recommended that the project:

Conduct an audit to identify potential embedded design flaws— problems that have not been detected through analysis, inspection, or  test activities and pose a significant risk to JWST schedule, cost, and  mission success;

Establish corrective actions to detect and correct human mistakes  during integration and test;

Establish a coherent, agreed-upon, and factual narrative on project  status and communicate that status regularly across to all relevant  stakeholders; and

Augment integration and test staff to ensure adequate long-term  staffing and improve employee morale.

In its response to the IRB’s report, NASA stated that it accepted the  report’s recommendations and had already begun implementing action in  response to many of them. Further, project officials told us that some of  the actions were underway before the IRB completed its review.

NASA Revised Schedule and Cost Commitments to Reflect Prior and Ongoing Technical Challenges

To develop a new schedule for JWST’s 2018 replan, NASA took into  account the remaining integration and test work and added time to the  schedule to address threats that were not yet mitigated. This includes 5.5  months to address an anomaly that occurred on the sunshield’s cover in  2018. The project also replenished its schedule reserves—which we  found in February 2018 had been consumed—so that they now exceed  the recommended levels. Both the project and IRB conducted schedule  risk assessments that produced similar launch dates. The project relied  on the replan schedule to determine its remaining costs because the  workforce necessary to complete the observatory represents most of the  remaining cost.

Following is additional information on the schedule and cost  considerations.

Schedule: JWST’s revised launch readiness date of March 2021 reflects  a consideration of the hardware integration and test challenges the  project has experienced, including adding time to:

Add snag guards for the membrane tensioning system—which helps  deploy the sunshield and maintain its correct shape—to prevent  excess cable from snagging,

Deploy, fold, and stow the sunshield, and

In addition, the project added extra time to the schedule to complete  repairs to the membrane cover assembly, which did not perform as  expected during acoustics testing in April 2018. The membrane cover  assembly shown in figure 3 is used to cover the sunshield membrane  when in the stowed position to provide thermal protection during launch.

After the anomaly occurred, the project halted spacecraft element testing,  investigated the anomaly, and found that the fasteners had come loose  due to a design change made to prevent the fasteners from damaging the  sunshield membrane. The design change caused the nuts to not lock  properly. According to project officials, due to the design of the membrane  cover assembly, the project was not able to conduct flight-like, stand- alone testing on the cover prior to spacecraft element testing. As a result,  the project did not discover the design issue until the hardware came  loose while installed on the spacecraft element. The project determined  that the repairs would take approximately 5.5 months.

The project’s replan also reflected schedule reserves above the level  required by Goddard Space Flight Center policy, which would have been  approximately 5 months at that time. The new schedule includes a total  of 293 days or 9.6 months of schedule reserves leading up to its  committed launch readiness date of March 2021. NASA approved a  JWST launch date of March 2021, but the project and the contractor are  working toward a launch date in November 2020. Figure 4 shows the  project’s new schedule following the 2018 replan, including how the  project distributed its schedule reserves through different integration and  test activities.

As part of its May 2018 study, the IRB reviewed the project’s schedule  and recommended a launch date of March 2021, which was subsequently  reflected in NASA’s new schedule for the program. In reviewing the  project’s schedule, the IRB found that the project had robust scheduling  practices for ensuring that the schedule represented a complete and  dynamic network of tasks that could respond automatically to changes.

This schedule also passed a standard health check with minimal errors  indicating that it was well constructed. However, the IRB noted that this  schedule does not account for certain types of unknown risks to the  program such as integration and test errors which can take many months  to resolve, or the potential need to remove a science instrument from the  observatory, which can have about a 1 year impact. As a result, the  program could experience additional delays if a risk of this magnitude is  realized.

Cost: The project’s new $9.7 billion life-cycle cost estimate is principally  driven by the schedule extension, which requires keeping the contractor’s  workforce to complete integration and test longer than expected.  Specifically, the project determined that almost all of the hardware had  been delivered and the remaining cost was predominantly the cost for the  workforce necessary to complete and test the observatory.

For the past 3 years, we have reported that Northrop Grumman’s ability to  decrease its workforce was central to JWST’s capacity to meet its long- term cost commitments. However, Northrop Grumman’s actual  workforce continued to exceed its projections. This was because it  needed to maintain higher workforce levels due to technical challenges,  including problems with spacecraft and sunshield integration and test. It  also needed to keep specialized engineers available when needed during  final assembly and test activities.

In developing the cost estimate supporting the 2018 replan, the project  used a Northrop Grumman workforce profile that is higher than previous  projections because Northrop Grumman now plans to maintain personnel  longer during integration and test. According to project officials, the  planned reduction of Northrop Grumman’s workforce is now more gradual  and conservative than the prior plan. For example, the Northrop  Grumman workforce will not start to significantly decline until the  observatory ships to the launch site, which is expected to occur in August  2020. As shown in Figure 5, the JWST workforce assembling the  observatory declines and the government and contractor workforce  necessary to manage and operate the observatory remains after the  internal launch readiness date of November 2020.

As seen in the above figure, the Space Telescope Science Institute  workforce, the contractor responsible for operating JWST, will remain  generally flat between fiscal years 2021 to 2026 when it operates the  observatory. The NASA civil service and support contractor will remain  relatively flat through November 2020 launch date and then decline. In  addition, the new cost estimate also took into account $61 million for  implementing the IRB recommendations and mission success  enhancements, funding for project cost reserves, and operations costs.

In June 2018, the NASA associate administrator—who is the project’s  decision authority—approved the project to proceed with its replan with a  March 2021 launch date and $9.7 billion in life-cycle costs based on the  Agency Program Management Council review and replan documents.  The associate administrator did not require the project to conduct an  updated Joint Cost and Schedule Confidence Level (JCL) analysis for this  replan. A JCL is an integrated analysis of a project’s cost, schedule, risk,  and uncertainty whose result indicates the probability of a project’s  success of meeting cost and schedule targets. NASA policy states that a  JCL should be recalculated and approved as a part of the rebaselining  approval process, but it is not required. In its replan decision memo,  NASA’s associate administrator explained that he did not require the  project to update the JCL because project costs are almost entirely  related to the workforce and most of the remaining planned activities will  be performed generally in sequence. Therefore, according to NASA’s  associate administrator, the total cost would be driven almost entirely by  the schedule because the workforce levels will remain the same through  delivery of the observatory. Both the project and independent estimators  used multiple schedule estimating methods to analyze the schedule for  the remaining work, and NASA’s associate administrator said these  analyses returned consistent, high confidence launch dates.

Project Has Used Some Schedule Reserve from Its 2018 Replanned Schedule with Challenging Integration and Test Work Remaining

The project’s ability to execute to its new schedule will be tested as it  progresses through the remainder of challenging integration and test  work. The project has yet to complete three of five integration and test  phases. The remaining phases include integration and test of OTIS, the  spacecraft element, and the observatory. Our prior work has shown that  integration and testing is the phase in which problems are most likely to  be found and schedules tend to slip. For a uniquely complex project  such as JWST, this risk is magnified as events start to become more  sequential in nature. As a result, it will continue to become more difficult  for the project to avoid schedule delays by mitigating issues in parallel.

As of November 2018, the project is about a week behind its replanned  schedule because repairs on the membrane cover assembly took longer  than planned. Completing the membrane cover assembly repairs and  returning the spacecraft to vibration testing was a key event for the  project to demonstrate that it could execute to its new schedule. When  the project developed its 2018 replanned schedule, it had planned to  complete the membrane cover assembly repairs and reinstall the  assembly onto the sunshield and restart spacecraft element integration  and test activities by November 6, 2018. The project allocated 4 weeks of  schedule reserves specifically for these repairs. However, the membrane  cover repairs proved more difficult than anticipated. For example, the  program had to address unanticipated technical challenges on the  membrane cover assemblies, including repairing tears and pin holes in  the covers discovered after the covers were removed. The project also  had to allot time to install bumpers, which are kapton tubes, to the  assembly to protect the composite material on a sunshield structure  during launch. The project identified the need to add the bumpers during  subassembly vibration testing.

As a result, as of November 2018, the project had used about 4.5 weeks  of schedule reserves to cover delays associated with these activities. The  use of reserves beyond what the project had planned for the repairs  pushed the restart of spacecraft element integration and test activities out  about a week to November 14, 2018. Figure 6 compares the project’s  initial membrane cover assembly schedule in June 2018 to the actual  schedule in November 2018.

While the project repaired the membrane cover assembly, it also used  this time to conduct risk mitigation activities on OTIS. For example, the  project worked to mitigate a design issue on the frill connections. The frill  is composed of a single layer of blankets placed around the outside of the  primary mirror used to block stray light (see figure 7).

A combination of modeling and inspections revealed that most of the frill  sections did not have as much slack as expected at the near-absolute  zero cryogenic temperatures of space. This caused shrinkage that put  stress on the edges of the outer ring of mirrors, which could affect the  stability of the optical mirror and image quality. The project loosened  these outer connections by adding a ring to the connecting points. As of  November 2018, project officials said they were in the process of verifying  the fix through inspections.

Examples of technical issues and risks that the project continues to face  during the remaining phases of integration and test include:

The project is working to mitigate a design issue on the sunshield  membrane tensioning system—which helps deploy the sunshield and  maintain its correct shape. In our February 2018 report, we found that  Northrop Grumman was planning to modify the design of the  membrane tensioning system after one of the sunshield’s six  membrane tensioning systems experienced a snag when conducting  folding and deployment exercises on the sunshield in October 2017.  The project and Northrop Grumman determined that a design  modification was necessary to fully mitigate the issue, which includes  modifying clips used to progressively release the cable tension and  adding guards to control the excess cable.

The project identified a concern that the depressurization of trapped  air in the folded sunshield membrane when the fairing separates to  release the JWST observatory may overly stress the membrane  material. The project is working with Arianespace—the company  responsible for operating JWST’s launch vehicle—and experts at the  Kennedy Space Center to resolve this concern. Officials estimated  that a design solution would be in place in mid-2019. However, if the  project determines that it needs to reinforce the membrane covers to  survive excessive residual pressure as it works on this design  solution, a multi-month schedule delay could occur.

As of November 2018, the project has mitigated 21 of its 47 hardware  and software risks to acceptable levels, and reviews these risks  monthly for any changes that might affect the continued acceptability  of the risk. Five of these 21 risks are related to the project’s more  than 300 potential single point failures—several of which are related  to the deployment of the sunshield. The project is actively working to  mitigate the remaining 26 risks to acceptable levels or closure prior to  launching.

The project also has several first-time and challenging integration and test  activities remaining. For example, the project must integrate OTIS and the  completed spacecraft element and test the full observatory in the final  integration phase, which includes another set of challenging  environmental tests. See figure 8 for an image of OTIS and the spacecraft  element prior to being integrated.

As previously discussed, the project also has two remaining deployments  of the sunshield, and prior deployments have taken longer than planned.  To help mitigate the risks associated with the deployments, the project  added additional time for deployments in the 2018 replanned schedule  based on lessons learned from prior deployments. The two remaining  deployments are to occur after spacecraft element integration and test  and again after observatory integration and test.

The JWST project office is required to evaluate whether the project can  complete development within its revised cost and schedule commitments  at its next major review—the system integration review—planned for  August 2019. This review is to occur after the project has completed two  major tasks—OTIS and spacecraft element integration and test. The  review is to evaluate whether the project (1) is ready to enter observatory  integration and test, and (2) can complete remaining project development  with acceptable risk and within its cost and schedule constraints. NASA  guidance does not require projects to conduct a JCL at this review.  However, project officials said that they plan to conduct another schedule  risk analysis in the future. They do not intend to complete a new JCL for  the same reasons they did not complete one for the 2018 replan— because costs are almost entirely related to the workforce and can be  derived from a schedule that takes into account known risk.

While not required, conducting a JCL prior to the system integration  review would inform NASA about the probability of meeting both its cost  and schedule commitments. If the project proceeds with its plan to  conduct only a schedule risk analysis, NASA would be provided only with  an updated probability of meeting its schedule commitments. Our cost  estimating best practices recommend that cost estimates should be  updated to reflect changes to a program or kept current as it moves  through milestones and as new risks emerge. In addition, government  and industry cost and schedule experts we spoke with noted that  integration and testing is a critical time for a project when problems can  develop. These experts told us that completing a JCL is a best practice  for analyzing major risks at the most uncertain part of project execution.  Conducting a JCL at system integration review—a review that occurs  during the riskiest phase of development, the integration and test phase— would allow the project to update its assumptions of risk and uncertainty  based on its experiences in OTIS and spacecraft element integration and  test. The project could then determine how those updated assumptions  affect overall cost and schedule for the JWST project.

As noted above, the project has many risks to mitigate, technical  challenges to overcome, and challenging test events to complete, which  could affect the project’s schedule and risk posture. Further, the project  has an established history of significant cost growth and schedule delays.  In its June 2018 letter notifying an appropriate congressional committee  of its updated cost and schedule commitments, NASA acknowledged that  recent cost growth for the project will likely impact other science missions.  Conducting a JCL at system integration review would provide NASA and  Congress with critical information for making informed resource decisions  on the JWST project and its affordability within NASA’s portfolio of  projects more broadly.

NASA Is Augmenting Oversight of Contractor and Project Performance, and Identified the JWST Project Manager as Responsible for Sustaining Changes

NASA has taken steps to augment oversight of the contractor and project  following the discovery of the embedded design flaws and workmanship  errors that contributed to the project’s most recent schedule delays and  cost increases. See table 1 for examples of changes NASA has made to  contractor and project oversight—some of which NASA self-identified and  others that were in response to IRB recommendations. The IRB made 31  recommendations that ranged from improving employee morale to  improving security during transporting JWST to its launch site.

NASA has also used award fees to try to incentivize Northrop Grumman  to improve its performance. In a July 2018 hearing on the JWST program  before the House Science, Space, and Technology Committee,  Administrator Bridenstine stated that NASA had reduced the available  award fee through commissioning by $28 million out of a total of about  $60 million. Northrop Grumman also did not earn its full award fee in the  two most recent periods of performance that NASA assessed.

For the performance period of April 1, 2017 to September 30, 2017,  Northrop Grumman earned approximately 56 percent of the available  award fee. Reasons that NASA cited for its evaluation of award fees  in this period included workmanship errors on the propulsion system,  schedule delays, as well as issues with schedule execution,  management, and quality control.

For the period of October 1, 2017 to March 31, 2018, Northrop  Grumman earned none of the available award fee. Northrop  Grumman’s overall score was driven by an “unacceptable” rating in  schedule and cost due to delays and in anticipation of exceeding the  project’s $8 billion cost cap. Northrop Grumman received an  “excellent” rating under the technical category, but the evaluation  noted ongoing issues with quality controls, which resulted in delays.  For example, the process steps for applying voltage to the  spacecraft’s pressure transducers were not clear enough, which  resulted in technician error and irreparable damage to the hardware.

According to Northrop Grumman officials, the contractor has started to  take action to try to improve its quality assurance processes. Officials  described actions that ranged from rewriting hardware integration and test  procedures to starting efforts to change aspects of the company’s culture  that contributed to quality control issues. For example, in July 2018,  Northrop Grumman initiated a JWST mission assurance culture change  campaign to increase focus on product quality and process compliance.  This effort includes having inspectors affirm by signature that they have  personally inspected, verified, and confirmed that all aspects of an activity  meet quality standards. According to the form instructions, if the inspector  is uncertain on compliance or if instructions are unclear, workers are to  halt work, investigate and assess the situation, and request help to  resolve the situation. Project and Northrop Grumman officials provided an  example of these changes working. During a manual deployment of a  radiator panel, a Northrop Grumman employee discovered that a flap  used as thermal protection for a radiator was installed incorrectly and  reported the error. Northrop Grumman technicians found that this flap had  been swapped with another flap in the process of moving them to be  installed and corrected the problem before work proceeded.

Further, NASA and Northrop Grumman are conducting audits to try to  minimize the risk of failures during the remaining phases of integration  and test. These audits are conducted on items that have not been fully  tested, are in workmanship-sensitive areas, or have had a late design  change. The first phase of the audit was completed in September 2018  and found no major design issues or hardware rework required. The  project plans to audit other areas through at least spring 2019, but will  add audits if needed.

The JWST oversight structure includes a number of positions that could  be responsible for ensuring that the recent augmentations to contractor  and project oversight are sustained through launch (see table 2).

In response to our review, NASA officials clarified that the project  manager has sole responsibility for ensuring that these improvements are  sustained through launch. Further, these officials stated that the project  office is responsible for monitoring these changes at the project level and  at Northrop Grumman. The project manager’s continued focus on these  efforts will be important because:

The project is implementing a wide span of improvement efforts,  ranging from more on-site coverage at the contractor facility to cultural  improvements, which will now need to be sustained for an additional  29 months.

The project has had recurring issues with effective internal and  external communication as well as defining key management and  oversight responsibilities, both of which are important to sustaining  oversight. For example, the Independent Comprehensive Review  Panel identified communication problems—between the JWST project  and Science Mission Directorate management as well as between  NASA and Northrop Grumman—and that the project’s governance  structure lacked clear lines of authority and accountability. In  December 2012, we found the JWST project had taken several steps  to improve communication—such as instituting meetings that include  various levels of NASA, contractor, and subcontractor management— but the IRB’s findings in 2018 indicate that communication and  governance issues have resurfaced in some areas. For example,  the IRB found that communication with key stakeholders including the  science community, Congress, and NASA leadership, has been  variable and at times inconsistent.

The project may encounter new schedule pressures as it proceeds  through integration and test. A senior NASA official with expertise in  workmanship issues told us that schedule pressure is a key reason for  increased quality problems on projects. For example, this official said  that companies tend to give experts leniency to operate without the  burden of quality assurance paperwork when schedule pressures  arise, which can lead to workmanship errors. While JWST project  officials told us they do not view this as applicable to their project, the  perspective regarding potential schedule pressures and workmanship  is important to keep focus on given the magnitude of technical  challenges and delays the project has faced.

We will continue to monitor the project’s efforts at maintaining these  oversight augmentations in future reviews, given that less than a year has  passed since the project began implementing many of them. Moreover,  the project may find that some actions will be required of officials outside  the project, particularly since the communication problems identified by  the IRB may well extend to headquarters’ interaction with stakeholders  from the science community, industry, and the Congress.

JWST is one of NASA’s most expensive and complex science projects,  and NASA has invested considerable time and resources on it. The  project first established its cost and schedule baseline in 2009. Since  then, the project made progress by completing two of five phases of  integration and test, but has also experienced significant cost growth and  schedule delays. However, the project did not complete a JCL analysis as  part of its second replan. Between now and its system integration review  planned for August 2019, the JWST program will have to continue to  address technical challenges and mitigate risks. Conducting a JCL would  better inform decision makers on the status of the project as they  determine whether the project can complete remaining project  development with acceptable risk and within its cost and schedule  constraints. Given the project is now on its third iteration of cost and  schedule commitments, conducting a JCL is a small step that NASA can  take to demonstrate it is on track to meet these new commitments.

We are making the following recommendation to NASA:  The NASA Administrator should direct the JWST project office to conduct  a JCL prior to its system integration review. (Recommendation 1)

We provided a draft of this report to NASA for comment. In written  comments, NASA agreed with our recommendation. NASA expects to  complete the JCL by September 2019, prior to the system integration  review. The comments are reprinted in appendix II. NASA also provided  technical comments, which have been addressed in the report, as  appropriate.

We are sending copies of this report to the appropriate congressional  committees, the NASA Administrator, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions on matters discussed in this report,  please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix III.

Appendix I: Elements and Major Subsystems of the James Webb Space Telescope (JWST) Observatory

Appendix II: Comments from the National Aeronautics and Space Administration

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Molly Traci (Assistant Director),  Karen Richey (Assistant Director), Jay Tallon (Assistant Director), Brian  Bothwell, Daniel Emirkhanian, Laura Greifner, Erin Kennedy, Jose  Ramos, Sylvia Schatz, Roxanna Sun, and Alyssa Weir made key  contributions to this report.

The James Webb Space Telescope, the planned successor to the Hubble Telescope, is one of NASA's most complex projects.

Since fall 2017, NASA has delayed the telescope's planned launch 3 times, due to factors like workmanship errors. NASA has taken some steps to ensure that the telescope can meet its new launch date of March 2021, such as increasing its oversight of contractors.

We recommended NASA update an analysis to determine the likelihood of meeting its new launch date and cost estimate for the James Webb Space Telescope.

NASA's acquisition management has been on our High Risk list since 1990 due to persistent cost and schedule growth.