Auto Recalls: NHTSA Should Take Steps to Further Improve the Usability of Its Website

The number of vehicles affected by safety defect recalls increased sharply in recent years—from nearly 13 million in 2011 to over 51 million in 2016. Once a defect is identified, auto manufacturers are required to send written notification to vehicle owners by mail. NHTSA also aims to enhance awareness of auto recalls by providing information on its website, NHTSA.gov .

The Fixing America's Surface Transportation Act includes a provision requiring GAO to study the use of publicly available safety recall information. This report addresses: (1) how consumers and industry stakeholders use such information and (2) how easy to use do consumers find the auto recall areas of NHTSA.gov, among other objectives. To understand consumers' use of auto recall information and to test website usability, GAO conducted 12 focus groups with 94 consumers who had a recall. Focus groups were held in six locations selected for population and geographic variation. GAO identified key website usability practices and requested an evaluation by website usability professionals. GAO reviewed statutes, regulations, and NHTSA documents, and interviewed industry stakeholders—including 10 manufacturers selected based on sales market share and other factors.

Consumers, manufacturers, and auto dealers use publicly available auto recall information differently. For example, the 94 consumers in 12 focus groups that GAO conducted used this information to decide whether to repair their vehicles. These consumers overwhelmingly cited safety risk and convenience as the two most influential factors they considered. Most consumers reported a preference for receiving recall notification by at least one electronic means, such as by e-mail or text message, in addition to mail. However, only 7 of 94 consumers reported receiving electronic notifications, suggesting a gap between the industry's auto recall notification practices and consumers' preferences. (See fig.). In response to a mandate in law, in September 2016, the National Highway Traffic Safety Administration (NHTSA) issued a proposed rule that, if finalized, would require manufacturers to notify consumers about auto recalls by electronic means in addition to mail.

Most consumers in GAO's focus group website usability tests found the auto recall areas of NHTSA's website—NHTSA.gov—easy to use; however, some consumers experienced difficulties when asked to complete auto recall related tasks. For example, when consumers attempted to search for recalls affecting their specific vehicles, some found the search results confusing, leading them to question the accuracy of the results. Similarly, some consumers were hampered in searching for recalls by their vehicles' year, make, and model because the website did not always display model options using plain language. GAO found that the auto recall areas of NHTSA.gov do not always reflect federal and industry key website usability practices, and that an independent evaluation conducted by website usability professionals at GAO's request identified similar issues. NHTSA is in the process of consolidating its websites and plans to conduct a website usability study of NHTSA.gov with consumers after the consolidation is complete. However, the agency has not determined a completion date for the consolidation effort—an essential step for organizations to effectively guide their information technology efforts. Without establishing a completion date and taking interim steps to improve the usability of NHTSA.gov, consumers will likely continue to experience difficulties, which may limit the effectiveness of the website's primary means of providing consumers with information about recalls affecting their vehicles.

GAO recommends that NHTSA determine a completion date for its website consolidation effort and take interim steps to improve the usability of NHTSA.gov by addressing the website usability difficulties GAO identified. The Department of Transportation concurred with the recommendations.

The number of vehicles affected by safety defect vehicle recalls (auto  recalls) has increased dramatically in recent years—rising from nearly 13  million in 2011 to over 51 million in 2016. This increase is driven in part by  several large-scale recalls, such as the ongoing recall of 34 million  vehicles with defective Takata air bag inflators that have caused at least  13 deaths and more than 220 injuries in the United States.

Auto recalls seek to address a wide variety of safety risks, such as wiring  system problems that may result in vehicle fires or windshield wiper  assemblies that fail to operate properly. When safety defects are  identified, auto manufacturers are required to notify consumers and  remedy (repair) the problem without charge. The National Highway Traffic  Safety Administration (NHTSA) monitors the percentage of defective  vehicles that manufacturers ultimately remedy (completion rates).  However, according to NHTSA’s Strategic Plan 2016–2020, auto recall  completion rates are unacceptably low, leaving vehicles with potentially  deadly safety defects on the road. In calendar year 2014—the latest year  for which data are complete—67 percent of recalled light vehicles had  been remedied. NHTSA aims to enhance the public’s awareness of auto  recalls and improve completion rates by, for example, providing  information on its website, NHTSA.gov  The Fixing America’s Surface Transportation Act (FAST Act) requires  GAO to study the use of publicly available safety recall information. This  report addresses the following objectives: (1) How do consumers and  industry stakeholders use publicly available auto recall information? (2)  How easy or difficult to use do consumers find the auto recall areas of  NHTSA.gov? (3) What steps, if any, has NHTSA taken to raise consumer  awareness about auto recalls and how has NHTSA evaluated the  effectiveness of these steps?

For purposes of this report we define publicly available auto recall  information to include information on the auto recall areas of NHTSA.gov,  such as examples of notification letters that manufacturers mail to  consumers. This report focuses on safety defect vehicle recalls affecting  passenger vehicles that are initiated when a defect in a vehicle or vehicle  equipment creates an unreasonable safety risk, as determined by NHTSA  or a manufacturer.

To inform the first two objectives, we conducted 12 focus groups with new  and used vehicle owners who had experienced an auto recall in the last  24 months. Each focus group was split into two sessions: (1) a discussion  session to explore participants’ thoughts, experiences, and preferences  about auto recall information and (2) a session to test the usability of the  auto recall areas of NHTSA.gov. We conducted focus groups at six  locations across the country to provide population and geographic  dispersion, with each group including 7 or 8 consumers for a total of 94  participants. Half of the focus groups were comprised of consumers who  had completed the repair and the remaining half included consumers who  had not completed the repair. We selected focus group participants based  on age, income, gender, education level, race, and ethnicity to ensure we  collected a range of perspectives.

We evaluated transcripts of each focus group session using  systematic content analysis to identify how consumers used auto  recall information and to evaluate how easy or difficult consumers  found the auto recall areas of NHTSA.gov to use.

We also analyzed participants’ responses to the questionnaires we  administered for each session to quantify responses regarding how  consumers used auto recall information—including how they received  and preferred to receive auto recall notifications—and participants’  assessments of NHTSA.gov’s usability while performing certain auto  recall tasks on the website.

Since we did not select a representative sample of participants, the  results of our focus group analyses are not generalizable to all vehicle  owners.

To determine how industry stakeholders use auto recall information, we  interviewed representatives from 10 auto manufacturers, selected based  on each manufacturer’s sales market share, place of ownership, and  experience with auto recalls, collecting a range of perspectives on how  manufacturers use auto recall information. We also interviewed four  franchised dealerships and three independent auto dealerships across  the country. The results of these interviews are not generalizable to all  auto manufacturers and dealerships, but provide insights about how  some industry stakeholders use auto recall information. To understand  NHTSA’s role in the auto recall process as well as that of other  stakeholders, we also interviewed NHTSA program officials, industry  groups, and consumer associations, among others.

To evaluate how easy or difficult consumers find the auto recall areas of  NHTSA.gov to use, we first analyzed guidance documents from NHTSA  and other federal agencies to identify key website usability practices. For  example, we analyzed the General Services Administration’s (GSA) and  the Department of Health and Human Services’ Research-Based Web  Design & Usability Guidelines, which includes quantified, peer-reviewed  guidelines intended to help federal agencies improve the design and  usability of their information-based websites. In addition, we reviewed  federal standards for internal control related to communicating quality  information externally. During our usability testing sessions, we asked  consumers to attempt to complete auto recall tasks—the primary means  NHTSA.gov provides for consumers to access information about auto  recalls affecting their vehicles—and discuss their experiences. We then  compared consumers’ experiences with the usability of the website  against these practices. To corroborate the results of our usability testing  sessions, we also requested that five website usability professionals from  GSA’s Federal User Experience Community conduct an independent  evaluation of the auto recall areas of NHTSA.gov.

To determine any steps NHTSA has taken to raise consumer awareness  about auto recalls and how NHTSA evaluates the effectiveness of any  steps, we reviewed relevant statutes, regulations, and agency  documents, such as NHTSA’s strategic planning and performance  reports. We reviewed performance management practices as provided in  the Government Performance and Results Act of 1993 (GPRA), the  GPRA Modernization Act of 2010, and standards for internal control in the  federal government to identify any opportunities for improvement. We  also interviewed responsible agency officials and discussed NHTSA’s  public awareness efforts during interviews with industry stakeholders  (described above). We analyzed the results of these interviews along with  the focus group discussions to identify perspectives on the effectiveness  of NHTSA’s public awareness steps. See appendix I for more information  on our objectives, scope, and methodology, including the organizations  we interviewed for this engagement.

We conducted this performance audit from October 2016 to December  2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Safety defect vehicle recalls (auto recalls) are initiated when a defect in a  vehicle or vehicle equipment creates an unreasonable safety risk, as  determined by NHTSA or a manufacturer. After a recall is initiated,  manufacturers are required to provide written notification to vehicle  owners via First-Class Mail within 60 days and remedy the defect.  Franchised dealers—which sell or lease an auto manufacturer’s new  vehicles—perform the recall remedy.

Before manufacturers send recall notification letters to affected vehicle  owners, NHTSA reviews draft letters and envelopes to ensure they  include required information about the safety defect. Required information  includes, among other things, a clear description of the safety defect, an  evaluation of the risk to vehicle safety, and a statement that the  manufacturer will remedy the defect without charge. See appendix II for  an example of a notification letter.

The number of vehicles affected by safety defect vehicle recalls has  increased dramatically since 2011 (see fig. 1). The increase reflects, in  part, several large-scale recalls. For example, in 2014, General Motors  initiated a recall of over 8 million vehicles with faulty ignition switches.  Similarly, according to NHTSA in 2014 and 2015, some passenger  vehicle manufacturers began recalling Takata air bag inflators, recalls that  have grown to include approximately 34 million vehicles and 19 auto  manufacturers. For the Takata recall, NHTSA issued various orders and  established a Coordinated Remedy Program under which the agency  oversees the supply of remedy parts and risk-based prioritization of  vehicles for repair, and manages related recalls with the assistance of an  Independent Monitor. The Independent Monitor assesses compliance  with the applicable orders issued by NHTSA and makes  recommendations aimed at enhancing the remedy program.

According to NHTSA’s Strategic Plan 2016–2020, this unprecedented  recall activity encouraged the agency to improve its system for identifying  and addressing defective vehicles. For example, the plan states that  NHTSA’s “vision is to achieve a 100-percent completion rate for every  recall by improving communication at every level, at every step of the  way.” Thus, according to the plan, NHTSA and the auto industry have  committed to identifying and implementing effective strategies to inform  consumers of safety defects and envision that their coordination will  bolster recall efforts to improve completion rates.

NHTSA reported that annual completion rates for passenger vehicle  recalls have remained relatively flat, ranging from 63 to 67 percent  between calendar year 2011 and calendar year 2014. See appendix III  for completion rates by vehicle component and vehicle type. In part, to  improve communication and encourage consumers to complete repairs,  NHTSA and manufacturers provide auto recall information to the public  on their websites. For example, certain motor vehicle manufacturers are  required to allow consumers to search a vehicle’s recall remedy status on  the Internet using the vehicle identification number (VIN). NHTSA also  provides publicly available auto recall information on its website, including  examples of recall notification letters.

In December 2016, NHTSA began consolidating its websites into  NHTSA.gov to provide a single access point for its auto recall content.  One of these websites, safercar.gov, was once NHTSA’s primary method  of communicating auto recall information to consumers; however, the  agency is in the process of moving this information to NHTSA.gov.  NHTSA’s Strategic Plan 2016–2020 states that the agency wants  NHTSA.gov to be a comprehensive user-friendly platform that serves as  the premier source of vehicle safety information by, for example,  improving the website’s search capabilities. NHTSA also aims to  encourage consumers to use its website’s auto recall information through  its communications program.

NHTSA’s Office of Communications and Consumer Information (OCCI) is  the primary office responsible for implementing the agency’s public  communication efforts. OCCI intends to increase public engagement with  the agency’s information through its social media channels, such as  Instagram, Twitter, and Facebook. The amount OCCI obligated to support  the agency’s auto recall efforts has increased from nearly $.5 million in  fiscal year 2011 to about $2.5 million in fiscal year 2016. According to  NHTSA officials, these obligations supported various efforts, including  public awareness campaigns, an auto recall hotline, advertising agencies,  exhibits at auto shows, and NHTSA’s mobile application.

Auto Recall Information Use Varies, and Most Consumers in Focus Groups Preferred Electronic Recall Notifications in Addition to Mail

Consumers in Our Focus Groups Primarily Considered Safety Risk and Convenience when Using Auto Recall Information to Make Repair Decisions

As part of our focus group discussion sessions, consumers selected  safety risk and convenience as the two most influential factors they  considered when using auto recall information to decide whether to  complete repairs.

All factors considered: During each session, we first asked consumers  to describe all the factors they considered. Across the sessions,  consumers shared a wide variety of factors including availability of a  loaner vehicle, time to schedule and complete the repair, safety risk,  and other factors. For example, some consumers had not yet repaired  their vehicles because they were “just waiting” for parts to become  available. Other consumers considered their previous customer  service experiences at the franchised dealership or the distance they  would need to travel to complete the repair. For example, one  consumer at our rural focus group location told us it would take  roughly 2 hours to reach the dealership’s repair shop.

Most influential factors considered: After the discussion of all factors,  we then asked each consumer to select the single most influential  factor they considered. Consumers in the sessions overwhelmingly  selected safety risk and convenience as the two most influential  factors (see table 1).

More than half of consumers in our focus group discussion sessions  selected safety risk as the most influential factor they considered when  making repair decisions. They told us that their perception of the risk  influenced whether or not they repaired their vehicle. For instance, some  consumers stated that they completed repairs immediately, because the  risks “sounded serious” or that they considered the defect a “safety  concern.” Conversely, some consumers said they did not complete the  repairs because the defect “didn’t sound very urgent.”

While each recall notification letter is required to include an evaluation of  the risk to vehicle safety reasonably related to the defect, consumers in  our focus group sessions shared mixed opinions about the quality and  clarity of safety risk information included in the notification letter they  received. For example, some consumers told us the letter’s safety risk  information seemed vague. For instance, one consumer told us the  letter’s description of the safety defect did not clearly state the chances of  an increased risk of injury and so he “had to figure out  on his  own.” In addition, some consumers commented that the safety risk  information could be more prominent in the notification letter, that the  letter could emphasize the severity of the risk, or that the letter could  describe the risk in simpler language. However, other consumers stated  the notification letter they received adequately described the recall’s  safety risk. In June 2011, we recommended that NHTSA modify the  requirements for defect notification letters to include additional information  to obtain readers’ attention. In 2013, NHTSA responded to our  recommendation by requiring manufacturers to include the statement  “IMPORTANT SAFETY RECALL” at the top of auto recall notification  letters.

Focus Group Participant’s Comment  “I don't want to be without a car for half the  day or stay  with my kids all  day.”

Consumers in our focus group discussion sessions selected convenience  as the second most influential factor they considered in making repair  decisions. While some consumers described the “hassle” of the repair  and being “too busy” to schedule and fix the defect, other consumers told  us they repaired their vehicles more easily because, for example, they  could take advantage of previously scheduled service appointments to  also repair the defect. Also, some consumers in our sessions stated that  the letter or notification they received could better address the  inconvenience of the recall, for example by including better estimates of  how long repairs might take. In addition, some consumers recommended  the letter include options for scheduling needed repairs.

As we discuss later in the report, NHTSA officials told us they continue to  work with auto manufacturers to identify ways to encourage consumers to  complete needed repairs, while representatives from some manufacturers  we met with described specific steps they have taken to address some of  the inconveniences consumers may experience in completing repairs. For  example, one manufacturer facilitated a pilot program for a third-party service provider in conjunction with dealers to repair vehicles at the  owner’s home or place of work, while another manufacturer told us they  work with individual dealers to hold events specifically for recall repairs  when consumers can come in to have repairs performed after normal  business hours.

Industry Stakeholders’ Use of Publicly Available Auto Recall Information Varies

Industry stakeholders’ use of auto recall information varies because these  stakeholders play different roles in the auto recall process. Auto  manufacturers are primarily responsible for providing auto recall  information to the public and others, including NHTSA and auto dealers.  Franchised dealers are responsible for performing the recall remedy for  manufacturers and therefore use manufacturer-provided information for  that purpose. Specifically, all of the franchised dealers we interviewed told  us they identify recalls on new vehicles in their inventory primarily by  accessing auto recall information through internal manufacturer  databases. These franchised dealers may also use information from  third-party providers or publicly available auto recall information on  NHTSA’s website to identify recalls affecting used vehicles.

Independent dealers—which are not generally authorized by  manufacturers to perform recall remedies—may use publicly available  auto recall information to identify open recalls. Specifically, 2 of the 3  independent dealers we met with told us they use NHTSA’s VIN look-up  tool to search for open recalls affecting vehicles in their inventory before  selling them to consumers. However, these dealers told us that the  current design of the tool takes too much time to use, because it requires  users to search each VIN individually. For example, one dealer told us  each search took about 15 seconds to perform, resulting in significant  time and cost because the dealership has tens of thousands of vehicles in  its inventory. These dealers told us being able to search multiple VINs in  a single search (i.e., VIN-batch search) could save them time or money.

Representatives from the Alliance of Automobile Manufacturers stated  they—in coordination with other industry stakeholders—are working with  a third-party provider to develop a search tool that would address this  concern by enabling VIN-batch searches for use by government  agencies, such as state departments of motor vehicles, and commercial  entities. The group anticipates the tool will be available in the first half of  2018.

Most Consumers in Our Focus Groups Prefer to Receive Recall Notification by Electronic Means in Addition to Mail

Although the vast majority of consumers who participated in our focus  group discussion sessions reported a preference to receive auto recall  notification by mail, most preferred to receive notifications by at least one  additional electronic means such as e-mail, phone calls, and text  messages. Eighty of the 94 consumers in our sessions reported a  preference for receiving notification by mail, and all but 4 reported actually  receiving mailed notification (see fig. 2). However, 69 of the 94  consumers in our sessions also reported a preference for receiving recall  notification by electronic means, but only 7 reported actually receiving at  least one type of electronic notification. This result suggests a gap  between industry recall notification practices and notification preferences  for most consumers in our focus groups, especially for younger  consumers who were more likely to report a preference for notification by  electronic means. For complete results of the questionnaire we  administered to consumers for the discussion session, see appendix IV.  As we discuss later in this report, in September 2016, NHTSA issued a  Notice of Proposed Rulemaking (NPRM) that proposes to require auto  manufacturers to notify consumers about auto recalls by electronic means  in addition to First-Class Mail. NHTSA officials told us the agency is   working with the administration on NHTSA’s regulatory portfolio and  priorities, including this rulemaking.

Some manufacturers told us they use additional methods to reach  consumers, including notifying consumers by electronic means and  translating recall information into Spanish. For example, representatives  from one manufacturer told us they always notify consumers by e-mail  before sending out the required First-Class Mail letter notification. These  representatives told us using multiple recall notification means resulted in  higher recall completion rates. In addition, eight of the remaining nine  manufacturers told us they use supplemental electronic means  notification on a case-by-case basis—generally using additional means to  improve recall completion rates—while four manufacturers stated they  consider safety risk severity when deciding when or how to use additional  notification means for individual recalls. Also, representatives from 3 of  the 10 manufacturers we spoke with told us they translate the entire  mailed notification letter into Spanish.

Most Consumers in Focus Groups Found the Auto Recall Areas of NHTSA.gov Generally Easy to Use, but Some Experienced Difficulties

Usability Testing with Consumers Found the Auto Recall Areas of NHTSA’s Website Generally Easy to Use

In late 2016, NHTSA launched its redesigned NHTSA.gov website,  including the auto recall areas consumers assessed during our testing  sessions. According to responses to a questionnaire we administered  during our testing sessions, 78 of the 94 consumers found the auto recall  areas of NHTSA.gov either “somewhat” or “very easy” to use (see fig. 3).  See appendix V for complete participant responses to the questionnaire  we administered to each consumer.

To inform the development of the redesigned website, NHTSA worked  with a contractor to conduct a usability study in 2015 to evaluate users’  reactions to the agency’s websites, including NHTSA.gov. According to  agency officials, NHTSA implemented several changes based on the  findings from the usability study, including:  the creation of a dedicated “recalls” area of NHTSA.gov, and  the ability for users to access the VIN look-up tool in three different  ways—on the homepage, in the “recalls” area, and through direct links  either in a NHTSA e-mail for subscribers or from an external website.

In addition, NHTSA officials told us that Department of Transportation  (DOT) and NHTSA staff meet as needed to discuss the website and  consider improvements. For example, the officials said they monitor user  searches for the relevance and accuracy of results and adjust the search  software to better assist users in finding auto recall information. Officials  also told us the agency collects a variety of other information about how  visitors use NHTSA.gov, including how visitors access the website, and  makes adjustments accordingly. For instance, NHTSA incorporated  responsive web design as part of the agency’s ongoing consolidation  effort—meaning the site is optimized for viewing on desktop, tablet, and  mobile devices. In addition to monitoring searches and how visitors  access NHTSA.gov, NHTSA officials told us they collect and consider  online survey data to make website improvements and use web-analytic  software to monitor, for example, where visitors choose to exit the  website. Officials stated that such monitoring activities have allowed  NHTSA to identify and correct problems with NHTSA.gov.

We did not directly evaluate the accessibility of the auto recall areas of  NHTSA.gov to ensure the ability of people with physical or mental  disabilities to use the website. However, NHTSA officials provided us  with an overview of several steps the agency takes to ensure NHTSA.gov  complies with website accessibility requirements. For example, according  to officials, NHTSA subscribes to a service that provides monthly  accessibility scans of the agency’s websites.

Consumers in Our Focus Groups Identified Opportunities to Improve the Usability of Certain Auto Recall Tasks on NHTSA’s Website

While most consumers in our usability testing sessions generally found  the auto recall areas of NHTSA’s website easy to use, some consumers  experienced difficulties completing tasks we asked them to perform (see  table 2). Specifically, during each testing session we asked participants to  perform tasks using the primary means NHTSA.gov provides for  consumers to access information about auto recalls affecting their  vehicles:  searching for auto recalls using their vehicle’s VIN;  searching for auto recalls using their vehicle’s year, make, and model;  and  locating NHTSA’s auto recall notification e-mail subscription service.

In addition, an evaluation we requested to corroborate the results of our  consumer usability testing, identified similar issues. As discussed below,  consumers experienced these difficulties because the auto recall areas of  NHTSA.gov do not always reflect federal and industry key website  usability practices, which describe standards and guidelines for making  websites easy to use. Following such practices can assist agencies in  creating quality websites while providing the flexibility necessary to meet  organizational needs. Website usability is particularly important for  agencies, such as NHTSA, that are responsible for conveying safety  information to the public. Federal standards for internal control state that  agencies should communicate quality information externally and select  appropriate methods for communicating with the public.

While most consumers in our usability testing sessions found searching  for recalls by VIN somewhat or very easy, some consumers found the  search results did not provide the information they were seeking. When  we asked consumers to perform VIN searches, they generally found the  VIN look-up tool easy to use—88 of 94 consumers found searching with a  VIN either somewhat or very easy. But some consumers experienced  difficulties performing this task. Specifically, some consumers who had  had their vehicles repaired expected to find the completed recall on the  search results page. However, they were confused because the page is  designed to display only open (i.e., unrepaired) recalls, not completed  (i.e., repaired) recalls—leading these consumers to question the accuracy  of the results. In addition, the evaluation conducted by website usability  professionals found that, when an error occurred during a VIN search, the  error message was too difficult to locate on the search results page. The  evaluation recommended the error message have greater weight and  more prominence on the page.

Federal key website usability practices state that agencies should ensure  that results of user searches provide the precise information being  sought, and in a format matching users’ expectations. When users are  confused by search results, or do not immediately find what they are  searching for, they become frustrated and may abandon the search or the  website entirely. Since NHTSA launched the VIN look-up tool in August  2014, the number of VIN searches performed has increased (see fig. 4).  According to NHTSA officials, major increases occurred in mid-2015— when the Takata air bag inflator recalls were announced—and in early  2017, when NHTSA made the VIN look-up tool search function available  on NHTSA.gov and displayed it prominently on the website. Ensuring the  usability of NHTSA’s VIN look-up tool is particularly important because it  is the only way on NHTSA.gov for a consumer to determine whether their  specific vehicle has an open safety recall.

Recall Search Using Vehicle Year, Make, and Model  Some consumers’ vehicle year, make, and model searches were  hampered by the information required to conduct an accurate search, as  the content on the website is not always in plain language. We asked  consumers to perform a recall search using their vehicles’ year, make,  and model, and 78 of 94 consumers found the task to be either somewhat  or very easy. However, some consumers found that they did not know  enough information about their specific vehicles to feel confident that they  were searching for the correct vehicle. For example, a year, make, and  model search for a 2009 Toyota Tacoma may ask the consumer to  choose among vehicle options, including “2009 TOYOTA TACOMA  REGULAR CAB W/SAB RWD/AWD.” Acronyms such as “W/SAB”— which stands for “with side air bags”—may be confusing to consumers.

Federal key website usability practices state that federal agencies should  write website content using plain language, so website visitors can easily  find and use what they need.

Focus Group Participant’s Comment  “I think [the Recall Notification E-mail System  Sign-Up is] poorly placed. I had to scroll to  find it. I had to search for it. You want  at  the top .”

Recall Notification E-mail System Sign-Up  Some consumers suggested improvements to make the Recall  Notification E-mail System Sign-Up easier to locate on the homepage.  NHTSA first made its Recall Notification E-mail System Sign-Up available  in March 2008. Of the 94 consumers in our testing sessions, 66 found it  either “somewhat” or “very easy” to find the Recall Notification E-mail  System Sign-Up—making this the least easy of the three tasks we asked  consumers to perform. Specifically, several consumers said the Recall  Notification E-mail System Sign-Up should include a clearer description,  be easier to find, and be located at the top of the homepage (see fig. 5).  These improvements are particularly important because some consumers  in our focus group sessions told us that the ability to sign up for auto  recall e-mail notifications was the most useful part of the auto recall areas  of NHTSA.gov.

The website evaluation conducted by website usability professionals  recommended that NHTSA streamline its homepage with more of a focus  on primary website tasks. The evaluation also found that users must  move through too many pages to sign up for recall e-mails.

Federal key website design and usability practices state that agencies  should put important items closer to the top of the page, where users can  better locate the information. Key practices also state that agencies  should design their websites so users can successfully complete the most  common tasks in the fewest number of steps.

The website usability difficulties that consumers in our focus groups  experienced may be due to the fact that NHTSA has not studied the  website’s usability since the agency redesigned NHTSA.gov in late 2016  and, therefore, may have been unaware of these difficulties prior to our  review. NHTSA plans to conduct a website usability study with consumers  after the consolidation effort, discussed above, is complete. However,  NHTSA could not provide a general time frame for conducting the study  because it has not yet determined when the consolidation effort will be  complete. We have previously reported that it is essential for  organizations to effectively guide their information technology efforts by  establishing timelines to complete them, among other strategic planning  best practices. Without establishing a completion date for its website  consolidation effort, the website usability difficulties we identified may  persist and limit the effectiveness of NHTSA’s primary means of providing  consumers with safety recall information about their vehicles on  NHTSA.gov.

NHTSA Has Initiated Activities to Raise Consumer Awareness about Recalls, but It Is Too Early to Evaluate the Agency’s Efforts Public Awareness Campaign

In January 2016, NHTSA launched a national advertising campaign  encouraging consumers to check for open recalls using the agency’s VIN  and year, make, and model look-up tools. Through March 2017, NHTSA  spent about $1 million on its Safe Cars Save Lives campaign, which  sponsors advertisements on Google, Facebook, and other media platforms. For example, Google might place NHTSA’s advertisement  above other search results, when a consumer typed certain keywords— such as “recall,” “airbag recalls,” or “safercar.gov”—into the search.

NHTSA evaluated the campaign’s effectiveness by monitoring website  traffic performance reports to determine how frequently consumers  clicked on NHTSA-sponsored advertisements and ultimately searched for  open recalls using the agency’s look-up tools. NHTSA also compared  results across media platforms and adjusted the campaign’s strategy to  improve performance. For example, NHTSA optimized advertisements on  mobile devices, since mobile-device users performed more recall  searches than other users. According to NHTSA data, the awareness  campaign resulted in consumers performing 1.1 million recall searches  through March 2017—a cost of about $0.90 per search. Agency data  indicate that this cost generally decreased as NHTSA improved the  campaign’s strategy. Agency officials told us NHTSA plans to spend  another approximately $1.8 million on Safe Cars Save Lives from  September 2017 through September 2018 due to the campaign’s  effectiveness in raising the public’s awareness about auto recalls.

NHTSA began implementing a mandated 2-year pilot grant program  intended to evaluate the feasibility and effectiveness of informing  consumers about open auto recalls during state vehicle registration. In  September 2016, NHTSA solicited applications to participate in the  program, wherein selected states would inform consumers—at no  charge—about open recalls using all means that permit consumers to  register vehicles within the state (e.g., in person, Internet, and mail).

According to NHTSA, only one state applied for the grant. In September  2017, NHTSA awarded the sole applicant $223,000.

Under the program, the grantee needs to collect and report program  performance data, including the extent to which open recalls have been  identified and repaired. In addition, the grantee must report whether  certain notification means were more effective than others and what could  be done to improve the program. Upon completion of the pilot program,  NHTSA is required to evaluate the extent to which open recalls identified  have been remedied. Auto manufacturers we met with were generally  supportive of the program. Specifically, representatives from 9 of the 10  manufacturers told us that notifying consumers about open recalls during  vehicle registration can raise consumer awareness or improve recall  completion rates.

In September 2016, NHTSA issued a Notice of Proposed Rulemaking  (NPRM), which proposes to require auto manufacturers to notify  consumers about open recalls by electronic means—such as e-mails,  phone calls, and text messages, in addition to First-Class Mail. As we  described earlier, auto manufacturers are currently required to notify  consumers about safety recalls affecting their vehicles via First-Class  Mail. According to NHTSA, the NPRM aims to aid in efficiently and  effectively improving recall completion rates, by proposing that  manufacturers provide notification using electronic means in addition to  First-Class Mail.

Consumers in our focus groups as well as auto manufacturers and  consumer associations we interviewed generally supported additional  notification using electronic means.

Consumers in our focus groups: As we discussed earlier, 69 of the  consumers in our focus group discussion sessions reported they  would prefer to receive additional notification by at least one type of  electronic means. However, only 7 consumers actually received such  notifications—suggesting a gap between industry notification practices  and notification preferences for these consumers.

Auto manufacturers: Representatives from 9 of 10 manufacturers we  interviewed told us they generally support providing notification using  electronic means. Although the NPRM proposes a broad definition of  electronic means to give manufacturers flexibility to determine the  most effective means, these representatives also shared  implementation concerns. For example, representatives from 1 of  the 9 manufacturers told us that—although the company collects e- mail addresses from some customers for other purposes—not all  customers provide e-mail addresses, and those collected are not  always accurate. As we discussed previously, most manufacturers we  met with currently use supplemental electronic means notification on a  case-by-case basis.

Consumer associations: Similarly, both consumer associations we  interviewed told us additional electronic notification can help reach  consumers who do not complete repairs after receiving initial mailed  notification.

NHTSA’s proposal would maintain manufacturer reporting requirements,  though it may result in additional reporting. This additional information  could help the agency evaluate the effectiveness of various means of  consumer notification. We previously found that NHTSA may be able to  use manufacturers’ data to identify what factors make some recalls more  or less successful than others. We recommended that NHTSA use the  recall data it collects to analyze particular patterns or trends that may  characterize successful recalls and determine whether these factors  represent best practices.

If the NPRM is finalized, manufacturers would provide NHTSA with  representative copies of the newly required electronic notifications, in  addition to mailed notifications, and would specify the electronic means  used, such as e-mail or text message. According to NHTSA officials, this  information could allow the agency to track and evaluate the effectiveness  of various notification means used by manufacturers by, for example,  comparing completion rates across means—a key step in identifying best  practices that could encourage consumers to complete repairs. However,  it is too early for NHTSA to conduct such an evaluation, because the  agency has not issued a final rule. NHTSA officials told us the agency is  working with the administration on NHTSA’s regulatory portfolio and  priorities, including this rulemaking.

NHTSA has also taken steps to collaborate with industry stakeholders  and explore consumer education best practices. For example, in April  2015 NHTSA hosted a day-long workshop that brought together auto  industry stakeholders to examine public education of the recall process.  During the workshop, participants identified current barriers to the public’s  awareness of auto recalls and discussed potential solutions to address  them, such as using text messages and social media to communicate with younger consumers and using different delivery methods for recall  notices. Similarly, in January 2016 NHTSA and 18 auto manufacturers  adopted a set of Proactive Safety Principles to explore and employ new  ways to increase safety recall participation rates. For example, NHTSA  and auto manufacturers agreed to share industry best practices and  policies based on lessons learned from ongoing safety recalls.

The Independent Monitor of Takata in conjunction with NHTSA has also  issued a set of coordinated communications recommendations based on  consumer research, best practices observed during the Takata recall, and  discussions with manufacturers. For example, the recommendations  encourage manufacturers to: pursue a “multi-touch” communications strategy that employs non- traditional means, such as e-mail and text messages;  convey risk in clear, accurate and urgent terms; and  include a clear “call to action” designed to facilitate prompt and  efficient scheduling of repairs.

According to NHTSA officials, the agency relies on auto manufacturers to  evaluate the effectiveness of these efforts. However, agency officials  told us NHTSA reviews manufacturers communication plans as part of  the Takata recall’s Coordinated Remedy Program and provides ongoing  recommendations on manufacturers’ communication language, approach,  and strategies.

With the recent steep increase in safety defect vehicle recalls and  continued low recall completion rates, it is vital for consumers to be able  to easily access and use publicly available auto recall information.

NHTSA has taken important steps to improve its website—which provides  safety recall information to consumers—resulting in most consumers in  our focus groups finding the website easy to use. However, the difficulties  some experienced in attempting to complete essential auto recall tasks  demonstrated that NHTSA.gov does not always reflect key website  usability practices for website design. Although NHTSA plans to conduct  a website usability study with consumers after consolidating its websites,  it has not determined a completion date for this effort—an essential step  for organizations to effectively guide their information technology efforts.  Without such a date, the website usability difficulties may persist and limit  the effectiveness of NHTSA.gov in providing consumers with recall  information about their vehicles. By addressing these difficulties in the  interim, NHTSA can better assure that consumers obtain this information,  which can be vital to their safety.

We are making the following two recommendations to NHTSA:  The Administrator of NHTSA should determine a completion date for the  agency’s website consolidation effort. (Recommendation 1)

The Administrator of NHTSA should, while the agency continues its  website consolidation effort, take interim steps to improve the usability of  the auto recall areas of NHTSA.gov by addressing the website usability  difficulties we identified. (Recommendation 2)

We provided a draft of this report to DOT for review and comment. In its  written comments, reproduced in appendix VI, DOT stated that it  concurred with our recommendations.  The department also provided  technical comments, which we incorporated as appropriate.

We are sending copies of this report to relevant congressional  committees, the Secretary of Transportation, and the Administrator of  NHTSA. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or flemings@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

This report examines the use of publicly available auto recall information  for safety defects affecting passenger vehicles. The report addresses the  following objectives: (1) How do consumers and industry stakeholders  use publicly available auto recall information? (2) How easy or difficult to  use do consumers find the auto recall areas of NHTSA.gov? (3) What  steps, if any, has the National Highway Traffic Safety Administration  (NHTSA) taken to raise consumer awareness about auto recalls and how  has NHTSA evaluated the effectiveness of these steps?

We define publicly available auto recall information to include information  on the auto recall areas of NHTSA.gov, such as examples of notification  letters that manufacturers mail to consumers. This report focuses on  safety defect vehicle recalls affecting passenger vehicles that are initiated  when a defect in a vehicle or vehicle equipment creates an unreasonable  safety risk, as determined by NHTSA or a manufacturer.

To determine how consumers use publicly available auto recall  information, we conducted and analyzed transcripts and questionnaires  from 12 consumer focus groups we conducted with used and new vehicle  owners who had experienced an auto recall in the last 24 months. Each  focus group was split into two sessions: (1) a discussion session to  explore participants’ thoughts, experiences, and preferences about auto  recall information and (2) a website usability testing session. Also, we  administered a questionnaire as part of each of these sessions. For the  discussion session, we asked consumers about the recall notification  process and how they used the recall information, and for the website  usability testing session, we asked consumers to fill in a questionnaire  during the session itself as they assessed the usability of the auto recall  areas of NHTSA’s website. We conducted the 12 focus groups at six  locations across the country, with each group including 7 or 8 consumers  for a total of 94 participants. Half of the focus groups were comprised of  consumers who had completed the repair and the remaining half included  consumers who had not completed the repair.

We selected the six focus group locations to provide population and  geographic dispersion. To ensure geographic dispersion, we selected at  least one location in each U.S. Census region (see table 3). To ensure  population dispersion, we selected Metropolitan Statistical Areas  representing a range of population sizes based on 2015 U.S. Census  estimates. To ensure our selection included the perspectives of vehicle  owners in geographically distant or isolated communities, we also  selected a rural location, which we defined as a city or town that has a  population of less than 50,000 inhabitants and is not an urbanized area  contiguous and adjacent to a city or town that has a population of greater  than 50,000 inhabitants.

Using information provided by the participants, we selected focus group  participants based on age, income, gender, education level, race, and  ethnicity to ensure we collected a range of perspectives on auto recall  information use. However, since we did not select a representative  sample of participants, focus group results are not generalizable to all  vehicle owners.

During focus group discussion sessions, we asked participants to discuss  factors they considered when deciding whether to repair their recalled  vehicle and then to select the single most influential factor. Each of the 12  focus group sessions was audio recorded and transcriptions were  created; transcripts served as the record for each group. We then  evaluated those transcripts using systematic content analysis to identify  the factors consumers considered when deciding whether to complete  repairs and any suggested improvements to the auto recall  communication process. The analysis was conducted in three steps. First,  two analysts independently developed a code framework and then  worked together to resolve any discrepancies. Second, each transcript  was coded independently by analysts using the framework and any  discrepancies were resolved by both analysts agreeing on the coding of  the associated statement by a participant. Third, if needed, another  analyst adjudicated any continued disagreement between coders.

Because the transcripts did not include a unique identifier for each focus  group participant, we conducted our analysis of focus group session  discussions at the group level (i.e., of the 12 focus groups we conducted).  We also administered and analyzed a questionnaire as part of each  discussion session to quantify responses regarding consumers’ use of  auto recall information, including how they received and preferred to  receive auto recall notifications. Our analysis of the questionnaire  responses was conducted at the individual consumer level (i.e., of the 94  consumers who participated).

These focus group sessions were structured, guided by a moderator who  used a standardized list of questions to encourage participants to share  their thoughts, experiences, and preferences. We also conducted two  pretest focus groups at our headquarters and made some revisions to the  focus group guide prior to beginning the sessions with consumers.  Methodologically, focus groups are not designed to demonstrate the  extent of a problem or to generalize results to a larger population or  provide statistically representative samples or reliable quantitative  estimates. Instead, they are intended to generate in-depth information  about the reasons for the focus group participants’ thoughts, experiences,  and preferences on specific topics.

The projectability of the information produced by our focus group  sessions is limited. For example, the information includes only the  responses from the vehicle owners from the 12 selected groups and their  individual responses to questions we asked. The experiences and  preferences expressed may not reflect other vehicle owners’ thoughts  and preferences. In addition, while the composition of the groups was  designed to ensure a range of age and education levels, among the other  criteria mentioned previously, the groups were not constructed using a  random sampling method.

To determine how industry stakeholders use auto recall information, we  interviewed selected auto manufacturers, selected franchised and  independent auto dealerships, NHTSA program officials, and other  industry stakeholders. Specifically, we interviewed representatives from  the following 10 auto manufacturers, selected based on each  manufacturer’s sales market share (small, medium, and large), place of  ownership (foreign and domestic), and experience with auto recalls (lower  to higher based on the average annual number of auto recall campaigns  and average market share of each manufacturer from 2010 to 2014) to  collect a range of perspectives on how manufacturers use auto recall  information:

To understand the perspective of auto dealers, we interviewed four  franchised dealerships, one in each of the four U.S. Census regions  where we conducted focus groups with consumers. We also interviewed  three independent auto dealerships in two U.S. Census regions. The  results of these interviews are not generalizable to all auto manufacturers  and dealerships, but provide insights about how some industry  stakeholders use auto recall information.

We conducted interviews with NHTSA program officials to understand  NHTSA’s role in the auto recall process. In addition, we interviewed other  stakeholders, including the Independent Monitor of Takata, which assists  NHTSA in overseeing the Takata recall, as well as officials from  consumer associations and other industry groups (see table 4).

To evaluate how easy or difficult consumers find the auto recall areas of  NHTSA.gov to use, we reviewed various website usability resources to  understand federal and industry key website usability practices for making  websites easy to use, such as focusing on design and how easily users  can find information. In addition, we reviewed federal standards for  internal control related to communicating quality information externally.  During our usability testing sessions, we asked consumers to attempt to  complete auto recall tasks—the primary means NHTSA.gov provides for  consumers to access information about auto recalls affecting their  vehicles—and discuss their experiences. We then compared consumers’  experiences with the usability of the website against these practices.

To identify key website usability practices, we analyzed guidance  documents from NHTSA and other federal agencies. For example, we  analyzed the General Services Administration’s (GSA) and the  Department of Health and Human Services’ Research-Based Web  Design & Usability Guidelines, which includes quantified, peer-reviewed  guidelines intended to help federal agencies improve the design and  usability of their information-based websites. We also analyzed GSA’s  Requirements for Federal Websites and Digital Services, and the U.S.  Digital Services Playbook to identify key practices for making websites  easy to use. Identified key practices are: (1) design and content— focusing on the layout, headers, and design; (2) navigation—how easily  users can find information; (3) clarity—the ability to read and digest content; (4) identity and purpose—whether the site clearly presents its  purpose; and (5) accessibility—the ability of people with physical or  mental disabilities to use the site.

To analyze the results of focus group website testing sessions, we  performed a systematic content analysis of the session transcripts using  the same content analysis methods described above and an analysis of  the questionnaire we administered to each participant during the website  usability sessions. Specifically, we analyzed the transcripts from the  website usability testing sessions to account for consumers’ experiences,  including their initial impressions of the website and any suggested  usability improvements. We also analyzed the results of the questionnaire  that each participant completed where participants were asked to mark  responses regarding their experience including an assessment of the  usability of the auto recall areas of NHTSA.gov. Our analysis of the  results from the questionnaire responses was conducted at the individual  consumer level (i.e., of the 94 consumers who participated) while our  analysis of focus group session discussions was conducted at the group  level (i.e., of the 12 focus groups we conducted).

To corroborate the results of usability testing sessions we conducted with  the consumers in our focus groups, we requested that five website  usability professionals from GSA’s Federal User Experience Community  conduct an independent evaluation of the auto recall areas of NHTSA.gov  against federal and industry key website usability practices (described  above). The website usability professionals developed a website  usability evaluation form, which they used to individually evaluate the auto  recall areas of NHTSA’s website. The website usability professionals then  met to form a consensus and provided us with one final group evaluation  of the website usability of the auto recall areas of NHTSA.gov. Also,  although neither our usability testing nor the website usability evaluation  conducted by website usability professionals directly addressed  accessibility, we interviewed responsible agency officials about how the  agency assesses the accessibility of NHTSA.gov.

We also requested and analyzed website data provided by NHTSA to  understand how consumers access and use NHTSA.gov. Requested data  included the number of subscribers to NHTSA’s Recall Notification E-mail  System Sign-up; the number of weekly vehicle identification number (VIN)  searches performed on NHTSA.gov from August 2014 through May 2017;  and NHTSA.gov usage data by device (i.e., usage by mobile devices,  tablets, and desktop computers). We assessed the reliability of these data  by reviewing any supporting documents provided by the agency and  interviewing responsible NHTSA officials, and concluded the data were  sufficiently reliable for our reporting purposes.

While we did not independently review the usability of auto  manufacturers’ auto recall websites, we requested and reviewed the  results of any audits that NHTSA performed of these websites, including  whether the websites met statutory and regulatory requirements for  providing auto recall information to the public. We then corroborated any  audit findings by reviewing the auto recall websites of the selected auto  manufacturers that we interviewed.

To determine any steps NHTSA has taken to raise consumer awareness  about auto recalls and how NHTSA evaluates the effectiveness of any  steps, we reviewed relevant statutes, regulations, and proposed rules,  including the Fixing America’s Surface Transportation Act and a Notice of  Proposed Rulemaking related to recall notification methods. We also  reviewed agency and other documents that describe or evaluate  NHTSA’s public awareness activities. For example, we analyzed  NHTSA’s strategic planning documents—such as NHTSA’s Strategic  Plan 2016–2020—to identify ongoing public awareness activities along  with their related goals, objectives, or performance metrics. Similarly, we  requested and analyzed any documents NHTSA uses to evaluate the  effectiveness of its public awareness activities, including performance  reports for NHTSA’s ongoing Safe Cars Save Lives campaign.

To assess the reliability of data included in these performance reports— such as VIN searches performed—we reviewed agency documentation  and interviewed agency officials about the reliability, accuracy, and  completeness of the data and determined the data were sufficiently  reliable for our reporting purposes. We reviewed performance  management practices as provided in the Government Performance and  Results Act of 1993 (GPRA), the GPRA Modernization Act of 2010, and  standards for internal control in the federal government to identify any  opportunities for improvement. We also performed a literature review to  identify any related published articles and research studies.

To understand how NHTSA implements and evaluates any public  awareness activities, we also interviewed responsible agency officials  from NHTSA’s Office of Communications and Consumer Information and  other offices. In addition, we discussed NHTSA’s public awareness efforts  during interviews with industry stakeholders, including selected auto  manufacturers, selected franchised and independent auto dealerships,  and other industry stakeholders. We analyzed the results of these  interviews along with the focus group discussions we conducted with  consumers (discussed above) to identify perspectives on the  effectiveness of NHTSA’s public awareness steps.

Appendix II: Example of an Auto Manufacturer’s Recall Notification Letter

Appendix III: Annual Recall Completion Rates by Vehicle Component and Vehicle Type

The National Highway Traffic Safety Administration (NHTSA) is required  to conduct a biennial analysis of vehicle safety recall completion rates  and submit the results of its analysis in a report to certain congressional  committees. The report must include, among other things, the annual  recall completion rate by vehicle type and vehicle component (such as  brakes, fuel systems, and air bags) for each of the 5 years preceding the  year the report is submitted. According to NHTSA’s May 2017 report,  completion rates for all vehicles combined ranged between 63 percent  and 67 percent between calendar year 2011 and calendar year 2014 (see  table 5). However, NHTSA reported wider variation when the recall  completion rates are broken down by vehicle type. Similarly, the report  found that completion rates for most component categories fall within a  range of 60 percent to 75 percent (see table 6).

The annual completion rate is a volume-based, weighted metric, such that  the more vehicles affected by the recall, the more weight or influence it  has on the computed rate.

Appendix IV: Focus Group Participants’ Responses to Recall Notification Questionnaire

Focus group participants responded to a questionnaire we administered  to collect information on consumers’ auto recall notification preferences  during our discussion sessions. Table 7 shows participants’ responses to  the administered questionnaire, by age group. We present these  responses by age group, because consumers’ notification preferences  may vary according to their ages.

Appendix V: Focus Group Participants’ Responses to Website Usability Questionnaire

Focus group participants responded to a questionnaire we administered  to collect information on the usability of NHTSA.gov during our usability  testing sessions. Table 8 shows focus group participants’ responses to  the administered questionnaire, by age group. We present these  responses by age group, because consumers’ website usability needs or  preferences may vary according to their ages.

Appendix VI: Comments from the Department of Transportation

Appendix VII: GAO Contact and Staff Acknowledgments

In addition to the individual named above, H. Brandon Haller (Assistant  Director); Katherine Blair; Jason Blake; Melissa Bodeau; Alicia Cackley;  William Colwell (Analyst in Charge); Lacey Coppage; Elizabeth Dretsch;  Jaci Evans; Marcia Fernandez; Sarah Kaczmarek; Malika Rice; Todd  Schartung; and Andrew Stavisky made key contributions to this report.