National Institute of Standards and Technology: Additional Review and Coordination Could Help Meet Measurement Service Needs and Strengthen Standards Activities

The U.S. Department of Commerce's NIST provides measurement services and supports standards that promote U.S. competitiveness. For example, NIST provides calibrations for manufacturing equipment and reference materials used in testing. NIST also supports private sector organizations in developing standards to help ensure product performance, among other things, such as Wi-Fi. In recent years, NIST has sought to improve the delivery of its services and documentary standards activities.

GAO was asked to review NIST measurement services and standards-support activities. This report examines (1) the challenges NIST faces in providing measurement services and supporting documentary standards development, and (2) the extent to which NIST has taken steps to address these challenges and how those steps align with federal guidance and policy. GAO analyzed testimony, reports, laws, and policies; conducted focus groups with academics and industry representatives; and interviewed various stakeholders.

The National Institute of Standards and Technology (NIST) faces challenges in providing measurement services and supporting private sector development of specifications for products' designs or performance—referred to as “documentary standards.” Based on reviews of relevant testimony, reports, and other documents; interviews with stakeholders; and focus groups with academics and industry representatives, GAO identified challenges including:

Identifying and prioritizing what measurement services, such as calibrating large force-measurement tools used by aerospace manufacturers, or what documentary standards activities, such as serving as a technical advisor on fire safety standards, are most needed by U.S. industry, and

Coordinating with other federal agencies on standards development issues.

NIST has taken steps to address these challenges, including industry outreach and reviews of measurement services and standards activities. However, some efforts do not fully align with federal guidance or NIST policy. For example, NIST's measurement-services and standards-activity reviews have not included a comprehensive examination of how these services and standards activities align with stakeholder needs. Federal internal control standards call for managers to use quality information to determine if the agency is meeting its objectives. Comprehensively reviewing NIST's measurement services and documentary-standards activities would provide NIST with greater confidence that its services and activities align with stakeholders' needs.

GAO also found that NIST coordinates with other agencies on standards development and related activities, but that some efforts do not fully align with specific leading practices GAO has previously identified for enhancing and sustaining interagency collaboration. For example, NIST and other agencies coordinate on standards activities through a NIST-chaired interagency committee. However, GAO found that the committee has not updated its charter since 2000—contrary to leading practices to update and monitor collaborative agreements. GAO also found that NIST has not worked with other committee members to fully clarify agencies' roles and responsibilities. Without ensuring that member agencies' roles and responsibilities are current and fully clarified, NIST and other agencies may miss opportunities to strengthen coordination.

GAO is making seven recommendations, including that NIST comprehensively review measurement services and documentary-standards activities, and work with other agencies to take steps to strengthen interagency coordination. The Department of Commerce agreed with six recommendations and disagreed with one, citing risks to the private-sector-led U.S. standards system. GAO clarified its recommendation and continues to believe this action is needed, as discussed in the report.

Commercial transactions rely on a buyer and seller having a shared  understanding about certain aspects of the goods being exchanged, such  as their size or other technical specifications. For example, motorists  expect that the volume of fuel dispensed from a gasoline pump accurately  matches the volume for which they are charged. Standards help define  the technical aspects or capabilities of materials, devices, machines, and  other products to ensure their performance and interoperability.  Standards include units of measurement, such as the gallon, and  standards that can describe the performance or design of a product,  process, or test—referred to as “documentary standards.” Wi-Fi, for  example, is a set of documentary standards that define a particular type  of wireless communication network. Devices that follow the same Wi-Fi  standard will be compatible regardless of manufacturer, thus providing  consumers with more choices among similar products.

The National Institute of Standards and Technology (NIST) is an agency  within the Department of Commerce (Commerce) that, since 1901, has  provided measurement services and tools and has helped develop and  maintain key measurement and documentary standards to help U.S.  industry compete. Specifically, NIST’s mission is to promote U.S.  innovation and industrial competitiveness by advancing measurement  science, standards, and technology in ways that enhance economic  security and improve quality of life. NIST’s measurement services include,  among other things, calibrating equipment, such as industrial  thermometers, to ensure their accuracy and precision. NIST also sells  standard reference materials that researchers and manufacturers can use  to test the purity, strength, or other attributes of their products. NIST’s  measurement services help businesses ensure that their products meet  relevant standards and give consumers confidence in the products they  purchase. Similarly, NIST supports the development of measurement and  documentary standards in many ways. NIST plays a leading role in  developing some standards, such as standards that support many basic  units of measurement and certain documentary standards on the  operation of federal information technology systems. However, the United  States has a voluntary, consensus-based standards system, whereby  most documentary standards are developed collaboratively by producers  and users through private-sector standards development organizations  (SDO). Consequently, NIST staff may participate in SDO-led standards  development activities by, for example, serving as technical advisors.

Because advances in science and technology continue to drive growth in  many commercial sectors, NIST’s measurement services and support for  developing documentary standards can directly affect innovation and the  nation’s economy. To fulfill its mission, NIST relies on the expertise and  research of its staff across a number of laboratories. NIST reorganized its  laboratory structure in 2010, in part, to improve the delivery of the  agency’s services. Under the previous laboratory structure, mission- based activities were often spread across numerous laboratories. In  testimony before the House Committee on Science and Technology’s  Subcommittee on Technology and Innovation in March 2010, the Director  of NIST stated that restructuring the agency’s laboratories would make  NIST more effective in delivering its services to customers. Since that  time, NIST has taken additional steps designed to improve its  measurement services and documentary standards activities.

You asked us to examine NIST’s measurement services and standards  development activities. This report examines (1) the challenges NIST  faces in providing measurement services and supporting documentary- standards development; and (2) the extent to which NIST has taken steps  to address any challenges and how those steps align with relevant federal  guidance and policy.

To identify any challenges NIST may face in providing measurement  services and supporting documentary-standards development, we began  by performing a literature review, including reports on NIST from the  National Academies of Sciences, Engineering, and Medicine (National  Academies); NIST’s Visiting Committee on Advanced Technology  (VCAT); and other sources. We supplemented our review of the  challenges identified in these sources by analyzing the responses of  participants in focus groups we organized of NIST stakeholders  comprised of: (1) researchers and (2) representatives working with  industry, including commercial entities and state metrology laboratories.

We selected researchers to participate in our focus groups from university  scientists in engineering and the physical and biological sciences. We  selected industry participants for the focus groups to reflect a range of  industrial sectors, including (1) sectors that contribute the most to U.S.  gross domestic product, and (2) sectors that Commerce has identified as  representing U.S. export opportunities. We also included representatives  from the National Conference on Weights and Measures, a nonprofit  association of state and local weights and measures officials, federal  agencies, manufacturers, retailers, and consumers that addresses  consumer measurement needs. We conducted 3 focus groups for  representatives from industry and 2 focus groups for researchers. Each  focus group included from 5 to 8 individuals. In total, our focus groups  included 31 stakeholders.

We also collected information on the challenges that NIST faces during  36 interviews, including 17 interviews with current and former NIST  officials, 10 interviews with officials from other federal agencies, and 9  interviews with representatives from SDOs and other stakeholders. The  10 interviews we conducted with other federal agency officials included 8  agency standards executives—senior level officials with knowledge of,  and experience in, standards-related issues at their agencies and who  are responsible for coordinating their agency’s participation in SDOs,  among other responsibilities.

To evaluate the steps NIST has taken to address challenges in providing  measurement services and in supporting documentary-standards  development, we drew upon our focus groups, interviews with NIST staff,  and reviews of NIST documentation that described the agency’s  measurement services and standards development activities, such as  agency policies, orders, and publications. We also conducted a review of  existing literature as well as relevant laws, NIST policies, and other  agency guidance documents. We compared the steps NIST has taken to  address the challenges it faces in providing measurement services and  supporting standards development to these policies and guidance. We  also assessed how well NIST’s efforts align with Standards for Internal  Control in the Federal Government and selected leading practices for  sustaining and enhancing interagency collaboration that we identified in  previous work.

We conducted this performance audit from July 2016 to July 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

NIST carries out its measurement services and documentary-standards  development activities across several agency laboratories. NIST’s  standards activities include participation in private-sector standards  development organizations that conduct most standards development in  the United States, and federal law and guidance provide direction to  agencies when they participate in this process.

NIST’s Management of Measurement Services and Documentary Standards Activities

NIST’s work underlies much of our nation’s business and public  infrastructure, from helping to ensure the quality of air and water to  helping to ensure the security of online financial transactions. This work  includes providing measurement services, such as calibrations of  equipment and reference materials used to ensure the accuracy and  reliability of a wide range of scientific and industrial devices, and support  for the development of documentary standards by the private sector. As  of July 2017, NIST employed approximately 3,500 federal personnel and  hosted 4,000 associates, who include guest researchers and  collaborators, student interns, facility users, and contractors at its  locations in Gaithersburg, MD and Boulder, CO.

Under its Associate Director for Laboratory Programs, NIST’s activities  span seven laboratory programs that cover a wide range of subject  matter, such as bioscience and health, energy, manufacturing, and public  safety and security (see figure 1). The seven laboratories are divided into  divisions and groups of scientists and engineers who perform research in  a certain field or discipline, and may also provide measurement services  or participate in standards activities. In addition, NIST has three offices  that primarily deal with measurement services: (1) the Office of Reference  Materials within the Material Measurement Laboratory; (2) Calibrations  Services within the Physical Measurement Laboratory; and (3) the Office  of Weights and Measures also within the Physical Measurement  Laboratory. Further, the Standards Coordination Office, which is also  under the Associate Director for Laboratory Programs, conducts  standards-related activities and provides guidance to NIST staff on  participation in documentary standards activities.

In 2010, NIST reorganized its laboratory structure, in part to improve the  agency’s provision of measurement services. Under the new structure,  the Physical Measurement Laboratory includes staff that handles most of  the agency’s measurement standards and calibrations. The Material  Measurement Laboratory includes staff supporting materials science and  produces most of the agency’s standard reference materials. According to  the NIST Director at the time of the reorganization, managing related  research and measurement services together would allow the agency to  improve its services.

NIST’s measurement services encompass calibrations, standard  reference materials, and standard reference data, among others. NIST  provides calibration services for about 700 different types of devices and  has over 1,200 different types of reference materials available (see figure  2). For example, NIST performs calibrations on many different types of  thermometers for both scientific and industrial uses. For customers that  have unique calibration needs, NIST can perform special tests tailored to  their specific circumstances. NIST also performs detailed analysis of  certain materials to precisely characterize their properties and makeup  and provides these reference materials for use by others. For example,  NIST produces a number of food-related reference materials that allow  companies to accurately determine the nutritional content of their  products.

NIST has established a formal quality-control system covering the  calibrations, special tests, and standard reference materials provided by  the agency. The NIST quality system is intended to provide customers  with confidence in the quality of NIST’s measurement services and create  an environment of continual improvement for NIST management and  staff. The quality system is described in policies and procedures  governing the agency’s measurement services. Specifically, the NIST  Quality Manual for Measurement Services, NIST-QM-I, contains NIST- wide policies and procedures and additional sub-level quality  documentation contains policies and procedures established and  maintained by each Division or Office to meet its technical needs. The system is overseen by the NIST Quality Manager, a position within the  Standards Coordination Office, and the NIST Measurement Services  Council, comprised of the Quality Manager and other agency officials,  who report to the Associate Director for Laboratory Programs.

Private sector calibration and testing companies may use NIST’s  measurement services to provide NIST-traceable services, meaning that  the accuracy and precision of the private company’s service has been  documented and compared to NIST’s capabilities. This process allows  these companies to provide services to consumers who do not need the  high level of certainty provided by NIST while still providing assurance  that their measurements are sufficient for their needs. As shown in figure  3, NIST performed calibrations on about 13,000 individual devices per  year and provided about 30,000 reference materials per year or more  from fiscal year 2006 to fiscal year 2016. NIST also provides standard  reference data—such as detailed technical data on various elements,  materials, and chemicals—and keeps time with its atomic clock in  Boulder, CO, and broadcasts it. NIST also accredits public and private- sector laboratories to perform calibrations and other tests through the  National Voluntary Laboratory Accreditation Program. Such accreditation  shows, among other things, that the measurement services provided by  these labs comports with certain federal and international requirements  for calibration and testing.

NIST’s standards activities support the development and use of standards  to enhance the economic and technological competitiveness of the United  States. There are various types of standards including measurement  standards that define specific units, such as the kilogram, and  documentary standards that describe the performance or design of a  particular product, process, or test. NIST develops and refines numerous  measurement standards and collaborates with other national metrology  institutes across the world through the General Conference on Weights  and Measures, the Bureau International des Poids et Mesures, and other  organizations. This work includes supporting the International System of  Units, which includes the kilogram, meter, second, and other units of  measurement that form the basis for NIST’s calibration services.  Measurement standards ensure that these units are consistently used  and applied around the world. Documentary standards, in comparison,  can specify how a product is designed or made, or they may establish  performance standards that define the product by function rather than  material. For example, documentary standards define Wi-Fi radios,  certain aspects of building codes, and safe design for children’s toys,  among other things. Both of these standards help define the properties  and functions of today’s products and provide businesses and consumers  with confidence that products will work as expected before purchase.

The U.S. Documentary Standards-Setting Process and NIST Participation

In the United States, documentary standards are generally developed by  the private sector through an open, consensus-based process overseen  by various SDOs. Private sector companies in the United States choose  when it is in their interest to participate in standards development. Many  SDOs follow similar processes in the development of standards, and  generally adhere to certain principles, including openness, balance of  interests, and consensus. Specifically, once an SDO agrees to develop a  new or revised standard, a committee is formed of representatives with  subject-matter expertise from companies, nonprofit organizations, and  government agencies. The representatives serve on a voluntary basis,  and the committee drafts the standard. SDOs may have certain  requirements for participants, such as payment of membership dues, to  fully participate. In the process of creating or revising documentary  standards, certain committee members will take on leadership roles, such  as chairing committee meetings or leading writing of draft standards or  other documents. Generally, a committee will use a consensus-based  process to vote on whether to approve the draft standard. For example, to  approve a draft standard, some SDOs require a supermajority, at least  two-thirds, of the members who cast ballots as well as resolution of any  negative comments.

Documentary standards define the technical  aspects or capabilities of materials, devices,  machines, and other products to ensure their  performance and interoperability. For  example, in 1990, the IEEE Standards  Association began work to develop a  documentary standard for allowing devices to  connect wirelessly to the internet. The IEEE  wireless networking working group,  designated as 802.11, approved its first  standard in 1997 and has since approved a  series of amendments and improvements to  the standard. In 1999, a group of companies  formed the Wi-Fi Alliance to help drive usage  of the 802.11 standard and provide  consumers with information on products that  implement the standard. The Wi-Fi Alliance  coined the brand Wi-Fi and developed  certification procedures to show that devices  using the 802.11 standard from different  vendors are interoperable and provide a  consistent user experience. Devices that  comply with the standard are able to  wirelessly transfer data within a local area.  Within 2 years of the standard’s initial  approval, the first devices using the standard  were available to consumers and 21 years  later wireless networks have become  commonplace in libraries, coffee shops, and  homes around the world. Originally intended  for linking home or office computers, the  standard has been implemented in a growing  array of devices including lightbulbs and other  household items. Hundreds of companies now  incorporate Wi-Fi into their products,  leveraging the ubiquity of the standard to  improve the value of their products and give  consumers options for meeting their  networking needs.

Several, large private sector organizations help create documentary  standards in the United States. The American National Standards  Institute (ANSI) is a membership organization that accredits numerous  U.S. SDOs that oversee the creation, promulgation, and use of over  10,000 American National Standards. Other U.S.-based organizations  that develop standards for domestic and international use include ASTM  International, IEEE, and the National Fire Protection Association. ANSI  also serves as the U.S. representative to two Geneva-based international  organizations that support the creation of global standards, the  International Organization for Standardization (ISO) and the International  Electrotechnical Commission (IEC). When followed, international  standards may reduce technical barriers to trade by reducing conflict  among domestic standards in various nations and allowing companies to  produce a single product for multiple markets. For example, ISO  encompasses 163 national standards organizations and is a major source  of international standards. In 2000, NIST and ANSI signed a  memorandum of understanding to, among other things, improve  communication and coordination among the private and public sector on  voluntary standards issues.

Staff across many of NIST’s laboratories participate in documentary- standards development activities. NIST policy encourages staff  participation in domestic- and international-standards activities whenever  such participation is in the public interest and is compatible with NIST’s  mission, policies, positions, priorities, and available resources. In 2016,  NIST reported that staff participated in 114 SDOs. In limited policy areas,  where a national priority has been identified in statute, regulation, or  administration policy, active engagement or a convening role by the  federal government may be needed to accelerate standards development  and implementation.

Federal Agencies’ Documentary Standards Activities and Relevant Law and Guidance

Federal agencies may use or help develop documentary standards for  several reasons, including (1) to procure goods or services, (2) to  incorporate standards into agency regulations, or (3) to improve agency  operations or further agency policy goals. For example, the General  Services Administration uses standards to specify packaging, marking,  and labeling of products purchased for government use and for  descriptions of the products themselves; the Consumer Product Safety  Commission has incorporated various consensus standards into its  regulations of consumer products; and the Department of Energy uses a  number of consensus standards to help operate its contractor-run  laboratories, among other uses. As a result, a number of federal agencies  participate in a range of standards development activities that span many  different areas of national need.

Federal law and guidance provide that where possible, agencies are to  use voluntary, private sector standards instead of creating their own  unique standards and are to establish certain responsibilities in  overseeing and coordinating these efforts. The National Technology  Transfer and Advancement Act of 1995 (NTTAA) states that federal  agencies are generally to use technical standards developed or adopted  by voluntary-consensus standards bodies, and in doing so are to consult  with voluntary, private-sector consensus standards bodies and participate  with such bodies in the development of technical standards when such  participation meets certain conditions. NTTAA, as amended, also  provides that it is a function of NIST to coordinate the use of private  sector standards by federal agencies emphasizing where possible the  use of standards developed by private-sector, consensus organizations.

Furthermore, the Trade Agreements Act of 1979 (1979 act) directs the  Secretary of Commerce to keep adequately informed regarding  international standards-related activities and identify those that may  substantially affect the commerce of the United States. The Secretary is  also to monitor the adequacy of U.S. representation in private  international standards activities. The 1979 act says that the  representation of U.S. interests before any private international standards  organizations is to be carried out by a private person recognized as an  organization member. Further, the 1979 act establishes a process for  the Secretary to follow if the Secretary has reason to believe that such  participation will not result in the adequate representation of U.S. interests  or if there is no current organization member. These tasks have been  delegated to NIST. NIST’s memorandum of understanding with ANSI also  describes NIST’s role under the NTTAA, OMB Circular A-119, and the  1979 act to, among other things, ensure adequate representation of U.S.  interests in all relevant international standards organizations and to  coordinate federal activities in voluntary standards.

In addition, Office of Management and Budget (OMB) Circular A-119, as  revised in 2016, sets forth the policy for federal participation in the  development and use of voluntary consensus standards. Federal  representatives are encouraged to participate actively in standards  development activities and to be fully involved in discussions and  technical debates, register opinions, and serve in leadership positions if  selected, among other things. A-119 directs the Secretary of Commerce,  who has delegated this responsibility to NIST, to foster implementation of  the Circular. Further, A-119 provides for a NIST-chaired interagency  group called the Interagency Committee on Standards Policy (ICSP). The  ICSP is composed of agency standards executives—senior-level officials  who are broadly engaged in the agencies’ standards activities. A-119  directs standards executives to coordinate their agencies’ views when  they participate in the same standards activities so as to present,  whenever feasible, a unified position and, when not feasible, mutual  recognition of differences. A-119 directs the ICSP to coordinate with  certain entities with a view to encouraging more strategic and coordinated  federal participation in the development and use of standards in regard to  regulatory policy. According to the ICSP charter, the ICSP has the  objective to promote effective and consistent standards policies in  furtherance of U.S. domestic and foreign goals and, to this end, to foster  cooperative participation by the federal government and U.S. industry and  other private organizations in standards activities, and its purpose is to  ensure effective federal participation in domestic- and international- standards activities.

Further, in 2012 the Executive Office of the President (EOP) issued a  memo for federal agencies to clarify principles guiding federal  government engagement in standards activities that can help address  national priorities. According to the memo and A-119, federal  engagement in standards activities should be guided by the following  strategic objectives:

Produce timely, effective standards and efficient conformity  assessment schemes that are essential to addressing an identified  need.

Achieve cost-efficient, timely, and effective solutions to legitimate  regulatory, procurement, and policy objectives.

Promote standards and standardization systems that promote and  sustain innovation and foster competition.

Enhance U.S. growth and competitiveness and ensure non- discrimination, consistent with international obligations.

Facilitate international trade and avoid the creation of unnecessary  obstacles to trade.

To address these strategic objectives, the memo notes that the federal  government works with the private sector to address common standards- related needs, while taking on a more active role where necessary to  ensure a rapid, coherent response to national challenges. The memo also  identifies responsibilities for agencies, such as periodically reviewing their  standards activities to identify gaps in representation for mission-critical  areas as part of their long-range planning and ensuring effective intra-  and inter-agency coordination of engagement in standards development  activities.

NIST Faces Several Challenges in Providing Measurement Services and Supporting Documentary- Standards Development

We identified three areas where NIST faces challenges in providing  measurement services and supporting documentary-standards  development, based on our literature review, NIST stakeholder focus  groups, and interviews with stakeholders and agency officials. First, the  breadth of U.S. industry and the number of SDOs, among other factors,  make identifying and prioritizing measurement service and standards  needs and communicating with stakeholders about NIST’s services  challenging. Second, ensuring adequate U.S. representation in  international standards activities can be challenging due to the number of  activities and other factors. Third, the involvement of multiple agencies  and interdisciplinary issues makes coordinating among federal agencies  on documentary standards challenging.

NIST Faces Challenges Identifying and Prioritizing Measurement Services and Documentary- Standards Activities, and Communicating with Stakeholders Can Be Challenging

The breadth of U.S. industries and commercial sectors with  measurement-service and documentary-standards needs, and other  factors, make it challenging to identify and prioritize among these needs,  and make it challenging for NIST to ensure stakeholders are aware of the  agency’s services.

Identifying measurement-service and documentary-standards  needs: Identifying measurement-service and documentary-standards  support needs can be challenging, according to participants in all five of  our focus groups and other NIST stakeholders we interviewed.  Participants and stakeholders identified several factors that contribute to  these challenges, including difficulty identifying needs across the breadth  of U.S. industries and standards development activities, and difficulties  presented by emerging, crosscutting, or interdisciplinary technology  areas.

The breadth of U.S. industries and standards development activities can  make it difficult to identify their measurement service needs. NIST’s  potential customer base covers the entirety of the U.S. manufacturing  sector and many service sectors, including small- and medium-sized  enterprises, as well as federal agencies and state and local governments.  Identifying needs across the full range of this customer base can be a  challenge, according to participants in all five of our focus groups and  other stakeholders. Further, NIST officials noted that even within an  industry sector, stakeholders may have differing views on the industry’s  measurement service needs, which can make it harder to determine  whether or how NIST should take action to meet those needs.

Similarly, the diversity of documentary standards activities across many  SDOs may make it difficult to identify when industry needs NIST staff  participation in documentary standards efforts. There are no restrictions  on which organizations may develop standards, and therefore, the total  number of SDOs is not precisely known. However, ANSI estimates that  there are hundreds of such bodies in the United States, and NIST has  reported participating in 114 separate SDOs. Participants in four of our  focus groups and two agency standards executives said that it can be  difficult to keep track of SDOs or standards development activities, and  NIST standards officials noted that the breadth of active SDOs and  volume of their activities was an ongoing challenge. Similarly, three  agency standards executives we interviewed said that identifying  standards activities of interest to their agencies is challenging due to the  large number of activities.

Furthermore, emerging, crosscutting, or interdisciplinary technology areas  can be a challenge, according to participants in all five of our focus  groups. For example, participants in three focus groups discussed the  difficulties faced by organizations that work in areas that combine multiple  areas of technical expertise. A participant in one focus group cited  electronic health records as an example of an interdisciplinary  technology, as it includes biomedical research, public health research,  and information technology. Another participant cited increasingly high- tech development in biological devices that involve physics, engineering,  and mathematics. Participants said that organizations need to coordinate  across disciplines and break down communication barriers to address  these challenges. Additionally, representatives from one SDO we  interviewed as well as an agency standards executive we spoke with  highlighted the difficulty associated with predicting the trajectory of future  change in emerging technologies. NIST officials noted that taking action  to support the measurement service and standards needs of emerging  technologies may be more challenging where there is a lack of industry  consensus on how a technology will develop.

Prioritizing among needs: Prioritizing among different measurement  services can also be challenging. Participants in all of our five focus  groups said that NIST must prioritize among measurement service needs  because it does not have the resources to provide services for all industry  needs. Participants in three of our five focus groups described challenges  balancing between continuing older measurement services that serve  current needs and creating new services. Further, a 2017 review of the  activities of NIST’s Material Measurement Laboratory by the National  Academies found that stakeholders have high demands for the laboratory  and that it faces challenges balancing between maintaining ongoing  efforts and initiating new efforts. A participant in one focus group also said  that it can be difficult to prioritize between services that have broad use  and those that are vital to narrower customer bases. For example, NIST  performs calibrations for thermometers across a wide range of  temperatures for use in many different sectors. On the other hand, NIST’s  million-pound deadweight machine provides calibrations for very large  force gauges used by aerospace manufacturers and the U.S. military.  Focus group participants and NIST officials said that the volume of  services provided may not reflect the value of the service to the industry,  because a single calibration can support many millions of dollars of  economic activity.

When prioritizing staff participation in documentary standards activities,  NIST faces a similar challenge. Specifically, the abundance of ongoing  standards development activities means NIST staff may have to choose  among several standards development activities in their areas of  expertise. While some staff may have expertise that is closely linked to a  small number of SDOs and activities, others may have expertise in  foundational technologies that have relevance to numerous activities.  Further, a participant in one focus group and two agency standards  executives we interviewed stated that standards need to be revised from  time to time, for example, to incorporate new technologies, and these  revisions may compete for time and attention against new standards  efforts in related areas. While individual SDOs can plan for and prioritize  among their new standards efforts and revisions, NIST staff who  participate in standards development efforts across a number of SDOs  may still have to choose among contemporaneous efforts.

Communicating with stakeholders: Communicating with stakeholders  about NIST’s measurement services can be challenging, according to  participants in all five of our focus groups and other stakeholders.  Specifically, the breadth of potential users of NIST’s measurement  services makes it more difficult for NIST to communicate with industry  about its needs and NIST’s services.

Participants in four of the five focus groups said that it can be difficult for  potential users to be aware of and understand the services NIST provides  relevant to their needs. For example, participants in one focus group  described concerns regarding how well they, and industry stakeholders  generally, understand the extent of NIST’s capabilities within their areas  of expertise. Participants in this focus group cited benefits of having NIST  and industry staff perform site visits to elucidate each other’s needs and  capabilities. One NIST calibrations official we interviewed said that some  commercial sectors, such as the automotive industry, may be  underserved by NIST’s services due to a limited understanding of how  NIST could help companies remain innovative and competitive. However,  participants in one focus group said that NIST’s engagement with the  industrial community is generally quite strong.

Other stakeholders suggested that NIST faces an increasingly difficult  task educating potential customers about its services because those  customers may have less technical expertise today than they did in the  past. Participants in one focus group, officials from the Department of  Energy, and a NIST reference material official said that the portion of the  nation’s workforce trained in measurement and standards issues is  shrinking and that industry representatives now have less experience in  these matters than they used to. Accordingly, NIST now communicates  with stakeholders who have less expertise about its measurement  services.

Ensuring Adequate U.S. Representation in International Documentary-Standards Activities Can Be Challenging

According to NIST standards officials, focus group participants, other  stakeholders, and a NIST report on U.S. representation in international  documentary-standards activities, ensuring adequate U.S. representation  in these activities can be challenging. Several factors, such as the large  number of international standards activities occurring across numerous  industry sectors, underlie this challenge and make it difficult for NIST to  ensure adequate U.S. representation.

First, the breadth of the global economy and the volume of international  documentary-standards development activities make ensuring adequate  U.S. representation challenging, according to NIST standards officials,  stakeholders, participants in two focus groups, and literature we  reviewed. For example, NIST standards officials, two agency standards  executives, participants in one focus group, and literature we reviewed  said that the large number of SDOs and volume of international standards  activities presents a challenge to NIST. A participant in a different focus  group also said that in some cases, industry is reliant on NIST to provide  them with information on international standards activities relevant to  them. As the number of activities increase, it can be difficult to maintain a  comprehensive understanding of what is happening in various industry  sectors and standards areas. Further adding to this challenge, several  sources of information we collected identified a significant increase in the  number of international standards activities or the relative participation of  other countries in these activities, for example:

NIST officials and participants in one focus group said that  international SDOs, such as the ISO, are expanding their efforts to  create global standards. Participants said that ISO’s efforts could  conflict with existing standards that U.S. industry uses.

Two stakeholders we interviewed said that U.S. industry also faces  increasing competition from other countries, such as China, which, in  some cases, is overwhelming the ability of U.S. industry to participate.  Further, according to a 2012 testimony to Congress by the director of  NIST’s Standards Coordination Office, other countries have made  significant investments in their standards efforts and have attempted  to increase their participation in international standards activities.  According to the testimony, other countries increasingly view  standards as a tool to increase their international competitiveness and  are developing strategies and tactics to play a greater role in  standardization, such as increasing their participation and leadership  in international standards bodies.

A 2014 NIST report on U.S. representation in international SDOs  showed that the United States fell from first in 2005 to second in 2012  in the number of experts participating in one international SDO, the  IEC, which produces standards for electric and electronic products,  systems, and services. Specifically, the number of technical experts  from the top ten countries that participate in the IEC was 5,528 in  2005 and 9,199 in 2012—an increase of 66 percent. However,  participants from outside the United States were responsible for 85  percent of the increase. Additionally, the report showed that the  United States fell from third in 1999 to fourth in 2012 in the number of  IEC standards proposals submitted. The report showed that the  number of new IEC standards proposed by all countries was 70  proposals in 1999 and 124 proposals in 2012—an increase of 77  percent. However, the percentage of U.S. country proposals out of all  country proposals fell from 19 percent in 1999 to 14 percent in 2012.

NIST standards officials said that there could be additional factors driving  changes in U.S. stakeholder participation in international SDO activities.  For example, NIST officials said that while U.S. stakeholder participation  in ISO and IEC may have declined in some cases, some U.S.  stakeholders have increased participation in other international SDOs  whose standards are better suited for their industry.

Second, what constitutes adequate representation is currently unclear,  according to NIST’s 2014 report on U.S. representation in ISO and IEC  activities and NIST standards officials we interviewed. According to  NIST’s 2014 report, there are no guidelines or definitions given for what is  deemed to be adequate representation of U.S. interests in international  standards activities. Further, NIST standards officials said that it was not  clear what circumstances would lead NIST to use the process established  under the 1979 act if U.S. representation in an international SDO may be  potentially inadequate. NIST officials also said that defining what would  constitute adequate U.S. representation at international SDOs and  collecting the information to help assess the adequacy of U.S.  representation would be difficult and the definition and metrics could vary  by industrial sector.

Third, according to participants in two focus groups, the large number of  companies and other stakeholders that could be involved in or have an  interest in various international SDO activities under the U.S. system of  private-sector-led standards development can make ensuring adequate  U.S. representation a challenge. For example, participants in two focus  groups said that NIST would need to consult with numerous industry  stakeholders or SDOs to facilitate representation in situations where U.S.  representation was inadequate. Further, as we mentioned above,  documentary standards needs in emerging, crosscutting, or  interdisciplinary technology areas can be a challenge. Literature we reviewed highlighted the need for NIST or other agencies to help bring  together industries or other stakeholders that may not have a history of  collaborating to resolve standards issues.

Coordinating Federal Agencies’ Participation in Developing Documentary Standards Can Be Challenging

Fulfilling NIST’s role to work with other agencies to coordinate use of and  participation in standards activities under the NTTAA, as chair of the  ICSP, and in implementing OMB Circular A-119 is challenging due to (1)  the involvement of multiple federal agencies in documentary standards  activities, and (2) increasingly interdisciplinary technology areas.

Multiple agency involvement: Multiple federal agencies are involved in  documentary standards activities, a situation that can make coordinating  agencies’ activities challenging, according to participants in all five of our  focus groups, and some stakeholders and agency standards executives  we interviewed. Because multiple agencies are involved in documentary  standards, agency efforts can be fragmented. Fragmentation refers to  those circumstances in which more than one federal agency (or more  than one organization within an agency) is involved in the same broad  area of national need and opportunities exist to improve service  delivery. We have previously reported that fragmentation of federal  efforts occurs in a number of areas and can lead to challenges to  effective coordination. As we mentioned earlier, federal agencies may  use or help develop documentary standards for several reasons,  including (1) to procure goods or services, (2) to incorporate standards  into agency regulations, or (3) to improve agency operations or further  agency policy goals. As a result, a number of federal agencies participate  in a range of standards development activities that span many different  areas of national need. Further, while some documentary standards  issues may only affect the mission or activities of a limited number of  agencies, other issues may affect many agencies.

Participation by multiple federal agencies in documentary standards  activities can be beneficial, according to some focus group participants,  stakeholders, and agency standards executives. Some focus group  participants, stakeholders, and agency standards executives identified  examples of federal participation in which agencies could leverage their  different strengths and expertise. For example, participants in four focus  groups and some stakeholders we interviewed noted NIST’s unique role  as a non-regulatory and neutral agency in facilitating the development of  standards. These participants and stakeholders said that, in combination  with NIST’s technical capability, this role allowed NIST to gain trust and  cooperation from industry in advancing standards development, whereas  industry may view regulatory agencies as less neutral. Participants in one  focus group said that this role was also helpful to regulatory agencies  because these agencies, such as the Food and Drug Administration,  would not be able to work as closely with industry in regard to solving  technical standards problems or assisting industry because of their  regulatory role. Further, two agency standards executives said that some  standards activities benefit from the expertise of multiple agencies. For  example, one agency standards executive said that evaluating whether  standards or product specifications in other countries constituted a barrier  to trade required the expertise and participation of different agencies. Two  stakeholders we spoke to also said that participation by all relevant  federal agencies in standards activities is beneficial because the agencies  can provide technical expertise and are important stakeholders for  standards efforts because agencies regulate industry, develop policy, and  procure goods from the private sector.

At the same time, participants in all five of our focus groups, some  stakeholders, agency standards executives, and NIST officials we  interviewed cited challenges in coordinating documentary standards  among multiple federal agencies, for example:

Some stakeholders, agency standards executives, and another  federal standards official we interviewed said that communication  between federal agencies on their standards activities can be a  challenge. For instance, three agency standards executives and one  stakeholder said that it can be difficult to identify when other agencies  are working on the same standards areas, and two of the standards  executives said it can be difficult to know who to contact in other  agencies to coordinate efforts.

Three agency standards executives said that it can be difficult for  standards executives to be fully aware of all standards activities in  their department or agency. One standards executive also said that  some standards executives have split responsibilities and are not full  time, a situation that may make it difficult for these executives to  devote sufficient time to understanding their agencies’ standards  activities, particularly in large agencies. Further, NIST officials said  that there is an uneven level of interest and focus on standards as a  policy issue among federal agencies, generally.

Participants in three focus groups cited differing priorities and  interests among federal agencies as a challenge to coordinating on  standards activities. For example, participants in one focus group said  that different interests and priorities among financial regulatory  agencies posed a challenge to coordination on cybersecurity  standards. As we reported in December 2015, NIST undertook a  collaborative process that involved federal agencies as well as  nonfederal stakeholders in developing a cybersecurity framework in  response to executive order and legislative requirements.  Participants in one focus group noted NIST’s efforts to work with a  variety of public and private-sector stakeholders but also said that  financial regulatory agencies have their own cybersecurity regulations  that may not align with NIST’s framework. In a February 2018 report  on implementation of the cybersecurity framework, we noted the  complex regulatory and cybersecurity environment of the financial  sector and noted that sector representatives said that agencies’  differing cybersecurity requirements led to competition among various  cybersecurity frameworks.

NIST officials also said that it can be challenging for federal agencies  to harmonize their views on standards because they each have  individual missions and priorities that may lead to varying views.  Similarly, a 2011 National Science and Technology Council report  cited a lack of coordination among agencies with interests in  standards activities as having a negative impact on government  effectiveness. The report noted that agency objectives may not  always be aligned and that they may be providing redundant support  or competing with one another.

An additional complexity to coordination of federal agencies’ documentary  standards activities is that some standards issues may have multiple  venues for interagency coordination. NIST officials said that agencies  participated in the U.S. private-sector-led standards system and that there  were a number of different organizations and groups through which  federal agencies shared information, depending on the standards activity.  There are at least four groups including the ICSP that provide interagency  coordination on standards issues generally. According to NIST officials,  interagency coordination also occurs through the National Science and  Technology Council’s committees and subcommittees. Further, individual  documentary standards areas may have additional interagency  coordination venues. For instance, interagency coordination on  cybersecurity standards also occurs through the Interagency International  Cybersecurity Standardization Working Group, according to a NIST 2018  draft report. This group was established by the National Security  Council’s Cyber Interagency Policy Committee to coordinate on major  issues in international cybersecurity standardization and enhance U.S.  federal agencies’ participation in these efforts. Furthermore, agencies  may coordinate amongst themselves informally on specific standards  interests, according to two standards executives.

Interdisciplinary technology areas: Documentary standards  development issues have become increasingly interdisciplinary— potentially creating challenges to coordinating agencies’ standards  activities, according to our prior work, literature we reviewed, focus  groups we conducted, and stakeholders we interviewed. As described  above, technology areas, including emerging areas of technology such as  electronic health records, can cut across disciplines. According to  literature we reviewed, our prior work, and stakeholders we interviewed,  interdisciplinary standards can be more difficult to develop or implement  because they can be complex and involve a broader range of industry  and government stakeholders with potentially different interests and  needs. Standards areas are also becoming increasingly interdisciplinary,  according to literature we reviewed, one stakeholder, and participants in  one focus group. In addition, participants in all five focus groups and  some stakeholders we interviewed said that it can be challenging to  facilitate interagency coordination on interdisciplinary standards areas.  For example, some participants and some stakeholders said that these  standards areas can involve the need for collaboration among multiple  agencies that can have different roles and responsibilities, priorities, or  levels of expertise. Further, two stakeholders said that it could be a  challenge for federal agencies to identify these areas.

Our prior work, participants in two of our focus groups, and an agency  standards executive identified several examples of interdisciplinary  standards areas that present challenges to interagency coordination:  In February 2018, we reported that protecting the nation against  complex and growing cybersecurity threats required coordination  between 10 different agencies, 9 of which had responsibility for  coordinating implementation of NIST’s cybersecurity framework  across 16 different critical infrastructure sectors.

In November 2016, we reported that improved interagency  coordination could help to address challenges SDOs face in using  forward-looking climate information—an interdisciplinary standards   area that requires expertise from multiple agencies—and we made a  related recommendation.

One focus group participant identified open source software and a  participant in a different focus group identified machine learning and  artificial intelligence as interdisciplinary standards areas needing  increased coordination among federal agencies.

One agency standards executive also said that federal coordination  could be beneficial for developing standards for “Big Data” because  multiple federal agencies have expertise in Big Data that is not being  leveraged to create standards that could facilitate appropriate use of  Big Data-related technology and techniques.

Establishing a private-public partnership  to coordinate standards development with  different stakeholders.   Implementing a testing framework.

According to NIST officials, developing documentary standards for  interdisciplinary technologies can be more resource intensive because of  the need to pull together expertise from different disciplines and potential  competition among SDOs in developing a standard. However, NIST  officials also noted that standards development for interdisciplinary  technologies may not always be more challenging than other types of  standards development efforts, when an SDO has willing participants with  the necessary expertise. For example, according to a 2010 VCAT report,  NIST established a public-private Smart Grid Interoperability Panel to  identify, prioritize, and address new and emerging requirements for this  interdisciplinary standards area involving many stakeholders and  agencies. According to VCAT, the panel allowed for a wide range of  participating stakeholders and served as an effective way to determine  and incorporate the different needs and interests of participants in a  framework that enabled further development of smart grid standards.

NIST Has Taken Steps to Address Challenges in Providing Measurement Services and Supporting Documentary Standards, but Some Efforts Could Be Improved NIST’s Efforts Help Address Challenges to Identifying and Prioritizing Needs but Could More Fully Align with Federal Guidance and NIST Policy

NIST works to address the challenges it faces in providing measurement  services and supporting documentary-standards development in a variety  of ways, but opportunities exist to improve some efforts. First, NIST’s  efforts help address challenges to identifying and prioritizing  measurement-service and documentary-standards needs, but some  efforts do not fully align with federal guidance or NIST policy. Second,  NIST’s efforts help support U.S. representation in international standards  organizations but may not fully implement its role and address the  challenge it faces. Third, NIST’s efforts support federal agency  coordination on standards issues but do not fully align with selected  leading collaboration practices.

NIST takes a variety of steps to identify agency stakeholders’  measurement-service and documentary-standards needs, has  procedures to support the prioritization of measurement services and  standards activities in the agency, and has developed mechanisms for  communicating with stakeholders. While these steps help NIST address  challenges to identifying and prioritizing needs and communicating with  stakeholders, some aspects of these efforts do not fully align with federal  guidance or NIST policy.

Identifying measurement-service and documentary-standards  needs: NIST identifies stakeholders’ measurement-service or  documentary-standards needs in various ways, including discussions with  industry by NIST technical staff, attendance at trade shows and scientific  professional society meetings, workshops hosted by NIST on technology  areas of interest, and participation in planning activities of SDOs, among  other ways. However, NIST does not regularly perform a comprehensive  assessment of its measurement services and standards activities to  identify and address any gaps between the agency’s efforts and industry  needs.

Genome in a Bottle   The Genome in a Bottle consortium is one of  several ongoing collaborations among the  National Institute of Standards and  Technology (NIST), Stanford University, and  various industry and government partners that  focus on measurements and standards  supporting the newest developments in  biology. Genome sequencing involves  determining the chemical building blocks of  deoxyribonucleic acid (DNA) or ribonucleic  acid (RNA) and can give insights into the  genes carried by an individual and how and  when they are activated. Since the completion  of the Human Genome Project in 2003 which  first sequenced the whole genome of a  human, scientists have worked to make whole  human genome sequencing faster and less  expensive. Genome in a Bottle aims to  develop the tools needed to allow clinical use  of whole human genome sequencing. These  tools include reference materials that allow  laboratories to ensure the reliability and  accuracy of their sequencing equipment,  increasing laboratories’ capability to perform  genetic testing, medical diagnoses, and  customize drug therapies.

NIST’s primary method for assessing stakeholders’ needs is through  outreach by individual technical staff and their expertise in relevant  disciplines and related industries. Measurement services officials said  that NIST’s staff work closely with their respective industry stakeholders  and others to understand their measurement service needs. They stated  that NIST staff engage with industry through direct contact at conferences  and trade shows, company and NIST laboratory visits, training, NIST-led  workshops, through their ongoing research activities, and other activities.  NIST also collaborates with other metrology organizations to identify  measurement service needs and advance measurement science. For  example, measurement services officials described NIST’s participation in  international organizations that develop strategic plans for calibrations  and measurement standards, such as the Bureau International des Poids  et Mesures and Inter-American Metrology System. These organizations  allow national metrology institutes, like NIST, to collaborate with their  peers and other stakeholders to improve the world’s measurement  standards and services.

NIST also collaborates with government, industry, and research  institutions on emerging issues through various collaboration  mechanisms. For example, NIST formed the Genome in a Bottle  Consortium in 2011. It provides an open forum for discussion and  planning for reference materials and other measurement infrastructure  needed to use human genomic sequencing in clinical settings. Similarly,  NIST’s Advanced Materials Center of Excellence allows the agency to  work with universities, a government lab, and others to address research  and development needs related to designing novel materials for  manufacturing. Furthermore, since 1905 NIST has participated in  activities of the National Conference on Weights and Measures. Recent  activities of this group include developing measurement practices and  measurement standards to ensure that ride-sharing companies accurately  measure time and distance charges.

The National Institute of Standards and  Technology (NIST) began an active mercury  reduction campaign in 2007 and stopped  calibrating mercury thermometers entirely on  March 1, 2011. NIST's Temperature and  Humidity Group is actively participating in  several U.S. and international phasing out  efforts to identify alternative thermometers for  a broad range of measurement applications,  and to coordinate efforts to replace mercury- based instruments. For example, the  Minamata Convention on Mercury is a global  treaty to protect human health and the  environment from the adverse effects of  mercury and includes a phase out of the use  of mercury in products and processes.  According to NIST officials, NIST worked  closely with one standards development  organization, ASTM International, to develop a  new standard for the manufacture and  selection of digital thermometers. This  standard describes three different types of  digital-thermometer sensors and defines  different classes of devices based on  accuracy and, according to NIST, allowed  ASTM to revise over 750 additional standards  to replace required mercury thermometers  with digital thermometers. NIST provides  calibrations for all three types of sensors to  the worldwide measurement standard, the  International Temperature Scale.

Furthermore, NIST examines trends in the measurement services it  provides to better understand industry needs. In particular, NIST conducts  individualized testing for companies known as special tests that can give  the agency insight into industry’s needs. Special tests comprise  calibrations and related measurements that are unique to the customer  and are not part of NIST’s regular catalogue. A company may request a  special test, for example, to evaluate a prototype product or measurement  technology. According to NIST officials, NIST uses special tests as a way  to meet industry needs and also understand what kinds of measurement  services industry may need more of in the future.

Similarly, NIST standards officials also described staff expertise as  important for identifying stakeholders’ needs for support in the  development of documentary standards. In particular, standards officials  described staff participation in roadmapping activities—used to identify  and plan for future standards activities in certain fields—sponsored by  NIST or SDOs as important opportunities for staff to assess the standards  landscape and identify needs. For example, NIST officials noted the  importance of NIST participation in ANSI’s standards panels and  collaboratives, some of which are co-led by NIST staff, for identifying  standards needs. NIST also participates in SDO administrative groups,  such as the ANSI Government Members Forum, that can alert NIST to  important international and domestic standards activities. Through  participation in SDOs, NIST standards officials said that NIST may obtain  information on international standards activities in which U.S. industry  representation is needed. NIST officials said that developing new  documentary standards can take from a year and a half up to a decade to  complete, and accordingly, NIST considers what the standards industry is  likely to need in the next 1 to 5 years.

Various thermometers at NIST’s temperature  calibration lab.

In addition, NIST gathers information on how its efforts align with  stakeholders’ needs through feedback from industry customers and  external reviews by the National Academies, VCAT, and others. For  example, NIST asks the users of its calibration services and standard  reference materials to respond to customer satisfaction surveys. NIST  measurement services officials said that while a small percentage of  customers respond to the surveys, the information gathered provides  useful input on what new services customers need. By 2018, NIST  implemented new information systems to track its measurement services’  sales and customers, and is evaluating if the agency’s outreach to these  customers can be improved using the new systems. According to NIST  officials, NIST also receives feedback when stakeholders contact the  agency through phone, email, or the NIST website. Further, the National  Academies evaluated NIST’s Material Measurement Laboratory in 2017  and has reviewed every NIST laboratory since the 2010 reorganization. In  a 2017 report, the National Academies recommended that NIST’s  Material Measurement Laboratory develop a strategy to balance between  existing product support and the research, production, and certification of  new standard reference materials. Recent VCAT reports have also  considered how well NIST identifies measurement-service and  documentary-standards needs of its customers and assessed the  agency’s services:  In a 2009 report, VCAT examined NIST’s participation in standards  development in three specific areas and found that NIST’s technical  expertise, its reputation as an unbiased and neutral party, and its  extensive participation in standards activities strongly position NIST to  address the standards-related challenges of the 21st century.

In a 2010 report, the VCAT found that NIST’s analysis and planning  practices for its measurement services tended to be driven by bottom- up initiatives more than high level strategy and in 2010 and 2011  reports the VCAT recommended, among other things, that NIST  perform additional assessments of its measurement services.

In 2012 and 2013 reports, the VCAT found that NIST’s participation in  standards activities has helped the agency identify industry needs  related to advanced manufacturing.

Additionally, in 2015, NIST received a peer review of its measurement  services by experts from other national metrology institutes. According to  the peer review summary, most of the eight individuals from other  national metrology institutes who reviewed NIST’s measurement services  found that the agency covered major needs, and the reviewers gave NIST  additional feedback on areas for expansion the agency should consider.

Participants in three of our five focus groups said that when NIST focuses  on a specific area, its efforts to understand industry needs can be very  effective. For example, participants in one group said that NIST creates  an open environment for discussions with industry and companies feel  comfortable approaching the agency with their needs. Participants in  another group said NIST’s regular contact with and surveys of state  metrology labs help the agency understand their needs. However,  participants in all five of the focus groups said that NIST’s capacity for  outreach is limited. For example, participants in one focus group said that  NIST’s outreach efforts can be driven by the personal relationships NIST  staff develop with stakeholders and therefore do not scale to the large  size of U.S. industry. NIST officials said that its measurement services  and documentary-standards support activities serve different populations  of stakeholders and that identifying the needs of NIST’s measurement  services’ stakeholders is more manageable than with documentary  standards. Specifically, by working with measurement equipment  manufacturers, NIST officials said that understanding the needs of its  measurement services’ stakeholders was manageable. However, NIST  officials said it is more difficult to know the measurement-service needs of  industry stakeholders that work with equipment manufacturers rather than  with NIST directly.

NIST’s efforts to identify industry needs are supported by agency policy,  and NIST has controls in place to evaluate the efficacy of the  measurement services it provides. NIST policy directs staff to consider  stakeholders’ measurement service needs and assign responsibility for  assessing measurement services to agency management. Specifically,  the NIST Quality Manual, which contains the agency’s policies and  procedures governing its measurement services, describes meeting and  anticipating the needs of measurement services’ users as a goal for the  agency. The manual encourages staff to identify improvements to  measurement services and assigns ultimate responsibility for providing  services that meet industry, academia, and other government agency  needs to the Associate Director for Laboratory Programs. Further, the  Quality Manual requires multiple levels of review of the agency’s  measurement services, including internal audits at the division level,  quarterly management review by measurement services officials, and  peer-review by a team of experts from other NIST divisions. The  assessments are to provide NIST with assurance that its measurement  services, and especially the calibration and measurement capabilities,  continue to be in compliance with its quality management system.  Further, NIST officials told us that the agency was considering  measurement services as part of its strategic-planning efforts, but those  efforts were preliminary at the time of our review.

For documentary standards, NIST’s policies for staff participation in  standards development encourages staff participation in domestic- and  international-standards development activities whenever such  participation is in the public interest and is compatible with NIST’s  mission, policies, positions, priorities, and available resources. NIST’s  standards participation policy also provides that the Associate Director for  Laboratory Programs conduct periodic reviews of the effectiveness of  NIST’s participation in documentary standards activities, with support  from the Standards Coordination Office. Additionally, the policy directs  NIST managers to annually review records of SDO participation by staff in  their divisions and calls for laboratory and division managers to  periodically review activities to identify gaps in representation for mission- critical areas. Further, NIST officials said that across both measurement  services and documentary-standards support activities, the efforts of its  staff to meet stakeholders’ needs are assessed via employee  performance reviews, among other means.

NIST’s multi-level assessments of its measurement services and  documentary standard development activities help ensure their quality  and help to identify stakeholders’ needs; however, these assessments do  not comprehensively identify and assess gaps in NIST’s services or how  well they align with stakeholder needs. For example, NIST’s efforts to  identify measurement service needs for individual technology areas or  industries, or to evaluate the services provided by its labs—both areas of  strength for NIST—may not identify gaps in service needs for technology  areas not evaluated or that cut across NIST’s labs. Officials working on  calibration services and reference materials told us that NIST has not  performed a comprehensive assessment of how well its services address  industry needs since a 2006 assessment of the U.S. measurement  system. NIST measurement services officials raised concerns about the  value of this type of review, describing the agency’s 2006 assessment as  time consuming and ultimately of limited use in identifying unmet  measurement needs. However, members of the 2015 peer review of  measurement services said that NIST would benefit from strategic  assessments to identify and assess gaps in programs and a calibration  official told us that it is difficult for NIST to recognize if it is not effectively  reaching stakeholders. Similarly, NIST officials told us that the Associate  Director for Laboratory Programs does not perform a periodic review of  the effectiveness of NIST’s standards participation, despite the agency’s  standards participation policy calling for such a review. One standards  official said NIST generally does not comprehensively assess standards  needs because of the number and diverse nature of standards activities.

Federal standards for internal control direct management to use quality  information to determine if the agency is meeting its objectives and to  identify, analyze, and respond to significant changes that could present  risks to achieving its objectives. In addition, the 2012 EOP memo on  standards activities in areas of national priority states that agencies  should periodically review their standards activities to identify gaps in  representation for mission-critical areas as part of their long-range  planning. Revising NIST’s policies to provide for periodic comprehensive  management reviews of NIST’s measurement services would augment  NIST’s ongoing efforts to assess how well its services align with  stakeholder needs and identify any gaps. Conducting comprehensive  reviews of NIST’s measurement services and documentary standards  activities would provide NIST with greater confidence that its activities align with stakeholders’ needs, consistent with internal control standards.  Conducting such reviews would also help NIST address the  recommendations made by its recent external reviews and could be used  to support NIST’s efforts to develop the strategic plan called for by the  American Innovation and Competitiveness Act.

Prioritizing among needs: NIST has a process for deciding when new  measurement services are warranted; however, decision-making about  documentary-standards development activities is decentralized, and NIST  management and staff may not have clear guidelines or sufficient  information to support decision-making about new standards activities.

NIST has processes to guide decision-making about measurement  services; this guidance helps address the challenges focus group  participants identified given that NIST cannot address all needs. Before  choosing to develop a new measurement service, such as a new  standard reference material or calibration service, NIST’s Office of  Reference Materials and Calibration Services office, respectively,  consider the need for and priority of the service. One NIST official said  that because NIST cannot cover all measurement services that may be  needed by the private sector, the measurement services program focuses  on the areas where NIST may have the most impact. NIST has  procedures in place to evaluate proposals for new services. For example,  before NIST develops a new reference material, Office of Reference  Materials and relevant laboratory staff annually review proposals for new  materials by evaluating factors such as the potential user base for the  material, related legislative or regulatory requirements, and whether the  material could be produced by other organizations, such as private  companies. The Office of Reference Materials also considers these  factors when considering extending reference materials it already  provides. Further, because developing a new reference material can be  time consuming, NIST is currently evaluating the creation of a suite of  reference materials called “research-grade materials” that could address  high priority areas with a lower level of precision than NIST’s standard  reference materials. According to NIST officials, research-grade materials  are one way of providing this type of measurement service faster to meet  the needs of U.S. industry.

Similarly, new calibration proposals are reviewed by Calibration Services  management on a quarterly basis and are evaluated on factors such as  stakeholder need and potential impact. In addition, NIST extends its  reach through its work with private sector test and calibration companies  that also serve the needs of U.S. industry. For example, the National  Voluntary Laboratory Accreditation Program allows NIST staff to directly  interact with test and calibration laboratories and provides opportunities to  share NIST’s expertise and improve services offered by these  laboratories. Further, a focus group participant and government  laboratory officials described industry association meetings as important  opportunities to find out about cutting-edge capabilities and potential  future measurement capabilities offered by NIST and others.

NIST does not have a similar formalized process to support consistent  decision-making across NIST laboratories and divisions about  participation in new documentary standards activities. As described  previously, NIST’s policy for staff participation in standards development  encourages staff participation in domestic- and international-standards  development activities. Additionally, NIST guidance directs staff to  participate in SDO activities based on their unit’s mission and goals, and  the technical competence required, among other factors, and advises that  staff may choose to accept leadership positions in these activities, such  as the secretary or chair of a standards committee. OMB Circular A-119  and the 2012 EOP memo on standards also encourage agencies to play  a variety of roles in the standards process, such as serving as  chairpersons or other official capacities. Focus group participants had  mixed opinions on when NIST staff should take on leadership roles.  Participants in four of our focus groups said that NIST staff are  particularly suited to leadership roles, and some attributed this conclusion  to the technical expertise of NIST staff or their ability to act neutrally  among competing companies. However, participants in two focus groups  said that NIST is better suited to a technical advisor role.

However, NIST policy and guidance do not describe when it is  appropriate for staff to take leadership roles in standards development  activities. Individual staff in consultation with their supervisors determine  what standards activities, if any, they should participate in and their  appropriate role in the standards development activity. According to NIST  officials, other levels of management may be involved in the decision- making process to varying degrees depending upon whether participation  in an SDO activity aligns with a NIST priority or where involvement entails  international travel, among other factors. A guidance document for staff  encourages them to attend additional training provided by the Standards  Coordination Office, and standards officials we interviewed told us that  the training and informal guidance provided by the office could help staff  in leadership positions; however, such roles may entail additional time  commitments. Although some stakeholders have expressed interest in  increasing NIST’s participation and leadership in standards activities,  doing so could entail tradeoffs between these activities and other NIST  priorities. Without clear agency guidance on staff participation in  standards development activities, such as the factors staff could consider  when deciding to take on leadership or other more active roles, NIST  cannot be assured that decisions on the time staff commit to standards  activities are being made consistently across the agency and in  accordance with agency priorities.

Further, NIST’s ability to ensure participation is appropriately prioritized  across the many documentary-standards development activities in which  its staff could be involved is limited by incomplete information. Staff are  directed in NIST policy to record their participation in standards activities  in a centralized database, including a description of the SDO, specific  activity, and role of the NIST participant. According to a NIST standards  official, the database may be used by laboratory managers to assess the  standards activities of their staff. However, NIST’s database does not  contain information regarding NIST staffs’ time commitment, information  that could be used by management to assess the resources required for  participation in these activities. NIST guidelines also direct staff to  document their goals and time commitments for standards activities in  their individual performance plans, but the data are not included in the  standards participation database.

According to NIST officials, determining the time spent on documentary  standards can be difficult. Specifically, staff participate in standards  development within their areas of expertise and often this work is closely  related to their research activities at NIST. Because of this confluence,  the amount of time staff spend on a particular standards activity may be  unclear. The time spent at meetings or directly drafting or responding to  standards documents will also depend on the amount of consensus on  the standards committee, consensus that may not be known ahead of  time. NIST standards officials told us that the self-reported data currently  in the database are sufficient for laboratory management to identify what  activities staff are participating in, and that management can then ask  individual staff for additional information. However, NIST does not have  data at an aggregate level on the time staff commit to or expect to commit  to these activities. Standards for internal control require agencies to use  quality information to achieve the entity’s objectives, such as by using  relevant data from reliable internal and external sources in a timely  manner based on the identified information requirements. While staff  document their roles in documentary standards activities, without  information on the estimated amount of time staff commit to these  activities, NIST management may not have the information needed to  comprehensively assess how staff distribute their limited time and  attention. Although it may not be feasible to determine the exact amount  of time spent on documentary standards activities, information on  estimated amount of time could help inform staff decision-making on  when to accept leadership roles in standards development activities and  could inform management on trends in time commitments to these  activities across the agency’s laboratories and divisions.

Communicating with stakeholders: NIST takes a number of steps to  address the challenges it faces communicating with its diverse  stakeholders about its measurement services and documentary  standards activities. NIST measurement services officials described the  primary goal of their stakeholder outreach efforts as informing potential  customers of the services NIST provides. The officials described multiple  avenues for reaching potential customers of NIST’s measurement  services, including: attendance at workshops, trade shows, and professional societies;

NIST’s measurement services websites; email and newsletter correspondence with current customers; direct contacts between individual staff and stakeholders; and  research papers and other scholarly activities.

For example, NIST distributes a newsletter to customers that includes  information on upcoming changes to the agency’s standard reference  materials.

NIST has also taken steps to better target its stakeholder communication.  For example, NIST measurement services officials described an effort to  evaluate customer interest in NIST’s standard reference materials, as  expressed through contact with NIST staff at trade shows. As a result of  this analysis, NIST reduced the number of trade shows at which it  advertised these materials—focusing on those trade shows that were  identified as having the greatest number of interested attendees. More  broadly, by 2018 NIST implemented new information systems supporting  its measurement services sales, inventory, and customer relationship  management. Measurement services officials described efforts currently  underway to take advantage of these systems to better target customers  by, for example, providing email notifications to customers of new  materials or improved measurements of current materials.

NIST’s Efforts Help Support U.S. Representation in International Documentary Standards Activities but May Not Fully Address the Challenge

NIST’s efforts help support U.S. representation in international standards  organizations but may not fully implement its role and address the  challenges it faces.

NIST works to support U.S. industry’s efforts to ensure that its interests  are adequately represented in international standards activities. For  example, NIST officials said that NIST staff participate broadly in  international standards activities that are aligned with NIST’s priorities  and share their technical expertise in various committees. Through its  participation in SDOs, NIST may obtain information on international  standards activities in which U.S. industry representation is needed. NIST  also shares information on international standards activities with U.S.  industry. NIST hosts the World Trade Organization Inquiry Point, a U.S.  government website that serves as a communications hub for information  on international standards and related issues. Through the website, U.S.  industry and other stakeholders receive notifications of standards-related  regulations and procedures, as well as the basis and objective for  proposed measures, among other information provided. The website also  provides a mechanism to circulate comments on proposed measures.  Further, we noted above that NIST participates in ANSI standards panels  and collaboratives and the ANSI Government Members Forum. NIST’s  participation in these bodies can alert NIST to important international  standards activities.

NIST officials said that when NIST has become aware of concerns about  U.S. representation at an international standards activity within a federal  government area of responsibility, it has led efforts to ensure adequate  representation in those activities. For example, according to NIST  officials, NIST:

Worked with the U.S. Patent and Trademark Office to help identify  another organization to represent U.S. interests in a standards activity  that affected U.S. intellectual property after the original organization  decided not to continue participating.

Established and administered the U.S. technical advisory group for a  new ISO technical committee on biotechnology after industry and  many federal agencies chose not to participate.

Took on a leadership role to represent U.S. interests in a standards  activity at the International Telecommunication Union when no U.S.  telecommunications companies took on responsibility for representing  U.S. industry in the activity.

NIST has also issued three reports on U.S. representation in international  SDOs. In June 2014 NIST issued its most recent report on U.S.  representation in two international SDOs, the ISO and IEC—its two prior  reports were published in 2000 and 1988. The 2014 report describes  U.S. representation in ISO and IEC activities from 1966 through 2012. As  noted previously, the report also describes U.S. memberships and roles  in ISO and IEC standards development committees, and includes data  comparing U.S. representation on these committees with representation  from other countries.

While NIST has helped support U.S. representation in international  documentary standards activities, NIST has not developed a mechanism  to implement the role delegated to it under the 1979 act to address  circumstances when U.S. representation at international standards  organizations may be inadequate. As noted previously, the 1979 act  directs the Secretary of Commerce to coordinate with the U.S. Trade  Representative (USTR) and keep adequately informed regarding  international standards-related activities and identify those that may  substantially affect the commerce of the United States. The Secretary is  also to monitor the adequacy of U.S. representation in private  international standards activities. Further, the 1979 act establishes a  process for the Secretary to follow to address circumstances in which  U.S. representation may be inadequate, specifically:  If the Secretary, after an inquiry, has reason to believe that the  participation by an organization member in the proceedings of a  private international standards organization will not result in the  adequate representation of United States interests that are, or may  be, affected by the activities of such organization (particularly with  regard to the potential impact of such activity on the international  trade of the United States) the Secretary shall immediately notify the  organization member concerned.

The organization member has a 90-day period following the  Secretary’s notification to demonstrate its willingness and ability to  represent adequately U.S. interests. If the organization member  demonstrates willingness and ability, the Secretary should take no  further action.

If the organization member either does not respond or does not  demonstrate the requisite willingness or ability to represent U.S.  interests or there is no organization member of the private  international standards organization—the Secretary is to make  arrangements to provide for the adequate representation of U.S.  interests.

Although NIST has reported on the extent to which the U.S. participates  in some Geneva-based international standards development activities,  these reports do not assess the adequacy of this participation. NIST  officials we interviewed said that the agency does not assess whether  U.S. interests are adequately represented and does not have definitions  of or guidelines for what constitutes adequate representation. NIST  officials also told us that the agency has not evaluated the circumstances  under which it would follow the procedures under the act for addressing  inadequate representation. Federal standards for internal control indicate  that management should identify, analyze, and respond to risks related to  achieving an agency’s objectives. As noted previously, the large number  of international standards activities occurring across numerous industry  sectors, among other factors, present challenges to ensuring adequate  U.S. representation in international standards activities.

In commenting on a draft of this report, Commerce stated that a  determination to follow the statutory process in the 1979 act would carry  significant risk of being perceived by the national and international  standards community as a U.S. government change of policy relating to  the nation’s private-sector-led standards system. However, the  memorandum of understanding between NIST and ANSI states that  NIST’s role, under the NTTAA, OMB A-119, and the 1979 act is “to  ensure adequate representation of U.S. interests in all relevant  international standards organizations.” Further, NIST has previously taken  action in some cases in international standards activities within a federal  government area of responsibility, as described above. Additionally, the  ongoing contacts between NIST and SDOs and staff participation in  standards activities can help NIST keep adequately informed on  international-standards-related activities.

Without a mechanism to identify and respond to circumstances when U.S.  representation at international SDOs may be inadequate, such as  guidelines for what constitutes adequate representation and when and  how to follow the process under the 1979 act, NIST may miss  opportunities to take action in furtherance of its mission to support U.S.  competitiveness by helping to ensure adequate U.S. representation in  international standards activities. Alternatively, given Commerce’s  concerns about the statutory process in the 1979 act, NIST could develop  a legislative proposal that allows NIST to ensure adequate U.S.  participation in international standards activities while addressing those  concerns.

NIST’s Efforts Help Support Interagency Coordination, but Opportunities Exist to Better Follow Leading Practices for Enhancing Collaboration

NIST supports coordination among federal agencies on documentary  standards issues as chair of the ICSP, as well as additional coordination  efforts outside of the ICSP. However, aspects of the ICSP’s efforts do not  fully align with selected leading practices for enhancing and sustaining  interagency collaboration. These practices can help agencies manage  fragmentation and other coordination challenges.

NIST has taken several steps to address the challenge of interagency  coordination on documentary standards issues through its efforts as chair  of the ICSP:

NIST and member agencies have a charter that outlines the purpose,  functions, and membership of the ICSP, among other information.  According to the charter, the ICSP was established to advise the  Secretary of Commerce and the heads of other federal agencies in  matters relating to standards policy. The ICSP’s purpose under the  charter is to ensure effective participation by the federal government  in domestic- and international-standards activities, among other  things. NIST officials we interviewed said that the ICSP meets three to  four times per year and that the purpose of the ICSP is to promote  effective participation by federal agencies in the standards process— when it is within an agency’s mission and in the public interest—but  not to force their participation. According to NIST officials, NIST tries  to demonstrate the benefits of participation and encourages other  agencies to participate actively in relevant standards activities.

NIST chairs and supports ICSP activities, including providing  administrative services, organizing meetings, and developing agendas  and reports. We previously reported in 2012 that designating a lead  agency can assist in driving accountability and providing for continuity  of leadership for a collaborative effort. According to NIST officials we  interviewed, ICSP meetings are open to agencies outside of the  member agencies. Further, NIST officials said they routinely invite  staff from non-member agencies when the committee plans to discuss  items of particular interest to them. NIST officials said that the ICSP  network allows NIST to provide a knowledge base for other federal  agencies, and to help federal staff understand standards policy and  participation in SDOs, among other things.

According to ICSP members and NIST officials, NIST facilitates the  sharing of best practices on broad standards topics affecting multiple  agencies through the ICSP. As of June 2018, 30 federal agencies  have identified participants to the ICSP, while 5 agencies have vacant  positions on the committee. NIST officials provided several examples  of its information-sharing activities:

NIST led ICSP efforts to facilitate discussion on, and manage  revisions to, key guidance regarding federal agencies’ standards  efforts in OMB Circular A-119.

NIST led ICSP efforts to promote awareness on and share  information related to the development of corporate social  responsibility standards in ISO.

NIST created an ICSP working group on conformity assessment  to help address issues that were frequently being raised during  ICSP meetings.

NIST invites speakers to share information with ICSP members on  various standards-related issues. For example, NIST invited  officials from ANSI to present information on standards areas of  agency interest and also invited the members of the SDO leading  efforts on smart grid standards to brief agencies on the SDO’s  efforts.

According to six agency standards executives we interviewed, the ICSP  helps members to share information, including best practices and to have  a general awareness of pertinent documentary standards topics. Three  agency standards executives said that the ICSP helps standards  executives to know each other on a personal basis so that they know  whom to contact to coordinate on standards activities. Bringing agency  standards executives together through the ICSP can also spur  coordination among agencies if a topic of mutual interest is identified. For  example, two agency standards executives said that questions and  suggestions raised by agencies at the ICSP led to coordination with other  agencies on a standards area of mutual interest.

NIST also coordinates with individual agencies on documentary  standards activities outside of the ICSP. For example, according to  participants in four focus groups, relevant NIST and VCAT reports, and  congressional testimonies we reviewed, NIST is particularly strong in  bringing relevant federal agencies and other stakeholders together to  develop standards and frameworks for individual interdisciplinary  technology areas. Such coordination can occur through a variety of  methods or groups, such as workshops, that address standards-related  issues. Additionally, four agencies’ standards executives said that they  coordinated extensively with NIST on specific standards activities. Two of  the agency standards executives described how collaboration with NIST  on research helped inform their agencies’ standards activities. NIST also  offers training, such as NIST’s standards boot camp, according to federal  standards executives. Five agency standards executives said staff from  their agencies attended the training and four standards executives said  the training had improved their staff’s competence in standards.

While coordination between NIST and other federal agencies on  documentary standards issues occurs in a variety of ways, the ICSP is  the primary body established to facilitate interagency coordination on  standards policy, according to NIST standards officials. However, some of  the ICSP’s efforts to support coordination among federal agencies on  standards issues do not fully align with selected leading practices for  interagency collaboration we identified in our previous work. Specifically,  the ICSP charter has not been updated; ICSP member agencies’ roles  and responsibilities have not been fully clarified; and the ICSP may not  include relevant members to carry out its functions. Additionally, we  reported in 2015 that while collaborative mechanisms differ in complexity  and scope, following leading practices can help manage fragmentation  and other coordination challenges.

ICSP charter: The ICSP charter has not been updated since it was  signed in October 2000. According to the charter, the need for and  mission of the ICSP was to be reexamined 3 years after the charter was  created. However, NIST officials said that the ICSP charter has not been  reexamined. We reported in 2012 that agencies that articulate their  agreements in formal documents can strengthen their commitment to  working collaboratively. We also reported that written agreements are  most effective when they are regularly updated and monitored. Further,  updating written agreements, such as the ICSP charter, can be an  opportunity for members to define common goals and purpose. In  addition, focus group participants, stakeholders we interviewed, and  literature we reviewed described broad changes in documentary  standards that have led to new challenges. For example, we previously  noted the challenges related to emerging interdisciplinary standards  issues and the increase in the number of international standards  activities, and both of these areas can benefit from federal coordination.

Further, the ICSP charter has not been updated to reflect the 2016  revisions to OMB Circular A-119 or the guidelines provided to agencies in  the 2012 EOP memo on engagement in standards activities to address  national priorities. For example, the revised A-119 notes several  executive orders relating to review and coordination that were not in  existence at the time of the charter’s creation. Additionally, the EOP  memo that outlines agency responsibilities for standards areas of national  priority was also not in existence at the time of the ICSP charter’s  creation. The EOP memo calls on agencies to ensure effective intra and  interagency coordination of engagement in standards development  activities. Without reexamining and updating the ICSP charter, as  necessary, NIST and other ICSP member agencies cannot be assured  that their collaborative efforts are best structured to address current  standards challenges.

ICSP member roles and responsibilities: ICSP member agencies’ roles  and responsibilities have not been fully clarified to an extent that would  help the ICSP fulfil its purpose, objectives, and functions to gather  information and make recommendations to the Secretary of Commerce to  strengthen standards policy and coordination. The ICSP charter outlines  two basic functions for the committee: (1) gathering, analyzing, and  maintaining current information about standards and other specified  related information and (2) on the basis of such information, and when  appropriate, making recommendations to the Secretary of Commerce to  achieve various standards-related objectives, such as strengthening  coordination of standards-related policies and activities among federal  agencies. The charter also specifies that the ICSP may create task  groups as appropriate. However, we found several areas in which the  ICSP charter could more fully clarify member agencies’ roles and  responsibilities in regard to implementing its functions, purpose, and  objectives, for example:

The ICSP charter does not fully clarify the ICSP role and member  agencies’ responsibilities for identifying and coordinating on  interdisciplinary standards issues that cross agency boundaries. Five  agency standards executives we interviewed as well as NIST  standards officials described the ICSP as primarily an information- sharing or networking body, with little role in establishing federal  positions on standards activities or policy, or making joint policy  decisions with respect to specific standards issues. While information  sharing is an important component of interagency coordination, OMB  Circular A-119 gives agency standards executives responsibility for  consulting with other relevant agencies on standards issues to avoid,  to the extent practicable, expressing inconsistent views on standards  issues. Furthermore, the 2012 EOP memo specifies that agencies  should periodically review their standards activities to identify gaps in  representation for mission-critical areas and should ensure effective  coordination of engagement in standards development activities. NIST  officials and one standards executive said that the ICSP could identify  emerging or interdisciplinary standards issues that may require more  active federal roles and coordination; however, the charter does not  specify the ICSP’s role and member responsibilities regarding  interdisciplinary standards areas that may cut across agencies.

The ICSP charter does not fully identify member agencies’  responsibilities for coordinating on international standards issues.  OMB Circular A-119 and the ICSP charter specify that the ICSP has a  role in coordinating federal agencies’ international standards activities.  However, the ICSP charter does not fully identify member agencies’  responsibilities toward fulfilling this role. NIST standards officials and  one agency standards executive said that the ICSP does not typically  address international standards issues, or policy, or coordinate federal  positions on international standards. While NIST officials said other  USTR-led efforts could help coordinate agencies’ international  standards activities, a USTR official told us that USTR does not have  the technical expertise needed to effectively coordinate multiple  agencies’ views on standards. Further, the USTR official, one NIST  standards official, and representatives from ANSI said that different  agencies’ positions are taken into account as part of the standards  development process at SDOs that have an open process, such as  ANSI. However, as mentioned previously, A-119 specifies that the  ICSP is to coordinate with other interagency entities with a view to  encouraging more strategic and coordinated federal participation in  the development and use of standards. Further, the ICSP charter also  specifies that the ICSP is to ensure effective federal participation in  international standards activities.

The ICSP charter does not fully identify member agencies’  responsibilities for developing joint recommendations to the Secretary  of Commerce. The charter describes eight areas in which the ICSP  shall make recommendations when appropriate, including to  strengthen agency coordination and to improve the efficiency of  standardization efforts within the federal government. Further, the  charter specifies an administrative process for voting on ICSP  recommendations. However, according to NIST officials, the ICSP has  never made a recommendation to the Secretary of Commerce to  address a standards-related issue. Further, NIST officials said that the  ICSP agencies have not shown interest in acting jointly. NIST officials  said that there may be circumstances in which making such a  recommendation would be appropriate, for example, to strengthen  interagency coordination on interdisciplinary standards issues,  although the officials said that the ICSP would try to address issues at  a lower level before elevating them to the Secretary of Commerce.  Further clarifying agencies’ responsibilities may help ensure that the  ICSP is able to meet this function of its charter.

The ICSP charter does not fully identify member agencies’ roles and  responsibilities for creating and participating in ICSP’s task groups.  We have previously reported that task groups can be an effective  mechanism for agencies to collaborate on joint challenges. NIST  standards officials said that one task group has been created and that  task groups may be appropriate either (1) when a standards issue  may require more focused and sustained monitoring to understand  possible effects on U.S. government activities and missions or (2)  when an ICSP member suggests the need for a task group and there  is a consensus among membership. However, these reasons are not  specified in the charter or other documents available on the ICSP  website. Further, while NIST officials said that no member has  requested the creation of a task group, two standards executives  identified standards coordination issues that they thought may benefit  from the creation of an ICSP task group. The charter specifies an  administrative process for voting to create an ICSP task group but  does not specify what the role of task groups are, why they would be  created, or ICSP member agencies’ responsibilities in determining the  need for and participating in task groups.

We have previously reported that to achieve a common outcome,  participating agencies should consider clarifying roles and  responsibilities. By agreeing on and clearly defining the roles and  responsibilities of the members as well as documenting those decisions,  collaborating agencies can clarify which agency will do what, organize  their joint and individual efforts, and facilitate decision-making. Without  ensuring that member agencies’ roles and responsibilities have been fully  clarified, NIST and the ICSP may miss opportunities to strengthen  agencies’ coordination on standards issues, and better ensure effective  coordination related to standards activities. Further, without fully clarifying  federal agencies’ roles and responsibilities, the ICSP may also miss  opportunities to address standards challenges noted above, which limit its  ability to support U.S. competitiveness.

ICSP membership: The ICSP may not include all relevant agencies as  members or invited observers. The ICSP is comprised of certain specified  agencies that are represented by their standards executives as described  in OMB Circular A-119. A-119 provides that federal standards  executives should be broadly engaged in the agency’s standards-related  activities so as to ensure intra-agency coordination and have sufficient  authority to ensure compliance with Circular A-119. In addition, the ICSP  charter allows the Secretary of Commerce to invite additional members— a role which has been delegated to NIST as chair of the ICSP. Consistent  with leading collaboration practices, it is important to ensure that all  relevant agencies are included in a collaborative effort. In addition,  participants should also have full knowledge of relevant resources in their  agency, and the skills and abilities to commit relevant resources and  contribute to the outcomes of the collaborative effort, among other  attributes.

While the chair of the ICSP said that agency standards executives are  engaged in understanding their standards activities and that participation  in the ICSP is strong, as we noted above, it can be difficult for standards  executives to be fully aware of all standards activities in their department  or agency, particularly in large agencies. In addition, an agency standards  executive, a NIST stakeholder, and NIST standards officials raised  concerns about whether standards executives have sufficient time to  understand all their agencies’ activities and needs, given their other  duties, or whether standards executives have the authority to fully  coordinate on standards activities. In some cases, agency officials other  than the standards executive may have greater knowledge and expertise  about specific standards issues. For example, sub-component offices and  agencies may have numerous standards-related activities, such as the  Food and Drug Administration within Health and Human Services.  According to interviews with two agency standards executives, some sub- component agencies may also have more knowledge and expertise in  significant standards areas, and some of these areas can affect multiple  agencies. According to one NIST standards official, NIST invites  additional agencies when it learns that an agency could potentially add  value but has not conducted a comprehensive assessment of ICSP  membership within the last 5 years. By assessing whether relevant  agencies and offices have been invited to participate as members or  observers, the ICSP would have greater assurance of its ability to ensure  effective participation by the federal government in domestic- and  international-standards activities. Further, having relevant parties involved  at the ICSP could enhance the ICSP’s efforts by ensuring the viewpoints  of all relevant agencies are considered.

In NIST’s role as the nation’s measurement science laboratory, NIST  works to improve how we design, build, and test the technologies around  us. Further, NIST’s measurement services and support of documentary  standards development can directly affect innovation and the nation’s  economy by helping companies produce better products and compete in  the global economy. However, factors such as the breadth of industry  needs, number of domestic- and international-standards development  activities, and the fragmented nature of federal agencies’ involvement in  documentary standards development create challenges to NIST’s ability  to fulfill its mission of promoting U.S. innovation and industrial  competitiveness.

NIST has taken steps to address these challenges. For example, NIST’s  expert scientists and engineers maintain close contact with industry  through a variety of mechanisms, and use their expertise to help identify  industry needs and to communicate about NIST’s services. NIST has also  established procedures to help the agency prioritize and evaluate the  effectiveness of its measurement services and ensure that supervisors  and laboratory management oversee agency staff participation in  documentary standards activities. However, NIST has not  comprehensively evaluated the extent to which its efforts align with  stakeholder needs. Conducting comprehensive reviews of its activities  would provide NIST with greater confidence that its activities align with  stakeholder needs and may help identify areas not currently served by  NIST.

Similarly, NIST could gain confidence in the effectiveness of its  participation in documentary-standards development activities: by completing the comprehensive review called for in NIST policy, by improving its guidance to staff, and by taking steps to enhance agency management’s understanding of  the time devoted to NIST’s current efforts.

NIST staff’s expertise and lack of bias make them valuable contributors in  documentary-standards development activities, but these individuals have  limited time for such activities. Without additional guidance regarding the  factors staff could consider when deciding to take more active roles in  standards development activities, NIST cannot be assured that decisions  on when to participate in such activities are made consistently. Further,  while determining the exact amount of time spent on any one standards  activity may not be possible due to overlap with other employee duties,  examining to the extent possible the aggregate amount of time NIST staff  expect to commit to standards development activities could help NIST  management assess the agency’s participation in these activities.

Further, NIST has a role, delegated by the Secretary of Commerce, to  ensure U.S. interests are adequately represented at private international  standards organizations but does not currently have a mechanism to use  the process under the Trade Agreements Act of 1979 to identify or  respond to circumstances when U.S. representation in international  standards activities may be inadequate. Developing a mechanism could  help ensure that NIST does not miss opportunities to ensure that the  United States is adequately represented in international standards  activities. Alternatively, NIST could develop a legislative proposal that  allows NIST to ensure adequate U.S. participation in international  standards activities while addressing any concerns.

Finally, as chair of the ICSP, NIST provides leadership to enhance  interagency coordination on documentary standards issues. However,  some of the ICSP’s efforts do not fully align with selected leading  practices for enhancing and sustaining interagency collaboration, as  identified in our previous work. Specifically, the ICSP charter has not  been updated since 2000 and certain roles and responsibilities for the  ICSP and its members are unclear. Further, the ICSP membership may  not include relevant members or observers to ensure effective  participation by the federal government in domestic- and international- standards activities. Updating the ICSP charter to affirm its mission and to  delineate appropriate roles and responsibilities of participating agencies  could strengthen interagency coordination through the ICSP on standards  development issues and help the ICSP fulfill its role as envisioned under  OMB guidance. Moreover, assessing whether relevant agencies or sub- component offices are invited to participate in the ICSP could provide the  ICSP with better assurance of its ability to effectively coordinate agencies’  standards activities.

We are making a total of seven recommendations to NIST, specifically:  The NIST Associate Director for Laboratory Programs should: update NIST policy to include periodic comprehensive management  review of the agency’s measurement services to assess gaps and  ensure alignment with stakeholders’ needs, and take steps to ensure  that the Associate Director completes the review of NIST’s standards  development activities (Recommendation 1).

The NIST Standards Coordination Office Director should: update NIST policy for staff participation in standards development  activities to provide additional guidance, such as the factors staff  could consider when deciding to take more active roles, including  leading efforts to develop standards (Recommendation 2); and assess the feasibility of collecting aggregate data on the estimated  amount of time staff spend on documentary standards activities  (Recommendation 3).

The Director of NIST should establish a mechanism—such as guidelines  for what constitutes adequate U.S. representation—to assess whether  U.S. representation in international SDOs is adequate, and when to follow  the statutory process for addressing inadequate U.S. representation. If  NIST determines that it is unable to implement the process described in  the 1979 act without conflicting with current standards policy, the Director  of NIST should develop a legislative proposal to address those concerns  (Recommendation 4).

The NIST Standards Coordination Office Director, working with other  ICSP member agencies, should:  review and, as necessary, update the ICSP charter (Recommendation  5);  clearly define ICSP roles and member agencies’ responsibilities, such  as for identifying and coordinating on interdisciplinary documentary- standards issues and for making recommendations, as appropriate, to  the Secretary of Commerce (Recommendation 6); and, assess whether additional agencies or sub-component offices should  be invited to participate as ICSP members or observers  (Recommendation 7).

We provided a draft of this report to the Departments of Commerce,  Defense, Energy, and Homeland Security, and Health and Human  Services, as well as the Consumer Product Safety Commission,  Environmental Protection Agency, General Services Administration, and  USTR for their reviews and comments. Commerce provided written  comments, which are reproduced in appendix III, in which it generally  concurred with six recommendations and disagreed with one. USTR  provided technical comments, which we incorporated as appropriate. The  remaining seven agencies informed us that they had no comments.

Commerce agreed with our recommendation regarding comprehensive  reviews of the agency’s measurement services and standards  participation, and stated that it will include requiring management review  of its measurement services in a future agency order and will have the  Standards Coordination Office support the Associate Director’s review of  staff participation in standards activities. Commerce also agreed with our  recommendations regarding guidance on staff participation in standards  activities and assessing the feasibility of collecting data on the time spent  on documentary standards activities. Specifically, Commerce said that it  will consider updates to guidance to staff and will report on the feasibility  of collecting data on the time spent on standards activities.

Commerce also agreed with recommendations on improving interagency  coordination in the ICSP. It stated that the chair of the ICSP will review  the charter and recommend any updates to the committee and will work  with existing ICSP members, alternates, and observers to identify other  agencies or sub-component offices that may be invited to participate.  While Commerce agreed with our recommendation on roles and  responsibilities of the ICSP and its members, it stated that the roles and  responsibilities of standards executives are effectively stated in OMB A- 119. A-119 describes the roles and responsibilities of standards  executives in general, such as their responsibilities to promote effective  use of agency resources and participation in standards bodies. However,  A-119 does not address specific roles and responsibilities with respect to  the activities of the ICSP, such as defining the role of the ICSP in  establishing federal positions on standards issues. In our 2012 report on  interagency collaborative mechanisms, we state that agencies working  together to define and agree on their respective roles and responsibilities  can help clarify who will do what and identify how to organize individual  and joint efforts. We believe that additional efforts by ICSP members to  clarify their roles and responsibilities within the framework of the ICSP will  improve the effectiveness of the ICSP as a coordinating body.

Commerce disagreed with the recommendation on ensuring adequate  U.S. representation in international SDOs, stating that any determination  to follow statutory process would itself carry significant risk of being  perceived by the national and international standards community as a  U.S. government change of policy relating to our private-sector-led  standards system. However, we do not perceive a conflict between the  private sector leading U.S. standards development, and NIST developing  a mechanism to respond to any instances where U.S. representation in  international standards efforts are inadequate. NIST has already taken  steps, in some cases, to identify or encourage private sector participation  in international standards efforts that lacked U.S. participation. Further,  the 1979 act does not define what constitutes adequate representation;  NIST can develop criteria to allow the agency to take action when  appropriate.

In carrying out the 1979 act, the agency could continue to encourage  private-sector stakeholders to address any areas determined to be  inadequate, allowing NIST to step in only as necessary when the relevant  private-sector entities are not adequately representing, and are not willing  and able to adequately represent, U.S. interests. We continue to believe  that a mechanism to identify and respond to circumstances when U.S.  representation at international SDOs may be inadequate would allow  NIST to consistently take action in furtherance of its mission to support  U.S. competitiveness. However, in consideration of the concerns raised  by Commerce, we have clarified our recommendation that NIST either  develop a mechanism to carry out the process described by the 1979 act,  or develop a legislative proposal to address any concerns arising from the  implementation of the act.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Commerce, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-3841 or neumannj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This report examines (1) the challenges the National Institute of  Standards and Technology (NIST) faces in providing measurement  services and supporting documentary-standards development and (2) the  extent to which NIST has taken steps to address any challenges and how  those steps align with relevant federal guidance and policy.

To identify any challenges NIST may face in providing measurement  services and supporting documentary-standards development, we began  by performing a literature review, including reports on NIST from the  National Academies of Sciences, Engineering, and Medicine, NIST’s  Visiting Committee on Advanced Technology (VCAT), congressional  committee hearings on NIST’s measurement services, and other  sources. We reviewed these sources to identify statements regarding  any challenges to NIST’s current measurement services and  documentary standards activities or statements recommending  improvements to these services or activities. For purposes of our  analysis, we included NIST’s efforts to support measurement standards  as a component of NIST’s measurement services.

We supplemented our literature review by holding focus groups with NIST  stakeholders including: (1) researchers and (2) representatives working  with industry, including commercial entities and states’ metrology  laboratories. To ensure our focus groups contained a diverse group of  stakeholders and viewpoints on NIST’s measurement services and  documentary standards activities, we included participants from a variety  of backgrounds. We selected researchers to participate in our focus  groups from university scientists in engineering and the physical and  biological sciences. We selected industry participants for the focus groups  to reflect a range of industrial sectors, including (1) sectors that comprise  greater than 1 percent of U.S. gross domestic product, according to  Bureau of Economic Analysis data and (2) sectors that the Department of  Commerce’s International Trade Agency has identified as U.S. export  opportunities. Across these sectors we selected industry participants  from the following categories: broadcasting and telecommunications;  chemical products and pharmaceuticals; computer and electronic  products and related services; building products and construction; finance  and insurance; food and beverage and tobacco products; health care and  social assistance; transportation; utilities and energy; defense products;  and other manufacturing. We also included representatives from the  National Conference on Weights and Measures, an organization of  commercial entities and state metrology laboratories, among others, that  addresses measurement of commercial products.

After developing our focus group structure and determining our participant  categories, we obtained feedback on our approach during discussions  with NIST officials and with representatives from standards development  organizations (SDO) selected from those that NIST most often  collaborates with—the American National Standards Institute, ASTM  International, and IEEE. We then used a snowball approach to identify  and invite individuals from across our participant categories. Starting with  individuals from several SDOs and the VCAT, we asked for suggestions  of individuals knowledgeable in measurement services and standards  needs of our participant categories. As we received responses and  contacted those individuals, we asked them to recommend additional  participants. Through this process, we identified and invited nearly 100  individuals to participate in our focus groups, and 58 individuals agreed to  participate. To organize our focus groups we asked the individuals who  agreed to participate to describe their expertise regarding measurement  services and standards development and familiarity with the industrial  sectors we identified. We then selected individuals for each focus group  based on availability and to include a mix of expertise. We conducted 3  focus groups for representatives from industry and 2 focus groups for  researchers. Each focus group included from 5 to 8 individuals. In total,  our focus groups included 31 stakeholders. We reviewed transcripts of  the focus groups to identify the challenges NIST faces in providing  measurement services and supporting documentary-standards  development.

We also collected information on the challenges that NIST faces during  36 interviews, including 17 interviews with current and former NIST  officials, 10 interviews with officials from other federal agencies, and 9  interviews with representatives from SDOs and other stakeholders. The  10 interviews we conducted with other federal agency officials included 8  agency standards executives—senior level officials with knowledge of,  and experience in, standards-related issues at their agencies and who  are responsible for coordinating their agency’s participation in SDOs,  among other responsibilities—or their alternates on the ICSP. The  agencies whose standards executives or other officials we interviewed  included the Departments of Defense, Energy, and Homeland Security,  and Health and Human Services, as well as the Consumer Product  Safety Commission, Environmental Protection Agency, General Services  Administration, and Office of the U.S. Trade Representative. GAO  reviewed our interview notes to identify challenges and NIST’s efforts to  address these challenges.

To evaluate the steps NIST has taken to address challenges in providing  measurement services and supporting documentary-standards  development, we drew upon our focus groups, interviews with NIST staff,  and reviews of NIST documentation that described the agency’s  measurement services and standards activities, such as agency policies,  orders, and publications. We also conducted a review of existing  literature, relevant laws, NIST policy, and other guidance documents to  identify federal requirements and guidance. For example, we reviewed  the National Technology Transfer and Advancement Act of 1995, Office  of Management and Budget’s Circular A-119, and Executive Office of the  President’s Memo on Principles for Federal Engagement in Standards  Activities to Address National Priorities, among other sources. We  compared the steps NIST has taken to address the challenges it faces in  providing measurement services and supporting standards development  to these policies and guidance.

To evaluate NIST’s current steps, we also considered our prior work on  federal standards for internal control and on interagency collaboration.  Internal control is a process created by an agency’s management and  other personnel that provides reasonable assurance that the objectives of  the agency will be achieved and comprises the plans, methods, policies,  and procedures used to fulfill the mission and objectives of the agency.  Standards for Internal Control in the Federal Government (known as the  Green Book), provide the overall framework for establishing and  maintaining an effective internal control system and require that agencies  perform and document certain actions to establish an effective internal  control system. These requirements include:  that management should identify, analyze, and respond to risks  related to achieving the defined objectives;  that management should use quality information to achieve the  entity’s objectives; and  that management documents the results of evaluations to identify  internal control issues.

Our work on interagency collaboration describes leading practices  agencies can engage in to enhance and sustain collaborative efforts and  describes seven key features to consider to implement these practices.  We selected the following five features relevant to NIST’s leadership of  the Interagency Committee on Standards Policy (ICSP) for review:

Outcomes and Accountability: Have short-term and long-term  outcomes been clearly defined? Is there a way to track and monitor  their progress?

Leadership: How will leadership be sustained over the long-term? If  leadership is shared, have roles and responsibilities been clearly  identified and agreed upon?

Clarity of Roles and Responsibilities: Have participating agencies  clarified roles and responsibilities?

Participants: Have all relevant participants been included? Do they  have the ability to commit resources for their agency?

Written Guidance and Agreements: If appropriate, have participating  agencies documented their agreement regarding how they will be  collaborating? Have they developed ways to continually update and  monitor these agreements?

We did not review ICSP collaboration with respect to key features  regarding resources or bridging organizational culture because we did not  fully examine the activities of all agencies participating in the ICSP.

We conducted this performance audit from July 2016 to July 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The following individuals participated in GAO’s five focus groups:  Allen Adler, Former Vice President of Enterprise Technology Strategy,  Boeing  Kathleen Almand, Vice President for Research, Data, and Analytics,  National Fire Protection Association  Karin Athanas, Government and Regulatory Affairs Manager, American  Association for Laboratory Accreditation (A2LA)

Robert Austin, Professor of Physics, Princeton University  Karl Bly, Quality Assurance Director, Vermont Thread Gage  Jerry Buendel, Weights and Measures Program Manager, Washington  State Department of Agriculture  Rita Colwell, Professor, University of Maryland, and Johns Hopkins  University School of Public Health  Ross Corotis, Professor of Engineering, University of Colorado Boulder  Denyette DePierro, Vice President and Senior Counsel, Center for  Payments and Cybersecurity, American Bankers Association  Don Detmer, Professor of Public Health Sciences, University of Virginia  Gail Folena-Wasserman, Senior Vice President, Biopharmaceutical  Development, MedImmune  Ruben G. Carbonell, Professor of Chemical Engineering, North Carolina  State University and Chief Technology Officer, National Institute for  Innovation in Manufacturing Biopharmaceuticals (NIIMBL)

Christopher Guay, Regulatory Fellow, Procter and Gamble  Kelvin H. Lee, Professor of Chemical and Biomolecular Engineering,  University of Delaware and Director, National Institute for Innovation in  Manufacturing Biopharmaceuticals (NIIMBL)

Hani Haider, Director of Orthopaedics Biomechanics & Advanced  Surgical Technologies Laboratory, University of Nebraska  Jennie Hwang, CEO and Principal, H-Technologies Group  Walter Jager, Principal, Environmentally Conscious Design (ECD)  Compliance  Karen Kafadar, Professor and Chair of Department of Statistics,  University of Virginia  Dave Kreitlow, Operations Manager, MTS Systems Corporation  Zhiyong Ma, Vice President and Director of Technology and  Manufacturing Labs, Intel  Kristin Macey, Director of Division of Measurement Standards, California  Department of Food and Agriculture  Josh Magri, Vice President and Counsel for Regulation and Developing  Technologies, Financial Services Roundtable  Dave Maisch, Director of Engineering and Industrial Affairs, PMC Lone  Star  Tod Sizer, Vice President of Mobile Radio Research Laboratory, Nokia  Bell Labs  Clifford Spiegelman, Distinguished Professor of Statistics, Texas A&M  University  Lonnie Spires, President and CEO, American Association for Laboratory  Accreditation (A2LA)

Appendix III: Comments from the Department of Commerce

Appendix IV: GAO Contact and Staff Acknowledgements

In addition to the contact named above, Chris Murray (Assistant Director),  Tind Shepper Ryen (Analyst-in-Charge), John Delicath, Justin Fisher, Eli  Harpst, Tricia Moye, Danny Royer, Andrew Stavisky, and Sarah Veale  made key contributions to this report.

The National Institute of Standards and Technology helps promote U.S. innovation and industrial competitiveness by providing measurement services and supporting the development of standards in collaboration with the private sector. For example, it provides calibrations for equipment used in high-precision manufacturing and it helps industry develop standards for fire safety, among other things.

We found NIST has taken steps to address challenges it faces in prioritizing and coordinating its work, but that it could better align its work with its policies and best practices. We made several recommendations on how to address these challenges.