In 2018, at the request of the Director of the National Institute of Standards and Technology (NIST), the National Academies of Sciences, Engineering, and Medicine formed the Panel on Review of the Information Technology Laboratory of the National Institute of Standards and Technology (the “panel”) and established the following statement of task:
The National Academies shall appoint a panel to assess independently the scientific and technical work performed by four of the divisions of the National Institute of Standards and Technology (NIST) Information Technology Laboratory. This panel will review technical reports and technical program descriptions prepared by NIST staff and will visit the facilities of the Information Technology Laboratory. The visit will include technical presentations by NIST staff, demonstrations of NIST projects, tours of NIST facilities, and discussions with NIST staff. The panel will deliberate findings in closed sessions of the panel meeting and will prepare a report summarizing its assessment findings.
NIST specified that the following four divisions of the Information Technology Laboratory (ITL) would be reviewed: the Applied and Computational Mathematics Division (ACMD), the Advanced Network Technologies Division (ANTD), the Computer Security Division (CSD), and the Applied Cybersecurity Division (ACD). The panel was not asked to review the three other divisions of the ITL: the Information Access Division (IAD), the Software and Systems Division (SSD), and the Statistical Engineering Division (SED).
The NIST Director requested that the panel focus its assessment on the following factors: the technical quality of the work; the scientific expertise of the staff; the adequacy of facilities, equipment, and human resources; and the effectiveness with which outputs of the work are disseminated.
All four of the divisions reviewed are located in Gaithersburg, Maryland, and nearby Rockville, Maryland (at the National Cybersecurity Center of Excellence [NCCoE]), and were visited by the panel on June 12-14, 2018.
The divisions described their purposes as follows: perform research in the mathematical sciences to nurture trust in NIST metrology and scientific computing (ACMD); establish the technical basis for trustworthy networking via standards, measurement science, test methods, reference implementations, and guidance (ANTD); cultivate information technology’s roots of trust (CSD); and improve the management of cybersecurity and privacy risk (ACD).
The four ITL divisions reviewed by the panel work in areas of major national importance and vital relevance to the security and stability of computing systems and networks. The activities of the four divisions provide good coverage of both technology and theoretical/mathematical infrastructure, and they complement one another well. The formation of the NCCoE, a federally funded research and development center (FFRDC), has added substantially to the vitality of ITL by strengthening and deepening its interactions with the broader cybersecurity communities in both the private and the public sectors.
Risk in the security of computing systems and networks is expected to worsen rather than improve in the foreseeable future. ITL needs to continue to invest and deepen its involvement in these areas. It is important for ITL to establish an increased level of preparedness for responding to emergencies and to develop the necessary infrastructure and processes.
ITL has a major role to play internationally and has the credibility to do so. Stronger emphasis on the international role of ITL as convener, facilitator, and promulgator of standards is essential. A more prominent and active international role will be of significant benefit to the country’s industrial and business sectors.
The detailed chapters on the four ITL divisions that form the core of this report provide overviews and appraisals and point to specific opportunities and challenges in each area. Three topics were highlighted in the findings of more than one division: staffing and recruitment, technical planning, and conferences and publications.
Staffing and Recruitment
The long-term vitality of ITL is closely intertwined with its ability to continuously renew and deepen its pool of talent. This requires a strategic approach and coordinated efforts across ITL subdivisions. Closer collaboration with the country’s universities can be helpful in this regard. In most cases, staffing is currently adequate to perform the assigned work. There are current and projected exceptions.
The ACMD is experiencing staffing stresses that may have an impact on its ability to meet its goal of providing comprehensive mathematical expertise for NIST. There is more demand for such expertise than can be met by the current ACMD staffing, and there is also an anticipation of substantial turnover due to the potential retirement of a significant fraction of staff in the near future.
RECOMMENDATION: The ACMD should evaluate its organizational and recruiting practices in order to better meet the challenges it faces. Ideas that should be considered include the use of contractors to broaden the pool of potential participants in the ACMD mission; the use of sabbatical opportunities for career staff to broaden the range of skills in response to new areas for ACMD; and development of a more effective pipeline for graduate students into ACMD through, for example, a broad-based university affiliates program. (Chapter 6)
There is need to increase the core full-time ANTD staff to address new areas of research such as the Internet of Things (IoT), machine learning, and 5G wireless, and to expand existing areas of activity, such as formal verification and model checking.
RECOMMENDATION: The ANTD should build up and grow expertise in new and emerging areas such as the Internet of Things, machine learning, and 5G wireless. (Chapter 6)
CSD’s Lightweight Cryptography project promises good potential application if it receives greater visibility and resources.
RECOMMENDATION: The CSD should consider adding staff to the Lightweight Cryptography project. (Chapter 6)
Another project whose impact could be amplified by additional resourcing and community outreach is the CSD’s Combinatorial Methods in Software Testing project. The project currently has only two staff members.
RECOMMENDATION: The CSD should consider adding staff to the Combinatorial Methods in Software Testing project to accelerate adoption of the project’s tools and techniques by the software development community. (Chapter 6)
The CSD’s Vulnerability Metrics project has a critical short-term need for supplemental staff to address the increased volume and backlog of submissions for Common Vulnerabilities and Exposures (CVE) scoring.
RECOMMENDATION: The CSD should devote additional short-term resources to Common Vulnerabilities and Exposures scoring until the backlog can be remediated. (Chapter 6)
In the projects that are strategic national cybersecurity resources, the CSD is performing functions of a national laboratory. However, the CSD does not have recruiting programs like the national laboratories for mid-career staff.
RECOMMENDATION: The CSD should emphasize recruiting of mid-career staff. (Chapter 6)
NIST is limited to hiring U.S. citizens as permanent staff, but it also maintains a foreign guest researcher program, the Professional Research Experience Program (PREP), which supports visiting scientists and students under NIST-sponsored J1 visas.
RECOMMENDATION: The ITL should expedite and grow the Professional Research Experience Program to hire more international graduate students from among those already at U.S. universities (e.g., as interns or cooperative researchers). (Chapter 6)
Recruiting, retention, and mentoring of women and minorities has been a major issue in science, engineering, technology, and mathematics programs in organizations generally. ITL managers have expressed agreement with the importance of recruiting and developing women and minorities.
RECOMMENDATION: The ITL should assess the effectiveness of its efforts to improve recruiting, retention, and mentoring of women and minorities. (Chapter 6)
The technical work at the ACMD is driven by collaborations between ACMD staff and scientists from other disciplines, largely from other units within NIST. This work is mostly chosen in a bottom-up fashion, with some informal guidance from the division leader, so that there is little overt strategic organization of the scientific work done.
Several of the ANTD projects had timelines and roadmaps, both short and long term. At the same time, these plans differ in their formats, making them hard to contrast with one another and evaluate thoroughly. A standard format template completed for each project could provide answers to a set of questions such as the following:
- What is the problem statement? (What is this project attempting to do?)
- Who is the ultimate customer? (Who will benefit if this project is successful?)
- Why should NIST use its resources to do this work? Why ANTD? Are adequate resources available?
- How does the proposed work build upon what already exists today in the external community?
- How will the results from the proposed work impact the external community?
- What are the measurable milestones that define the path toward success and completion?
- What is the execution plan? What resources will be used? What collaborations with other ITL/NIST organizations are needed to reach each milestone? What collaborations with industry or academia are needed to reach each milestone?
Conferences and Publications
All divisions reported that their staff members attend professional conferences and author peer-reviewed publications. Anecdotal, but not systematic, data on conference attendance and publications, including the number of attendees, presenters, authors, and collaborative studies and the quality of the conferences and journals, were not made available to the panel.
RECOMMENDATION: The ITL should perform a systematic assessment of the conferences at which its staff members have presented their research or otherwise attended. The ITL should consider whether attendance has been sufficiently frequent and whether the conferences are of sufficiently high quality, and it should maintain or increase, as appropriate, conference attendance. A similar assessment should be performed for publications in scholarly journals. (Chapter 6)
Applied and Computational Mathematics Division
As a research organization, the ACMD is very successful, considering several factors: executing high-quality research in applied and computational mathematics; meeting the needs of collaborators in diverse scientific disciplines; fulfilling its part of the institutional missions of NIST in metrology; and disseminating its work to broader communities. Especially noteworthy is its strength in mathematical analysis, particularly when used in tandem with simulation, which provides a high degree of scientific insight and is a distinctive strength of the ACMD. The ACMD successfully performs high-impact work in the areas of mathematics of metrology, high-performance computing and visualization, and materials modeling and simulation.
Notable accomplishments include the use of computer simulation in the development of a standard reference mortar to replace expensive oils in concrete rheometers; the design of standard reference artifacts for calibrating magnetic resonance imagers (MRIs); the deployment of community software for computing the physical properties of complex microstructures in solids from image data in
three dimensions; the development of an efficient method for generation of uniformly distributed random bitstrings from a quantum source, with applications in secure communications; and developing and maintaining the Digital Library of Mathematical Functions, a project to update the National Bureau of Standards’ renowned Handbook of Mathematical Functions.
The complexity of simulations of physical systems is rapidly increasing, and the processor architecture of computers used in simulation and modeling is becoming vastly more complex. There is a serious risk that the existing approach in ACMD of having a small number of people, or even a single staff member, implement complete simulation capabilities starting from scratch will no longer be feasible.
RECOMMENDATION: The ACMD should evaluate simulation software development practices in light of the disruptive changes in high-performance computing technology. (Chapter 2)
With the exception of staffing needs, ACMD resources appear adequate. The ACMD has an excellent group of career staff that requires expansion. A key facilities issue for the ACMD is access to evolving computing resources, although its current approach to sharing computing resources within NIST and extramurally appears to provide currently adequate access to computing capabilities.
ACMD staff members effectively perform a diverse set of activities in support of disseminating the outputs of their work. They publish extensively in high-visibility refereed journals and conference proceedings; distribute software, including micromagnetic modeling, tools for combinatorial testing of software, and the Digital Library for Mathematical Functions; and participate in standards setting for metrology, contributing to the publication of those standards in NIST reports.
Advanced Network Technologies Division
The ANTD has four major project areas: Network Resilience, Cloud Computing, Internet of Things, and Future Network Technologies. ANTD’s projects include testing methodology of indoor localization and tracking systems, wireless networks, wireless networking specifically for smart manufacturing, network resilience, robust interdomain routing, high-assurance domains, measurement for complex systems, a distributed algorithm for suppressing epidemic spread in networks, the NIST cloud computing program, software-defined networks and virtual networks, and information-centric networking.
Network localization and navigation (NLN) is an area in which NIST can play an important role in creating databases and through roadmapping exercises. ANTD’s work in indoor localization is commendable. The ANTD has created a methodology to allow apples-to-apples comparisons of different smartphone-based NLN applications.
The Wireless Networking for Smart Manufacturing project is examining reliable wireless networking for smart manufacturing and industrial IoT by investigating the radio frequency (RF) landscape, channel sounding and modeling, co-simulation, and scheduling. By identifying key technological components to form a framework and reference model, the ANTD could help industrial and academic developers to contribute technological innovations toward standards. This would help secure a leading role of the United States in smart manufacturing technologies.
In the Complex Systems project, the ANTD has undertaken an ambitious effort to understand some of the emergent behaviors that pervade across a swath of complex systems that include social media and networks.
The Software-Defined and Virtual Networks project provides an excellent example of a well-articulated and well-contextualized research vision. The uMon (User-defined traffic MONitoring) project adheres to ANTD’s goal of being a technology facilitator by creating a first step toward standardization.
While the ANTD has been active in pursuing projects in the information-centric networking (ICN) area, the future importance of such architectures is not assured. ANTD’s role could beneficially be
aligned with examining the fundamental networking problems that ICN is attempting to solve. The ANTD could also examine the efficacy of ICN as compared with traditional approaches. The division needs to leverage current efforts to explore commercially promising technologies such as mobile edge computing.
The Smart Grid is a key component of the nation’s energy strategic plan. NIST is one of the federal agencies with a statutory role in the Federal Smart Grid Task Force,1 led by the U.S. Department of Energy (DOE). Within the Smart Grid concept/architecture, the grid-to-end user interface is critical, and this is being addressed by ANTD staff. A roadmap for what the ANTD intends to accomplish in its Smart Grid project would be very helpful for evaluating the potential impact of this project.
ANTD’s future benchmarks and data sets in cloud computing and related emerging areas such as data analytics systems could facilitate developers’ and startup companies’ choosing from among numerous open-source systems and standards. NIST’s ability to convene stakeholders and assess pros and cons of various options makes this is an opportunity for real impact on both industry and academia.
RECOMMENDATION: The ITL should develop and publish benchmarks to be used for evaluating the performance of existing and proposed networks and network technologies in more areas and should develop simulators and make them available for researchers. (Chapter 3)
An important focus of research at NIST is measurement science for real systems. Today’s ecosystem of public clouds, Internet service providers (ISPs), and the Internet offer myriad opportunities for doing this.
RECOMMENDATION: The ITL should work with Internet service providers, public clouds, and data centers to collect data sets needed for NIST researchers to perform evaluations of the performance of existing networking solutions. If possible, the ITL should make those data sets available to industry and academia. (Chapter 3)
One of the ITL’s areas of expertise is the study of what problems need to be solved, rather than taking what is currently deployed and assuming that this was the right or only possible choice.
RECOMMENDATION: In its role as a technology facilitator, the ITL should study Internet problems and behaviors, outside the assumptions inherent in deployed standards. (Chapter 3)
Several ANTD projects focus on standardization and creation of International Organization for Standardization (ISO)2/NIST/Internet Engineering Task Force (IETF)3 standards. For example, the Cloud Computing initiative resulted in an ISO standard, and the Secure Border Gateway Protocol (BGPSec) resulted in an IETF standard. ANTD’s cloud computing effort involves work on catalyzing standards for Service Level Agreements (SLAs) into clouds, including federated clouds. NIST’s involvement in the development of the ISO 19086 standard, which has since been adopted by Microsoft, is a commendable first step toward expansion of the Cloud Computing project in the ANTD. ANTD staff members in the High Assurance Domains project are encouraging adoption of IETF standards within the government.
1 Further information is available at Department of Energy, “Federal Smart Grid Task Force,” https://www.energy.gov/oe/activities/technology-development/grid-modernization-and-smart-grid/federal-smartgrid-task-force.
Other ANTD projects focus on the creation of data sets. For example, the Indoor Localization project resulted in the PerFloc data sets.
Networking and network-based technologies such as cloud computing and data analytics systems are a fast-growing market. As such, increased investment in the ANTD would bolster existing expertise and enable ANTD growth into new areas as the need and opportunity arise.
Computer Security Division
CSD’s technology focus areas are cryptography, risk management, identity and access management, testing and validation, software security, vulnerability metrics and configurations, and emerging technologies.
The quality of work in CSD is uniformly excellent. Two CSD projects in particular are strategic national cybersecurity resources. The Cryptography project creates standards that are implemented by virtually every significant commercial encryption in a laptop computer, cell phone, or automated teller machine, and NIST’s cryptographic standards are widely adopted by industry groups. The National Vulnerability Database (NVD) and the associated Common Vulnerability Scoring System are widely used, not only in government but also by private-sector firms and vulnerability and risk assessment product vendors.
The Quantum-Resistant Cryptography (QRC) project is timely and important, and its importance is well understood not only in the cryptographic community but also among government and commercial customers of cryptography. But because no quantum computer capable of breaking deployed public-key cryptosystems is likely to exist for at least 20 years, the QRC’s impact will be felt on the time scale of decades. The Combinatorial Methods in Software Testing project is mature and has generated numerous highly cited publications. The tools and techniques developed by the project promise substantial impact on real-world software testing efficiency and effectiveness.
Other CSD projects could benefit from a clearer statement of the requirements that are driving them. The Access Control project has reached maturity and has had substantial academic and commercial impact, but it may have reached the point of diminishing return as an ITL activity.
RECOMMENDATION: The Access Control project’s resources should be directed toward more recently emergent risks in order to have higher impact. (Chapter 4)
The CSD has hired and retained appropriately expert staff in all of its project areas, but some projects could benefit from additional staff. Some emerging research areas will need to be staffed, and there are some issues relating to career progression and recruiting that could represent risks to the availability of necessary expertise in the medium term.
CSD’s Lightweight Cryptography project is much less well-known to its potential customers than its QRC Algorithms Standardization project and its NVD and associated Common Vulnerability Scoring System.
RECOMMENDATION: The CSD should take steps to publicize the Lightweight Cryptography program among potential users of the resulting algorithms—particularly Internet of Things vendors and customers. (Chapter 4)
Some emerging areas of research are currently being handled by existing CSD staff but will require dedicated experts as the areas mature. Additional staff expertise will shortly be required in the areas of multiparty computation, artificial intelligence (AI) and machine learning, high-performance computing security, and IoT security. The Pathways program has proven to be effective for recruiting scientific experts who eventually join CSD’s permanent staff.
CSD facilities and equipment are adequate, and the budget for CSD staff to attend conferences and host workshops is adequate.
The CSD disseminates its work via Federal Information Processing Standards (FIPS), guidance in the form of NIST Special Publications, tools and testing services, academic publications, workshops, and data references, including online products such as the National Vulnerability Database (NVD). The CSD has long been a prolific producer and effective disseminator of high-quality and frequently cited publications, broadly implemented standards, and influential guidance.
CSD’s impact is strong. Its guidelines and standards are widely adopted. However, evidence for the impact of many projects is anecdotal rather than systematic. Some projects have effective systematic impact metrics. Impact metrics would be very helpful in quantifying the effectiveness of the standards, guidance, and tools developed by the CSD.
RECOMMENDATION: Recognizing that impact is sometimes difficult to measure without deep insight into stakeholder products and processes, the ITL should work toward the development of impact metrics for projects in the CSD where development of such metrics is feasible. (Chapter 4)
While the CSD and the ACD have incorporated privacy recommendations into their respective Risk Management guidance documents, there are no metrics for privacy.
RECOMMENDATION: The CSD, in partnership with the ACD, should investigate and, if possible, develop and disseminate metrics for privacy. (Chapter 4)
Applied Cybersecurity Division
The ACD addresses its goal of improving the management of cybersecurity and privacy risk through outreach and application of standards and best practices whose adoption is deemed necessary to strengthen U.S. cybersecurity capabilities. Central to its approach is collaboration with industry, other federal agencies, state and local agencies, academia, international organizations, and others. The division consists of three groups: the National Initiative for Cybersecurity Education (NICE); the Cybersecurity and Privacy Applications Group; and the NCCoE.
The NCCoE was established in 2012 by NIST in partnership with the state of Maryland and Montgomery County, Maryland. The NCCoE is an FFRDC operated by the MITRE Corporation. It houses about 30 laboratories, where researchers define cybersecurity issues, develop technical descriptions of problems, and engage with technology vendors that have standards-based, commercially available products that can be used as part of an example implementation.
NCCoE’s Secure Interdomain Routing project is an effort to build a standards-based solution to a significant problem: spoofing routing information to hijack (reroute) packets on the Internet. The team has developed what appears to be a potentially cost-effective and thus practical solution for deployment, although a formal cost-benefit analysis has not yet been done.
The NCCoE initiated the Securing Wireless Infusion Pump project with the goal of applying the cybersecurity framework to devise a set of specific security measures that could enable health-care delivery organizations such as hospitals to use wireless infusion pumps for drug delivery without introducing undue risks. The project has delivered a valuable resource for direct application by hospitals and other health-care organizations. The publication Securing Wireless Infusion Pumps in Healthcare Delivery Organizations4 does provide some general recommendations for mitigating and responding to residual risks. However, there is no indication in the document that those recommendations resulted from
4 MITRE Corporation and NIST, 2017, Securing Wireless Infusion Pumps in Healthcare Delivery Organizations, Special Publication 1800-8, https://nccoe.nist.gov/projects/use-cases/medical-devices.
an adversarial analysis of the proposed solution. The infusion pump publication and other similar guidance documents produced by the NCCoE would benefit from such a review and also from a description of how these measures could be generalized to other venues.
It would be prudent for the ACD to consider how it would respond to a defeat scenario (a real-world safety or security problem after a technology has adhered to the best practices), technically and with communications to stakeholders, and to have a plan and assigned responsibilities ready in advance. Mature cybersecurity organizations create such response plans as a matter of course and find that they can mitigate many of the substantive and reputational consequences of dealing with such contingencies if they act quickly and consistently.
RECOMMENDATION: The ITL should consider putting together a rapid response plan of action to be invoked in the event of a real-world safety or security problem after a technology has adhered to the best practices and guidance from the NCCoE. To the extent that there is the potential for reputational damage to NIST as to the effectiveness of its best practices and guidance, the ACD should prepare in advance to proactively address issues that may arise. (Chapter 5)
RECOMMENDATION: The NCCoE should add an adversarial perspective to the solutions and guidance that are promulgated by the NCCoE laboratories. That would mean conducting an adversarial review (e.g., red-teaming) against these solutions and feeding the adversarial review results back into their process for purposes of defensive improvement. This may involve adding steps into the current NCCoE process before reference designs and documents are released from the laboratory; additional resources should be added if needed to accomplish including the additional steps. (Chapter 5)
RECOMMENDATION: The NCCoE should examine the university affiliates program with the federally funded research and development center contractor and consider how that program could be modified to enhance engagement with the existing university affiliates and how it could be improved to broaden participation with additional universities. (Chapter 5)
The creation, enhancement, and sustainment of the NIST Cybersecurity Framework is one of the key contributions of the ACD and of the NIST cybersecurity program. The ACD recently updated the framework in response to Executive Order 13800,5 and NIST has supported the framework by creating samples of framework profiles. The profiles are a critical resource for organizations that seek to adopt the framework. ACD is showing excellent commitment to sustaining the framework and enabling its adoption.
The National Initiative for Cybersecurity Education (NICE) framework provides classification of practitioner duties in both broad categories and specific professional roles. It has been generally accepted within the field and is being used to map certifications’ common bodies of knowledge. The NICE group is also seeking to encourage collaboration and development of enhanced educational and training materials. The NICE initiative is of high quality and high impact. This effort is being recognized nationally and internationally for filling a significant need and doing so in a detailed fashion. There is strong interaction with multiple communities—education, government, and private sector—and the work appears to be well accepted.
The ACD Privacy Engineering Program helps technology managers navigate privacy engineering concerns via guidance such as NISTIR 8062, integrates the new privacy sensibilities into existing NIST Special Publications, and participates in collaborations, workshops, and standards bodies. ACD has
5 The White House, 2017, “Executive Order on Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure,” May 11.
assembled talented staff with diverse perspectives that are carrying out this function with clarity and efficiency.
The NIST Identity and Access Management project has released a new family of guidelines (NIST Special Publications 800-63-3 and 800-63A-C) that focus on digital identity. This family of guidelines represents a significant evolution from the earlier versions of SP 800-63.
The ACD has built a first-rate team that focuses on the Cybersecurity Framework. The team combines a high level of cybersecurity expertise with an outstanding approach to stakeholder engagement and collaboration. The wide acceptance of the framework is ample testimony to their accomplishments in both technical cybersecurity and community engagement. Similarly, the NICE effort appears to be properly scoped and staffed. As the field continues to evolve, the NICE framework will need to continue to evolve as well, but the current personnel seem well positioned to track and incorporate changes as they occur.
In general, the scientific and technical talent was adequate for the projects and tasks that were undertaken by the ACD. Additionally, the research staff represented a diverse and inclusive mix of backgrounds, talent, and skills. For the research and testing conducted in the NCCoE laboratories, future research would benefit from having people with more adversarial experience in their backgrounds.
The ACD disseminates its outputs and interacts with extramural researchers and developers very well. Recognition of the Cybersecurity Framework is both industry-wide and worldwide and represents an example of a NIST project that is extremely well disseminated and recognized. Considerable effort is being devoted by the NICE project, with great effect, to dissemination of results and interaction with the community. The recently initiated Cybersecurity for the Internet of Things group appears to be connected with appropriate external organizations and is already in the process of developing a first document.
If problems arise in NCCoE products, NIST will need to be prepared to create updates and make users of the document aware that they have been released. The ACD, and in particular the NCCoE, would enhance its fulfillment of its practical, applied mission of strengthening the nation’s cybersecurity posture by proactively tracking and monitoring problems, attacks, and failures after solutions from the laboratories have been fielded.
RECOMMENDATION: The NCCoE should develop a process by which results from the field are systematically and proactively tracked and monitored after a project has been successfully transferred out of the NCCoE laboratory. The results from this proactive monitoring should then be disseminated (e.g., by the NIST Special Publications 1800 series) and appropriately incorporated into future NCCoE laboratory projects. (Chapter 5)