Summary

At the request of the Director of the National Institute of Standards and Technology (NIST), in 2020 the National Academies of Sciences, Engineering, and Medicine formed the Panel on Assessment of Selected Divisions of the National Institute of Standards and Technology (NIST) Information Technology Laboratory and established the following statement of task for the panel:

The NIST Director requested that the panel focus its assessment on the following factors:

1. The organization’s technical programs;
2. The portfolio of scientific expertise within the organization;
3. The adequacy of the organization’s facilities, equipment, and human resources; and
4. The effectiveness with which the organization disseminates its program outputs.

To accomplish the assessment, the National Academies assembled a panel of 24 volunteers whose expertise matched that of the work performed by the Information Technology Laboratory (ITL) staff.¹ On June 21-24. 2021, the panel conducted a virtual review (via Internet media) of the Information Access Division (IAD), the Software and Systems Division (SSD), and the Statistical Engineering Division (SED). During a plenary session, the panel received overview presentations by the acting NIST Director and the Director of the ITL. Subsequently, the panel spent approximately 1.5 days receiving presentations from and engaging in discussions with the staff at the three divisions reviewed. On the third day, the panel met in a closed session to deliberate on its findings and to define the contents of this assessment report. The panel met with NIST management on the fourth day to clarify open questions. The choice of projects to be reviewed was made by the ITL. The panel applied a largely qualitative approach to the assessment. Given the non-exhaustive nature of the review, the omission in this report of any particular ITL project should not be interpreted as a negative reflection on the omitted project. Crosscutting conclusions and recommendations are presented in this Summary. Additional conclusions and recommendations specific to individual ITL divisions are also presented in the body of this report.

TECHNICAL QUALITY OF THE WORK

The technical quality of the work is generally excellent, addressing the assigned mission areas. IAD, SSD, and SED continue to make significant contributions to address the ITL and NIST missions, national needs, and the needs of government, industry, and academic stakeholders. Amidst continuing technological and societal changes, there are opportunities for increased collaborations across projects to address common challenges in such areas as artificial intelligence, machine learning, health information technology, data science, and statistical methodology.

Trustworthy AI is an open problem. IAD has done a commendable job in laying out cross-cutting, high-level elements of trustworthiness, and there are additional cross-cutting contributions that it can make in terms of guidelines. However, standards will be highly dependent on the application area, and getting community trust depends on incorporating communities and their input at the design phase.

TECHNICAL EXPERTISE OF THE STAFF AND ADEQUACY OF STAFFING

Technical staff generally possess expertise adequate to perform their task. Some are among the best in the world in their areas of research. However, evolving needs in some areas (e.g., AI, ML, and data science) will produce expertise gaps that will have to be filled.

In terms of human resources, recruitment of highly qualified full-time staff has been identified as an overall challenge at ITL due to competition with industry and academia, which offer salaries that exceed the limits available to the ITL. It may be difficult for small teams to be effective due to lack of critical mass in some areas. The relatively large proportion of the permanent workforce at retirement age in some areas raises the potential need for succession planning to ensure maintenance of competence in core areas while meeting the demand for new areas of competency. Current and future ITL needs cannot be met without the addition of new competencies to complement existing strengths. New hires, to a large extent, will define the future and new activities of the SED.

The NIST Domestic Guest Researcher Program provides access for technically qualified U.S. citizens to NIST facilities and equipment while working with NIST staff on projects of mutual interest. Exchange programs would provide additional opportunities for collaboration and could serve as additional means of enticing recruits to work at NIST.

RECOMMENDATION: ITL should apply an aggressive, imaginative focus on hiring to replace retiring staff, to address important growth areas such as artificial intelligence, machine learning, and data science, and to fill specific gaps in the divisions. This effort should aspire to diversity targets.

RECOMMENDATION: ITL should plan and implement effective ways to recruit and retain a diverse workforce to ensure the appropriate staffing in areas of significant interest to national welfare and security, and to address severe competition from industry in areas such as artificial intelligence, cybersecurity, and the Internet of Things.

RECOMMENDATION: ITL should establish exchange programs with relevant government laboratories, academic institutions, and industry consortia to stimulate new ideas and problem areas, enhance competencies, and facilitate collaboration.

Properly managed, ITL’s diversity, equity, and inclusion strategy can add a helpful element to the recruiting process. The staff has performed commendably during the COVID-19 pandemic, collaboratively addressing critically important responses to the pandemic.

ADEQUACY OF FACILITIES AND EQUIPMENT

Post-pandemic planning gives NIST opportunities to continue some remote work, making NIST a more attractive environment for staff and giving NIST a needed edge in recruiting. Also, creative thinking about the new work environment can lead to more productive use of facilities, which could be considered during NIST’s ongoing laboratory and office renovations.

The ITL laboratory facilities are generally adequate and support well the activities of the divisions. There is a critical need, however, for improved computing capabilities that are needed to support complex computation and analysis. This is particularly the case for the growing number of projects that involve state-of-the-art machine learning models, which call for ever larger training sets and computational resources.

RECOMMENDATION: ITL should take steps to insure adequate resources, especially computing to support AI/ML and data science at sufficient scale.

RECOMMENDATION: To get access to the most modern resources, ITL should seek collaborations with other organizations in the public and private sectors, including other Government agencies. To achieve collaborative access, the ITL should examine its potential contributions to partnerships.

EFFECTIVENESS OF THE DISSEMINATION OF OUTPUTS

Each division disseminates its outputs widely, but the divisions vary in the relative emphasis placed on dissemination vehicles (e.g., publications, workshops, data repositories, standard reference data, and educational programs). Careful, systematic, and continuing analysis of the needs of specific stakeholder communities would improve the effectiveness of the dissemination of ITL’s outputs.

RECOMMENDATION: ITL should broaden its impact to non-technical stakeholders, policy makers, and the public.

RECOMMDNATION: ITL should improve messaging aimed at non-technical audiences such as policy makers, media, and the general public for the outputs of the Information Access, Software and Systems, and Statistical Engineering Divisions.

This would benefit NIST by providing greater acceptance of and support for NIST efforts.