1
The Charge to the Panel and the Assessment Process
At the request of the National Institute of Standards and Technology (NIST), the National Academies of Sciences, Engineering, and Medicine has, since 1959, annually assembled panels of experts from academia, industry, medicine, and other scientific and engineering environments to assess the quality and effectiveness of the NIST measurements and standards laboratories, of which there are now six,1 as well as the adequacy of the laboratories’ resources.
At the request of the Director of NIST, in 2020 the National Academies formed the Panel on Assessment of Selected Divisions of the National Institute of Standards and Technology (NIST) Information Technology Laboratory and established the following statement of task for the panel:
The National Academies shall appoint three panels to assess independently the scientific and technical work performed by the National Institute of Standards and Technology (NIST) Physical Measurement Laboratory, Information Technology Laboratory, and Center for Neutron Research. Each panel will review technical reports and technical program descriptions prepared by NIST staff and will visit the facilities of their respective NIST laboratory. Visits will include technical presentations by NIST staff, demonstrations of NIST projects, tours of NIST facilities, and discussions with NIST staff. Each panel will deliberate findings in closed session panel meetings and will prepare a separate report summarizing its assessment findings. The Panel on Assessment of Selected Divisions of the Information Technology Laboratory at the National Institute of Standards and Technology will review the following divisions of the NIST Information Technology Laboratory: Information Access, Software and Systems, and Statistical Engineering. This panel will not access restricted information; the report summarizing its assessment will contain only public release information.
The NIST Director requested that the panel focus its assessment on the following factors:
- The organization’s technical programs;
- The portfolio of scientific expertise within the organization;
- The adequacy of the organization’s facilities, equipment, and human resources; and
- The effectiveness with which the organization disseminates its program outputs.
The context of this technical assessment is the mission of NIST, which is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve the quality of life. NIST laboratories conduct research to anticipate future metrology and standards needs, enable new scientific and technological advances, and improve and refine existing measurement methods and services.
NIST specified that three of the six divisions of the Information Technology Laboratory (ITL) would be reviewed: the Information Access Division (IAD), the Software and Systems Division (SSD),
___________________
1 The six NIST laboratories are the Engineering Laboratory, the Physical Measurement Laboratory, the Information Technology Laboratory, the Material Measurement Laboratory, the Communication Technology Laboratory, and the NIST Center for Neutron Research.
and the Statistical Engineering Division (SED). The following ITL divisions were not reviewed because they had previously been reviewed (NIST may request their review in the future): the Applied and Computational Mathematics Division, the Computer Security Division, the Applied Cybersecurity Division, and the Advanced Network Technologies Division. In order to accomplish the assessment, the National Academies assembled a panel of 24 volunteers, whose expertise matches that of the work performed by ITL staff.2
On June 21-24, 2021, the panel conducted a virtual review (via Internet media). During a plenary session, the panel received overview presentations by the acting NIST Director and the Director of the ITL. Subsequently, the panel spent approximately 1.5 days receiving presentations from and engaging in discussions with the staff at the three divisions reviewed. These presentations and discussions were attended by the following three separate teams of panel members, with the chair, Philip Neches, attending portions of each session:
- For the IAD: Ruzena K. Bajcsy, Duane Blackburn, Stephanie A. Schuckers, Peter Tu, Mari Ostendorf, Christopher Manning, Mary Ellen Zurko, and Juan E. Gilbert;
- For the SSD: Mark Edward Dean, Suzanne Bakken, Don Eugene Detmer, J. Marc Overhage, Peter M. Kogge, Dorin Comaniciu, Phillip Colella, Ming C. Lin, Gerard J. Holzmann, and James Darnell;
- For the SED: Stephen B. Vardeman, Alyson G. Wilson, Janes O. Berger, Wing Hung Wong, and Peter A. Beling.
On the third day, the full panel met in a closed session to deliberate on its findings and to define the contents of this assessment report. The full panel met with NIST management on the fourth day to clarify open questions.
The panel’s approach to the assessment relied on the experience, technical knowledge, and expertise of its members. The panel reviewed selected examples of the technical research performed at ITL; because of time constraints, it was not possible to review ITL programs and projects exhaustively. The examples reviewed by the panel were selected by ITL. The panel’s goal was to identify and report salient examples of accomplishments, challenges, and opportunities for improvement with respect to the factors suggested above by the Director of NIST. These examples are intended collectively to portray an overall impression of the laboratory, while preserving useful suggestions specific to the projects and programs that the panel examined. The panel applied a largely qualitative, rather than quantitative, approach to the assessment.
Given the necessarily broad but not exhaustive nature of the review, omission in this report of any particular ITL program or project should not be interpreted as implying any negative reflection on the omitted program or project.
___________________
2 See the NIST Information Technology Laboratory homepage for information on organization and programs at https://www.nist.gov/itl/about-itl/, accessed May 17, 2021.
