An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2011 (2011)

1

The Charge to the Panel and the Assessment Process

At the request of the National Institute of Standards and Technology (NIST), the National Research Council (NRC) has, since 1959, annually assembled panels of experts from academia, industry, medicine, and other scientific and engineering environments to assess the quality and effectiveness of the NIST measurements and standards laboratories, of which there are now six,⁷ as well as the alignment of the laboratories’ activities with their missions. NIST requested that three of its laboratories be assessed in 2011: the Center for Nanoscale Science and Technology, the NIST Center for Neutron Research, and the Information Technology Laboratory (ITL). Each of these was assessed by a separate panel of experts; the findings of the respective panels are summarized in separate reports. This report summarizes the findings of the Panel on Information Technology.

For the fiscal year (FY) 2011 assessment, NIST requested that the panel focus on the following criteria as part of its assessment:

1. Assess the degree to which laboratory programs in measurement science, standards, and services achieve their stated objectives and fulfill the mission of the operating unit (laboratory);
2. Assess the technical merits and scientific caliber of the current laboratory programs relative to comparable programs worldwide; and
3. Assess the alignment between laboratory research and development (R&D) efforts and those services and other mission-critical deliverables for which the laboratory is responsible.

The context of this technical assessment is the mission of NIST, which is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve the quality of life. The NIST laboratories conduct research to anticipate future metrology and standards needs, to enable new scientific and technological advances, and to improve and refine existing measurement methods and services.

In order to accomplish the assessment, the NRC assembled a panel of 18 volunteers whose expertise matches that of the work performed by the ITL staff.⁸ The panel members were also assigned to six subgroups (division review teams), whose members’ expertise matched that of the work performed by staff in the six divisions in the ITL: Applied and Computational Mathematics Division (ACMD), Advanced Network Technologies Division (ANTD), Computer Security Division (CSD), Information Access Division (IAD), Software and Systems Division (SSD), and Statistical Engineering Division (SED).

_____________________

⁷ The six NIST laboratories are the Material Measurement Laboratory, the Physical Measurement Laboratory, the Engineering Laboratory, the Information Technology Laboratory, the Center for Nanoscale Science and Technology, and the NIST Center for Neutron Research.

⁸ See http://www.nist.gov/itl/ for more information on ITL programs. Accessed April 18, 2011.

The panel met at the NIST facilities in Gaithersburg, Maryland, on March 21-23, 2011. After the full panel had met for a session of welcoming comments from the NIST Acting Associate Director for Laboratory Programs and overview presentations on the ITL and six ITL programs by the laboratory’s management and staff, the panel divided into its six review teams, and each team (led by a team leader chosen from within the panel) then visited its respective ITL division for about a day. During these visits, the review team members attended presentations, tours, demonstrations, and interactive sessions with the ITL staff. Subsequently, the entire panel assembled for about a day and a half, during which it interacted with ITL and NIST management and also met in closed session to deliberate on its findings and to define the contents of this assessment report.

The approach of the panel to the assessment relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the technical areas of ITL activities. The panel reviewed selected examples of the technological research covered by the ITL; because of time constraints, it was not possible to review the ITL programs and projects exhaustively. The examples reviewed by the panel were selected by the ITL in consultation with the panel chair and NRC staff. The panel’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the following: the degree to which the ITL programs achieve their stated objectives and fulfill the ITL mission, the technical merit and scientific caliber of the ITL work, and the alignment between ITL R&D efforts and ITL services and other mission-critical deliverables. These examples are intended collectively to portray an overall impression of the laboratory, while preserving useful suggestions specific to projects and programs that the panel examined. The panel applied a largely qualitative rather than a quantitative approach to the assessment, although it is possible that future assessments will be informed by further consideration of various analytical methods that can be applied.

For its assessment, the panel relied primarily on presentations made by NIST and ITL managers and staff and by other researchers associated with NIST projects and programs, and on informational notes prepared by NIST and ITL staff for use by the panel. This report does not contain extensive citations of technical articles and reports. Other documents and resources used by the panel are cited in the report, as appropriate.

The comments in this report are not intended to address each program within the ITL exhaustively. Instead, this report identifies key issues. Given the necessarily non-exhaustive nature of the review process, the omission of any particular ITL program or project should not be interpreted as a negative reflection on the omitted program or project.

Chapter 2 of this report presents a more detailed summary of the panel’s assessment of the Information Technology Laboratory. Chapter 3 presents the panel’s assessment of the divisions within the laboratory. Chapter 4 summarizes the overall conclusions of the panel.