At the request of the Director of the National Institute of Standards and Technology (NIST), in 2015 the National Research Council1 formed the Panel on Review of the Information Technology Laboratory at the National Institute of Standards and Technology and formulated the following statement of task for the panel:
The National Research Council shall appoint a panel to assess the scientific and technical work performed by the National Institute of Standards and Technology (NIST) Information Technology Laboratory. This panel will review technical reports and technical program descriptions prepared by NIST staff and will visit the facilities at the Information Technology Laboratory. Visits will include technical presentations by NIST staff, demonstrations of NIST projects, tours of NIST facilities, and discussions with NIST staff. The panel will prepare a report summarizing its assessment findings.
NIST specified that three of the six divisions of the Information Technology Laboratory (ITL) would be reviewed: the Information Access Division (IAD), the Software and Systems Division (SSD), and the Statistical Engineering Division (SED). The following ITL divisions were not reviewed because they had recently been reorganized: the Applied and Computational Mathematics Division, the Computer Security Division, and the Advanced Networking Technologies Division. NIST plans to request their review in the future. The NIST Director requested that the panel focus its assessment on the following factors:
- Assess the organization’s technical programs.
- How does the quality of the research compare to similar world class research in the technical program areas?
- Is the quality of the technical programs adequate for the organization to reach its stated technical objectives? How could it be improved?
- Assess the portfolio of scientific expertise within the organization.
- Does the organization have world-class scientific expertise in the areas of the organization’s mission and program objectives? If not, what areas should be improved?
- How well does the organization’s scientific expertise support the organization’s technical programs and the organization’s ability to achieve its stated objectives?
- Assess the adequacy of the organization’s facilities, equipment, and human resources.
- How well do the facilities, equipment, and human resources support the organization’s technical programs and its ability to achieve its stated objectives? How could they be improved?
- Assess the effectiveness by which the organization disseminates its program outputs.
- How well are the organization’s research programs driven by stakeholder needs?
- How effective are the technology transfer mechanisms used by the organization? Are
_________________________
1 Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council are used in an historical context identifying programs prior to July 1.
these mechanisms sufficiently comprehensive?
- How well is the organization monitoring stakeholder use and impact of program outputs? How could this be improved?
This summary presents general observations about the ITL divisions reviewed and observations and recommendations specific to the divisions reviewed.
GENERAL OBSERVATIONS
ITL has a unique role as convener and facilitator of research and technology development. This role of the Laboratory and, more broadly, of NIST, is of great importance to the nation. ITL has a very broad range of projects, some with significant influence on public policy, such as the projects on voting, health-related information technology, forensics, and cyberphysical and cybersecurity systems. Resources are spread thin, even in these key projects. As described below, each division faces special challenges with respect to needed expertise and numbers of researchers. While there was modest evidence of a systematic, data-driven process of planning for personnel and resources, both short- and long-term, this planning needs improvement.
The ability to respond successfully to mandates and to generate new research projects in support of its core mission is essential to ITL. There was modest evidence of systematic horizon-scanning in the realms of government, industry, and the sciences, but ITL needs to improve its horizon scanning.
The role of convener and facilitator of research and technology development is a major part of ITL’s contribution. However, ITL’s outreach activities do not seem to make optimal use of current Internet-based capabilities.
DIVISION-SPECIFIC OBSERVATIONS, CONCLUSIONS, AND RECOMMENDATIONS
Information Access Division
The technical programs in the IAD are, in general, well organized, well staffed, and appropriate to the mission of ITL and to the IAD itself. The research teams enjoy broad peer recognition while supporting the NIST mission. The core competencies and facilities in this division are strong, and interactions with other parts of ITL and NIST are evident, if not always formalized. It is important that while maintaining momentum and leadership the IAD also consider enhancing engagement in new opportunities in areas such as data-driven information science, security policy, and health informatics. There was modest evidence of ongoing critical review of programs.
The portfolio of scientific expertise is appropriate and impressive for the traditional areas that the IAD focuses on, but core capabilities need to be enhanced in areas where the IAD aspires to make a major contribution.
Recommendation 1: The Information Access Division should review the current and proposed portfolios and examine the adequacy of its core competencies for ambitious new programs like data science, health informatics, and public safety.
Supporting the standards process places resource and attention-span demands on personnel. One option to reduce the demands could be to turn over standards materials to the International Organization for Standardization (ISO) and other standards development organizations (SDOs). However, this approach is sometimes associated with update cycles that are too long and possibly too onerous to meet the original intent of the standard, and it entails risk to IAD credibility.
Recommendation 2: The Information Access Division (IAD) should continually consider the pros and cons of turning over standards materials—for example, the Common Industry Format (CIF)—to the International Organization for Standardization (ISO) and other standards development organizations (SDOs). If the ISO-SDO channel remains the best or only approach, the IAD should consider how to plan proactively for updates as appropriate and should propose updates to the controlling SDO(s) whenever IAD efforts indicate they are needed or desired by the user community.
Facilities, equipment, and human resources are strong assets of IAD. As areas such as health informatics gain importance, there seems to be a tendency to make do with the existing core talent areas rather than hiring specialists—for example, trained biomedical informaticians and health data scientists—permanently or even for shorter periods of time. This practice needs to be reconsidered during strategic planning for the division. A prime example of where such expertise is needed is the electronic health record team.
Recommendation 3: The Information Access Division (IAD) should bolster the electronic health record team to ensure that this high-profile work is well supported in terms of information technology rigor as well as domain expertise. IAD should add scientists or contractors with strong clinical experience, include all stakeholders (physicians, nurses, technicians, pharmacists, and patients) in its empirical investigations, and participate in medical informatics communities such the American Medical Informatics Association(AMIA) and the congressionally chartered Patient-Centered Outcomes Research Institute (PCORI). Internally, the IAD should collaborate closely with the Text Retrieval Conference (TREC) team on medical tracks.
Collaborative teamwork with scientists and engineers inside and outside IAD, including virtual teamwork and in-person interactions, is of fundamental importance to the work of the IAD staff.
Recommendation 4: To maximize the mechanism of virtual teams and to foster contacts with other laboratories, the Information Access Division should give greater consideration to the formulation of clear policies and processes that support sabbaticals and appropriate travel.
The IAD value proposition with standards and conformance toolsets is outstanding and deserves continued nurturing and support. Its dissemination process would be enhanced by greater use of Internet-based methods.
Recommendation 5: The Information Access Division should apply more Internet-based methods to its dissemination and outreach process. Such methods as webinars, virtual meetings, and recordings should be considered.
Statistical Engineering Division
The SED is well-aligned and effective in its support of ITL and NIST missions. The division’s portfolio includes a wide variety of projects, including classical statistical metrology projects such as those on mass calibration weighing designs. Other work supports national and international best statistical practices in metrology and the development of new methods of measurement and uncertainty quantification for important specialized technical applications. The SED efforts are technically challenging and well executed. The division’s scientific expertise in statistics for metrology continues to be among the best in the world. The SED continues to have a low profile in the national and international
technical statistics community. If this profile were to be enhanced, SED could more fully benefit from and contribute to current statistical research.
Recommendation 6: The Statistical Engineering Division should take steps to raise its profile through technical publications in statistics venues and through educational efforts to improve the awareness of statistics educators of the vital role of measurement in data collection and the role of statistics in good measurement.
Attention to both metrology and the basic statistics field needs to be a division priority to move both areas forward and guarantee ongoing excellence in supporting NIST priorities.
Recommendation 7: The Statistical Engineering Division should seek stronger ties with the statistical research community in terms of both publications and human resources.
Progress has been made on human resource issues, but important challenges remain to provide technical staffing on important projects and in critical areas, as well as to anticipate future staff losses with potential retirement.
Recommendation 8: In hiring permanent employees, the Statistical Engineering Division (SED) should focus attention on (1) ensuring corporate memory of and expertise in the fundamental ongoing National Institute of Standards and Technology (NIST) standard reference material (SRM), calibration, and the experimental design work that has always been part of the SED portfolio, and on developing new capabilities in priority areas such as shape metrology and greenhouse gas monitoring, which seem to be particularly thinly staffed; (2) maintaining high overall technical competence; (3) improving staff diversity; and (4) balancing emphases on the noncore and core areas—for example, while forensics is a current laboratory-wide priority and SED is a key participant, when hiring for forensics the SED should address general expertise, so that other more thinly staffed critical areas are not neglected.
Recommendation 9: The Statistical Engineering Division should invigorate its program of graduate student internships to include two to four Ph.D. students per summer, and it should begin to develop a much broader and stronger program of visiting faculty researchers.
Publications, presentations, interacting with visiting researchers, and sponsoring interns are important means for engaging with the statistical and metrology communities.
Recommendation 10: The Statistical Engineering Division should monitor its yearly division total publications in the three categories—subject-matter journals, metrology journals, and statistics journals—and external research presentations, research visitor days, and internships sponsored.
Software and Systems Division
The SSD is engaged in several high profile areas, notably in voting, health information technology (IT), and cyberphysical systems (CPS). It is crucial that the SSD, in collaboration with ITL and NIST management, delineate the risks and rewards of each area in the context of organizational strategy. By defining the stakeholders and success metrics, it can match competence and resources to the risks and rewards.
The SSD work on cloud computing standards has had substantial government and industry impact, both within the United States and internationally. The NIST cloud-computing reference architecture is now internationally accepted, and it has led to a voluntary, consensus ISO standard.
The SSD group working on computational metrology has been successful in attracting strong scientific collaborators in the stem-cell therapy area and in collaborations that have generated both publicly distributed software and published scientific results. The group’s focus could be expanded to a broader set of areas. The inherent strengths of the SSD in computational metrology might be better served by a sharper focus on metrology methodology issues and metrology-related imaging standards.
The SSD group working on voting security and verification has played a major role in voting systems development and implementation for more than a decade. The group has a low public profile that is not commensurate with its contribution.
The SSD work on software assurance has produced useful data sets that can benefit the producers of software tools. The group has also had success in raising awareness about the existence of these data sets, including databases of common software risks and security vulnerabilities, and the existence of test cases. The group’s portfolio of activities could benefit from closer alignment with the work of other ITL units. More could be done by the SSD to develop effective software test tools and methodologies and to disseminate them—for example, for conformance and interoperability testing. The SSD could take a leading role in the development of software test tools.
The SSD group working on cyberphysical systems and the Internet of Things has developed a technically sound portfolio of activity that addresses some of the important issues in the area. The group has demonstrated leadership by launching projects such as the smart city grand challenge and the CyberPhysical Systems Public Working Group (CPS PWG). The recognition and engagement with the scientific community gained by the SSD in this area is not yet matched by a similar level of visibility and engagement in the industrial community.
The SSD group addressing electronic health records is working hard to respond to challenging mandates from the Office of the National Coordinator (ONC). However, it is not clear how the needs of the continuously evolving ONC strategy can best be addressed by ITL in a sustainable way. The current staff comprises capable computer scientists and information technologists but includes no biomedical/health informaticians.
It would be beneficial for the SSD to increase its engagement with the scientific and industrial communities.
Recommendation 11: The Software and Systems Division not only should participate with the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) joint technical committee working group on the Internet of Things (ISO/IEC JTC 1/WG 10) and the Institute of Electrical and Electronics Engineers (IEEE) project on the Standard for an Architectural Framework for the Internet of Things (P2413), but also should reach out to industrial consortia such as the Industrial Internet Consortium and the Open Interconnect Consortium.
Recommendation 12: The Software and Systems Division should make open to the scientific and industrial communities the framework that it put together for the Smart Grid, as well as tools and techniques developed in university projects such as Precision Timed (PRET) machines project at the University of California, Berkeley.
Recommendation 13: The Software and Systems Division should look into growing its approach to timing in connection with Internet of Things applications by considering protocols that are robust to clock drift.
It is not clear that the ONC has a comprehensive strategy that allocates the tasks needed for the nation to achieve interoperable EHRs and EHR systems. Within this context, the SSD is faced with making strategic choices as to the most effective focus for its EHR work.
Recommendation 14: For electronic health records (EHRs) the Software and Systems Division (SSD) should define a clear direction that allows it to either focus on limited objectives or add staff of the appropriate types to meet larger expectations. Because focusing on meaningful use of EHRs is intrinsically clinical in nature, the SSD should consider adding clinical informaticians to its staff.
