The Information Technology Laboratory is a very strong scientific and technical resource for the nation with a long history of making lasting contributions. The ITL is composed of six divisions: Applied and Computational Mathematics, Advanced Network Technologies, Computer Security, Information Access, Software and Systems, and Statistical Engineering. The ITL’s staff resources total 562, comprising the following: 322 technical federal staff members (57 percent of the total), 176 technical guest researchers and contractors (31 percent), 46 administrative staff (8 percent), 15 faculty (3 percent), and 3 postdoctoral researchers (1 percent). Of the ITL staff, 38 percent are computer scientists, 20 percent are mathematicians or statisticians, 17 percent are information technology specialists, 12 percent are engineers or physicists, 12 percent are administrative and support personnel, and 1 percent are social scientists. The following sections present summary assessments for each of the ITL divisions.
The Applied and Computational Mathematics Division provides leadership within NIST in the use of applied and computational mathematics to solve scientific and engineering problems that arise in measurement science and related applications. The division’s staff does research and development in mathematics and computation and participates with NIST colleagues in peer-to-peer collaborations on NIST problems. Further, the ACMD supports the national and worldwide science and engineering communities by developing and disseminating mathematical reference data, software, and related tools. The division works with internal and external groups to develop standards, test procedures, reference implementations, and other measurement technologies for scientific computation.
The ACMD is composed of four groups: Mathematical Analysis and Modeling, Mathematical Software, Computing and Communications Theory, and High Performance Computing and Visualization. The division has 57 federal employees; 51 are at the Gaithersburg, Maryland, NIST campus, and 6 are at the Boulder, Colorado, NIST campus. Three of the federal employees are postdoctoral researchers. The ACMD also has 29 associates, of whom 23 are guest researchers, 1 is a postdoctoral researcher, and 5 are students.
In regard to the charge of the Director of NIST to assess the degree to which laboratory programs in measurement science, standards, and services achieve their stated objectives and fulfill the mission of the ITL: the division has an impressive array of technical expertise, and it is attacking a wealth of research areas. The quality of the work meets the computational accuracy needs of consumer laboratories, which vary widely among applications (e.g., from accuracy sufficient for visualization to highly accurate computations for atomic structure). In most areas, the work of the division is very good and comparable to what one would find in top 10 to top 20 university applied/computational mathematics programs. In some cases—for example, the Digital
Library of Mathematical Functions—the work is unique and without peer in the broader community.
The work of the division is accomplished primarily through a culture of small “atomic” collaborations common in the work of applied mathematics. Impressively, members of the division have established effective collaborations with materials science and physics researchers.
On the downside, however, the culture of small, atomic collaborations may be a hindrance to attacking important areas that require the formation of collaborative teams from across many disciplines. However, the DLMF work demonstrated a capacity to form a larger team of collaborators than is typical of this division. This ability to form larger, effective collaborations may, in the long run, bode well for ACMD’s ability to cope with future multidisciplinary problem areas in which the ACMD will be called on to participate.
In regard to the charge of the Director of NIST to assess the technical merits and scientific caliber of the current laboratory programs relative to comparable programs worldwide: There are areas in which the accomplishments and capabilities of the ACMD are excellent and surpass the capabilities of comparable programs worldwide. The ability of staff to engage with physicists and materials scientists to use mathematics and computing to provide insight on experiments is among the most successful such efforts in the world. The Digital Library of Mathematical Functions is a unique and enduring accomplishment. The Quantum Communications Program seems to be among the best in the field.
In other areas the level of technology in the division work, while still excellent, is below the state-of-the-art level. That is, the work is excellent by academic standards in terms of the development of novel mathematical methods but is below the state of the art when measured against the work of the division’s peers in other mission agencies in meeting the goals of providing mathematical modeling and simulation expertise to the rest of their organizations. For example, in the area of large-scale partial differential equation (PDE) simulation, capabilities are mostly (but not exclusively) two-dimensional. In addition, the level of mathematical rigor varies widely across the division. Code verification is now viewed as an essential component for any large-scale simulation project. Very systematic convergence studies similar to those done in the Parallel Hierarchical Adaptive MultiLevel (PHAML) project might be adopted in other areas as a method of code verification.
As to the charge to assess the alignment between laboratory R&D efforts and those services and other mission-critical deliverables for which that laboratory is responsible: While the level of technical expertise in the ACMD is certainly sufficient to meet upcoming challenges, some significant cultural problems in the program composition and scientific culture in the division could inhibit the response of the ACMD to the challenges that the division views as critical to meeting its future needs.
The challenges facing the ACMD are multiscale and multiphysics, involve complex geometries and new applications, and are of necessity multidisciplinary. In addition, computational science is facing a major disruptive change in technology. This change is the transition to multiple computing cores on a single processor, for which many of the methods for parallelization and software engineering developed over the past 15 years lead to very low performance. Since this is an architectural change on the level of a single processor, it will affect scientific computing at all scales.
The response of the larger applied mathematics community to the vastly increased complexity of these problems has been to form larger teams, sustained over long periods of time, in order to span the broad range of intellectual capabilities and challenges required to make a successful attack on such problems. This is being done at universities, at funding agencies, and at other national laboratory systems. This kind of change has not yet taken place in the ACMD, and in fact it runs contrary to the more traditional “small science” model in applied mathematics that has served the ACMD well in meeting many of its mission requirements. It is certainly possible for the ACMD to form larger teams, as evidenced by the DLMF and (to some extent) by quantum communications projects. However, if the problems that NIST is investigating or will investigate are multiscale and multiphysics, then such an approach will need to become more widely used in the ACMD. Such a change may well require the ITL and NIST management to reconsider the role of the ACMD in the laboratory as well. Owing to resource constraints, it may be necessary to focus on a set of technologies and applications that lead to a high degree of synergy, possibly at the cost of reducing the range of stakeholders at NIST. Another possible approach to meeting these challenges, particularly in the simulation software area, would be to collaborate with the laboratories in other agencies that would bring strengths complementary to those of the ACMD (such as its success in collaborating with physical scientists) and with whom the division could share the responsibility of software development. The ITL and the ACMD should consider strategies to change the scientific culture of the ACMD to meet the increased challenges facing the division. In addition to broader, multidisciplinary simulations, discrete simulations might also be an area in which the ACMD can address important challenges that align with the NIST mission (network standard, for example).
The mission of the Advanced Network Technologies Division is to “provide the networking industry with the best in test and measurement research.”22 Its goals are “to improve the quality of emerging networking specifications and standards,” and “to improve the quality, reliability, resilience, robustness, security, and interoperability of networked systems.”23 The ANTD assists in the conception and development of national priorities in which networking is an essential enabler, and additionally it proactively participates in the implementation of those developments.
The ANTD is composed of two groups: the Emerging and Mobile Network Technologies and the Internet and Scalable Systems Metrology groups. The ANTD has a regular staff of 29, and there are 10 guest researchers.
The ANTD focus areas are core networking infrastructure, ad hoc networks, edge networks, complex systems, and networks and applications (cloud computing, smart grid, and health care). The division’s delivery mechanisms include specifications and guidelines, models and prototypes, test and measurement tools, reference implementations and demonstrations, journal and conference papers, workshops and conferences, and standards activities.
22 Donna F. Dodson, ANTD and CSD, “Advanced Network Technologies Division,” presentation to the panel, Gaithersburg, Maryland, March 21, 2011, p. 2.
23 Ibid., p. 3.
High-Level Observations and Recommendations
The Advanced Network Technologies Division is strong and healthy, exhibiting an awareness of and competence in achieving its mission within the ITL and within NIST as a whole. Morale and motivation across the division appear high. The ANTD leverages its relatively small size by taking advantage of opportunities for collaboration, both internally with other divisions and with external organizations. It understands its mission and strives to balance various demands on its attention, including mandates, supporting science, standards advocacy, and strategic collaborations. The division demonstrates an aptitude for responding to externally imposed agendas, often under tight time lines with limited resources, while attempting to maintain a solid scientific foundation.
Networking is a key enabling technology for many, if not all, of the high-priority activities of the ITL and NIST in general. National innovation and industrial competitiveness are fueled by a robust, secure, and accessible national infrastructure. The ANTD is and should continue to be at the forefront of the national discussion on vital issues, standards, and policies of networking and worldwide communications infrastructures.
Since the previous NRC review in 2009, the ANTD has made significant progress on that panel’s suggestion for expanding the division’s exposure in the greater community outside of specific standards organizations. The ANTD has increased the quantity and quality of its publications, including papers in conference proceedings and journals that the panel noted as being of high quality, and a staff member received a best-paper award. The ANTD also has widened its external collaborations, in some instances with grants and sometimes by seeking out companies, universities, and other research organizations with common interests.
Despite these observed accomplishments, the ANTD faces several challenges. First, it needs to acquire without delay a full-time director to articulate ANTD’s values and interests and to define effectively long-term directions for the division and its projects. Second, as a key collaborator in many projects, the ANTD finds itself understandably drawn into activities that address many national priorities; these activities are not only aligned with NIST’s charter but also positively impact national policy and the economy. However, as a consequence of so many commitments, the division seems to be understaffed, and the researchers’ ability to balance short-term and long-term NIST goals can be impaired. Ideally, near-term projects in areas such as smart grid and safety networks can be developed into core competencies, aligned with long-term strategic goals in these important areas.
Following are the recommendations of the panel with respect to the Advanced Network Technologies Division:
- The ITL should fill the position of chief of the ANTD with a permanent chief.
- ANTD management should address the understaffing issue in the division, and in particular it should ensure that there are adequate resources to handle both the short- and long-term needs of the division.
3. ANTD management should create a strategic roadmap for the technical work of the division. The roadmap should be useful in managing the division’s resources and portfolio of activities.
4. In addition to its portfolio of important activities, the ANTD should embrace the future importance of cloud computing and draw up a plan for ANTD’s leading role in a multidivision initiative within the ITL.
Project-Specific Observations and Recommendations
Internet Infrastructure Protection
In the area of Internet infrastructure protection, the ANTD has an exemplary portfolio that represents technical foundations, standards leadership, policy guidance, and adoption facilitation. The ANTD group provided an exceptionally lucid and compelling presentation for the panel, detailing the motivation, accomplishments, and impacts of its outstanding multifaceted work in this area. Particularly in the area of standards—with regard both to the definition of standards (e.g., in which group members have co-authored key standards documents) and to their adoption (e.g., IPv6 and the .gov deployment of the DNSSEC)—the group is at the forefront of the field. The group’s leadership here results in part from its sustained and long-term efforts in the area. Developing formal methods to prove security properties of the various systems under study might be a worthwhile research topic; although this topic had been explored in the past, recent developments may justify another look at it as a possible growth area.
The area of smart grids has tremendous impact and importance, and it fits into NIST’s mission for supporting infrastructures that increase U.S. competitiveness. This is new work for the division, and consequently the current work is somewhat narrowly focused and does not take advantage of the breadth of the possible challenges. This narrow focus may be because the group is responding to specific external requests. Networking researchers can and should play a significant role in helping to define and evaluate future smart grid architectures, including and beyond the communications aspects of smart grids. The networking field, with more than 40 years of combined experience in developing a highly scalable, informationally decentralized network-of-networks, may well be able to contribute architectural, measurement, management, control, and security techniques to the smart grid.
Public Safety Communications
The efforts of the ANTD in public safety communications are aimed at developing performance-analysis tools to evaluate deployment scenarios and to evaluate new technologies. These efforts are in collaboration with a number of federal agencies, including the Department of Homeland Security (DHS) and the National Institute for Occupational Safety and Health (NIOSH), as well as the Institute for Telecommunication Sciences in Boulder, Colorado. The activities thus far have been a conceptually straightforward evaluation of wireless-network deployment scenarios with various
channel models, with existing radio technologies. The private sector has likely developed similar (possibly proprietary) tools. Before embarking on a large-scale or long-term activity in this area, an analysis of existing tools (both public and private) should be conducted, and partnerships with any promising tool developers should be explored.
Body Area Networks
Body Area Networks is a cross-divisional program (involving collaborators from the ANTD and the Applied and Computational Mathematics, Statistical Engineering, Software and Systems, and Information Access divisions) with a number of outside collaborators. The panel was briefed only on the project developing a radio-frequency propagation model for implantable and wearable medical sensors. The group is making solid progress on the development of the radio propagation model using detailed models of the human body and on the demonstration of the use of these models to determine channel performance. These models have been included in a modeling tool. Model validation for body surface propagation has proven extremely valuable and has provided important insights; additional validation and continued (and extended) collaboration with external research groups will be valuable. Congratulations are in order for the group’s receipt of the Institute of Electrical and Electronics Engineers Personal, Indoor and Mobile Radio Communications Conference Best Paper Award.
Measurement Science for Complex Information Systems
The area of measurement science for complex information systems is an excellent example of core competency development through multidisciplinary, multidivisional research (e.g., involving researchers from the ANTD and the Statistical Engineering and the Applied and Computational Mathematics divisions). This work addresses an increasingly common challenge in networking (and, more generally, the emerging interdisciplinary field of network science) in which complex, large-scale systems must be evaluated over a high-dimensional parameter space. The interdisciplinary contributions here make the whole greater than the sum of the parts. The connection to a specific application (cloud computing in this case) and the choosing of model details (such as the virtual machine types and hardware platform classes) based on an Amazon Elastic Compute Cloud (EC2) data center, are beneficial. The transition from congestion control to a topical new area while retaining all previously acquired knowledge is a commendable accomplishment. This is a well-chosen area with strategic vision.
The Computer Security Division is composed of three groups: Cryptographic Technology, Systems and Emerging Technologies Security Research, and Security Management and Assurance. The CSD’s mission, “[to] provide standards … to protect our nation’s … information systems,”24 is being achieved exceptionally well. Generally speaking, the division’s activity can be divided into two categories: (1) activities that directly create and communicate mission-critical deliverables, and (2) basic research and
24 Donna F. Dodson, ANTD and CSD, “NIST Computer Security Division,” presentation to the panel, Gaithersburg, Maryland, March 21, 2011, p. 2.
development activities that support the identification, improvement, and creation of those mission deliverables.
Mission-critical deliverables include a broad range of standards (including cryptographic standards), technical and management guidelines, validation, testing, and measurement tools, and technical reference materials and specifications to support agency implementations. As a shining example, the NIST Special Publication 800* series is renowned for providing technically sound, unbiased, relevant guidelines that are frequently adopted voluntarily in private-sector procurements and practices and often mandated by the OMB for use by the federal government. It is generally agreed that NIST security guidelines are consistently superior to guidelines from other sources, which are often vague, unrealistic, incomplete, or idiosyncratic.
The division program includes the following: coordination of large new efforts such as the cryptographic hash competition and the Security Content Automation Protocol Program; independent good judgment as exercised in the specification of the transition to new key lengths; the development of conceptual taxonomies that structure research in new fields such as cloud computing; the development of validation of products, tools, and techniques through tools and trained outsiders; and outreach to agencies, industry, and academia through NIST-managed workshops, external conferences, and government committee participation. Given the broad scope of the publication series, some publications will need updating soon; the authors are aware of this need. The National Vulnerability Database (NVD) is another example of an outstanding collaborative effort that is widely used and appreciated across the federal government and by U.S. business. Efforts in the important area of authentication, such as the study of biometrics risks and participation in the public-private NSTIC digital identity initiative, tackle a messy, urgent problem for which no fully satisfactory solution is ready yet.
Research and development activities explore basic science and engineering that will sustain the division’s ability to create high-quality deliverables into the future as new challenges emerge. Several of these projects are closely related to the division’s programs in standards and technical guidance. For example, the work on combinatorial circuit analysis and on side channel and differential power analysis is related to the division’s work on cryptographic module validation. Such research work is important in that it provides the division with the technical know-how required to evaluate modules or components that implement cryptography.
The merit and caliber of the division’s mission deliverables are widely acknowledged as being world-leading. Historical examples of this leadership include the Advanced Encryption Standard (AES) introduced in 2001. The AES has been accepted internationally and is an exemplar for conducting open competitions for cryptographic standards. The CSD is at present conducting an international competition for a new secure hash algorithm, the so-called SHA-3, and all indications are that the quality and integrity of this process are at the same high levels. A significant number of other security-related standards have been developed or are in process now (block cipher modes, digital signatures, and key management, for example), and many influential special publications (SPs) have been written (SP 800-108, Recommendation for Key Derivation Using Pseudorandom Functions; SP 800-132, Recommendation for Password-Based Key Derivation, Part 1: Storage Applications; and SP 800-38B, Recommendation for Block-Cipher Modes of Operation: The CMAC Mode for Authentication).
Security tools such as the National Vulnerability Database have become highly regarded international resources that are critical to modern security engineering. These CSD activities are singular and are setting global standards for quality. Their value and success have led others to create derivative products (sometimes not as high quality) that may confuse users about the source of the products. Accordingly, NIST and the CSD should protect their brand in this space as much as possible. For example, developing quantitative analysis techniques and tools that define and estimate metrics from the NVD would be one way to add value and to further NIST’s branding in this area.
The merit and caliber of the division’s basic R&D activities are comparable to those of its peer, mission-oriented government laboratories such as those operated by the Department of Energy, Department of Defense (DoD), National Aeronautics and Space Administration, and National Institutes of Health. R&D activities of the division that are most closely aligned with current and future NIST mission deliverables include work in biometrics, cloud computing, virtualization, secure boot processes, test coverage efficiency, and mobile platforms. Those activities are of high quality and are comparable to applied academic, industrial, and government laboratory efforts.
There appears to be solid alignment between the division’s R&D efforts and its mission-critical deliverables. When a research project is being proposed and created, it will be leading edge to some degree—ahead of the specific standards and guidelines that it is intended to support. Such is inevitably the case: research has to run far enough ahead of the work on standards and guidelines to provide meaningful technical input. However, this constraint means that any given research program may prove to be inapplicable to technical standards, either because the research does not succeed or because the directions of the market, standards, and technology diverge from the direction of the research effort. As a general principle, given constraints on resources and the dynamic nature of IT security technology, the division should be mindful of the relevance of its research projects to the remainder of its mission and should be willing to sunset projects in those cases in which the project has begun to achieve industrial or commercial success or the focus of the project has diverged from the mainstream direction of information technology or from the division’s work on standards and guidelines. The Role Based Access Control Program appears to have achieved a measure of industrial success and is perhaps a candidate for handing off to industry.
A recommended new initiative for the division would be to undertake a formal collaboration with the Software and Systems Division in the area of secure software development. Application-level vulnerabilities are a growing focus of hostile attacks in both locally hosted and cloud computing environments. It is important that NIST bring together its resources in security and software development to contribute in this area and to prepare for the development of standards and guidelines that will help to raise the level of the nation’s security in this important area. The ITL should consider creating a collaborative effort between the CSD and the SSD that would be responsible for the creation of standards and guidelines on secure software and enterprise-level system development for use by government, industry, and academia. The CSD is also encouraged to investigate, as appropriate, NIST’s role in metrics and guidelines for privacy, a subject that was not specifically presented to the panel.
In sum, this well-managed division has a depth of expertise that contributes to task forces defending the United States against advanced cyberthreats. This contribution is particularly important given the ongoing national and international attention to the
cybersecurity challenge. Overall, staff in this division are spread thin, but they seem to be juggling priorities well. The morale in the division appears to be quite high, creating a positive work environment that has attracted top talent. These two facts—thinly spread resources and a history of attracting top talent—suggest that the division should consider hiring as a priority.
Organization and Operation
The mission of the Information Access Division is to conduct research of performance metrics and guidelines, standards and testbeds, and community-wide evaluations in order to support the development of advanced “technologies for intuitive, efficient, and effective access, manipulation, analysis and exchange of complex information.”25 The division’s projected benefits of its work include the expedited development of these technologies and their transition into the commercial marketplace and into government applications.
The IAD consists of five major groups: Retrieval, Image, Multimodal Information, Visualization and Usability, and Digital Media. The IAD also pursues four crosscutting program areas: biometrics technology, human language technology, human-system interaction, and multimedia technology. The division’s staff consists of 66 permanent employees, 12 contractors, 14 guest researchers, and 7 students—the IAD is the largest division in the ITL in terms of staffing.
The educational background of the personnel is roughly as follows: one-half has master’s degrees, and one-quarter each has doctoral or bachelor’s degrees. The range of technical activities covered by these five groups is diverse, yet they can be generally described and broadly grouped into types of activities as follows.
- Biometrics: The use of biometric features for personal identification, particularly related to face, fingerprint, iris, voice, and, more recently, multimodality, which uses a combination of features for the same purpose;
- Retrieval (and search): The retrieval of information in the form of image, text, video, and three-dimensional objects, in response to user queries;
- Human language technology: Community-wide technology evaluations that assess progress in speech and language for text analysis, machine translation, speech analysis, and speaker and language recognition;
- Visualization and usability: The study of usability by humans related to specific implements and systems such as electronic voting machines, health information records, and biometric devices; and
- Multimedia technology: Evaluations, standards, and challenge problems in video analysis, digital data preservation, three-dimensional imaging, and biomedical imaging.
25 Curt Barker and Leslie Collica, ITL and IAD, “Information Access Division (IAD),” presentation to the panel, Gaithersburg, Maryland, March 21, 2011, p. 5.
The biometrics and retrieval activities enjoy a long track record of accomplishments. The usability initiative is emerging and showing the potential of important impacts. The video analytics, multimedia event detection, and the retail surveillance projects exemplify emerging projects of smaller scales.
The funding structure of the division is unique among ITL divisions. Sixty percent of the division’s budget comes from OA funding (other agencies, such as from the DHS, DoD, FBI, and Intelligence Advanced Research Projects Activity), which is mostly security-related. Only about a quarter of the budget is classified as IAD base funding, and less than 15 percent is from other sources at NIST.
The output of the work in the division includes the following:
- Standards and guidelines: Defining the best practice along a specified dimension or aspect of the relevant technology development;
- Data repository: Acquiring, verifying, organizing, and supporting the various important standard data sets required for developing and evaluating the relevant technologies;
- Evaluation and benchmark results: Conducting and reporting impartial community-wide evaluation results to help gauge the progress in the relevant technology;
- Outreach: Organizing workshops (some, e.g., the Text Retrieval Conference [TREC],26 have been going on annually for almost 20 years) to present findings of the technology evaluations that bring together people from academia, industry, and government; hosting students and guest researchers; and
- Reports, publications, and software: Publishing and archiving relevant results and reports; providing benchmark software (e.g., fingerprint image quality and matching).
Division Accomplishments and Observations by the Panel
The IAD is responsible for meeting a number of high-value and critical national priorities and mandates. Several groups in the division are uniquely qualified to address these mandates. Many scientists in the IAD continue to play a major role commendably. Examples of these mandates include the USA PATRIOT Act, the Enhanced Border Security and Visa Entry Reform Act of 2002, the Help America Vote Act of 2002, and the Health Information Technology for Economic and Clinical Health Act of 2009.
Within the division’s charter, the groups demonstrate a high level of technical competence in conducting long-standing, large-scale technology evaluations in critical technologies that are extremely important to the government, business, and academia, not only in the United States but worldwide. Examples include open community evaluations, such as TREC, and large-scale biometric testing (e.g., fingerprint, face, iris, and speech). Given the staff’s expertise and its neutral role, NIST evaluations conducted by the IAD are considered fair and beneficial to the sponsors and to technology developers, and they foster research in academia.
26 TREC is a forum for evaluating technologies for detecting, searching, and retrieving query text in a large set of documents.
The division appears to take a proactive role in identifying new directions for pursuing its mission and is adapting well to meet the national mandates and progress in technology. Examples include the progression of the Retrieval Group from the retrieval of spoken documents in TREC to video retrieval in TRECVid (TREC for video “documents”). The Image Group started out mainly in fingerprint matching and evaluation, but it now has broadened its expertise and conducts large-scale evaluations in face, iris, and multimodal biometrics. Examples of some other new initiatives include the evaluation of computerized tomography (CT) lung image analysis and usability studies for biometrics.
The IAD demonstrates unique capabilities in establishing biometric data standards and large-scale performance evaluation benchmarks.
Following are the recommendations of the panel with respect to the Information Access Division:
- The Information Access Division supports the development of technologies and their transition into the commercial marketplace as well as government applications. The division currently relies on substantial and sustained amounts of other agency (OA) funding (approximately 60 percent of IAD funding). Most of the OA funding is security-related (from the Department of Homeland Security, the Department of Defense, the Federal Bureau of Investigation, and the Intelligence Advanced Research Projects Activity). The reports, standards, and evaluation studies of the IAD are closely followed by academia and industry. In light of increasing foreign dominance of the biometric industry, IAD’s reliance on OA funding, and IAD’s work in support of biometrics technology development, it is important that the IAD and the ITL remain mindful of the NIST mission to promote U.S. innovation and industrial competitiveness, and so IAD efforts should continue to place highest priority on the needs of the nation’s commerce even while pursuing activities involving international sponsors.
- The IAD should establish a long-term vision and identify some fundamental research problems (e.g., relevant measurement science) that it determines should be pursued in addition to the ongoing activities.
- The Image Group should highlight its work in biometrics and differentiate its work from that of other groups in the IAD that also deal with images.
- The NIST administration should initiate a review of the interagency negotiation process, joined by the individual and relevant external agencies if necessary, to reduce the unnecessary administrative impediments that may adversely affect the technical progress and jeopardize future contracts.
- The ITL should work with the NIST administration, and possibly the policy-defining organization, that is, the Department of Commerce, in reviewing the institutional approval process related to the Institutional Review Board (IRB) to maximize efficiency and minimize unnecessary latency.
- The IAD should enhance outreach and external interaction as well as personnel professional development. For example, a NIST-wide internship
program at the graduate level with a tracking mechanism may prove helpful in raising the institutional image, the promotion of IAD activities, and the recruitment of new talents.
7. The IAD should review its recruiting policy and practice so as to ensure a healthy influx of new and young talent.
The mission of the Software and Systems Division is “to accelerate the development and adoption of correct, reliable, testable software leading to increased trust and confidence in deployed software, systems, and products.”27 Four research groups comprise the SSD: Software Components, Information Systems, Interoperability, and Electronic Information. The SSD recently hired a new division chief and two managers. The division staff performs high-quality work and is having good impact on a number of high-visibility, national priority projects.
On the question of the degree to which the division’s programs in measurement, standards, and services achieve their objectives: The SSD work to date in software testing and conformance is strong, although the SSD should look to the future and position the division to get measurement, standards, and data sets out ahead of urgent needs. Regarding the overall merits and caliber of the program: The SSD efforts in niche areas are of unquestioned leadership, and in general the quality of work is comparable to that of its peers in any other government or private organization. In addition, inter- and intralaboratory collaborations are a key strength of several members of the SSD group. On the question of the alignment of division R&D efforts with its mission, the work in the SSD is well aligned with the missions of the SSD, the ITL, and NIST; however, application-level projects could be better leveraged such that the methods, techniques, and tools developed for one project are readily available for reuse in new projects. It is, of course, clear that customer needs must be met, but a continuing focus on the potential generalizability of the technologies developed within the context of a particular project will ensure that the SSD staff is ready to take on new projects more quickly and effectively.
Formal Performance Metrics
Foundational work can be challenging to measure objectively. One possible method for measuring the external impact of foundational work might be through applications that are built on the core methods and techniques. For example, the merits of software testing could be measured by assessing the improvements in voting systems subsequent to their having undergone rigorous NIST-based testing. Measuring applications should lead to increased emphasis on customer-oriented metrics. For example, Software Assurance Metrics and Tool Evaluation (SAMATE)-related improvement in static analysis tools (perhaps as measured at Static Analysis Tool Exposition [SATE] events) and overall quality of code (perhaps as measured by vendors of software testing tools) could augment more internal, process, or local metrics.
27 Ram D. Sriram, SSD, “Software and Systems Division,” presentation to the panel, Gaithersburg, Maryland, March 21, 2011, p. 8.
The new SSD leadership is in the early stages of developing a strategic plan. To accelerate the process, the leadership identified an outside expert who will work with it to help define goals and objectives. As part of developing the strategic plan, the group should re-examine its mission statement and should consider the relative benefits of aligning the SSD technical organizational structure with foundational methods rather than with specific applications. As projects come and go, the organization will thus be better positioned to bring together the appropriate expertise for new projects as they arise. Once such a new organizational structure is in place, it should become clear where there are gaps in expertise, and additional formally trained individuals in gap areas should be hired.
Externally mandated and funded projects are subject to sunsetting, and the team should be ready to let go of those application areas that are no longer priorities. Application-level projects are important opportunities to fulfill SSD’s mission, but it is important that the division continue to build the core approaches, methods, and technologies particularly so that, as new application-level projects arise, the group is well positioned to address them.
Reference Data Sets
A core role of the ITL is to generate and curate reference data sets. Although this role requires significant effort, other ITL divisions have been highly successful in such efforts. For example, the ITL work on the TREC project has been exemplary, and it would serve the SSD well if it were also to take a lead in generating, curating, and maintaining reference data sets, including, if possible, those in the health domain.
SSD Research Groups
Software Components Group
The SSD should consider technology developments that have occurred and are likely to occur in the first part of the 21st century, and it should reevaluate the activities of the Software Components Group. The National Software Reference Library (NSRL) for software for mobile devices is a good new effort. Additional energy should be applied to improving and speeding up the process of generating and distributing the reference library given the new, and now much more common, models of commercial software distribution. Examples include auto-updates and application downloads, software as a service, and cloud computing.
The software assurance projects such as SAMATE, SATE, and the SAMATE Reference Data Set (SRD) are solid, appropriate SSD projects. More metrics related to external impact should be used to evaluate the (likely very positive) impact of these projects. In particular, the SSD could quantify impact by comparing old and new tools against the same code base, and possibly it should gather statistics from tool vendors on
industry-wide improvements in software quality. On the SRD in particular, the SRD code snippets are too small (100 lines of code [LOC]). The SSD should add larger code examples—perhaps some from SATE—into the SRD (e.g., Emacs [~300K LOC C], Chrome [~3M LOC], BSD network stack).
Information Systems Group
The voting work being conducted by the Information Systems Group is moving apace. The group should consider the implications of the completion of the voting project, given that the Election Assistance Commission may sunset this project. SSD staff should aggressively publish on this work so that it, and, more generally, NIST, receive appropriate credit for their contributions.
On bioinformatics and bioimaging, the group is collaborating well with other laboratories at NIST. The SSD should capture best practices here, and they should be replicated across the ITL. The medical imaging work has the potential for impact by advancing the quality and consistency of measurement in clinical use. However, this work and other SSD efforts in the biosciences are less well aligned with the SSD mission than are some other SSD projects. If the SSD wishes to build strength here, it should clarify how this project contributes to its core foundational work and then recruit domain leadership in that area.
Test methods developed by the SSD for meaningful use certification have been somewhat helpful to the Electronic Health Record (EHR) vendor community and the Office of the National Coordinator for Health Information Technology (HIT). (This work allows EHR vendors to appear on the Certified HIT Product List.) As the funding from the American Recovery and Reinvestment Act of 2009 (Public Law 111-5) comes to an end, work on testing infrastructure for health IT has the potential for use in new application areas that the Interoperability Group should consider. The energy devoted by the group to “tools for tools” is well placed. It is helpful for NIST to bridge and help bring consistency across various interoperability groups (e.g., the Nationwide Health Information Network [NwHIN] and Integrating the Healthcare Enterprise [IHE]); it is nevertheless important to maintain both the perception and the reality of NIST neutrality, particularly in the IHE involvement.
Electronic Information Group
The smart grid framework and roadmap work is a contribution to the community. The specific work on 1588 time synchronization is interesting. The framework and roadmap document for the smart grid released as a draft is likely to have positive impact on the smart grid community and future industrial development.
The Statistical Engineering Division has a 65-year history of consistent and fundamental contributions to the central mission of NIST through the development and application of statistical methodology for metrology. This methodology supports the basic NIST activities of producing measurements and quantifying their associated uncertainties.
The SED advances its mission of supporting research in measurement science, technology, and the production of standard reference materials (SRMs) through numerous scientific collaborations within NIST and externally. The SED conducts statistical research and provides important training and educational activities within and beyond NIST. It serves as a unique national and international resource for the metrology and standards communities and more broadly in high-profile contexts where an acknowledged impartial broker of data analysis and interpretation is needed.
The SED is composed of two groups: one at the NIST campus in Gaithersburg, Maryland, and the other at the NIST campus in Boulder, Colorado. The Gaithersburg group has 20 technical staff members plus a support secretary; the Boulder group has 3 technical staff members and a secretary.
Achievement of Stated Objectives
The SED mission statement is that the SED “seeks to contribute to research in information technology, to catalyze scientific and industrial experimentation, and to improve communication of research results by working collaboratively with, and developing effective statistical methods for, NIST scientists and our partners in industry.”28 The division is highly successful in achieving these objectives outside of the ITL, where its contributions (inside and outside of NIST) are highly regarded, effective, and visible. The management and staff of the division also believe that there are additional opportunities for genuinely substantial contributions to existing efforts inside the ITL (as personnel limitations allow).
Inside NIST, the SED provides essential support for basic metrology efforts (for example, in the production of standard reference materials). The division’s personnel also have deep collaborations with NIST scientists in both traditional and emerging areas of science and technology, ranging from neutron physics, to the large-scale measurement of greenhouse gas concentrations, to basic signal processing, to the mapping of return values of extreme wind gusts. The division’s lead role in the new NIST (Innovations in Measurement Science) Shape Metrology Program is evidence of the quality of its work and effectiveness in NIST collaborations.
Outside of NIST, the SED responds on an emergency and ad hoc basis to national and international needs, is engaged on an ongoing basis with important U.S. researchers and agencies, and contributes materially to international metrology groups. Recent examples of important externally focused work of these types include the development of an oil budget calculator for the Deepwater Horizon oil spill; work with the Office of Law Enforcement Standards on body armor performance; collaborations with medical researchers on computer tomography and optical medical imaging; long-term work with
the U.S. Departments of Justice and Homeland Security and the Office of National Drug Control Policy; membership on technical committees of the International Organization for Standardization (ISO); contribution of basic material to new ISO standards development; and invited participation in important international workshops on mathematics and metrology.
The SED makes substantial educational contributions both inside and outside NIST and effectively disseminates sound statistical methodology for metrology. Division personnel are active in the development and delivery of metrology-related statistics short courses from Gaithersburg to South America and Europe. The division maintains and has recently updated the online Engineering Statistics Handbook,29and it develops and disseminates specialized software supporting best practices in statistical metrology. It has an excellent record of refereed publications, both papers done jointly with subject-matter scientists in fields of collaboration, and publications in the statistical (and statistical metrology) literature.
The most obvious opportunity for the SED to have additional impact is within the ITL itself. If the fairly severe need for a larger PhD-level staff can be alleviated, a number of emphases in the ITL could be catalyzed by more involvement of the SED. These possibilities include work in data management and visualization, work in pattern recognition and machine learning, and the laboratory’s work in forensics. The SED must be a central contributor to NIST work in forensics to the extent that the Daubert requirement30 for the evaluation and statement of potential error rates is to be upheld. The situation is not unlike SED’s role in developing quantifications of measurement uncertainties. While there is already some division activity in this area, notably in the National Ballistics Comparison Project, much more is possible and needed as personnel are available.
Technical Merits and Caliber of Work
The SED work is technically excellent and central to the overall mission of NIST. The division provides state-of-the-art support for traditional NIST activities such as the production of standard reference materials, calibration studies, and assessments of interlaboratory studies. It is an innovative partner in the development of complicated emerging measurement technologies like high-speed opto-electronic measurement. Its external reputation for excellence is easily established through a consideration of its solicitation to work on high-profile, pressing national and international issues such as the Deepwater Horizon oil spill disaster, the estimation of Colombian coca cultivation, and law enforcement body armor reliability; its steady stream of internal NIST and external awards (including three Department of Commerce Gold Medals in 2010 and the Youden Award from the American Statistical Association in 2009); its record of peer-reviewed publications in statistics, metrology, and basic sciences (which are strong in terms of both the venues used for dissemination and the number of papers published); and invitations to
30 The reference is to the 1993 Daubert v. Merrell Dow Pharmaceuticals, Inc., U.S. Supreme Court decision that set current rules and requirements for the admissibility of expert scientific testimony in federal courts. Among other things it requires that error rates of methods used to support expert scientific testimony be known and stated.
international service, both on society committees and in the providing of training in statistical best practice.
Alignment of Research with Laboratory Services and Deliverables
The alignment of the SED research with ITL services and deliverables is excellent. The SED projects and participations fully contribute to the ITL roles of advancing science and engineering, setting standards and requirements for unique scientific instrumentation and experiments, data, and communications. It is the very nature of good statistical research that it is driven by the real data-analysis needs of a substantive field. The SED research is excellent and thus both inherently and effectively collaborative in nature (inside and outside NIST).
Developments Since the 2007 and 2009 NRC Assessments
The SED management continues to be excellent and gaining momentum. It is proactive in finding and defining important projects and in aligning the division’s work with ITL and NIST priorities. It is proactive in marshaling appropriate funding and promoting international linkages. The leadership is technically active and effective, and the group morale appears to be high. The number and visibility of projects (including the new NIST Innovations in Measurement Science project) are increasing.
Two strong hires have been made since the previous NRC review, but two compensating retirements have prevented the growth in the number of PhDs strongly recommended in both the 200731 and 200932 reviews. There have been no new partnerships established with academicians and no division postdoctoral researchers since the 2009 review.
Increasing the number of PhDs in the SED is a critical need (identified, as noted above, in the 2007 and 2009 NRC reviews). This is particularly true in light of potential retirements and the exponential explosion of the complexity of statistical and measurement methodologies, and of NIST needs (related, e.g., to forensics work). At least one hire per year for the next 5 years is needed in order to cover both imminent retirements and any other attrition and to provide for growth of at least a 2 to 3 full-time-equivalent staff in the next 5 years in order to handle the increasing complexity and volume of SED involvement in important new initiatives and areas. In addition, an increase in visiting faculty and other scientists is needed to help foster long-term external collaborations. More than one-time events and/or visits and a much broader scope here would allow the division to leverage its trove of excellent problems and opportunities to
31 National Research Council, An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2007. Washington, D.C.: The National Academies Press, 2007.
32 National Research Council, An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National Academies Press, 2009.
increase not only its own output, but the quality and relevance of U.S. academic statistical research.
For the direct development of the SED staff, increased flexibility, support, and opportunity for visits by SED personnel to academic and non-academic institutions to establish long-term collaboration would be beneficial. So also would increased funding opportunities and support for participation in conferences and other professional statistical activities. The latter is important not only for the development of existing staff, but also to make the division attractive to the best new potential hires.
Opportunities for the Future
Overall, the situation in the SED is quite healthy. Important and fascinating opportunities outstrip the division’s present physical ability to respond. There is much important work going on within the ITL that could genuinely benefit from increased SED involvement and collaboration. If the small group in Boulder were larger, there would be the opportunity to collaborate with scientists at the National Center for Atmospheric Research and the National Oceanic and Atmospheric Administration. More could be done in the way of participation on editorial boards and the organization of sessions (at both statistical and metrological meetings). More could also be done to raise the SED profile in the statistics community. The division has access to some of the most interesting statistical problems in the world, and it does some of the most interesting physical-science-related work in the world. Statisticians broadly should know more about opportunities and the work at NIST.