An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Years 2004-2005 (2005)

The Information Technology Laboratory (ITL) has responsibility for information technology, telecommunications, mathematics, and statistics. The laboratory staff is organized in six divisions, as shown in Appendix A:

• Mathematical and Computational Sciences Division (MCSD),

• Advanced Network Technologies Division (ANTD),

• Computer Security Division (CSD),

• Information Access Division (IAD),

• Software Diagnostics and Conformance Testing Division (SDCTD), and

• Statistical Engineering Division (SED).

Appendix A also presents the staffing trends for the laboratory (see Figure A.5).

The Information Technology Laboratory is a vital resource, both to other NIST laboratories and directly to the United States and the world. It has an outstanding program—a conclusion based on six factors:

1. The work of many individual scientists in ITL is excellent.

2. Some of the work (for example, in security testing) has already had a major impact on formal national and international standards as well as standards agreed to by vendors across industry boundaries. Specific examples of standards impact are given by way of illustration throughout this chapter. Much of the work of ITL has the potential for making a significant difference in global commerce, and the Board expects that other work will likely have a similar impact on standards.

3. The ITL uses many mechanisms to disseminate its work effectively, including formal standards bodies, professional collaborations, technical papers, workshops, conferences, software, Web-based tools, and collaboration with other government agencies.

4. The disciplines in ITL are at the heart of major U.S. initiatives, including homeland security, electronic voting, and the supporting information systems for health care.

5. Two ITL divisions (the Mathematical and Computational Sciences Division and the Statistical Engineering Division) provide fundamental tools and expertise for many measurement, standards, and technology activities throughout NIST. All are a fundamental part of programs across NIST or with other government agencies. Thus, ITL scientists are enablers for other NIST programs as well as being direct contributors to information technology (IT)-specific activities.

6. The Board anticipates significant impact from the work reviewed during this assessment period based on its excellence and alignment with key needs and on a track record of impact from previous work.

Different laboratories have different missions and different pressures. Currently, ITL is active in a large number of mandated programs (principally in the areas of homeland security and information security) that constrain its activities in ways that university laboratories or some foreign government research laboratories, for example, are not constrained. Recognizing these limitations, ITL compares favorably with top-ranked U.S. government laboratories.

This conclusion does not apply to every ITL project reviewed. In any large research laboratory there will inevitably be varying levels of success across projects and some projects that are viewed as questionable by some reviewers; if this is not the case, the laboratory is not including enough higher-risk projects in its portfolio. In addition, no other laboratory used for comparison has a mission with a primary emphasis on standards and metrology, and this difference in mission must also be taken into account.

The ITL portfolio of projects has wide range and distribution. The Board believes that ITL compares favorably with other government laboratories. The primary concern of the Board comes in part from the success of ITL. Because information technology is at the heart of so many vital changes in our world, and because ITL has been very responsive to needs from Homeland Security, Help America Vote, and Electronic Health Records activities, for example, ITL finds itself at the heart of these and other mandated government initiatives. This role has required ITL to put more and more of its resources into increasingly short-term requirements. The Board is concerned that this shift will undermine the ability of ITL to anticipate future needs of both industry and government. Additional longer-term, stable funding would enable ITL to look at some longer-range areas and to backfill some skill areas that have become very thin or nonexistent during a multiyear period of declining funding.

In spite of the pressures of short-term, mandated programs and the challenges of low funding, morale seems to be very high in the laboratory. The recent NIST Employee Survey supports this observation, placing ITL first or second among the NIST laboratories in 10 of the 15 categories surveyed, and above the median in all categories.

Rather than covering all of the programs and projects within ITL, the discussions below provide examples within the assessment categories (technical quality and merit, relevance, effectiveness, and resources). The Board has highlighted a small number of programs that demonstrate the nature of the work in ITL. In many instances, a particular program could have been reiterated within each assessment category. However, because these are illustrations rather than reviews of all programs and projects, discussion of the same project is not repeated across categories. Hence, lack of mention of a particular program or project is not an indication that the work was somehow considered less important than those mentioned.

The Information Technology Laboratory ranks with the best of the U.S. government laboratories in the quality and merit of its technical work. The technical quality of the work is uniformly very high across all six divisions.

Four ITL scientists were recognized with the Department of Commerce Gold Medal for their work in Smart Cards. Along with two colleagues from the Manufacturing Engineering Laboratory, two ITL scientists were awarded the Silver Medal for their work on Two Dimensional Grid Standard Reference Material. Seven scientists from four different projects were awarded Bronze Medals. One staff member received the NIST 2004 Allen V. Astin Award for advancement in measurement technology. The long list of staff publications reflects the quality of work and the focus on standards groups both nationally and internationally. Outside awards to ITL staff members (e.g., appointments as an Institute of Electrical and Electronics Engineers fellow and an American Society for Quality fellow, receipt of the Technology Review Top 5 Patent from the Massachusetts Institute of Technology, several awards from the International Committee for Information Technology Standards, and selection to the National Academy of Engineering) further speak to the strong technical work of the laboratory.

The projects reviewed generally evinced high technical quality because of the caliber of the scientists, the significant accomplishments in the work, and the collaboration with other scientists enabling breakthrough work that could not be done in isolation. Following is a discussion of examples in quantum computing, the digital mathematical library, statistical key comparisons, digital health care, and computer security. These projects, selected from many, typify the highly collaborative work of ITL and demonstrate very different examples of the excellent work.

The quantum computing work is perhaps the highest-risk area of projects reviewed by the Board in that there is a long way to go before practical implementation, let alone standardization of quantum computing. Generally, ITL work in this area is a collection of individual projects functioning under a large umbrella. At some point, it would be important for ITL to have an interdisciplinary team review the components in order to establish unity of purpose and goals. Nevertheless, the technical work is outstanding. It is carried out with Defense Advanced Research Projects Agency (DARPA) funding, in collaboration with the NIST Physics Laboratory (PL) and Electronics and Electrical Engineering Laboratory (EEEL), and it cuts across multiple divisions of ITL. For example, an ITL staff member was a key contributor to a recent series of landmark NIST experiments demonstrating key steps for quantum information processing in ion systems: teleportation (Barrett et al., 2004) and the semiclassical Fourier transform (Chiaverini et al., 2005).

The work by ITL on the Quantum Information Systems project involves the Mathematical and Computational Sciences Division, Advanced Network Technologies Division, and Computer Security Division, working with scientists from PL and EEEL, with partial support from DARPA. The goal is to advance the science in quantum computing, leading to the ultimate goal of creating a useful quantum computer, and to develop secure quantum communications systems.

Recent results showed a 3 percent failure rate of quantum gates, which is several hundred times larger than the rate scientists had generally thought necessary to produce useful results. But ITL mathematicians have shown that even with this high error rate, a new fault-tolerant architecture can produce reliable and useful results. A NIST researcher cautioned that while this work reduced the gap between theory and practical reality, showing that quantum computing may be easier than had been thought, it will take a lot of work to build a useful quantum computer.

In quantum communications, two important steps have been taken: (1) network scientists from ANTD, in collaboration with PL colleagues, have achieved a quantum key exchange rate of 1.0 mega-

bits per second, the highest rate achieved over a free-space quantum link; and (2) CSD scientists have shown vulnerabilities in the proposed quantum key distribution protocols and are working on remedies for these.

The ITL has been addressing the move to digital health care systems. In 2003, spending for health care in the United States was $1.7 trillion, according to a 2004 study from the Centers for Medicare and Medicaid Services (Smith et al., 2005). The Institute of Medicine reported that 44,000 to 98,000 patients die each year from preventable medical errors (IOM, 2000). Subsequent research has found that patients are at the highest risk for medical error when they are prescribed and administered medication and when their care is transferred from one provider or facility to another.

Information technology provides a great deal of potential for decreasing error rates in both of these situations, as well as for reducing costs and increasing access to specialty care. Some of the relevant technologies include the following:

• Electronic health records (EHRs),

• Computerized physician order entry (CPOE) systems,

• Document management systems that enhance information sharing among health care providers,

• Wireless pervasive computing devices on a local or body area network,

• Telehealth clinical specialty applications,

• Digital medical imaging, and

• Collaborative decision-support tools at the point of care.

Both CPOE systems and EHRs rely on the Health Level (HL)7 standards effort with which ITL is heavily involved. CPOE systems are used specifically for pharmacy orders; they can reduce or eliminate errors related to illegible handwriting, easily confused drug names, and other factors. In addition, the emerging environment must be secure and reliable, it must protect privacy, and it must be able to interoperate across a wide range of independent facilities.

The ITL team has formed a crosscutting initiative looking at technology and standards issues across all of the ITL divisions. This is an excellent effort addressing an important problem area that requires both the technical and standards focus that NIST ITL can uniquely provide.

The individual components of the work in technology that supports health care are excellent. For example, the work on networking standards focused on performance metrics (throughput, delay, jitter, loss) in realistic medical scenarios in which there is a requirement for communication across disparate medical devices. The statisticians were also engaged in this part of the project. The security scientists were concerned with the protection of electronic records against various threat scenarios, assuring the ability to achieve reliable results while meeting the privacy standards of the Health Insurance Portability and Accountability Act of 1996 (HIPAA). The software group addressed the conformance testing and standards needed to share documents across multiple environments. The work described here represents only a sample of the health care initiative.

While this work has a framework that ties the projects together, today it is primarily a loose consortium of projects with a common motivation. This is an appropriate place to start, creating interest and motivation for work across the divisions of ITL. Ultimately, for maximum success, an integrated project structure will be needed in order to ensure best results not only for the components but for the whole.

The Digital Library of Mathematical Functions (DLMF) is another long-term project, but it involves much lower risk than that for quantum computing. Here the goal is to replace the NIST best seller, Abramowitz and Stegun’s Handbook of Mathematical Functions (first published in 1964), one of the most widely used mathematics books in the world but now extremely out of date. Rather than simply

updating the volume, MCSD has chosen to make advanced use of extensible markup language/math markup language (XML/MathML) software capabilities. The DLMF will have a substantial network of hyperlinked and cross-referenced formulas. In carrying out the project, ITL scientists have drawn on researchers from around the world for writing, reviewing, and testing new chapters. This work will be an important resource for scientific computation done around the world. The cutting-edge use of MathML for presenting mathematical formulas, the best existing example of this standard, is impressive.

Statistical methodology is at the heart of reliable and comparable measurements and standards, and hence it is central to the mission of NIST. Through the work carried out in one project, the international Mutual Recognition Arrangement (MRA) signed by the United States and more than 80 other nations established an official international policy for recognition of equivalence of weights and measures. This project was implemented through NIST and the National Metrology Institutes (NMIs) of other countries. The ITL is working with scientists across NIST and throughout the world developing and applying statistical methodologies for the key comparisons that implement the MRA and support the statements of NMIs’ certified measurement capabilities that are fundamental to world trade.

The Computer Security Division’s programs in security testing are outstanding. The Cryptographic Module Validation Program (CMVP) and Crypto-algorithm Validation Program (CAVP) are models that analogous laboratories around the world emulate. This is evidenced by the adoption (or planned adoption) of Federal Information Processing Standard (FIPS) 140-2 procedures by the United Kingdom and possible expansion to Korea, France, and Germany, and by ongoing work to incorporate FIPS 140-2 into the Common Criteria. These programs find a high rate of security flaws (20 to 50 percent in CMVP, depending on the previous experience of the submitter), and they find about 30 percent of the products to be nonconformant (via CAVP). Finding these problems before hackers do is a significant accomplishment. This excellent work will have impact on the increasingly important area of computer security.

A factor in evaluating the overall quality and merit of a laboratory is the balance of the work in the context of the laboratory mission. That involves asking whether there is an appropriate mix of short- and long-range projects and a sufficient number of high-risk, high-payoff projects in the portfolio to provide insurance for the future. The ITL displays good balance between short- and long-term projects, although it has moved toward more short-term projects over the years (this varies by division). “Long-term” is not synonymous with “high-risk” (for example, the Digital Library of Mathematical Functions is very long term and involves a large number of external people, but it is on a very clear path).

The majority of the short-term work arises in mandated programs and has external funding. The increased reliance on this funding and ITL’s central role in many mandates (homeland security, health care, and electronic voting, for example) is slanting the laboratory toward a narrow range of issues. The extent to which federal mandates may be enervating the divisions by refocusing work on security is an issue to be watched carefully by laboratory management in the future. Another danger of excessive short-term focus is that it could ultimately undermine the ability to anticipate new issues that will arise.

It is also necessary to ask whether the work is sufficiently distinct from work done elsewhere. Because of the unique NIST mission in standards and metrology, it generally makes little sense for NIST to use its scarce resources to compete with research work done elsewhere. Generally during this assessment period, ITL’s projects were either part of mandated work (e.g., related to homeland security); were essential to standards and metrology (e.g., involved Domain Name System [DNS] security); undergirded fundamental measurement science (the work of the Statistics Division fits in this category, as the scientists define the measurement and validation practices across NIST); or were sufficiently distinct from outside work. Of the projects reviewed, only in quantum computing does the Board

question the role of NIST, not because of the caliber of the work, but because this work is still in the fundamental science realm and is a long way from the issues of standardization.

The relevance of the efforts of ITL involved both its customer focus and its responsiveness. The laboratory has been responsive to many initiatives within NIST—homeland security, voting, World Trade Center analysis, and health care, to name a few. Relevant work on these initiatives has been carried out by all of the ITL divisions. As stated above, the laboratory may be becoming overwhelmed by mandated (and often underfunded) work, to the detriment of maintaining a stable, long-term laboratory able to respond to future as well as present needs.

For example, the Statistical Engineering Division statisticians have collaborated with scientists in the Building and Fire Research Laboratory in an analysis of the collapse of the Twin Towers at the World Trade Center. The SED has provided the statistical design for experiments and model simulations to discover the sequence of failure events and to determine the relative importance of the several failure modes (impact, fire, heat, structural elements). This one of many examples underscores a common theme in ITL. Since computing, mathematical modeling, and statistical analysis are fundamental to all areas of science today, ITL plays an important role in providing expert collaboration throughout NIST. This work is aided by software tools and training that the staff does for other parts of NIST. In turn, this keeps the ITL scientists abreast of key developing needs and feeds back into programs in ITL. The ITL staff does this so well that the contribution they make throughout NIST on other projects is sometimes taken for granted and underappreciated.

Relevance in ITL is not limited to those areas in which the laboratory responds to outside requests, however. The way that its scientists seek out key connections across all of ITL’s programs continues to be impressive. There is a notable and growing strategic nature to these connections. Any scientist can find someone interested in his or her work; it is another thing to be tied in to the customers whose use of the results can have high impact. This latter attitude has become ingrained in the thinking of ITL to the point that many of the ITL publications are in journals of interest to the customer, and very few projects are taken on with no sponsor. By necessity, this has meant that work has become more short term simply because it is focused on a defined problem. ITL should push back the boundaries to better anticipate future needs.

An example of this is the First Responders program in ANTD. During the events of September 11, 2001 (9/11), there was difficulty in linking all of the messages from the disparate response communications because of issues of wireless, localization, interference, and ad hoc networks. The network scientists have been developing testing methodologies and standards enabling future interconnection of a range of devices and networks.

Following voting irregularities in the 2000 elections, there has been a strong push to solve the problem with technology. The ITL sponsored the First Symposium on Building Trust and Confidence in Voting Systems, looking at a variety of ways to support electronic voting. ITL scientists have been working in the areas of security, interoperability, and human factors. The group has a goal of updating the standards in early 2006. This is good work, but the scientists should broaden their scope, looking at ways of ensuring the reliability of the voting record and dealing with potential hacking in the voting system.

The ITL has demonstrated remarkable agility during this review period. Scientists in the Software Diagnostics and Conformance Testing Division show absolute mastery of short-term projects, which often have multiple external industry stakeholders with differing goals and short time frames. The

Computer Security Division has taken on many mandated initiatives following 9/11 and maintained its excellent work. Similar stories can be found in all of the divisions of ITL.

As indicated above, however, this outstanding ability is a double-edged sword. The Board is concerned about the ability of ITL to anticipate future mandates with the decline in longer-term core projects.

In evaluating the effectiveness of ITL, the Board considered the extent to which each division and project linked to other communities, both internally to NIST and externally, how well they disseminated their output, and the impact of their work.

The Software Diagnostics and Conformance Testing Division has been successful in bringing a broad community of vendors together to agree on conformable standards that enable more interoperable software products. The division’s testing tools for conformance to XML standards have the attention of vendors across a wide spectrum. More recently, SDCTD has been engaged with health care standards (HL7 and IEEE1073) that will enable improved interoperability across diverse health care providers.

The ANTD has been at the forefront in driving the Internet Engineering Task Force (IETF) work on securing the Domain Name System for a number of years. At various points the DNS work stalled, and the ANTD scientists devised technical advances, tools, and specifications needed to get the work going again. ANTD staff members have led the IETF editorship for five core DNS security specifications and achieved closure on their adoption, a monumental task directly aligned with the NIST mission for standardization and with the national focus on securing cyberspace. The ITL has applied an impressive approach to promoting this standard and moving it toward deployment—not just writing and analyzing, but developing deployment and measurement tools for the Internet Service Providers, users, and government agencies desiring to put the technology into practical use.

In addition to the dissemination of their work through standards bodies and technical papers, the ITL divisions have led the organization of workshops and symposiums that bring together the technical research community and vendors to consider key areas. An example is the Text Retrieval Conference (TREC), at which those doing research in information retrieval from large text collections are brought together by ITL leaders on an annual basis. These conferences have been growing, as measured by numbers of participants as well as by the challenges of the problems that they are undertaking. The Information Access Division has clearly established itself as the leader in this important area.

Examples of other gatherings coordinated by ITL include the Biometrics Consortium Conference, a conference on Building Trust and Confidence in Voting Systems, and the SPAM Technology Workshop. In these examples and many others, people needing the results come together with those doing the work under the direction of NIST leaders, which enables very rapid dissemination of results. In the rapidly moving ITL fields where much of the standardization takes place through de facto standards and vendor products, the ITL scientists have done a masterful job of encouraging interchange and agreement while avoiding the trap of endorsing individual vendor products.

Other ways in which ITL effectively disseminates its work include standard software. For example, MCSD and SED each play a role in disseminating high-quality software for standard tasks. Web-based dissemination of the NIST/SEMATECH e-Handbook of Statistical Methods and the future availability of the Digital Library of Mathematical Functions will provide invaluable services to the worldwide community.

Work in biometrics constitutes a major area of work in the Information Access Division. The Department of Homeland Security’s US-VISIT Program—part of a continuum of security measures that begin overseas and continue through a visitor’s arrival in and departure from the United States—

requires all U.S. visa applicants to have fingerprint and face biometrics captured and stored for U.S. records. In addition, the U.S. government is considering biometrics for a new federal Personal Identity Verification card. Further deployment of biometrics is expected for passports worldwide, if a U.S.-initiated effort succeeds. The Image Group of IAD has been involved in all of these projects for tasks including the following: compiling databases (the fingerprint databases now comprise 128 million fingerprints for 18 million people); designing test methodologies and performing tests on fingerprint, face, and other recognition products; testing and making recommendations on aspects of fingerprint size and quality; and involvement with standards (e.g., MINEX04—the Minutia Interoperability and Exchange test).

Through IAD’s work, NIST has become a recognized authority for large-scale fingerprint databases and testing and holds a respected third-party testing role for biometrics systems. Because the U.S. government is leading the world in the required use of biometrics, and NIST benchmarks are used in decision making for U.S. government procurements, IAD has become very influential in fostering improved technologies and business progress in the biometrics industry. Funding of this work comes from congressionally appropriated biometrics initiative funds and from U.S. government agencies, including the Department of Justice, the Department of State, the Department of Homeland Security, and the Central Intelligence Agency.

The ITL has been concerned for a number of years about its spatially fragmented facilities. Now there is a plan to consolidate ITL groups located at the NIST campus in Gaithersburg, Maryland. This plan will go a long way toward addressing the natural collaboration issue that is created by multiple sites. Nonetheless, some people will continue to be located at the Boulder, Colorado, campus as long as there is need to support activities there, and even on the Gaithersburg campus there will be enough physical separation that not all collaboration will happen naturally. The Board continues to urge ITL to pioneer collaborative technology, both for effective interaction and to show the way for others in overcoming the geographic distance factor.

Recently, the NIST-wide high-performance computing cluster was decommissioned. The ITL and the Physics Laboratory have taken over ownership of the system and are working with the NIST Chief Information Officer to enhance it to provide a medium-scale computational facility for use by the two laboratories. The resulting facility remains much smaller than computing facilities at other government laboratories such as the Lawrence Livermore National Laboratory and the Lawrence Berkeley National Laboratory, but the Board believes that it meets current computing needs within ITL and is sufficient to maintain the necessary level of expertise in parallel programming within ITL.

Another issue is the low bandwidth of the Internet connection between NIST and the outside world. The ANTD has initiated a pilot project with Mid-Atlantic Crossroads to link as many as 64 machines inside NIST to a regional high-speed Internet hub that will provide connections to other government agencies (e.g., National Institutes of Health), major universities in the area, and other high-speed research networks. Again, this low-budget solution seems sufficient to meet current needs of ITL, but it may need to be expanded.

There are several issues in ITL relating to human resources. With past declines in budget and the growing focus on mandated areas, there are holes in certain skill areas. Of particular concern to the Board were certain areas of statistics, mathematical optimization, and geometry. Compounding this issue of staff shortages is the projected retirement of many on the staff. There is a need to add younger staff in key skill areas both for the present and for the future health of ITL.