National Academies Press: OpenBook
« Previous: 7. Building and Fire Research Laboratory
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

8
Information Technology Laboratory

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

PANEL MEMBERS

Louise H.Trevillyan, IBM T.J.Watson Research Center, Chair

Tony Scott, General Motors Corporation, Vice Chair

Michael Angelo, Compaq Computer Corporation

Bishnu S.Atal, AT&T Laboratories Research

Matthew Bishop, University of California at Davis

Linda Branagan, Secondlook Consulting

Jaime Carbonell, Carnegie Mellon University

Aninda DasGupta, Philips Consumer Electronics

Albert M.Erisman, Institute for Business, Technology, and Ethics

John R.Gilbert, Xerox Palo Alto Research Center

Roscoe C.Giles, Boston University

Stephen T.Kent, BBN Technologies

Jon R.Kettenring, Telcordia Technologies

Catherine C.Lasser, IBM T.J.Watson Research Center

John W.McCredie, University of California at Berkeley

Vijayan N.Nair, University of Michigan

Lawrence O’Gorman, Veridicom, Inc.

David R.Oran, Cisco Systems

Thomas Parenty, Consultant, Oakland, California

Craig Partridge, BBN Technologies

K.K.Ramakrishnan, TeraOptic Networks, Inc.

William Smith, Sun Microsystems

Stephanie M.White, Long Island University

Submitted for the panel by its Chair, Louise H.Trevillyan, and its Vice Chair, Tony Scott, this assessment of the fiscal year 2001 activities of the Information Technology Laboratory is based on a site visit by the panel on February 27–28, 2001, in Gaithersburg, Md., and documents provided by the laboratory.1

1  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Information Technology Laboratory Technical Accomplishments 2000, NISTIR 6558, National Institute of Standards and Technology, Gaithersburg, Md., October 2000; U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Report to the NRC Assessment Panel for Information Technology, National Institute of Standards and Technology, Gaithersburg, Md., February 2001; U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Information Technology Laboratory Publications 2000, National Institute of Standards and Technology, Gaithersburg, Md., January 2001.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

LABORATORY-LEVEL REVIEW

Technical Merit

According to laboratory documentation, the mission of the Information Technology Laboratory (ITL) is to strengthen the U.S. economy and improve the quality of life by developing and applying technology, measurements, and standards for information technology. According to the laboratory’s strategic plan, the laboratory has two primary functions: (1) to provide the U.S. information technology industry and key information technology users with the world’s best technical infrastructure and to increase the quality of software in industry, returning the best possible value to the economy and society, and (2) to provide the best in information technology services for all of NIST, thus enabling its staff to use information technology to improve their service delivery and efficiency and therefore provide the best value to their customers.

The mission of the Information Technology Laboratory is very broad, and the programs under way no only encompass technical and standards-related activities but also provide consulting services in mathematical and statistical techniques and computing support throughout NIST. To manage this wide array of projects, the laboratory is organized into eight divisions (see Figure 8.1). The Mathematical and Computational Sciences Division is responsible for developing and disseminating analytical and computational methods for solving scientific and engineering problems. The Advanced Networking Technologies Division works with industry to address the technical challenges of an increasingly connected world; areas of emphasis include wireless communications and pervasive, optical, and multimedia networks. The Computer Security Division addresses needs in information technology security by developing standards, metrics, tests, and validation programs for security in systems and services and by acting as advisors on issues related to information technology security risks and vulnerabilities. The Information Access Division is responsible for technologies that facilitate interactions with computing devices and focuses on relevant measurement methods and standards in areas like speech and image recognition, information retrieval, and usability technologies. The Convergent Information Systems Division’s goal is to investigate the exchange, storage, and manifestation of digital content and its feasibility and scalability for integrated systems, applications, and architectures. The Information Services and Computing Division provides support for information technology capabilities throughout NIST; its responsibilities include maintaining a high-performance network, providing Web, personal computer, and help-desk facilities, and supporting central administrative applications (such as those used in the accounting, procurement, and inventory offices). The Software Diagnostics and Conformance Testing Division focuses on emerging languages and products and provides tools like conformance test suites, reference implementations, and advanced testing methodologies for standards development and compliance. The Statistical Engineering Division collaborates with industry and staff across NIST and provides guidance on experimental design, statistical modeling, and data analysis. Overall, the panel found that the programs under way in all of these divisions are appropriate and well-aligned with the laboratory and divisional missions. Examples of excellent ongoing activities and more detailed discussion of specific issues observed by the panel are discussed in detail in the divisional reports later in this chapter.

A major reorganization of two divisions occurred this past year. The Distributed Computing and Information Services Division and the High-Performance Systems and Services Division were discontinued. The many information technology support functions of the laboratory, which had been spread between those two divisions, are now consolidated in the new Information Services and Computing Division. The High-Performance Systems and Services Division group that provided visualization

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

FIGURE 8.1 Organizational structure of the Information Technology Laboratory. Listed under each division are the division’s groups.

expertise to NIST scientists has joined the Mathematical and Computational Sciences Division and is strengthening the collaborative efforts that are the focus of that division. Finally, the research functions, like work on standards for electronic books, of the High-Performance Systems and Services Division formed the core of the new Convergent Information Systems Division. The new organization provides the laboratory with a more coherent structure in which the divisions can have a tighter focus on key functions and programs can be managed more effectively with opportunities for synergies clearly highlighted. Although the full consequences of the new structure cannot yet be evaluated, the panel is very

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

enthusiastic about this reorganization, particularly about its potential for improving information technology support services at NIST. The Information Services and Computing Division is clearly starting off in the right direction by working closely with the Information Technology Services Planning Team, and the panel believes that the effort to develop an information technology architecture plan, i.e., a blueprint for information technology support, will help the division to define its roles and responsibilities.

Overall, the panel is extremely pleased with the progress made in the Information Technology Laboratory since the last assessment. Under the guidance of the (relatively) new laboratory director, the management team as a whole has become significantly stronger. The “acting” titles are gone and two new division chiefs and a new deputy laboratory director have been hired. The new strategic plan2 more clearly lays out the laboratory’s goals and responsibilities and is organized so as to tie each division very specifically to the laboratory and NIST missions. Throughout the laboratory, the focus is sharpening, and the panel observed many instances of mature projects being appropriately concluded, existing activities being merged to produce programs that are more cohesive and better aligned with the laboratory’s mission, and new projects being started in areas that are important to industry and suitable for NIST. While a potential concern with an organization the size of the Information Technology Laboratory is its ability to respond to change in a timely manner, the panel has been pleased with the flexibility shown by the divisions. The Advanced Networking Technology Division and the Software Diagnostics and Conformance Testing Division are to be particularly commended on their processes for discontinuing mature activities, identifying important areas where NIST’s contributions are needed, and redeploying staff with the appropriate skills to fill the voids. The NIST director’s office does provide some competence funding to assist in the development of new programs and build new expertise in the laboratories, and the panel encourages Information Technology Laboratory staff to submit high-quality proposals in order to take advantage of this funding source as the laboratory programs evolve.

In the past, the panel has commented on the importance of interdivisional collaboration on programs within the Information Technology Laboratory. This year, there were several examples of staff working together across divisions to make good progress on interdisciplinary projects. However, the panel feels that some additional interactions might be necessary to ensure that work in the various divisions is consistent and nonoverlapping, to increase awareness of areas in which there could be collaborations and synergies, and to cross-train staff in areas where key problems need to be understood from a variety of technical perspectives. Areas in which the panel felt increased communication and coordination between divisions would be helpful include activities relevant to biometrics in the Computer Security, Convergent Information Systems, and Information Access Divisions and the streaming media efforts in the Convergent Information Systems and Advanced Networking Technology Divisions. The panel also continues to emphasize the importance of integrating the laboratory-wide efforts in pervasive computing and suggests that connections between this work and the projects related to interactive television in the Convergent Information Systems and Software Diagnostics and Conformance Testing Divisions could be strengthened. In some cases, active collaboration across divisional lines will be needed, but informal cooperation is also very valuable. For example, often an individual project is relevant to a variety of technical communities but the terminologies used by these communities can be quite different or even inconsistent. If project staff can learn this unfamiliar technology from their colleagues in other divisions, they will be a better position to communicate effectively with their potential customers and increase their credibility both inside and outside NIST.

2  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Information Technology Laboratory Strategic Plan, National Institute of Standards and Technology, Gaithersburg, Md., April 2000.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

Program Relevance and Effectiveness

The Information Technology Laboratory has a wide array of projects that effectively support the U.S. economy and enable the federal government to meet its responsibilities to the American people. The divisional reports discuss in detail the various mechanisms used in the divisions to ensure that NIST’s work is relevant and affects the laboratory’s customers, but a few projects are highlighted here to exemplify the breadth and effectiveness of the laboratory’s programs.

In the Statistical Engineering Division, the work on characterization of high-speed optoelectronic devices is a good example of a project that solves an important practical problem, develops cutting-edge scientific methods, and demonstrates the value of an effective collaboration across NIST laboratory boundaries. Working with staff from the Electronics and Electrical Engineering Laboratory, division researchers investigated issues related to accurately measuring the performance of high-speed optoelectronic devices such as photodiodes and sampling oscilloscopes. The result was the development of statistical signal processing techniques that are being used to correct for the effects of time-base distortion, timing jitter, and impedance mismatch in measurements using high-speed optoelectronic detectors. The approaches worked out have been incorporated into a new measurement service that will support the telecommunications, cable television, and fiber channel networks industries.

The selection process for the Advanced Encryption Standard was completed this year. The Rijndael algorithm was chosen as the new standard, and the Computer Security Division’s management of the process is a good example of how NIST can capitalize on its reputation as an unbiased, technically expert organization to achieve international impact in an important area. NIST organized an open international competition in which candidate algorithms were made publicly available so that they could be judged by a large number of expert cryptographers; this was an impeccable approach for selecting a high-quality new standard that had the support of the international security community. Security is critical to government and industry, and the NIST-led selection process provided the encryption user community with a strong, stable, generally accepted standard to meet current and future security needs. This activity received a great deal of scrutiny and public attention, so the Computer Security Division’s successful management of it adds luster to NIST’s reputation.

Another example of how the Information Technology Laboratory provides key support in standards is the work on Fortran 90 graphics in the Mathematical and Computational Sciences Division. The problem was that graphics software developed in Fortran was not portable across systems because of incompatible interfaces. To resolve the issue, the division developed a standardized Fortran graphics interface, which was endorsed by industry’s OpenGL Architecture Review Board. NIST made the new standard available on the Internet and provided implementation guidance to support and encourage use of the standard. The standard has already been incorporated into the product lines of major providers of Fortran compilers, including Compaq Computer Corporation and Lahey Computer Systems, Inc.

Other projects worthy of mention are the Braille reader produced by the Convergent Information Systems Division and the information technology architecture initiative in the Information Services and Computing Division. Work on the Braille reader prototype is consistent with the division’s mission of integrating diverse technologies, and the division’s efforts in this area resulted in a technical breakthrough that could reduce the cost of such devices by thousands of dollars. The information technology architecture plan is in its formative stages, but the panel believes that it is a solid approach for determining what platforms should be supported at NIST and for communicating to staff and management throughout NIST what services can be expected from this new division. Once the division’s responsibilities have been defined, appropriate resources should be allocated to support their efforts, and techniques to measure the quality of the services provided should be put into place.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

All of the divisions in the Information Technology Laboratory are actively involved in standardsrelated activities. Examples include standards for Java Numerics, broadband wireless access, advanced encryption, fingerprints, digital television, and XML. NIST contributions include management of standards development processes and coordination of the relevant parties, analysis and testing of proposed standards, and development of tools needed to test conformance to standards. The key factors in NIST’s success in information technology standards-related work are the laboratory staff’s reputation for providing unbiased, high-quality technical advice, data, and tools, and the timeliness of the laboratory’s efforts (i.e., involvement early in the standards development process). The panel commends the laboratory for its continuing commitment to standards-related work. The industries that use the many standards that have been improved by NIST contributions are appreciative of the value the Information Technology Laboratory provides, and the laboratory staff’s work on standards does help build an awareness within information technology communities of the important role NIST plays.

Traditionally, industry used open approaches to standards development (e.g., work within organizations like the IEEE). In recent years, however, companies have begun to supplement these approaches with standards development in consortia and other private groups, where membership is conditional upon payment of a fee or signing of an agreement limiting disclosure or use of intellectual property. The value of these nonopen groups is that they can often be faster and more efficient than the traditional public approaches. The potential downside to these groups is that companies have been known to exclude their competitors from a group and manipulate the process to develop standards that give their products a competitive advantage. However, to meet NIST’s mandate to support U.S. industry, the divisions of the Information Technology Laboratory in some cases must consider joining private organizations and consortia. While the panel, and NIST staff as well, recognize that it would be inappropriate for NIST to join organizations with especially restrictive policies on membership or prohibitive constraints enforced by intellectual property or nondisclosure agreements, not all nonopen standards processes are unfairly exclusionary. The laboratory should therefore establish a policy to help divisions decide when participation in closed consortia is appropriate and should consider how NIST can encourage industry to utilize open, or at least inclusionary, approaches to standards development. The panel notes that when the laboratory does decide to participate in a nonopen standards group, appropriate and timely support from the Department of Commerce legal services will be needed so that staff may play their crucial roles in the standards development process and industry may benefit from the measurement and evaluation tools developed at NIST.

A key element in the effectiveness of Information Technology Laboratory programs is the relationships NIST staff have built with companies, universities, and other government agencies. These interactions help staff gather input on current and future activities and disseminate results to relevant communities. However, for the laboratory to strengthen current relationships and establish new partnerships, the panel believes that NIST should increase the visibility of its efforts in information technology. In some areas, such as networking, this means careful consideration of the quality and reputation of various journals and conferences so as to ensure that NIST publishing and speaking efforts produce the maximum impact. In other areas, such as mathematics and statistics, collaborative articles in scientific publications should be supplemented by papers in mathematics or statistics journals. In general, the goal is to have NIST’s value recognized not only by technical researchers within industry, government, and academia but also by their senior management. Information Technology Laboratory managers have indicated that they plan to create a position with responsibility for enhancing the laboratory’s reputation outside NIST, and the panel encourages this step.

Overall, the panel is impressed with the programs under way and believes that the Information Technology Laboratory is well-positioned to have an even greater impact than in the past. It is clear that

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

the broad topics that the laboratory has chosen to work on are the right ones, and the laboratory’s efforts to address relevant issues within these areas can be expected to have positive effects nationally and internationally. However, the panel is somewhat concerned about the selection of individual projects within the broad topics. While the bottom-up approach to identifying activities continues to produce good and effective projects, it is important for the laboratory not to neglect the larger view, which should ensure that the most important questions within the laboratory’s purview are being addressed, that the diverse expertise available across the laboratory is fully exploited, and that appropriate balance is maintained among various types of laboratory activities (standards-related work, basic research, technology development, paper writing and conference attendance, and so on). The panel noticed a proliferation of small (often one-person) projects, and these may be a result of the laboratory’s not considering how its collection of individually appropriate projects can be fit together to meet larger goals.

Laboratory Resources

Funding sources for the Information Technology Laboratory are shown in Table 8.1. As of March 2001, staffing for the Information Technology Laboratory included 368 full-time permanent positions, of which 302 were for technical professionals. There were also 92 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

Last year, the panel raised concerns about the fact that the explosive growth in information technology in industry was not being matched by growth in the funding or staffing levels in the Information Technology Laboratory. The need for new standards and protocols, for analysis of issues related to scalability and feasibility, and for support of new software and hardware continues to affect every division; laboratory staff simply cannot work on all of the problems important to their customers. While increased resources have been provided for work in the computer security area, other divisions continue to have flat funding, and staffing is actually decreasing. The panel fears that owing to resource constraints the laboratory is being forced to neglect important issues, and it again recommends that NIST compile a list of areas that the Information Technology Laboratory cannot tackle with its current staff and funding levels and describe the potential negative impact on industry.

The panel is still concerned about the continued housing of the Mathematical and Computational Sciences Division and the Statistical Engineering Division in NIST North. The work of these divisions is highly collaborative, and staff feel that their separation from their research partners on the main NIST campus significantly decreases their effectiveness and productivity. It is not clear to the panel that all possible approaches to bringing these divisions back to the main campus have been explored. Creative solutions should be investigated by laboratory and NIST management, and—most importantly—staff from these divisions should be kept up to date on the efforts and results. If solutions cannot be found, the reasons need to be clearly communicated to personnel at NIST North. This issue needs to be resolved.

In general, morale at the laboratory appears to be extremely high. Personnel are generally happy with the changes in management and structure that have occurred over the past year, and the panel commends laboratory management as a whole for the overall improvement in the working environment. Staff retention continues to be good, especially in light of the highly competitive job market for people with information technology expertise. During its visits to the Information Technology Laboratory and its individual divisions, the panel spoke with staff members without management present (skip-level meetings). Several issues arose in these conversations. One is an apparent need for more career counseling and development and for clearer criteria, or communication of criteria, on how staff performance is measured. Some staff wanted career development advice because they sensed that there were

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

TABLE 8.1 Sources of Funding for the Information Technology Laboratory (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)a

NIST-STRS, excluding Competence

31.6

31.6

31.9

38.3

Competence

0.9

1.5

1.6

1.3

STRS-supercomputing

11.8

12.1

12.0

12.4

ATP

1.8

1.8

2.4

1.6

Measurement Services (SRM production)

0.0

0.0

0.0

0.4

OA/NFG/CRADA

10.6

8.4

9.9

13.7

Other Reimbursable

1.5

0.5

1.6

1.6

Agency Overhead

12.0

14.4

16.4

18.9

Total

70.2

70.3

75.8

88.2

Full-time permanent staff (total)b

362

381

381

368

NOTE: Funding for the NIST Measurement and Standards Laboratories comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST’s congressional appropriations but is allocated by the NIST director’s office in multiyear grants for projects that advance NIST’s capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST’s ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Funding to support production of Standard Reference Materials (SRMs) is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding, including that for Calibration Services, is grouped under “Other Reimbursable.”

aDue to the reorganization of the ITL that began in November of 2000 and became official in February 2001, the budget estimates and staff numbers for FY 2001 are as of March 2001.

bThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

few formal avenues for them to upgrade their skills or develop new expertise. Others, because they were uncertain about the mission and priorities of the laboratory, how their projects fit in, and whether they were valued by management, wanted to know how their performance is measured. Such insecurity is a normal byproduct of all the changes that the Information Technology Laboratory has experienced over the past several years, but the panel encourages management to be aware of the concerns and try to resolve them. The skip-level sessions caused the panel to become somewhat uneasy about single-person projects. These small projects are of concern for two reasons: (1) without a critical mass of relevant expertise, staff can grow technically isolated, which affects the quality of their work and their morale, and (2) isolation also may contribute to staff’s uncertainty about the value of their contribution.

Last year’s report mentioned that NIST’s network connectivity to the outside world was significantly poorer than that available to many universities and industrial research organizations via the Internet2 Project (12 Mbps for NIST versus 155 Mbps or faster on Internet2). This issue is discussed in detail in the section on the Information Services and Computing Division. The panel notes that this outdated level of technology limits the quality of the staff’s connections to their customers and NIST’s

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

ability to efficiently and effectively distribute its results to industry. It is important for the Information Technology Laboratory to seek NIST-wide solutions for this problem so that individual groups do not begin to try work-around approaches on their own, which could lead to duplicated efforts or divergent solutions.

DIVISIONAL REVIEWS

Mathematical and Computational Sciences Division

Technical Merit

According to division documentation, the mission of the Mathematical and Computational Sciences Division is to provide technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to U.S. industry.

The panel is very impressed with the current status of the division. The division is in excellent shape, with technically strong and effective projects and capable and well-respected leadership. Good people are working enthusiastically on tough, important problems, and the level of collaboration with staff throughout the NIST laboratories is high, which advances the division’s ability to make an impact.

The Mathematical and Computational Sciences Division currently contains four groups: Mathematical Modeling, Mathematical Software, Optimization and Computational Geometry, and Scientific Applications and Visualization. This last group was transferred to the division late in 2000 from the former High Performance Systems and Services Division and is a strong and appropriate addition to the Mathematical and Computational Sciences Division. Both the new and old groups in the division appear to have welcomed the move, and integration of the new team with the established staff and programs in the division is going well.

Overall, the technical work done in the division continues to be excellent, with many projects well focused on important objectives. There are several examples that particularly impressed the panel. One was the project on blind deconvolution of images, in which the division is investigating ways to deblur images without knowing the cause of the blur. NIST is advancing the state of the art in analysis of such images, in part to support microscopy work ongoing within NIST, but the division’s methods have also garnered a great deal of attention in the external scientific community. The work on time-domain algorithms for computational electromagnetics is already strongly influencing the relevant technical community, and the division’s products have the potential to supply NIST staff and their customers with useful tools. The project on parallel adaptive refinement and multigrid methods, through its software package PHAML, has already had a positive impact both inside and outside NIST; now staff are investigating the intriguing possibility that it could be applied to modeling the behavior of the elements of a quantum computer. Finally, the Matrix Market, a standard collection of sparse matrices with search tools, statistics, and visualizations, is an excellent example of mathematical reference data and a good fit for NIST’s mission. Although the division currently has limited resources devoted to this project, the collection continues to grow and is the second busiest Web site in all of the Information Technology Laboratory.

The work on the Digital Library of Mathematical Functions (DLMF) is an excellent fit with the division and NIST missions. This project has ambitious and important goals, and the panel was pleased to see that progress on assembling this reference site is on schedule; chapter contracts are in place and writing is moving ahead well. A major accomplishment of the past year was securing support for this project from the National Science Foundation. In the coming year, the key challenge will be bringing

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

the completed chapters online on the Web.3 The panel believes that there are two issues associated with this task. One is whether the Web version, as it becomes fully available, will be able to incorporate lessons from user experience into its design. The second question is how and if users will be able to obtain function values. The Handbook of Mathematical Functions,4 the hard copy predecessor to DLMF, included tables that constituted reference values from NIST. The data were useful for readers and also provided some ability to validate implementations. It is not clear what the corresponding capability should be in the Web-based version. One option is for it to somehow use the reportedly very stable International Mathematical Subroutine Library or the Numerical Algorithms Group special functions codes. Another option is to try to take advantage of other projects in the division and investigate whether working with the Java Numerics project to produce downloadable Java software is feasible. This question of how users could obtain function values is a fundamental issue, and the panel believes that it will require some original thinking to find the right solution.

In addition to the work on established and traditional projects, the Mathematical and Computational Sciences Division is exploring a number of important new directions, such as combinatorial methods, quantum computing, bioinformatics, and mathematical metrology. The challenge in these new areas will be to carve a focused agenda out of a huge realm of possibilities. Initiatives like the work on bioinformatics and the new DARPA proposal on quantum computing seem quite promising, but some overall strategic vision will be needed to define a coherent program in which NIST can be most effective.

Complementing the division’s exploration of new directions is its appropriate conclusion of some existing work, which will free up resources to invest in new efforts. The project on appearance of coated objects and the work for the Army in terrain modeling have achieved their main goals and are coming to an orderly close. Both projects have had an impact, and the division is correct to treat them as completed and replace them with new priorities.

Program Relevance and Effectiveness

The Mathematical and Computational Sciences Division performs highly effective work in mathematical modeling, software, and visualization. Demand for the results of these activities and for participation by staff in collaborative projects exceeds the time staff have available, and customers are very appreciative of the division’s efforts. The division has received glowing testimonials on the impact of its work from Ford, Hewlett-Packard, and other companies, from university research groups, and from staff in the other NIST laboratories, including materials scientists, physicists, and building and fire researchers.

The Java Numerics project is an example of an activity that demonstrates the way in which this division maximizes its impact and how NIST leadership plays a vital role in the chaotic and fast-moving world of information technology standards. Three important steps by the division ensured that this project would make a difference. First, staff recognized that Java would become an important tool for numerical computing but that its design contained significant problems. Then the division brought together a technical community to address the problems and propose solutions. Finally, NIST staff

3  

The introductory page for the DLMF is available online at <http://dlmf.nist.gov/>, and a mockup of it, including links to draft outlines and chapters, is available online at <http://dlmf.nist.gov/Contents/>.

4  

M.Abramowitz and I.A.Stegun, eds., Handbook of Mathematical Functions, with Formulas, Graphs, and Mathematical Tables, National Bureau of Standards, Gaithersburg, Md., 1964.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

convinced Sun Microsystems, the owner of the standard, to support their efforts and endorse the solutions. This last step was a remarkable achievement, given the technical and commercial conflict that existed over Java at that time. The panel applauds the division’s work in this area and notes two crucial factors in the success: the original vision that this was an important area where NIST was needed to serve as a neutral broker and the effective leveraging of the strong technical credibility of the division and its staff.

The Java Numerics effort is just one example of how the division’s standards work affects a broad array of communities. In fact, NIST’s impact in this arena often extends beyond U.S. borders, and several division projects have well-chosen international components. In the Java Numerics project, staff convened an international forum to help improve a commercial standard used around the world. Similarly, division staff convened a group of relevant international vendors and provided a conformance tester to facilitate the development and adoption of the Interoperable Message Passing Interface protocol standard to support the writing of parallel scientific applications. NIST personnel also work through and with existing groups and external organizations. The division participates in the Basic Linear Algebra Subprograms Technical Forum, an international consortium coordinating standardization effort for BLAS. In the F90GL project, NIST and industry collaborated to develop a standard that has been officially adopted by an international organization (OpenGL); the division’s reference implementation for this standard is now in several commercial software products.

In addition to specific project-related activities, division personnel actively serve technical communities in a wide variety of ways, including as members of professional society boards, participants and leaders of working groups, and editors of journals. Examples of the important roles filled by staff are editor in chief of the Association for Computing Machinery’s Transactions on Mathematical Software, chair of the Society for Industrial and Applied Mathematics’ Activity Group on Special Functions, co-chairs of the Java Grande Forum Numerics Working Group, and chair of the International Federation for Information Processing’s Working Group 2.5 on numerical software standards. These critical positions held by division staff contribute to the division’s high reputation for technical excellence and its credibility.

Staff participation in committees is one way in which people outside NIST are kept informed about the division’s activities and results. A number of other, more formal dissemination mechanisms are also employed by the division. In the past year, division staff have published 25 articles in refereed journals and many reports and have given 53 invited lectures, numerous short courses, and seminars. The division has long been a leader in using the Internet to disseminate information; an example of their pioneering early work in this area is the Guide to Available Mathematical Software (GAMS).5 This Web site, which was begun in 1994, is still remarkably effective and has about 400 downloads per day. GAMS is now just one of the division’s many useful and popular Web sites; currently, the division operates the second-busiest Web site at NIST and the seven busiest Web sites in ITL.

The high technical merit and the effectiveness of the division’s work are a result of the quality of its personnel and of their reputation at NIST and in the larger mathematical and scientific communities. The value placed on NIST staff (and the relevance of the work they do) is reflected in the awards they have received. In the past year, one staff member received the Arthur S.Flemming Award recognizing her fundamental contributions in fields like probability and stochastic modeling and the impact of her work via extensive close collaborations with scientists and engineers. Also in 2000, a team from the new Scientific Applications and Visualization Group received a NIST Bronze Medal for its efforts to

5  

The Guide to Available Mathematical Software is available online at <http://gams.nist.gov/>.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

produce images of Bose-Einstein condensates. This work was of value to NIST and the physics community in more ways than one: It contributed to the understanding of a fundamental physical system and it helped debug the computational simulation of this system. Not only is information about this technical achievement being published in the visualization literature, but the vivid representation of the Bose-Einstein condensate has caused the results to be featured prominently in the physics literature and the popular science press.

The panel is impressed with the strong, positive outlook for the Mathematical and Computational Sciences Division and the relevance and effectiveness of the past and ongoing projects. However, as the division moves forward, there are a number of important strategic issues to be faced, especially in light of the fact that the potential activities for this division far outstrip the resources available. The panel therefore wishes to comment on issues worth considering as the division decides its priorities and tries to maximize the effectiveness of its programs.

The individual projects in the division are very strong, but they need to be supported by periodic examination of the big picture. This step back is necessary to determine where the holes are in the research portfolio and what emerging key areas are being overlooked. The division should be careful to consider a long-range view of overall programmatic goals when hiring new staff or developing new projects. One area that could be worth contemplating is financial issues. Subjects like modeling financial options and dealing with cost as an independent variable in design and manufacturing are of growing importance for the U.S. economy and are highly mathematical in nature, yet they do not seem to have been taken into account in the division’s current agenda.

A related issue is the need to build a complete picture of customer needs at a strategic level. The Mathematical and Computational Sciences Division has historically demonstrated the ability to combine technical excellence with the fulfillment of NIST’s mission, to come up with projects that make good strategic sense; recent examples include GAMS, DLMF, Java Numerics, and the work on finite element analysis of materials microstructures. While these projects have been effective and appropriate, the panel notes that the decisions to launch them to appear to have been driven by excited and visionary researchers with mentoring and guidance from management, rather than by top-down strategic planning. While building on the inspirations of the technical staff is an essential element of the division’s success, management’s contribution of a broader view, backed by global strategic designs, is also critical.

For the division’s work with other NIST laboratories, a primary source of projects is the web of connections that division staff have built up with personnel across the campus. The strength of this approach is that successful technology deployment takes place readily in a collaboration based on past relationships and mutual trust. Its weakness is that such deployment may never happen if the relevant people do not know one another. While networks of individuals and their successes should not be ignored or discouraged, it would also be advantageous to strategically consider what mathematical or computational technologies might be relevant or where different opportunities for collaborations might lie. This is accomplished not by asking other NIST laboratories what they think they need from the division but by holding strategic sessions on what is needed and what new technologies might make possible.

Even when additional appropriate areas for the division have been identified using the mechanisms described above, it is unrealistic in light of the limited resources to expect the division to be able to take on all projects that would be of value to NIST. Technologies are changing rapidly, and the division might benefit by developing a surveillance program to maintain awareness of what is being developed at NIST and in the rest of the world. This information could be used to help decide what areas the division should work on itself and in what areas it could direct its customers to other organizations and scientists.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

The panel’s final observation was that the division’s organizational structure seemed to be used more to divide up administrative tasks than to identify staff with particular technological areas. Since most of the problems that the division tackles for its customers involve multiple disciplines, the division is to be commended for the fluidity of its organizational boundaries and its ability to form project teams that combine all the relevant capabilities. However, it is important not to entirely neglect the focus on an individual discipline that comes with providing a clearly defined connection between an organizational group and a particular type of expertise or subject area. Such a focus assures excellence in a specific discipline, readily allowing the surveillance of a particular field and creating a natural structure for recruiting and connecting to university research. The panel is not suggesting that the boundaryless environment that currently exists in the division be discouraged or impeded in any way but rather that it be complemented by an effort to sharpen the disciplinary focus of the current groups. The development of a specific technological identity for each group will be particularly valuable as the division refines its programs in discrete mathematics and optimization.

Division Resources

Funding sources for the Mathematical and Computational Sciences Division are shown in Table 8.2. As of March 2001, staffing for the Mathematical and Computational Sciences Division included 39 full-time permanent positions, of which 36 were for technical professionals. There were also 14 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

Staff morale is good in the division, and retention of personnel is high, especially considering the current competitive job climate. This positive situation is the result of a number of factors, including good communication and trust between laboratory management and division staff and staff enthusiasm

TABLE 8.2 Sources of Funding for the Mathematical and Computational Sciences Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)a

NIST-STRS, excluding Competence

4.3

3.3

3.6

4.0

Competence

0.1

0.2

0.2

0.1

STRS-supercomputing

0.6

0.7

0.6

3.1

ATP

0.2

0.1

0.1

0.3

OA/NFG/CRADA

0.3

0.4

0.7

1.0

Other Reimbursable

0.0

0.0

0.1

0.0

Total

5.5

4.7

5.3

8.5

Full-time permanent staff (total)b

30

30

27

39

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe difference between the FY 2000 and FY 2001 funding and staff levels reflects the reorganization of ITL in which the visualization group was moved out of the Convergent Information Systems Division and into this division.

bThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

for their projects. The panel particularly commends the division chief for creating a level of energy and excitement that helps make NIST an attractive place to work. While there still exist some issues related to facilities (as discussed below), past concerns about morale are greatly reduced, and this change reflects positively on division managers and the new laboratory management.

While the current collection of staff is well-qualified and stable, valuable expertise in math software and in optimization that was lost a few years ago still has not been restored. As the division continues to move into such new areas as bioinformatics and quantum computing, new personnel will be needed to strengthen the division’s capabilities in discrete mathematics and optimization. The new programs will also require strategic vision and leadership, and division management must identify someone, either within NIST or from outside, who can provide direction in these emerging areas. The value of such a vision can be seen, for example, in the quality and impact of the division’s work in mathematical software.

The primary facilities issue continues to be the location of the Mathematical and Computational Sciences Division at NIST North. As the panel emphasized in past reports, the separation of the division staff from personnel in the rest of the NIST Measurements and Standards Laboratories hinders the formation and execution of collaborations. Laboratory and NIST management have an obligation to seriously consider all possible means by which this division and the Statistical Engineering Division (for which the issues are similar) could be moved back to the main campus. If such a move is absolutely impossible, the reasons need to be communicated clearly to the staff, and NIST should aggressively investigate new tools and technologies to facilitate distance collaborations. The division’s use of such tools would have a dual benefit: advancing the cooperative efforts under way and testing the strengths and weakness of these tools in order to provide guidance and recommendations to other staff at NIST, who often have research partners from around the United States.

Other facilities issues include what the panel perceived to be an overall strain on the infrastructure. Even in NIST North, the physical space available to the division is tight, and this has limited the division’s hiring of student interns. The video link to NIST’s Boulder campus was out of order during the panel’s visit to the division. The staff is particularly concerned about declining library resources. The mathematics literature is very specialized, so often individual journals or publications do not have many readers, which is the traditional metric for value. However, individual references in mathematics often have an extremely high impact in terms of effort saved, so access to good reference materials is particularly necessary. While the limitations described here are a familiar story in many laboratories, the growth of NIST’s customer base in the national information technology industry is certainly not being reflected by growth in the Information Technology Laboratory in general or the Mathematical and Computational Sciences Division in particular.

Advanced Networking Technologies Division

Technical Merit

According to division documentation, the mission of the Advanced Networking Technologies Division is to provide the networking industry with the best in test and measurement technology.

This mission statement is appropriate and accurately reflects the NIST and laboratory missions within the context of technologies relevant to this division’s work. The division focuses on using test and measurement technologies to improve the quality of networking specifications and standards and to improve the quality of networking products based on public specifications. Overall, the division’s activities are relevant and effective, and the work spans several of the areas currently important in networking research.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

For a second consecutive year, the panel has observed notable improvements in the division’s work. More synergy exists among the ongoing programs and more communication and collaboration among the research groups. Division projects are generally well focused on achieving specific and valuable goals and are well-directed in support of the NIST mission. Equally important is the good technical quality of the work being done on these projects.

The Advanced Networking Technologies Division consists of three groups: High Speed Network Technologies, Wireless Communication Technologies, and Internetworking Technologies. The areas of technology under investigation at this time are Media Convergence and Security, Agile Networks, and Programmable Networks, and the nine division programs all fall into one or more of these categories. Below, the panel describes some of the highlights and issues observed in its assessment of these activities.

In the Advanced Networking Technologies Division’s work on pervasive computing, two projects support the development of networking standards for relevant devices. The first focuses on issues surrounding how to craft the various ubiquitous wireless standards (e.g., IEEE 802.15 and IEEE 802.11) so that they do not conflict within the unlicensed 2.4-GHz band. The original designers of the relevant standards all assumed that their technology would be the sole user of this band, but now it is clear that coexistence will almost certainly be necessary. NIST has taken an important leadership role on questions related to reconciling the standards. Division staff have done some formal modeling of Bluetooth and have developed tools to simulate the interference that may occur between devices using the different standards. While the NIST results demonstrate that critical interference problems arise, they also show that various techniques can improve the ability of the different standards to coexist. These are valuable findings, and the panel is particularly impressed by how aggressively and effectively the division tackled the problem. The potential for interference problems was only recognized in late 1999, and by the beginning of 2001 NIST staff were briefing IEEE standards bodies about their results and proposed solutions to the problem. This timely information will allow the IEEE groups to incorporate the division’s solutions into the standards.

The second effort in the area of networking standards for pervasive computing devices focuses on the analysis of the resource discovery protocols being developed for ubiquitous computing systems. Division staff are modeling and analyzing protocols to evaluate their ability to perform various functions (mainly using formal models) and to see how they scale with different network sizes. Current efforts include work on modeling service descriptions for Jini, UpnP, and other protocols. The use of Rapide as the architectural description language is appropriate as it appears to capture the salient abstractions and allows exploration of constraints and possible constraint violations. The panel believes that, overall, this is a high-quality project that continues to make progress; recent steps forward include proposing a set of measurements, deriving quantitative measures, and building an emulation environment. The panel’s only suggestion is that, since the relevant standards bodies will be making critical decisions soon, the division should put extra effort into circulating its results to these bodies within the next several months.

In the area of Agile Networks, one division program focuses on technologies and standards for mobile ad hoc networks (MANETs) and smart sensor networks. The work on MANETs concentrates on analysis but also includes simulations using OPNET. One question that might be worth considering within this effort is how to exploit that fact that computation is becoming cheaper than high-rate wireless. In the sensors area, the emphasis is mostly on distributed detection, but the reason for this emphasis was not clear to the panel. The synergy between the two areas is good, but the panel suggests that the MANET work might productively evolve toward investigating networking protocols and distributed algorithms in order to better support sensor networking.

Also in Agile Networks, the division continues to work on modeling, evaluation, and research of lightwave networks (MERLiN). In the effort on wavelength division multiplexing (WDM) modeling, staff

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

have put together a virtual topology simulator for wavelength assignment and dynamic reconfiguration and are also evaluating protection and restoration algorithms using simulation and measurement tools. A related effort is the project on multiprotocol label switching (MPLS) optical restoration and recovery, and the panel commends the refinement of the goals in both of these activities over the past year; the resulting synergy is impressive.

The project on Internet telephony (voice over IP) is moving in the right directions and increasing its relevance. The development of a session initiation protocol (SIP) interoperability test tool is appropriate, and the Web-enabled SIP load generation and trace capture elements of this tool have the potential to be very useful. Good work is being done on a NIST simulator that allows one to measure voice quality on models of various networks, and the panel believes that this work could evolve into an effective project. The division is not currently looking at the pieces that surround basic call signaling, like telephony routing over IP, telephone number mapping, and call routing, and the panel believes that staff should seriously consider whether the current focus on basic SIP issues is suitable. Perhaps addressing questions related to the other elements around SIP-based call signaling would be a more effective way for NIST to contribute in this area.

One of the division’s security efforts has focused on developing reference implementations for Internet Protocol Security (IPsec) technologies. The results from this work have proven useful to the community, and the project now is winding down and has no plans for future activities in this area. The emphasis is shifting to work on critical infrastructure in collaboration with the Computer Security Division. This is an important project, and the panel believes the shift is appropriate. Other potential new areas that would utilize the expertise available at NIST include security for content distribution (under consideration by the division), quantum key distribution systems, and, possibly, multicast key distribution. While these new areas might all be valuable, the panel members also urge the division to be alert for potential new issues in high-performance IPsec extensions that are expected to arise in the next year or two.

Overall, the panel is very pleased with the division’s ability to sunset activities either because the stated goals have been accomplished or because technical innovations require a shift in focus. The division has demonstrated an impressive agility and has shown an ability to jump into an area at an early point and select work with significant potential impact. An example of a successful effort now winding down is the work on broadband wireless access. This project contributed by providing results from fixed point-point 30-GHz band modeling and analysis that were used in the IEEE 802.16 standards and by modeling the performance of CDMA-2000 and supplying models for the Cadence toolset. These accomplishments have made an impact, and concluding the project is an appropriate move. The project on active networking could also be wrapped up, with the expertise refocused on network management applications. More resources and effort could also be devoted to the work on pervasive computing, an area in which NIST could make a real difference.

As noted above, the panel was impressed with the closer collaborations that have developed between staff working on different projects; specific examples of synergistic pairings include WDM modeling and MPLS; IP quality of service and voice over IP; and sensor networks and MANET. Effective utilization of the expertise and results available in other groups within the division is a good way to leverage a project’s resources and maximize NIST’s impact.

Program Relevance and Effectiveness

Staff of the Advanced Networking Technologies Division continue to be active in a variety of industry organizations, including the Internet Engineering Task Force (IETF), the IEEE, and the International Telecommunications Union. NIST personnel are well respected by the staff of these standards

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

organizations and by the communities they serve. The value of the division’s standards-related efforts are realized in several ways. Most often, technical work done at NIST, such as modeling and analysis or development of testing tools and evaluation criteria, provides a greater understanding of the implications of proposed standards or supplies solutions to problems that could arise in standards development. NIST’s familiarity with the networking community and its reputation for an unbiased technical approach is also useful in determining what issues have inspired the standards effort and in defining the technical space on which the standards bodies should be focusing. Past examples of the impact of the division’s work include the use of modeling and analysis results in the IEEE 802.16 standards on broadband wireless access. Current work on modeling of interference in the unlicensed 2.4-GHz band will probably play a key role in determining what standards are adopted in this area, and the division’s work has the potential to have significant impact over the next year.

The standards-related activities mentioned above take place mainly through professional organizations, and the standards are “open”—that is, the process by which they are developed and implemented is open to the public, and representatives from any institution may participate. In recent years, industry has begun to supplement this traditional approach with other models of standards development. Interested parties come together in consortia or other private groupings to decide on a standard. Participation usually is conditional on paying a fee, signing a nondisclosure agreement, or agreeing to controls on intellectual property, or a combination of these conditions. Such models have grown in popularity because they can be faster and more efficient that open standards development processes. The panel, and the division, recognize that the “closed” system is somewhat antithetical to the NIST and governmental philosophy of supporting all U.S. companies and the public in an open manner. However, to carry out the NIST mission of strengthening the U.S. economy, the division must be able to impact the standards that will be used in the networking community no matter how they are developed. Therefore, NIST should develop a policy on this issue and criteria for deciding when and how to participate in these consortia. Some of the closed standards groups are actually very inclusive, with minimal burdens placed on participants; others may be designed to exclude potential competitors and should not be endorsed by NIST. Therefore, NIST should also consider whether it could develop a strategy for encouraging the IT community to continue to utilize open or at least quasi-open models of standards development.

The Advanced Networking Technologies Division assumes a leadership role in the networking communities, in part by virtue of the standards activities described above. However, it is important for the staff to build awareness of NIST’s expertise and maintain its reputation in other ways. The division does publish in journals and conference proceedings, and its personnel attend a variety of meetings. These activities are appropriate, but the panel suggests that the journals and conferences could be selected more carefully. Strategic decisions about which meetings and periodicals would provide the widest dissemination and afford the greatest impact for NIST results would allow the division to burnish its reputation and position itself as a key element of the networking community.

Division Resources

Funding sources for the Advanced Networking Technologies Division are shown in Table 8.3. As of March 2001, staffing for the Advanced Networking Technologies Division included 21 full-time permanent positions, of which 18 were for technical professionals. There were also 5 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

The primary issue for the Advanced Networking Technologies Division is its limited number of full-time permanent staff. The total number of full-time permanent technical professionals dropped by 4 (18 percent) in 2000. While the group continues to do relevant and effective work, the heavy reliance on guest

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

TABLE 8.3 Sources of Funding for the Advanced Networking Technologies Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)

NIST-STRS, excluding Competence

4.1

4.9

4.0

4.0

Competence

0.0

0.2

0.3

0.2

ATP

0.3

0.3

0.5

0.2

OA/NFG/CRADA

1.5

1.2

1.7

2.6

Total

5.9

6.6

6.5

7.0

Full-time permanent staff (total)a

26

30

27

21

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

researchers (about 24 people) is increasing. The division continues to be dependent on visitors to support the mission-critical projects. This approach has its risks, primarily the potential for unexpected delays or the premature termination of an important effort when a temporary staff member leaves NIST. It also has benefits, particularly in the added manpower and capabilities that allow the division to expand its programs; these people also interject new ideas and provide information about advances and activities at other institutions. The panel believes that the benefits are currently outweighing the risks.

In addition to growing more reliant on visiting staff, the division is also increasing its fiscal support from external sources (mainly other government agencies). This sort of supplement to the core funding of the division can be very valuable and provides evidence that division programs are relevant to outside institutions. However, external funds usually expire at the end of each fiscal year, and this can interfere with the division’s ability to make and execute long-term plans.

The panel was pleased to observe that morale within the division is quite good and that the staff are enthusiastic about their work.

Computer Security Division

Technical Merit

According to division documentation, the mission of the Computer Security Division is to improve information systems by raising awareness of information technology risks, vulnerabilities, and protection requirements, particularly for new and emerging technologies; by researching, studying, and advising agencies of information technology vulnerabilities and devising techniques for the cost-effective security and privacy of sensitive federal systems; by developing standards, metrics, tests, and validation programs to promote, measure, and validate security in systems and services, to educate consumers, and to establish minimum security requirements for federal systems; and by developing guidance to increase secure information technology planning, implementation, management, and operation.

The division’s programs directly support this mission and are also consistent with the laboratory and NIST missions. Measurements and standards related to privacy and security are crucial for protecting

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

electronic commerce, critical infrastructure, personal privacy, and private and public assets, all of which are essential for the U.S. economy and the public welfare.

The projects under way in the Computer Security Division are very appropriate activities for NIST, and the technical quality of the work is high. The division is organized into two groups: Security Technology and Systems and Network Security. In addition to the ongoing projects in these groups, the division also has several new activities as part of the fiscal year 2001 federal initiative on critical infrastructure protection. Below the panel discusses some of the highlights and issues associated with the programs reviewed this year.

In the Security Technology Group, a major recent triumph is the selection of the Rijndael algorithm as the draft Advanced Encryption Standard (AES), as announced in October 2000. This decision came after an international competition in which the stability, security, and performance of a number of proposed algorithms were examined by cryptographers around the world. NIST effectively managed this process by testing candidates, hosting several international conferences, and providing a forum for public analysis of the options. By virtue of the Computer Security Division’s work on this project, NIST played an appropriate role in the discussions and provided a key service to the international security community. The high technical quality of the work and the balance with which the division staff led what was widely perceived as an open and fair process enhanced both the visibility and reputation of NIST.

The Security Technology Group also contains a number of other projects. In their work on cryptographic security testing and the program on cryptographic module validation, staff want to ensure that the cryptographic modules being used to provide security function correctly. Work on public key infrastructure (PKI) continues and includes both technical activities, such as developing algorithms to help commercial products conform to standards, and managerial responsibilities, like leading the Federal PKI Technical Working Group. A critical element of the division’s technical activities is examining the security requirements of PKI components. These PKI-related efforts reflect the need to develop a general infrastructure for key management. Finally, the group has a project on developing a cryptographic standards toolkit and another on providing standards for and guidance in the use of cryptographic technologies. These efforts are of high technical quality and produce results that affect software development, electronic commerce, and other areas of information technology.

In the Systems and Network Security Group, the staff continue to manage the National Information Assurance Partnership (NIAP) program, which focuses on developing Common Criteria protection profiles and investigating issues related to the use of these profiles in developing security requirements for the federal government. This project is timely and relevant. The division is also working with companies and other government agencies to develop a private information technology security testing industry in support of the Common Criteria. On the international front, staff continue to help put mutual recognition testing agreements with other nations into place; these documents will govern recognition of laboratories that evaluate products relative to the Common Criteria. Also as part of NIAP, the division is investigating smart card technologies and their potential to support information technology security. Overall, the goal of the NIAP program is to provide quicker, more effective security evaluations and standardize baseline security requirements for particular environments and products.

The Systems and Network Security Group also has a project specifically focused on intrusion detection testing. The motivation for this work is the popularity of many intrusion detection systems, whose performance has, however, seldom been subjected to standardized testing. The division plans to adapt a test methodology developed at Lincoln Laboratories and to gather data to provide vendors with a rigorous test methodology; the panel finds this approach appropriate and well thought out. Enabling the evaluation of the intrusion detection systems would increase their effectiveness and benefit consumers, who could better compare the available products. Another product that will aid researchers,

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

developers and users of computer systems is the ICAT Metabase. In this work, staff are collecting vulnerabilities into a searchable database that supplements existing public databases by being more precise and detailed.

There are a variety of other appropriate projects under way in the Systems and Network Security Group. With immediate applications in existing and emerging technologies is the work on testing for authorization management, vulnerability analysis, intrusion detection, and mobile code. Within this emerging technologies project, the division has a productive collaboration with the Advanced Networking Technologies Division on an IPsec interoperability testbed and reference implementation. This is a good example of how division staff routinely participate in cross-divisional efforts and provide support for internal NIST activities. Another area in which Computer Security Division expertise might be productively applied to a program in a different division is biometrics; the Convergent Information Systems Division activities in this field would benefit from advice about issues of interest to and vocabulary used in the security community.

Within the Systems and Network Security Group, an activity that is critical to the security of the information infrastructure and will become important in the future is the work on mobile agents. These agent systems are widely used in electronic commerce and the Internet, and the division is focused on improving their security and the security of those who use them. Finally, work with a very long horizon for practical implementation is the quantum information network project. This work is being done in collaboration with the NIST Physics Laboratory and is part of a Competence project funded by the NIST Director’s office. This is a speculative research effort with great potential. The goal is to establish a quantum communication testbed and build components of a scalable quantum information network. If successful, the division’s work could be a showcase and an invaluable resource for national efforts in quantum computing and cryptography. This work is consistent with the division’s mission as it will build NIST expertise in an area that may be important for future computer security measurements and standards.

Program Relevance and Effectiveness

The Computer Security Division’s programs significantly impact a wide range of constituencies. For example, the selected AES algorithm, the result of a NIST-managed evaluation process, will be used by both government and industry for years to come. The international computer security community embraced and applauded the process, and the positive publicity gained by the division and NIST is remarkable. Other division work that is expected to have a broad governmental and industrial impact is the cryptographic testing and module validation project. One hundred and fifty modules have been validated (62 in 2000), and the division’s efforts in this area will benefit users and organizations by enabling trustworthy cryptography. This project is often cited as a good example of how properly focused security evaluation criteria can be applied successfully and productively. In other areas, the frequent use of the IPsec interoperability testbed and the central role of division staff in the smart card consortium, as well as staff participation in and leadership of national and international standards bodies and committees, demonstrate the value to industry of the division’s efforts and how NIST personnel are uniquely qualified and positioned to perform important functions and services for the computer security community.

The division has a large number of standards-related activities, including developing cryptography and security standards as Federal Information Processing Standards and serving on national and international standards committees. Standards activities are consistent with the goals of the division and vitally important to the NIST support of U.S. industry and government. The division’s work in this area will positively affect the way customers use and implement cryptographic and other security mechanisms.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

One example is the work on the Common Criteria mutual recognition agreements. The division’s efforts to establish these agreements, to accredit private U.S. laboratories for Common Criteria testing, and to develop protection profiles for the Common Criteria will all contribute to the efficient adoption and deployment of systems meeting the Common Criteria.

In general, the division is guided by the belief that standards developed by or with the participation of industry should be used unless they involve poor practices from a security perspective. The panel finds this to be a reasonable approach. However, when adopting standards, care must be taken to ensure that NIST is not endorsing specific protocols developed by companies outside open standards environments. The panel believes that the division should aim to sanction only standards that resulted from open development processes, such as processes within inclusive standards organizations or at NIST itself. Standards that are developed via an exclusive process, by a single company or by a group that does not include all relevant parties, have the potential to be used as a tool for gaining a competitive advantage or limiting the entry of new products and businesses into a market. Also, an open standards development process has a better chance of catching flaws in proposed standards, whereas exclusive processes make it easier for companies to argue against fixing the standard when the change would delay deployment of a new product or interfere with products already on the market. Since NIST’s mission is to strengthen the U.S. economy by supporting industry as a whole, rather than an individual company at the expense of other businesses, it is appropriate for the Computer Security Division to encourage the use of open standards development processes.

While the panel strongly recommends that the Computer Security Division be involved in open standards development processes, it also acknowledges that there are times when NIST unavoidably encounters intellectual property issues in the course of its activities. In some cases, CRADAs have proven to be an effective means to handle these issues and to facilitate collaborations with industry. An example is the CRADA currently in place to govern the division’s relationship with Boeing on the mobile agent project. In general, however, CRADAs are complicated to put in place and the delays in implementing the agreements can slow progress on a particular activity. As a result, the division’s use of them has declined, and they are now only used when absolutely necessary. The trend appears to be towards more participation in open consortia and more attendance at invitation-only workshops.

Division Resources

Funding sources for the Computer Security Division are shown in Table 8.4. As of March 2001, staffing for the Computer Security Division included 40 full-time permanent positions, of which 37 were for technical professionals. There were also 13 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

The panel is very pleased to see that the division’s budget is projected to grow significantly in fiscal year 2001. This sharp increase is due to receipt of funding for NIST’s contributions to the federal government initiative on critical infrastructure protection. Beyond core (STRS) funding, which will continue at about the same level as last year, the Computer Security Division will receive $2.9 million for research and development activities in this area, another $2.9 million for the Computer Security Expert Assist Team (CSEAT) activity, and $0.5 million to administer a $5 million grants program funding research in this field.6 The panel has several concerns about the CSEAT and grants programs, which are discussed below. In general, however, the significant increase in total base funding is a

6  

All numbers are estimates as of January 2001.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

TABLE 8.4 Sources of Funding for the Computer Security Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)

NIST-STRS, excluding Competence

6.3

5.9

6.6

12.4

ATP

0.0

0.2

0.3

0.5

OA/NFG/CRADA

3.7

2.3

1.4

1.4

Other Reimbursable

0.0

0.1

0.1

0.1

Total

10.0

8.5

8.4

14.4

Full-time permanent staff (total)a

44

48

43

40

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

positive development because it reduces the division’s need to rely on external funding (money from other agencies) tied to particular projects of interest to the external agency. With greater independence, the division has more control over the scope and direction of its programs and is better able to focus its work in support of the division, laboratory, and NIST missions. Staff morale is also improved.

The CSEAT activity would be related to the Systems and Network Security Group’s collection of programs designed to provide U.S. government agencies with technical and programmatic guidance in the area of computer security policy, management, and operations. These activities are entirely suitable for NIST staff, and the laudable goal of advising and assisting government agencies is, in general, appropriately matched with the skill sets within the Computer Security Division. The panel’s biggest concern in this area, however, is the proposed CSEAT. The team would be based at NIST and would provide assessment and advice to government agencies in need of significant computer security help. The panel was presented with information about NIST plans for this group and was deeply concerned that this activity had the potential to be an inappropriate use of NIST resources. If the team is required to provide detailed security checking for the agencies, the division will need to find and hire very specialized personnel. There are very few people who are capable of doing the system analyses such a commitment would need, and those people are unlikely to come work at NIST. This approach would also be very costly. If, instead, the team is only required to provide a minimal security check, then NIST would be doing tasks commercial outfits can accomplish equally well, in part because those outfits have significantly more resources than are currently allotted for CSEAT. In fact, the limited resources for CSEAT mean that the division would have to outsource many responsibilities anyway. In either scenario, CSEAT’s effectiveness would be seriously in doubt. The panel recommends that the division reconsider CSEAT’s emphasis on performing penetration studies for government agencies and refocus the effort to support existing NIST programs in other ways—for example, through training and education. Given the tight job market for security experts and the limited staffing available in the division, NIST should be wary of taking on too much responsibility in too many areas. The panel notes that when it visited NIST in the spring of 2001, the goals and responsibilities of the CSEAT effort were very much in flux, and the panel appreciates the division’s responsiveness to members’ comments and hopes that the effort can be successfully redirected.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

Another part of the critical infrastructure protection is the grants program. The panel believes that the current level of funding for this program ($5 million to be distributed in grants) is too small to have an impact. Numerous grants funded at low amounts will provide neither the breadth nor the depth of research required to make meaningful contributions to the massive and complex problem of how to secure the nation’s critical infrastructure. The panel hopes that the fiscal year 2001 funding is only the beginning of a significant effort and that future funding will expand so that the program can adequately support the needed effort across the United States.

There are a number of positions open for technical, managerial, and support personnel that the division is planning to fill when the hiring freeze associated with the change of administration is lifted. The division recognizes that the job market is tight for people with relevant expertise. In the fiscal year 2000 assessment report, the panel raised concerns about the number of “acting” division managers. This year, the panel was pleased to find that a number of management changes had occurred, including removal of “acting” from the titles of the division chief and the Security Technology Group Leader and several promotions. Morale in the division has improved, and the division chief certainly deserves some credit for this, as well as for providing high-quality technical leadership.

Information Access Division

Technical Merit

According to division documentation, the mission of the Information Access Division is to accelerate the development of technologies that allow intuitive, efficient access, manipulation, and exchange of complex information by facilitating the creation of measurement methods and standards.

The Information Access Division is composed of four groups: Speech, Image, Retrieval, and Visualization and Usability. These groups are engaged in a very broad array of activities, from document searching, filtering, and understanding to image recognition and matching to standards and evaluation tools for usability. The panel feels that the programs are going very well this year and offers some comments on the division’s recent achievements.

The Speech Group works with DARPA, the National Security Agency, and the spoken language research community in industry and at universities to develop standardized speech databases and metrics for evaluating state-of-the-art speech and speaker recognition systems and uses these databases to coordinate benchmark testing within the community. The tools developed in this group have played an important role in the surge of speech recognition programs on the market, and the size and scope of the group’s efforts have grown in recent years, reflecting the increasing commercial interest in spoken language technologies. The emerging areas that the group has started projects in include spoken document retrieval, entity detection and tracking, natural spoken dialogue, and topic detection and tracking.

One of the relatively new activities for the Speech Group is the DARPA Communicator project. There are a number of organizations involved in this project, and the overall focus is on enabling airline flight travel planning via telephone. NIST’s work has included the creation of a centralized data collection system, and its future work will include supporting this system and assisting in the analysis of evaluation data. NIST has only recently become involved with this project, and it is too early to judge the impact of the overall project or the effect of NIST’s contributions. However, the Information Access Division should monitor progress on this work closely and should evaluate whether the work of the division is critical to the project and the tasks are appropriate for it to be engaged in.

In the Image Group, long-standing but evolving programs in the areas of fingerprint and face

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

recognition continue, and a new effort related to multimedia standards is under way. The fingerprint effort, which is supported by the Federal Bureau of Investigation, focuses on standards, databases, and technologies for use by federal, state, and local law enforcement. Within this program, a variety of tasks are usually under way, some nearing completion, some just starting up. This past year, work on the standard data format for the interchange of fingerprint, facial, and scar mark and tattoo information ended with the publication of the ANSI standard in September 2000, and the NIST special database on fingerprint minutia from latent and matching tenprint images was released in July 2000. Ongoing activities include the universal latent workstation and a public domain pattern-level classification automation system for fingerprints (PCASYS). The workstation project pushes the state of the art for computer-assisted law enforcement efforts, and PCASYS, a recently developed, cutting-edge system for fingerprint matching, is demonstrating substantial improvement in recognition over current technologies. To supplement these efforts, the division has begun several new projects this year. One is work on high-resolution fingerprint images where the focus is mainly on 1000 pixels per inch (ppi) images, and another is a timely, appropriate effort on sharing fingerprint data via the Internet. In this latter effort, the panel wonders if a collaboration between the Image Group and the Visualization and Usability Group could be established to work on effective ways to display the data.

Building on successful past work in the Face Recognition Technology (FERET) program of the mid-1990s, the Information Access Division has begun a new effort in human identification. This project is funded by DARPA and NIJ and is slated to run through 2002 with the possibility of being extended to 2004. It will take advantage of the division’s existing expertise in face recognition, shifting it to serve antiterrorist and anticriminal purposes and expanding it beyond just the face to incorporate characteristics of the whole body, including gait. As a first step, division staff have constructed a preliminary database of videos containing staged images of the whole body, gait, and face. The panel suggests that this collection should be expanded to include natural and crowd-scene video sequences. The panel also cautions that given the success of the FERET program for faces, NIST may be inclined to neglect the facial recognition element of the new human identification program. Since FERET largely examined only segmented faces, much challenging work remains to be done on both face and body recognition in a crowd, and results in this area will be interesting and important for the application of this work to the real world.

In an entirely new area, the Image Group is beginning to get involved in multimedia standards for MPEG and JPEG files. The goal is to help advance these standards by filling leadership positions on standards committees, hosting Web-based data repositories for these committees, and making a MPEG-7 system prototype available for testing and analysis. While it is too early to assess the technical merit of these efforts, the panel notes that the roles the division has outlined for itself in this multimedia area are very appropriate and closely related to the division’s existing expertise. Academic and corporate communities are very interested in multimedia standards, and past MPEG and JPEG releases have become the main standards for image and video, so NIST work in this area could have a broad impact. The only concern is that the type of activities planned for this project, such as standards committee participation and data collection, require a great deal of staff time, so management must be careful to ensure that the time allocated to these activities is adequate.

The Retrieval Group manages the Text Retrieval Conference (TREC) program, which includes the internationally renowned workshop series and the work on developing tests to evaluate the performance of information management systems. While this activity has been going on for a number of years, the specific elements of the NIST work continue to evolve. The focus has expanded from information retrieval techniques to general information management issues—for instance, How can questions be formulated and data be organized to more efficiently connect information seekers with the data they

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

need? The division is continually revising the testing and evaluation programs it provides in order to stay one step ahead of the commercial developers and meet the emerging needs of the international information management communities. This year, tasks that were concluded include the work on spoken document retrieval and European cross-language information retrieval, and new efforts were begun on video retrieval and Arabic cross-language information retrieval.

The Visualization and Usability Group is working mainly on tools to help people design more efficient Web interfaces. A primary focus is the Common Industry Format Testing of Usability Evaluation Reports (CIFter) project, which aims to create benchmark test data for Web usability evaluation methodologies. The first step is to identify which usability approaches are particularly successful, and the group recently released a CD of a static version of The Motley Fool Web site with guidelines for user testing. Data gathered from evaluation of this site will be collected in the form of CIF reports, and this first testbed will help NIST staff develop benchmarks for usability. This experiment has already demonstrated that recognizing and correcting inefficiencies can make a significant difference in usability. The high level of corporate and university interest in improved usability is demonstrated by the participation of Boeing and Microsoft in NIST work and by the array of organizations represented at the fourth Industry Usability Reporting Project Workshop, hosted by NIST in November 2000. The panel hopes that the Visualization and Usability Group’s efforts will help transform this interest into action, as many organizations have indicated a desire to enhance Web usability but noticeable improvements appear to be slow in arriving.

Several of the groups in the Information Access Division participate in the laboratory-wide pervasive computing program, which focuses on the integration of multiple technologies, such as speech and image recognition systems, and the networking of multiple data types, such as voice, image, and video. The overall objective for NIST is to speed progress in this field by enabling methods to test interoperability of components and to measure the relative performance of different integration approaches. In the Information Access Division, staff are working on projects related to the smart space concept for meeting rooms and the development of a pervasive computing testbed. In the past, the panel emphasized that the division needed to concentrate on facilitating the infrastructure as opposed to building components, and this year the division appears to have appropriately refocused its efforts on measuring interactions between devices. For example, a substantial accomplishment of the past year was the development of an application programming interface that will enable interoperable plug-in of algorithms and devices.

The pervasive computing program has components across the Information Technology Laboratory. While personnel are using the unique expertise and facilities available in the different groups to tackle diverse aspects of the pervasive computing problem, cross-divisional cooperation is still necessary. A regularly scheduled get-together or workshop might help staff share information informally and consult about issues where the experience of others may be relevant. Such free exchanges of information might help catch overlapping efforts or potential synergies early and could also increase the critical mass on various projects by effectively utilizing the expertise from across the laboratory. The value of cooperative efforts is not limited to interactions within a given program; for example, increased connections between pervasive computing and network protocol efforts might be productive for both efforts. In the area of biometrics, currently a focus of the Convergent Information Systems Division, stronger interactions with the Information Access Division personnel from the Image Group might prove fruitful.

Program Relevance and Effectiveness

In the Speech Group, the NIST-administered benchmark tests have produced significant benefits for the emerging commercial industry in automatic speech recognition. The NIST tests support companies’

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

efforts to advance the capabilities of their technologies, and the performance (i.e., the accuracy) of the speech recognition systems that were developed using the division’s tests has improved continuously over the years. IBM and Dragon Systems, two of the companies that participated in NIST benchmarking, currently have commercial speech dictation products on the market, and BBN/Verizon is developing a product in this area as well. The division’s efforts have clearly contributed to the quality of these products and to an efficient development process by the companies.

In addition to administering the benchmark tests developed at NIST, the Speech Group makes available software with which researchers can measure and provide diagnostic information on the performance of speech recognition systems. These software packages are very popular with developers from all sectors. For example, in the past year and a half, the Information Access Division’s standard scoring software, known as SCLITE, has been accessed from more than 638 domain addresses, including 106 .com addresses, 84 .edu, 22 .gov, and a large number of international addresses.

In addition to these projects on speech recognition system testing, the Speech Group also supports activities spearheaded by other government agencies. One example is the DARPA Communicator Project. Another is the work on conversational telephone speech recognition and speaker recognition for the NSA. NIST is providing test sets and performance benchmarks for this effort. The panel notes that the speech research community has been involved in such testing and benchmarking efforts for many years. At this time the performance of the relevant algorithms seems to have plateaued (the quality of the algorithms is low and improving slowly), and perhaps NIST should consider what impact it expects to have from this work.

In the Image Group, the human identification project’s focus on antiterrorism and criminal identification tasks makes it highly relevant to the current concerns of law enforcement communities. Protecting government buildings and airports via this type of technology serves important objectives for justice and society, although the panel notes that law enforcement agencies and security organizations will have to deal with privacy issues when they implement the technologies. The long-term impact of NIST’s work on this project cannot be judged yet, but the division has held two workshops and has involved over 20 universities and research laboratories in its efforts. This level of participation is a good sign of early interest in the division’s activities, and NIST should monitor outside involvement to ensure that these organizations’ connections with the division continue. Their participation will indicate if the project is producing results relevant to this community.

The Information Access Division’s fingerprint-related efforts are primarily targeted at federal, state, and local law enforcement communities. The relevance of this work for the FBI, and its satisfaction with past NIST results, is clearly indicated by the FBI’s continued financial support of the division’s projects, but the Bureau is not the only organization to benefit from NIST’s work in this area. Many agencies and companies have requested and used NIST results and derivations of NIST results to enhance their products and effectiveness. For example, the ANSI NIST fingerprint standard has already been requested by about 100 entities around the world. The databases advance the state of the art of current and new law enforcement systems, and the technologies developed at NIST or using NIST products can be transferred to other communities, as is happening in the biometrics area. In some of the division’s newer work, this crossover potential is particularly clear; the results of the high-resolution fingerprint images project should be of interest to both law enforcement communities and researchers working on personal authentication technologies.

The panel is very supportive of the ongoing fingerprint program and of the division’s efforts to ensure that the projects under way evolve to maintain NIST’s focus on cutting edge standards and technology work. However, given that this project is entirely supported by external funding, a great deal of attention must be paid to whether all of the tasks are appropriate for NIST and the Information Access

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

Division. The work being done is certainly relevant and valuable to NIST’s customers in law enforcement, but not all of it is necessarily advancing the state of the art. In the few cases where the division is merely executing a specific well-defined task for other agencies, management should seriously consider whether these projects (such as some of the latent fingerprint activities) should be transferred to other organizations, thereby allowing NIST to focus all of its energies and resources on more general research-oriented efforts.

The Retrieval Group hosts the well-respected TREC program. A primary reason the Information Access Division continues to devote significant time and effort to this program and its customers are so enthusiastic is that no group other than NIST could bring an appropriately neutral and unbiased approach to performance evaluating and testing tasks. Not only do commercial entities value the TREC meetings and products, but the NIST work provides important support to the federal government’s intelligence community (as can be seen by the Department of Defense’s significant financial contributions). Through TREC, the division is able to bring relevant parties from universities, companies, and the government together for a technical discussion of research and development efforts supported by accurate, unbiased data from NIST testing methods. The annual TREC meeting attracts a large number of people from around the world (in 1999, there were 170 invited participants from 16 countries), and the software developed at NIST is used and respected globally. The Information Technology Laboratory should take advantage of this international reach and reputation as it looks to increase its visibility.

The pervasive computing program is focused on very relevant technologies that are currently receiving a great deal of attention in the research community, specifically at universities. NIST has effectively reached out to build partnerships with a number of industrial and academic organizations for this program, and the level of attendance at the laboratory’s annual workshops is high (despite a snowstorm that prevented 114 registered attendees from getting to Maryland, the January 2000 meeting still had 160 participants). The pervasive computing effort has longer-range goals than other division projects, but its potential impact on industry and society is quite high. Preliminary groundwork has been laid for the NIST effort and should facilitate good progress in the coming years. As the number of workshops, papers, and systems produced by this program grows, the panel will be better able to assess the technological effectiveness of NIST work in this area.

Division Resources

Funding sources for the Information Access Division are shown in Table 8.5. As of March 2001, staffing for the Information Access Division included 39 full-time permanent positions, of which 33 were for technical professionals. There were also 9 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

The percentage of the division’s funding that comes from external sources (mainly other government agencies) is high and growing, from 32 percent in fiscal year 1998 to an estimated 54 percent in fiscal year 2001. While the outside funding in general supports relevant, important, and appropriate NIST activities, the panel cautions that the division must remain vigilant to ensure that its work with other agencies contributes to the NIST mission of advancing standards and measurement tools for new technologies and utilizes unique division expertise or facilities to play a critical role in the external projects. The panel is very supportive of the long-term externally funded activities in the division, including the TREC program and the fingerprint work; these efforts are appropriate and have significant impact on the communities that NIST serves in support of the U.S. economy. However, the Information Technology Laboratory is not to be discouraged from asking tough questions about the audience, general value, and technical stature of its continuing projects to ensure that it remains at the cutting edge

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

TABLE 8.5 Sources of Funding for the Information Access Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)

NIST-STRS, excluding Competence

4.6

4.5

4.6

4.5

ATP

0.4

0.2

0.1

0.0

OA/NFG/CRADA

2.3

3.2

4.0

5.4

Total

7.3

7.9

8.7

9.9

Full-time permanent staff (total)a

41

40

39

39

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

of relevant fields and focuses on emerging technologies and industries as needed. When possible, tasks that do not utilize the full potential of NIST’s expertise and facilities can and should be passed on to other agencies, industry or professional organizations, or private companies.

Convergent Information Systems Division

Technical Merit

According to division documentation, the mission of the Convergent Information Systems Division is to conduct research and development into integrated systems, architectures, and applications and infrastructure for the exchange, storage, and manifestation of digital content and to explore their scalability, feasibility, and realization for new applications.

The division plays a unique role within the ITL by focusing on the integration of systems applications and concepts from across the laboratory that can be brought together to develop prototype technologies. It then works with industry to develop relevant standards, interoperability protocols, and reference implementations, and these efforts are entirely consistent with the NIST and laboratory missions. The Convergent Information Systems Division was formed in October 2000 from the research components of the former High Performance Systems and Services Division, and the panel believes that creation of this much smaller but very focused division was an appropriate step that is allowing division staff and management to concentrate their energies on research projects relevant to industry.

The division runs high-quality programs supported by a staff with core competencies in hardware and software at both the component and system levels. This collection of expertise and NIST’s reputation for producing unbiased, first-rate technical results allow the division to serve as a bridge between traditional computer technologies and consumer markets, as it is currently doing in its work on digital television and on optical disk storage. The division comprises two groups (Distributed Systems Technologies and Information Storage and Integrated Systems), and division staff are currently working in seven areas.

In the Convergent Information Systems Division’s work on optical disk storage, the primary focus is on measurement methods and testing systems for disks based on optical technologies (e.g., digital

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

versatile disks [DVDs] and compact disks [CDs]). The overall goal is to enable and encourage interoperability between these storage media, which are growing in number, and the various devices designed to record on and play from them. Examples of the useful products the division has developed include a DVD optical reflectance reference disk (now being widely distributed by Warner) and a specification and test software to determine the interoperability of multiread DVD and CD-RW (read/ write) and various devices.

The division continues its strong efforts in biometrics. A past success in this area was the facilitation of the formation of the inclusive, industry-wide BioAPI consortium. Recent accomplishments include the definition of a biometric application programmer interface (BioAPI) specification and its adoption by industry; the specification will allow for the interoperability of biometric data objects and plug-ability of biometric devices and the development of a common file format for the exchange of biometric data and information. Now the division is building on past achievements and utilizing existing expertise to look at the interaction between smart cards and biometrics. For example, the division is supporting the U.S. military’s effort to select a viable system for personnel identification that utilizes both smart card and biometric technologies. As the biometrics program expands, staff could benefit from input from and collaborations with the Computer Security Division and the Information Access Division.

In the project on Digital TV Application Software Environment (DASE), the Convergent Information Systems Division continues to build strong relationships with industries interested in interactive digital television (DTV) and to develop and facilitate the use of relevant standards. A major accomplishment of this project is the development of a prototype reference implementation based on an industry-proposed specification for a DTV API. The work was timely, and NIST’s implementation helps assure the industry that a free, public version of the implementation will be available for vendors to exploit for commercial purposes. An important next step will be the development of conformance tests and testbeds in conjunction with staff from the Software Diagnostics and Conformance Testing Division. Other future plans for this project include setting up an authoring station and a testbed delivery chain for interactive DTV. Such a facility would allow small advertisers and businesses that could not otherwise afford access to such technologies to develop and test their interactive content before broadcasting, and it could jump-start a new segment of the market by speeding development of localized interactive content. Finally, the division may expand the reference implementation to incorporate APIs used in the OpenCable Application Platform (OCAP) specification. This effort would effectively utilize the DASE expertise available in the division and would benefit industry by helping vendors develop unified API implementations for receivers that aim to deliver content from terrestrial as well as cable networks.

The Convergent Information Systems Division has for several years played a key role in the emerging electronic book (e-book) industry. Recent accomplishments include the standardization of formats for such books, the development of browser software that reads content developed in conformance with those standards, and the development of a prototype Braille reader. The first two results have been widely disseminated to industry, and the last project is a technical breakthrough utilizing the cutting edge in electromechanical technologies. A key element of this successful program has been division staff’s ability to bring together the leaders of this emerging and fractious industry at conferences and to help build the consensus necessary for companies to embrace an industry standard.

A relatively new program for the division is the work on digital cinema. The entertainment industry has begun trials on ways they can use this new technology, such as for cinema transmission to theaters and for projection systems for display in theaters. The division is getting involved in this area because it believes that if NIST provides test and measurement services for digital cinema acquisition devices,

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

transmission facilities, storage devices, and display systems, the industry will be able to develop and commercialize new technologies faster and will be encouraged to create and use products that conform to standards that enable interoperability. To this end, the division plans to set up a state-of-the-art facility in which industry groups and vendors can test the pieces of the digital cinema chain. The panel believes such a facility is appropriate for NIST and applauds the staff’s work on the well-attended digital cinema workshop in January 2001.

The area of trust management and digital rights management could certainly benefit from standards. Consumers are suffering because many of the products on the market are not interoperable with different consumer electronics or information technology systems. However, there are a number of barriers to the successful development and adoption of standards. One is that the terms “trust management” and “digital rights management” (DRM) mean different things to different members of different industries. There is also a great deal of competition among vendors, each of which wants to establish its technology as the forerunner in these emerging markets, and a number of industry consortia and standards bodies that are trying, with little success, to impose order. For the division to work effectively in this area, it must make careful decisions about which results are most likely to be utilized by the industry and which partnerships are necessary and sufficient to encourage the widespread use of consistent standards. It also needs to utilize the expertise available in other divisions of the Information Technology Laboratory (e.g., the Computer Security Division) to ensure that the projects under way recognize the needs of the many relevant communities and that division staff understand the disparate languages used by the different industries.

To date, the Convergent Information Systems Division has focused specifically on developing a framework for financial agent secure transactions (FAST) that interoperably supports trustworthy exchanges between parties in transactions conducted over networks. This framework was developed in cooperation with government organizations, financial institutions, and e-commerce vendors. The panel believes that this project has been successful, and NIST’s work in this area was well received by the industry. Future efforts will focus on developing industry consensus around a common system for the exchange of trust messages and management of digital rights.

The panel is pleased to be able to report that, in addition to starting up new projects like the digital cinema projects, the division is also phasing out completed work on display characterization, cluster computing, and MultiKron time synchronization. The cluster computing project developed software to distribute computation tasks across a networked farm of processors or personal computers; this package has helped NIST researchers perform complex, computationally intense tasks faster and more efficiently. The MultiKron project developed a hardware reference design and Unix driver software that allows networkwide time synchronization with microsecond accuracy. This is a useful tool in distributed systems and may be relevant to the new work on trust management, where accurate, networkwide agreement on time is an essential element of building and managing trust. In all of these projects, the division will cease work altogether or will maintain a minimal level of effort to ensure that NIST staff who utilize these capabilities continue to be adequately served.

Overall, the panel believes that the quality of work done in division projects is high. However, no metrics were evident that would allow division management or the panel to quantify the quality of project results. Such metrics would allow the division to benchmark its results and then, as the division matures, work against this baseline to drive improvement of the products it delivers to industry. The division might consider investigating whether the various metrics available for assessing the quality of software development project deliverables, like the Capability Maturity Model developed at the Carnegie Mellon Software Engineering Institute, are applicable to NIST efforts, or whether seeking ISO certification for specifications and test systems developed at NIST would be valuable.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Program Relevance and Effectiveness

The Convergent Information Systems Division delivers a variety of results that are relevant to and valued by industry. Projects under way perform three main functions for NIST customers: working with industry to define and develop standards; developing benchmarks, infrastructures, and facilities for testing and measurement; and evaluating technologies and making recommendations to the U.S. government. Below the panel discusses the activities in the division and how they impact (or may impact) industry. Several of the projects are fairly new, so it can be difficult to assess customer satisfaction because NIST results have not yet been completed or transferred to industry.

The division’s work on optical disk storage is aimed at increasing interoperability of media and devices. This growing market could fragment quickly if the many competing (and incompatible) technologies for DVD recording solutions continue to be marketed. NIST is a neutral party, so the division is in a good position to develop standards, measurement methods, and test systems that are free of the biases that currently hinder industry cooperation on efforts to produce a merged specification for recordable DVDs. For example, the projects on a DVD optical reflectance reference disk and on a specification and test system for multiread DVD and CD-RW systems have both resulted in unique products that are of great value to the companies that manufacture disks and devices as well as to the consumers, who benefit from increased system compatibilities. The work on testing the lifetime of DVD-R disks using weather and stress cycles is particularly relevant to government agencies and other users of optical storage technologies, as lifetime data will help these customers determine if the medium is appropriate for long-term storage of important data.

A slightly unusual element of the optical disk storage program is the division’s involvement with a small company that is developing and marketing a new kind of optically based storage medium, the fluorescent multilayered disk. This technology is attracting interest from industry stalwarts like Philips, and by developing measurement and testing systems for fluorescent multilayered disk systems early in the technology development process, the division is prepared to help establish a new technology that could be integrated effectively with recognized standards and systems, like the increasingly popular DVDs.

In the biometrics and smart cards area, the division’s leadership in organizing workshops has brought together interested parties, including vendors, government organizations, and industry consortia. The biometric file format and the BioAPI specification will provide the technical elements needed to foster interoperability between devices and systems in this growing industry.

The Convergent Information Systems Division’s work on a prototype implementation for the DTV API is a very important project, highly responsive to industry needs. By providing a free, public reference version of the implementation, NIST facilitates industry’s acceptance of the standard, which in turn will jump-start commercial efforts in this area by making it easier for companies to quickly bring interactive television products to market. The easy availability of the source code for this implementation is what makes it so attractive to industry. The panel is somewhat concerned that issues related to intellectual property restrictions by Sun Microsystems have delayed the widespread transfer of this implementation to industry, but it is hopeful that efforts of the division, the Advanced Television Systems Committee (ATSC), and Sun to resolve the issue will be successful and broad dissemination can commence. Even with the restrictions, some vendors have already begun using the NIST source code to start prototype implementations of commercial products. An important element of the DASE project has been the division’s organization of workshops. The First Annual DASE Symposium was held in May 2000; it attracted participants from throughout the relevant industries and helped bring together factions in the emerging interactive television industry. The second such symposium will be held in June 2001, and it, too, will be organized and hosted by NIST.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

The work on electronic books is the most mature of the division’s active projects, and the effects of the NIST work on the burgeoning electronic book industry are a good demonstration of the value this division’s work can have. The standard format for digital books will provide a basis for interoperability between content products and reading and display devices. The Open EBook browser software, which is a free, public reference implementation for reading the standard format, will allow content vendors to test their products and be assured that the technology to read content authored in the EBook format exists. The division expects commercial versions of this browser, based on the NIST-developed implementation, to be available soon. The prototype Braille reader will significantly reduce the cost of these readers (the technologies demonstrated in this prototype will help lower the price of such products by thousands of dollars) and will offer a much easier reading experience for the visually impaired. Some vendors are already considering commercialization of the division’s model.

The project on digital imagery and printing is very new, so it is difficult to assess the sophistication of this program relative to the state of the art or to describe effects of the NIST work. However, the panel does believe that the area is an appropriate one for NIST to explore and that the potential for impact is significant. The project is working on the following problem: When an image of an object is viewed on a monitor or printed on paper, there is no guarantee that the colors in the image will match those of the object in real life. This problem is of particular interest to companies involved in e-commerce and e-entertainment and their customers. Currently the division is investigating the relevant technologies available in industry and is positioning itself appropriately to help define a new standard in this area.

The digital cinema project is aimed at customers like movie producers and distributors and makers of display equipment. The division has done an excellent job of reaching out to this community to educate them about the need for and value of standards. A key accomplishment was the organization of the first Digital Cinema Conference at NIST in January 2001. This meeting brought together users and vendors from across relevant industries and not only increased communications among them but also established NIST as the focal point for industry’s efforts to achieve consensus on standards and common formats.

The area of trust and digital rights management is relevant to a number of industries. In the information technology community, various trials are ongoing to validate mechanisms for delivery of digital content in which the content is secure, copyrights are safeguarded, and payments are securely made to the right players along the delivery chain. NIST’s work in this area began narrowly with the effort on the FAST framework, where the primary customers were the financial and distributed electronic commerce industries, who will certainly benefit from a common standard. As this project expands, NIST expects to establish a DRM testbed to enable systems to be tested for interoperability and standards conformance across content, products, and transaction systems relevant to a number of industries.

Since the trust management effort could be relevant to the needs of the growing number of industries interested in distributing content digitally, the panel suggests that the Convergent Information Systems Division might consider expanding its participation in some of the standards groups, such as the Copyright Protection Technical Working Group (CPTWG), the Secure Digital Music Initiative (SDMI), and the TV Anytime Forum’s DRM work. There are many issues on which NIST could have an impact. One is the diversity of DRM solutions and the lack of interoperability between “secure container formats” and playback devices from different vendors. Here, organizing a workshop on electronic content distribution (ECD) might be an effective way to bring parties together and foster discussions of interoperability; putting together an ECD chain on which industry vendors could test their content and systems for interoperability might encourage the vendors to get involved. The model for this effort would be the division’s successful work on the Bio API and the biometric data exchange file

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

format specification. Another issue is how to evaluate and compare various DRM solutions. Methods for measuring these systems’ requirements in terms of “security instructions per second” would help device manufacturers select appropriate DRM systems for their target applications. As noted in the section on technical merit, this is a very contentious area, where a number of groups are operating with conflicting goals and not necessarily pure motives. It is important that the division not be caught in the cross fire.

In all of these activities, the dissemination of NIST results to relevant industries is of the utmost importance if the division is to play one of its primary roles: catalyst for industrial development and the commercialization of new technologies. The panel is very impressed with the amount of outputs from this rather small division, whose staff, in fiscal year 2000, produced 20 publications, gave 33 talks, and organized 5 NIST-sponsored conferences. Important products of the division’s work also include measurement systems and specifications like those set up for optical data storage media and devices and for display characterization. The research activities in the convergent systems area have also resulted in six patent actions over the last 4 years.

In addition to producing and disseminating results, the division staff communicate and build relationships with relevant industries through their participation in standards organizations, sponsoring of symposia, and collaborations with universities, companies, and industrial consortia. The panel is particularly impressed by the division’s work on fostering industrywide cooperation and facilitating adoption of standards via groups and conferences like the Open Electronic Book Forum, the Biometric Consortium, the Digital Cinema Conference, and the Annual DASE Symposia. Participation in standards groups like the ATSC has been valuable, and the panel encourages the division to get involved in even more external organizations, if possible. Some of the groups were mentioned above: They include CPTWG, SDMI, TV Anytime, and OCAP, as well as the DVD Forum and the Society of Cable Telecommunications Engineers.

Division Resources

Funding sources for the Convergent Information Systems Division are shown in Table 8.6. As of March 2001, staffing for the Convergent Information Systems Division included 13 full-time permanent positions, of which 11 were for technical professionals. There were also 15 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

The Convergent Information Systems Division is very young, and funding levels are still somewhat in flux. The panel did not see any evidence of projects being scaled back or not begun as a consequence of scarcity of resources or lack of access to equipment or facilities. However, given the small size and brief history of this division, shortfalls may occur due to the uncertainties associated with funding from ATP and other government agencies. The area most likely to be cut back if overall funding is lacking is support for travel, training, and related activities. The panel would be particularly disappointed if this were to occur, as the division is to be congratulated for having spent sufficiently in the past to keep staff up to date on the latest technological advances and in touch with the relevant communities. As noted above, active participation by staff in industry consortia and conferences is an important element in the division’s ability to foster standards development.

Like the funding, the staffing levels for the division’s projects are somewhat uncertain. In some areas, the number of personnel is adequate, while in others, the projects are slightly below critical mass (for example, two more people are needed for the digital cinema work, two more in the trust management and DRM area, and one more in both biometrics and optical disk storage). This situation may be resolved when some of the projects that are winding down free up staff time. Currently, the division

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

TABLE 8.6 Sources of Funding for the Convergent Information Systems Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)a

NIST-STRS, excluding Competence

2.4

2.6

2.2

2.2

Competence

0.1

0.0

0.0

0.0

STRS-supercomputing

9.5

9.9

10.0

0.6

ATP

0.5

0.6

0.8

0.4

OA/NFG/CRADA

0.4

0.4

0.8

0.4

Other Reimbursable

0.7

0.0

0.8

0.0

Agency Overhead

5.3

6.7

7.4

0.0

Total

18.9

20.2

22.0

3.6

Full-time permanent staff (total)b

71

75

81

13

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe significant difference between the FY 2000 and FY 2001 funding and staff levels reflects the reorganization of the Information Technology Laboratory, in which the information technology service groups and the visualization group were moved out of this division to the Information Services and Computing Division and the Mathematical and Computational Sciences Division.

bThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

relies heavily on student interns and guest researchers. This dependence does have a downside during periods of transition, when progress is slowed as experience and responsibilities are transferred. However, morale in the division is high and retention of permanent staff is good, so the division does have a stable core of personnel. Also, the division programs that bring in external personnel for fixed periods of time have proven very successful in attracting international and young talent that add to the division’s capabilities and advance NIST programs, while providing the visitors with valuable experiences in science and technology during the formative period of their careers.

The panel was pleased to learn that the division has developed a business plan to describe its program goals and the potential impact of its work over the next 5 years. The only concern is that— given the fluid nature of the technologies that the division works with and the rapid changes in focus these industries often undergo—a business plan, which usually describes costs and return on investments for a clearly defined product or technology, is not an appropriate way to describe 5 years’ worth of division activities. A vision or strategic plan might be a more suitable way to provide guidance for divisional programs and planning efforts, with business plans drawn up for specific established projects or for the whole division on shorter time scales. In any case, the panel does encourage these sorts of plans, as they will help the division orient its programs and results so that they meet customers’ needs and will help the division manage resources more efficiently. A description of the potential impact on industry or society of the division’s projects is certainly useful, both to remind staff of the ultimate goal of their work and to explain the value of NIST to external organizations. Descriptions of impact may include estimates of the economic value in time saved or increased revenues for companies that utilize the division’s results.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

Information Services and Computing Division

Technical Merit

According to division documentation, the mission of the Information Services and Computing Division is to improve the productivity of NIST staff through the efficient and effective delivery of computing technologies, information management, and support.

The division’s mission and vision directly support the goals and objectives presented in the laboratory’s strategic plan.7 One of the primary recommendations in last year’s assessment was that laboratory management should review the responsibilities and charters of its various computing support organizations with the goal of developing a more coherent, effective, and less confusing system through which IT services could be provided throughout NIST. The panel is pleased to report that NIST has been very responsive to the panel’s suggestions and that there has been significant progress in the right direction. One step is the recent reorganization of the Information Technology Laboratory to place all of the service groups within one division. Another step is the effort to define an institution-wide vision for IT services at NIST. The panel was very pleased with what it observed during this year’s assessment but does caution that expectations are high and that NIST is just beginning the long and difficult process of putting a stable, practical system for IT support into place.

The work on defining an institution-wide vision for information technology (IT) services has two elements: a new IT architecture plan (ITAP) and a desktop support plan. In early 2000, NIST reorganized an IT Services Planning Team (ITSPT), consisting of personnel from across NIST, to tackle issues related to IT support. The panel was pleased to see that personnel from the service division were more actively involved in this team’s work over the past year, as recommended in the last assessment report. Currently the ITSPT, assisted by external consultants and laboratory personnel, is developing the ITAP. This NIST-wide plan will specify which systems should be centrally supported and which should be supported locally by the individual groups, divisions, or laboratories that are their primary users. Management believes that the ITAP framework will provide a sustainable mechanism for identifying and documenting IT architecture specifications throughout NIST. The panel believes this effort is commendable and necessary and is looking forward to reviewing the ITAP upon its completion later this year. Another element of the effort to put a shared vision for IT services in place at NIST is the work of the Office of the Chief Information Officer and the ITSPT to develop an expanded plan for desktop support services. This plan will identify standards for the service functions and will define a new business model for how the services will be provided. A clear plan should improve the ability of division staff to provide quality, cost-effective support for the many desktops throughout NIST.

The panel believes that this effort to develop the ITAP and the desktop support plan, which is being undertaken in order to have a clear and uniform blueprint describing NIST-wide IT services, is an important goal for both the laboratory and NIST. Without these plans, service will be spotty, ineffective, and wasteful of resources. A strong information technology infrastructure is crucial to the basic mission of NIST, and the Information Services and Computing Division should play a strong role in developing, supporting, and improving this infrastructure. Managers and staff throughout NIST need to develop and buy into a shared understanding of what common IT services the Information Services and Computing Division will be responsible for providing. Once the expectations have been clearly defined

7  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Information Technology Laboratory Strategic Plan, National Institute of Standards and Technology, Gaithersburg, Md., April 2000.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

and are understood by all, NIST must allocate the appropriate level of resources to the division to allow it to accomplish its defined objectives. In this environment, the division can then be held accountable and expected to efficiently provide the agreed-on level of services.

These efforts to clarify the responsibilities of the division are timely, in that the ITL has recently been restructured to unify all service functions within one division. The new Information Services and Computing Division includes all of the service groups in the old Distributed Computing and Information Services Division, as well as the service-oriented groups from the old High Performance Systems and Services Division and some support personnel from other laboratories at NIST. The new division now consists of six groups: Enterprise Systems Administration, Web and Internet Technology Services, Administrative Computing Support, PC Support, High Performance Systems, and Network and Telecommunications Systems. These groups are providing important services to the NIST laboratories and administrative units and are currently involved in several activities to reduce the ongoing costs and improve the effectiveness and efficiency of the NIST computing infrastructure.

Examples of the key services supplied by division personnel include centralized high-performance computing capabilities, advanced connectivity, common centralized e-mail, shared calendars, centrally coordinated Web pages, help desk support, file servers, and site-licensed software. Such services are commonly required of IT support units in industry and academia, and the panel is pleased to see that the division coordinates these activities efficiently, using a low ratio of staff to customers. The panel notes that current initiatives, like the development of a consolidated help desk and a life-cycle planning model for administrative applications, are good and appropriate investments for the division.

Information security at NIST remains one of the highest priorities of the Director of the Information Technology Laboratory and the Deputy Director of NIST, and the Information Services and Computing Division is actively contributing to the comprehensive efforts to maintain the safety and security of NIST IT systems. The division operates and supports the NIST firewall program, the NIST security implementation and incident response program, and the new NIST PKI program. The division’s work is supplemented by a security working group within the ITSPT and by a NIST computer security officer, who provides oversight and direction.

Program Relevance and Effectiveness

The reorganization leading to the creation of the Information Services and Computing Division began in November 2000 and was finalized in early February 2001. This change in structure is so recent that the panel is unable to discuss any demonstrated impact of the consolidation of the service groups. However, the panel is very optimistic that the unified collection of personnel, the enhanced planning activities, and the new management team will together produce a significantly improved IT support environment for NIST in the coming years.

The Information Services and Computing Division is appropriately focused on ways to improve its activities and is currently putting together quality support analysis teams to lay the groundwork for a formalized implementation of Baldrige quality standards. One of the teams is looking at ways of determining and implementing meaningful performance metrics that address availability, quality, customer satisfaction, and cost issues related to IT services. This effort is particularly important because institutions wishing to assess the performance and value of their information technology support organizations currently lack meaningful and practical measurements. In the face of this need, the division could develop a useful tool for its work at NIST and at the same time contribute to industry at large by defining, using, and adapting better IT support metrics. In past reports, the panel recommended that the division take a leadership role in developing metrics to describe IT service users’ experiences, and the

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

current effort is a promising step in this direction. Initial results from the performance metrics team are expected in 2001, and the panel is very interested in seeing the outputs.

Division Resources

Funding sources for the Information Services and Computing Division are shown in Table 8.7. As of March 2001, staffing for the Information Services and Computing Division included 131 full-time permanent positions, of which 106 were for technical professionals. There were also 13 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

Recruiting and retention programs are working well. Over the past year, the division experienced approximately 5 percent turnover in its staff, a low rate for an IT support organization in a major metropolitan area. Although the division has lost a few key people, its initiatives in hiring and its embrace of flexible approaches such as telecommuting, when appropriate, seem to be effective, and morale appears to be high. The panel did observe that management at the group and division level was entirely male and encourages division and laboratory management to look for ways to increase diversity. Female leaders do exist at the team level.

Recently, the director of the Information Technology Laboratory was named acting NIST Chief Information Officer (CIO). It may be difficult for one individual to carry out the responsibilities of both positions. However, the value of having the IT service group integrated into the Information Technology Laboratory has been observed in the past, and the panel understands that the organizational location of the IT support unit was a primary factor in determining the CIO for NIST. The role of CIO does not

TABLE 8.7 Sources of Funding for the Information Services and Computing Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)a

NIST-STRS, excluding Competence

0.6

0.6

0.9

0.3

STRS-supercomputing

0.9

0.9

0.9

7.7

Measurement Services (SRM production)

0.0

0.0

0.0

0.1

OA/NFG/CRADA

0.1

 

0.0

0.7

Other Reimbursable

0.6

0.4

0.6

1.5

Agency Overhead

6.3

7.1

8.2

17.3

Total

8.5

9.0

10.6

27.6

Full-time permanent staff (total)b

62

72

77

131

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe difference between the FY 2000 and FY 2001 funding and staff levels reflects the reorganization of ITL in which the visualization group was moved out of the Convergent Information Systems Division and into this division.

bThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

appear to be clearly articulated, and the panel expects the responsibilities to evolve over time as the Department of Commerce clarifies what is expected of this person and as NIST progresses in defining the appropriate level of IT services to be provided by a central organization.

As noted in previous assessments, NIST does not appear to participate in the academic and industrial research community’s collaborative efforts to develop and utilize high-performance networks. Several laboratory research groups have stated that their dissemination and communication efforts would be significantly improved if they were able to connect to the world outside NIST at a higher bandwidth (e.g., it would be easier for external groups to access large reference data files such as those produced in the Information Access Division’s speech recognition work). The panel recommends that the division explore relationships with organizations such as the University Corporation for Advanced Internet Development, the university-led nonprofit consortium that coordinates Internet2 activities among research and academic institutions around the world. Involvement with such an organization would allow the division to determine if and how NIST staff could be provided with broader bandwidth. The division is already studying NIST researchers’ needs in high-performance scientific computing, and advanced networking technologies can be an important element in the delivery of high-performance computing capabilities.

Software Diagnostics and Conformance Testing Division

Technical Merit

According to division documentation, the mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, Conformance to standards, and correctness; to participate with industry in the development of forward-looking standards; and to lead efforts for Conformance testing, even at the early development stage of standards.

The division’s work in designing Conformance and diagnostic tests and developing reference implementations for standards bodies clearly fulfills its mission and is consistent with the goals of both the laboratory and NIST missions. The division’s philosophy is to concentrate on key areas at the forefront of technology (that is, to get involved early), to partner with industry (that is, to fill voids where companies cannot or will not work), to transfer relevant technology developed at NIST, and to move on to new projects. The panel finds this focus on timely involvement and close collaboration with industry appropriate and effective. The division is organized into three groups: Standards and Conformance Testing, Software Quality, and Interoperability.

The Standards and Conformance Testing Group develops Conformance tests and reference implementations, performs research into better ways to do Conformance testing, and develops standards jointly with industry. Some of the work is focused on standards for electronic commerce, such as the efforts on extensible markup language (XML) technologies and the XML Registry and Repository. In conjunction with an XML standards organization, staff are establishing the means by which software and systems will communicate over the Internet. This group is also responsible for the division’s component of the Information Technology Laboratory-wide program on pervasive computing. Initially, the focus was on demonstrating the feasibility of these embedded computing systems, but now the division has moved on to issues related to ensuring software quality for these complex systems. A current project is the development of a three-dimensional graphical simulation tool that will enable cost-effective measurement and testing of pervasive computing systems. The staff are also leveraging expertise and knowledge from their previous research on architecture definition languages (ADLs) in their work on rigorously specifying service discovery protocols. For example, simulation of the ADL

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

implementation of a standard protocol for pervasive computing has already led to the led to discovery of an inconsistency in the Jini protocol.

In addition to interdivisional work on pervasive computing, the Software Diagnostics and Conformance Testing Division also has a strong collaboration with the Statistical Engineering Division on software testing using statistical methods. This consists of a competence project funded by the NIST director’s office as well as a companion project within the Software Quality Group. The division’s work in this area is concentrating on component testing. This focus is appropriate, because the development of high-quality, reusable components would produce significant cost reductions in the software industry.

The Software Quality Group develops software diagnostic tools, reference data, and methods to automate software testing and also performs research in formal methods. The group’s testing research is making significant contributions to this field. In their work on the generation of automatic tests for software, staff have developed an innovative method that uses mutations or counterexamples to generate comprehensive tests from formally specified system requirements and are currently working on a metric to quantify the amount of coverage these tests provide. They are also developing systems that can translate between the different formal models used in industry, academia, and government; the hope is for these translators to make formal methods efforts much more interoperable and hence more widely used and effective. Overall, the Software Diagnostics and Conformance Testing Division is leading the way for industry and government in the usage of formal methods to analyze and improve standards specifications and develop conformance tests. Already the division is applying the formal methods developed during the automatic test generation project to the work in XML conformance testing and to the object-oriented portion of the competence project on software testing by statistical methods.

The Interoperability Group focuses on federal agency needs in interoperability. Its activities include providing technical support to voluntary standards committees and helping government and industry achieve interoperability through the application of products developed at NIST. Appropriate and useful projects under way include the National Software Reference Library (NSRL) and the work on verification of computer forensics tools. For the NSRL, the group is developing reference data sets that contain standard examples of common computer programs and files. The file signatures from these reference sets can then be compared to computer files that have been seized in the course of an investigation in order to identify unaltered common programs. By using the NSRL data set to eliminate a significant number (40 to 95 percent) of nonpertinent files automatically, investigators can concentrate on searching potentially relevant computer content, thus increasing their efficiency and saving hundreds of staff-hours. In a related project, division staff are planning to hold focus groups on defining quality requirements and aim to develop rigorous testing procedures for computer forensics tools. These specifications and tests will be used to ensure that the tools yield objective, repeatable, reproducible results that hold up in court. In other projects, staff from this group work through the federal Chief Information Officers Council to ensure that federal agency requirements are taken into account by voluntary standards committees and support the NIST paperless office effort utilizing digital signatures.

The panel continues to be impressed by the timely and effective way in which projects are phased out throughout the division. The software industry tends to change technologies and shift priorities rapidly, and the division has demonstrated that it is able to react both quickly and appropriately. This year, the projects on instructional management systems, real-time Java, and the error, fault, and failure repository were all concluded and the freed resources allocated to other projects.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Program Relevance and Effectiveness

The Software Diagnostics and Conformance Testing Division provides necessary technical leadership to industry through its work on standards, reference implementations, and conformance testing. In the standards area, the division’s efforts contribute significantly to software quality and interoperability, and many organizations and areas could benefit from the expertise available in this division. In fact, based on their reputation for helping to create high-quality standard specifications, division staff are invited to participate in numerous standards bodies and must constantly set priorities based on where the division can have the greatest impact. The panel continues to be impressed by the decisions the division makes in this area and by the effectiveness of its efforts.

The success of the division in the standards arena is due to two factors: (1) its early involvement in efforts to define standards and (2) its work on technical tools to support those efforts. Whenever possible, NIST staff develop reference implementations and conformance test suites, the tools needed to facilitate industry’s adoption of and compliance with standards, in parallel with the definition of the standards specification and well before industry implementations have begun. Taking advantage of NIST’s reputation for unbiased, high-quality technical advice, the division works within the community as a neutral party to resolve errors in the specifications and provide feedback on implementation issues before final standards are crystallized.

Currently, the Software Diagnostics and Conformance Testing Division is emphasizing activities related to standards efforts relevant to electronic commerce, including XML and interactive TV. Electronic commerce applications are growing exponentially, changing the way business is conducted and how consumers behave. The division is appropriately participating in standards committees and developing test suites in this area at a very early stage, and the value placed on its efforts is clear from testimonials offered by personnel from the organizations trying to establish standards in these emerging areas. The executive director of the Organization for the Advancement of Structured Information Standards (OASIS) writes: “The OASIS XML conformance work is greatly enhanced by the leadership of NIST. NIST’s mission is standards and testing. Their many years of experience in providing quality, comprehensive conformance test suites are a tremendous asset to OASIS.” The chair of the Society of Motion Picture and Television Engineers’ Declarative Data Essence Group is equally supportive of the NIST plan to develop an interactive television testbed: “PLEASE DO IT! Everyone I’ve talked to is excited about your proposal….” Given the ubiquity of the Internet and the many potential applications of electronic commerce technologies, it is not unreasonable to expect that the division’s work in this area will have an impact worldwide.

In all of the Software Diagnostics and Conformance Testing Division programs, the focus on emerging technologies and on maintaining close partnerships with industry ensures that NIST efforts will have a significant impact. By building strong connections with companies and industry organizations, the division can involve them in the collaborative development of tests and implementations so that technology transfer is not a hurdle tackled at the end of an activity but an ongoing effort, smoothly integrated throughout the life of the project. This approach allows the division to stay in close contact with industry participants and get continuous feedback on work performed and results delivered to companies and standards bodies. An example of how the division uses this model effectively is the work with Ford Research Laboratories on automatic test generation from formal specifications. The division’s work in this area will simplify the development of conformance tests and allow these tests to cover a wider range of potential faults, thus decreasing the number of errors and increasing software quality. Already a number of corporations, including Ford Motor Company Powertrain Division and Argus, a developer of secure operating systems, are using the reference implementation developed at

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

NIST to automatically generate tests from a formal description of their product requirements. Ford is now commercializing this product, and the division has successfully ensured the transfer of NIST results and broad dissemination of a useful tool.

Division Resources

Funding sources for the Software Diagnostics and Conformance Testing Division are shown in Table 8.8. As of March 2001, staffing for the Software Diagnostics and Conformance Testing Division included 35 full-time permanent positions, of which 32 were for technical professionals. There were also 15 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

Retention of staff in the Software Diagnostics and Conformance Testing Division has been reasonably good for a number of reasons. One is that the division provides personnel with a chance to work in interesting areas and allows them to participate fully in their projects. Management effectively shares information about the “big picture” with everyone in the division and includes even the most junior staff in all strategy meetings and interactions with NIST customers. (The fact that the division provides funding and encourages the travel required for building customer relationships also has a positive effect on morale.) Pay increases have been reasonable, and flexible working hours and granting of leave add to the sense of a supportive working environment. NIST provides the staff with a high degree of both autonomy and stability, and the division appears to be an attractive place to work. Some turnover has occurred in the past year, but morale remains high as those who departed did so to do different types of work (usually still within NIST) or retire, while the new people all are very excited to be at NIST. The technology labor market underwent some significant changes in the year 2000, including widespread decreases in compensation based on the dropping value of stock options. The division may benefit from these changes, as the stability of NIST may make it an attractive alternative for many qualified potential employees.

TABLE 8.8 Sources of Funding for the Software Diagnostics and Conformance Testing Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)

NIST-STRS, excluding Competence

4.6

4.8

4.8

4.7

Competence

0.5

0.6

0.5

0.4

ATP

0.4

0.4

0.6

0.2

OA/NFG/CRADA

2.2

0.6

1.0

1.8

Other Reimbursable

0.2

0.0

0.0

0.0

Total

7.9

6.4

6.9

7.1

Full-time permanent staff (total)a

41

39

37

35

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

The Software Diagnostics and Conformance Testing Division continues to be hampered by the slowness of some NIST and Department of Commerce support functions. While there do not appear to be any immediate difficulties concerning the amount of capital resources, staff have remarked that equipment can be very difficult to procure, even when authorization has been obtained and funds are available. Long delays in procuring equipment can seriously hamper the division’s ability to support rapidly changing technologies, particularly in the hardware-dependent pervasive computing effort. Also, the poor responsiveness of the Department of Commerce legal department remains an ongoing problem. Although the 3-year struggle to officially join the World Wide Web Consortium appears to finally have been resolved, there is no sign of any improvement in the division’s ability to enter into a legal agreement in a timely manner. The division needs this capability because NIST’s impact in software standards is dependent on working with industry (often via consortia arrangements) early in the standards development process.

Statistical Engineering Division

Technical Merit

According to division documentation, the mission of the Statistical Engineering Division is to advance measurement science and technology by collaborating on NIST multidisciplinary research and by formulating, developing, and applying statistical methodology for the collection and analysis of data critical to NIST scientists and engineers.

The division continues to be involved in a broad range of activities including collaborative research with scientists at NIST, promulgation of measurement standards, experiment planning and inference, and development of new statistical methodology. While division staff are making important contributions to projects in the Information Technology Laboratory, a large portion of the division’s current efforts (about 80 percent) involves work with people outside the laboratory. This far-reaching array of collaborations represents a continuing shift of the division’s focus back to the tradition of working on important problems across all of NIST.

As in past years, the Statistical Engineering Division is successfully promoting the use of state-of-the-art statistical methods in metrology, experimentation, and data analysis across NIST. The staff are also involved in collaborative research that has led to the development of new methodologies. The projects described below show the value and diversity of the division’s contributions over the past year.

The work on characterization of high-speed optoelectronic devices is an excellent example of a research project that is making fundamental contributions to an important scientific problem while also developing cutting-edge statistical methodology. In the Optoelectronics and Radio Frequency Technology Divisions of the NIST Electronics and Electrical Engineering Laboratory, staff are working on techniques to accurately measure the performance of high-speed optoelectronic devices, such as photodiodes and sampling oscilloscopes. These techniques are critical in the design of high-performance systems that take advantage of the potential bandwidth of optical fiber. Statistical Engineering Division staff have contributed to the NIST efforts by developing statistical signal processing techniques to correct for the effects of time-base distortion, timing jitter, and impedance mismatch in measurements using high-speed optoelectronic detectors. Recent accomplishments include new results on jitter estimation and uncertainty analysis of time-base distortion. The productivity of this effort is demonstrated by the seven publications (three in refereed journals and four in conference proceedings or technical digests) that NIST staff produced in this area in 1999 and 2000.

Another project with EEEL is the work on new measurement methods for characterizing the permit-

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

tivity and loss tangent properties of dielectric materials. Statistical Engineering Division staff are contributing by developing statistical methods for the design and analysis of experiments, optimal estimation of dielectric property parameters, and quantification of uncertainty.

The division’s work can have international implications. Staff are working with Building Environment Division personnel from the NIST Building and Fire Research Laboratory on a multiyear assessment of the comparability, equivalence, and traceability of thermal conductivity hot-plate measurements at the national laboratories of five countries. NIST staff have designed experiments and analyzed data for these international intercomparisons, and the results have demonstrated that one of the laboratories consistently produced outlying values and that laboratory-material interactions exist. The ultimate goal of this project is to help establish worldwide equivalence for hardware and protocols used in hot-plate measurements as well as to define a globally accepted international reference material in thermal conductivity.

The work on comparing cigarette ignition properties is also in collaboration with the Building and Fire Research Laboratory. Prompted in part by the Cigarette Safety Act of 1984, past NIST work empirically established the feasibility of developing fire-safe cigarettes and led to the development of standardized test methods for assessing ignition propensity. Over the past year, at the request of the Federal Trade Commission, NIST staff have been studying how experimental cigarettes and test equipment have changed over time and evaluating the ignition properties of commercial cigarettes.

Division staff are not neglecting good opportunities for appropriate collaborations within the Information Technology Laboratory. One example is the work on ranking algorithms for face recognition with staff from the Information Access Division. The goal of this project is to develop a test suite that will allow researchers to compare the performance of various algorithms for identifying still images of humans without detailed knowledge of the algorithms. While this effort is still in its early stages, division staff are contributing to progress on the project by providing advice on database development and empirical evaluation of the algorithms and systems, investigating various relevant techniques, and developing new methods for comparing and ranking the algorithms.

The collaborative nature of the division’s work is not limited to cooperative efforts within NIST. One of the key activities currently under way in the division is the NIST/SEMATECH Engineering Statistics Internet Handbook. This 5-year project is producing a Web-based book intended to help practitioners design experiments and analyze data using good statistical practices, even when they cannot work closely with an experienced statistician. The book covers applications in a variety of areas, including measurement process characterization, process monitoring and improvement, and reliability. While this reference will certainly be helpful to many of the division’s partners at NIST, the intended audience is much broader, and preliminary versions8 of the handbook have been received enthusiastically by both industrial practitioners and the academic community. Current plans call for the final version of the Handbook to be released in May 2001 and to be featured in a session at that month’s Quality and Productivity Research Conference.

The above projects are just a few examples of the many diverse activities under way in the Statistical Engineering Division. Division scientists have also been involved in research on statistical methodology and inference, investigating basic questions stimulated by the applications at NIST. They continue

8  

The NIST/SEMATECH Engineering Statistics Handbook is available online at <http://www.itl.nist.gov/div898/handbook/index.html>.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

to provide statistical consulting on the many measurement services provided by NIST, and they also reach out to other parts of the U.S. government to supply statistical advice needed in areas such as prediction of retirements at NIST and impartial methodologies for draft lottery selections. An emerging effort for the division is increasing NIST’s role in international metrology and standards. As noted above, staff are already working with other NIST laboratories on international intercomparisons. Two division staff members are serving on standards committees, and some collaborations with foreign standards organizations are being developed. The division is also represented on the NIST-wide task group on voting standards, which is investigating how NIST might productively play a role in the national discussion on improving election systems.

Program Relevance and Effectiveness

The efforts of the Statistical Engineering Division have a broad impact on the work of the NIST scientists and engineers with whom they collaborate and also on industry practices in general. Looking again at the examples described above, the approaches developed during the project on characterization of high-speed optoelectronic devices have been incorporated into a new measurement service that will benefit industries involved in optical fiber and wireless communications as well as in ethernet and fiber channel networks. The work on characterizing dielectric materials will facilitate the design of new devices and may yield significant savings in industry’s research and development costs. The work on fire-safe cigarettes is not industry-focused but deals with an important public safety concern and is consistent with the NIST mission to improve the quality of life. The NIST/SEMATECH Engineering Statistics Handbook will certainly increase the productivity and accuracy of the scientists and engineers who use it, and it also has potential value as an educational tool.

The division’s primary mode of operation is collaborative, and through these collaborations, the division’s results are immediately communicated to and used by relevant parties. However, as mentioned above, the methodologies and outcomes of the division’s work are often relevant to a wider audience, making publications and presentations a necessary element of dissemination efforts. Over the past year, division staff published about 30 articles in refereed journals and 36 papers in conference proceedings or other outlets, and 25 more articles have been submitted. This level of output is very good, and the publications are complemented by the staff’s involvement in professional activities, including talks at conferences, memberships on committees, and service on editorial boards.

The Statistical Engineering Division is certainly producing quality work that appropriately supports the NIST mission. However, the panel believes that the impact of the division could be greatly enhanced. The long-term challenge for NIST is to reestablish the division’s stature as a premiere statistical research and consulting organization. A concerted effort is needed to bolster the division’s profile and visibility among professional statisticians in order to restore the division’s standing, which has slowly declined over the past 30 years. Reclaiming a reputation for excellence and a role as a national resource is certainly possible. Few other groups in the United States have access to the range of interesting and challenging statistical problems accessible to the Statistical Engineering Division. The division’s recently revised long-term plans include placing a higher priority on publishing papers in statistical journals, participating in professional activities, hosting statistical symposia and workshops, and developing internship programs with academia, and the panel endorses these tactics. In fact, the panel firmly believes that the revitalized Statistical Engineering Division can play a pivotal role for U.S. industry by promoting industrial statistics and by helping to link key statistical groups in academia and industry and at U.S. national laboratories.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

TABLE 8.9 Sources of Funding for the Statistical Engineering Division (in millions of dollars), FY 1998 to FY 2001

Source of Funding

Fiscal Year 1998 (actual)

Fiscal Year 1999 (actual)

Fiscal Year 2000 (actual)

Fiscal Year 2001 (estimated)

NIST-STRS, excluding Competence

2.8

2.9

3.0

3.1

Competence

0.3

0.5

0.6

0.6

STRS-supercomputing

0.5

0.1

0.0

0.0

Measurement Services (SRM production)

0.0

0.0

0.0

0.3

OA/NFG/CRADA

0.0

0.1

0.1

0.3

Total

3.6

3.6

3.7

4.3

Full-time permanent staff (total)a

21

23

19

17

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

aThe number of full-time permanent staff is as of January of that fiscal year, except in FY 2001, when it is as of March.

Division Resources

Funding sources for the Statistical Engineering Division are shown in Table 8.9. As of March 2001, staffing for the Statistical Engineering Division included 17 full-time permanent positions, of which 15 were for technical professionals. There were also 6 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers.

The panel is delighted to report that, after 3 years of searching, the Statistical Engineering Division now has a permanent chief. The new chief has been at NIST for less than a year but has already had a major positive impact on the atmosphere, morale, and functioning of the division. She has involved division members in discussions of how to do long-term planning, organize individual project efforts, set standards, and establish balance among division programs. The division is also developing a structured performance appraisal process and an organized approach to recognizing staff contributions, both of which should facilitate priority setting, address staff concerns about being overloaded, and ensure that personnel receive full and timely credit for their efforts. The panel commends all of these programs and notes that they are consistent with recommendations made in past assessments. The new division chief is providing exactly the right kind of direction needed to move this division forward.

The biggest and most immediate challenge facing the division is the need to rebuild the staff by recruiting high-quality people with the right mix of skills. Areas that are especially important include statistical computing, large databases, and Bayesian inference. A major obstacle to the division’s recruiting efforts is the fact that NIST’s starting salaries for new Ph.D.s are not competitive with what they could obtain in industrial research laboratories. This gap, which is substantial, is particularly severe for researchers working at the interface between computer science and statistics. The division needs sufficient salary flexibility to attract the best candidates at the entry level. In addition, resources and slots should be made available to allow the division to recruit at a senior level, where several key people left NIST in the past few years. While the panel encourages the division, with the support of

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×

Information Technology Laboratory management, to actively and aggressively seek to hire people, it cautions that it will not be possible, or even wise, to fill all of the open slots in 1 or 2 years. Restoring the division to full strength will take at least 3 to 5 years, and management at all levels should be committed to sustaining the recruiting efforts over a long period.

The panel continues to be concerned about the relative isolation of the division at NIST North. For the staff to be physically separated from their partners in the NIST laboratories negatively affects their productivity and the effectiveness of their collaborations. It may also impede recruiting. Given the interdisciplinary nature of the division’s work and the value of its contributions to projects throughout NIST, the panel once again urges NIST management to thoroughly investigate all possible approaches to relocating the Statistical Engineering Division to appropriate quarters on the main campus.

MAJOR OBSERVATIONS

The panel presents the following major observations:

  • The panel is extremely pleased with the progress made in the Information Technology Laboratory since the last assessment and appreciates the responsiveness and openness of laboratory management and staff to the panel’s requests and suggestions.

  • Increasing the visibility of the work done in the Information Technology Laboratory is an important goal for management. Enhancing the laboratory’s reputation outside NIST will improve and expand customer relationships. Publications in respected journals and presentations at quality conferences are key elements in this outreach effort.

  • Industry is increasingly using closed consortia instead of open processes to develop standards. In some cases, these closed groups are fairly inclusive and can be the most effective forum for NIST staff to impact industry standards. The Information Technology Laboratory should consider developing a policy on when participation in closed consortia is appropriate as well as on how NIST can encourage industry to utilize open, or at least inclusive, approaches to standards development.

  • The unification of the information technology support functions in the new Information Services and Computing Division was an excellent move, and the panel expects the new organization to provide a better structure for delivering more coherent services to NIST. However, questions remain about what exactly the responsibilities of this division will be and what levels of staffing and resources will be necessary for the division to meet its obligations. The panel urges the division and the Information Technology Services Planning Team to conclude work on the new Information Technology Architecture Plan in a timely manner so that the level and type of NIST-wide services can be defined and communicated to users and division staff alike.

  • The Mathematical and Computational Sciences Division and the Statistical Engineering Division are still located in NIST North, despite reasonable concerns that this separation negatively affects the ability of staff in these divisions to collaborate with scientists on the main campus. The panel strongly urges laboratory and NIST management to thoroughly explore creative solutions to this problem and to communicate these efforts to division personnel. This issue needs to be resolved.

Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
This page in the original is blank.
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 229
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 230
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 231
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 232
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 233
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 234
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 235
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 236
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 237
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 238
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 239
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 240
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 241
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 242
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 243
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 244
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 245
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 246
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 247
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 248
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 249
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 250
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 251
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 252
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 253
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 254
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 255
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 256
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 257
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 258
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 259
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 260
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 261
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 262
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 263
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 264
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 265
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 266
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 267
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 268
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 269
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 270
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 271
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 272
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 273
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 274
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 275
Suggested Citation:"8. Information Technology Laboratory." National Research Council. 2001. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001. Washington, DC: The National Academies Press. doi: 10.17226/10204.
×
Page 276
Next: 9. Microelectronics »
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2001 Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume represents the 42nd annual assessment by the National Research Council (NRC) of the technical quality and relevance of the programs of the Measurement and Standards Laboratories of the National Institute of Standards and Technology (NIST). This report provides judgments regarding the overall state of the NIST Measurement and Standards Laboratories (MSL),and offers findings to further increase the merit and impact of NIST MSL programs. It also offers in-depth reviews of each of the seven laboratories of the MSL, with findings aimed at their specific programmatic areas.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!