National Academies Press: OpenBook
« Previous: 7 Building and Fire Research Laboratory
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Chapter 8

Information Technology Laboratory

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

PANEL MEMBERS

Brian W. Kernighan, Lucent Technologies, Co-Chair

Richard E. Nance, Virginia Polytechnic Institute and State University, Co-Chair

Bishnu S. Atal, AT&T Laboratories Research

Mary Ellen Bock, Purdue University

Allen L. Brown, Jr., Cinebase Software, Inc.

Josephine Cheng, IBM Santa Teresa Laboratory

Dorothy E. Denning, Georgetown University

James L. Flanagan, Rutgers University

Roscoe C. Giles, Boston University

Andrew S. Grimshaw, University of Virginia

Carl M. Harris, George Mason University

Thomas P. Kehler, Conclusive Software, Inc.

Sandra M. Lambert, Lambert & Associates

Radia Perlman, Sun Microsystems

James L. Phillips, Boeing Shared Services Group

Jerome Sacks, National Institute of Statistical Sciences

Ahmed N. Tantawy, IBM Corporation

Raymond T. Yeh, International Software Systems, Inc.

Eddie L. Zeitler, Charles Schwab & Company

Submitted for the panel by its Co-Chairs, Brian W. Kernighan and Richard E. Nance, this assessment of the fiscal year 1998 activities of the Information Technology Laboratory is based on a site visit by the panel on March 17–19, 1998, and documents provided by the laboratory.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

LABORATORY-LEVEL REVIEW

Laboratory Mission

The mission of the Information Technology Laboratory (ITL) is to “promote the U.S. economy and public welfare by providing technical leadership for the nation's measurement and standards infrastructure for information technology.” The Information Technology Laboratory carries out this mission by working with industry, research, and government organizations to develop and demonstrate tests, test methods, reference data, proof of concept implementations, and other infrastructural technologies that are needed by U.S. industry to produce information technology systems that are usable, secure, scalable, and interoperable.

The ITL's role includes four components: research, measurement, standards, and service to both industry and other NIST laboratories. This combination of roles is unique among the laboratories at NIST. It is important that ITL management and NIST senior management recognize that ITL is an especially heterogeneous laboratory. Each division has an important role to play, but the weighting of the four components varies greatly across them. The aggregated contributions of ITL must match the needs of NIST and industry, but no uniform profile will match the individual divisions. Forcing each division into a common model would be inappropriate; instead, the individual contributions should be acknowledged and appreciated and explained to the community at large in formal mission statements, annual assessments, and day-to-day behavior.

Progress has been made in integrating the collaborative research functions of the Statistical Engineering Division (SED) and Mathematical and Computational Sciences Division (MCSD) into the rest of the laboratory. The task is not yet complete, however, and there are still strains, uncomfortable situations, and morale issues. This situation requires continuing work on the part of ITL management to state clearly and consistently the importance of the divisions and what their roles are; on the part of the management and staff of the divisions to be sensitive to the importance of working within the ITL environment; and on the part of NIST management itself to recognize the NIST-wide role and importance of these divisions as a fundamental part of NIST 's mission.

It is clear that ITL personnel involved in service activities on behalf of other parts of NIST (for example, providing networking or support of supercomputers) do not feel that their work is adequately recognized or appreciated outside of ITL. Because one of the attractions of support work at NIST is the chance to work with researchers on technically exciting projects, it is important that everyone involved feel that their contributions are important and appreciated. To the extent that this is not true, it will be even harder to attract good support personnel in this area. Morale and work satisfaction are two quite important factors in motivating these support staff, whose talents can command much higher salaries in the industrial and commercial sector.

Technical Merit and Appropriateness of Work

The quality of the programs in the laboratory is discussed in detail in each divisional report. With a few exceptions, also detailed in the divisional reports, the programs are well aligned with laboratory and divisional missions.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

The ITL's planning process continues to improve. The laboratory has a coherent picture of its emerging goals, focused on the information technology industry in the United States and consistent with the overall NIST mission.

The effects of significant and conscious planning by the laboratory are reflected in the new focus and in better-defined rules for project selection and phasing out. The laboratory has assumed an increased emphasis on both industrial needs and the unique resources and skills of the ITL. Laboratory managers and staff have made many hard decisions, and the resulting set of programs reflects a clearer vision of the future of information technology and of the role of the laboratory. The time lines that accompany this year's reports from ITL are valuable, and their continued use is encouraged. The inclusion of staffing profiles would further enhance their usefulness.

Most of the work of the laboratory appears to be suitable and appropriate for its mission, though there are a few projects that should be examined more closely. In general, the criteria for selecting projects should include explicit support from industry groups or consortia relevant to ITL's mission, the need for the unique NIST role of neutral arbiter, and the need for a competence unique to ITL. The panel endorses the ITL strategy of targeting fairly young technologies, so that changes can be made before industry practices have hardened. It is in these early stages that the tests, advice on standardization, and reference implementations provided by the laboratory are most valuable to industry. The ITL's major contributions to standards will come through such early involvement, which helps voluntary standards groups define and strengthen rapidly evolving standards. The laboratory should place more emphasis on participation in such industrial standards groups and less on NIST's position as an arbiter of conformance to mature standards.

Impact of Programs

The ITL is unique among NIST laboratories in that services and consulting for other NIST laboratories are a major part of its program. The ITL thus has two sets of customers to address: internal NIST customers and external customers.

The panel met with an ITL-selected sampling of internal customers of its support and consulting services. Overall, this group was very positive about the quality of services received and collaborations undertaken with ITL. This is in contrast to the feelings of underappreciation expressed by members of the ITL support and consulting divisions.

The ITL continues to be responsive to the need for increased involvement with industry. Many divisions are focused on customers and make a real effort to seek new partners. Although there are still areas that might be improved, the focus, alertness, and conscientiousness of the effort are laudable. The laboratory has also done a good job in disseminating its results to the industrial community; innovative use of Web technology continues to be used to good effect here, both for dissemination of information and for tracking of usage patterns. The laboratory is encouraged to continue and expand such activities.

The components from which the ITL was formed have a long history of industrial impact. Some of this is direct, as in the security work of the Computer Security Division and the support for test corpora (suites of tests and test data for verifying software performance) and technology evaluation conferences of the Information Access and User Interfaces Division. In these cases,

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

ITL's perceived neutrality gives it a unique opportunity to contribute. Other contributions are more indirect, as in the support provided by the Mathematical and Computational Sciences Division and the Statistical Engineering Division to research in other parts of NIST. It is important that this support be recognized when the impact of ITL is evaluated. Efforts to make the value of this support more visible to other laboratories of NIST and to their industrial contacts need to continue.

Since the previous assessment, ITL has made significant progress in its tracking of industrial contacts and impact. This is a difficult but worthwhile task, as such information can be used to help focus efforts in areas that appear most important to industry and to improve the laboratory's ability to demonstrate the value of its programs to NIST management, Congress, and potential industrial customers. The use of explicit time lines for projects is also a useful addition to project tracking and planning.

Laboratory Resources

Funding sources1 for the Information Technology Laboratory (in millions of dollars) are presented below:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

31.2

30.4

Competence

0.7

1.0

STRS-Supercomputing

12.3

12.3

ATP

2.1

1.5

MEP

0.1

0.0

OA/NFG/CRADA

9.9

11.9

Other Reimbursable

1.9

1.6

Agency Overhead

11.0

12.5

Total

69.2

71.2

1  

The NIST Measurement and Standards Laboratories funding comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST's congressional appropriations, but it is allotted by the NIST director's office in multiyear grants for projects that advance NIST's capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST's ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Manufacturing Extension Partnership (MEP) funding reflects support from NIST's MEP for work done at the NIST laboratories in collaboration with or support of MEP activities. Funding to support production of Standard Reference Materials is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding including that for Calibration Services is grouped under Other Reimbursable. In addition to performing research and providing calibrations and reference data, the Information Technology Laboratory also provides services to the other NIST laboratories. This work is supported through STRS funding designated specifically for the supercomputing facilities and through NIST overhead.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Staffing for the Information Technology Laboratory currently includes 362 full-time permanent positions, of which 295 are for technical professionals. There are also 87 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

A large majority of ITL funding is from NIST internal appropriations. This comes from three sources: the Scientific and Technical Research and Services, which is the direct appropriation from Congress for the NIST laboratories; payments from NIST-wide overhead to support services provided by the ITL to all of NIST; and Industrial Technology Services, which includes the Advanced Technology Program and the Manufacturing Extension Partnership. The remainder of the laboratory 's funding is from other agencies and from sales and services related to reference materials produced by the laboratory. Note that in the charts in this report, the portion of the STRS funding that is specifically allotted for the supercomputing facilities is listed separately. The panel was pleased that the laboratory's dependence on OA funding is running at what appears to be a comfortable level, under 20 percent.

The ITL director has directed that the Mathematical and Computational Sciences Division and the SED move toward a new funding paradigm wherein 20 percent of each unit's funding comes from sources external to MCSD, e.g., from other ITL divisions or other NIST laboratories or from other government sources. If this were to develop, it would have some advantages for these divisions. It would provide a vehicle for division personnel to become familiar with, and involved with, a much broader collection of NIST projects. It would add support to develop a larger group with more technical diversity and would be in a better position to contribute to the overall mission of NIST. However, it is clear that there are formidable obstacles to making this transition, from both within and outside these two divisions. From within, many people were attracted to these units, in part, because staff have not traditionally been expected to seek funding for their own activities. To make a successful transition away from this practice, staff involved must understand the advantages of it and need for it. Moreover, groundwork must be laid at the other NIST laboratories. They traditionally have not paid for support help from mathematicians, and there are indications that they would find the transition to such support problematic due to their own budget constraints.

The present ITL staff are strong and qualified, but the difficulties in hiring new personnel that were noted last year appear to be even more serious this year. Currently an intense market demand exists for information technology professionals. NIST management needs to consider whether the starting salary levels can be adjusted to be competitive in this environment. As noted above, one way to compensate for lower salaries is freedom from issues like getting grants.

The Advanced Measurement Center for Information Technology (AMCIT), a joint research and development institute with the University of Maryland, is another innovation that might help bring a flow of contributors and potential future hires, such as interns, graduate students, and visiting faculty. All ITL divisions should be involved with AMCIT because its highly visible activities should provide benefits through the attraction of eminent scientists that can in turn serve to draw younger talent. The potential here is extremely important to realizing the broader mission of NIST and helping ITL to support that mission.

Significant hardware investments will be needed to bring NIST up to the level of a world-class operation. The supercomputers are badly out of date, and individual computer services need continual upgrading on a short (2- to 3-year) cycle.

A perception widely shared by ITL staff is that internal NIST systems for procurement, personnel, and so on, are far too slow and cumbersome, especially when compared with normal

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

business practice. Activities such as finalizing a job offer, which might take a week in industry, can take months at NIST. This puts NIST at a disadvantage in recruiting.

DIVISIONAL REVIEWS

Mathematical and Computational Sciences Division
Mission

The mission of the Mathematical and Computational Sciences Division is to provide analytical and computational methods for solving scientific, engineering, and information technology problems of critical importance to U.S. industry. This mission is discharged through a program of advanced research in selected areas of applied and computational mathematics and collaboration with technical experts in other NIST divisions, industry, and academia.

In general, since the previous year, there is improved alignment of the work of MCSD with the ITL mission and its focus on metrology. MCSD has strengthened its work in this area by initiating the Digital Library of Mathematical Functions project. This is a significant undertaking and very consistent with the notion of advancing metrology and standards in the computational mathematics portion of the ITL mission. It also adds to the work in this same area represented by the Statistical Reference Datasets (StRDs) project being conducted with the SED and the MCSD's ongoing work on the Matrix Market and sparse BLAS (Basic Liner Algebra Subprograms). In each of these cases, MCSD is directly enhancing standards and metrology for the computational sciences. The larger scientific community has benefited by NIST work in this area for many years.

A continued effort ought to be made to develop more projects that align with the ITL focus on metrology. The panel emphasized again the observation made a year ago that the addition of one or more discrete mathematicians would aid in this endeavor. It is also important to preserve the role of optimization in the division in light of recent staff departures in that area. The new agreement with the University of Maryland on metrology in information technology ought to be viewed—by both ITL management and MCSD personnel—as a potential vehicle to spur further such MCSD work.

The panel finds the MCSD program to be in general conformance with the overall NIST mission. However, the panel also applauds the efforts of the ITL director to help link MCSD personnel with personnel from other NIST laboratories when those laboratories engage in strategic discussions with outside customers on projects involving MCSD. If others in NIST view MCSD personnel as strategic partners, it will help raise the visibility of MCSD across NIST as well as outside NIST.

Technical Merit and Appropriateness of Work

The work of the MCSD staff continues to be of high quality and to be highly respected in the applied mathematics community at large. The division has an unusual (but quite solid) portfolio of projects, some of which are important and innovative. As noted above, they have

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

added a new project on the digital replacement of the classical Handbook of Mathematical Functions,2 wherein the formulas, graphs, and tables are to be expanded into downloadable formats, dynamic graphics, tables on demand, reference algorithms, and so on. This work is very much in the spirit of the traditional role of the division, and its highlighting as a major new initiative of ITL is welcomed.

The major continuing projects are solid as well, and the general reaction the work has received both inside and outside of NIST has been positive. Special examples of important continuing work include the Matrix Market (a Web-based database of test problems for numerical linear algebra), Micromagnetic Modeling (a collaboration with the NIST Materials Science and Engineering Laboratory to develop computational tools for micromagnetic modeling of materials), and JazzNet (the development of a “personal supercomputer” from a dedicated cluster of PCs).

Impact of Programs

The effectiveness of the dissemination of the division's results varies widely. The Mathematical Software Group facilitates access to mathematical software tools for NIST users and the external community via the Guide to Available Mathematical Software and other projects. The division's statistics show that the GAMS Web site continues to record a significant number of hits, a clear indication of widespread interest. Many other projects have good descriptive material available on the Web as well. However, a conscious effort focused on technology transfer would undoubtedly lead to broader usage of MCSD results within NIST. For example, MCSD's seminar series aimed at researchers across NIST might be pursued more aggressively with the objective of making other NIST scientists and engineers aware of MCSD work and how they might benefit from it. Such outreach takes time away from other ongoing work, and its benefits might take time to accrue. But over time, it would likely lead to stronger support of the division by other laboratories and ITL divisions and to a more varied set of collaborative projects as well.

The division's direct impact on industry has also been most significant through the output of the Mathematical Software Group: GAMS, the Matrix Market, the sparse BLAS, and the work on the open/graphics language (Open/GL) graphics system. MCSD members have also been active professionally, with papers in refereed journals, talks at professional meetings, and professional journal editorships. Such activities are extremely important, not only as a vehicle for dissemination of results, but also as a validation of the technical quality of the work carried out and as a means for helping personnel keep abreast of research being conducted elsewhere.

The division also has an impact on industry through the collaborative work done with scientists in other NIST laboratories. The efforts of the ITL director to increase the direct interaction of MCSD collaborators with industrial customers of the other NIST laboratories will, if successful, facilitate a more direct understanding of the impact of the MCSD work. Consideration should be given to finding other ways to measure such impacts.

2  

Abramowitz, M., and I. Stegum, eds., 1964. Handbook of Mathematical Functions, Applied Mathematics Series SS, National Bureau of Standards, U.S. Government Printing Office, Washington, D.C. Also available from Dover Press.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Resources

Funding sources for the Mathematical and Computational Sciences Division (in millions of dollars) are as follows:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

4.1

3.8

Competence

0.1

0.1

STRS-Supercomputing

1.0

0.7

ATP

0.2

0.2

OA/NFG/CRADA

0.2

0.3

Total

5.6

5.1

Staffing for the Mathematical and Computational Sciences Division currently includes 30 full-time permanent positions, of which 28 are for technical professionals. There are also 15 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

The division's computer facilities seem quite adequate to support the workload, and the panel heard no complaints of any particular problems in maintenance and general support services.

Last year, the panel was concerned that the location of MCSD in NIST North had a detrimental effect on the level of the division's ability to expand its collaboration with scientists in the main campus laboratories through informal encounters. However, the panel heard fewer complaints this year and suspects that there has been a certain amount of adaptation. Furthermore, the realities of the long-term realignment of the entire NIST campus suggest that patience will be its own reward and that the MCSD staff should continue to reach out to the campus laboratories as much as possible. The management and staff should not let new ventures be restricted by minor problems of geography.

The MCSD has been through considerable transition in the past few years, first with the formation of ITL and the naming of a director who wanted to change some directions the division had taken in the past, and then with the subsequent retirement of their division chief. These events have led to morale problems in the past year. However, in the panel's view, the division is now past the low point and things are looking up for the division as a whole. There appears to be a fresh opportunity, with the naming of a new division chief, for the division to move forward with renewed spirit and new ideas. Discussions with candidates ought to be held with that thought in mind. Morale would also be improved by better communication to staff of ITL directions and their implications for MCSD personnel. It is clear to the panel that morale in the past year was adversely affected by the perception on the part of many in the division that they were given no clear sense of direction. Staff often thought they knew divisional priorities, only to be presented with different ones from management. The panel is not in a position to understand how these problems in communication came about, but they are real, and it will take a concerted effort on the part of the ITL director and all MCSD management to overcome them.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Advanced Network Technologies Division

The Advanced Network Technologies Division mission is to deliver measurement science to the communications industry. Despite the broad definition of the division's mission, all the division's projects are well justified. NIST can fill a unique role in standards by:

  • Evaluating competing draft standards and extracting the technical differences among them. NIST is particularly effective here because it is seen as an “honest broker”;

  • Providing freely available reference source code;

  • Providing interoperability testing in a convenient and private manner; and

  • Developing conformance tests.

Each of the division's projects involves at least one of the above activities.

Technical Merit and Appropriateness of Work

The Advanced Network Technologies Division is performing state-of-the-art research in many areas. For example, the effort to develop a Java-based mechanism for multimedia collaboration among participants on different platforms is an exciting and important new technology.

The division has also developed Web-based interoperability testing, which is a clever alternative to technology evaluation conferences. It reduces the expense of gathering and interconnecting independent implementations to test and allows testing in a risk-free private environment. This Web-based tester is currently being used to support Internet Security Protocol (IPsec) testing and will be expanded to provide services for other systems. IPsec is an important technology that has up to now been slow at being standardized and deployed, and the division's reference code will almost certainly speed up its deployment.

The use of formal specifications by the High-Speed Network Technologies Group (in concert with Bell Laboratories) provides a precise specification of protocols and could lead to automatic generation of conformance tests. The Software Conformance Group is doing similar work, and collaboration may be in order.

Although projects in internetworking technologies and multimedia seem appropriate, it is not certain that these efforts should focus totally on asynchronous transfer mode (ATM) technology. Backbones might be constructed in other ways, such as with pure fast Internet Protocol (IP) routers, or IP routers that achieve ATM speed through tag switching. Furthermore, IEEE Protocol 802.14 for cable television modems is attracting considerably less industry interest than it did last year and should be considered for phasing out.

Impact of Programs

The Advanced Network Technologies Division is active in the Internet Engineering Task Force (IETF), the ATM Forum, and the International Telecommunications Union, which are

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

currently the major standards-setting groups in networking. The division is also involved with the Cross-Industry Working Team and the International Multimedia Teleconferencing Consortium.

Probably the most important protocol, and the one whose success the division has the most opportunity to influence, is IPsec. The IPsec reference code is clearly popular with industry; 30 companies have applied for the code thus far. Some are using it for reference, others for code base, and others for testing purposes. The division was a coauthor of the IPsec protocol specifications at the request of IETF. Similarly, the division's Web-based interoperability tester is also clearly of interest to industry. Although it is very new, and NIST is not necessarily notified when someone uses the tool, at least 14 companies are known to have tested against it.

Another important area is IP quality of service, which needs NIST 's help. Reservation Protocol is assumed nonscalable, and many Internet service providers are refusing to deploy it. This protocol and its alternatives need to be studied by unbiased bodies like NIST.

NIST can play a unique role in moving standards forward and getting them deployed. Standards bodies often get delayed, sometimes by years, because they are forced to compare competing complete proposals. A company may have a large financial and marketing stake in having their proposal chosen or individuals may have great ego invested in their proposal. It is invaluable to have an independent body like NIST separate out the individual subproblems from the various proposals and compare approaches for each subproblem. This leads to a fair process for arriving at consensus. The division has provided such services in the case of the hybrid fiber coaxial Media Access Control protocol, where they identified common portions of 18 separate proposals and synthesized a single coherent protocol.

Once a standard has been adopted, NIST can play a major role in facilitating its deployment. The division is focusing its efforts on the right parts of this process, including making source code available for reference, providing code base or interoperability testing, creating conformance tests, creating interoperability tools, and hosting technology evaluation conferences.

In addition to working with standards committees, the division has worked directly with industrial collaborators, including AT&T, Bay Networks, Cisco Systems, COMSAT, and IBM. The division also collaborates with universities and other government agencies, including the Defense Advanced Research Projects Agency (DARPA) and other NIST laboratories and divisions.

Resources

Funding sources for the Advanced Network Technologies Division (in millions of dollars) are as follows:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

3.8

3.3

ATP

1.1

0.3

OA/NFG/CRADA

0.7

1.8

Total

5.6

5.4

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Staffing for the Advanced Network Technologies Division currently includes 26 full-time permanent positions, of which 21 are for technical professionals. There are also six nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

The division's researchers are enthusiastic and capable, and their morale seems high. In the past year, the division hosted 22 faculty and guest researchers. Some of these adjuncts are full-time guests who are in residence for 1 to 2 years; others act more as consultants and work with the division for as few as 10 hours per month. (The breakdown between these two categories is unknown.) The use of both full-time and part-time guest researchers is a creative solution to the problems of recruiting full-time permanent employees in an area where the private sector can pay significantly more than the government, but care must be taken to make sure that the goals of the guests are consistent with the division objectives.

Staffing is of particular concern in the Wireless Communications Group. Technical plans in this area are sound, but the group is experiencing difficulty in recruiting top talent due to noncompetitive salaries. The salary deficiency can be somewhat offset by offering liberal travel to conferences, educational opportunities, flexibility in pursuit of individual research interests, and an environment free of bureaucratic overhead such as grant-writing. If a few exceptions in salary could be made, a few “superstars” could act as a magnet to attract other good researchers who will trade off salary against the opportunity to work with the superstars. In an effort to jump-start some research in this area, the division is submitting joint proposals that will leverage personnel from other NIST laboratories and divisions. This is a good short-term solution, but it cannot be a long-term substitute for fully staffing the group.

The division has an unusually high ratio of OA to base funding (OA is $1.8 million; base is $3.3 million). The division management claims this does not affect the mission because the division researchers are in a position to choose collaborations compatible with their mission. Nevertheless, care must be taken to ensure there is no adverse impact on the primary mission because of OA demands.

Computer Security Division
Mission

The mission of the Computer Security Division is to enable organizations and individuals to use information technology with the assurance and trust that the confidentiality, integrity, reliability, and availability of resources are protected.

The security programs support the mission by promoting the development of technologies, standards, and practices that provide information security. The main thrust is testing and evaluation against widely accepted standards to provide improved assurance. The division also has related activities in education and training and in research on emerging technologies. All of these activities support the mission of the laboratory and of NIST. They also support NIST's charge under the Computer Security Act of 1987 to address IT security issues for unclassified federal applications.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Technical Merit and Appropriateness of Work

The division's programs not only employ state-of-the-art concepts and technology but also advance the state of the art in the areas of technology, standards, testing, and evaluation. Noteworthy programs include the Advanced Encryption Standard, public-key infrastructure (PKI), specification-based testing and evaluation through accredited laboratories, crypto module validation, National Information Assurance Partnership (NIAP), Common Criteria (CC), internetworking security through the IPsec standard, key recovery, role-based access controls, Federal Computer Incident Response Capability (FedCIRC), and mobile agents. The division is at the forefront of activity in these areas.

The division's focus is on enabling technologies, including cryptography and key recovery, and on enabling infrastructures, including PKI, internetworking security, security management, and criteria and assurance. All of the programs are appropriate for the division, in terms of both accomplishments and plans.

The division's name does not reflect the scope of the security program, which extends far beyond “computers” to include networks and systems.

Impact of Programs

The division does a good job of disseminating its results through publications, meetings, and its Web site. Its programs are having a significant impact on industry and other government agencies. A few examples follow.

In 1977, NIST issued the Data Encryption Standard. That algorithm became the encryption standard for government and industry, both nationally and internationally. Since then, there is unanimous agreement that the world needs a new standard for electronic commerce (EC), banking, and secure communications, but no organization or standards group had stepped forward to provide one. The division took on this task. The new Advanced Encryption Standard will almost certainly become the next world standard. The approach taken to develop the standard, namely, openness and extensive collaboration with industry and academia, should guarantee acceptance of this standard.

PKI services are essential for secure EC and communications; yet products and systems do not interoperate. Working with 10 industrial CRADA partners, the division has contributed substantially to the development of Minimum Interoperability Specifications for PKI components. This specification will enable the much-needed interoperability. Similarly, the division's project to provide a public-domain reference implementation of IPsec is making a large contribution towards interoperability.

The security of most information technology products and systems currently available is abysmal. Producers rarely evaluate their systems in any rigorous way, and products are shipped with gaping holes in security. Until recently, the standard documented approach to evaluation —Department of Defense 5200.28-STD Department of Defense Trusted System Evaluation Criteria, “The Orange Book”—was extremely costly and generally irrelevant to industry needs. The division plays a leadership role in making security evaluation and assurance relevant and affordable. The division 's Trust Technology Assessment Program, which aims to enable accredited commercial firms to perform specification-based security testing and evaluation,

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

will promote the availability of products and systems that provide security assurances. This effort is part of the NIAP program, a joint program of NIST and the National Security Agency. NIAP focuses on activities related to the CC, a joint North American-European effort to develop criteria for security evaluation and assurance. NIAP will promote the development of tests, methods and tools for assurance across a broad spectrum of information security technologies. This effort is likely to have high impact. A Cryptographic Module Validation program that follows this approach has already brought significant benefits to industry and government. Through this program, the Department of Defense has been able to procure quality encryption in software instead of hardware, saving well over $100 million. Because rigorous security standards and methodologies for security functionality and assurance do not exist, building a national capability through accredited laboratories within the framework of the internationally recognized CC will be a significant contribution. It will help U.S. companies market their products in Europe and elsewhere, when certification by accredited laboratories may be required. Once the capability is developed and refined, it may be possible to migrate the capability into companies for the purpose of self-testing during product development.

Resources

Funding sources for the Computer Security Division (in millions of dollars) are presented below:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

6.8

5.6

OA/NFG/CRADA

4.5

4.0

Total

11.3

9.6

Staffing for the Computer Security Division currently includes 44 full-time permanent positions, of which 39 are for technical professionals. There are also six nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

The division has successfully hired energetic and well-qualified staff. Although salaries are not competitive, the division has been able to recruit based on the opportunity to work on significant and nationally recognized projects.

The division has done an excellent job of prioritizing projects and allocating resources to efforts that are likely to have the highest impact. It has also done well leveraging the resources of industry and other government agencies. Recognizing the importance of information security, ITL management has been extremely supportive of the division, providing resources for programs and new hires. Any further resources that can be given to the division are likely to be utilized well, as they could be used to accelerate and broaden programs.

Current space limitations are a concern; the panel was told a proposal has recently been put forward by ITL to alleviate this situation.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

The division has responded to federal agencies' requirements for handling security incidents by establishing the FedCIRC, which provides assistance to federal agencies when needed. The start-up funding for FedCIRC, which was obtained through the Government Information Technology Services Committee, runs out at the end of this fiscal year. The funding to continue FedCIRC is well beyond the division's internal resources. Because FedCIRC provides government-wide services, it would seem logical that FedCIRC be cross-governmentally funded. The panel supports the continuation of FedCIRC, but if funding for the FedCIRC program is not renewed next year, the division should not attempt to maintain this important service on internal funds but transition the responsibility to other owners.

Information Access and User Interfaces Division
Mission

The Information Access and User Interfaces Division stated that its mission is to accelerate the development of technologies that allow intuitive manipulation and exchange of, and efficient access to, complex information by facilitating the creation of measurement methods and standards; to coordinate and provide evaluation methodologies, test suites and corpora, prototypes, workshops, and standards and guidelines to enable faster transition into the commercial marketplace in collaboration with industry, academia, and government; and to research technologies such as the digitization and representation of multimedia data, the use of spoken and written natural language, and visual interactive modalities for search and presentation of that information.

The mission of the division is well aligned with the laboratory's goal of meeting industry needs for objective, neutral measurement methods, tests, and standards for information technology. The mission is also consistent with NIST's charge to conduct research programs that also address industry's need for measurement and infrastructure technology.

Technical Merit and Appropriateness of Work

The division's activities fall into four major areas: spoken natural language processing, natural language processing and information retrieval, visual image processing, and visualization and virtual reality. Overall the programs in the division are quite appropriate and in most cases are supported by serious customers.

Working with DARPA and the spoken language research community, the Spoken Natural Language Processing Group has played a key role in designing tests for evaluation of the state-of-the art speech recognition research systems and in providing metrics to measure the progress made in such systems. The group has also developed techniques for providing diagnostic information to the research community; such information helps to identify strengths and weaknesses of the systems.

The Natural Language Processing and Information Retrieval Group organizes and runs the Text Retrieval Conference, which has both industrial and academic participants, to develop testing methodology for evaluating text retrieval. Among other work, the group created testing material

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

and methodology for evaluating speech-based document retrieval systems. This work is timely in view of the importance of retrieving both text and spoken information from large databases. The group collaborated with the Spoken Natural Language Processing Group to develop a state-of-the-art voice-actuated document-retrieval system.

The face and fingerprint recognition work in the Visual Image Processing Group is very good and is leading to standards that are being adopted by law enforcement agencies. The customers, such as the Federal Bureau of Investigation, are providing serious support to NIST for this work.

The division's programs in visualization and virtual reality do not yet show a coherent direction that points to significant progress. The Web usability testing program does not appear to provide much useful information and does not merit serious attention. What is really needed in this area are metrics for proper evaluation of multimodal interface technologies beyond mouse and keyboard.

Impact of Programs

The speech corpora developed by the division have helped the research community to make steady progress in improving the performance of speech recognition systems over the past 10 years. The results of these benchmark tests have helped the community to keep track of its progress. The most recent speech corpora, based on TV and broadcast news, have also helped the community to move towards real applications.

The division's fingerprint recognition work had a strong impact on law enforcement work, making it possible to access fingerprint databases in a speedy manner. The face recognition work has resulted in more reliable means of identifying persons and in providing access to such data.

Resources

Funding sources for the Information Access and User Interfaces Division (in millions of dollars) are as follows:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

4.8

4.5

ATP

0.0

0.2

MEP

0.1

0.0

OA/NFG/CRADA

2.0

3.0

Total

6.9

7.7

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Staffing for the Information Access and User Interfaces Division currently includes 41 full-time permanent positions, of which 38 are for technical professionals. There are also 13 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

These resources are sufficient for effective execution of all the programs in the division.

High Performance Systems and Services Division
Mission

The High Performance Systems and Services Division (HPSSD) enables effective application of high-performance computing and communications (HPCC) systems in support of NIST and its interactions with industry, academia, the federal government, and the public by (1) conducting research, development, and evaluation of innovative measurement and test methods, system architectures, and software technologies for improved scalability, functionality, flexibility, reliability, and economy for HPCC; (2) serving as a testbed for R&D in high-performance computing (HPC) and information technologies, and gaining experience in the deployment of these technologies and assessing their functional capabilities, interoperability, and operational characteristics; (3) serving as a responsive, effective mission-critical resource spanning computational, communication, mass storage, security, archival, and scientific visualization services; and (4) providing and managing state-of-the-art facilities that integrate and support an enterprise-wide heterogeneous information technology environment for NIST.

With minor exceptions, the division programs are clearly in accord with the division's and NIST's missions. The program emphasis, in terms of budget, people, and effort, is strongly in the service areas supporting mission goals (3) and (4). There are smaller-scale efforts supporting the research mission (1) and the testing mission (2). This approach is appropriate considering the history and the scope of the division.

Technical Merit and Appropriateness of Work

This discussion of technical merit is organized according to the three areas of HPSSD program activity: service, research, and testing.

Service. The service function dominates the division's work. The service functions of the HPSSD include responsibilities for telephone services, networking, and high-performance computing.

Telephone. The planned telephone upgrades are necessary given the growth in usage on the site and the lack of support from the vendor of the legacy phone system. It is hoped that the planned acquisition will take into account data-networking needs and leave room for future expansion.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Network. Planned capacity upgrades to the network are required to meet expected bandwidth needs for data communication, video teleconferencing, and multimedia. The panel encourages high bandwidth connection to the outside, including very high speed Backbone Network Series and future Internet 2 sites.

The planned full-scale firewall is an excellent first step in implementing computer security for NIST. It is not a panacea for all security issues. Note particularly that the interaction of the phone system and the firewall needs to be considered carefully (e.g., unprotected modem access inside the firewall should not be allowed).

High-Performance Computing. HPC support is a key service role for HPSSD. By enabling NIST researchers to make the best use of state-of-the-art resources, HPSSD enables other divisions to compete in their disciplines. To this end, it is essential that the HPSSD track the technological developments in HPC.

HPSSD maintains a diverse stable of hardware resources. Some of these, heavily used in the past, are now antiquated and are no longer state of the art or cost-effective. For example, the Cray C-90 and Convex ought to be phased out. Alternatives for users who must have Cray vector access in the future might include leased time at the National Science Foundation, Department of Energy, or other centers. State-of-the-art resources such as the SP-2, the Origin 2000, and PC clusters should be scaled up to provide needed levels of service. Simultaneous exploration of clustered computers and distributed shared memory for problems not amenable to cluster computing should be encouraged. Further, considerably more capability is needed to remain competitive, let alone to be first class. To get a sense of scale, HPSSD currently maintains an eight-processor Origin 2000. In contrast, university computer centers often feature 32 to 64 processors and range up to as many as 192 processors.

HPSSD has made strides in educating and helping the user community exploit HPC resources. The panel applauds their shift toward real collaboration with domain scientists rather than acting as mere programmers. This has accelerated the shift toward parallel computing at NIST. The panel encourages HPSSD to educate and train all orphaned Crazy and Convex users in the use of modern HPC equipment. This training will require additional resources.

HPSSD-developed tools to aid in the use of HPC systems represent good support for the NIST community. The needs of NIST users are similar to those of other high-end computational scientists. The panel encourages HPSSD to maintain contact with recent large-scale HPC computational science efforts such as the National Science Foundation 's Partnerships for Advanced Computational Infrastructure, the Department of Energy's Accelerated Strategic Computing Initiative, and the Department of Defense's Defense Modernization Programs.

Research. The research component of HPSSD is relatively small. It is not as comprehensive as might be desired but consists of various small projects with varying degrees of conformance to the division mission. For example, it is unclear that the electronic book (E-book), pattern recognition, and photonic interconnection for wireless communication projects fit within the division mission. The Photonic Interconnect project, in fact, seems a better fit to the Advanced Network Technologies Division. Work on the properties of flat-panel displays is marginally related to the mission and should at least be coordinated with the NIST Electronics and Electrical Engineering Laboratory and the NIST Physics Laboratory. However, research into PC clusters

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

for HPC is an exciting area being investigated throughout the world, and work in HPSSD is state of the art and appropriate to the division mission. Projects are at various stages of development although all seem to have clear time lines.

Testing. The division's participation in development of a testbed for scalable computing over an interoperable message passing interface (IMPI) is very appropriate and helps move the industry toward truly flexible scalable computing.

The division could provide a useful service to the broader community by testing and benchmarking HPC hardware and software; however, given the division's level of staffing, and the necessity to remain a neutral arbiter, the division's decision not to do so may be appropriate.

Impact of Programs

The panel interviewed several NIST users of the division's service programs who were very positive about services received. User response to the division has also been very positive. This was especially evident for the collaborative model the division uses in developing parallel applications and for services supporting administrative computing systems.

The division's publication level has increased substantially, which improves the reputation and visibility of NIST in this discipline. Ongoing activities in HPC have not yet had a significant impact on that industry. There is a potential for future impact through the division's work on IMPI and clusters. Time synchronization and digital video disk (DVD) projects also represent technologies with potential industrial impact outside of the HPC industry. Also, a division staff member was recognized by the Video Electronics Standards Association for his leadership in standards issues relating to digital display interfaces.

Resources

Funding sources for the High Performance Systems and Services Division (in millions of dollars) are presented below:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

1.7

1.7

Competence

0.5

0.1

STRS-Supercomputing

8.3

9.8

ATP

0.5

0.5

OA/NFG/CRADA

0.8

0.4

Other Reimbursable

0.9

0.7

Agency Overhead

4.9

5.1

Total

17.6

18.3

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Staffing for the High Performance Systems and Services Division currently includes 71 full-time permanent positions, of which 51 are for technical professionals. There are also 17 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

In order for NIST to have first-class computational resources, more money will be necessary for capital expenditures on computers and networking. NIST management also needs to understand that traditional vector computing is not a cost-effective solution, either from the viewpoint of hardware maintenance or from the viewpoint of software migrated from the desktop. In addition, in the rapidly changing HPC environment, the procurement cycle must be reduced and costs amortized over a small number of years. A generation for HPC is 2 to 3 years.

As with the whole of ITL, salary is a major problem in hiring new staff. As in other areas of ITL, flexibility and research opportunities offset the salary disincentives. However, on the service side, this effect is much reduced. Therefore there is a risk that it will become more and more difficult to maintain staff in service positions.

Distributed Computing and Information Services Division
Mission

The mission of the Distributed Computing and Information Services Division (DCISD) is to provide an integrated suite of information technology resources and services that meets the needs of NIST staff in an efficient and effective manner and that provides a sound basis for meeting future needs and enabling innovative applications of information technology.

The division provides software for scientific computing, support for local PCs and scientific workstations, electronic information dissemination services, and administrative and management applications support, directly supporting service functions and indirectly supporting the measurements, standards, and research functions of NIST. The division provides services and supports infrastructure to help the NIST staff, its collaborators, and its clients to address the measurement and standards needs of many industry sectors; therefore, its mission conforms to and helps to carry out the overall NIST mission to promote U.S. economic growth by working with industry to develop and apply technology, measurements, and standards.

Technical Merit and Appropriateness of Work

The DCISD is involved in a number of activities to reduce the maintenance cost and improve the effectiveness and efficiency of NIST computing. Examples are unifying e-mail services, implementing a software checkout program, providing training on Microsoft Office Suite™, defining workstation security policy, negotiating site-wide license agreements, investigating Web training, and disseminating information via the Web. In addition to the continual service activities provided by DCISD, six projects are in progress as of the review. Although the Year 2000 compliance project, NT Enterprise Architecture, electronic funds transfer (EFT) disbursements, and similar projects are necessary for supporting NIST needs, collaborative projects such as EC, PKI implementation, Intranet streaming audio and video implementation, and

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

EFT draw DCISD personnel into research activities with other ITL divisions. Involving DCISD personnel in such a way is an inducement to the hiring and retention of good people.

Impact of Programs

The DCISD has carried out many upgrade projects at NIST effectively and efficiently. The division's support of a NIST-wide World Wide Web and its use to disseminate information on NIST programs is good. Feedback received by the panel from some internal customers was positive, although this sample may have been weighted toward customers who understood available technologies and the services they receive. There is a need for the division to conduct frequent customer surveys to obtain feedback on the effectiveness of the service portion of its organization.

Because the division focuses on serving the IT needs of its NIST staff, its program has no direct impact on industry. However, the division could take an active role as a testbed for the IT industry, for example, advising the industry on the Year 2000 efforts and sharing its experience on scalability and reliability of NT clusters.

Resources

Funding sources for the Distributed Computing and Information Services Division (in millions of dollars) are as follows:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

0.4

0.5

STRS-Supercomputing

0.9

0.9

OA/NFG/CRADA

0.2

0.2

Other Reimbursable

0.5

0.6

Agency Overhead

5.7

7.0

Total

7.7

9.2

Staffing for the Distributed Computing and Information Services Division currently includes 62 full-time permanent positions, of which 57 are for technical professionals. There are also seven nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

Resources are probably the division's biggest challenge. The division is continually understaffed due to the competitive job market for the skills it needs and the uncompetitive salary that government pay scales restrict it to. As noted previously, the opportunity to spend some time working on collaborative research projects with other ITL divisions has been an inducement that partially compensates for salary inequities.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

The division finds itself trying to keep pace with changes in technology while continuing to support old systems (both hardware and software) for users who do not see the need to upgrade. The recent implementation of Microsoft Office as the standard office software at NIST has occurred in principle, but not entirely in practice. The division faces tough decisions on which systems it should support centrally out of overhead funds and which should be left to individual operating units to support directly.

The continued upgrade of all systems every 3 to 4 years would reduce the complexity of maintenance and improve the effectiveness of the division's staff, but the appropriate funding for the upgrade must be allocated. It is hard to imagine a state-of-the-art organization still using 386 and 486 machines, but that is the case at NIST. An active division effort to educate and make recommendations to the NIST community on hardware and software upgrades may help bring about acceptance of and demand for newer technologies.

Software Diagnostics and Conformance Testing Division
Mission

The mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, conformance to standards, and correctness; participate with industry in the development of forward-looking standards; and lead efforts for conformance testing, even at the early stages of development of standards.

Technical Merit and Appropriateness of Work

The division's work in designing conformance tests is state of the art and clearly relevant to the division's mission. The division's work in development of virtual reality modeling language (VRML) test suites continues, with much progress made in the past year. The division has now released tests that cover 60 percent of VRML specifications and has completed development of a reference parser and scene graph generator for VRML. To this point, the division has focused only on static testing but is planning work on testing dynamic VRML worlds. There may be a potential to accelerate and improve this work through collaborations with the Information Access and User Interfaces Division, which has efforts in both VRML and virtual worlds.

Work on Java test suites has only recently been initiated, despite the widespread use of the language in industry. The division has demonstrated inconsistencies in Java virtual machine (VM) implementations, and is developing a transparent VM that allows viewing of VM components during execution. This work is quite relevant to industrial users, but it is not clear that NIST has a unique role to play, as industry is already quite active in this area.

The division has exciting interactions with researchers from the SED on software testing by statistical methods. Although this project is based on a concept developed almost 20 years ago, the practical realization of this concept will be highly useful to industry. The enthusiasm of the participating staff from both divisions demonstrates the fruitfulness of the collaboration for all partners.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

Continuing work in role-based access control, in collaboration with the Computer Security Division, is of good technical quality but lacks a sense of direction. The concept is very useful for industry, and program management might consider strategies for increasing the visibility and impact of the efforts. The division's related work in electronic commerce is innovative both in the attempt to use an instruction management system and in the design of an architecture for electronic commerce. Designing for future standards based on this architecture could be very important. Again, a clear focus is necessary for this work to have an impact.

A Competence project to develop automated software test generation using formal methods is a very ambitious project but worth the preliminary efforts being expended. Smartcard vendors have already expressed interest in using such a method to generate tests for smartcard verification.

Resources

Funding sources for the Software Diagnostics and Conformance Testing Division (in millions of dollars) are presented below:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

5.2

4.4

Competence

0.0

0.5

ATP

0.3

0.3

OA/NFG/CRADA

1.5

1.9

Other Reimbursable

0.4

0.3

Total

7.4

7.4

Staffing for the Software Diagnostics and Conformance Testing Division includes 41 full-time permanent positions, of which 34 are for technical professionals. There are also 10 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

The panel was impressed with the high level of enthusiasm displayed by the staff of the division. The staff members seem to have a good sense of the direction that the division is taking and are engaged in the division mission.

Statistical Engineering Division
Mission

The SED states that its mission is to catalyze experimentation, enhance research, and improve communication of results by working collaboratively with, and developing effective statistical methods for, NIST scientists and partners in industry. The role of this division is now articulated in the ITL mission statement. In support of its mission, ITL provides technical

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×

leadership and collaborative research in critical infrastructure technologies that promote better development and the use of information technology, in addition to high-quality services and supporting infrastructure to help the NIST staff, its collaborators, and its clients address the measurement and standard needs of many industry sectors.

Technical Merit and Appropriateness of Work

The SED continues to play an integral and successful role in the core business of NIST, with high-quality work and collaborations ongoing throughout the NIST laboratories. It also appears to be unusual for a division of this size to have three NIST Competence funding awards.

The division's work with SEMATECH on an online Internet Engineering Statistics Handbook for scientists and engineers in industry has progressed well and will be a highly useful and important product. All chapters are expected to be completed in the coming year. Another important and useful project is the award-winning Statistical Reference Datasets. The purpose of this Web-based service, now in place, is to improve the accuracy of statistical software by providing reference data sets with certified computational results that enable objective evaluation of statistical software.

The division continues to pursue efforts to more efficiently process the analysis of the uncertainty associated with the measured quantities and characteristics certified in NIST Standard Reference Materials. They have increased the number of their analyses significantly without a corresponding increase in resources. It is not clear if much more efficiency is possible at this point. The division's program of short courses and short tutorials for the NIST community serves to provide some contact with other scientists and engineers at NIST in spite of the SED's isolated location at NIST North. Other laboratories and other ITL divisions report favorably on their impact.

The division's annual report3 is an impressive catalog of research activities and collaborations with multiple NIST laboratories, including other divisions in ITL. The percentage of effort devoted to projects with other ITL divisions has grown to 10 percent, a significant increase over last year. The panel is pleased by this increase and sees the possibility for still more mutually productive interactions with other ITL divisions. A large number of publications in high-quality journals is strong evidence of the technical merit of the work.

Impact of Programs

The StRD and the Engineering Statistics Handbook are too recently available on the World Wide Web to evaluate their impact, but counting the number of hits would allow for some estimate of their use. The large number of publications by the division is an indication of the dissemination of their work. Much of the industrial impact is indirect through collaborations with other divisions and laboratories in NIST.

3  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Report of Activities: Statistical Engineering Division, NIST internal document, National Institute of Standards and Technology, Gaithersburg, Md., 1998.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Resources

Funding sources for the Statistical Engineering Division (in millions of dollars) are as follows:

 

Fiscal Year 1997

Fiscal Year 1998 (estimated)

NIST-STRS, excluding Competence

2.6

2.4

Competence

0.1

0.3

STRS-Supercomputing

0.5

0.4

OA/NFG/CRADA

0.0

0.2

Other Reimbursable

0.1

0.0

Total

3.3

3.3

Staffing for the Statistical Engineering Division currently includes 21 full-time permanent positions, of which 19 are for technical professionals. There are also 10 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers.

The demands on SED by other laboratory personnel for collaborations and consulting need to be balanced with the effort to increase interactions with other ITL divisions. The unfilled positions and no-growth budget have affected the ability of SED staff to respond to the full demand for its expertise.

The development of the Advanced Measurement Center for Information Technology, an ITL collaboration with the University of Maryland, is a promising avenue for reinforcing SED and other ITL division interactions and for bringing new external scientific expertise into “software metrology.” SED should take full advantage of this opportunity to supplement staffing by these interactions.

MAJOR OBSERVATIONS

The panel presents the following major observations.

  • ITL and NIST management should explicitly recognize and support the varied roles of the divisions within ITL and measure their performance and allocate their resources with a clear eye to the differing work profiles within the divisions. One size does not fit all.

  • New policies that require MCSD and SED to seek nonoverhead resources should be reassessed carefully; this is not a step to be taken lightly.

  • The collaborative and research functions of the Statistical Engineering and Mathematical and Computational Sciences Divisions still need to be more fully integrated into the ITL's activities and mission.

  • Information technology is a dynamic area with an extremely high demand for skilled people in all roles. NIST must work toward increased compensation for such employees and must find creative ways to attract and retain such employees.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
  • NIST must find ways to express appreciation and give credit to those in supporting roles or whose collaborations are behind the scenes.

  • The ITL would benefit from more detailed tracking of the results and consequences of its technical activities.

  • Maintaining old technologies such as the Cray supercomputer or outdated software on older desktop machines is not resource efficient. Equipment should be upgraded, and only common hardware and software should be supported out of overhead funds.

  • The departure of the SED chief in the past year has led to the appointment of an acting chief. Filling this position rapidly with a permanent division chief is essential to staff morale and necessary to accelerate the growth of SED interactions with other ITL divisions.

Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 179
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 180
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 181
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 182
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 183
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 184
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 185
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 186
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 187
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 188
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 189
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 190
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 191
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 192
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 193
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 194
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 195
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 196
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 197
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 198
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 199
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 200
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 201
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 202
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 203
Suggested Citation:"8 Information Technology Laboratory." National Research Council. 1998. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998. Washington, DC: The National Academies Press. doi: 10.17226/9515.
×
Page 204
Next: Appendixes »
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories, Fiscal Year 1998 Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!