An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1997 (1997)

Ralph Z. Roskies, University of Pittsburgh, Co-Chair

Brian W. Kernighan, Lucent Technologies, Co-Chair

Bishnu S. Atal, Lucent Technologies

Cheryl Y. Begandy, Alcoa Technical Center

Mary Ellen Bock, Purdue University

Allen L. Brown, Jr., Cinebase Software, Inc.

Dorothy E. Denning, Georgetown University

Edward A. Feigenbaum, Chief Scientist of the Air Force

Andrew S. Grimshaw, University of Virginia

Carl M. Harris, George Mason University

Thomas P. Kehler, CONNECT, Inc.

Ellen M. Knapp, Coopers & Lybrand

Sandra M. Lambert, Lambert & Associates

Martin C. Libicki, Institute for National Strategic Studies

Carl Machover, Machover Associates Corporation

Richard E. Nance, Virginia Tech

Jerome Sacks, National Institute of Statistical Sciences

Daniel L. Solomon, North Carolina State University

Ahmed N. Tantawy, IBM Corporation

Raymond T. Yeh, International Software Systems, Inc.

Eddie L. Zeitler, Charles Schwab & Company

Submitted for the panel by its Co-Chairs, Ralph Z. Roskies and Brian W. Kernighan, this assessment of the fiscal year 1997 activities of the Information Technology Laboratory is based on a site visit by the panel on March 19–21, 1997, and on documents provided by the laboratory.

The Information Technology Laboratory (ITL) stated that its mission is to stimulate U.S. economic growth and industrial competitiveness through technical leadership and collaborative research in critical infrastructural technologies to promote better development and use of information technology. The Information Technology Laboratory carries out this mission by working with industry, research, and government organizations to develop and demonstrate tests, test methods, reference data, proof of concept implementations and other infrastructural technologies that are needed by U.S. industry to produce information technology systems that are usable, secure, scaleable, and interoperable.

The focus and clarity of this mission indicate significant progress since the panel's 1995 assessment of the laboratory. The laboratory 's emphasis on metrology for information technology and on the establishment of a national test and measurement infrastructure for information technology is quite promising. This vision points in a sound direction for the future and integrates well with the overall NIST mission. The panel recognizes the importance of standards for interoperability and system integration and therefore hopes that the mission is broadly construed to include work on standards and their underlying technology.

Though the panel approves of the goals laid out in this statement, this mission as currently constructed does not reflect the present composition of the Information Technology Laboratory. The laboratory embraces three distinct functions, all of which are important and vital to NIST: (1) the new focus on information technology metrology; (2) computing services; and (3) collaborative research in computational and statistical science. The latter two elements are not adequately reflected in the mission and were not well represented in the laboratorywide goals presented to the panel. These two areas have a long, established role at NIST, and their continued contribution is critical.

In the panel's discussion of this lack of coherence, members noted that many institutions have responded to difficulties of this sort by moving the service functions into a separate organization. In this instance, that would include essentially all of the Distributed Computing and Information Services Division and perhaps parts of other divisions. Currently, laboratory management spends a disproportionate amount of time on the service component, yet staff are discouraged and frustrated. The rest of the NIST community often believes that they are not well served and that technology support should be a free commodity.

In addition, the panel believes that the collaborative research function of the Statistical Engineering and Mathematical and Computational Sciences Divisions could be integrated more thoroughly with the rest of the Information Technology Laboratory. The NIST-wide role and importance of these divisions and their work are not featured prominently enough in the laboratory mission statement, yet their functions are basic to NIST's mission. At the same time, these groups have not yet adapted to the current organizational structure and its requirements. In particular, the Mathematics and Computational Science Division needs to be more responsive regarding work on issues related to information technology. In this division, significantly greater sensitivity to customer needs and outreach is needed. But other divisions within the Information Technology Laboratory could also be more supportive and make more of an effort to work actively with their mathematics and statistics colleagues.

The quality of the programs in the laboratory are discussed in detail in each divisional report.

The Information Technology Laboratory has been especially responsive to the need for increased involvement with industry. Many some divisions have a proactive focus on customers and make a real effort to seek new partners. Although some areas still need to be improved, the focus, alertness, and conscientiousness of the effort are laudable. The laboratory has also done a good job in disseminating its results to the industrial community, particularly through innovative use of Web technology. This method is easy to use, cost effective, and readily accessible to both large and small companies.

The components from which the Information Technology Laboratory was formed have a long history of industrial impact, although such interactions were not explicitly part of their mission. Some examples of past programs still in use today include various standards for computer languages, like Ada, Cobol, and Structured Query Language (SQL). These languages have mature, stable user communities, and so there is now little need for NIST's special competence in these areas. Related projects at the laboratory are being passed on to private companies or industrial consortia while being phased out at NIST. This shift reflects a general trend away from paper standards but is also a result of the laboratory's move away from classical ex post facto standards, particularly Federal Information Processing Standards.

This trend away from the traditional paper standards (as produced by committee meetings) and toward early de facto standards (as defined by test suites, conformance tests, and reference implementations) requires the Information Technology Laboratory to remain in close touch with the communities in which standards are evolving. The present effort to develop a set of tests for the Virtual Reality Modeling Language (VRML) is a good example of this kind of activity: The impetus for the project came from an industry consortium rather than an individual company. In addition, NIST is perceived correctly as neutral and can provide the particular competence appropriate for a standard that requires considerable mathematical and graphical sophistication. Past and current work to provide reference datasets, tests, and test services also has significant industrial impact. Some current examples are work on optical character recognition in handwriting and fingerprints, programs for conformance testing, and technology evaluation conferences (known as “bake-offs”) for speech recognition software. In each of these cases, the Information Technology Laboratory's combination of expertise and neutrality provide a unique resource and have significant impact on industry. The statistical and mathematical work of the laboratory also has significant effect on industry, both directly through the dissemination of informational packages like the “Guide to Available Mathematical Software,” and indirectly, through the collaborations between personnel in the Statistical Engineering and Mathematical and Computational Sciences Divisions and other scientists throughout NIST.

A key element of industrial interaction should be a mechanism to assess the laboratory's impact on U.S. companies. A clear method of tracking industrial contacts would assist in such assessments as well as improve the laboratory's ability to demonstrate the value of its programs

to NIST management, Congress, and other potential industrial customers. The laboratory maintains a database of industrial contacts and activities, but it appears to be more a list of names and meetings than an organized repository of information. A more complete record would document how a given activity began, who has been involved from NIST and from outside, what work has been done, how much time has been devoted to the project on each side, what adjustments have been made over its life cycle, what external publications or artifacts have been produced, which industries have been affected, how they have benefited from NIST's work, what acknowledgments of the laboratory's efforts have been made, what possible connections to future activities or other industries exist, and further data of this sort. The purpose of such information gathering is not to impose reporting requirements on the laboratory, but to make it easier for NIST to accurately assess its external interactions and impact, to help demonstrate NIST's value, and to concentrate available resources most effectively.

A large majority of Information Technology Laboratory funding is NIST internal appropriations. This comes from three sources: (1) the Scientific and Technical Research and Services (STRS), which is the direct appropriation from Congress for the NIST laboratories; (2) payments from NIST-wide overhead to support services provided by the Information Technology Laboratory to all of NIST; and (3) Industrial Technology Services (ITS), which includes the Advanced Technology Program and the Manufacturing Extension Partnership. The remainder of the laboratory's funding is from other agencies (OA) and from sales and services related to reference materials produced by the laboratory. In the charts in this report, the portion of the STRS funding specifically allotted for Consolidated Scientific Computing Systems (CSCS), the supercomputing facilities, is listed separately.

Funding for the Information Technology Laboratory (in millions of dollars):

The panel supports the continuing reduction in the laboratory's dependence on OA funding. However, some level of outside support is valuable, not only as a source of additional resources, but also as a validation of the laboratory's efforts. It is important to strike a balance between these benefits of OA funding and the danger that resources will be diverted to projects that do not support the laboratory 's mission.

The space allotted to the Information Technology Laboratory is adequate at the moment but allows no room for future growth. Increasing external collaborations (particularly in the Computer Security and Software Diagnostics and Conformance Testing Divisions) will require expanded facilities. The Mathematical and Computational Sciences and Statistical Engineering Divisions are now located in NIST North. This placement inhibits their collaborations with the scientists in other NIST laboratories, although it may improve the interactions among the divisions within the Information Technology Laboratory.

The major problem with technical resources is that the NIST-wide digital network “backbone” of 100 Mbits/sec is insufficient for the present and future needs of all the laboratories. Computer facilities are mostly adequate at the moment, but a computer generation is only 2 to 3 years long and equipment must be upgraded frequently. The technical support for individual workstations seems inadequate, especially in the Mathematical and Computational Sciences and Statistical Engineering Divisions. The cause of this lack of support was not clear to the panel.

In general, the allocation of service resources is a NIST-wide issue that has a significant effect on the Information Technology Laboratory. The distribution of services is not governed by any market mechanism but is instead supported through overhead and hence appears free. Thus the other laboratories tend to undervalue the services provided by the Information Technology Laboratory. Further comments on this issue are made in the section on Laboratory Planning.

At the moment, the staff of the Information Technology Laboratory includes 420 full-time permanent (FTP) positions, of which 340 are for technical professionals. There are also 45 guest researchers. Current staff are quite strong and qualified, but the laboratory is experiencing difficulties in hiring new personnel. There is intense market demand for information technology professionals. The Information Technology Laboratory is handicapped because the starting salary levels at NIST are not competitive in this environment. One way of compensating for lower-than-market salaries is to provide a better-than-market work environment, including opportunities for publication.

The panel was pleased to see the progress in planning made by the Information Technology Laboratory over the last 2 years. The laboratory has a coherent picture of its emerging goals, focused on the information technology industry in the U.S. and consistent with the overall NIST mission. The laboratory has made considerable progress in understanding the role of metrology in information technology and the ways in which metrology and testing can help U.S. industry be more productive and competitive (see e.g., the recent NIST white paper on Metrology for Information Technology).¹ The move away from performing conformance tests and toward developing the tests and the related methodology for industrial use is a positive step.

The effects of significant and conscious planning by the laboratory are reflected in the new focus and in better-defined rules for project selection and sunsetting. Under the laboratory director's effective leadership, the laboratory has assumed an increased emphasis on both industrial needs and the unique resources and skills of the Information Technology Laboratory. Laboratory managers and staff have made many hard decisions, and the resulting set of programs projects a clearer vision of the future of information technology and of the role of the laboratory. However, timelines are an important part of project planning but were missing from most of the projects presented to the panel. Such life-cycle descriptions assist in assessing the progress of programs and the effectiveness of the planning process.

The Information Technology Laboratory has taken significant steps toward establishing sound criteria for resource allocation among programs. Nevertheless, the panel believes that there is still some room for improvement in the way projects are chosen. A few projects still appear to be continuing for historic rather than programmatic reasons; these are cited in the divisional assessments. The criteria noted by the laboratory as key elements in the selection process —such as impetus from industry, the need for NIST to serve as an honest broker, or the presence of a special competence available only in the Information Technology Laboratory—are generally useful in deciding whether an activity is appropriate for the laboratory. It is also highly desirable that projects target fairly young technologies so that changes can be made before industry practices have hardened. It is in these early stages that the tests, advice on standardization, and reference implementations provided by the laboratory are most valuable to industry. The Information Technology Laboratory's major contributions to standards will come through such preemptive activities that help voluntary standards groups define and strengthen rapidly evolving standards. Participation in such industrial standards groups is a more important activity for the Information Technology Laboratory than its work as an arbiter of conformance to mature standards. The panel was also concerned that in some cases the laboratory has insufficient understanding of the status of research being done external to NIST before undertaking projects.

The amount of planning done by the laboratory is reflected in the copious documentation of the planning process. The materials provided to the panel were of high quality, and spoke directly to issues raised in previous assessments. Now that the need for laboratorywide planning efforts is decreasing, the task of unifying divisional management and staff behind the new mission is the primary challenge.

The Mathematical and Computational Science Division stated its mission to be providing technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to American industry. This division has created a unified analytical and computational setting in which advanced scientific and engineering problems can

be solved. The components of such a setting include: the development and analysis of theoretical descriptions of phenomena (mathematical modeling); the design and analysis of the requisite computational methods and experiments; the transformation of these methods into efficient numerical algorithms for high-performance computers; the implementation of these methods in high-quality mathematical software; and the distribution of this software to potential clients.

The mission of the Mathematical and Computational Sciences Division reflects its historical role of providing analytical and computational tools and engaging in collaborative research with other NIST divisions. These services are critical to the NIST mission because they provide vital computational capabilities. However, neither the laboratory nor the division mission statement reflects the role that this division could play within the Information Technology Laboratory. The laboratory director has placed an increased emphasis on research in information technology metrology, yet such a shift in focus is not apparent in the Mathematical and Computational Sciences Division. This confusion is reflected in the absence of information technology in the divisional mission statement as well as in the lack of a well-articulated role for the Mathematical and Computational Sciences Division in the laboratory 's strategic plan. The paucity of collaborative activities between the Mathematical and Computational Sciences Division and the other divisions of the Information Technology Laboratory is a symptom of this disjointedness.

The work of the Mathematical and Computational Sciences Division is generally of high quality. The staff is well respected in the applied and computational mathematics research community, and the division often attracts some of the best in this field to come to NIST for terms as guest researchers. Examples of particularly innovative or important works in progress in the division include the Matrix Market, a Web-based repository of matrix test data that has had more that 8,000 visitors who have downloaded more than 18,000 matrices; a Web-based set of basic linear algebra subprograms (sparse BLAS) for sparse matrices, which allows users to generate source code from the Web; and process models that enable the calibration of complex devices such as coordinate measuring machines or the brain probes used during neural surgery.

The division's work on compression algorithms raised some concerns for the panel. It was unclear to what extent the group at NIST was aware of the state of the art in this field and understood the most pressing issues in the area. For example, it was unclear that the group was aware of the types of compression problems that would come up in a compression discussion over the Internet.

The Mathematical Software Group has disseminated its work effectively through the World Wide Web. Their products have reached an extraordinarily high number of working computational scientists in industry, academia, and other research facilities. The Web has served not merely as a new medium for publication, but also as a tool to add value to their work (as in the case of sparse BLAS, which is capable of generating actual code that can perform fundamental sparse matrix operations). In general, this innovative use of the Web is commendable.

The entire division has not focused enough on defining a realistic measure of the value of its work. A great deal of this division' s modeling and simulation work is done in collaboration with scientists in other NIST laboratories; an example is work on a paint weathering simulation with personnel from the Building and Fire Research Laboratory. The other laboratories then interact with their industrial partners. Because this division is not in direct contact with U.S. companies, it is obliged to define a new way to measure its effectiveness. One approach would be to treat collaborators from other laboratories as the customers and to assess both the depth of their interest in and the benefits they receive from working on projects with Mathematical and Computational Sciences Division staff. Some quantitative analysis of the value added by this division's efforts on industrially relevant projects with other laboratories could supplement the present counting of publications as a measure of effectiveness.

Funding for the Mathematical and Computational Sciences Division (in thousands of dollars):

Divisional personnel includes 51 FTP staff, of whom 48 are technical professionals.

There is confusion about whether Information Technology Laboratory resources are available to support the Mathematical and Computational Sciences Division's workstations and to provide computing services. Although funding for these activities is apparently in the laboratory budget, the division has not seen the resulting services.

In general, the division's facilities are of good quality. However, the location of the Mathematical and Computational Sciences Division in NIST North has had a direct detrimental impact on the ability to collaborate with scientists in other laboratories. The remote location cuts down on informal interactions and the chance encounters with colleagues that lead to identification of new research opportunities.

If this division is to expand its technical capabilities in a direction better suited to an emphasis on information technology, it needs to hire a discrete mathematician.

In the past, the division has had no official planning process to ensure unity of purpose in its efforts. Although it is important that personnel from all levels be able to suggest possible projects, the inauguration of formal meetings with the leadership from other NIST laboratories to determine general mathematical and computational science needs will probably bring some order to divisional project selection. A more structured process could also allow laboratory management to work on aligning divisional programs with the Information Technology Laboratory's mission.

The Mathematical Software Group has made a significant effort to relate its work to the overall mission of helping industry by providing tests and test data, performing interoperability testing, and developing metrics to assess software performance. However, in general divisional personnel have done little outreach to other Information Technology Laboratory divisions involved in information technology metrology work. The Mathematical and Computational Sciences Division staff does not yet have a strategy to foster and identify joint proposals with the other Information Technology Laboratory divisions in this field. Such an effort might include educating their colleagues in applications of modeling, simulation, and other computational methods to research in the areas that interest other divisions. It is worth noting that there has been a corresponding lack of proposals or projects suggested to the Mathematical and Computational Sciences Division by the other divisions. It is the responsibility of laboratory management to provide the encouragement, environment, and recognition necessary to foster such collaborations.

The Advanced Network Technologies Division stated that its mission is to expedite the development and deployment of next-generation networking technologies for the transmission of multimedia data streams to enable heterogeneous, collaborative computing.

The division is to be commended for working on projects closely related to both the laboratory and the NIST missions.

Much of the work presented to the panel had high technical merit and was appropriate. Of particular value were the projects on the next generation Internet Protocol Suite (IPS), the modeling and analysis of Asynchronous Transfer Mode (ATM) network protocols, the development of conformance tests, and the work on Private Network Node Interfaces. NIST's reputation as an unbiased evaluator of technologies led to this division's simulations of Hybrid Fiber/Coaxial (HFC) networks being used by the Institute of Electrical and Electronics Engineers (IEEE) 802.14 standards group to choose a preferred Medium Access Control (MAC) protocol. The multimedia, video-on-demand work for the Digital Audio Visual Council (DAVIC) was of

good scientific quality but it was not timely; the division's efforts in this field came in an area that had already been tried and essentially abandoned by the outside community. In the panel 's opinion, the intention to steer this group's efforts toward “video-on-the-net” and potentially toward using Java in such work is a much more appropriate direction. The division also performs interoperability testing, coordinates “bake-offs” of transmission protocols for International Multimedia Teleconferencing Consortium (IMTC), and is involved in work on videoconferencing. In general, the Advanced Network Technologies Division has successfully undergone a significant shift in mission and culture. One example is the change in focus from open systems interconnection (OSI) networks to projects related to the Internet and multimedia.

The division has done an excellent job of disseminating information through the ATM forum, the Internet Engineering Task Force, DAVIC, IMTC, the Cross Industry Working Team, and IEEE as well as through individual interactions with assorted companies, agencies, and universities. The Internetworking Technologies Group has made particularly effective use of joint projects to disseminate its programs. A good example of this group's efforts are the testbeds for internetwork protocol version 6 (IPv6) and the “6-Bone,” an international virtual IPv6 backbone.

The division's programs—particularly its work on the IPS and on simulations for ATM and HFC networks—do have industrial impact. The IPS project is a key piece of a large national and international effort, and the ATM and HFC simulators have been used by industry forums to model their networks. The video-on-demand program probably has little current industrial impact, but the group has developed good technical skills that could be used in other video-related efforts as DAVIC and similar organizations evolve.

The panel was not given information that would allow specific evaluation of the effectiveness of the division's impact assessment. Much of the information was anecdotal rather than quantitative in nature.

Funding for the Advanced Network Technologies Division (in thousands of dollars):

Divisional personnel includes 32 FTP staff, of whom 25 are technical professionals.

The division appears to have adequate funding, equipment, and facilities but is understaffed. Roughly 10 slots in the division are unfilled, including all of the spaces allotted to the Wireless Communications Technology Group.

Generally, the division is attentive to the needs of industry, universities, and other federal agencies and has planned effectively to provide leadership and support in appropriately chosen areas. The IPS and ATM/HFC programs are adequate and well planned. The multimedia effort appears to be in transition, and no formal plans for it were presented. The Wireless Group is dormant, but an effort is being made to hire a group manager who would be responsible for staffing the group. Both the redirection of the video-on-demand effort and the reorganization of the Wireless Group are key issues for the division.

Information technology is a dynamic field, and projects must be constantly reexamined for relevance. For example, the division's videoconferencing research has not recently been reviewed to determine if an opportunity and need for standards and interoperability testing exists. Also, though sponsoring and organizing “bake-off” activities is appropriate work for the division, the venues of such events are currently limited. Perhaps the large professional audiences in attendance at exhibitions and conferences such as the Special Interest Group on Computer Graphics would benefit from observing and participating in these activities.

The Computer Security Division stated its mission as follows: With the growth of electronic commerce and the increased use of distributed systems linked by networks, the need to ensure the security of data and the privacy of information becomes critical. The Computer Security Division provides guidance and technical assistance to government and industry in the protection of unclassified automated information systems.

The Computer Security Division has aligned its programs closely with the laboratory's mission. The division's strategy is built on its mission, and the programs are focused appropriately on the following six technical thrusts: cryptographic technology and applications, advanced authentication, public key infrastructure (PKI), internetworking security, criteria and assurance, and security management and support.

This division is superbly run and does its job well. Work of particular importance to the information security community includes the division 's cooperative effort with industry to develop an Advanced Encryption Standard; the Federal Computer Incident Response Capability, which provides incident response services to federal agencies; development of the Common

Criteria for testing and evaluation of security systems and products; and establishment of a national Testing Competency Center.

The present array of projects is appropriate, but the field of computer security can change quickly. The panel recognizes three issues that it believes affect divisional programs. First, although PKIs are important and standards to facilitate their use and interworking are needed, the legal implications of PKI are still unsettled. A digital signature should be a proxy for a real one, which has always served as a legally binding act. The first major court case on the acceptability of digital signatures has yet to be heard—but it will. When it does, many of the questions that this division is struggling to answer will be decided by the courts or by various legislatures. The uncertain future of PKI implies that the Computer Security Division must be careful not to overinvest in programs that might become obsolete or irrelevant when the courts join the debate.

Second, cryptography is only one aspect of systems security; assurance has much wider properties. The Common Criteria Program is useful and will probably expand as federal agencies come under pressure to demonstrate the security of their systems to auditors. There will be a growing drive to move assurance up-market from a product to a system, even though the technological base for such a shift is weak. This division may be affected by the demand for such a change.

Finally, in recent years, the International Organization for Standardization (ISO) has developed a series of process standards governing quality assurance (ISO 9000) and environmental management (ISO 14000). Within the next few years, the Europeans will likely try to have ISO do the same for systems security. If such an activity is dominated by the European security industry, it could seriously impede the ability of U.S. companies to extend their networks into Europe. If only for defensive purposes, the United States will need an institution that is prepared to participate intelligently in such deliberations either directly, as the official American arbiter of standards, or indirectly through representatives from U.S. firms.

Each of the division's activities has been selected for its potential benefits to both the public and private sectors of the information security community. The industrial impact of the division's programs is monitored to ensure that each project not only has “customers” but also has several groups who serve as partners or collaborators in the work. The division identifies and maintains a close relationship with these collaborators to ensure the most effective use of skills and resources. The types of organizations that can be considered collaborators are universities, commercial research and development organizations, standards bodies, federal, state and local governments, and some foreign institutions.

Funding for the Computer Security Division (in thousands of dollars):

Divisional personnel includes 40 FTP staff, of whom 35 are technical professionals.

Significant growth in the division's resources is planned, both through higher STRS funding from Congress and through a National Security Agency (NSA) grant for work on a joint test center. An increase in funding is well justified by the division's plans for new and expanded work; however, there are two potential pitfalls. Overall, NIST's physical facilities resources are scarce. Therefore, if the Information Technology Laboratory does not carefully husband its available space, this division will not have room to grow. In addition, there is concern over whether new resources will be spent on programs that remain aligned with the current laboratory mission. Without close scrutiny, potential involvement in projects such as the National Information Infrastructure Protection Program could adversely affect the division's activities by allocating scarce resources to tasks that do not directly support the laboratory mission.

Management has recently begun to emphasize appropriate planning, and the division's planning process has improved significantly over the past few years. Each project is reviewed by asking why NIST should do the work and whether one or more of NIST's collaborators could do the same work. In the spring of 1997, the division held an off-site meeting to carry out a 5-year vision exercise. The results should provide significant guidance to the division in the future. Such a vision is an important part of ensuring that the long-term view is routinely addressed in divisional planning and prioritization. There is no staff member currently designated to concentrate on forecasting what will be needed in computer security.

Although many of the division's programs are in support of the metrology mission of the laboratory, research is also being done on generating new security standards. This division may also be asked one day to provide basic support and guidance to civilian government agencies as computer security issues enter the mainstream. With this wide range of tasks, the Computer Security Division programs might suffer if judged using the narrow criteria employed by other divisions.

The Information Access and User Interfaces Division stated that its mission is to accelerate the development of technologies that allow intuitive, efficient access, manipulation, and exchange of complex information by facilitating the creation of measurement methods and standards. These technologies include the digitization and representation of multimedia data, and the use of spoken and written natural language and visual interactive modalities for search and presentation of that information. Through collaboration with industry, academia, and government, the division coordinates and provides evaluation methodologies, test suites and corpora, prototypes, workshops, and standards and guidelines to enable faster transition into the commercial marketplace.

The mission of the division is well aligned with the laboratory's goal of stimulating the growth and adoption of new technology by helping industry to develop better, higher-quality products. This mission is also consistent with NIST's charge to conduct research programs that address industry's need for measurement and infrastructure technology.

The division's activities fall in four major areas: spoken natural language processing, natural language processing and information retrieval, visual image processing, and visualization and virtual reality. These programs are designed to provide reference corpora for U.S. industries and to coordinate benchmark testing within the research and development community.

Overall, the division's programs are of high quality. The activities have contributed significantly to the advancement of state-of-the-art systems and are considered significant by the research community. The NIST-coordinated benchmark tests using standard databases have supported objective evaluation of various systems and improved the performance of these systems. The programs in visual image processing and in visualization and virtual reality are relatively new, and it is difficult to comment on their success. However, their goals are appropriate and address emerging industry needs in the processing of multimedia information.

The division works closely with the research community in industry and universities. Research results are shared at workshops and through timely publication. The World Wide Web is another valuable avenue for disseminating information about division programs and services. It is important to use such mechanisms not only to disseminate NIST work, but also to learn about the research other institutions are conducting in fields of interest to this division. For example, in the Text Processing and Information Retrieval Group, the panel was concerned by an apparent lack of awareness of relevant work on search engines being done outside NIST.

The speech, text, and other corpora distributed by NIST on CD-ROMs are an extremely valuable component of the division's work. These programs have had a strong impact on the

industry and led to the development of new information access and user interface technologies. Nonetheless, there is no systematic process in place that allows the division to track and assess the results of such projects or to measure their impact quantitatively over time.

Funding for the Information Access and User Interfaces Division (in thousands of dollars):

Divisional personnel includes 47 FTP positions, of whom 41 are technical professionals.

The panel considers these resources adequate for carrying out the programs effectively. Current levels of OA support are appropriate.

Historically, the Information Access and User Interfaces Division has received a great deal of funding from the Defense Advanced Research Projects Agency and the National Security Agency. Many current programs have evolved from projects designed for and funded by these agencies. As dependence on outside support decreases, the division will be responsible for developing a proper internal process for project selection and planning. A key element of such a program is alignment with the divisional and laboratory missions. In addition, a thorough understanding of current research being done outside NIST on any proposed program is also important.

The High-Performance Systems and Services Division stated that its mission is to enable effective application of high-performance computing and communications systems (HPCC) in support of NIST and its interactions with industry, academia, the federal government and the public by conducting research, development, and evaluation of innovative measurement and test methods, system architectures and software technologies for improved scaleability, functionality, flexibility, reliability, and economy of HPCC; serving as a testbed for research and development

in high-performance computing and information technologies, and gaining experience in the deployment of these technologies and assessing their functional capabilities, interoperability and operational characteristics; serving as a responsive, effective mission-critical resource spanning computational, communication, mass storage, security, archival, and scientific visualization services; and providing and managing state-of-the-art facilities that integrate and support an enterprise-wide heterogeneous information technology environment for NIST.

The High Performance Systems and Services Division has combined service and research very effectively. The research programs have been reshaped and are now consistent with the new laboratory vision. Many of the activities have had an impact on the significant players in the high-performance computing industry. The division has also made a serious attempt to understand who its NIST customers are and how the division's work can benefit them.

In a few cases, such as work on a electronic book or on the optical pattern recognition testbed, the panel was unclear how these projects fit the division's mission. Division management conceded that these activities reflected the interests of the personnel assigned to the division in the recent reorganization. The division's name does not currently describe the total scope of its programs; a more inclusive name like “Systems Lab and Services” might be more appropriate.

The research highlights presented to the panel were impressive and have had a significant impact on industry. These projects included WebSubmit, a Web-based user interface that simplifies access to NIST 's IBM SP2 supercomputer, thus making parallel computing accessible to a broader user community. There is a serious effort under way to improve this product further through a collaborative project with the Computer Security Division. The division has also developed two tools designed to improve performance of parallel programs: MultiKron, a performance measurement instrument, and S-Check (sensitivity checker), a tuner for complex concurrent programs. New and unforeseen uses for the MultiKron technology previously developed by the division have also been found. S-Check could potentially have a large impact on the field of parallel computing. Preparing for widespread dissemination usually includes external evaluation of the product. Another project that combines testing work and research is the effort on message passing interface (MPI) interoperability. At the request of the leading vendors of high-performance computing, the division serves as an unbiased assessor of the interoperability of various MPI implementations and is developing testing protocols for the industry.

On the whole, the panel was presented with little information about the service functions of the division. It was therefore difficult to assess how well the service needs of NIST were being met. The upgrade from the Cray Y/MP 4 to the Cray C96 appears to have gone smoothly and greatly expanded computing capability. The division continues to increase the number of nodes in the IBM SP2, but in general, the transition of the user load to parallel machines is progressing too slowly; the SP2 is still not heavily utilized. It is important that users recognize and take advantage of the unique capabilities of parallel processing; to treat this machine as a collection of single workstations is a waste. To ensure that this transition occurs, in-house consulting expertise in parallelization is crucial to expedite the shift of a substantial portion of

NIST's user base to parallel computers, which are emerging as the most cost-effective high-end machines.

The division is beginning to consider whether clusters might also be a cost-effective and efficient alternative to supercomputers for NIST's less demanding computing requirements. Close examination of the workload characteristics of the laboratories may suggest different possible interconnection methods, such as fast ethernet or Myrinet. This activity is an appropriate task for the service part of the division, but the panel is somewhat skeptical that a cluster testbed can be a rich source of research results, because this field has already been well explored outside of NIST.

For each of the moderately mature projects, the division has identified private-sector customers interested in its technology. Some examples are IBM and Ford's interest in WebSubmit; the incorporation of a functional subset of MultiKron into modern microprocessors by Intel; and the freeware distribution of S-Check by Silicon Graphics, Inc., and its potential commercialization by Hewlett-Packard. The work on MPI interoperability is supported by the Interoperability Message Passage Interface (IMPI) Steering Committee, which includes most of the major high-performance computing vendors. This division is extremely conscious of the need not only to spread information about its activities, but also to demonstrate the extent of this dissemination to NIST and to the panel. The use of timelines to art the development of projects and their potential and actual industrial impact is an effective approach.

Funding for the High-Performance Systems and Services Division (in thousands of dollars):

Divisional personnel includes 75 FTP positions, including 53 for technical professionals.

The projects described in this assessment clearly demonstrate how good development projects can grow out of this division's service role. For example, the work on WebSubmit began as an effort to encourage scientists from the Chemical Science and Technology Laboratory to use the SP2 more often. Customer impact appeared to play a significant role in their choice of projects.

A sensible allocation policy for the high-end machines is needed. The new supercomputer and the upgraded parallel machine are still relatively uncrowded, and therefore the division has not yet been forced to prioritize demands. A plan that is developed now, rather than after oversubscription becomes a problem, would be more useful. The division intends to form a laboratorywide user's group to give feedback in the planning process. The division's approach to prioritizing help requests in parallelization is similar. At the moment, there is little demand, and the division staff are proactively seeking out collaborations with productive NIST scientists who could quickly benefit from parallel computing. As the interest in using parallel processing increases, the division may need to develop a formal criterion based on a project's possible scientific impact to determine who most deserves such assistance in parallel computing.

In some areas, division personnel do not clearly know the status of related work outside of NIST. Careful studies of what is available from outside sources are missing from the planning process for new projects. Using such information could help avoid duplication of effort.

According to the mission statement of this division, as provided by NIST, the Distributed Computing and Information Services Division provides the information technology resources, supporting infrastructure, applied research, and assistance to NIST staff, collaborators, and clients for application in the conduct of scientific, engineering and administrative applications, and in the dissemination of information. This includes: an easy-to-use, robust, secure distributed heterogeneous environment with support for desktop systems and workstations; network capabilities; information services; visualization and graphics services; access to external and mobile users; common computing environments, information access tools, software development tools and specialized applications software; site-wide hardware maintenance for standardized desktop systems and workstations; site-wide software licensing, maintenance, and repositories for standardized platforms and applications; and large-scale testbeds, advanced prototypes and reliable systems as part of the continuous improvement in scope and quality of service.

The Distributed Computing and Information Services Division is a new group, created during the formation of the Information Technology Laboratory to unite various service functions. These information processing services (which include support for both computers and networks) currently being treated as a subsidiary piece of a single laboratory. However, in many other institutions they are housed instead in a separate information processing operation that provides technology services to the entire institution. Such an operation can charge directly for its services, rather than providing them under the umbrella of overhead payments, as is done at

NIST. The costs charged for such services are then justified by a system that assesses the quality of technical support provided. Such an approach leads both to an honest cost-accounting system and to appropriate recognition in the community of the value of the work performed. At NIST, the current approach seems to lead to an undervaluing of the technical support within the wider NIST community. There is no process in place that objectively measures of the quality of service, so there is no way to evaluate the support provided or track year-by-year improvements. In industry, a services operation of this sort is often appropriately headed by a Chief Information Officer who reports directly to the company president and who is responsible for ensuring quality service.

The Distributed Computing and Information Services Division is chartered to provide limited hardware and software support to all NIST operating units. The major accomplishments of its first year have been to create and enhance many information technology services, such as e-mail, calendaring, and mail-list processing. A significant portion of this division's efforts have been devoted to the successful replacement of previously fragile NIST-wide services with considerably more robust programs and systems.

Funding for the Distributed Computing and Information Services Division (in thousands of dollars):

Divisional personnel includes 66 FTP positions, including 56 for technical professionals.

The Distributed Computing and Information Services Division is understaffed for the service job it is assigned. Although there may be enough slots allotted to fulfill the tasks assigned to this division, there are also a significant number of unfilled job openings. Although new staff members have been hired, the division has an ongoing loss of personnel. This turnover apparently happens because NIST cannot pay market-rate salaries for the information technology professionals needed. In addition to this personnel shortage, there is not enough funding for the software tools required to manage a state-of-the-art distributed computing environment. The

lack of necessary instrumentation affects the division's effectiveness not only in maintaining a distributed computing environment but also in monitoring or measuring the quality of service it offers.

This division supports administrative computing for NIST, computing services in the general NIST research community, and computing for computer science research. This broad range of responsibilities leads to conflicts in the allocation of resources. Moreover, the current structure leads to an unnatural separation of distributed computing service support and networking service support. Though this division may reflect the organization of high-performance computing and networking research in separate groups, it is an inappropriate and inefficient way to manage the technical support functions of a client server computing service.

The Software Diagnostics and Conformance Testing Division stated that its mission is to develop software testing tools and methods that improve quality, conformance to standards, and correctness. This division also participates with industry in the development of forward-looking standards and leads efforts for conformance testing, even at the early development stage of standards.

This mission statement is consistent with the goals of the Information Technology Laboratory and of NIST. It also encompasses a broad range of possible activities for the division.

Several of the projects undertaken in the Software Diagnostics and Conformance Testing Division represent an interesting contrast to the traditional conformance testing paradigm. One such project is a Virtual Reality Modeling Language (VRML) test suite. A strong impetus from users convinced the division that NIST's involvement at the early stages of VRML development could have a large impact on the technical community. The application domain for VRML is potentially substantial, and work in this area will highlight NIST's usefulness to the industry. The tests will develop concurrently with the language and will thus enable sensitive points in the language's definitions to be identified. The resultant high quality of the finished product will demonstrate the importance of careful specification during a language's evolution.

The Role Based Access Control (RBAC) project is a different but still unconventional approach to standards work. Working with the Computer Security Division on product development, the Software Diagnostics and Performance Testing Division's goal is to simultaneously produce a corresponding test suite that will evolve with the product from concept

to implementation. The final output will be a Web-based program to control information access through an explicit model of the roles and relationships within an enterprise or organization. This project has been considered high risk from the outset. Its success will be measured by industry recognition of this new approach and by whether patents are submitted by NIST personnel.

The Unravel Program Slicer is a long-standing project within this division. The version for the C language has been completed and is attracting some interest from companies such as Ford and MITRE as well as government agencies such as the NSA and the Nuclear Regulatory Commission. The Commission initially sought this technology about 10 years ago and has helped fund the development process. The division is now considering whether similar Slicers could be built for C++ or Java. The panel believes that the large community of practicing professionals working with Java might find such a tool useful. However, in general, since the slicing technology is quite old, extending it to other applications might not support the division's mission.

The project on Standard Reference Data for Software Quality is an interesting new undertaking designed to gather error, fault, and failure data across projects and organizations. The ability to generate profiles and recognize common sources of errors would be exceedingly helpful to the software industry.

The division has decided to conclude its long-time support of SQL Test Suite development efforts and to allow industry to conduct future testing activities. Several appropriate companies have expressed interest, and the transition is under way. For example, testing was scheduled to begin at the National Software Testing Laboratory (NSTL) on July 1, 1997, and the test method and procedures used by NSTL will be traceable back to NIST's SQL test method and procedures. In addition, NIST's test suite is currently available for no charge on the World Wide Web. NIST will continue to send an expert in object-oriented programming to the meetings of the national SQL standards committee (X3H2) but will no longer serve as official international representative of that committee. NIST will also be available to the SQL community to answer questions and provide advice, although active work on SQL conformance test development will now occur in the private sector and not at NIST. The Open Group, an international consortium of information technology users and suppliers, is considering developing a conformance test suite for SQL3 and providing certification for SQL3 implementations based on that suite. Reactions from database users and producers regarding NIST's decision have varied, and some industry representatives have expressed strong disapproval of the laboratory's position. Nevertheless, the panel believes that NIST has properly considered the issue. This conclusion of the SQL testing program is part of and consistent with the Information Technology Laboratory's shift in focus away from traditional paper standards and the performance of conformance tests and toward work on conformance development for emerging technologies. This transition is strongly endorsed by the panel.

The panel was pleased to find that this division makes use of less traditional indicators of effectiveness, such as patent applications and early industry adoption. Of course, the usual method of result dissemination, such as reports, journal publications, and conference presentations, are also used and tracked.

The programs presented to the panel had various levels of industrial involvement. In the case of the work on VRML, the division's efforts in direct response to requests from users. This project is applying verification procedures during the language definition process. Taking this approach will allow a novice community with little experience in developing a language standard to generate a high-quality product. The division's tactics are a highly original approach to standardization activities, and the panel expects this work to be quite effective.

The RBAC project represents an alternative to traditional conformance-based testing. Early industry reaction has been positive and encouraging. Standard Reference Data for Software Quality is also a new project that has had little time to garner industry reactions. The panel believes that the data produced from such a project would be welcomed by the software industry and widely used.

The division is clearly able to identify industrial users. However, direct feedback from these users about the benefits of the division 's work would be more convincing. For example, comments made in a published paper demonstrate a clear appreciation of the public availability of the Unravel tool, but the panel was not supplied with any information directly from industrial users. The ad hoc nature of the impact data is insufficient, and the division would benefit from more direct input from its customer community. It is necessary to keep systematic records and to track all such interactions over the course of a project.

Funding for the Software Diagnostics and Conformance Testing Division (in thousands of dollars):

Divisional personnel includes 54 FTP positions, including 46 for technical professionals.

The panel finds that the core skills of the staff are adequate, and no resource limitations were noted. However, as the NSA becomes more involved in the RBAC effort with this division and the Computer Security Division, the lack of available space for NSA researchers may be a constraint.

The division presented a diverse array of projects to the panel. This range of topics raised some concerns about what procedures the division uses to select projects. The stated selection criteria and the characteristics of the projects described to the panel were not well connected. The project selection process did not obviously include a well-defined set of goals or a unifying theme, such as automatic test generation. It was also unclear whether an intended outcome was attached to each project considered for future work.

The stated mission of the Statistical Engineering Division is to catalyze scientific and industrial research through the application of statistical methods to the experimentation and data analysis underlying the empirical information gathering processes critical to NIST scientists and engineers.

This division has always had a clear sense of direction, as evidenced by its continued dedication to its client-collaborators. This focus on service and joint projects has not been lost during the laboratory 's reorganization. However, the panel sees some inconsistencies between the stated goals of the Information Technology Laboratory and the mission of the Statistical Engineering Division. For example, the role of this division is not well articulated in the laboratory mission statement. Furthermore, the main task of this division—provide high-quality statistical consulting and collaboration to all of NIST—is not explicitly included in the laboratory's list of guiding values, as laid out in the Strategic Plan.² Similarly, the mission statement of the Statistical Engineering Division makes no reference to developing statistical applications for information technology. The white paper, “Computational Science in the Information Technology Laboratory,” fails to make a compelling case for the central role of computational and statistical science within the laboratory.³

The Statistical Engineering Division has played an integral and successful role in the core business of NIST. It continues to do high-quality work in general and in its traditional collaborations throughout the laboratories. This division has been actively developing Web products, most notably the SEMATECH (semiconductor manufacturing technology) Handbook

and the Statistical Reference Datasets. These efforts demonstrate the power of Web technology and efficiently disseminate NIST products.

The division is making a notable drive to routinize development of Standard Reference Materials (SRMs). This approach will allow the division to use resources more efficiently and to free up time for projects related to information technology. Another important program is the division's ongoing role in the statistical education of other NIST staff. The division's annual report is an impressive demonstration of the diversity of the division's collaborators within NIST. The division also plans summer meetings with the leadership from other NIST laboratories to get feedback on current joint projects and to assist in selecting future work in high-priority areas.

In addition, the division appears to be making a good-faith effort to develop significant relationships with the divisions of the former Computer Systems Laboratory. The goal is to direct one-third of the division's efforts toward projects in information technology. Examples of such work can be seen in the division's half dozen “ITL Project Starts,” which include a project on algorithms for random number testing with applications in computer security. Another positive step is the recent hiring of an expert in complex data sets whose experience is directly relevant to problems in information technology. The national statistics community is beginning to extend statistical science to help solve new problems of information technology; this division is in an excellent position to provide leadership in such pursuits by fostering contacts with industry, academic groups, and other national laboratories.

In general, this division continues to affect industry indirectly through collaborations with other divisions and laboratories. In this case, assessment of the resulting benefits to U.S. companies is best carried out by the collaborating units. In addition, the division's work on the World Wide Web promises to have a direct and important impact. The effectiveness of such products will be indicated by how often the sites are visited and their contents downloaded.

Funding for the Statistical Engineering Division (in thousands of dollars):

Divisional personnel includes 30 FTP positions, including 27 for technical professionals.

This division's location at NIST North is a serious problem. For a group whose major focus is on consulting and collaboration, close proximity to current and potential clients is essential. Effective collaboration is enhanced by chance encounters. Divisional personnel have reported a drop in interactions with other NIST scientists, which will slow down this division's efforts to fulfill its collaborative mission.

Another issue is confusion about availability of staff resources to support computing systems in the division. Hardware and software maintenance currently appears inadequate. The laboratory budget indicates an annual allocation of more than $0.5 million for such support in the Statistical Engineering and Mathematical and Computational Sciences Divisions, but the staff of these divisions report difficulty in obtaining timely service.

The Information Technology Laboratory's mission does not explicitly refer to the value of the Statistical Engineering Division's role within NIST or to its contribution to solving problems in information technology. This omission has a negative effect on the division's morale. Furthermore, the laboratory may not provide adequate resources for this division to continue carrying out its traditional consulting mission while giving high priority to new intralaboratory collaborations on information technology. Without such supplemental support, the staff of the Statistical Engineering Division will be unable to maintain the current high quality of their activities.

• The Information Technology Laboratory's mission statement does not reflect the importance either of the service functions performed by several of its divisions or of the collaborative and research activities of the Statistical Engineering and Mathematical and Computational Sciences Divisions. Both omissions affect staff morale. For the service groups, this problem is compounded by the fact that their work is not appreciated by the rest of NIST, in part because it is paid for out of overhead and hence perceived as free. In the collaborative divisions, their work has not been fully integrated into the Information Technology Laboratory's activities and mission because insufficient efforts have been made both by the divisions in question and by the rest of the laboratory.

• The Information Technology Laboratory has made innovative use of the World Wide Web, both in research and as a dissemination tool.

• Information technology is a dynamic area with an extremely high demand for skilled people in all roles. The Information Technology Laboratory 's ability to attract and retain such employees is limited by NIST 's salary structure.

• The Information Technology Laboratory has made significant progress in refining how older programs are selected for termination and new areas chosen for funding. However, the laboratory still does not pay enough attention to issues such as who has initiated a potential

project (whether NIST, industry, or other government agencies) and what truly unique competence or quality the laboratory can provide in that area.

• The Information Technology Laboratory has increased its involvement with industry by being more proactive about seeking out new partners. However, the laboratory's database of industrial contacts does not contain detailed enough information on the results and consequences of its technical activities. The ability of the laboratory and NIST to justify the value of their work may be impeded by this insufficient tracking.

• The current NIST-wide digital network “backbone” of 100 Mbits/sec is insufficient for current and future needs of NIST programs.