Chapter 8

Information Technology Laboratory



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Chapter 8 Information Technology Laboratory

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 PANEL MEMBERS Ralph Z. Roskies, University of Pittsburgh, Co-Chair Brian W. Kernighan, Lucent Technologies, Co-Chair Bishnu S. Atal, Lucent Technologies Cheryl Y. Begandy, Alcoa Technical Center Mary Ellen Bock, Purdue University Allen L. Brown, Jr., Cinebase Software, Inc. Dorothy E. Denning, Georgetown University Edward A. Feigenbaum, Chief Scientist of the Air Force Andrew S. Grimshaw, University of Virginia Carl M. Harris, George Mason University Thomas P. Kehler, CONNECT, Inc. Ellen M. Knapp, Coopers & Lybrand Sandra M. Lambert, Lambert & Associates Martin C. Libicki, Institute for National Strategic Studies Carl Machover, Machover Associates Corporation Richard E. Nance, Virginia Tech Jerome Sacks, National Institute of Statistical Sciences Daniel L. Solomon, North Carolina State University Ahmed N. Tantawy, IBM Corporation Raymond T. Yeh, International Software Systems, Inc. Eddie L. Zeitler, Charles Schwab & Company Submitted for the panel by its Co-Chairs, Ralph Z. Roskies and Brian W. Kernighan, this assessment of the fiscal year 1997 activities of the Information Technology Laboratory is based on a site visit by the panel on March 19–21, 1997, and on documents provided by the laboratory.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 LABORATORY-LEVEL REVIEW Laboratory Mission The Information Technology Laboratory (ITL) stated that its mission is to stimulate U.S. economic growth and industrial competitiveness through technical leadership and collaborative research in critical infrastructural technologies to promote better development and use of information technology. The Information Technology Laboratory carries out this mission by working with industry, research, and government organizations to develop and demonstrate tests, test methods, reference data, proof of concept implementations and other infrastructural technologies that are needed by U.S. industry to produce information technology systems that are usable, secure, scaleable, and interoperable. The focus and clarity of this mission indicate significant progress since the panel's 1995 assessment of the laboratory. The laboratory 's emphasis on metrology for information technology and on the establishment of a national test and measurement infrastructure for information technology is quite promising. This vision points in a sound direction for the future and integrates well with the overall NIST mission. The panel recognizes the importance of standards for interoperability and system integration and therefore hopes that the mission is broadly construed to include work on standards and their underlying technology. Though the panel approves of the goals laid out in this statement, this mission as currently constructed does not reflect the present composition of the Information Technology Laboratory. The laboratory embraces three distinct functions, all of which are important and vital to NIST: (1) the new focus on information technology metrology; (2) computing services; and (3) collaborative research in computational and statistical science. The latter two elements are not adequately reflected in the mission and were not well represented in the laboratorywide goals presented to the panel. These two areas have a long, established role at NIST, and their continued contribution is critical. In the panel's discussion of this lack of coherence, members noted that many institutions have responded to difficulties of this sort by moving the service functions into a separate organization. In this instance, that would include essentially all of the Distributed Computing and Information Services Division and perhaps parts of other divisions. Currently, laboratory management spends a disproportionate amount of time on the service component, yet staff are discouraged and frustrated. The rest of the NIST community often believes that they are not well served and that technology support should be a free commodity. In addition, the panel believes that the collaborative research function of the Statistical Engineering and Mathematical and Computational Sciences Divisions could be integrated more thoroughly with the rest of the Information Technology Laboratory. The NIST-wide role and importance of these divisions and their work are not featured prominently enough in the laboratory mission statement, yet their functions are basic to NIST's mission. At the same time, these groups have not yet adapted to the current organizational structure and its requirements. In particular, the Mathematics and Computational Science Division needs to be more responsive regarding work on issues related to information technology. In this division, significantly greater sensitivity to customer needs and outreach is needed. But other divisions within the Information Technology Laboratory could also be more supportive and make more of an effort to work actively with their mathematics and statistics colleagues.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Technical Merit and Appropriateness of Work The quality of the programs in the laboratory are discussed in detail in each divisional report. Industrial Impact The Information Technology Laboratory has been especially responsive to the need for increased involvement with industry. Many some divisions have a proactive focus on customers and make a real effort to seek new partners. Although some areas still need to be improved, the focus, alertness, and conscientiousness of the effort are laudable. The laboratory has also done a good job in disseminating its results to the industrial community, particularly through innovative use of Web technology. This method is easy to use, cost effective, and readily accessible to both large and small companies. The components from which the Information Technology Laboratory was formed have a long history of industrial impact, although such interactions were not explicitly part of their mission. Some examples of past programs still in use today include various standards for computer languages, like Ada, Cobol, and Structured Query Language (SQL). These languages have mature, stable user communities, and so there is now little need for NIST's special competence in these areas. Related projects at the laboratory are being passed on to private companies or industrial consortia while being phased out at NIST. This shift reflects a general trend away from paper standards but is also a result of the laboratory's move away from classical ex post facto standards, particularly Federal Information Processing Standards. This trend away from the traditional paper standards (as produced by committee meetings) and toward early de facto standards (as defined by test suites, conformance tests, and reference implementations) requires the Information Technology Laboratory to remain in close touch with the communities in which standards are evolving. The present effort to develop a set of tests for the Virtual Reality Modeling Language (VRML) is a good example of this kind of activity: The impetus for the project came from an industry consortium rather than an individual company. In addition, NIST is perceived correctly as neutral and can provide the particular competence appropriate for a standard that requires considerable mathematical and graphical sophistication. Past and current work to provide reference datasets, tests, and test services also has significant industrial impact. Some current examples are work on optical character recognition in handwriting and fingerprints, programs for conformance testing, and technology evaluation conferences (known as “bake-offs”) for speech recognition software. In each of these cases, the Information Technology Laboratory's combination of expertise and neutrality provide a unique resource and have significant impact on industry. The statistical and mathematical work of the laboratory also has significant effect on industry, both directly through the dissemination of informational packages like the “Guide to Available Mathematical Software,” and indirectly, through the collaborations between personnel in the Statistical Engineering and Mathematical and Computational Sciences Divisions and other scientists throughout NIST. A key element of industrial interaction should be a mechanism to assess the laboratory's impact on U.S. companies. A clear method of tracking industrial contacts would assist in such assessments as well as improve the laboratory's ability to demonstrate the value of its programs

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 to NIST management, Congress, and other potential industrial customers. The laboratory maintains a database of industrial contacts and activities, but it appears to be more a list of names and meetings than an organized repository of information. A more complete record would document how a given activity began, who has been involved from NIST and from outside, what work has been done, how much time has been devoted to the project on each side, what adjustments have been made over its life cycle, what external publications or artifacts have been produced, which industries have been affected, how they have benefited from NIST's work, what acknowledgments of the laboratory's efforts have been made, what possible connections to future activities or other industries exist, and further data of this sort. The purpose of such information gathering is not to impose reporting requirements on the laboratory, but to make it easier for NIST to accurately assess its external interactions and impact, to help demonstrate NIST's value, and to concentrate available resources most effectively. Laboratory Resources A large majority of Information Technology Laboratory funding is NIST internal appropriations. This comes from three sources: (1) the Scientific and Technical Research and Services (STRS), which is the direct appropriation from Congress for the NIST laboratories; (2) payments from NIST-wide overhead to support services provided by the Information Technology Laboratory to all of NIST; and (3) Industrial Technology Services (ITS), which includes the Advanced Technology Program and the Manufacturing Extension Partnership. The remainder of the laboratory's funding is from other agencies (OA) and from sales and services related to reference materials produced by the laboratory. In the charts in this report, the portion of the STRS funding specifically allotted for Consolidated Scientific Computing Systems (CSCS), the supercomputing facilities, is listed separately. Funding for the Information Technology Laboratory (in millions of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 31.015 32.195 CSCS (Supercomputing) 12.098 12.290 Overhead (NIST) 9.528 11.505 ITS 1.269 0.768 OA 12.667 9.636 Reference Materials 0.192 0.889 Total 66.769 67.283 The panel supports the continuing reduction in the laboratory's dependence on OA funding. However, some level of outside support is valuable, not only as a source of additional resources, but also as a validation of the laboratory's efforts. It is important to strike a balance between these benefits of OA funding and the danger that resources will be diverted to projects that do not support the laboratory 's mission.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 The space allotted to the Information Technology Laboratory is adequate at the moment but allows no room for future growth. Increasing external collaborations (particularly in the Computer Security and Software Diagnostics and Conformance Testing Divisions) will require expanded facilities. The Mathematical and Computational Sciences and Statistical Engineering Divisions are now located in NIST North. This placement inhibits their collaborations with the scientists in other NIST laboratories, although it may improve the interactions among the divisions within the Information Technology Laboratory. The major problem with technical resources is that the NIST-wide digital network “backbone” of 100 Mbits/sec is insufficient for the present and future needs of all the laboratories. Computer facilities are mostly adequate at the moment, but a computer generation is only 2 to 3 years long and equipment must be upgraded frequently. The technical support for individual workstations seems inadequate, especially in the Mathematical and Computational Sciences and Statistical Engineering Divisions. The cause of this lack of support was not clear to the panel. In general, the allocation of service resources is a NIST-wide issue that has a significant effect on the Information Technology Laboratory. The distribution of services is not governed by any market mechanism but is instead supported through overhead and hence appears free. Thus the other laboratories tend to undervalue the services provided by the Information Technology Laboratory. Further comments on this issue are made in the section on Laboratory Planning. At the moment, the staff of the Information Technology Laboratory includes 420 full-time permanent (FTP) positions, of which 340 are for technical professionals. There are also 45 guest researchers. Current staff are quite strong and qualified, but the laboratory is experiencing difficulties in hiring new personnel. There is intense market demand for information technology professionals. The Information Technology Laboratory is handicapped because the starting salary levels at NIST are not competitive in this environment. One way of compensating for lower-than-market salaries is to provide a better-than-market work environment, including opportunities for publication. Laboratory Planning The panel was pleased to see the progress in planning made by the Information Technology Laboratory over the last 2 years. The laboratory has a coherent picture of its emerging goals, focused on the information technology industry in the U.S. and consistent with the overall NIST mission. The laboratory has made considerable progress in understanding the role of metrology in information technology and the ways in which metrology and testing can help U.S. industry be more productive and competitive (see e.g., the recent NIST white paper on Metrology for Information Technology).1 The move away from performing conformance tests and toward developing the tests and the related methodology for industrial use is a positive step. 1   U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Information Technology Laboratory Planning and Project Reports, NIST internal document, National Institute of Standards and Technology, Gaithersburg, Md., 1997, pp. 101–122.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 The effects of significant and conscious planning by the laboratory are reflected in the new focus and in better-defined rules for project selection and sunsetting. Under the laboratory director's effective leadership, the laboratory has assumed an increased emphasis on both industrial needs and the unique resources and skills of the Information Technology Laboratory. Laboratory managers and staff have made many hard decisions, and the resulting set of programs projects a clearer vision of the future of information technology and of the role of the laboratory. However, timelines are an important part of project planning but were missing from most of the projects presented to the panel. Such life-cycle descriptions assist in assessing the progress of programs and the effectiveness of the planning process. The Information Technology Laboratory has taken significant steps toward establishing sound criteria for resource allocation among programs. Nevertheless, the panel believes that there is still some room for improvement in the way projects are chosen. A few projects still appear to be continuing for historic rather than programmatic reasons; these are cited in the divisional assessments. The criteria noted by the laboratory as key elements in the selection process —such as impetus from industry, the need for NIST to serve as an honest broker, or the presence of a special competence available only in the Information Technology Laboratory—are generally useful in deciding whether an activity is appropriate for the laboratory. It is also highly desirable that projects target fairly young technologies so that changes can be made before industry practices have hardened. It is in these early stages that the tests, advice on standardization, and reference implementations provided by the laboratory are most valuable to industry. The Information Technology Laboratory's major contributions to standards will come through such preemptive activities that help voluntary standards groups define and strengthen rapidly evolving standards. Participation in such industrial standards groups is a more important activity for the Information Technology Laboratory than its work as an arbiter of conformance to mature standards. The panel was also concerned that in some cases the laboratory has insufficient understanding of the status of research being done external to NIST before undertaking projects. The amount of planning done by the laboratory is reflected in the copious documentation of the planning process. The materials provided to the panel were of high quality, and spoke directly to issues raised in previous assessments. Now that the need for laboratorywide planning efforts is decreasing, the task of unifying divisional management and staff behind the new mission is the primary challenge. DIVISIONAL ASSESSMENTS Mathematical and Computational Sciences Division Mission The Mathematical and Computational Science Division stated its mission to be providing technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to American industry. This division has created a unified analytical and computational setting in which advanced scientific and engineering problems can

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 be solved. The components of such a setting include: the development and analysis of theoretical descriptions of phenomena (mathematical modeling); the design and analysis of the requisite computational methods and experiments; the transformation of these methods into efficient numerical algorithms for high-performance computers; the implementation of these methods in high-quality mathematical software; and the distribution of this software to potential clients. The mission of the Mathematical and Computational Sciences Division reflects its historical role of providing analytical and computational tools and engaging in collaborative research with other NIST divisions. These services are critical to the NIST mission because they provide vital computational capabilities. However, neither the laboratory nor the division mission statement reflects the role that this division could play within the Information Technology Laboratory. The laboratory director has placed an increased emphasis on research in information technology metrology, yet such a shift in focus is not apparent in the Mathematical and Computational Sciences Division. This confusion is reflected in the absence of information technology in the divisional mission statement as well as in the lack of a well-articulated role for the Mathematical and Computational Sciences Division in the laboratory 's strategic plan. The paucity of collaborative activities between the Mathematical and Computational Sciences Division and the other divisions of the Information Technology Laboratory is a symptom of this disjointedness. Technical Merit and Appropriateness of Work The work of the Mathematical and Computational Sciences Division is generally of high quality. The staff is well respected in the applied and computational mathematics research community, and the division often attracts some of the best in this field to come to NIST for terms as guest researchers. Examples of particularly innovative or important works in progress in the division include the Matrix Market, a Web-based repository of matrix test data that has had more that 8,000 visitors who have downloaded more than 18,000 matrices; a Web-based set of basic linear algebra subprograms (sparse BLAS) for sparse matrices, which allows users to generate source code from the Web; and process models that enable the calibration of complex devices such as coordinate measuring machines or the brain probes used during neural surgery. The division's work on compression algorithms raised some concerns for the panel. It was unclear to what extent the group at NIST was aware of the state of the art in this field and understood the most pressing issues in the area. For example, it was unclear that the group was aware of the types of compression problems that would come up in a compression discussion over the Internet. The Mathematical Software Group has disseminated its work effectively through the World Wide Web. Their products have reached an extraordinarily high number of working computational scientists in industry, academia, and other research facilities. The Web has served not merely as a new medium for publication, but also as a tool to add value to their work (as in the case of sparse BLAS, which is capable of generating actual code that can perform fundamental sparse matrix operations). In general, this innovative use of the Web is commendable.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Industrial Impact The entire division has not focused enough on defining a realistic measure of the value of its work. A great deal of this division' s modeling and simulation work is done in collaboration with scientists in other NIST laboratories; an example is work on a paint weathering simulation with personnel from the Building and Fire Research Laboratory. The other laboratories then interact with their industrial partners. Because this division is not in direct contact with U.S. companies, it is obliged to define a new way to measure its effectiveness. One approach would be to treat collaborators from other laboratories as the customers and to assess both the depth of their interest in and the benefits they receive from working on projects with Mathematical and Computational Sciences Division staff. Some quantitative analysis of the value added by this division's efforts on industrially relevant projects with other laboratories could supplement the present counting of publications as a measure of effectiveness. Resources Funding for the Mathematical and Computational Sciences Division (in thousands of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 3,596.4 4,825.9 CSCS (Supercomputing) 1,595.3 1,090.0 ITS 164.0 225.0 OA 104.7 190.5 Total 5,460.4 6,331.4 Divisional personnel includes 51 FTP staff, of whom 48 are technical professionals. There is confusion about whether Information Technology Laboratory resources are available to support the Mathematical and Computational Sciences Division's workstations and to provide computing services. Although funding for these activities is apparently in the laboratory budget, the division has not seen the resulting services. In general, the division's facilities are of good quality. However, the location of the Mathematical and Computational Sciences Division in NIST North has had a direct detrimental impact on the ability to collaborate with scientists in other laboratories. The remote location cuts down on informal interactions and the chance encounters with colleagues that lead to identification of new research opportunities. If this division is to expand its technical capabilities in a direction better suited to an emphasis on information technology, it needs to hire a discrete mathematician.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Planning In the past, the division has had no official planning process to ensure unity of purpose in its efforts. Although it is important that personnel from all levels be able to suggest possible projects, the inauguration of formal meetings with the leadership from other NIST laboratories to determine general mathematical and computational science needs will probably bring some order to divisional project selection. A more structured process could also allow laboratory management to work on aligning divisional programs with the Information Technology Laboratory's mission. The Mathematical Software Group has made a significant effort to relate its work to the overall mission of helping industry by providing tests and test data, performing interoperability testing, and developing metrics to assess software performance. However, in general divisional personnel have done little outreach to other Information Technology Laboratory divisions involved in information technology metrology work. The Mathematical and Computational Sciences Division staff does not yet have a strategy to foster and identify joint proposals with the other Information Technology Laboratory divisions in this field. Such an effort might include educating their colleagues in applications of modeling, simulation, and other computational methods to research in the areas that interest other divisions. It is worth noting that there has been a corresponding lack of proposals or projects suggested to the Mathematical and Computational Sciences Division by the other divisions. It is the responsibility of laboratory management to provide the encouragement, environment, and recognition necessary to foster such collaborations. Advanced Network Technologies Division Mission The Advanced Network Technologies Division stated that its mission is to expedite the development and deployment of next-generation networking technologies for the transmission of multimedia data streams to enable heterogeneous, collaborative computing. The division is to be commended for working on projects closely related to both the laboratory and the NIST missions. Technical Merit and Appropriateness of Work Much of the work presented to the panel had high technical merit and was appropriate. Of particular value were the projects on the next generation Internet Protocol Suite (IPS), the modeling and analysis of Asynchronous Transfer Mode (ATM) network protocols, the development of conformance tests, and the work on Private Network Node Interfaces. NIST's reputation as an unbiased evaluator of technologies led to this division's simulations of Hybrid Fiber/Coaxial (HFC) networks being used by the Institute of Electrical and Electronics Engineers (IEEE) 802.14 standards group to choose a preferred Medium Access Control (MAC) protocol. The multimedia, video-on-demand work for the Digital Audio Visual Council (DAVIC) was of

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 good scientific quality but it was not timely; the division's efforts in this field came in an area that had already been tried and essentially abandoned by the outside community. In the panel 's opinion, the intention to steer this group's efforts toward “video-on-the-net” and potentially toward using Java in such work is a much more appropriate direction. The division also performs interoperability testing, coordinates “bake-offs” of transmission protocols for International Multimedia Teleconferencing Consortium (IMTC), and is involved in work on videoconferencing. In general, the Advanced Network Technologies Division has successfully undergone a significant shift in mission and culture. One example is the change in focus from open systems interconnection (OSI) networks to projects related to the Internet and multimedia. The division has done an excellent job of disseminating information through the ATM forum, the Internet Engineering Task Force, DAVIC, IMTC, the Cross Industry Working Team, and IEEE as well as through individual interactions with assorted companies, agencies, and universities. The Internetworking Technologies Group has made particularly effective use of joint projects to disseminate its programs. A good example of this group's efforts are the testbeds for internetwork protocol version 6 (IPv6) and the “6-Bone,” an international virtual IPv6 backbone. Industrial Impact The division's programs—particularly its work on the IPS and on simulations for ATM and HFC networks—do have industrial impact. The IPS project is a key piece of a large national and international effort, and the ATM and HFC simulators have been used by industry forums to model their networks. The video-on-demand program probably has little current industrial impact, but the group has developed good technical skills that could be used in other video-related efforts as DAVIC and similar organizations evolve. The panel was not given information that would allow specific evaluation of the effectiveness of the division's impact assessment. Much of the information was anecdotal rather than quantitative in nature. Resources Funding for the Advanced Network Technologies Division (in thousands of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 3,161.6 3,858.4 ITS 250.0 175.0 OA 1,548.6 1,105.6 Total 4,960.2 5,139.0 Divisional personnel includes 32 FTP staff, of whom 25 are technical professionals.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 NIST's user base to parallel computers, which are emerging as the most cost-effective high-end machines. The division is beginning to consider whether clusters might also be a cost-effective and efficient alternative to supercomputers for NIST's less demanding computing requirements. Close examination of the workload characteristics of the laboratories may suggest different possible interconnection methods, such as fast ethernet or Myrinet. This activity is an appropriate task for the service part of the division, but the panel is somewhat skeptical that a cluster testbed can be a rich source of research results, because this field has already been well explored outside of NIST. Industrial Impact For each of the moderately mature projects, the division has identified private-sector customers interested in its technology. Some examples are IBM and Ford's interest in WebSubmit; the incorporation of a functional subset of MultiKron into modern microprocessors by Intel; and the freeware distribution of S-Check by Silicon Graphics, Inc., and its potential commercialization by Hewlett-Packard. The work on MPI interoperability is supported by the Interoperability Message Passage Interface (IMPI) Steering Committee, which includes most of the major high-performance computing vendors. This division is extremely conscious of the need not only to spread information about its activities, but also to demonstrate the extent of this dissemination to NIST and to the panel. The use of timelines to art the development of projects and their potential and actual industrial impact is an effective approach. Resources Funding for the High-Performance Systems and Services Division (in thousands of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 2,324.4 2,244.7 CSCS (Supercomputing) 4,471.8 9,038.1 Overhead (NIST) 4,947.9 5,212.0 ITS 249.1 0.0 OA 1,050.0 1,118.7 Reference Materials 0.0 843.9 Total 13,043.2 18,457.4 Divisional personnel includes 75 FTP positions, including 53 for technical professionals.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Planning The projects described in this assessment clearly demonstrate how good development projects can grow out of this division's service role. For example, the work on WebSubmit began as an effort to encourage scientists from the Chemical Science and Technology Laboratory to use the SP2 more often. Customer impact appeared to play a significant role in their choice of projects. A sensible allocation policy for the high-end machines is needed. The new supercomputer and the upgraded parallel machine are still relatively uncrowded, and therefore the division has not yet been forced to prioritize demands. A plan that is developed now, rather than after oversubscription becomes a problem, would be more useful. The division intends to form a laboratorywide user's group to give feedback in the planning process. The division's approach to prioritizing help requests in parallelization is similar. At the moment, there is little demand, and the division staff are proactively seeking out collaborations with productive NIST scientists who could quickly benefit from parallel computing. As the interest in using parallel processing increases, the division may need to develop a formal criterion based on a project's possible scientific impact to determine who most deserves such assistance in parallel computing. In some areas, division personnel do not clearly know the status of related work outside of NIST. Careful studies of what is available from outside sources are missing from the planning process for new projects. Using such information could help avoid duplication of effort. Distributed Computing and Information Services Division Mission According to the mission statement of this division, as provided by NIST, the Distributed Computing and Information Services Division provides the information technology resources, supporting infrastructure, applied research, and assistance to NIST staff, collaborators, and clients for application in the conduct of scientific, engineering and administrative applications, and in the dissemination of information. This includes: an easy-to-use, robust, secure distributed heterogeneous environment with support for desktop systems and workstations; network capabilities; information services; visualization and graphics services; access to external and mobile users; common computing environments, information access tools, software development tools and specialized applications software; site-wide hardware maintenance for standardized desktop systems and workstations; site-wide software licensing, maintenance, and repositories for standardized platforms and applications; and large-scale testbeds, advanced prototypes and reliable systems as part of the continuous improvement in scope and quality of service. The Distributed Computing and Information Services Division is a new group, created during the formation of the Information Technology Laboratory to unite various service functions. These information processing services (which include support for both computers and networks) currently being treated as a subsidiary piece of a single laboratory. However, in many other institutions they are housed instead in a separate information processing operation that provides technology services to the entire institution. Such an operation can charge directly for its services, rather than providing them under the umbrella of overhead payments, as is done at

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 NIST. The costs charged for such services are then justified by a system that assesses the quality of technical support provided. Such an approach leads both to an honest cost-accounting system and to appropriate recognition in the community of the value of the work performed. At NIST, the current approach seems to lead to an undervaluing of the technical support within the wider NIST community. There is no process in place that objectively measures of the quality of service, so there is no way to evaluate the support provided or track year-by-year improvements. In industry, a services operation of this sort is often appropriately headed by a Chief Information Officer who reports directly to the company president and who is responsible for ensuring quality service. Technical Merit and Appropriateness of Work The Distributed Computing and Information Services Division is chartered to provide limited hardware and software support to all NIST operating units. The major accomplishments of its first year have been to create and enhance many information technology services, such as e-mail, calendaring, and mail-list processing. A significant portion of this division's efforts have been devoted to the successful replacement of previously fragile NIST-wide services with considerably more robust programs and systems. Resources Funding for the Distributed Computing and Information Services Division (in thousands of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 966.3 540.0 CSCS (Supercomputing) 2,482.8 948.1 Overhead (NIST) 4,134.2 5,831.6 ITS 115.7 33.3 OA 50.0 86.7 Reference Materials 0.0 15.0 Total 7,749.0 7,454.7 Divisional personnel includes 66 FTP positions, including 56 for technical professionals. The Distributed Computing and Information Services Division is understaffed for the service job it is assigned. Although there may be enough slots allotted to fulfill the tasks assigned to this division, there are also a significant number of unfilled job openings. Although new staff members have been hired, the division has an ongoing loss of personnel. This turnover apparently happens because NIST cannot pay market-rate salaries for the information technology professionals needed. In addition to this personnel shortage, there is not enough funding for the software tools required to manage a state-of-the-art distributed computing environment. The

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 lack of necessary instrumentation affects the division's effectiveness not only in maintaining a distributed computing environment but also in monitoring or measuring the quality of service it offers. Planning This division supports administrative computing for NIST, computing services in the general NIST research community, and computing for computer science research. This broad range of responsibilities leads to conflicts in the allocation of resources. Moreover, the current structure leads to an unnatural separation of distributed computing service support and networking service support. Though this division may reflect the organization of high-performance computing and networking research in separate groups, it is an inappropriate and inefficient way to manage the technical support functions of a client server computing service. Software Diagnostics and Conformance Testing Division Mission The Software Diagnostics and Conformance Testing Division stated that its mission is to develop software testing tools and methods that improve quality, conformance to standards, and correctness. This division also participates with industry in the development of forward-looking standards and leads efforts for conformance testing, even at the early development stage of standards. This mission statement is consistent with the goals of the Information Technology Laboratory and of NIST. It also encompasses a broad range of possible activities for the division. Technical Merit and Appropriateness of Work Several of the projects undertaken in the Software Diagnostics and Conformance Testing Division represent an interesting contrast to the traditional conformance testing paradigm. One such project is a Virtual Reality Modeling Language (VRML) test suite. A strong impetus from users convinced the division that NIST's involvement at the early stages of VRML development could have a large impact on the technical community. The application domain for VRML is potentially substantial, and work in this area will highlight NIST's usefulness to the industry. The tests will develop concurrently with the language and will thus enable sensitive points in the language's definitions to be identified. The resultant high quality of the finished product will demonstrate the importance of careful specification during a language's evolution. The Role Based Access Control (RBAC) project is a different but still unconventional approach to standards work. Working with the Computer Security Division on product development, the Software Diagnostics and Performance Testing Division's goal is to simultaneously produce a corresponding test suite that will evolve with the product from concept

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 to implementation. The final output will be a Web-based program to control information access through an explicit model of the roles and relationships within an enterprise or organization. This project has been considered high risk from the outset. Its success will be measured by industry recognition of this new approach and by whether patents are submitted by NIST personnel. The Unravel Program Slicer is a long-standing project within this division. The version for the C language has been completed and is attracting some interest from companies such as Ford and MITRE as well as government agencies such as the NSA and the Nuclear Regulatory Commission. The Commission initially sought this technology about 10 years ago and has helped fund the development process. The division is now considering whether similar Slicers could be built for C++ or Java. The panel believes that the large community of practicing professionals working with Java might find such a tool useful. However, in general, since the slicing technology is quite old, extending it to other applications might not support the division's mission. The project on Standard Reference Data for Software Quality is an interesting new undertaking designed to gather error, fault, and failure data across projects and organizations. The ability to generate profiles and recognize common sources of errors would be exceedingly helpful to the software industry. The division has decided to conclude its long-time support of SQL Test Suite development efforts and to allow industry to conduct future testing activities. Several appropriate companies have expressed interest, and the transition is under way. For example, testing was scheduled to begin at the National Software Testing Laboratory (NSTL) on July 1, 1997, and the test method and procedures used by NSTL will be traceable back to NIST's SQL test method and procedures. In addition, NIST's test suite is currently available for no charge on the World Wide Web. NIST will continue to send an expert in object-oriented programming to the meetings of the national SQL standards committee (X3H2) but will no longer serve as official international representative of that committee. NIST will also be available to the SQL community to answer questions and provide advice, although active work on SQL conformance test development will now occur in the private sector and not at NIST. The Open Group, an international consortium of information technology users and suppliers, is considering developing a conformance test suite for SQL3 and providing certification for SQL3 implementations based on that suite. Reactions from database users and producers regarding NIST's decision have varied, and some industry representatives have expressed strong disapproval of the laboratory's position. Nevertheless, the panel believes that NIST has properly considered the issue. This conclusion of the SQL testing program is part of and consistent with the Information Technology Laboratory's shift in focus away from traditional paper standards and the performance of conformance tests and toward work on conformance development for emerging technologies. This transition is strongly endorsed by the panel. The panel was pleased to find that this division makes use of less traditional indicators of effectiveness, such as patent applications and early industry adoption. Of course, the usual method of result dissemination, such as reports, journal publications, and conference presentations, are also used and tracked.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Industrial Impact The programs presented to the panel had various levels of industrial involvement. In the case of the work on VRML, the division's efforts in direct response to requests from users. This project is applying verification procedures during the language definition process. Taking this approach will allow a novice community with little experience in developing a language standard to generate a high-quality product. The division's tactics are a highly original approach to standardization activities, and the panel expects this work to be quite effective. The RBAC project represents an alternative to traditional conformance-based testing. Early industry reaction has been positive and encouraging. Standard Reference Data for Software Quality is also a new project that has had little time to garner industry reactions. The panel believes that the data produced from such a project would be welcomed by the software industry and widely used. The division is clearly able to identify industrial users. However, direct feedback from these users about the benefits of the division 's work would be more convincing. For example, comments made in a published paper demonstrate a clear appreciation of the public availability of the Unravel tool, but the panel was not supplied with any information directly from industrial users. The ad hoc nature of the impact data is insufficient, and the division would benefit from more direct input from its customer community. It is necessary to keep systematic records and to track all such interactions over the course of a project. Resources Funding for the Software Diagnostics and Conformance Testing Division (in thousands of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 6,771.0 5,095.0 ITS 279.4 235.0 OA 3,440.0 2,458.1 Total 10,490.4 7,788.1 Divisional personnel includes 54 FTP positions, including 46 for technical professionals. The panel finds that the core skills of the staff are adequate, and no resource limitations were noted. However, as the NSA becomes more involved in the RBAC effort with this division and the Computer Security Division, the lack of available space for NSA researchers may be a constraint.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 Planning The division presented a diverse array of projects to the panel. This range of topics raised some concerns about what procedures the division uses to select projects. The stated selection criteria and the characteristics of the projects described to the panel were not well connected. The project selection process did not obviously include a well-defined set of goals or a unifying theme, such as automatic test generation. It was also unclear whether an intended outcome was attached to each project considered for future work. Statistical Engineering Division Mission The stated mission of the Statistical Engineering Division is to catalyze scientific and industrial research through the application of statistical methods to the experimentation and data analysis underlying the empirical information gathering processes critical to NIST scientists and engineers. This division has always had a clear sense of direction, as evidenced by its continued dedication to its client-collaborators. This focus on service and joint projects has not been lost during the laboratory 's reorganization. However, the panel sees some inconsistencies between the stated goals of the Information Technology Laboratory and the mission of the Statistical Engineering Division. For example, the role of this division is not well articulated in the laboratory mission statement. Furthermore, the main task of this division—provide high-quality statistical consulting and collaboration to all of NIST—is not explicitly included in the laboratory's list of guiding values, as laid out in the Strategic Plan.2 Similarly, the mission statement of the Statistical Engineering Division makes no reference to developing statistical applications for information technology. The white paper, “Computational Science in the Information Technology Laboratory,” fails to make a compelling case for the central role of computational and statistical science within the laboratory.3 Technical Merit and Appropriateness of Work The Statistical Engineering Division has played an integral and successful role in the core business of NIST. It continues to do high-quality work in general and in its traditional collaborations throughout the laboratories. This division has been actively developing Web products, most notably the SEMATECH (semiconductor manufacturing technology) Handbook 2   U.S. Department of Commerce, Technology Adminitration, National Institute of Standards and Technology, Information Technology Laboratory Strategic Plan, Revised, NIST internal document, National Institute of Standards and Technology, Gaithersburg, Md., 1997, p. 2. 3   U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Information Technology Laboratory Planning and Project Reports, NIST internal document, National Institute of Standards and Technology, Gaithersburg, Md., 1997, pp. 31–41.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 and the Statistical Reference Datasets. These efforts demonstrate the power of Web technology and efficiently disseminate NIST products. The division is making a notable drive to routinize development of Standard Reference Materials (SRMs). This approach will allow the division to use resources more efficiently and to free up time for projects related to information technology. Another important program is the division's ongoing role in the statistical education of other NIST staff. The division's annual report is an impressive demonstration of the diversity of the division's collaborators within NIST. The division also plans summer meetings with the leadership from other NIST laboratories to get feedback on current joint projects and to assist in selecting future work in high-priority areas. In addition, the division appears to be making a good-faith effort to develop significant relationships with the divisions of the former Computer Systems Laboratory. The goal is to direct one-third of the division's efforts toward projects in information technology. Examples of such work can be seen in the division's half dozen “ITL Project Starts,” which include a project on algorithms for random number testing with applications in computer security. Another positive step is the recent hiring of an expert in complex data sets whose experience is directly relevant to problems in information technology. The national statistics community is beginning to extend statistical science to help solve new problems of information technology; this division is in an excellent position to provide leadership in such pursuits by fostering contacts with industry, academic groups, and other national laboratories. Industrial Impact In general, this division continues to affect industry indirectly through collaborations with other divisions and laboratories. In this case, assessment of the resulting benefits to U.S. companies is best carried out by the collaborating units. In addition, the division's work on the World Wide Web promises to have a direct and important impact. The effectiveness of such products will be indicated by how often the sites are visited and their contents downloaded. Resources Funding for the Statistical Engineering Division (in thousands of dollars):   Fiscal Year 1996 Fiscal Year 1997 (estimated) STRS (NIST) 3,006.2 3,273.1 CSCS (Supercomputing) 263.0 550.0 OA 19.6 64.0 Reference Materials 132.0 30.0 Total 3,420.8 3,917.1 Divisional personnel includes 30 FTP positions, including 27 for technical professionals.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 This division's location at NIST North is a serious problem. For a group whose major focus is on consulting and collaboration, close proximity to current and potential clients is essential. Effective collaboration is enhanced by chance encounters. Divisional personnel have reported a drop in interactions with other NIST scientists, which will slow down this division's efforts to fulfill its collaborative mission. Another issue is confusion about availability of staff resources to support computing systems in the division. Hardware and software maintenance currently appears inadequate. The laboratory budget indicates an annual allocation of more than $0.5 million for such support in the Statistical Engineering and Mathematical and Computational Sciences Divisions, but the staff of these divisions report difficulty in obtaining timely service. Planning The Information Technology Laboratory's mission does not explicitly refer to the value of the Statistical Engineering Division's role within NIST or to its contribution to solving problems in information technology. This omission has a negative effect on the division's morale. Furthermore, the laboratory may not provide adequate resources for this division to continue carrying out its traditional consulting mission while giving high priority to new intralaboratory collaborations on information technology. Without such supplemental support, the staff of the Statistical Engineering Division will be unable to maintain the current high quality of their activities. MAJOR OBSERVATIONS The Information Technology Laboratory's mission statement does not reflect the importance either of the service functions performed by several of its divisions or of the collaborative and research activities of the Statistical Engineering and Mathematical and Computational Sciences Divisions. Both omissions affect staff morale. For the service groups, this problem is compounded by the fact that their work is not appreciated by the rest of NIST, in part because it is paid for out of overhead and hence perceived as free. In the collaborative divisions, their work has not been fully integrated into the Information Technology Laboratory's activities and mission because insufficient efforts have been made both by the divisions in question and by the rest of the laboratory. The Information Technology Laboratory has made innovative use of the World Wide Web, both in research and as a dissemination tool. Information technology is a dynamic area with an extremely high demand for skilled people in all roles. The Information Technology Laboratory 's ability to attract and retain such employees is limited by NIST 's salary structure. The Information Technology Laboratory has made significant progress in refining how older programs are selected for termination and new areas chosen for funding. However, the laboratory still does not pay enough attention to issues such as who has initiated a potential

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 project (whether NIST, industry, or other government agencies) and what truly unique competence or quality the laboratory can provide in that area. The Information Technology Laboratory has increased its involvement with industry by being more proactive about seeking out new partners. However, the laboratory's database of industrial contacts does not contain detailed enough information on the results and consequences of its technical activities. The ability of the laboratory and NIST to justify the value of their work may be impeded by this insufficient tracking. The current NIST-wide digital network “backbone” of 100 Mbits/sec is insufficient for current and future needs of NIST programs.

OCR for page 177
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1997 This page in the original is blank.