National Academies Press: OpenBook
« Previous: Building and Fire Research Laboratory
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

8

Information Technology Laboratory

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

PANEL MEMBERS

Louise H. Trevillyan, IBM T.J. Watson Research Center, Chair

Tony Scott, General Motors Corporation, Vice Chair

Bishnu S. Atal, AT&T Laboratories Research

Matthew Bishop, University of California, Davis

Linda Branagan, Construct Internet Design

Jaime Carbonell, Carnegie Mellon University

Josephine Cheng, IBM Santa Teresa Laboratory

Albert M. Erisman, The Boeing Company

John R. Gilbert, Xerox Palo Alto Research Center

Roscoe C. Giles, Boston University

Andrew S. Grimshaw, University of Virginia

Thomas P. Kehler, Recipio

Jon R. Kettenring, Telcordia Technologies

John W. McCredie, University of California, Berkeley

Vijayan N. Nair, University of Michigan

Lawrence O' Gorman, Veridicom, Inc.

Thomas Parenty, Consultant, Oakland, California

Craig Partridge, BBN Technologies

Radia Perlman, Sun Microsystems Laboratories

James L. Phillips, The Boeing Company

K.K. Ramakrishnan, AT&T Laboratories Research

Stephanie M. White, Long Island University

Eddie L. Zeitler, Charles Schwab & Company

Submitted for the panel by its Chair, Louise H. Trevillyan, and its Vice Chair, Tony Scott, this assessment of the fiscal year 2000 activities of the Information Technology Laboratory is based on a site visit by the panel on February 23-24, 2000, in Gaithersburg, Md., and documents provided by the laboratory.1

1  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, 1999 Information Technology Laboratory Technical Accomplishments, NISTIR 6365, National Institute of Standards and Technology, Gaithersburg, Md., 2000.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

LABORATORY-LEVEL REVIEW

Laboratory Mission

According to laboratory documentation, the mission of the Information Technology Laboratory (ITL) is to strengthen the U.S. economy and improve the quality of life by working with industry to develop and apply technology, measurements, and standards in information technology.

In support of this mission, the ITL works with industry, research, and government organizations to develop the information technology standards, conformance tests, test methods, reference data, proof-of-concept implementations, and other tools that are needed by U.S. industry to produce usable, secure, scalable, and interoperable information technology systems.

The ITL has eight divisions. Four of these—the Advanced Network Technologies, Computer Security, Information Access and User Interfaces, and Software Diagnostics and Conformance Testing Divisions—are focused primarily on performing research and developing measurement methods and standards in information technology in support of the ITL and NIST missions. The four other divisions of ITL are devoted to service and collaborative support. The High Performance Systems and Services Division and the Distributed Computing and Information Services Division provide support for computer systems and software used in ITL and other NIST laboratories, as well as for the NIST-wide infrastructure. The Mathematical and Computational Sciences Division and the Statistical Engineering Division provide technical assistance and advice on analysis, modeling, and statistics through collaborations with staff throughout NIST. This range of activities makes ITL an especially heterogeneous laboratory, with attendant management challenges. Each division has a key role to play and all of the components—research, measurement, standards, and services—should be valued appropriately by management.

An area that may need some management attention is communication of the laboratory and divisional missions to the staff. In its interactions with employees from across ITL, the panel found some confusion about what the mission statements were, what they meant, and how they related to the projects that staff were working on. It also found a belief that some “customers” were more valued than others. For example, some staff believed that more credit is given for work done with people in industry than for work with people in government and that more credit is given for work with other government agencies (for example, the Federal Bureau of Investigation (FBI)) than for work with other staff within NIST.

It has been more than 4 years since the ITL was formed by combining two laboratories. Management at all levels has turned over since the merger, and ITL seems to have largely solved the problems associated with the transition. In general, the various divisions seem to be pulling together to form a coherent team. However, some areas still require attention. In particular, the Statistical Engineering Division has been trying to recruit a permanent division chief. The panel believes that a high priority should be given to finding a strong leader to fill this position, a person who can help the division redefine its long-term strategic plan and carry out its important role within ITL and across NIST. This issue is discussed further in the review of the Statistical Engineering Division.

Technical Merit and Appropriateness of Work

Examples of the many excellent projects under way in the laboratory are discussed in detail in the divisional reports. Overall, the programs are appropriate and well aligned with laboratory and divisional missions.

Last year, the panel noted that the ITL planning process had improved, and the panel observed continued progress this year, including the development of a laboratory-wide strategic plan for 2000

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

2004.2 New projects are started based on their relevance to U.S. industry, and the work examined by the panel generally had specific, focused goals. An example is the work on the Extensible Markup Language (XML) done in the Software Diagnostics and Conformance Testing Division. This project is notable because it involves a major standard on which industry was making little progress and it also effectively leverages skills and approaches developed in a now terminated project on Virtual Reality Modeling Language (VRML). An important new activity is ITL 's critical contribution in emerging standards for local wireless networks, where NIST can again play the role of honest broker to help harmonize many competing technologies. Another aspect of the improved planning process is the existence of clear termination criteria, which have been useful for sunsetting projects. For example, when the Real-Time Java project and the work on role-based access control (RBAC) satisfied their completion criteria in 1999, they were ended.

One concern of the panel is that, with the explosive growth of information technologies, more metrology work is needed on faster schedules. This increased pace implies that decisions on which standards NIST should support must be made quickly, often with incomplete information, and that in the future ITL will have to take more risks in project initiation. It is important for NIST to get involved early in areas where standards are not set yet and there is no industry consensus. Proactive work by ITL is vital to shape industry products and behavior; because technology changes so rapidly, coming in late in the process would be ineffective. Good examples of areas in which ITL has already demonstrated the benefits of proactive involvement include the work on electronic books in the High Performance Systems and Services Division, the project on architectural description languages (ADLs) in the Software Diagnostics and Conformance Testing Division, and the standards development work in the Java Numerics project in the Mathematical and Computational Sciences Division.

This year, the panel viewed two major new programs in the ITL, one in pervasive computing and the other in security. At the time of last year's panel meeting, the pervasive computing initiative was just beginning, and the fiscal year 1999 assessment contained several cautionary statements regarding the panel's views of the appropriate role of ITL in this field. Specifically, the report said that the laboratory should concentrate on topics where NIST possesses special expertise or has a unique role to play, such as involvement in early standards activities, coordinating industry consortia, and building reference implementations. This year, the panel was pleased to discover that the laboratory had made significant progress on more clearly defining the initiative. The NIST workshops on smart spaces and pervasive computing were well connected and provided a good start for building relationships with the community. The panel also approved of many of the individual projects, especially in the Advanced Network Technologies Division. However, there are still some cases in which there has not been a focus on producing work unique to ITL; specific examples are discussed in the divisional reports. The laboratory as a whole has still not managed to define the specialized role of NIST in this area: Exactly how should the elements of pervasive computing be measured, where are standards most necessary, and what tools will be needed for these tasks? To ensure that ITL has an impact, it must continue to strive to identify its mission in interoperability, standardization, and metrology for pervasive computing. Work in this field is rapidly expanding in industry and academia, and NIST must take care not to duplicate other efforts but rather to complement and support emerging technologies. This challenge is an issue both in individual projects and at the level of the laboratory as a whole. The pervasive computing work would benefit from a formal management-coordination structure across the entire laboratory. Such a structure would assist in answering the questions posed above and would also

2  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, ITL Strategic Plan January 2000, National Institute of Standards and Technology, Gaithersburg, Md., 2000.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

facilitate collaboration across divisional lines. While the panel observed that informal cooperation already exists among ITL divisions, such cooperation is critical to the success of ITL in the pervasive computing area, and there must be a management system in place to ensure that the effort hangs together coherently across the laboratory.

The other major program planned for the ITL is a collection of three security initiatives proposed in the President's budget for fiscal year 2001. While the final details and shape of the program will not be clear until the budget process is complete, provisional plans were presented to the panel. The three components of this effort are (1) the Institute for Information Infrastructure Protection, through which NIST will coordinate the distribution of approximately $50 million in grants for security technology development; (2) the critical infrastructure protection activities, for which NIST is slated to receive $5 million per year to identify best security practices and to develop tools to detect and remedy security vulnerabilities; and (3) the Expert Review Team, which would be funded at about $2 million per year and would create eight new permanent positions at NIST with responsibility for identifying existing vulnerabilities in federal computer systems and providing advice on fixing them. ITL's Computer Security Division is well known and well respected internationally, so ITL is an obvious home for this collection of security programs. The panel is looking forward to heating next year about the progress that has been made on these security initiatives but has two warnings for the laboratory. First, the market for personnel with computer security expertise is extremely tight. The ITL Computer Security Division has been losing staff, so these initiatives would require NIST to recruit replacements as well as fill a significant number of new slots. In the current environment, it may be very difficult to find these people in a timely fashion. Second, the projected funding for the initiatives does not match the scope of the programs and goals presented to the panel. For these programs to be successful, projects must be chosen carefully to leverage existing expertise of ITL staff, and specific and achievable goals must be carefully defined.

Finally, the panel continues to be concerned about the absence of a coherent strategy and architecture for providing information technology support to NIST. The current situation has services being supplied by an amalgam of groups, including the High Performance Systems and Services and the Distributed Computing and Information Services Divisions in ITL, several external companies (by means of outsourcing contracts), and, occasionally, significant collections of staff in units outside ITL (e.g., as the Manufacturing Engineering Laboratory's Service Administration). This is a costly and inefficient system. The NIST-wide Information Technology Services Planning Team (ITSPT) formed to tackle the problem held its first meeting in February 2000. The group includes representatives from the management of all NIST operating units, and the panel applauds this initiative. To most efficiently determine the best system for information technology support within NIST, the panel suggests that two key ingredients be added to the ITSPT's effort. First is the formation of a broadly based users' forum, including scientific and administrative staff from throughout NIST. These end users are significantly affected by the NIST service structure, and a forum would allow the ITSPT to receive feedback about the current system, to solicit input into the evaluation process, and to communicate ideas and proposed architectures. Second is the involvement of current information technology support staff in the evaluation and (potential) reorganization; the management and staff of the High Performance Systems and Services and the Distributed Computing and Information Services Divisions have valuable experience in the information technology (IT) needs of NIST's heterogeneous organization. To define and implement an effective and efficient information technology services architecture, NIST-wide management, end users, and support staff will all have to buy into the new system; by involving them at the start, the chances for success are increased. The panel notes that with the reevaluation being undertaken by the ITSPT, NIST also has an opportunity to study what gains in efficiency and effectiveness can be achieved

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

by increasing the standardization of its IT infrastructure. Standardization cannot be dictated but instead should be developed in the best interests of a broad range of customers. To move in the direction of more standardization will require working closely with the end users in the laboratories to identify (1) research that requires experimental platforms outside of any standards and (2) areas where standard platforms would be adequate for the work and bring value through commonality of training, support, and procurement.

Impact of Programs

The ITL continues to have a positive impact on the industrial IT community. Its portfolio of projects is extensive and appropriate and generally chosen to maximize NIST's impact. ITL has been effective in influencing industry, especially in areas such as XML, where industry was stalled in coming to agreement. The recent work on numerical computation in Java is another example of how ITL can have a positive impact on industry. The laboratory uses many mechanisms to interact effectively with industry. A wide variety of consortia have been set up by and for information technology companies, and NIST plays a role in the most important of them. The laboratory also makes good use of industry roadmaps as a guide to initiating new projects. Where no roadmaps exist, ITL uses its influence as a neutral party to help project industry's needs through the use of publications, workshops, and collaboration with industry consortia. The panel singled out the following ITL activities as having had a great impact on U.S. industry: specifications for instructional management systems (IMS) for distance learning, development of role-based access control protocols, modeling of material microstructures, conformance testing for XML, the Text Retrieval Conference (TREC) series, drafting of the Common Criteria for information security, and visualization work with other NIST laboratories. More information about these projects and many others that are filling critical needs in the industrial IT community is provided in the divisional assessments.

In addition to interacting with external organizations, ITL also provides vital services to the entire community within NIST. Laboratory staff provide support for networking, high-performance computing, desktop boxes and workstations, and other systems vital to NIST' s infrastructure. The Statistical Engineering and the Mathematical and Computational Sciences Divisions provide valuable support in their specialties to other NIST laboratories and within ITL. All of these support activities are crucial to NIST, and ITL should make it clear to the employees providing these services that their contributions are appreciated and that their work in support of NIST's staff is essential to the valuable role NIST plays in the U.S. economy.

Laboratory Resources

Funding sources for the Information Technology Laboratory are shown in Table 8.1. As of January 2000, staffing for the Information Technology Laboratory included 381 full-time permanent positions, of which 310 were for technical professionals. There were also 85 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The number of computer scientists at NIST has increased by 143 percent since 1980. Even if all of these new people were in ITL (which they are not), this would still be an extremely modest increase considering the explosion of the IT industry in the last 20 years. While the panel would not argue that the staff of ITL must increase exactly in proportion to the increase in the U.S. IT industry, it does appear that the ITL is seriously understaffed in relation to the job it has to do.

Owing to slight declines in STRS funding and increases in personnel costs (e.g., salary and benefits), ITL can now support fewer full-time permanent staff members. Several divisions (especially

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

TABLE 8.1 Sources of Funding for the Information Technology Laboratory (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

31.2

31.6

31.6

30.4

Competence

0.7

0.9

1.5

1.5

STRS-supercomputing

12.3

11.8

12.1

12.0

ATP

2.1

1.8

1.8

1.2

MEPa

0.1

0.0

0.0

0.0

Measurement Services (SRM production)

0.0

0.0

0.0

0.2

OA/NFG/CRADA

9.9

10.6

8.4

8.7

Other Reimbursable

1.9

1.5

0.5

1.5

Agency Overhead

11.0

12.0

14.4

16.1

Total

69.2

70.2

70.3

71.6

Full-time permanent staff (total)b

420

362

381

381

NOTE: Funding for the NIST Measurement and Standards Laboratories comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST's congressional appropriations but is allocated by the NIST director's office in multiyear grants for projects that advance NIST's capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST's ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Funding to support production of Standard Reference Materials (SRMs) is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding, including that for Calibration Services, is grouped under “Other Reimbursable.”

aManufacturing Extension Partnership.

b The number of full-time permanent staff is as of January of that fiscal year.

Computer Security and Statistical Engineering) have lost key people in recent years, so while the present collection of employees is technically strong and well qualified, maintaining staffing levels and hiring new people are serious issues for ITL and division management. Personnel in the laboratory are already stretched very thin, with many single-person projects and, in some areas, what appears to be overreliance on temporary guest researchers. ITL has in general been successful at recruiting new personnel when the budget allows, and the laboratory also has an impressive retention rate among existing staff. (Turnover in ITL is much lower than turnover at many of the panel members' companies.) However, the demand for skilled people in the IT industry is extremely great, and NIST should be vigilant about tracking the effectiveness of its recruiting and retention efforts. ITL should consider developing a contingency plan to be implemented if the current success begins to wane or if the opportunity or need suddenly arises to hire a significant number of new staff. It has been suggested that agencies such as the Defense Advanced Research Projects Agency (DARPA) have found mechanisms that allow them to offer somewhat higher salaries and that NIST might look into employing similar methods.

The types and uses of information technologies are exploding, and new developments occur at “Internet speed.” The rapid changes necessitate continually increasing the number of standards at a continually increasing pace. Given the current staffing levels in ITL, the panel is concerned that there

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

may not be enough personnel in place to cover basic standards needs. Owing to its limited resources, ITL is not participating in some arenas where its expertise would be valuable and its work consistent with the NIST mission. Examples include interdomain routing standards, multicasting, Internet measurements (performance and reliability), hardware and software design, and related intellectual property modeling and simulation standards. Work with international standards bodies could also be increased. The panel is not claiming that programs in these areas would be of higher priority than ITL's current programs, but rather that work on them would also be of significant benefit to industry if ITL had the resources to pursue them. In summary, the panel believes that ITL is already underfunded, and it appears that the laboratory is losing, rather than gaining, staff positions. The panel believes that it might be useful for ITL to attempt to compile a list of important standards that are not being addressed by ITL due to understaffing and to quantify the potential loss to U.S. industry because NIST is not participating in these standards.

The panel also evaluated the state of NIST's data communications network. ITL includes staff who are consumers of the data communications services as well as staff who provide these services to all of NIST. The panel was pleased to learn that ITL expects to have wired NIST's entire internal network for switched high-speed (100 megabits per second (Mbps)) Ethernet by the end of 2002. The only issue on this front is that the expected completion date was recently pushed back from 2001, and the panel believes that a significant effort should be made to ensure that the installation is truly complete in 2002. The panel was more concerned to find that NIST's connectivity to the outside world is rather poor. NIST is using a 12-Mbps outside line at a time when the top 170 research universities in the United States are joining Internet2 using links of 155 Mbps or faster. (The Advanced Network Technologies Division does have its own 155-Mbps link to a D.C.-area testbed, but this link is not available to other divisions for a variety of technical and budgetary reasons.) NIST should move aggressively to improve the speed of its connectivity to the outside world. In the panel 's experience, much of the benefit of higher-speed connectivity, such as facilitation of distance collaborations and information sharing, cannot be accurately predicted or appreciated until the new capability is actually installed, but NIST still must recognize the need to keep pace with comparable institutions to ensure the continued high quality of its work.

In general, morale in ITL seems to be quite high. Panel members met with bench-level employees in “skip-level” interviews (interviews without laboratory-level management present). The panel is pleased to note that ITL staff are almost uniformly enthusiastic about their work at NIST. They especially valued the intellectual freedom available at NIST and the interactions with their NIST and ITL colleagues. However, in some divisions, staff felt that the tendency to have one-person projects and the reliance on guest researchers often limited the opportunities for formal project collaboration with peers at NIST. Staff members are in general satisfied with the laboratory facilities and office space and feel that they have access to the equipment necessary for doing their jobs. Concerns about the isolation of some divisions in NIST North continue and are discussed in detail in the divisional-level reviews. As was noted last year, there is some frustration with the Department of Commerce bureaucracy. The procurement process has been singled out by the staff for particular criticism, especially the $2500 limit on streamlined purchases. When the cost of a purchase exceeds this amount, a more complex process is used that causes delays. As an example, some employees asserted that the cycle for procuring computer equipment was so long that the equipment could be obsolete by the time it arrived. Difficulties with Department of Commerce legal services also continue. Three years after the Software Diagnostics and Conformance Testing Division's original request for approval of its membership in the World Wide Web Consortium, the request was denied by the legal department, despite the fact that other government agencies are official members of this consortium.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

DIVISIONAL REVIEWS

Mathematical and Computational Sciences Division
Division Mission

According to division documentation, the mission of the Mathematical and Computational Sciences Division is to provide technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to U.S. industry. The division focuses on the development and analysis of theoretical descriptions of phenomena (mathematical modeling), the design and analysis of requisite computational methods and experiments, the transformation of these methods into efficient numerical algorithms for high performance computers, the implementation of these methods in high-quality mathematical software, and the distribution of the software to NIST and industry partners.

The programs of the Mathematical and Computational Sciences Division conform very well to the division mission and to the ITL and NIST missions. Current programs provide mathematical modeling and analysis advice and tools for other NIST laboratories and for U.S. industry, and the division is pursuing activities to improve measurement methods and standards for mathematical work and software.

The panel has seen notable improvement in the overall planning of programs and projects in the division. The Mathematical and Computational Sciences Division strategic plan put together this past year gives a blueprint for building on current activities and developing new programs to address current and future needs. The strategic plan has three thrusts. The thrust on measurement and calibration for the virtual sciences could have a significant industrial impact; it holds great challenges, but the division is well positioned to address them. The thrust on evolving architecture of tools, libraries, and information systems for computational science and engineering will build on many of the division's highest-impact capabilities and activities. The thrust on emerging needs of the NIST laboratories is a vital direction for the division and for NIST. However, to succeed in the last effort, the division will have to cultivate new technical expertise to address the growing need for analysis and modeling in such areas as complex networks, large data sets, and interacting computer systems. Adding personnel will be necessary to extend the division's capabilities. The division should also continue efforts to expand technical interactions with personnel from other ITL divisions.

To fulfill the mission of the division, a healthy balance must be maintained between the work that supports specialized areas within NIST and the more application-independent methods with a broader external audience. The current portfolio of division projects contains an appropriate mix of these two types of programs. A potential challenge to sustaining this balance arises from the constant or decreasing number of staff in the Mathematical and Computational Sciences Division, coupled with the long-term nature of the collaborations with personnel in the other NIST laboratories. These enduring, multiyear relationships are fruitful, and the work produced helps NIST meet its goals. However, because there are so many long-standing demands on the division staff 's time, it is difficult to expand the number of projects or to form new collaborations with unexplored portions of NIST.

Technical Merit and Appropriateness of Work

The overall quality of the programs in the Mathematical and Computational Sciences Division is very high. The merit of the division's work is illustrated by the continuing recognition accorded to various projects, such as the object-oriented finite-element (OOF) analysis for the modeling of real material microstructures, which is one of the winners of Industry Week Magazine's 1999 Technology of the Year award. The OOF system is a general software tool that is used by a number of large companies,

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

and the project has attracted external support from the Department of Energy and includes collaborative work with a major research university, the Massachusetts Institute of Technology.

The Mathematical and Computational Sciences Division consists of three groups. Most of the projects in the Mathematical Modeling Group are done in collaboration with researchers in other NIST laboratories. The emphasis is on domain-specific problems, and the results are of interest mainly to scientists involved in these particular areas. The Optimization and Computational Geometry Group also tends to work on modeling and analyses that are specific to particular applications. By contrast, the work of the Mathematical Software Group focuses on the development of algorithms and software in support of general computational science; thus, the results of this work are typically useful to a broad spectrum of scientists both inside and outside NIST.

Many Mathematical and Computational Sciences Division programs support the NIST mission by providing the mathematical modeling and analysis needed for the success of important projects in other NIST laboratories. For example, the work on OOF was developed as part of the Materials Science and Engineering Laboratory's project on material microstructures. Also, work on modeling high-speed milling has been done with the Physics and Manufacturing Engineering Laboratories. In collaboration with staff from the Building and Fire Research, Physics, and Manufacturing Engineering Laboratories, analysis is being provided to support a project on computer graphic rendering of material surfaces. This project might also be relevant to graphics work in the entertainment industry, which is moving increasingly toward physics-based rendering.

Other Mathematical and Computational Sciences Division programs align directly with the measurement and standards mission of the ITL. In the Java Numerics project, division personnel have been leaders in establishing arithmetic standards for the Java programming language so that this language can be used in scientific computing applications requiring both run-time efficiency and numerical accuracy. A project focused on the development of a code for the numerical solution of partial differential equations also developed F90GL, a Fortran binding for the OpenGL three-dimensional multivendor graphics user interface standard. Key divisional personnel are leading the standardization effort for basic linear algebra subprograms for sparse matrices (sparse BLAS), which are computational units widely used in the scientific computation community. The output of the most ambitious current project in the division, the Digital Library of Mathematical Functions (DLMF), will be a Web-based update of the Handbook of Mathematical Functions,3 one of the most heavily used references from NIST's predecessor, the National Bureau of Standards.

Still other Mathematical and Computational Sciences Division programs support the division mission by advancing the methodologies and tools available for solving scientific problems of interest to U.S. industry. The division's Guide to Available Mathematical Software (GAMS) and its Matrix Market projects provide software references and test problems used extensively in the broad scientific computing community. The division's work in parallel adaptive multigrid methods and software to solve partial differential equations has been widely disseminated; this algorithm, pioneered in NIST's PHAML (Parallel Hierarchical Adaptive Multilevel project), is now being incorporated into PLTMG (Piecewise Linear Triangle Multigrid), one of the leading partial differential equations programs in use today. During the past few years, the JazzNet project demonstrated how a “personal supercomputer ” could be fashioned very inexpensively by harnessing clusters of personal computers; large commercial vendors now are beginning to offer products that use this approach.

3  

M. Abramowitz and I.A. Stegun, eds., Handbook of Mathematical Functions, with Formulas, Graphs, and Mathematical Tables, National Bureau of Standards, Gaithersburg, Md., 1964.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Impact of Programs

The overall impact of the programs in the Mathematical and Computational Sciences Division is significant and growing. The division correctly views its customers as falling into four categories: other ITL divisions, other NIST laboratories, the IT industry, and the computational science community. Examples of the division's positive impact on the latter three groups are noted in the above discussion of technical merit. The division is clearly continuing its strong support of and collaboration with scientists in other NIST laboratories, including teaching courses for NIST personnel on such topics as Fortran 90 and nonnumerical computational methods. In contrast, relatively few Mathematical and Computational Sciences Division activities are tied to those of other ITL divisions. However, the panel is pleased to note that forums for technical interchanges with these other divisions have been initiated and expects that the meetings will lead, over time, to further joint activity.

Many of the division's projects have had a clear and important impact on the information technology industry (largely but not entirely through the computational science community, an active element in that industry). For example, in the Java Numerics project, the division's technical expertise and NIST's role as a neutral broker have had a significant impact on an important industry standard. In 1999, Sun Microsystems enthusiastically embraced the project's first set of recommendations, with an effect on literally every user of Java. This project is a good example of NIST proactively discerning a need (rather than simply responding to industry requests) and having great impact on a core industry issue. Another example of industrial impact is the OOF project mentioned above: it has been widely used in industrial research and development settings, and a new collaboration with General Electric has secured DOE funding to support the adaptation of OOF for use in a manufacturing setting at General Electric. The panel also notes that the DLMF could have a significant impact. The NBS handbook on which the DLMF is based is still cited frequently today, and the number of citations is rising. The online version of this information will be very valuable to users in industry and academia, and new National Science Foundation (NSF) support for the project will move the work significantly closer to completion and to realizing the impact throughout the scientific community.

The Mathematical and Computational Sciences Division disseminates its results through several mechanisms, including the Internet, for which it has a long history of pioneering and effective use. This tradition continues with the GAMS (which had about 7 million hits in 1999; about 5000 external pages now contain links to the GAMS Web site) and the Matrix Market (which disseminates standard test data for scientific computation and has more than 100 users per day). Division staff also publish results in scientific journals and speak at conferences. Many results are disseminated by making software publicly available; examples in 1999 include two releases from the Java Numerics project—composite Java benchmarks packaged in SciMark and the Java Matrix Package (JamPack) for numerical linear algebra —and the object-oriented micromagnetic framework (OOMMF) modeling software.

Individual staff members contribute to the U.S. scientific community not only through their work on division projects but also by serving on important committees. Three staff members serve as editors of major professional journals, and several division staff are on working committees dealing with modeling and computational standards issues. The Association for Computing Machinery (ACM) recently awarded its Outstanding Contribution Award to the division chief for his innovative efforts in electronic dissemination during his tenure on the ACM Publications Board and as editor-in-chief of ACM Transactions on Mathematical Software. This award confirms the professional stature of the division leadership and reflects the division's long tradition of significant activities in areas central to NIST's goal of promoting dissemination of standards and technology through a variety of mechanisms, including

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

electronic publication, digital libraries, and standard repositories of mathematical software and reference data.

Division Resources

Funding sources for the Mathematical and Computational Sciences Division are shown in Table 8.2. As of January 2000, staffing for the Mathematical and Computational Sciences Division included 27 full-time permanent positions, of which 24 were for technical professionals. There were also 14 non-permanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

Staff members are of high technical caliber, are generally well connected professionally, have an appreciation for work being done elsewhere, and effectively strive to extend the state of the art in their areas of expertise. The value of the staff's work is confirmed by their solid record of papers appearing in external, peer-reviewed journals, as well as in other technical publications and as presentations at conferences. The practice of retaining leading university research faculty as part-time members of the staff and of having postdoctoral fellows and summer students also helps ensure that divisional staff keep abreast of important research at other locations. In the prominent DLMF project, the external authors and editors are world experts in their fields. The contributions from this board of outside experts will provide essential technical information, and their involvement ensures that the library will maintain a very high profile in the scientific computing community. In 1999, the division secured funding from the NSF to support the DLMF. This fulfills a recommendation made in last year's report; moreover, the project leaders are exploring innovative ways to provide ongoing support for the DLMF once it is complete and in use. As mentioned last year, plans for the DLMF must be continually reviewed in light of the speed of change of the digital publication technology that underlies the DLMF design.

As mentioned earlier in this report, the panel is concerned about the limited number of full-time, permanent staff currently employed in the division. Ongoing loss of personnel by attrition places the division's entire program at risk. The division's distinguished reputation within NIST, its positive staff morale, and its high stature and visibility in the external community all imply that it would be able to

TABLE 8.2 Sources of Funding for the Mathematical and Computational Sciences Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

4.1

4.3

3.3

3.5

Competence

0.1

0.1

0.2

0.1

STRS-supercomputing

1.0

0.6

0.7

0.6

ATP

0.2

0.2

0.1

0.1

OA/NFG/CRADA

0.2

0.3

0.4

0.8

Other Reimbursable

0.0

0.0

0.0

0.1

Total

5.6

5.5

4.7

5.2

Full-time permanent staff (total)a

51

30

30

27

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

attract world-class personnel if it had the resources to add more staff. (The division reported that in recent years, the first choice for each open position has accepted NIST's offer.) Of particular concern is the size of the Optimization and Computational Geometry Group. This group has been small for some time, and with the loss of two optimization people in the past 2 years, the remaining three people do not provide critical mass. Yet the research encompassed by this group's mission is essential for the Mathematical and Computational Sciences Division and for NIST. In the past three assessments, the panel stressed the need for expanded work in discrete mathematics. The goal of meeting the emerging needs of the NIST laboratories, as described in the division's strategic plan, cannot be achieved without adding staff in the domains covered by the Optimization and Computational Geometry Group. Last year, the panel suggested seeking postdoctoral fellows as a way to strengthen this group. However, given the recent shifts in the job market and the continued attrition in the Optimization and Computational Geometry Group, this strategy would no longer be feasible or indeed sufficient to meet the challenges the division is facing. A top priority should be to bring in a broadly respected, energetic, capable leader for this group. This person should be given the authority and the resources to develop plans to support divisional strategic goals and to hire several other people capable of helping to reach these goals.

Facilities and equipment are for the most part adequate, although the location of the Mathematical and Computational Sciences Division in NIST North, separate from all of the other NIST laboratories and half of ITL, is a continuing barrier to more effective communication with staff in other parts of NIST. There is a general impression of increasing strain on the infrastructure; for example, the loss of NIST's central UseNet service hinders the ability to participate in technical communities and eliminates what was once a useful channel for division personnel to respond to user questions about NIST software and resources.

Advanced Network Technologies Division
Division Mission

According to division documentation, the mission of the Advanced Network Technologies Division is to provide the networking industry with the best in test and measurement technology.

This mission statement is appropriate and accurately reflects the NIST and laboratory missions in the context of technologies relevant to the division's work. This year, the programs under way in the division are integrated much more effectively than the group of projects presented to the panel in 1999, and the individual projects are better focused both on specific goals and on contributing to NIST's role in measurement and testing.

Technical Merit and Appropriateness of Work

The Advanced Network Technologies Division currently emphasizes six areas of technology: wireless communications, pervasive network computing, internetworking technologies, optical networking, multimedia networking, and programmable networking. The overall quality of the projects is quite high; specific outstanding examples are discussed below. The division is playing an active leadership role, especially in response to industry requests. A key element in designing effective standards is defining the appropriate problem space, and the Internet Engineering Task Force (IETF) and other standards bodies are not always successful in this necessary step. The division's work in multimedia, wireless, and internetworking standards during the past year has been valuable to the community.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

The division's wireless communications activities are focused on four areas: third-generation (3G) wireless, mobile ad hoc networks (MANETs), local multipoint distribution service (LMDS), and the emerging standards for local wireless networks. The 3G modeling effort is very promising and should be continued, but the most vital and exciting work is on local wireless standards. All three nascent standards (802.11, Bluetooth, and HomeRF) make use of the Federal Communications Commission (FCC) unlicensed band for transmission, and as things stand now, the three standards will interfere with each other, a tremendous problem because networks utilizing them for communication are all intended for use in the same environments (home and workplace). Never before in the commercial standards arena have so many competing technologies tried to share the same wireless band. In this situation, leaving the problem to be solved by free-market competition could be disastrous because the competing standards will interfere with each other in ways that will confuse the consumer. NIST's contributions in this area, especially over the next year, could be invaluable. This is a multibillion dollar industry problem and NIST is uniquely positioned and qualified to save the industry from itself by working with it to harmonize the standards so that networks conforming to the different standards can at least coexist. NIST must aggressively support the division 's activities in this area.

In this division's contribution to the ITL-wide pervasive computing initiative, staff have built a small testbed to study the different proposed protocols (such as Jini and T-Spaces) for locating resources in a picocell wireless local area network (LAN). NIST staff have gotten ahead of industrial laboratories in this area and are positioned to play a leadership role in this emerging field. Industry will benefit greatly from the dissemination of this group's experiences and from its continued guidance. Although the problems in this area are not nearly as critical as the difficulties with the wireless standards, it is another area to which increased NIST resources could be very effectively applied.

The Internetworking Technologies Group has effectively integrated its quality of service research into a coherent package focused on voice-over Internetwork Protocol (IP) and its requirements. The voice-over IP community is expanding the base of users of the NIST network emulator, and the panel applauds the division's efforts to transition the emulator to a commercial entity. The NIST switch reference code for multiprotocol label switching has also turned out to be popular and is likely to be useful to the community.

In optical networking, the wavelength division multiplexing (WDM) effort has focused on modeling optical networks, resulting in the development of the modeling evaluation and research of lightwave networks (MERLiN) tool, whose primary uses are for WDM network planning and WDM protocol performance analysis. Integrating NIST's MERLiN with the Optical Communication Systems Simulator (OptSim) and Virtual InterNetwork Testbed is a creative approach. NIST has made MERLiN publicly available, and there has already been a discernible benefit to industry. The number of staff working in this area is small, but the effort now has an effective direction and should be encouraged.

In the multimedia networking area, the project on image indexing is very promising.

Impact of Programs

The Advanced Network Technologies Division continues to be active in a variety of industry organizations, including the IETF, the Asynchronous Transfer Mode (ATM) Forum, the Cross-Industry Working Team, and the International Telecommunication Union. Division personnel are well respected by the staff of these standards bodies and by the communities they serve. The potential impact of the division's work over the next year is very high. As noted above, the local wireless networking community desperately needs unbiased quantitative guidance to address problems of wireless interference between standards.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

In addition to their active participation on standards committees, the staff utilize a variety of other mechanisms to disseminate NIST results to the networking community. Examples include the 14 referenced publications produced by the Multimedia Networking Group, the bake-off on Internet key exchange interoperability, held in San Diego in January 2000, and attendance and presentations at meetings such as the Seventh International Conference on Optical Communications and Networks (January 2000), the DARPA Next Generation Internet Principal Investigators ' Meeting (December 1999), the Seventh Institute of Electrical and Electronics Engineers (IEEE) Workshop on Future Trends of Distributed Computing Systems (December 1999), and the Sixteenth International Conference on Communications (June 1999).

Division Resources

Funding sources for the Advanced Network Technologies Division are shown in Table 8.3. The significant reduction in internal (STRS) funding in the past year has forced division personnel to scramble to secure support from external sources. This effort distracts staff from their technical projects and affects the division's ability to perform work on “Internet timescales,” where results are needed quickly in order to impact rapidly evolving technologies. In this division, the work on wireless standards and on pervasive computing are both subject to the pressures of Internet time.

As of January 2000, staffing for the Advanced Network Technologies Division included 27 full-time permanent positions, of which 22 were for technical professionals. There were also eight nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The Advanced Network Technologies Division relies heavily on a significant number of guest researchers. This dependence on visitors for support of mission-critical projects is risky, but for the time being the benefits of this approach—expansion of the workforce at NIST and injection of new ideas and information about work at other institutions —are outweighing the disadvantages (the potential for unexpected delays or even terminations of projects when the guest researchers leave NIST).

Despite the issues related to decreases in funding, the panel was pleased to note that the enthusiasm and morale among staff in the division were higher than observed during the previous assessment. One area of particular concern to the panel was in the Internetworking Technologies Group, where just as the Internetwork Protocol Security (IPsec) technology is being deployed, both of the division's security

TABLE 8.3 Sources of Funding for the Advanced Network Technologies Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

3.8

4.1

4.9

3.7

Competence

0.0

0.0

0.2

0.3

ATP

1.1

0.3

0.3

0.2

OA/NFG/CRADA

0.7

1.5

1.2

2.1

Total

5.6

5.9

6.6

6.3

Full-time permanent staff (total)a

32

26

30

27

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

experts are leaving for other assignments within NIST. The division should aggressively search for qualified replacements with skills in this important area.

Computer Security Division
Division Mission

According to division documentation, the mission of the Computer Security Division is to improve information systems security by raising awareness of IT risks, vulnerabilities, and protection requirements, particularly for new and emerging technologies; researching, studying, and advising agencies of IT vulnerabilities and devising techniques for the cost-effective security and privacy of sensitive federal systems; developing standards, metrics, tests, and validation programs to promote, measure, and validate security in systems and services, to educate consumers, and to establish minimum security requirements for federal systems; and developing guidance to increase secure IT planning, implementation, management, and operation.

The division's programs are all in direct support of its articulated mission and are consistent with the overall missions of the Information Technology Laboratory and of NIST. IT work is essential to protect electronic commerce and critical infrastructure, which are key elements for strengthening the U.S. economy and promoting public welfare.

Technical Merit and Appropriateness of Work

The panel found a consistently high level of technical merit and appropriateness in the programs undertaken by the Computer Security Division. The process of evaluating candidates for the Advanced Encryption Standard (AES) continues to progress efficiently; five finalists were announced in August 1999. This activity is enhancing NIST's reputation and is establishing its primacy in encryption on an international level. In addition, the division's work in examining public key infrastructure (PKI) interoperability, cross-certification, and policy, including the Bridge Certification Authority effort, is relevant and important.

The division addresses its responsibility to improve the security of products used by both government and industry through its National Information Assurance Partnership as well as through ITL bulletins and other published guidelines.

Processes are in place for reviewing and terminating projects, as happened with the conclusion of work on RABC and the drafting of the Common Criteria. The interpretation of health care security requirements, including the Health Insurance Portability and Accountability Act of 1996, in terms of a Common Criteria protection profile provides a useful service to an industry that is struggling to define and address security issues. In general, the division is very careful to focus its efforts on emerging areas in which it can make a unique contribution. The new project on the security of mobile agents is one such area, in which work done at NIST will have significant impact on the evolving use of the Internet and intelligent networks.

Impact of Programs

The Computer Security Division's programs have significant impact across a wide range of constituencies. In the area of standards, the results of the AES evaluation process will affect both government and industry for years to come. A doubling of the number of validated cryptographic modules and the publishing for comment of a draft of Federal Information Processing Standard 140-2 together

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

directly benefit the United States by providing trustworthy cryptography for industry and government use. The frequent use of the IPSec interoperability testbed and the central role of division staff in the smart card consortium are clear indications of the value of NIST's efforts to industry.

In collaborations with companies in which there are intellectual property issues, Cooperative Research and Development Agreements (CRADAs) are an effective means to facilitate interactions with industry. An example is the CRADA with Boeing for the work on mobile agent security. In general, however, CRADAs are complicated to put in place and can slow progress on a particular activity; use of CRADAs has therefore declined in situations where they are not absolutely necessary. The trend is toward more participation in open consortia and at invited workshops. There are 45 instances of division staff serving on relevant committees, and this participation includes filling key roles such as editorship of the IETF PKI Working Group Requests for Comments.

The drafting of the Common Criteria has been completed, and 20 countries are expected to have Mutual Recognition Agreements (MRAs) by the end of this year. The size of this group clearly indicates that the division's work in this area will have an ongoing effect on the international community.

Division Resources

Funding sources for the Computer Security Division are shown in Table 8.4. As of January 2000, staffing for the Computer Security Division included 43 full-time permanent positions, of which 38 were for technical professionals. There were also five nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

In its 1999 report, the panel raised concerns over the combined trends of decreasing budget and increasing staff, as well as the significant reliance on other agency (OA) funding. In the past year the staff has been reduced by about 10 percent and funding has remained almost constant. The division expects an additional $600,000 of external funding, and if this support does materialize, the division will have adequate resources for fiscal year 2000. Although there is still a significant amount of OA funding in the division's budget, the percentage is decreasing. Continuing this trend will give the division greater control over the scope and direction of its projects and increase the degree to which its staff can support the missions of the division, the ITL, and NIST.

TABLE 8.4 Sources of Funding for the Computer Security Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

6.8

6.3

5.9

6.0

ATP

0.0

0.0

0.2

0.1

OA/NFG/CRADA

4.5

3.7

2.3

1.4

Other Reimbursable

0.0

0.0

0.1

0.0

Total

11.3

10.0

8.5

7.5

Full-time permanent staff (total)a

40

44

48

43

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

It is too early to assess the impact that funding from the President 's proposed computer security initiative would have on NIST's programs in this area, including the work on critical infrastructure protection (CIP). However, the panel was presented with a significant number of activities scheduled to be undertaken by the Expert Review Team proposed in this initiative. The work relates to fixing existing vulnerabilities in federal computer systems, and the proposed funding 4 for this group was definitely inadequate to accomplish the stated list of goals. In fact, in all three sections of the new initiative, the ITL must be careful to clearly define goals and limit the scope of the projects to ensure that there is not a mismatch between resources and objectives. In addition, NIST should not underestimate the difficulty in the current market of finding and hiring computer security experts in a timely fashion to staff the new programs.

Computer Security Division staff routinely participate in cross-divisional efforts and provide support for internal NIST activities. For example, the IPSec interoperability testbed and reference implementation work has been done in conjunction with the Advanced Network Technologies Division, and the Computer Security Division has also provided technical support for the use of digital signatures for internal NIST work flow.

The organization of the Computer Security Division into two groups, each with three sections, facilitates its work. The panel was somewhat concerned when it learned that at the time of its site visit, both the division and the Security Technology Group had acting directors. Having people without “acting” in their title in these positions would eliminate an unnecessary distraction for the division.

Information Access and User Interfaces Division
Division Mission

According to division documentation, the mission of the Information Access and User Interfaces Division is to accelerate the development of technologies that allow intuitive, efficient access, manipulation, and exchange of complex information by facilitating the creation of measurement methods and standards.

This mission statement reflects the heterogeneous nature of the division, whose charter encompasses speech, language, vision, and Web technologies, including their support, integration, and measurement and the establishment of common standards. This mission is consistent with NIST's overall mission of providing standards, measurements, and infrastructure and ensuring their availability.

Technical Merit and Appropriateness of Work

The Information Access and User Interfaces Division consists of four groups: Spoken Natural Language Processing, Natural Language Processing and Information Retrieval, Visual Image Processing, and Visualization and Virtual Reality.

The Spoken Language Processing Group works with DARPA and the spoken language research community in industry and at universities to develop standardized speech databases and metrics for evaluating state-of-the-art speech recognition systems and to coordinate benchmark tests using the standard databases. During the past decade, these tests served as an indispensable tool for measuring the progress of automatic speech recognition and contributed to a steady improvement in the performance

4  

The proposed support for the Expert Review Team would be a one-time allocation of $3 million, plus continuing funding at $2 million per year.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

of these systems. The group has also developed measures that provide important diagnostic information on the performance of speech recognition systems.

The NIST objective in the speech recognition projects is to guide the research community toward achievable improvements and, at the same time, to encourage significant breakthroughs—a difficult task. In some of the projects, the balance between incremental advances and big developments has tilted toward the former. Although this tilt is not in itself reason to cease work in a specific area, it is a signal to examine the projects in which NIST's work is still enabling the community to progress in giant steps, with an eye toward determining what factors contributed to the giant steps. Perhaps lessons from the continuing success of these activities can be used to jump-start NIST programs in areas in which industry seems to have plateaued.

The Natural Language Processing and Information Retrieval Group works with industry, academia, and other government agencies to promote the use of more effective and efficient techniques for manipulating (largely) unstructured textual information, including methods for browsing, searching, and presenting the information. NIST has over the past decade established internationally recognized benchmarks for evaluating information retrieval (IR) systems, exercised in an annual Text Retrieval Conference. TREC is composed of multiple tracks for evaluating query-based retrieval, cross-lingual retrieval, filtering, distributed and large-scale retrieval, and, most recently, question answering as focused passage retrieval, as embodied in the TREC-8 evaluation suite. NIST also provides benchmark collections, query suites, relevance judgments, and other crucial support resources to the industry and academic researchers in IR. The evaluation metrics (including 11-point precision curves) are disseminated in NIST software and have become widely accepted, driving focused progress in research. TREC provides substantial assistance to industry, including Web-based companies, by permitting neutral comparative analysis of technologies and providing quantitative measures of progress on developing technologies. TREC's success in providing measurement methods that drive both research on and commercialization of new information technologies is an impressive example of how NIST fulfills its mission. The panel strongly endorses continuation of this effort and suggests that TREC may provide a model for other NIST activities beyond its current application to IR technologies.

NIST is already moving to evaluate other language technologies beyond IR, in close connection with the DARPA-funded Translingual Information Detection, Extraction, and Summarization (TIDES) initiative and associated activities in academia and industry. TIDES supports translingual document retrieval, multidocument text summarization, and related technologies such as machine translation for information management and information visualization. NIST's objective is to provide standard, widely applicable (and hopefully widely accepted) evaluation metrics and evolving test suites for these technologies. TIDES is a major extension of TREC, as the group moves beyond IR into related and important technologies that are inherently more difficult to measure, evaluate, and compare objectively. The new TIDES initiative is not without risk, and its success or failure will become evident over the next 1 to 2 years. However, it is NIST's mission to develop objective metrics for emerging high-impact technologies, and this project is aimed at that exact goal for a set of information technologies that are extremely pertinent for the Internet economy. Due to the rapid evolution of such technologies, NIST needs to enter the field early in order to have a strong positive impact in both academic and commercial efforts. A wait-and-see attitude would marginalize the NIST staff, and without the necessary measurement technologies, the U.S. information industry would be hindered in its efforts to maintain international leadership in information technology. Therefore, the risks associated with NIST's entry into this area are justifiable, although the dangers must be explicitly acknowledged. Fortunately, the approaches that led to successes in TREC are expected to serve as guiding principles for the greater and more diverse challenges of TIDES. The panel recommends that the TIDES evaluation project be targeted

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

more at general text-processing technologies and less at specific DARPA tasks, if this is possible in a DARPA-funded project.

The Visual Image Processing Group works to support the technology of image recognition in government and industry by developing new image recognition methods, developing techniques for the evaluation of existing methods, and providing technology transfer to the commercial imaging and document conversion industry. This group is working on projects primarily funded by internal government organizations, so there is less emphasis on encouraging commercial markets. It focuses specifically on face recognition and fingerprint standards.

In the face recognition area, the group has begun a new effort to design test and database parameters. This effort builds on a previous NIST effort in this area that ended in 1996. This project is timely and will be welcomed by both producers and users of face recognition technology. As they accumulate data, the companies that make face recognition products should be invited to submit the data and provide input on testing procedures. Another new project, human identification at a distance, is in a very early phase; staff are assessing various possibilities, including face, gait, and iris recognition, to determine the scope of the project. Because results from work in this area will most likely be relevant mainly to government agencies (e.g., for identifying terrorists at airports) rather than to industry, the work is considered as part of the research component of the division 's portfolio.

In the fingerprint standards area, the Visual Image Processing Group is finalizing the American National Standards Institute (ANSI)/NIST-ITL-1-2000 fingerprint minutia format standard for law enforcement; this activity is sponsored in part by the division's continuing funding from the Federal Bureau of Investigation (FBI). Also for use by law enforcement agencies is a new project on chain of custody. The goal is to establish a digital record to track and validate the handling of fingerprint data from initial point of capture through court appearance by using digital signatures. The panel thought it might also be appropriate to attach the presiding officer's fingerprint to the record. This project would be a natural opportunity for collaboration with the NIST cryptography group. Another new project, supported by government funding, is the latent fingerprint workstation, developed in collaboration with the FBI and Mitretek to process and match latent fingerprints. The panel encourages NIST to use commercial off-the-shelf processing algorithms and matchers rather than spending time on researching and developing in-house engines. The NIST focus should be on integration, measurement, and usability analysis.

The Visualization and Virtual Reality Group conducts research in visualization and virtual environment technologies in order to demonstrate the utility and feasibility of visual displays of information to industry and government. The group works to ensure that technology advances in three-dimensional information visualization and data exploration can be used effectively for the access, manipulation, and exchange of complex information. It achieves its mission by working with private industry, academia, and government to develop evaluation methodologies and reference sets; by conducting research in visualization to support the development of these test and evaluation methods; and by building visualization and virtual reality prototype systems for application areas such as manufacturing, information retrieval, and collaborative work.

Three-dimensional imaging and modeling is a relatively new endeavor on the World Wide Web, and despite the emerging importance of three-dimensional environments and interfaces, the community still lacks widely accepted standards. Potential applications include collaborative engineering and manufacturing, virtual world modeling and simulation for education, and three-dimensional presentations for sales and marketing. NIST serves the Web3D consortium as a recommender of and repository for emerging standards, and it is also starting in-house projects, such as the development of Java virtual environment device interface (JVEDI) software. These activities in three-dimensional modeling and for

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

the Web3D consortium are very important, but the panel believes that NIST's efforts should be more tightly focused on areas in which they can have maximal potential impact. Adjustments would include shifting work on standards and metrics from VRML to XML, the environment emerging as the de facto standard in industry; focusing on a single testbed application in manufacturing or a complex service environment (such as a hospital), perhaps based on a close collaboration with an external partner (small demonstrations are not sufficient, and narrowing the scope would allow the group to tackle meaningful problems and obtain clear recognition from external researchers); and seriously exploring projects with leveraged impact, such as Jini device integration.

The Smart Space Integration project is part of an interdivision effort on pervasive computing. The objective is to build a testbed for investigating the integration and communications of diverse computing components in an associated environment. This project has great potential for influencing a nascent field by contributing to application programming interface (API)-level interchange (e.g., swapping of speech recognizers and other components). Industry's work on smart spaces is evolving rapidly and includes a broad range of new and different technologies, so NIST's best chance at significant and timely impact would come from focusing on a singular objective. However, at least in the testbed built to date, the NIST demonstration includes many components developed in-house, some specifically for this project (e.g., head tracking algorithm, speaker recognition software, and body tracking algorithm). This customization seems to be contrary to the objective of integrating off-the-shelf components and testing their mix-and-match capabilities. Components appropriate for use in this study are available commercially or from researchers at universities, and NIST staff should take advantage of these sources rather than developing their own equipment, because the time spent developing components rather than investigating their integration could prevent NIST from seizing the opportunity to perform timely, groundbreaking research in the area of smart spaces (e.g., stress testing APIs and investigating system-level metrology). The division could also benefit from the expertise on convergent systems being developed in the High Performance Systems and Services Division.

Impact of Programs

Both industry and government have benefited greatly from the work done in the Information Access and User Interfaces Division. The databases and metrics developed by the Spoken Natural Language Processing Group have contributed to the creation and refinement of commercial speech recognition systems by several companies. The Natural Language Processing and Information Retrieval Group has fostered advances in information retrieval metrology, which have led to widespread technology improvements and commercial applications. TIDES, the new project on wider-range multilingual text processing technology, has the potential to have an equally significant impact. In the Visual Image Processing Group, the completion of ANSI/NIST-ITL-1-2000 and the delivery of the latent workstation to the FBI both had a positive effect on government capabilities in the fingerprint standards area. Future impact is possible from the new projects in face recognition testing, human identification at a distance, and chain of custody. To maximize the benefits from NIST work in these areas, the scope of the projects could be extended beyond government to industrial applications. In the Visualization and Virtual Reality Group, it is too early to determine the impact of the work on three-dimensional modeling and three-dimensional Web interfaces, but the potential impact is vast if the group is able to focus on one or two substantial initiatives (rather than four or five small ones) and can quickly foster the evolution and development first of standards, and later metrics.

The work in smart spaces is just over a year old. Although staff have begun to demonstrate some communication between components, the project has not yet reached testbed status, let alone achieved

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

TABLE 8.5 Sources of Funding for the Information Access and User Interfaces Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

4.8

4.6

4.5

4.5

ATP

0.0

0.4

0.2

0.0

MEP

0.1

0.0

0.0

0.0

OA/NFG/CRADA

2.0

2.3

3.2

2.1

Total

6.9

7.3

7.9

6.6

Full-time permanent staff (total)a

47

41

40

39

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

the goal of mixing and matching the components. Therefore, industry has not yet benefited from the division's work in this field. However, interest in the NIST activity can be gauged from the number of companies attending the NIST workshop on this topic and partnering with the division on the smart space project.

Division Resources

Funding sources for the Information Access and User Interfaces Division are shown in Table 8.5. As of January 2000, staffing for the Information Access and User Interfaces Division included 39 full-time permanent positions, of which 35 were for technical professionals. There were also 10 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The division has 7 research laboratory spaces, 48 Sun workstations, 13 Silicon Graphics workstations, 90 personal computers, image scanners, image and video capture systems, computer-controlled and charge-coupled device cameras, color display projectors, speech collection equipment, fingerprint collection equipment, and optical tables with lasers, lenses, beam splitters, and holographic storage systems. This collection of equipment and facilities seems adequate for the division to conduct its current array of projects.

High Performance Systems and Services Division
Division Mission

According to division documentation, the mission of the High Performance Systems and Services Division is to enable effective application of high-performance computing and communications systems in support of the U.S. information technology industry and NIST by conducting research, development, and evaluation of advanced hardware and software components, new architectures, novel application technologies, and innovative measurement and test methods for improved computing performance, scalability, functionality, interoperability, flexibility, reliability, and economy; serving as a testbed for research and development in high-performance computing and information technologies such as embedded computing, displays, and data storage, gaining experience in the deployment of these technolo

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

gies and developing metrics for the representative technologies; serving as a responsive, effective mission-critical resource spanning computational, communication, mass storage, security, archival, and scientific visualization services; and providing and managing state-of-the-art computing and networking facilities that integrate and support an enterprise-wide heterogeneous information technology environment for NIST.

The division mission includes research and service components. The service and support elements of the division's activities and mission conform to the NIST mission indirectly by enabling other NIST staff to fulfill that mission. NIST staff are also supported by the Distributed Computing and Information Services Division. As discussed in the laboratory overview, large-scale cooperative efforts are needed in the ITL to ensure that NIST has a system in place to provide world-class information systems support to its personnel. As part of these efforts, the missions of the High Performance Systems and Services and Distributed Computing and Information Services Divisions should be coordinated.

The High Performance Systems and Services Division is currently working to refine and focus the research component of its mission. The panel commends this effort and the proposed theme of “providing technical leadership in the area of convergent information systems.” This theme captures critical elements of the division's currently successful projects and its planned future research. While a concept such as convergence, although powerful, appears at first glance to be too vague and general to provide operational guidance, it seems clear to the panel that the division and its management have for the most part been able to locate the areas in which NIST activities can have a high impact in a timely manner and in which the limited resources available to the division can still be leveraged to produce high-quality results. Work on convergent information systems is very consistent with the laboratory mission and will allow the ITL to take its proper place as a leader in standards and technology with respect to information systems. As expertise is developed in convergent information systems, the division should coordinate its plans in this area with systems integration work being done by other ITL divisions, such as the laboratory-wide initiative on pervasive computing.

Technical Merit and Appropriateness of Work

Overall, the research and service work of the High Performance Systems and Services Division is excellent. Particular examples of the high quality of the division's efforts include the open standards for electronic books, the collaborative scientific computing efforts such as visualization of Bose-Einstein condensates, the baseline digital TV applications software environment (DASE) implementation, and work on testing methods for digital versatile discs (DVDs). Previous and continuing work on computing clusters, interoperable message passing interface (IMPI) specifications, and conformance testing are also exemplary. The publication of articles and papers and the visibility of the division's work within and outside NIST testify to the high quality of the activities under way. The panel continues to be impressed by the successful implementation of an appropriate planning process. The division has done a thoughtful job of systematically introducing, building, and terminating projects. The division is composed of five groups: Information Storage and Integrated Systems, Distributed Systems Technologies, Scientific Applications and Visualization, Network and Telecommunications Systems, and High Performance Systems Usage.

The research elements of the division's programs are progressing impressively. Plans and activities include work that incorporates standards, metrology, and links to industry consortia. In the Information Storage and Integrated Systems Group, the projects on a Braille reader and on DVDs are well grounded and timely. In the Distributed Systems Technologies Group, the new project on microelectronic commerce aims to diminish barriers to small-scale e-commerce; early efforts include work on interoperability

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

for authentication applications. A broad range of issues impact e-commerce, and the panel had some concerns that it may be difficult for the staff to focus the NIST efforts on a manageable set of goals within this range and that the level of human and fiscal resources scheduled for the work may make it difficult to build the expertise and gather the information needed for the start-up phase of the activity.

The service and support functions of the division continue to impress the panel. The Scientific Applications and Visualization Group provides collaborative support for scientists in other NIST laboratories. This year, staff successfully completed NIST's transition away from the Cray to the Silicon Graphics Origin parallel computer. Projects under way include work with researchers in the Electronics and Electrical Engineering, Chemical Science and Technology, Physics, Materials Science and Engineering, and Building and Fire Research Laboratories. In the Network and Telecommunications Systems Group, the ongoing improvements in the campus network are essential to maintaining the NIST campus as a productive research facility. The completion date for these improvements was pushed back from 2001 to 2002, and they should not be delayed any further.

The High Performance Systems Usage Group administers the central computing facility and supports the networking system on campus. Current issues for this group are maintaining and improving the firewall and upgrading the storage systems for scientific data. In the former area, the staff completed installation of the firewall for the NIST campus in 1999. Because this system, which began only as protection for NIST's supercomputing facilities, has expanded to enclose the entire laboratory, the management and policies of the firewall now need to have more input from the broader community of users and their IT support people. Mechanisms for such input should be discussed at the laboratory level. A possible approach to strengthen the firewall program would be to endow the High Performance Systems and Services and the Distributed Computing and Information Services Divisions with a solid responsibility for information security.

In the area of archival storage systems for NIST data, the division should focus on making maximum use of commercially available archival storage systems rather than being tempted to build its own. If existing technology can meet NIST's needs, there is no reason to absorb the high costs of developing and maintaining a custom system. However, the idea of leveraging the Information Storage and Integrated Systems Group's expertise and leadership in writable DVD storage has real merits if the efforts remain consistent with emerging standards and future commercially supportable systems.

Impact of Programs

A key element of the division's research efforts has been their impact on and interaction with industry. Commendable examples include the work of division personnel on international standards efforts, such as the leadership of the Open Electronic Book Forum and the participation of significant numbers of companies in division workshops, conferences, and related activities in the areas of electronic books, biometrics, DVD standards, and DASE.

In the Information Storage and Integrated Systems Group, the work on a rotating-wheel-based Braille display is part of an ongoing effort to improve accessibility to text produced by an electronic book, desktop computer, or personal digital assistant. This project is highly visible and could impact a large number of information technology users. Also in this group, the panel believes that participation in the U.S. Biometric Consortium and work on the biometric API interoperability testbed will allow NIST to make important contributions to this growing industry.

On the service and support side of the division's work, the whole of NIST continues to benefit from the efforts of the High Performance Systems and Services Division staff. Users were transitioned away from the increasingly obsolete technology of the Cray in a careful, thorough, and effective way. The

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

leader of the Scientific Applications and Visualization Group deserves credit for an excellent job of managing this major effort. Phone system upgrades and the network upgrade have also had a positive impact on NIST staff throughout the laboratories. However, network speeds to the outside are still slow compared with other national computational facilities and even compared with universities now on Internet 2. While staff report that there has not been much demand from NIST researchers for high-speed outside network access, the panel believes that pressure for this capability will arise very soon (driven by both internal needs and external expectations) and that NIST should move to provide the service in a timely way.

In the Scientific Applications and Visualization Group, work with the Physics Laboratory on the visualization of vortices in Bose-Einstein condensates is a high-impact, high-visibility project for NIST; the images produced were featured on the cover of the December 1999 issue of Physics Today. The group also developed the IMPI conformance tester, which is an industry resource.

Division Resources

Funding sources for the High Performance Systems and Services Division are shown in Table 8.6. As of January 2000, staffing for the High Performance Systems and Services Division included 81 full-time permanent positions, of which 59 were for technical professionals. There were also 21 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

During fiscal year 2000, the division experienced a management transition, but the panel notes that both the past and the current division chiefs have provided excellent leadership for the division. The High Performance Systems and Services Division is leading the ITL in its efforts to recruit, recognize, and promote individuals from underrepresented groups, and the group, project, and division management teams include women and minorities. Mechanisms for hiring U.S. postdoctoral research associates more flexibly and with less administrative overhead than allowed by the current National Research Council process might permit more effective recruitment of women and minorities.

TABLE 8.6 Sources of Funding for the High Performance Systems and Services Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

1.7

2.4

2.6

2.0

Competence

0.5

0.1

0.0

0.0

STRS-supercomputing

8.3

9.5

9.9

10.2

ATP

0.5

0.5

0.6

0.4

OA/NFG/CRADA

0.8

0.4

0.4

1.0

Other Reimbursable

0.9

0.7

0.0

1.1

Agency Overhead

4.9

5.3

6.7

8.1

Total

17.6

18.9

20.2

22.8

Full-time permanent staff (total)a

75

71

75

81

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

Staffing is very thin in some areas. The project on wavelength division multiplexing was canceled in part because critical staff left NIST. The use of high school and college students in the Information Storage and Integrated Systems Group is a model for outreach and education, but it is not a good omen for the level of personnel that key projects can afford. Key elements of the electronic book effort were supported last year by temporary additions to the budget, which has strained the funding allotted to this area. The panel believes that direct internal support of this activity is needed.

The Department of Commerce has suggested a department-wide pooling of computational resources. This effort would be laudable, if idle resources could be found. Within NIST, utilization of the infrastructure is very efficient, and there is no spare computational power or time. Therefore, the department's attempt to coordinate these resources may not bear fruit unless there is significant idle capacity of an appropriate type elsewhere in the department.

In general, the panel believes that the activities of this division, especially as linked by the theme of convergent information systems, are appropriate and vital for NIST's role in supporting the U.S. information technology industries. Unfortunately, there is a mismatch between the many potential implementations of projects in this area and the current level of resources devoted to the work. A substantial level of investment of personnel and technical resources in this area would be strategically sound. Additional resources would deepen the support for existing work, ensure timely participation and leadership in standards and metrology activities, and perhaps also allow expansion into new areas consistent with NIST's mission and valuable to U.S. industry. Examples include security for distributed high-performance computing systems, biomimetic computing, work on large-scale scientific data systems and archives, software for implementing a shared memory programming model on general (distributed memory) computers, and focused and deep efforts on electronic commerce.

Distributed Computing and Information Services Division
Division Mission

According to division documentation, the mission of the Distributed Computing and Information Services Division is to provide the information technology resources, supporting infrastructure, applied research, and assistance to NIST staff, collaborators, and clients for application in the conduct of scientific, engineering and administrative applications and in the dissemination of information.

The Distributed Computing and Information Services Division's mission directly supports the goals and objectives presented in the ITL strategic plan.5 Last year, the panel noted that this division appeared to have little involvement in or knowledge of the long-range planning activities described by ITL senior management. In contrast, this year's Distributed Computing and Information Services Division operational plans and long-range goals appear to have been integrated into the ITL-wide strategic planning process. The final plan appropriately emphasizes the importance of providing information technology support to NIST staff. The division provides software for staff workstations, comprehensive desktop support services, electronic information dissemination services, file servers, and administrative and management applications support. The strategic plan describes an intention to embark on a comprehensive computer security program, an increased focus on centrally managed information technology infrastructure resources, the integration and modernization of NIST administrative information systems,

5  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, ITL Strategic Plan January 2000, National Institute of Standards and Technology, Gaithersburg, Md., 2000.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

and increased desktop support activities. The panel commends ITL both on its strategic plan and on conducting an inclusive planning process.

The panel is somewhat concerned about what seems to be an artificial distinction between the services provided by the Distributed Computing and Information Services Division and those performed by the High Performance Systems and Services Division (e.g., support of desktop machines versus support of the networks that these machines are on). The panel suggests that reviewing and clearly defining the missions and responsibilities of each division might provide a more coherent and less confusing environment for end users of ITL's support services. If the support staff are to remain separate, then the service functions and missions of these two divisions should be better coordinated.

Technical Merit and Appropriateness of Work

The Distributed Computing and Information Services Division is involved in several activities to reduce ongoing costs and hence improve the effectiveness and efficiency of the overall NIST computing infrastructure. Services such as common centralized e-mail, shared calendars, centrally coordinated Web pages, help-desk support, file servers, and site-licensed software are common practices throughout the IT community, and the division does a good job of coordinating them with a low ratio of staff to customers. For example, the division successfully completed an array of Y2K remediation and support projects for all of NIST.

However, as noted last year, NIST still seems to be without a shared, institution-wide vision about which services should be provided centrally and which should be provided locally in the other operating units throughout NIST. In meetings with users from seven NIST laboratories, the panel learned of several examples of independent contracts for outside commercial support services. In addition, some operating units such as the Manufacturing Engineering Laboratory manage and run their own support organizations. The issues related to defining an information technology support architecture for NIST are discussed in the laboratory-level overview, and the panel reiterates its previous insistence on first determining and then communicating the appropriate long-term balance between centrally and departmentally supported laboratory-wide IT services. Efforts have begun on this task, and the panel urges that the activity continue to be given a high priority and that the Distributed Computing and Information Services Division be involved in these efforts.

Impact of Programs

The Distributed Computing and Information Services Division provides a wide range of important support services to staff throughout NIST. Customers indicate through the automated customer service satisfaction survey that these services are well received, and the division is working on ways of involving users in additional service-related measurements. The panel believes that the division might be able to increase its NIST-wide impact by engaging in high-level, strategic discussions with staff from other laboratories. Collaborative planning requires partnerships at all levels, from senior management to technical end users. In meetings with the users, the panel learned that such consultations had not occurred recently but would be received enthusiastically. Potential topics for these consultations include the exchange of information about emerging technology needs of researchers and about new IT capabilities, discussion of how to establish metrics for support, and general confirmation of the significance of the support units. It is difficult for a service organization to gain respect as a valued partner in an environment in which one party gives orders and the other takes them, with little discussion of long-term goals and possible alternative solutions.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

Because the Distributed Computing and Information Services Division focuses on serving the information technology needs of NIST staff, its programs impact industry only indirectly. However, if the division began to work on the development of practical performance metrics for support organizations, the results would not only be consistent with its service mission within NIST but could also be of direct benefit to industry. There is a significant lack of meaningful and practical measurements available to organizations wishing to assess the performance and value of their IT support organizations. Distributed Computing and Information Services Division staff could attempt to develop metrics that assess information technology “as seen by the user.” Such measures would need to be systems-level measures, meshing evaluations of hardware, networks, and software into a single metric that provides accountability for service. This is not an easy task, but it is not impossible. Boeing, for example, has end-to-end availability measures, and Microsoft has developed internal metrics as well. Although a great deal of additional development is needed in this area, work would not have to start from scratch. There is also the potential for significant formal research collaborations between service personnel and the ITL research and development staff on metrics. Support-level research could best be done by the service people with assistance from staff who have experience with developing metrics in difficult environments. With this type of collaboration, the Distributed Computing and Information Services Division could experiment with different approaches to information technology metrics in the support environment and could make a significant contribution to NIST and industry at large by developing, using, and adapting innovative solutions.

Division Resources

Funding sources for the Distributed Computing and Information Services Division are shown in Table 8.7. As of January 2000, staffing for the Distributed Computing and Information Services Division included 77 full-time permanent positions, of which 71 were for technical professionals. There were also eight nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

TABLE 8.7 Sources of Funding for the Distributed Computing and Information Services Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

0.4

0.6

0.6

0.7

STRS-supercomputing

0.9

0.9

0.9

0.8

OA/NFG/CRADA

0.2

0.1

0.0

0.0

Other Reimbursable

0.5

0.6

0.4

0.3

Agency Overhead

5.7

6.3

7.1

7.1

Total

7.7

8.5

9.0

8.9

Full-time permanent staff (total)a

66

62

72

77

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

The ratio of information technology service personnel to all personnel supported is quite low (the Gaithersburg campus houses approximately 3000 people). Between January 1999 and January 2000, the division added six full-time technical professionals, but the total budget for the division dropped slightly over the same period. Recruiting and retention programs are working well for the division. The new division chief reports that the turnover rate for the past year was approximately 5 percent, a low figure for an IT support organization in a major metropolitan area.

During the fall of 1999, division staff conducted a desktop support benchmarking study with personnel at the National Institutes of Health. This study identified several best practices that the division plans to examine in more depth to understand how they might be applied in the NIST environment. In addition, the Personal Computer Support Group performed a Baldrige self-assessment to identify future training needs and to investigate ways of improving the effectiveness of internal procedures. The panel applauds these efforts by the division to make the best possible use of its resources.

Software Diagnostics and Conformance Testing Division
Division Mission

According to division documentation, the mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, conformance to standards, and correctness; to participate with industry in the development of forward-looking standards; and to lead efforts for conformance testing, even at the early development stage of standards.

The division's work on designing conformance and diagnostic tests and developing reference implementations with participation from industry clearly fulfills its mission and is consistent with the goals expressed in both the laboratory and the NIST missions.

Technical Merit and Appropriateness of Work

The Software Diagnostics and Conformance Testing Division is providing industry with necessary technical leadership and is developing reference implementations and conformance test suites. The continued focus on new technologies and on maintaining close partnerships with industry ensures that division programs have a significant impact. To support standards work, the division performs research in areas that improve conformance testing. Research on how to develop an automatic generator of tests from formal specifications is proceeding well; recent accomplishments include an improved test minimizer tool and a new coverage measure and report tool. This research is of significant value because the generated tests would be inexpensive to develop and, by definition, would be consistent with the specifications. The joint project with the Statistical Engineering Division to investigate whether statistical analysis can ensure that software conforms to a specification seems to be well under way, after a hiatus last year. This work remains both intriguing and potentially valuable, and the panel is pleased to see resources applied to it by both divisions and by other NIST Laboratories (Manufacturing Engineering, Electronics and Electrical Engineering, and Chemical Science and Technology). Over the past year, significant progress has been made on the project that aims to develop a repository of error, fault, and failure data; the public repository system has been set up, and eight projects are already in this system (nine more are due to be installed soon). Companies can use the data and NIST analysis of the data to improve their software development processes and the reliability of their products. Thus, there is a strong need to capture such data and make them available in the public domain. Newly obtained data will be installed outside the NIST firewall shortly, and more than 1000 users from companies and

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

agencies such as Boeing, Johnson Space Center, and Viacom have already downloaded the repository management tool.

To maximize the leverage of NIST work, the division aims to influence standards development in the early stages; therefore researchers focus on studying emerging technologies where standards work will be needed but has not yet begun. An example of this approach can be seen in the project investigating ADLs for component-based software development. This research is definitely needed: industry and academia are producing good work on ADLs, but the expressive and analytical capabilities vary among the ADLs because each has been developed to meet a different need.

The planning and documentation methods used by the Software Diagnostics and Conformance Testing Division could serve as a model for other ITL divisions. The process has used clearly identified criteria to select state-of-the-art programs with clearly defined priorities and goals, well-identified roles, and measurable contributions to national and international standards organizations. Examples include the work on conformance tests for XML, for distributed interactive learning systems, and for the interactive DASE in collaboration with the High Performance Systems and Services Division. Directions can change quickly within the software industry. The division carefully monitors these changes and acts appropriately. Some projects have been downsized. Very limited resources will be applied to work on the next generation of the VRML, and both the Real-Time Java project and the work on role-based access control (RBAC) satisfied their completion criteria and were concluded. The software copymarking project was also terminated, eliminating the concerns in last year 's report regarding its nondisclosure restrictions. As old projects end, the division can expand into new areas. The ITL has initiated a laboratory-wide program on pervasive computing. The panel expects that the Aroma/Air Java project, this division's contribution to the pervasive computing effort, will provide needed diagnostic tools and serve as a nucleus for software standardization.

Impact of Programs

The Software Diagnostics and Conformance Testing Division's programs continue to have a positive impact in many ways. To disseminate information about technologies developed at NIST, division staff publish reports in external journals and books, write internal (but publicly available) NIST reports, and give presentations at conferences, often at industry's request. In fiscal year 1999, the total outputs for the division were 53 external publications, 14 internal documents, and 52 presentations.

The RBAC project produced four patents (two issued and two pending). This work was completed last year, and the technology developed is now being utilized in the division's Healthcare Information Systems project, which is supported by the Department of Veterans Affairs (VA) and the NIST Advanced Technology Program. A remote procedure call broker has been developed and is in use throughout the VA system, and work is under way on an RBAC authentication mechanism for similar deployment. In addition, the specification for an authorization framework for distributed health care environments, which was developed as part of this project, has been adopted by the Object Management Group and is being implemented by industry partners.

The laboratory provides well-recognized benefits to standards consortia, including unbiased conformance tests, reference implementations, and recommendations on how to improve standards to make them more precise and more generic. For example, NIST is very active on IEEE, Department of Defense, and other standards committees that support electronic distributed learning, an area with the potential to supply the next big wave of Internet-based applications. Developers of instructional management systems (IMS) have already recognized NIST as an important contributor to the successful development of the first version of specifications. Another vital NIST role is providing a neutral setting

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

TABLE 8.8 Sources of Funding for the Software Diagnostics and Conformance Testing Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

5.2

4.6

4.8

4.7

Competence

0.0

0.5

0.6

0.5

ATP

0.3

0.4

0.4

0.4

OA/NFG/CRADA

1.5

2.2

0.6

1.0

Other Reimbursable

0.4

0.2

0.0

0.0

Total

7.4

7.9

6.4

6.6

Full-time permanent staff (total)a

54

41

39

37

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

for discussing conformance testing issues. Examples of this division 's skill at this task can be seen in its successful leadership (as team chair) on two important standards committees: the DASE Conformance Working Group and the Organization for the Advancement of Structured Information Standards (OASIS) XML Conformance Subcommittee. Industry has provided many testimonials to the great value of the division 's standards work, including “The OASIS-NIST XML Conformance test suite is critical for our industry ” (Norbert Mikula, chief technical officer, OASIS); “NIST made strong contributions (X3D) and resolved our knottiest problems ” (Don Brutzman, board of directors, Web3D Consortium); and “The high quality test suite and certification program is an invaluable resource for ATA [Air Transport Association]” (Robert Peel, Director of Airworthiness and Standards, ATA).

Division Resources

Funding sources for the Software Diagnostics and Conformance Testing Division are shown in Table 8.8. As of January 2000, staffing for the Software Diagnostics and Conformance Testing Division included 37 full-time permanent positions, of which 33 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

At this time, the division does not have any problems with equipment or resources.

Statistical Engineering Division
Division Mission

According to division documentation, the mission of the Statistical Engineering Division is to contribute to research in information technology, to catalyze scientific and industrial experimentation, and to improve communication of research results by working collaboratively with, and developing effective statistical methods for, NIST scientists and its partners in industry.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

The division continues to investigate an impressive array of problems and to be involved in a wide range of activities. The majority of the division's projects involve collaborations with scientists and engineers across all of NIST. This breadth represents a subtle but noteworthy shift in priorities from recent years. The emphasis is now closer to the historic and natural focus of the division: seeking out the best opportunities for impact, wherever they are found, rather than attempting to concentrate on problems within the ITL domain.

Technical Merit and Appropriateness of Work

Statistical Engineering Division personnel apply state-of-the-art statistical techniques to the projects undertaken by staff members throughout NIST laboratories. When existing techniques are inadequate, division staff develop new approaches. Since the need for statistical support is pervasive across NIST, this mode of operation is not just appropriate but absolutely essential for the institute to fulfill its measurement and standards mission. Below, the panel briefly describes several of the division's projects to illustrate the range of areas covered and the types of projects undertaken:

  • The Effect of Query Choice on Information Retrieval: This work develops graphical analysis methods that enable assessment of the effects and interactions of query topics, query formulations, and searching systems in information retrieval problems. The approach is built on sophisticated statistical methods, including multidimensional scaling. Increased understanding of the effects of varying query formulation can help improve the relevance of documents returned from a search of the World Wide Web. The project is in collaboration with the Information Access and User Interfaces Division of the ITL.

  • Statistical Modeling for Polymer Temperature and Pressure Measurement: The goal of this project is to relate the fluorescence of a dye that is added to a polymer during processing to the temperature and pressure of the material. The relationship is nicely systematic but appears to be quite complicated to describe mathematically. By taking a very creative approach, a model of feasible complexity has been developed that accurately represents the phenomenon under study. This project is in collaboration with the Polymers Division of the Materials Science and Engineering Laboratory.

  • Statistical Measure of Image Sharpness for Scanning Electron Microscopes (SEMs): A practical and effective numerical measure of image sharpness was developed using a bivariate measure of kurtosis computed from the two-dimensional Fourier transform of an SEM image. An automated system using this algorithm has been implemented by industry to improve performance of SEMs used in semiconductor wafer production lines. This work was done in collaboration with the Precision Engineering Division of the Manufacturing Engineering Laboratory.

  • Magnetic Trapping of Ultracold Neutrons and Determination of the Mean Lifetime of the Neutron: This collaboration supports a world-class experiment in fundamental physics that will provide essential data for astrophysical theories relating to the Big Bang. The project involves researchers from the Ionizing Radiation Division of the NIST Physics Laboratory and from Harvard University, Los Alamos National Laboratory, and the Hahn-Meitner-Institut in Berlin. The most recent statistical contribution has been to develop improved models and estimation methods for estimating the half-life of the neutron. The statistical modeling and analysis methods are also guiding refinements to the experiment design as the project progresses.

  • Statistical Models of Fourier Transform/Ultraviolet Spectral Estimation: This project is aimed at developing statistical models and algorithms to improve NIST's chemical measurement services in the area of Fourier transform spectrometry in the ultraviolet range. The improvements are based on

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

more accurate models of the measurement system responses, plus development of efficient computational algorithms for statistical fitting and estimation. This work is in collaboration with the Analytical Chemistry Division of the Chemical Science and Technology Laboratory and with the High Performance Systems and Service Division of the ITL.

In addition to the above projects, the division is also in the enviable position of being associated with three Competence projects (cross-laboratory collaborations funded by the NIST director to build expertise in a new area). One of these projects is the work on Bayesian metrology. The panel believes that there is considerable unrealized potential in this effort and that the division should leverage its efforts by tapping the expertise of outside researchers, perhaps through formal collaborations, workshops, or visiting appointments. One of the goals should be to facilitate interactions between statisticians and metrologists.

The division is working on a strategic plan that should help put its current efforts into a larger context and identify new opportunities and issues. The panel was informed that the plan will include new research areas such as information technology, biotechnology, combinatorial chemistry, and methods for dealing with massive and new forms of data. The plan will also recognize the importance of continually upgrading the division's capabilities, both by increasing computing power and by expanding the skills and expertise of staff. Currently, the division is looking to fill two vacant technical staff positions; recruiting people with the capabilities needed to pursue new research directions will be an important step toward preparing the division to meet future challenges. The panel supports the strategic planning effort and urges that completion of the plan be a high priority for division management.

Impact of Programs

As illustrated above, the Statistical Engineering Division engages in a myriad of interactions and has substantial impact on various projects across NIST's laboratories. For example, the division's work with the Manufacturing Engineering Laboratory on an image sharpness monitor for SEMs was recognized with the Department of Commerce Silver Medal award. The division also continually seeks out new opportunities. Staff efforts to build new relationships with personnel in other NIST laboratories, although successful, are seriously impeded by the division's relative isolation in NIST North. The panel strongly urges that the Statistical Engineering Division be relocated to the main campus. The contributions of this division are such that it should be as physically close to the center of gravity of NIST activities as possible in order to promote easy and spontaneous interactions. Lowering the cost (in time and effort) of collaborations and reducing the number of missed opportunities would easily justify a move.

Beyond collaborative work on specific projects, division personnel naturally provide general statistical education to NIST staff. During the past year, the division offered NIST scientists a tutorial lecture series on analysis of variance and an oversubscribed course on practical lessons in statistical uncertainty analysis. Division personnel also gave about 20 invited lectures to external audiences. In a similar spirit, the division ran three workshops specifically aimed at industrial researchers; the topics were methods of uncertainty analysis, advanced mass measurements, and uncertainty calculations in chemical measurements.

Division Resources

Funding sources for the Statistical Engineering Division are shown in Table 8.9. As of January 2000, staffing for the Statistical Engineering Division included 19 full-time permanent positions, of

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

TABLE 8.9 Sources of Funding for the Statistical Engineering Division (in millions of dollars), FY 1997 to FY 2000

Source of Funding

Fiscal Year

1997 (actual)

Fiscal Year

1998 (actual)

Fiscal Year

1999 (actual)

Fiscal Year

2000 (estimated)

NIST-STRS, excluding Competence

2.6

2.8

2.9

2.5

Competence

0.1

0.3

0.5

0.6

STRS-supercomputing

0.5

0.5

0.1

0.0

Measurement Services (SRM production)

0.0

0.0

0.0

0.2

OA/NFG/CRADA

0.0

0.0

0.1

0.1

Other Reimbursable

0.1

0.0

0.0

0.0

Total

3.3

3.6

3.6

3.4

Full-time permanent staff (total)a

30

21

23

19

NOTE: Sources of funding are as described in the note accompanying Table 8.1.

a The number of full-time permanent staff is as of January of that fiscal year.

which 17 were for technical professionals. There were also seven nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The division staff also includes four people who have joint appointments at NIST and as faculty members of universities (two in Boulder, two in Gaithersburg). The panel continues to strongly support such arrangements because the contributions of these people are invaluable for the work of the division. Given the small size of the Boulder contingent (five permanent NIST employees), the division relies particularly heavily on faculty members, and their services are essential to support NIST researchers at the Boulder facility.

The technical staff in the Statistical Engineering Division publish regularly and serve on a variety of committees. In 1999, the division produced 29 publications and technical reports, and personnel served on six standards committees and four professional society committees. In addition, one staff member is on the editorial boards of the Journal of Applied Statistics and the Journal on Total Quality Management, and two other staff members were guest editors for a special issue of the International Journal of Imaging Science and Technology. Another staff member was recently elected as a fellow of the American Statistical Society (ASA), bringing the total number of ASA fellows in the division to five.

The division appears to have ample resources at its disposal to increase its outreach into the statistics community, hire postdoctoral research associates, engage in workshops, and otherwise fulfill its mission. At the Boulder facility, there are some issues involving office space, equipment, and courses; these problems deserve attention so that staff members there are supported on a par with their colleagues in Gaithersburg.

Selection of a new division chief must be one of the highest-priority items. In the current job market, this may not be an easy position to fill. The panel recommends the selection of a strong technical visionary who is well known in the statistics community—someone who can help to hone, communicate, and realize a vision for the division that will increase its status and impact. The ITL director may find it helpful to form an external advisory committee to help locate viable candidates for this position.

The panel encourages the division not to wait for a new division chief to come on board and to take advantage of strong support from the new ITL director to execute its internal mission and increase its

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×

involvement with the external statistics research community in ways that advance its mission. Given the wonderful array of challenging problems to which the division can contribute—an array that would be coveted by almost any statistics group—there is every reason to believe that the future could and should be extremely bright.

MAJOR OBSERVATIONS

The panel presents the following major observations:

  • The panel appreciates the openness and responsiveness of ITL management and staff to its requests and suggestions.

  • The planning process within ITL continues to improve. This year, the panel was particularly pleased by the laboratory-wide strategic plan and the increased emphasis on criteria for beginning and ending projects and the definition of specific goals.

  • The panel is very concerned about the constant or, in some cases, decreasing staffing levels within ITL. This situation is due in large part to the limited funding available to cover salaries. The panel observes that the situation has two consequences: the first is that ITL personnel are spread very thin and some projects have only single-point coverage; the second is that many areas exist in which NIST participation could make a significant difference but ITL cannot afford to be active at this time.

  • The new initiatives in computer security represent an exciting opportunity for the ITL. However, care must be taken to ensure that the scope of the programs is aligned with the somewhat limited allotment of funding and the number of personnel available.

  • Since topics in information technology increasingly cross the laboratory's divisional boundaries, the panel believes that there has to be more emphasis on interdisciplinary collaboration. Although there are no explicit barriers to cross-divisional work, there seems to be little encouragement or support for such projects. A good example of an area in which cross-divisional cooperation must take place is the Pervasive Computing Initiative, which touches almost every division in some way. The panel believes that in such situations, there must be formal coordination among the divisions to ensure that every division that should participate is doing so, that efforts neither conflict nor are duplicated, and that potential synergies are effectively exploited.

  • The Statistical Engineering Division plays a fundamental role in the execution of the NIST mission throughout the Measurement and Standards Laboratories. To ensure that this group can continue to meet the diverse statistical needs of NIST activities and contribute to the national statistics community, it is important for ITL management to reinforce the message that the work of the division is valued and to identify and hire a strong leader for the division as soon as possible. In addition, both effectiveness and morale would be improved if the division could be moved from NIST North to the main campus. Staff and projects in the Mathematical and Computational Sciences Division would also benefit from such a relocation.

  • Provision of information technology support to NIST continues to be an issue. The panel applauds the latest efforts to define a coherent system for services. Some elements, such as a user's forum and input from current support staff, could be added to this effort to increase its chance of success.

Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 219
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 220
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 221
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 222
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 223
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 224
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 225
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 226
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 227
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 228
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 229
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 230
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 231
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 232
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 233
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 234
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 235
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 236
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 237
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 238
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 239
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 240
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 241
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 242
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 243
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 244
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 245
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 246
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 247
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 248
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 249
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 250
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 251
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 252
Suggested Citation:"Information Technology Laboratory." National Research Council. 2000. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000. Washington, DC: The National Academies Press. doi: 10.17226/9979.
×
Page 253
Next: Appendix A: Charge to the Board and Panels »
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2000 Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!