Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 219
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 8 Information Technology Laboratory
OCR for page 220
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 PANEL MEMBERS Louise H. Trevillyan, IBM T.J. Watson Research Center, Chair Tony Scott, General Motors Corporation, Vice Chair Bishnu S. Atal, AT&T Laboratories Research Matthew Bishop, University of California, Davis Linda Branagan, Construct Internet Design Jaime Carbonell, Carnegie Mellon University Josephine Cheng, IBM Santa Teresa Laboratory Albert M. Erisman, The Boeing Company John R. Gilbert, Xerox Palo Alto Research Center Roscoe C. Giles, Boston University Andrew S. Grimshaw, University of Virginia Thomas P. Kehler, Recipio Jon R. Kettenring, Telcordia Technologies John W. McCredie, University of California, Berkeley Vijayan N. Nair, University of Michigan Lawrence O' Gorman, Veridicom, Inc. Thomas Parenty, Consultant, Oakland, California Craig Partridge, BBN Technologies Radia Perlman, Sun Microsystems Laboratories James L. Phillips, The Boeing Company K.K. Ramakrishnan, AT&T Laboratories Research Stephanie M. White, Long Island University Eddie L. Zeitler, Charles Schwab & Company Submitted for the panel by its Chair, Louise H. Trevillyan, and its Vice Chair, Tony Scott, this assessment of the fiscal year 2000 activities of the Information Technology Laboratory is based on a site visit by the panel on February 23-24, 2000, in Gaithersburg, Md., and documents provided by the laboratory.1 1 U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, 1999 Information Technology Laboratory Technical Accomplishments, NISTIR 6365, National Institute of Standards and Technology, Gaithersburg, Md., 2000.
OCR for page 221
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 LABORATORY-LEVEL REVIEW Laboratory Mission According to laboratory documentation, the mission of the Information Technology Laboratory (ITL) is to strengthen the U.S. economy and improve the quality of life by working with industry to develop and apply technology, measurements, and standards in information technology. In support of this mission, the ITL works with industry, research, and government organizations to develop the information technology standards, conformance tests, test methods, reference data, proof-of-concept implementations, and other tools that are needed by U.S. industry to produce usable, secure, scalable, and interoperable information technology systems. The ITL has eight divisions. Four of these—the Advanced Network Technologies, Computer Security, Information Access and User Interfaces, and Software Diagnostics and Conformance Testing Divisions—are focused primarily on performing research and developing measurement methods and standards in information technology in support of the ITL and NIST missions. The four other divisions of ITL are devoted to service and collaborative support. The High Performance Systems and Services Division and the Distributed Computing and Information Services Division provide support for computer systems and software used in ITL and other NIST laboratories, as well as for the NIST-wide infrastructure. The Mathematical and Computational Sciences Division and the Statistical Engineering Division provide technical assistance and advice on analysis, modeling, and statistics through collaborations with staff throughout NIST. This range of activities makes ITL an especially heterogeneous laboratory, with attendant management challenges. Each division has a key role to play and all of the components—research, measurement, standards, and services—should be valued appropriately by management. An area that may need some management attention is communication of the laboratory and divisional missions to the staff. In its interactions with employees from across ITL, the panel found some confusion about what the mission statements were, what they meant, and how they related to the projects that staff were working on. It also found a belief that some “customers” were more valued than others. For example, some staff believed that more credit is given for work done with people in industry than for work with people in government and that more credit is given for work with other government agencies (for example, the Federal Bureau of Investigation (FBI)) than for work with other staff within NIST. It has been more than 4 years since the ITL was formed by combining two laboratories. Management at all levels has turned over since the merger, and ITL seems to have largely solved the problems associated with the transition. In general, the various divisions seem to be pulling together to form a coherent team. However, some areas still require attention. In particular, the Statistical Engineering Division has been trying to recruit a permanent division chief. The panel believes that a high priority should be given to finding a strong leader to fill this position, a person who can help the division redefine its long-term strategic plan and carry out its important role within ITL and across NIST. This issue is discussed further in the review of the Statistical Engineering Division. Technical Merit and Appropriateness of Work Examples of the many excellent projects under way in the laboratory are discussed in detail in the divisional reports. Overall, the programs are appropriate and well aligned with laboratory and divisional missions. Last year, the panel noted that the ITL planning process had improved, and the panel observed continued progress this year, including the development of a laboratory-wide strategic plan for 2000
OCR for page 222
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 2004.2 New projects are started based on their relevance to U.S. industry, and the work examined by the panel generally had specific, focused goals. An example is the work on the Extensible Markup Language (XML) done in the Software Diagnostics and Conformance Testing Division. This project is notable because it involves a major standard on which industry was making little progress and it also effectively leverages skills and approaches developed in a now terminated project on Virtual Reality Modeling Language (VRML). An important new activity is ITL 's critical contribution in emerging standards for local wireless networks, where NIST can again play the role of honest broker to help harmonize many competing technologies. Another aspect of the improved planning process is the existence of clear termination criteria, which have been useful for sunsetting projects. For example, when the Real-Time Java project and the work on role-based access control (RBAC) satisfied their completion criteria in 1999, they were ended. One concern of the panel is that, with the explosive growth of information technologies, more metrology work is needed on faster schedules. This increased pace implies that decisions on which standards NIST should support must be made quickly, often with incomplete information, and that in the future ITL will have to take more risks in project initiation. It is important for NIST to get involved early in areas where standards are not set yet and there is no industry consensus. Proactive work by ITL is vital to shape industry products and behavior; because technology changes so rapidly, coming in late in the process would be ineffective. Good examples of areas in which ITL has already demonstrated the benefits of proactive involvement include the work on electronic books in the High Performance Systems and Services Division, the project on architectural description languages (ADLs) in the Software Diagnostics and Conformance Testing Division, and the standards development work in the Java Numerics project in the Mathematical and Computational Sciences Division. This year, the panel viewed two major new programs in the ITL, one in pervasive computing and the other in security. At the time of last year's panel meeting, the pervasive computing initiative was just beginning, and the fiscal year 1999 assessment contained several cautionary statements regarding the panel's views of the appropriate role of ITL in this field. Specifically, the report said that the laboratory should concentrate on topics where NIST possesses special expertise or has a unique role to play, such as involvement in early standards activities, coordinating industry consortia, and building reference implementations. This year, the panel was pleased to discover that the laboratory had made significant progress on more clearly defining the initiative. The NIST workshops on smart spaces and pervasive computing were well connected and provided a good start for building relationships with the community. The panel also approved of many of the individual projects, especially in the Advanced Network Technologies Division. However, there are still some cases in which there has not been a focus on producing work unique to ITL; specific examples are discussed in the divisional reports. The laboratory as a whole has still not managed to define the specialized role of NIST in this area: Exactly how should the elements of pervasive computing be measured, where are standards most necessary, and what tools will be needed for these tasks? To ensure that ITL has an impact, it must continue to strive to identify its mission in interoperability, standardization, and metrology for pervasive computing. Work in this field is rapidly expanding in industry and academia, and NIST must take care not to duplicate other efforts but rather to complement and support emerging technologies. This challenge is an issue both in individual projects and at the level of the laboratory as a whole. The pervasive computing work would benefit from a formal management-coordination structure across the entire laboratory. Such a structure would assist in answering the questions posed above and would also 2 U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, ITL Strategic Plan January 2000, National Institute of Standards and Technology, Gaithersburg, Md., 2000.
OCR for page 223
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 facilitate collaboration across divisional lines. While the panel observed that informal cooperation already exists among ITL divisions, such cooperation is critical to the success of ITL in the pervasive computing area, and there must be a management system in place to ensure that the effort hangs together coherently across the laboratory. The other major program planned for the ITL is a collection of three security initiatives proposed in the President's budget for fiscal year 2001. While the final details and shape of the program will not be clear until the budget process is complete, provisional plans were presented to the panel. The three components of this effort are (1) the Institute for Information Infrastructure Protection, through which NIST will coordinate the distribution of approximately $50 million in grants for security technology development; (2) the critical infrastructure protection activities, for which NIST is slated to receive $5 million per year to identify best security practices and to develop tools to detect and remedy security vulnerabilities; and (3) the Expert Review Team, which would be funded at about $2 million per year and would create eight new permanent positions at NIST with responsibility for identifying existing vulnerabilities in federal computer systems and providing advice on fixing them. ITL's Computer Security Division is well known and well respected internationally, so ITL is an obvious home for this collection of security programs. The panel is looking forward to heating next year about the progress that has been made on these security initiatives but has two warnings for the laboratory. First, the market for personnel with computer security expertise is extremely tight. The ITL Computer Security Division has been losing staff, so these initiatives would require NIST to recruit replacements as well as fill a significant number of new slots. In the current environment, it may be very difficult to find these people in a timely fashion. Second, the projected funding for the initiatives does not match the scope of the programs and goals presented to the panel. For these programs to be successful, projects must be chosen carefully to leverage existing expertise of ITL staff, and specific and achievable goals must be carefully defined. Finally, the panel continues to be concerned about the absence of a coherent strategy and architecture for providing information technology support to NIST. The current situation has services being supplied by an amalgam of groups, including the High Performance Systems and Services and the Distributed Computing and Information Services Divisions in ITL, several external companies (by means of outsourcing contracts), and, occasionally, significant collections of staff in units outside ITL (e.g., as the Manufacturing Engineering Laboratory's Service Administration). This is a costly and inefficient system. The NIST-wide Information Technology Services Planning Team (ITSPT) formed to tackle the problem held its first meeting in February 2000. The group includes representatives from the management of all NIST operating units, and the panel applauds this initiative. To most efficiently determine the best system for information technology support within NIST, the panel suggests that two key ingredients be added to the ITSPT's effort. First is the formation of a broadly based users' forum, including scientific and administrative staff from throughout NIST. These end users are significantly affected by the NIST service structure, and a forum would allow the ITSPT to receive feedback about the current system, to solicit input into the evaluation process, and to communicate ideas and proposed architectures. Second is the involvement of current information technology support staff in the evaluation and (potential) reorganization; the management and staff of the High Performance Systems and Services and the Distributed Computing and Information Services Divisions have valuable experience in the information technology (IT) needs of NIST's heterogeneous organization. To define and implement an effective and efficient information technology services architecture, NIST-wide management, end users, and support staff will all have to buy into the new system; by involving them at the start, the chances for success are increased. The panel notes that with the reevaluation being undertaken by the ITSPT, NIST also has an opportunity to study what gains in efficiency and effectiveness can be achieved
OCR for page 224
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 by increasing the standardization of its IT infrastructure. Standardization cannot be dictated but instead should be developed in the best interests of a broad range of customers. To move in the direction of more standardization will require working closely with the end users in the laboratories to identify (1) research that requires experimental platforms outside of any standards and (2) areas where standard platforms would be adequate for the work and bring value through commonality of training, support, and procurement. Impact of Programs The ITL continues to have a positive impact on the industrial IT community. Its portfolio of projects is extensive and appropriate and generally chosen to maximize NIST's impact. ITL has been effective in influencing industry, especially in areas such as XML, where industry was stalled in coming to agreement. The recent work on numerical computation in Java is another example of how ITL can have a positive impact on industry. The laboratory uses many mechanisms to interact effectively with industry. A wide variety of consortia have been set up by and for information technology companies, and NIST plays a role in the most important of them. The laboratory also makes good use of industry roadmaps as a guide to initiating new projects. Where no roadmaps exist, ITL uses its influence as a neutral party to help project industry's needs through the use of publications, workshops, and collaboration with industry consortia. The panel singled out the following ITL activities as having had a great impact on U.S. industry: specifications for instructional management systems (IMS) for distance learning, development of role-based access control protocols, modeling of material microstructures, conformance testing for XML, the Text Retrieval Conference (TREC) series, drafting of the Common Criteria for information security, and visualization work with other NIST laboratories. More information about these projects and many others that are filling critical needs in the industrial IT community is provided in the divisional assessments. In addition to interacting with external organizations, ITL also provides vital services to the entire community within NIST. Laboratory staff provide support for networking, high-performance computing, desktop boxes and workstations, and other systems vital to NIST' s infrastructure. The Statistical Engineering and the Mathematical and Computational Sciences Divisions provide valuable support in their specialties to other NIST laboratories and within ITL. All of these support activities are crucial to NIST, and ITL should make it clear to the employees providing these services that their contributions are appreciated and that their work in support of NIST's staff is essential to the valuable role NIST plays in the U.S. economy. Laboratory Resources Funding sources for the Information Technology Laboratory are shown in Table 8.1. As of January 2000, staffing for the Information Technology Laboratory included 381 full-time permanent positions, of which 310 were for technical professionals. There were also 85 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. The number of computer scientists at NIST has increased by 143 percent since 1980. Even if all of these new people were in ITL (which they are not), this would still be an extremely modest increase considering the explosion of the IT industry in the last 20 years. While the panel would not argue that the staff of ITL must increase exactly in proportion to the increase in the U.S. IT industry, it does appear that the ITL is seriously understaffed in relation to the job it has to do. Owing to slight declines in STRS funding and increases in personnel costs (e.g., salary and benefits), ITL can now support fewer full-time permanent staff members. Several divisions (especially
OCR for page 225
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 TABLE 8.1 Sources of Funding for the Information Technology Laboratory (in millions of dollars), FY 1997 to FY 2000 Source of Funding Fiscal Year 1997 (actual) Fiscal Year 1998 (actual) Fiscal Year 1999 (actual) Fiscal Year 2000 (estimated) NIST-STRS, excluding Competence 31.2 31.6 31.6 30.4 Competence 0.7 0.9 1.5 1.5 STRS-supercomputing 12.3 11.8 12.1 12.0 ATP 2.1 1.8 1.8 1.2 MEPa 0.1 0.0 0.0 0.0 Measurement Services (SRM production) 0.0 0.0 0.0 0.2 OA/NFG/CRADA 9.9 10.6 8.4 8.7 Other Reimbursable 1.9 1.5 0.5 1.5 Agency Overhead 11.0 12.0 14.4 16.1 Total 69.2 70.2 70.3 71.6 Full-time permanent staff (total)b 420 362 381 381 NOTE: Funding for the NIST Measurement and Standards Laboratories comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST's congressional appropriations but is allocated by the NIST director's office in multiyear grants for projects that advance NIST's capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST's ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Funding to support production of Standard Reference Materials (SRMs) is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding, including that for Calibration Services, is grouped under “Other Reimbursable.” aManufacturing Extension Partnership. b The number of full-time permanent staff is as of January of that fiscal year. Computer Security and Statistical Engineering) have lost key people in recent years, so while the present collection of employees is technically strong and well qualified, maintaining staffing levels and hiring new people are serious issues for ITL and division management. Personnel in the laboratory are already stretched very thin, with many single-person projects and, in some areas, what appears to be overreliance on temporary guest researchers. ITL has in general been successful at recruiting new personnel when the budget allows, and the laboratory also has an impressive retention rate among existing staff. (Turnover in ITL is much lower than turnover at many of the panel members' companies.) However, the demand for skilled people in the IT industry is extremely great, and NIST should be vigilant about tracking the effectiveness of its recruiting and retention efforts. ITL should consider developing a contingency plan to be implemented if the current success begins to wane or if the opportunity or need suddenly arises to hire a significant number of new staff. It has been suggested that agencies such as the Defense Advanced Research Projects Agency (DARPA) have found mechanisms that allow them to offer somewhat higher salaries and that NIST might look into employing similar methods. The types and uses of information technologies are exploding, and new developments occur at “Internet speed.” The rapid changes necessitate continually increasing the number of standards at a continually increasing pace. Given the current staffing levels in ITL, the panel is concerned that there
OCR for page 226
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 may not be enough personnel in place to cover basic standards needs. Owing to its limited resources, ITL is not participating in some arenas where its expertise would be valuable and its work consistent with the NIST mission. Examples include interdomain routing standards, multicasting, Internet measurements (performance and reliability), hardware and software design, and related intellectual property modeling and simulation standards. Work with international standards bodies could also be increased. The panel is not claiming that programs in these areas would be of higher priority than ITL's current programs, but rather that work on them would also be of significant benefit to industry if ITL had the resources to pursue them. In summary, the panel believes that ITL is already underfunded, and it appears that the laboratory is losing, rather than gaining, staff positions. The panel believes that it might be useful for ITL to attempt to compile a list of important standards that are not being addressed by ITL due to understaffing and to quantify the potential loss to U.S. industry because NIST is not participating in these standards. The panel also evaluated the state of NIST's data communications network. ITL includes staff who are consumers of the data communications services as well as staff who provide these services to all of NIST. The panel was pleased to learn that ITL expects to have wired NIST's entire internal network for switched high-speed (100 megabits per second (Mbps)) Ethernet by the end of 2002. The only issue on this front is that the expected completion date was recently pushed back from 2001, and the panel believes that a significant effort should be made to ensure that the installation is truly complete in 2002. The panel was more concerned to find that NIST's connectivity to the outside world is rather poor. NIST is using a 12-Mbps outside line at a time when the top 170 research universities in the United States are joining Internet2 using links of 155 Mbps or faster. (The Advanced Network Technologies Division does have its own 155-Mbps link to a D.C.-area testbed, but this link is not available to other divisions for a variety of technical and budgetary reasons.) NIST should move aggressively to improve the speed of its connectivity to the outside world. In the panel 's experience, much of the benefit of higher-speed connectivity, such as facilitation of distance collaborations and information sharing, cannot be accurately predicted or appreciated until the new capability is actually installed, but NIST still must recognize the need to keep pace with comparable institutions to ensure the continued high quality of its work. In general, morale in ITL seems to be quite high. Panel members met with bench-level employees in “skip-level” interviews (interviews without laboratory-level management present). The panel is pleased to note that ITL staff are almost uniformly enthusiastic about their work at NIST. They especially valued the intellectual freedom available at NIST and the interactions with their NIST and ITL colleagues. However, in some divisions, staff felt that the tendency to have one-person projects and the reliance on guest researchers often limited the opportunities for formal project collaboration with peers at NIST. Staff members are in general satisfied with the laboratory facilities and office space and feel that they have access to the equipment necessary for doing their jobs. Concerns about the isolation of some divisions in NIST North continue and are discussed in detail in the divisional-level reviews. As was noted last year, there is some frustration with the Department of Commerce bureaucracy. The procurement process has been singled out by the staff for particular criticism, especially the $2500 limit on streamlined purchases. When the cost of a purchase exceeds this amount, a more complex process is used that causes delays. As an example, some employees asserted that the cycle for procuring computer equipment was so long that the equipment could be obsolete by the time it arrived. Difficulties with Department of Commerce legal services also continue. Three years after the Software Diagnostics and Conformance Testing Division's original request for approval of its membership in the World Wide Web Consortium, the request was denied by the legal department, despite the fact that other government agencies are official members of this consortium.
OCR for page 227
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 DIVISIONAL REVIEWS Mathematical and Computational Sciences Division Division Mission According to division documentation, the mission of the Mathematical and Computational Sciences Division is to provide technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to U.S. industry. The division focuses on the development and analysis of theoretical descriptions of phenomena (mathematical modeling), the design and analysis of requisite computational methods and experiments, the transformation of these methods into efficient numerical algorithms for high performance computers, the implementation of these methods in high-quality mathematical software, and the distribution of the software to NIST and industry partners. The programs of the Mathematical and Computational Sciences Division conform very well to the division mission and to the ITL and NIST missions. Current programs provide mathematical modeling and analysis advice and tools for other NIST laboratories and for U.S. industry, and the division is pursuing activities to improve measurement methods and standards for mathematical work and software. The panel has seen notable improvement in the overall planning of programs and projects in the division. The Mathematical and Computational Sciences Division strategic plan put together this past year gives a blueprint for building on current activities and developing new programs to address current and future needs. The strategic plan has three thrusts. The thrust on measurement and calibration for the virtual sciences could have a significant industrial impact; it holds great challenges, but the division is well positioned to address them. The thrust on evolving architecture of tools, libraries, and information systems for computational science and engineering will build on many of the division's highest-impact capabilities and activities. The thrust on emerging needs of the NIST laboratories is a vital direction for the division and for NIST. However, to succeed in the last effort, the division will have to cultivate new technical expertise to address the growing need for analysis and modeling in such areas as complex networks, large data sets, and interacting computer systems. Adding personnel will be necessary to extend the division's capabilities. The division should also continue efforts to expand technical interactions with personnel from other ITL divisions. To fulfill the mission of the division, a healthy balance must be maintained between the work that supports specialized areas within NIST and the more application-independent methods with a broader external audience. The current portfolio of division projects contains an appropriate mix of these two types of programs. A potential challenge to sustaining this balance arises from the constant or decreasing number of staff in the Mathematical and Computational Sciences Division, coupled with the long-term nature of the collaborations with personnel in the other NIST laboratories. These enduring, multiyear relationships are fruitful, and the work produced helps NIST meet its goals. However, because there are so many long-standing demands on the division staff 's time, it is difficult to expand the number of projects or to form new collaborations with unexplored portions of NIST. Technical Merit and Appropriateness of Work The overall quality of the programs in the Mathematical and Computational Sciences Division is very high. The merit of the division's work is illustrated by the continuing recognition accorded to various projects, such as the object-oriented finite-element (OOF) analysis for the modeling of real material microstructures, which is one of the winners of Industry Week Magazine's 1999 Technology of the Year award. The OOF system is a general software tool that is used by a number of large companies,
OCR for page 228
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 and the project has attracted external support from the Department of Energy and includes collaborative work with a major research university, the Massachusetts Institute of Technology. The Mathematical and Computational Sciences Division consists of three groups. Most of the projects in the Mathematical Modeling Group are done in collaboration with researchers in other NIST laboratories. The emphasis is on domain-specific problems, and the results are of interest mainly to scientists involved in these particular areas. The Optimization and Computational Geometry Group also tends to work on modeling and analyses that are specific to particular applications. By contrast, the work of the Mathematical Software Group focuses on the development of algorithms and software in support of general computational science; thus, the results of this work are typically useful to a broad spectrum of scientists both inside and outside NIST. Many Mathematical and Computational Sciences Division programs support the NIST mission by providing the mathematical modeling and analysis needed for the success of important projects in other NIST laboratories. For example, the work on OOF was developed as part of the Materials Science and Engineering Laboratory's project on material microstructures. Also, work on modeling high-speed milling has been done with the Physics and Manufacturing Engineering Laboratories. In collaboration with staff from the Building and Fire Research, Physics, and Manufacturing Engineering Laboratories, analysis is being provided to support a project on computer graphic rendering of material surfaces. This project might also be relevant to graphics work in the entertainment industry, which is moving increasingly toward physics-based rendering. Other Mathematical and Computational Sciences Division programs align directly with the measurement and standards mission of the ITL. In the Java Numerics project, division personnel have been leaders in establishing arithmetic standards for the Java programming language so that this language can be used in scientific computing applications requiring both run-time efficiency and numerical accuracy. A project focused on the development of a code for the numerical solution of partial differential equations also developed F90GL, a Fortran binding for the OpenGL three-dimensional multivendor graphics user interface standard. Key divisional personnel are leading the standardization effort for basic linear algebra subprograms for sparse matrices (sparse BLAS), which are computational units widely used in the scientific computation community. The output of the most ambitious current project in the division, the Digital Library of Mathematical Functions (DLMF), will be a Web-based update of the Handbook of Mathematical Functions,3 one of the most heavily used references from NIST's predecessor, the National Bureau of Standards. Still other Mathematical and Computational Sciences Division programs support the division mission by advancing the methodologies and tools available for solving scientific problems of interest to U.S. industry. The division's Guide to Available Mathematical Software (GAMS) and its Matrix Market projects provide software references and test problems used extensively in the broad scientific computing community. The division's work in parallel adaptive multigrid methods and software to solve partial differential equations has been widely disseminated; this algorithm, pioneered in NIST's PHAML (Parallel Hierarchical Adaptive Multilevel project), is now being incorporated into PLTMG (Piecewise Linear Triangle Multigrid), one of the leading partial differential equations programs in use today. During the past few years, the JazzNet project demonstrated how a “personal supercomputer ” could be fashioned very inexpensively by harnessing clusters of personal computers; large commercial vendors now are beginning to offer products that use this approach. 3 M. Abramowitz and I.A. Stegun, eds., Handbook of Mathematical Functions, with Formulas, Graphs, and Mathematical Tables, National Bureau of Standards, Gaithersburg, Md., 1964.
OCR for page 229
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 Impact of Programs The overall impact of the programs in the Mathematical and Computational Sciences Division is significant and growing. The division correctly views its customers as falling into four categories: other ITL divisions, other NIST laboratories, the IT industry, and the computational science community. Examples of the division's positive impact on the latter three groups are noted in the above discussion of technical merit. The division is clearly continuing its strong support of and collaboration with scientists in other NIST laboratories, including teaching courses for NIST personnel on such topics as Fortran 90 and nonnumerical computational methods. In contrast, relatively few Mathematical and Computational Sciences Division activities are tied to those of other ITL divisions. However, the panel is pleased to note that forums for technical interchanges with these other divisions have been initiated and expects that the meetings will lead, over time, to further joint activity. Many of the division's projects have had a clear and important impact on the information technology industry (largely but not entirely through the computational science community, an active element in that industry). For example, in the Java Numerics project, the division's technical expertise and NIST's role as a neutral broker have had a significant impact on an important industry standard. In 1999, Sun Microsystems enthusiastically embraced the project's first set of recommendations, with an effect on literally every user of Java. This project is a good example of NIST proactively discerning a need (rather than simply responding to industry requests) and having great impact on a core industry issue. Another example of industrial impact is the OOF project mentioned above: it has been widely used in industrial research and development settings, and a new collaboration with General Electric has secured DOE funding to support the adaptation of OOF for use in a manufacturing setting at General Electric. The panel also notes that the DLMF could have a significant impact. The NBS handbook on which the DLMF is based is still cited frequently today, and the number of citations is rising. The online version of this information will be very valuable to users in industry and academia, and new National Science Foundation (NSF) support for the project will move the work significantly closer to completion and to realizing the impact throughout the scientific community. The Mathematical and Computational Sciences Division disseminates its results through several mechanisms, including the Internet, for which it has a long history of pioneering and effective use. This tradition continues with the GAMS (which had about 7 million hits in 1999; about 5000 external pages now contain links to the GAMS Web site) and the Matrix Market (which disseminates standard test data for scientific computation and has more than 100 users per day). Division staff also publish results in scientific journals and speak at conferences. Many results are disseminated by making software publicly available; examples in 1999 include two releases from the Java Numerics project—composite Java benchmarks packaged in SciMark and the Java Matrix Package (JamPack) for numerical linear algebra —and the object-oriented micromagnetic framework (OOMMF) modeling software. Individual staff members contribute to the U.S. scientific community not only through their work on division projects but also by serving on important committees. Three staff members serve as editors of major professional journals, and several division staff are on working committees dealing with modeling and computational standards issues. The Association for Computing Machinery (ACM) recently awarded its Outstanding Contribution Award to the division chief for his innovative efforts in electronic dissemination during his tenure on the ACM Publications Board and as editor-in-chief of ACM Transactions on Mathematical Software. This award confirms the professional stature of the division leadership and reflects the division's long tradition of significant activities in areas central to NIST's goal of promoting dissemination of standards and technology through a variety of mechanisms, including
OCR for page 243
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 leader of the Scientific Applications and Visualization Group deserves credit for an excellent job of managing this major effort. Phone system upgrades and the network upgrade have also had a positive impact on NIST staff throughout the laboratories. However, network speeds to the outside are still slow compared with other national computational facilities and even compared with universities now on Internet 2. While staff report that there has not been much demand from NIST researchers for high-speed outside network access, the panel believes that pressure for this capability will arise very soon (driven by both internal needs and external expectations) and that NIST should move to provide the service in a timely way. In the Scientific Applications and Visualization Group, work with the Physics Laboratory on the visualization of vortices in Bose-Einstein condensates is a high-impact, high-visibility project for NIST; the images produced were featured on the cover of the December 1999 issue of Physics Today. The group also developed the IMPI conformance tester, which is an industry resource. Division Resources Funding sources for the High Performance Systems and Services Division are shown in Table 8.6. As of January 2000, staffing for the High Performance Systems and Services Division included 81 full-time permanent positions, of which 59 were for technical professionals. There were also 21 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. During fiscal year 2000, the division experienced a management transition, but the panel notes that both the past and the current division chiefs have provided excellent leadership for the division. The High Performance Systems and Services Division is leading the ITL in its efforts to recruit, recognize, and promote individuals from underrepresented groups, and the group, project, and division management teams include women and minorities. Mechanisms for hiring U.S. postdoctoral research associates more flexibly and with less administrative overhead than allowed by the current National Research Council process might permit more effective recruitment of women and minorities. TABLE 8.6 Sources of Funding for the High Performance Systems and Services Division (in millions of dollars), FY 1997 to FY 2000 Source of Funding Fiscal Year 1997 (actual) Fiscal Year 1998 (actual) Fiscal Year 1999 (actual) Fiscal Year 2000 (estimated) NIST-STRS, excluding Competence 1.7 2.4 2.6 2.0 Competence 0.5 0.1 0.0 0.0 STRS-supercomputing 8.3 9.5 9.9 10.2 ATP 0.5 0.5 0.6 0.4 OA/NFG/CRADA 0.8 0.4 0.4 1.0 Other Reimbursable 0.9 0.7 0.0 1.1 Agency Overhead 4.9 5.3 6.7 8.1 Total 17.6 18.9 20.2 22.8 Full-time permanent staff (total)a 75 71 75 81 NOTE: Sources of funding are as described in the note accompanying Table 8.1. a The number of full-time permanent staff is as of January of that fiscal year.
OCR for page 244
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 Staffing is very thin in some areas. The project on wavelength division multiplexing was canceled in part because critical staff left NIST. The use of high school and college students in the Information Storage and Integrated Systems Group is a model for outreach and education, but it is not a good omen for the level of personnel that key projects can afford. Key elements of the electronic book effort were supported last year by temporary additions to the budget, which has strained the funding allotted to this area. The panel believes that direct internal support of this activity is needed. The Department of Commerce has suggested a department-wide pooling of computational resources. This effort would be laudable, if idle resources could be found. Within NIST, utilization of the infrastructure is very efficient, and there is no spare computational power or time. Therefore, the department's attempt to coordinate these resources may not bear fruit unless there is significant idle capacity of an appropriate type elsewhere in the department. In general, the panel believes that the activities of this division, especially as linked by the theme of convergent information systems, are appropriate and vital for NIST's role in supporting the U.S. information technology industries. Unfortunately, there is a mismatch between the many potential implementations of projects in this area and the current level of resources devoted to the work. A substantial level of investment of personnel and technical resources in this area would be strategically sound. Additional resources would deepen the support for existing work, ensure timely participation and leadership in standards and metrology activities, and perhaps also allow expansion into new areas consistent with NIST's mission and valuable to U.S. industry. Examples include security for distributed high-performance computing systems, biomimetic computing, work on large-scale scientific data systems and archives, software for implementing a shared memory programming model on general (distributed memory) computers, and focused and deep efforts on electronic commerce. Distributed Computing and Information Services Division Division Mission According to division documentation, the mission of the Distributed Computing and Information Services Division is to provide the information technology resources, supporting infrastructure, applied research, and assistance to NIST staff, collaborators, and clients for application in the conduct of scientific, engineering and administrative applications and in the dissemination of information. The Distributed Computing and Information Services Division's mission directly supports the goals and objectives presented in the ITL strategic plan.5 Last year, the panel noted that this division appeared to have little involvement in or knowledge of the long-range planning activities described by ITL senior management. In contrast, this year's Distributed Computing and Information Services Division operational plans and long-range goals appear to have been integrated into the ITL-wide strategic planning process. The final plan appropriately emphasizes the importance of providing information technology support to NIST staff. The division provides software for staff workstations, comprehensive desktop support services, electronic information dissemination services, file servers, and administrative and management applications support. The strategic plan describes an intention to embark on a comprehensive computer security program, an increased focus on centrally managed information technology infrastructure resources, the integration and modernization of NIST administrative information systems, 5 U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, ITL Strategic Plan January 2000, National Institute of Standards and Technology, Gaithersburg, Md., 2000.
OCR for page 245
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 and increased desktop support activities. The panel commends ITL both on its strategic plan and on conducting an inclusive planning process. The panel is somewhat concerned about what seems to be an artificial distinction between the services provided by the Distributed Computing and Information Services Division and those performed by the High Performance Systems and Services Division (e.g., support of desktop machines versus support of the networks that these machines are on). The panel suggests that reviewing and clearly defining the missions and responsibilities of each division might provide a more coherent and less confusing environment for end users of ITL's support services. If the support staff are to remain separate, then the service functions and missions of these two divisions should be better coordinated. Technical Merit and Appropriateness of Work The Distributed Computing and Information Services Division is involved in several activities to reduce ongoing costs and hence improve the effectiveness and efficiency of the overall NIST computing infrastructure. Services such as common centralized e-mail, shared calendars, centrally coordinated Web pages, help-desk support, file servers, and site-licensed software are common practices throughout the IT community, and the division does a good job of coordinating them with a low ratio of staff to customers. For example, the division successfully completed an array of Y2K remediation and support projects for all of NIST. However, as noted last year, NIST still seems to be without a shared, institution-wide vision about which services should be provided centrally and which should be provided locally in the other operating units throughout NIST. In meetings with users from seven NIST laboratories, the panel learned of several examples of independent contracts for outside commercial support services. In addition, some operating units such as the Manufacturing Engineering Laboratory manage and run their own support organizations. The issues related to defining an information technology support architecture for NIST are discussed in the laboratory-level overview, and the panel reiterates its previous insistence on first determining and then communicating the appropriate long-term balance between centrally and departmentally supported laboratory-wide IT services. Efforts have begun on this task, and the panel urges that the activity continue to be given a high priority and that the Distributed Computing and Information Services Division be involved in these efforts. Impact of Programs The Distributed Computing and Information Services Division provides a wide range of important support services to staff throughout NIST. Customers indicate through the automated customer service satisfaction survey that these services are well received, and the division is working on ways of involving users in additional service-related measurements. The panel believes that the division might be able to increase its NIST-wide impact by engaging in high-level, strategic discussions with staff from other laboratories. Collaborative planning requires partnerships at all levels, from senior management to technical end users. In meetings with the users, the panel learned that such consultations had not occurred recently but would be received enthusiastically. Potential topics for these consultations include the exchange of information about emerging technology needs of researchers and about new IT capabilities, discussion of how to establish metrics for support, and general confirmation of the significance of the support units. It is difficult for a service organization to gain respect as a valued partner in an environment in which one party gives orders and the other takes them, with little discussion of long-term goals and possible alternative solutions.
OCR for page 246
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 Because the Distributed Computing and Information Services Division focuses on serving the information technology needs of NIST staff, its programs impact industry only indirectly. However, if the division began to work on the development of practical performance metrics for support organizations, the results would not only be consistent with its service mission within NIST but could also be of direct benefit to industry. There is a significant lack of meaningful and practical measurements available to organizations wishing to assess the performance and value of their IT support organizations. Distributed Computing and Information Services Division staff could attempt to develop metrics that assess information technology “as seen by the user.” Such measures would need to be systems-level measures, meshing evaluations of hardware, networks, and software into a single metric that provides accountability for service. This is not an easy task, but it is not impossible. Boeing, for example, has end-to-end availability measures, and Microsoft has developed internal metrics as well. Although a great deal of additional development is needed in this area, work would not have to start from scratch. There is also the potential for significant formal research collaborations between service personnel and the ITL research and development staff on metrics. Support-level research could best be done by the service people with assistance from staff who have experience with developing metrics in difficult environments. With this type of collaboration, the Distributed Computing and Information Services Division could experiment with different approaches to information technology metrics in the support environment and could make a significant contribution to NIST and industry at large by developing, using, and adapting innovative solutions. Division Resources Funding sources for the Distributed Computing and Information Services Division are shown in Table 8.7. As of January 2000, staffing for the Distributed Computing and Information Services Division included 77 full-time permanent positions, of which 71 were for technical professionals. There were also eight nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. TABLE 8.7 Sources of Funding for the Distributed Computing and Information Services Division (in millions of dollars), FY 1997 to FY 2000 Source of Funding Fiscal Year 1997 (actual) Fiscal Year 1998 (actual) Fiscal Year 1999 (actual) Fiscal Year 2000 (estimated) NIST-STRS, excluding Competence 0.4 0.6 0.6 0.7 STRS-supercomputing 0.9 0.9 0.9 0.8 OA/NFG/CRADA 0.2 0.1 0.0 0.0 Other Reimbursable 0.5 0.6 0.4 0.3 Agency Overhead 5.7 6.3 7.1 7.1 Total 7.7 8.5 9.0 8.9 Full-time permanent staff (total)a 66 62 72 77 NOTE: Sources of funding are as described in the note accompanying Table 8.1. a The number of full-time permanent staff is as of January of that fiscal year.
OCR for page 247
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 The ratio of information technology service personnel to all personnel supported is quite low (the Gaithersburg campus houses approximately 3000 people). Between January 1999 and January 2000, the division added six full-time technical professionals, but the total budget for the division dropped slightly over the same period. Recruiting and retention programs are working well for the division. The new division chief reports that the turnover rate for the past year was approximately 5 percent, a low figure for an IT support organization in a major metropolitan area. During the fall of 1999, division staff conducted a desktop support benchmarking study with personnel at the National Institutes of Health. This study identified several best practices that the division plans to examine in more depth to understand how they might be applied in the NIST environment. In addition, the Personal Computer Support Group performed a Baldrige self-assessment to identify future training needs and to investigate ways of improving the effectiveness of internal procedures. The panel applauds these efforts by the division to make the best possible use of its resources. Software Diagnostics and Conformance Testing Division Division Mission According to division documentation, the mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, conformance to standards, and correctness; to participate with industry in the development of forward-looking standards; and to lead efforts for conformance testing, even at the early development stage of standards. The division's work on designing conformance and diagnostic tests and developing reference implementations with participation from industry clearly fulfills its mission and is consistent with the goals expressed in both the laboratory and the NIST missions. Technical Merit and Appropriateness of Work The Software Diagnostics and Conformance Testing Division is providing industry with necessary technical leadership and is developing reference implementations and conformance test suites. The continued focus on new technologies and on maintaining close partnerships with industry ensures that division programs have a significant impact. To support standards work, the division performs research in areas that improve conformance testing. Research on how to develop an automatic generator of tests from formal specifications is proceeding well; recent accomplishments include an improved test minimizer tool and a new coverage measure and report tool. This research is of significant value because the generated tests would be inexpensive to develop and, by definition, would be consistent with the specifications. The joint project with the Statistical Engineering Division to investigate whether statistical analysis can ensure that software conforms to a specification seems to be well under way, after a hiatus last year. This work remains both intriguing and potentially valuable, and the panel is pleased to see resources applied to it by both divisions and by other NIST Laboratories (Manufacturing Engineering, Electronics and Electrical Engineering, and Chemical Science and Technology). Over the past year, significant progress has been made on the project that aims to develop a repository of error, fault, and failure data; the public repository system has been set up, and eight projects are already in this system (nine more are due to be installed soon). Companies can use the data and NIST analysis of the data to improve their software development processes and the reliability of their products. Thus, there is a strong need to capture such data and make them available in the public domain. Newly obtained data will be installed outside the NIST firewall shortly, and more than 1000 users from companies and
OCR for page 248
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 agencies such as Boeing, Johnson Space Center, and Viacom have already downloaded the repository management tool. To maximize the leverage of NIST work, the division aims to influence standards development in the early stages; therefore researchers focus on studying emerging technologies where standards work will be needed but has not yet begun. An example of this approach can be seen in the project investigating ADLs for component-based software development. This research is definitely needed: industry and academia are producing good work on ADLs, but the expressive and analytical capabilities vary among the ADLs because each has been developed to meet a different need. The planning and documentation methods used by the Software Diagnostics and Conformance Testing Division could serve as a model for other ITL divisions. The process has used clearly identified criteria to select state-of-the-art programs with clearly defined priorities and goals, well-identified roles, and measurable contributions to national and international standards organizations. Examples include the work on conformance tests for XML, for distributed interactive learning systems, and for the interactive DASE in collaboration with the High Performance Systems and Services Division. Directions can change quickly within the software industry. The division carefully monitors these changes and acts appropriately. Some projects have been downsized. Very limited resources will be applied to work on the next generation of the VRML, and both the Real-Time Java project and the work on role-based access control (RBAC) satisfied their completion criteria and were concluded. The software copymarking project was also terminated, eliminating the concerns in last year 's report regarding its nondisclosure restrictions. As old projects end, the division can expand into new areas. The ITL has initiated a laboratory-wide program on pervasive computing. The panel expects that the Aroma/Air Java project, this division's contribution to the pervasive computing effort, will provide needed diagnostic tools and serve as a nucleus for software standardization. Impact of Programs The Software Diagnostics and Conformance Testing Division's programs continue to have a positive impact in many ways. To disseminate information about technologies developed at NIST, division staff publish reports in external journals and books, write internal (but publicly available) NIST reports, and give presentations at conferences, often at industry's request. In fiscal year 1999, the total outputs for the division were 53 external publications, 14 internal documents, and 52 presentations. The RBAC project produced four patents (two issued and two pending). This work was completed last year, and the technology developed is now being utilized in the division's Healthcare Information Systems project, which is supported by the Department of Veterans Affairs (VA) and the NIST Advanced Technology Program. A remote procedure call broker has been developed and is in use throughout the VA system, and work is under way on an RBAC authentication mechanism for similar deployment. In addition, the specification for an authorization framework for distributed health care environments, which was developed as part of this project, has been adopted by the Object Management Group and is being implemented by industry partners. The laboratory provides well-recognized benefits to standards consortia, including unbiased conformance tests, reference implementations, and recommendations on how to improve standards to make them more precise and more generic. For example, NIST is very active on IEEE, Department of Defense, and other standards committees that support electronic distributed learning, an area with the potential to supply the next big wave of Internet-based applications. Developers of instructional management systems (IMS) have already recognized NIST as an important contributor to the successful development of the first version of specifications. Another vital NIST role is providing a neutral setting
OCR for page 249
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 TABLE 8.8 Sources of Funding for the Software Diagnostics and Conformance Testing Division (in millions of dollars), FY 1997 to FY 2000 Source of Funding Fiscal Year 1997 (actual) Fiscal Year 1998 (actual) Fiscal Year 1999 (actual) Fiscal Year 2000 (estimated) NIST-STRS, excluding Competence 5.2 4.6 4.8 4.7 Competence 0.0 0.5 0.6 0.5 ATP 0.3 0.4 0.4 0.4 OA/NFG/CRADA 1.5 2.2 0.6 1.0 Other Reimbursable 0.4 0.2 0.0 0.0 Total 7.4 7.9 6.4 6.6 Full-time permanent staff (total)a 54 41 39 37 NOTE: Sources of funding are as described in the note accompanying Table 8.1. a The number of full-time permanent staff is as of January of that fiscal year. for discussing conformance testing issues. Examples of this division 's skill at this task can be seen in its successful leadership (as team chair) on two important standards committees: the DASE Conformance Working Group and the Organization for the Advancement of Structured Information Standards (OASIS) XML Conformance Subcommittee. Industry has provided many testimonials to the great value of the division 's standards work, including “The OASIS-NIST XML Conformance test suite is critical for our industry ” (Norbert Mikula, chief technical officer, OASIS); “NIST made strong contributions (X3D) and resolved our knottiest problems ” (Don Brutzman, board of directors, Web3D Consortium); and “The high quality test suite and certification program is an invaluable resource for ATA [Air Transport Association]” (Robert Peel, Director of Airworthiness and Standards, ATA). Division Resources Funding sources for the Software Diagnostics and Conformance Testing Division are shown in Table 8.8. As of January 2000, staffing for the Software Diagnostics and Conformance Testing Division included 37 full-time permanent positions, of which 33 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. At this time, the division does not have any problems with equipment or resources. Statistical Engineering Division Division Mission According to division documentation, the mission of the Statistical Engineering Division is to contribute to research in information technology, to catalyze scientific and industrial experimentation, and to improve communication of research results by working collaboratively with, and developing effective statistical methods for, NIST scientists and its partners in industry.
OCR for page 250
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 The division continues to investigate an impressive array of problems and to be involved in a wide range of activities. The majority of the division's projects involve collaborations with scientists and engineers across all of NIST. This breadth represents a subtle but noteworthy shift in priorities from recent years. The emphasis is now closer to the historic and natural focus of the division: seeking out the best opportunities for impact, wherever they are found, rather than attempting to concentrate on problems within the ITL domain. Technical Merit and Appropriateness of Work Statistical Engineering Division personnel apply state-of-the-art statistical techniques to the projects undertaken by staff members throughout NIST laboratories. When existing techniques are inadequate, division staff develop new approaches. Since the need for statistical support is pervasive across NIST, this mode of operation is not just appropriate but absolutely essential for the institute to fulfill its measurement and standards mission. Below, the panel briefly describes several of the division's projects to illustrate the range of areas covered and the types of projects undertaken: The Effect of Query Choice on Information Retrieval: This work develops graphical analysis methods that enable assessment of the effects and interactions of query topics, query formulations, and searching systems in information retrieval problems. The approach is built on sophisticated statistical methods, including multidimensional scaling. Increased understanding of the effects of varying query formulation can help improve the relevance of documents returned from a search of the World Wide Web. The project is in collaboration with the Information Access and User Interfaces Division of the ITL. Statistical Modeling for Polymer Temperature and Pressure Measurement: The goal of this project is to relate the fluorescence of a dye that is added to a polymer during processing to the temperature and pressure of the material. The relationship is nicely systematic but appears to be quite complicated to describe mathematically. By taking a very creative approach, a model of feasible complexity has been developed that accurately represents the phenomenon under study. This project is in collaboration with the Polymers Division of the Materials Science and Engineering Laboratory. Statistical Measure of Image Sharpness for Scanning Electron Microscopes (SEMs): A practical and effective numerical measure of image sharpness was developed using a bivariate measure of kurtosis computed from the two-dimensional Fourier transform of an SEM image. An automated system using this algorithm has been implemented by industry to improve performance of SEMs used in semiconductor wafer production lines. This work was done in collaboration with the Precision Engineering Division of the Manufacturing Engineering Laboratory. Magnetic Trapping of Ultracold Neutrons and Determination of the Mean Lifetime of the Neutron: This collaboration supports a world-class experiment in fundamental physics that will provide essential data for astrophysical theories relating to the Big Bang. The project involves researchers from the Ionizing Radiation Division of the NIST Physics Laboratory and from Harvard University, Los Alamos National Laboratory, and the Hahn-Meitner-Institut in Berlin. The most recent statistical contribution has been to develop improved models and estimation methods for estimating the half-life of the neutron. The statistical modeling and analysis methods are also guiding refinements to the experiment design as the project progresses. Statistical Models of Fourier Transform/Ultraviolet Spectral Estimation: This project is aimed at developing statistical models and algorithms to improve NIST's chemical measurement services in the area of Fourier transform spectrometry in the ultraviolet range. The improvements are based on
OCR for page 251
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 more accurate models of the measurement system responses, plus development of efficient computational algorithms for statistical fitting and estimation. This work is in collaboration with the Analytical Chemistry Division of the Chemical Science and Technology Laboratory and with the High Performance Systems and Service Division of the ITL. In addition to the above projects, the division is also in the enviable position of being associated with three Competence projects (cross-laboratory collaborations funded by the NIST director to build expertise in a new area). One of these projects is the work on Bayesian metrology. The panel believes that there is considerable unrealized potential in this effort and that the division should leverage its efforts by tapping the expertise of outside researchers, perhaps through formal collaborations, workshops, or visiting appointments. One of the goals should be to facilitate interactions between statisticians and metrologists. The division is working on a strategic plan that should help put its current efforts into a larger context and identify new opportunities and issues. The panel was informed that the plan will include new research areas such as information technology, biotechnology, combinatorial chemistry, and methods for dealing with massive and new forms of data. The plan will also recognize the importance of continually upgrading the division's capabilities, both by increasing computing power and by expanding the skills and expertise of staff. Currently, the division is looking to fill two vacant technical staff positions; recruiting people with the capabilities needed to pursue new research directions will be an important step toward preparing the division to meet future challenges. The panel supports the strategic planning effort and urges that completion of the plan be a high priority for division management. Impact of Programs As illustrated above, the Statistical Engineering Division engages in a myriad of interactions and has substantial impact on various projects across NIST's laboratories. For example, the division's work with the Manufacturing Engineering Laboratory on an image sharpness monitor for SEMs was recognized with the Department of Commerce Silver Medal award. The division also continually seeks out new opportunities. Staff efforts to build new relationships with personnel in other NIST laboratories, although successful, are seriously impeded by the division's relative isolation in NIST North. The panel strongly urges that the Statistical Engineering Division be relocated to the main campus. The contributions of this division are such that it should be as physically close to the center of gravity of NIST activities as possible in order to promote easy and spontaneous interactions. Lowering the cost (in time and effort) of collaborations and reducing the number of missed opportunities would easily justify a move. Beyond collaborative work on specific projects, division personnel naturally provide general statistical education to NIST staff. During the past year, the division offered NIST scientists a tutorial lecture series on analysis of variance and an oversubscribed course on practical lessons in statistical uncertainty analysis. Division personnel also gave about 20 invited lectures to external audiences. In a similar spirit, the division ran three workshops specifically aimed at industrial researchers; the topics were methods of uncertainty analysis, advanced mass measurements, and uncertainty calculations in chemical measurements. Division Resources Funding sources for the Statistical Engineering Division are shown in Table 8.9. As of January 2000, staffing for the Statistical Engineering Division included 19 full-time permanent positions, of
OCR for page 252
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 TABLE 8.9 Sources of Funding for the Statistical Engineering Division (in millions of dollars), FY 1997 to FY 2000 Source of Funding Fiscal Year 1997 (actual) Fiscal Year 1998 (actual) Fiscal Year 1999 (actual) Fiscal Year 2000 (estimated) NIST-STRS, excluding Competence 2.6 2.8 2.9 2.5 Competence 0.1 0.3 0.5 0.6 STRS-supercomputing 0.5 0.5 0.1 0.0 Measurement Services (SRM production) 0.0 0.0 0.0 0.2 OA/NFG/CRADA 0.0 0.0 0.1 0.1 Other Reimbursable 0.1 0.0 0.0 0.0 Total 3.3 3.6 3.6 3.4 Full-time permanent staff (total)a 30 21 23 19 NOTE: Sources of funding are as described in the note accompanying Table 8.1. a The number of full-time permanent staff is as of January of that fiscal year. which 17 were for technical professionals. There were also seven nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. The division staff also includes four people who have joint appointments at NIST and as faculty members of universities (two in Boulder, two in Gaithersburg). The panel continues to strongly support such arrangements because the contributions of these people are invaluable for the work of the division. Given the small size of the Boulder contingent (five permanent NIST employees), the division relies particularly heavily on faculty members, and their services are essential to support NIST researchers at the Boulder facility. The technical staff in the Statistical Engineering Division publish regularly and serve on a variety of committees. In 1999, the division produced 29 publications and technical reports, and personnel served on six standards committees and four professional society committees. In addition, one staff member is on the editorial boards of the Journal of Applied Statistics and the Journal on Total Quality Management, and two other staff members were guest editors for a special issue of the International Journal of Imaging Science and Technology. Another staff member was recently elected as a fellow of the American Statistical Society (ASA), bringing the total number of ASA fellows in the division to five. The division appears to have ample resources at its disposal to increase its outreach into the statistics community, hire postdoctoral research associates, engage in workshops, and otherwise fulfill its mission. At the Boulder facility, there are some issues involving office space, equipment, and courses; these problems deserve attention so that staff members there are supported on a par with their colleagues in Gaithersburg. Selection of a new division chief must be one of the highest-priority items. In the current job market, this may not be an easy position to fill. The panel recommends the selection of a strong technical visionary who is well known in the statistics community—someone who can help to hone, communicate, and realize a vision for the division that will increase its status and impact. The ITL director may find it helpful to form an external advisory committee to help locate viable candidates for this position. The panel encourages the division not to wait for a new division chief to come on board and to take advantage of strong support from the new ITL director to execute its internal mission and increase its
OCR for page 253
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES: Fiscal Year 2000 involvement with the external statistics research community in ways that advance its mission. Given the wonderful array of challenging problems to which the division can contribute—an array that would be coveted by almost any statistics group—there is every reason to believe that the future could and should be extremely bright. MAJOR OBSERVATIONS The panel presents the following major observations: The panel appreciates the openness and responsiveness of ITL management and staff to its requests and suggestions. The planning process within ITL continues to improve. This year, the panel was particularly pleased by the laboratory-wide strategic plan and the increased emphasis on criteria for beginning and ending projects and the definition of specific goals. The panel is very concerned about the constant or, in some cases, decreasing staffing levels within ITL. This situation is due in large part to the limited funding available to cover salaries. The panel observes that the situation has two consequences: the first is that ITL personnel are spread very thin and some projects have only single-point coverage; the second is that many areas exist in which NIST participation could make a significant difference but ITL cannot afford to be active at this time. The new initiatives in computer security represent an exciting opportunity for the ITL. However, care must be taken to ensure that the scope of the programs is aligned with the somewhat limited allotment of funding and the number of personnel available. Since topics in information technology increasingly cross the laboratory's divisional boundaries, the panel believes that there has to be more emphasis on interdisciplinary collaboration. Although there are no explicit barriers to cross-divisional work, there seems to be little encouragement or support for such projects. A good example of an area in which cross-divisional cooperation must take place is the Pervasive Computing Initiative, which touches almost every division in some way. The panel believes that in such situations, there must be formal coordination among the divisions to ensure that every division that should participate is doing so, that efforts neither conflict nor are duplicated, and that potential synergies are effectively exploited. The Statistical Engineering Division plays a fundamental role in the execution of the NIST mission throughout the Measurement and Standards Laboratories. To ensure that this group can continue to meet the diverse statistical needs of NIST activities and contribute to the national statistics community, it is important for ITL management to reinforce the message that the work of the division is valued and to identify and hire a strong leader for the division as soon as possible. In addition, both effectiveness and morale would be improved if the division could be moved from NIST North to the main campus. Staff and projects in the Mathematical and Computational Sciences Division would also benefit from such a relocation. Provision of information technology support to NIST continues to be an issue. The panel applauds the latest efforts to define a coherent system for services. Some elements, such as a user's forum and input from current support staff, could be added to this effort to increase its chance of success.
Representative terms from entire chapter: