The report of the Board on Assessment of NIST Programs is one part of NIST's overall process of planning and performance evaluation. Input into long-term strategic planning comes from sources such as interaction with industry, industry “roadmaps,” science and technology conferences and workshops, strategic planning studies, and priorities expressed by the Administration and legislated by Congress. Performance evaluation has internal and external components. Each laboratory tracks quantitative output metrics such as the number of research papers published, the number of invited talks given by staff, the number and prestige of national awards given to staff, the number of Standard Reference Materials (SRMs) provided to industrial customers, and so on. The NIST Office of Economic Analysis conducts retrospective studies to estimate a rate of return on investment for selected successful NIST programs. The Visiting Committee on Advanced Technology, a federal advisory committee mandated in the NIST charter, provides a review of overall Institute strategy. Customer feedback provides a qualitative indicator of program impact. The Board on Assessment of NIST Programs provides NIST with its only independent external review of the Measurement and Standards Laboratories by technical experts and focuses on the technical quality and relevance of the ongoing programs at the laboratories.
The process used by the Board and its panels involves review of a significant sample of the ongoing technical programs in the Measurement and Standards Laboratories. It also includes checks for the existence of criteria and procedures for program selection and “sunsetting, ” maintenance of staff skills, and other important aspects of a high-quality research program to ensure that the programs reviewed are representative of the outcome of an orderly, coherent planning process. The Board oversees seven panels, each corresponding to one of the seven Measurement and Standards Laboratories at NIST. These panels are organized into subgroups corresponding to the divisions of each laboratory. A subgroup visits each division for a 1- to 2-day meeting. The purpose of these meetings is to interact with bench-level scientists and engineers to review their ongoing research and technical service work and to meet with division management to discuss the division's plans, resources, and technical outreach and dissemination. After these subgroup meetings, each panel meets for a 2- to 3-day period. The panels receive briefings from the management of each laboratory on the laboratory's major accomplishments of the past year, its mission, plans, resources, outreach, and other relevant topics. The panel meeting provides an opportunity for additional technical interaction with NIST staff and for discussion with NIST management. Based on the findings of its ad hoc subgroups and on additional overview information obtained at the panel meeting, each panel writes a report on the state of the laboratory it has reviewed. The Board then meets to consider the reports of the panels and to discuss overarching themes and opportunities with NIST managers to obtain information for its own report.
This chapter represents the results of the Board's consideration of the overall state of the NIST Measurement and Standards Laboratories. It is organized according to four topics: Technical Merit, Appropriateness of Programs, Impact of Programs, and Resources. Subsequent chapters of this volume contain the reports of each panel reporting to the Board.
The technical merit and the appropriateness of the work in all of the laboratories continue to be very high. The staff represents one of the world's finest assemblages of scientific and
engineering expertise. The breadth of research programs and laboratory expertise provides leading-edge information and services to both emerging and mature industries. Quantitative measures, such as the number of papers published in peer-reviewed journals, the number of invited talks given by staff members, and the number of awards received, continue to demonstrate that NIST staff and their work are respected and valued by the global technical community. Of particular note is the award of a 1998 National Medal of Science, the nation 's highest scientific honor, to John W. Cahn of the Materials Science and Engineering Laboratory (MSEL). The award cited “his profound influence on the course of materials and mathematics research, and his immense impact on three generations of materials scientists, solid-state physicists and mathematicians.”
In the spring of 1998, NIST initiated a series of internal evaluation workshops titled “Best in the World.” Each workshop focuses on a specific area of measurement or technology (for example, temperature, time and frequency, capacitance, force) and seeks to assess NIST's capability in that area relative to other international organizations and researchers. This activity has allowed NIST to ascertain areas in which it is the world leader, is among the best in the world, is at the state of the art, or is below state of the art. In addition to these NIST-wide evaluations, some major laboratories have engaged in their own benchmarking activities. For example, in 1998 the Chemical Science and Technology Laboratory (CSTL) held workshops to benchmark the SI (International System of Units) base units of temperature (the Kelvin) and amount of substance (the mole). Future workshops will focus on several SI-derived units, including pressure, humidity, and flow rate. These evaluations have shown that NIST is best in class or state of the art in most areas examined. The results are being used by NIST to determine whether its efforts are sufficient for anticipated U.S. industry needs in those areas where it is not state of the art or whether more resources must be devoted to improving performance in critical areas. The Board commends NIST's engagement in benchmarking activities to demonstrate and assure the world-class nature of its research and services. These activities are a vital part of maintaining the high technical quality of programs.
APPROPRIATENESS OF PROGRAMS
NIST's primary mission is to promote U.S. economic growth by working with industry to develop and apply technology, measurements, and standards. 1 The Board and panels found that the laboratories' ongoing programs continue overall to be effectively directed toward NIST's mission to industry. Projects and programs examined by the Board and panels in general represented appropriate and judicious use of NIST resources.
To help assure mission focus, each laboratory has in place a process for program and project selection, review, and sunsetting. The criteria for choosing programs and projects, as judged by the resulting portfolio of programs, are good. Projects are being terminated when they are mature, complete, or deemed to be of lower priority. For example, the Manufacturing Engineering Laboratory transitioned the Secretariat of the now-mature Standard for the Exchange of Product Data away from NIST. Opportunities for research in new venues are identified and undertaken. The Physics Laboratory identified the opportunity to perform
U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Guide to NIST, NIST Special Publication 858, National Institute of Standards and Technology, Gaithersburg, Md., July 1998.
absolute radiometric measurement of space-based remote-sensing instruments by placing calibrated sources on the International Space Station. These are only two of many possible examples of good results obtained from the laboratories' prioritization processes and criteria.
The maturity of these selection processes varies from laboratory to laboratory. In some laboratories, it is clear that the selection criteria are well understood and used by all levels of technical staff. In other laboratories, the criteria are not universally applied or have not been communicated effectively to staff. However, since the previous assessment, all of the major laboratories showed improvement in their use of an effective prioritization process.
In determining industry need for a measurement, standard, service, or measurement technology, the laboratories make use of technology roadmaps produced by various sectors, such as the Semiconductor Industry Association's (SIA's) National Technology Roadmap for Semiconductors (NTRS). Where such roadmaps do not exist, the laboratories often use workshops and other meetings with industrial representatives to forecast future measurement and standards needs. The Board commends such activities and recommends an even more proactive role for NIST as a facilitator of activities to define future industry needs in metrology. NIST can catalyze the development of roadmaps or other technology forecasts by industries that have not yet organized to produce such documents.
IMPACT OF PROGRAMS
Project selection criteria that consider industry need for a measurement, standard, or measurement technology have helped assure the impact of the laboratories' programs. An increased use of technology roadmaps by the laboratories is apparent, as are efforts to develop such guidance where none exists. It is also clear that NIST's priorities have allowed for movement into areas having an impact on emerging industries such as biotechnology and information technology.
Many of the biggest technical challenges facing industry are interdisciplinary in nature, and so to achieve maximum impact NIST must also ensure that its programs can cross disciplinary and laboratory boundaries when appropriate. The benefits of such collaboration between laboratories include selection of the best tools available to apply to a problem, proper application of technical expertise, and the most efficient use of available resources. In some instances, the laboratories have very clearly embraced intra- and interlaboratory collaboration. For example, the National Advanced Manufacturing Testbed, led by the Manufacturing Engineering Laboratory (MEL), embraces collaboration with each of the other major laboratories. There are instances, however, where laboratories do not appear to be taking sufficient advantage of the opportunities for collaboration. For example, existing and proposed efforts in optical technology should involve collaboration between the Physics Laboratory, the Electronics and Electrical Engineering Laboratory (EEEL), the MSEL, the CSTL, and the Information Technology Laboratory (ITL), but such interaction was not evident to the panels. Similarly, ITL, MEL, and the Building and Fire Research Laboratory (BFRL) all have programs developing open systems architectures for their customers, but the laboratories do not appear to extract the potential benefits from each other's efforts. Achieving the greatest possible impact from NIST programs requires recognizing and capitalizing on all such opportunities for collaborative efforts.
One NIST activity that is of significant benefit to industry is its participation in international standards activities. Many NIST scientists and engineers participate in national and international standards-setting committees, as participants, advisors, and chairs. The economic impact of such work, which facilitates product compatibility and acceptance of U.S. products in foreign markets, is difficult to measure but is surely large. While recognizing that the U.S. system of participation in standards activities is driven by the private sector and relies on the donation by private companies of their employees' time and expenses, the Board encourages NIST to take an even more proactive leadership role in facilitating such activities. Where U.S. leadership (or even participation) in important standards activities is lacking, NIST can serve as a catalyst for building interest in the private sector and increasing its participation in standards-setting efforts.
In its 1998 report, the Board noted that the laboratories use a variety of measures of the quality and quantity of program output and that identifying and implementing best practices in the measurement of impact might make these measures more comprehensive and consistent and increase their usefulness. In 1999, the Board and panels saw increased effort in the area of measuring impact and recommend a continued focus on progress in this area. The Board recommends that the practice of defining the criteria for success at the time of program or project implementation become more universally applied in the laboratories.
Program impact is, of course, dependent on the successful dissemination of program results. NIST researchers are very successful at reaching their peers in the technical community through publications, talks at professional meetings, and other traditional scientific and engineering means of communication.
NIST has also continued to increase and improve its use of the World Wide Web to disseminate data, information on services, and other technical results. This is an excellent dissemination tool, allowing NIST to reach a potentially much more diverse audience. Recent innovations at NIST have included the development of the capacity to perform certain calibrations and measurement comparisons remotely via the Internet. MEL's SIMnet (Inter-American System of Metrology) Program, for example, has established the capacity to perform real-time measurement intercomparisons between national laboratories of Organization of American States (OAS) member countries. Use of the Web by the NIST laboratories is generally good and appropriate. NIST has made progress in ensuring that Web pages are up-to-date. Continued attention to maintenance of pages is required. The laboratories are also learning how to extract useful information about their customer base and the utility of their products from information on hits to the NIST Web sites.
In its 1998 report, the Board noted that the NIST name is not as well recognized as it should be. The Board encouraged NIST to do more to increase its recognition with the outside community, as an aid to disseminating its work, increasing its impact, and broadening the pool of potential recruits for positions. NIST leadership appears to be aware of this untapped opportunity and is discussing plans to increase the visibility of the Institute. The Board encourages such efforts.
To illustrate the impact possible from NIST programs, the Board has chosen to highlight two examples, DNA Measurements and Standards for Forensic and Clinical Laboratories, and Cybernetic Building Systems. Many other high-quality programs with solid impact could be used as examples. The Board chose to focus on these two because they represent technology development that is now essentially complete, with the technology having been transferred from NIST to industry; they illustrate a sustained program rather than a one-time effort; they illustrate
the importance of interdisciplinary technology to industrial customers; they show a clear stakeholder impact; and they demonstrate clear international technical leadership.
DNA Measurements and Standards for Forensic Laboratories
The use of DNA testing in the prosecution of criminals began in the 1980s; the first conviction on the basis of DNA evidence occurred in 1986. Analysis was originally performed only by independent commercial laboratories. Because no rigorous and universally accepted standard for these measurements was available, the soundness of early DNA evidence was questioned by many experts.
In 1988, the Federal Bureau of Investigation (FBI) brought together representatives of commercial laboratories, state crime laboratories, and other relevant parties to form a technical working group (TWG) on DNA analysis methods. Recognizing NIST competence in the development of measurement protocols and standards, the FBI invited CSTL's Biotechnology Division to be one of the founding members of the TWG. NIST assumed the role of developing quality assurance standards for those DNA analysis technologies that were being adopted by the TWG.
The first NIST DNA Profiling Standard was released in 1992. Standard Reference Material (SRM) 2390 was recognized with an R&D 100 Award as one of that year's top 100 technological advances. Shortly after its release, three organizations (Collaborative Testing Services, Cellmark Diagnostics, and the College of American Pathologists) established commercial proficiency testing programs. Commercial DNA testing laboratories could now have their testing procedures certified according to an accepted scientific standard, greatly enhancing the reliability of such evidence and its acceptance in the courts.
Since that first SRM, CSTL, in cooperation with EEEL's Office of Law Enforcement Standards, has continued to provide standards for new DNA profiling techniques and other new methods of forensic analysis. SRMs were developed for polymerase chain reaction (PCR) analysis and for PCR analysis of short tandem repeat (STR) as used on degraded DNA samples. The laboratory is currently working on an SRM for mitochondrial DNA sequencing, another emerging forensic tool. CSTL staff members continue to participate in TWGs on emerging technologies, such as DNA sequencing chips. NIST is responding to industry's need for new standards and protocols as industry develops and adapts new techniques for analysis. The availability of NIST SRMs has reduced forensic testing costs while improving the tests' accuracy and providing a legally defensible traceability for this measurement system nationwide. CSTL is currently engaged in an economic impact evaluation to estimate the magnitude of the economic benefit these SRMs have returned.
Cybernetic Building Systems
As computer controls of building systems, such as heating, ventilation, and air-conditioning (HVAC) systems, became more common in the 1970s and 1980s, manufacturers developed proprietary control systems for their equipment. Equipment from different manufacturers could not cross-communicate with each other. To ensure interoperability,
building owners and engineers became locked into buying products from one firm, rather than basing purchases on reliability, performance, and efficiency.
The BFRL staff recognized the need for a standard communication protocol for building control systems in the early 1980s but found no industry acceptance at that time. By 1987, a facilities manager at Cornell University, plagued by the problems that multiple building control communications systems posed in managing a full campus of buildings, convinced the American Society of Heating, Refrigerating, and Air-conditioning Engineers (ASHRAE) to initiate a national standards committee on building control systems. Now that the customer had recognized the problem, BFRL stepped forward to become part of the solution.
BFRL played several roles in the ASHRAE committee and its standards development activities. In addition to carrying out its own technology development, BFRL facilitated interaction between the various equipment manufacturers. This is illustrated by BFRL's development of a “virtual test shell” for the emerging control standard, which allowed BFRL to work one-on-one with equipment manufacturers to test their ideas about the communication protocol and assure its technical soundness before implementation. BFRL developed testing tools and software based on the proposed standard, while manufacturers devised candidate implementations. The NIST Building Automation and Controls Network (BACnet) Interoperability Testing Consortium was established in 1993. Participants included manufacturers that collectively represent the dominant share of the HVAC control market in North America and a majority of the world market. Manufacturers of life-safety systems and lighting control systems were also represented. Manufacturers sent their implementations to NIST for testing on the BFRL testbed. NIST provided a neutral venue for testing, allowing competitors to bring their prototypes together for a noncompetitive, nonjudgmental evaluation. The results of this testing allowed further technical refinement of the standard, and companies participating in the consortium benefited from the advance understanding of the standard that they developed. The resulting BACnet communication protocol became an ASHRAE standard in 1995. It is currently in the prestandard phase of consideration by the European Union. It has been implemented in about 5,000 buildings worldwide on all seven continents.
The successful development and implementation of the BACnet protocol provides an enabling technology that now allows BFRL to work with industry partners to integrate building control systems in such a way as to increase building efficiency, safety, and security. This is the concept driving current BFRL work on cybernetic building systems. Building emulator testing technology, developed in the mid-1980s and including the development of a “virtual cybernetic building” in France, is synthesized with BACnet and BFRL expertise in fault detection in building control systems. The resulting testbed will provide a means of examining the performance of various building control systems and their interactions under both normal and adverse (including emergency) situations. Unanticipated feedback loops and failure modes of building systems can be detected and addressed without the need to subject an actual building to adverse conditions. Researchers and service providers could use the testbed to study system interactions, evaluate control algorithms, test new systems and services, and develop standards for integrated systems that will enhance safety, reliability, system performance, and building efficiency.
The Board is pleased to note that the quality of the staff that NIST is able to attract and retain remains very high. Morale among the staff is good and is reflected in a good retention rate in most areas. More attention must be paid to retention of technician support, particularly in BFRL and ITL. The loss of a highly skilled technician can mean the loss of expertise and experience that are difficult if not impossible to replace, and good technical support staff can be key to retaining technical professional staff. The management and compensation of technicians should reflect the fact that experienced technicians can be as valuable to a program as PhD scientists or engineers. The Board notes that those areas where morale and retention are highest among technical support staff, often in spite of salaries that are not competitive with the private sector, are those areas in which the support staff feels respected and valued for knowledge and capabilities. A good example is the machine shop staff at JILA. Machinists frequently team with scientists in design of equipment, providing valuable input that helps experiments succeed in the shortest time with the least expense. The machinists report feeling respected for their abilities and find their work interesting and challenging. Scientific staff members obviously respect their capabilities, praise their abilities without prompting, and value them as team members.
The question of succession planning and employee cross-training continues to be a challenge for NIST. This is critical in three circumstances: in cases in which critical expertise lies with a single expert, in the development of leadership and management skills within staff, and in cases in which skill refreshment is needed to move an employee to a higher-priority program. Since the previous assessment, NIST leadership has established a management training program, the Leadership and Management Development Pilot project. This pilot training program features three training series: Performance Management, Effective Communications, and Strategic Leadership at NIST. Each series involves approximately 30 hours of training. The Board is pleased to see action being taken on this issue and will look for its impact on recruitment and retention of management staff in the future.
The Board continues to see slow progress in facilities improvement. The occupancy of the newly constructed Advanced Chemical Sciences Laboratory represents a tremendous improvement of facilities available on the Gaithersburg campus. However, most facilities issues on the Gaithersburg campus, including environmental health and safety issues, will not be fully addressed until completion of the planned Advanced Measurement Laboratory and renovation of the existing laboratories. These improvements are dependent on appropriations for construction and renovation. Some renovation work has also been possible at the Boulder campus, but in both instances, the need for interim planning and remediation continues while a more permanent solution is implemented. In the meantime, it is necessary to work around poor air quality, poor temperature and humidity control, excessive vibration and power fluctuations, and other deficiencies. Such methods used to work around these problems contribute to extra cost, program delays, and inefficient use of staff time.