Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 311
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 9 Measurement Services
OCR for page 312
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 PANEL MEMBERS Kenneth O. MacFadden, Honeywell, Inc., Chair David Abell, Agilent Technologies John M. Ball, U.S. Army Primary Standards Laboratory, Redstone Arsenal Anthony J. Berejka, Independent Consultant, Huntington, New York Duane J. Christy, Mahr Federal, Inc. Terrence M. Conder, 3M L.F. Eason, North Carolina Department of Agriculture and Consumer Services David Holkeboer, Independent Consultant, Ft. Myers Beach, Florida M. Lynne Neumann, Laboratory Accreditation Bureau James D. Olson, The Dow Chemical Company Cecil W. Schnieder, CEC Technologies, PC Ralph Truitt, Corning Incorporated David H. Youden, Eastman Kodak Company Submitted for the panel by its Chair, Kenneth O. MacFadden, this assessment of the NIST Measurement Services Program is based on a formal meeting of the panel on April 3-5, 2002, in Gaithersburg, Md., and on documents provided by NIST.
OCR for page 313
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 PURPOSE OF THE REVIEW The Panel for Measurement Services was established because of a continuing recognition by both the Board on Assessment of NIST Programs and the leadership of NIST of the importance of managing and assessing multidisciplinary programs in a way that transcends the organizational lines of the institute. Measurement services (Standard Reference Materials, Standard Reference Data, and Calibration Services, and the National Voluntary Laboratory Accreditation Program) are a long-standing part of NIST’s portfolio of programs, and this review is intended to consider how well these services are being developed, supported, and delivered throughout the NIST Measurement and Standards Laboratories. This panel includes members who participate in three of the seven standing laboratory assessment panels (the panels for the Physics Laboratory, the Manufacturing Engineering Laboratory, and the Chemical Science and Technology Laboratory) in addition to at-large members appointed especially for this review. The panel was charged with assessing the quality of measurement services carried out by the NIST laboratories and disseminated by the NIST Office of Measurement Services, with a focus on evaluating the following: The range of measurement services within the NIST laboratories; How measurement services are managed and coordinated across NIST, with emphasis on functions performed at the NIST level, including how effectively NIST maintains quality control over its measurement services; How effectively NIST coordinates its measurement services with customer needs in industry, government, and academia (including how NIST gathers and uses information on customer needs and customer feedback); How effectively current NIST measurement services meet industry needs; and The adequacy of NIST human resources, equipment, and facilities dedicated to measurement services, with an emphasis on the challenges of refreshing skill sets and equipment. In a meeting with the panel, the NIST director posed these additional questions: Does NIST approach its service planning effectively from a strategic point of view? Does NIST measure the impact of its measurement services successfully? Is NIST ensuring that U.S. industry has access to NIST-traceable measurements on all appropriate scales? Does NIST effectively align customer expectations with its goals and resources? NIST MEASUREMENT SERVICES ACTIVITIES NIST presented the panel with a summary of its measurement services activities, which are viewed by the institute as a primary mechanism for transferring the results of the investments it makes in measurement science and engineering. The services provide industry with direct links to high-accuracy, primary national measurement standards, calibration of measurement capabilities, and national databases. Every laboratory at NIST supports measurement services activities. The measurement services are these: Calibration services: Direct calibration of instruments and their traceability to national standards; Standard Reference Materials (SRMs): The NIST-certified reference materials traceable to primary national standards and suitable for self-calibration in customers’ laboratories;
OCR for page 314
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 TABLE 9.1 Total Budget for Measurement Services as of June 15, 2002 (in millions of dollars) Technology Services Laboratories Total Standard Reference Data 3.1 4.4 7.5 Calibrations 0.9 6.8 7.7 Standard Reference Materials 3.9 6.7 10.6 National Voluntary Laboratory Accreditation Program 4.4 0.2 4.6 Total 12.3 18.1a 30.4 aAllocated to those laboratories that support measurement services activities. Standard Reference Databases (SRDs): Accessible databases containing data from NIST and other sources that have been critically evaluated to ensure quality and standardized nomenclature; and National Voluntary Laboratory Accreditation Program (NVLAP): Accreditation of laboratories performing specific tests. The budget for NIST measurement services is shown in Table 9.1. Measurement services programs are full cost recovery programs with the exception of SRD, which operates under special legislation (15 U.S.C. 290). NIST currently provides more than 500 calibration services and more than 1,300 SRMs, supports more than 80 databases and online data systems, and has performed close to 800 accreditations. Over time, NIST has used a variety of organizational structures to manage and provide its measurement services. Today, the NIST Measurement and Standards Laboratories are responsible for the technical aspects of NIST measurement services. The seven NIST laboratories individually contain the expertise in measurement capability and technical management that underpins NIST services. The NIST Technology Services unit provides support and business services for NIST measurement services. Under this organizational structure, the majority of planning for individual measurement services comes from the individual NIST laboratories, as an outgrowth of their normal planning exercises. Oversight of the NIST measurement services is provided by the Measurement Services Advisory Group (MSAG), which is composed of the directors of the seven NIST laboratories and the director of Technology Services. The MSAG meets approximately 10 times per year to review NIST policy issues and the quality of services provided to NIST customers. The MSAG efforts seek to ensure quality service and technology to NIST measurement services customers and uniform policies for measurement services throughout NIST. ASSESSMENT OF NIST MEASUREMENT SERVICES The assessment of NIST measurement services should include both an assessment of the technical merit of the relevant research programs in the Measurement and Standards Laboratories and an assessment of the quality of services provided through Technology Services. During this panel’s review of the information provided by NIST, it was often difficult to separate the technical components that lead to the services provided from the process of providing the services. As much as possible, the panel focused its efforts on the service component of measurement services, since the technical merit of the projects under way in each of the NIST laboratories has been reviewed in the previous chapters of this volume by other panels.
OCR for page 315
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 In addressing the components of its charge, the panel decided to assess measurement services from the perspective of a service-supplier business. The charge to the panel, and hence the assessment, falls naturally into three categories: customer needs, NIST capabilities, and operations management. In this light, the range of services appears to be meeting customer needs through its flexible, distributed management structure and a highly talented technical staff dedicated to customer service. However, as discussed below, there appears to be a need for more NIST-wide, servicewide strategy, planning, policy, sharing, and accountability. Customer Needs The panel is impressed with how actively NIST staff seek customer input. Staff members utilize a variety of mechanisms to obtain customer input and feedback on needs in measurement services. They participate in technical conferences, visit industrial customers, and participate in national and international standards committees. They utilize industry road maps and technology forecasts to understand anticipated developments. They organize NIST-sponsored workshops to convene industry representatives in order to consider specific areas of measurement needs. Independent surveys—for example, direct surveys and input from the National Measurements Requirements Committee of the National Conference of Standards Laboratories International (NCSLI)—are used to seek customer feedback on services provided and input on services desired. The panel found definite awareness of and interest in customer needs at the working level for all the services it examined. Pride in customer service is evident throughout the staff that provides measurement services. Responsiveness to customer needs is also apparent throughout measurement services. Maintaining the preeminence of its expertise internationally in measurements and standards science is the most basic way in which NIST responds to its customer needs for measurement services. The laboratories are also clearly expanding their services and capabilities as necessary in response to customer requests for new or improved services. Staff respond not only to technical requests but also to feedback on customer service. An outstanding example of this type of response is the Web-based NIST Information System to Support Calibrations (ISSC)—an information system that can manage almost every aspect of information related to calibration service requests. Purchase orders and test requests can be documented, equipment tracked, calibration progress monitored, and reports issued through this system. It can be used for internal control purposes and can also be accessed by customers through a password-protected system in order to check the status of their service requests. The system gives customers timely information on their calibrations and provides NIST with a tool to analyze and improve calibration turnaround time. Another service improvement introduced in response to customer needs is the recent clarification of the NIST Traceability Policy. This comprehensive statement of NIST policy concerning its support of measurement traceability was placed on the NIST Web site along with frequently asked questions about traceability. The policy, which makes clear the burden of private parties making claims of traceability to NIST to prove that traceability to their customers, has been frequently accessed over the Web. NIST provides methods to assist users in establishing traceability, including accreditation services, through NVLAP. NIST has also responded to customer needs with new methods for delivery of service, such as the use of secondary standards suppliers through its NIST Traceable Reference Material (NTRM) Program for gas suppliers and through the use of the Internet to perform certain electrical calibrations remotely (“e-calibrations”).
OCR for page 316
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 It is clear to the panel that the MSAG has been an indispensable organizational structure for identifying customer needs that cut across all NIST measurement services. The MSAG has been responsible for developing and implementing policy to meet those needs—for example, its response to the need for a clear, NIST-wide statement on traceability. As the MSAG model of leadership matures, the panel expects that it will have an even greater positive impact on the quality of NIST measurement services and their responsiveness to customer needs. While the panel was pleased with much of what it saw occurring in measurement services, it believes that opportunities exist to strengthen NIST’s overall portfolio of services and to increase its effectiveness for customers. Significant customer input and feedback on individual services is gathered by NIST staff members, but the panel finds no mechanism for rolling that feedback up to a higher level and for sharing it among staff across NIST who work on measurement services. While some of this information is likely unique to particular services, much of it may reflect more general issues and concerns. The MSAG does provide a mechanism for gathering and analyzing customer feedback more universally across NIST, but this still appears to be happening in a somewhat ad hoc fashion. To get full benefit of the information that NIST staff currently gather with respect to their customers and from them, NIST needs a more formal process for centrally capturing that information, analyzing it, and sharing it across the institute. The panel is impressed with the capabilities of the ISSC and the opportunities that it provides for transferring information to and gathering information from customers. The panel sees value in expanding the use of such an online system to all NIST measurement services and to measurement services provided internally to NIST customers. The MSAG should examine the value of expanding the availability of ISSC-like capabilities throughout the measurement services portfolio. The capability to gather and analyze the “customer voice” obligates NIST, if it wishes to be perceived as responsive, to respond to the customer with information on its own performance. To that end, and also to improve the internal management of services, measurement services need more uniform, easily understood metrics that can be presented to the customer and staff succinctly. These so-called dashboard metrics could track and present data on such process attributes as turnaround time, service backlog, and customer satisfaction. Internally, NIST should develop metrics to understand customer needs for speed of service versus cost, and it should adjust its processes and services according to the analysis of those metrics over time. NIST is constrained to price services for cost recovery, and it is important to evaluate whether customers perceive the value for the price and to adjust services accordingly. The panel sees opportunities to expand outreach to the end users of NIST services. NIST presented examples of effective and innovative dissemination methods—Web-based data and services; measurement training seminars; and Recommended Practice Guides, booklets that provide practical, easy-to-understand advice and guidance on performing common measurement techniques, addressed to researchers throughout the country who may have recourse to these measurement methods only infrequently. Each of these dissemination tools is being used effectively in specific areas of the measurement services program—for example, Recommended Practice Guides have been issued primarily in areas related to materials science measurements. The panel believes that MSAG should examine such methods for broader use across measurement services. The panel also notes that few of the NIST laboratories seem to have strong ties to NIST Manufacturing Extension Partnership Centers. These centers provide NIST with another opportunity to disseminate the results of its work, and they seem to be particularly appropriate vehicles for expanding customer awareness of measurement services. NIST presented the panel with results of retrospective studies of the economic impact of specific NIST services and products. The panel suggests that NIST utilize prospective marketing studies for
OCR for page 317
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 proposed new services. This would provide the MSAG with a perspective on the primary customer for its services and on the secondary effects that the services might have, prior to the commitment of resources to service development and support. With such information, NIST could better target its investments to areas with maximum impact on U.S. competitiveness. NIST Capabilities Regarding the most important consideration with respect to NIST capabilities, the quality of the staff members who work on measurement services, the panel is impressed. Technical staff developing and supporting measurement services are, in general, world-class, and in many instances best-in-the-world practitioners of their particular measurement expertise. The morale of technical staff members supporting measurement services appears to be high, and their pride in the services they provide to customers is clearly evident. Commitment to quality of service and to appropriate traceability is apparent in their approach to their work. In addition to the measurement experts who directly provide the services, NIST also draws on the in-house expertise of its world-class statisticians, who assure the correct analysis and reporting of uncertainty in the values assigned to SRMs and measurements, and of accreditation experts in NVLAP. Overall, NIST has an excellent human resource base on which to support its measurement services. In general, the equipment supporting measurement services is good, although some significant exceptions may exist, as noted in the laboratory assessments in previous chapters of this report. Significant enhancements in capability are anticipated when the Advanced Measurement Laboratory (AML) is available for occupancy. This facility will enable NIST to continue world-class measurement services in areas of substantial importance for U.S. industry. As with all organizations, NIST struggles under the limitations of its funding and other resources, and so effective use of these resources is critical. Because of the current decentralized management of measurement services, the panel was not able to obtain a clear picture of the total resources devoted to these services. Several areas of possible concern presented themselves. No overall demographics for staff that develop and support measurement services are available. NIST provided no data to indicate whether the portfolio of measurement services rests on a base of staff with the appropriate age and experience to ensure continuation of core services into the future, or whether the available staff resources are being used appropriately. For example, it appeared to the panel in its brief time in the laboratories that a number of Ph.D.’s are spending inordinate amounts of time on technician-level work. Greater use of technicians would increase the overall efficiency of measurement services and enable NIST to better respond to more of its customer needs. Consideration of NIST-wide measurement services staff demographics by the MSAG might enable better utilization of staff, better succession planning, and better cross-training of staff involved in measurement services. No overall plan for equipment and facilities is in place. MSAG does not have a broad picture of which programs may be in need of major equipment upgrades in the near future in order to ensure continuity of services; nor is there a plan to ensure the required facilities for those measurement services programs that will not be housed in the new AML. The panel does not understand the policy regarding fees for SRDs. Some SRDs are available for purchase, while others are given away at no charge. The panel wonders if a uniform policy for charging would generate more complete cost recovery and additional revenues for SRD development. The panel was presented with data that indicate that SRM sales, an important source of revenue for measurement services through fees, are decreasing over time. No analysis of this trend was presented to the panel. These sales may be decreasing for positive reasons—for example, the increased use by
OCR for page 318
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 industry of secondary standards traceable to NIST such as NTRMs—or for negative reasons, such as lack of industry interest in specific SRM offerings or the higher cost of NIST SRMs in comparison with the cost of NRTMs. The MSAG needs to investigate the reasons for this drop-off in order to understand it and take appropriate actions. On a similar note, even though there appears to be a conscious effort to assess NIST impact and the value of particular measurement services, a few SRMs were rather low-technology in approach and might be migrated out-of-house in order to free up NIST resources to work on development and support of SRMs requiring more sophisticated technical expertise. The MSAG should review the SRM portfolio to determine whether such opportunities exist. Operations Management The flexible, distributed management of measurement services as currently embodied through the sharing of responsibilities between the laboratories and Technology Services, with coordination occurring through the MSAG, appears to be working well. Placing the responsibility for identification of customer needs and for technical decision making in the laboratories, which have the technical expertise and customer knowledge needed to perform these tasks, makes sense and is serving NIST well. At the same time, the coordination being performed by the MSAG is working well and has resulted in some significant improvements in policy and use of the resources available for measurement services. The current management system works well to ensure that delivered services are reliable, and responsiveness to customers’ expressed needs is strong. The resulting culture existing among measurement services staff places a high value on precision, accuracy, and appropriate traceability of measurements. Extensive use of domestic and international key comparisons and of check standards was observed. Partnerships with other national measurement institutes (NMIs), which have focused primarily on technical collaborations to ensure the quality of a measurement performed in both institutes, are now being expanded to agreements to accept one another’s standards in certain areas. The panel believes that the overall operations of measurement services could be strengthened by the development of a strategic plan. While each current program appears to meet an identified customer need, the panel and, indeed, MSAG have no way of judging whether the portfolio as a whole addresses the most critical customer needs. Strategic planning for measurement services could enable the MSAG to target its investments toward programs and measurements with the highest leveraging in the U.S. economy. It could serve as the basis for pursuing additional targeted collaborations with other NMIs in order to maximize the services available to U.S. customers. The Strategic Focus Areas identified in the ongoing NIST-wide strategic planning exercise provide an excellent template upon which a measurement services plan can be built. The panel also sees great benefit to be had from sharing of best practices. The panel saw excellent mechanisms for communicating with customers, delivering services more efficiently, and expanding services available to U.S. industry—for example, the ISSC, the use of e-calibrations, and the recent cooperative agreement with Germany’s Physikalisch Technische Bundesanstalt for certain SRMs. Many of these practices could be more widely utilized to good effect. It appears that no process is in place for sharing, dissemination, and implementation of such “best practices” NIST-wide. The panel suggests that the MSAG devote some time to such an exercise. As noted above, the panel also believes that management of measurement services could be im-
OCR for page 319
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 proved if the MSAG developed and used common metrics for the performance of measurement services NIST-wide. Currently, metrics are monitored for each service at the level of the service provider. It does not appear that metrics are universally applied for all services. Even if some metrics are being used for all services offered, this information still has to be rolled up into overall metrics for measurement services so that the MSAG has the information it needs to assess performance and make subsequent programmatic decisions. NIST has in place a NIST quality system to document the steps it takes to ensure the quality of the calibration and testing services on which its customers depend. Without meaning to imply that the current quality system is insufficient, the panel suggests that the NIST quality system more visibly conform to the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) systems that its customers use and are familiar with. The panel suggests that NIST self-declare voluntary compliance with those provisions of ISO/IEC 17025 that are applicable and appropriate to the case of an NMI. Adoption of ISO/IEC 17025 as a template for describing the NIST calibration and testing services program would make it easier for NIST customers (many of whom are ISO/IEC 17025-compliant or accredited) and other outsiders to understand certain aspects of the NIST measurement services program. Furthermore, some documentation of the manner in which NIST responds to specific clauses of the standard would provide valuable guidance to other calibration laboratories for developing or assessing their own quality programs. The MSAG should accept responsibility as quality coordinator for the measurement services programs and ensure that the quality system is implemented across NIST in all applicable programs. MAJOR OBSERVATIONS The panel presents the following major observations: The current system of flexible distributed management of NIST measurement services provides the capacity for positive customer relationships and responsiveness, excellent technical decision making, and structured coordination through the internal Measurement Services Advisory Group. Excellent grassroots connection with the customers of measurement services is evident, and much information on customer needs is gathered through such channels. To extract maximum value from such information, a process is needed to gather and analyze it centrally and to disseminate it across NIST. The MSAG should develop an overall strategic plan for measurement services that is consistent with the overall NIST strategic plan being developed. This would help ensure that the services offered are addressing the most critical customer needs and are providing those measurements that have the most leverage in the U.S. economy. The Strategic Focus Areas identified in the NIST strategic planning exercise provide a good template on which to build a measurement services strategy. The plan would also provide the basis for the needed facility and staff succession plans. While staff involved in measurement services receive significant feedback from their customers with respect to customer needs, the use of prospective marketing studies would help the MSAG better target those services that would have the greatest impact on U.S. competitiveness. Metrics for measurement services are needed to provide the MSAG with tools to assess performance and take necessary programmatic action. The panel recommends that the MSAG develop “dashboard” metrics that can also be used to give customers and staff succinct, easy-to-understand
OCR for page 320
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 2002 measures of NIST performance and demonstrate NIST’s commitment to continual improvement in its programs and services. The MSAG should engage in a “best practices” exercise to propagate the use of the most effective and innovative means of identifying and meeting customer needs. The MSAG should expand the NIST quality system to include a statement of voluntary compliance with the ISO/IEC 17025 quality standard where applicable and appropriate to a national measurement institute.
Representative terms from entire chapter: