Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 197
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Chapter 8 Computer Systems Laboratory
OCR for page 198
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 PANEL MEMBERS O.R. Pardo, Bechtel/Parsons Brinckerhoff, Chair Herbert D. Benington, Paramax Systems Corporation Dorothy E. Denning, Georgetown University Clarence G. Feldmann, Consultant, Winchester, Massachusetts James George, Mesa Graphics Robert E. Kahn, Corporation for National Research Initiatives Sandra M. Lambert, Citicorp Roger R.A. Morton, Eastman Kodak Company Lawrence R. Rabiner, AT&T Bell Laboratories Michael B. Spring, University of Pittsburgh Raymond T. Yeh, International Software Systems, Inc. Submitted for the panel by its Chair, O.R. Pardo, this assessment of the fiscal year 1994 activities of the Computer Systems Laboratory is based on site visits by individual panel members, a formal meeting of the panel on March 9-11, 1994, in the Gaithersburg, Maryland, facilities of the National Institute of Standards and Technology, and on the annual report of the Computer Systems Laboratory.
OCR for page 199
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 LABORATORY OVERVIEW Mission The mission of the Computer Systems Laboratory (CSL) is to help government and U.S. industry increase their productive and reliable use of computer and related telecommunications systems, improve security management and technology of computer and related telecommunications systems, and work closely with the users and producers of computer and telecommunications systems to improve the competitive posture of the U.S. computer-related industry. Strategy CSL improves the productive and reliable use of computers by developing standards, profiles of standards, guidelines, conformance tests, technical methods, measurement techniques, and prototypes and by providing advice on planning, deployment, and use of new information technologies. CSL improves security management and technology by developing timely and innovative solutions to security problems and technical, management, physical, and administrative standards and guidelines for the cost-effective security and privacy of sensitive information. CSL contributes to U.S. industrial competitiveness by developing and transferring to government and industry generic methods and techniques that facilitate technology development and improvement, collaborating with industry and industrial users to advance standards development and common implementation approaches, performing research to support technical activities, and participating in NIST-wide and government-wide planning for new technology-related initiatives. Resources CSL's full-time-equivalent (FTE) staff ceiling for fiscal year 1994 is 259. CSL currently has 282 staff members, of whom 234 are full-time permanent staff. The remainder are part-time permanent and intermittent staff. (Part-time faculty and student staff are counted against the FTE ceiling.) In addition, there are 32 guest researchers. The FTE ceiling limits CSL's performance. Fiscal year 1994 funding is estimated at $17.5 million from NIST Scientific and Technical Research and Services (STRS) funds, $1.1 million from NIST Competence Building Program funds, $0.5 million from NIST Information Technology Services funds, and $18.2 million from other (federal) agency (OA) funds. CSL has a wide range of computer and communications equipment, some of which is dated, much of which is powerful and current. CSL space at the NIST campus consists principally of the second floor and portions of the basement, first, and third floors of the Technology Building, which it shares with other NIST units. This space is not adequate to meet CSL's laboratory and office requirements.
OCR for page 200
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Assessment of Strategy CSL's strategy, as stated above, is consistent with its mission and programs. CSL's development of standards, profiles of standards, guidelines, and conformance tests is excellent and relevant to national need. CSL's use of workshops, consortia, and surveys is extensive and should continue. CSL could improve its strategic and technical planning. The strategies of each division are not completely uniform or clearly stated; however, many of the individual programs have clear, specific strategies. Two divisions have resource-based plans with milestones. The predominant absence of consistent, clearly defined division-level missions, strategies, and plans raises panel concerns as to whether CSL is working on the highest-priority programs, whether the programs represent a complete set, and whether CSL programs have and are achieving long-range goals. Given that new NIST resources will be distributed by priority, CSL and division missions and strategies should be better focused. CSL program priorities should be based on plans derived from CSL strategy. CSL programs demonstrate an excellent understanding of the open systems environment (OSE) and open systems communications models, but it is not always clear that CSL's program planning and assessment are based on a clear understanding of the trends in the information technology industry. Also, CSL's planning and assessment are unduly influenced by OA funding sources. If CSL's budget shifts to greater STRS funding, CSL should develop models of its customer base and technology base to assure that its technical programs mirror the needs of its clientele. Technical planning based on such models promotes top-down planning; however, given CSL's historical role and continuing activities with individual government agencies, bottom-up planning must also continue. Balance between top-down and bottom-up planning at CSL will be affected by the extent of CSL's activities on the National Information Infrastructure (NII). Policy and other NII-related activities may or may not mesh with CSL's current mission. Nevertheless, the convergence of technologies, notably the anticipated ubiquity of the Internet, the spread of wide area networks, and the advent of sophisticated computer services, suggests that major changes in information technology will happen irrespective of federal policy. CSL should be proactive in defining NII requirements in general and for federal customers in particular. NII requirements, in turn, should influence CSL's research agenda, particularly at the cutting edge, for example, the research agenda of the Advanced Systems Division. A top-down approach could serve both as a basis for assessing CSL's support of commercial research and as a guide for standards-setting activities. Top-down planning could also provide user perspective when resources are allocated for competing metrology needs. A major challenge in fiscal year 1994 is the planned merger of the Computing and Applied Mathematics Laboratory (CAML) and CSL to create the Information Technology Laboratory (ITL). This merger provides a major opportunity to focus existing resources and programs on the NII. However, management must carefully approach the merger in order to maintain staff morale, and, most importantly, maintain and strongly support core capabilities in applied mathematics, statistical engineering, software and communications testing, spoken and written language corpora development, and standards development. The staff of the two laboratories should work together to define the challenges and opportunities of the merger, providing input on the organization and programs to the NIST director.
OCR for page 201
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Assessment of Technical Programs CSL's programs are extensive in technical scope and number, with many fine, well-received accomplishments since the panel's fiscal year 1993 assessment. Highlights deserving special mention include the following. Program planning. In response to 1993 panel discussions, the Computer Security Division developed excellent plans for each of its programs, which included detailed activity descriptions, milestones, forecast completion dates, funding sources, staffing, and relationships to other projects. Five-year plan. The Systems and Network Architecture Division's 5-year plan included current and planned projects. Most importantly, it included the criteria for their selection. Virtual library. This activity supports both the NII and NIST's overall mission. A broad consortium (the National Library Association) was used to establish program requirements and the need for standards. Speech corpora. The Advanced Systems Division's Speech Recognition Group continued its production of new speech corpora CD-ROMs and its cooperation with researchers outside NIST to share software and to promote benchmarking. The group's work ensures that future commercial speech recognition products can be developed within a framework that is universally understood. Software standards. The Information Systems Engineering Division's Software Standards Program validates several computer languages and maintains a CSL Validated Products List that has a large influence on government acquisitions. The division's meticulous work is particularly important to the U.S. software industry. The “virtual library” that will put the CSL products and information bases on-line, and CSL's effort to step back and examine the standards, guidelines, and testing development process for process improvements, are particularly noteworthy. The panel's biggest concern about CSL's technical programs is the cap on FTE staff . The best that CSL can do is to focus on efficiency to meet its mission. Several CSL programs require attention: Parallel processing. The Advanced Systems Division is doing good work in parallel processing; however, given the resources of the Parallel Processing Group, the panel expected more progress. Software correctness. Although useful, the Software Corrections Program seems out of place in the Systems and Network Architecture Division. The program seems to come from the bottom-up, i.e., to be built around an individual 's interest and skills, rather than from the top-down, that is, as a response to a need identified by management. High-integrity software systems assurance. The “program slicer” is a useful tool. Given that most errors occur in the specification and software design stage, however, the Systems and Software Technology Division should give priority to the early stages of the software engineering process rather than continuing this tool's enhancement.
OCR for page 202
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 The above highlights are discussed in more detail under the assessments of the various divisions. Recommendations The following are the panel's recommendations for CSL as a whole. CSL management must carefully approach the merger with CAML to form ITL in order to maintain staff morale and maintain and support CSL core capabilities. The staff of the two laboratories should work together to define the challenges and opportunities of the merger, providing input on the organization and programs to the NIST director. CSL must work creatively to overcome, as best it can, its severe staffing limitations. Inclusion of part-time faculty and students in the FTE ceiling is counterproductive; guest staffing should be controlled by funding, space, and mutual benefit. CSL needs larger, more flexible space. If available space is off-campus, the space should be close enough to NIST facilities that a collegial relationship can be readily maintained with other NIST laboratories. With a mission expanded by the merger of CSL and CAML, ITL will need increasingly high bandwidth connections with the rest of NIST to assure full working ability and space on the main campus for the expanded visitor program. CSL should, with active division buy-in, develop a model for its customer base and technology base. CSL should revise its strategies against its technology base, and, using the guidelines in Setting Priorities and Measuring Results at the National Institute of Standards and Technology (NIST, Gaithersburg, Md., January 1994), assess its current programs and plan new ones. Using its model of the customer base, CSL should look for additional participants for current workshops and consortia or identify new workshops and consortia. Each CSL division should upgrade and perfect its multiyear plans, with detailed activity descriptions, milestones, forecast completion dates, funding sources, staffing, and relations to other NIST projects. CSL divisions should plan jointly. Joint programs involving more than one division (or laboratory) should have a single plan, showing the activities of each division, group, or laboratory. The panel recommends a skills survey and a summary of available versus needed skills, which could be used to guide hiring and transfer within CSL. CSL should assess the value of its “best-in-class” examples of projects that support industrial competitiveness. These “best-in-class” projects (with authentication by external experts) could be used as project role models and justification for future activities. CSL's formal and informal networking with industry appears to be increasing and should continue to be a priority.
OCR for page 203
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 CSL needs to adopt clear and consistent metrics to monitor and assess its work. For example, CSL should categorize desktop computers used in the emerging NII and track the extent to which CSL standards, guidelines, and other products are used in those computers. CSL Responses to Fiscal Year 1993 Recommendations CSL responses to the fiscal year 1993 panel assessment were, in general, positive and proactive. CSL collaborated in standards setting, demonstrated open systems applications in the Electronic Commerce Program, and extended the scope of NIST's open systems activities to cover interoperability among distributed, heterogeneous systems and the use of the Open Systems Environment Profile as a framework for organizing its seemingly disparate activities, as requested in the panel's fiscal year 1993 assessment. Given below are some of the panel's fiscal year 1993 recommendations (quoted from the fiscal year 1993 assessment), followed by CSL's responses and panel comments. “The Computer Systems Laboratory should adopt the strategic planning framework as outlined in Visiting Committee on Advanced Technology Annual Report 1993 (Figure 2, p. 5; Visiting Committee on Advanced Technology, National Institute of Standards and Technology, Gaithersburg, Md., January 1994)” (p. 216). CSL will use the strategic planning framework in rewriting its strategic plan. Materials provided to the panel were organized around the framework's format. The panel commends the response to date and stresses the need for continued refinement of planning, reporting, and measuring. “As a prelude to strategic planning, the Computer Systems Laboratory should categorize current and planned projects as either fundamental research or applied research to determine the balance between these two categories, whatever their source of funding” (p. 216). CSL considers its work to be almost entirely applied research. Funding sources for CSL work are driven by the NIST budget process and NIST's strategic planning. The panel believes that some of CSL's work has a broader, hence more basic, application and recommends that CSL reconsider this recommendation. “The Computer Systems Laboratory should determine specific needs of U.S. industry, civilian agencies, and Department of Defense agencies for its services” (p. 217). CSL continues to identify and assess requirements through workshops, forums, management meetings, participation in standards-development activities, contacts with agencies and users, collaboration with industry through agreements and other activities, and many additional interactions. The National Performance Review (NPR), the NII, and the High Performance Computing and Communications (HPCC) initiative have all identified and documented requirements for computer, communications, and security technologies. CSL will initiate efforts to develop and record formal documentation of the requirements identified. The panel believes CSL should do more than its response indicates. CSL will rely on other government programs to provide requirements and assessments. CSL has programs to provide the necessary requirements; however, it is not obvious that CSL has a thorough approach to requirements gathering, program prioritization, and progress assessment. “To minimize internal inefficiencies and maximize external value, CSL could emulate industry's increasingly customer-driven approach by inviting its government clients to participate
OCR for page 204
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 in annual assessments of CSL's performance” (p. 218). CSL participates in Department of Commerce efforts for total quality management (TQM), which addresses these concerns. These efforts include customer assessments of performance. The panel will continue to monitor the effectiveness of CSL in using clients in the assessment process. “The Computer Systems Laboratory should arrange for increased collaboration among the major users and promulgators of computer-related standards in prioritizing, setting, and adopting standards” (p. 218). This issue is addressed by the administration in The National Information Infrastructure: Agenda for Action (Information Infrastructure Task Force, Washington, D.C., September 1993), which describes the role of the government in promoting the development of the NII. The Agenda for Action called for NIST to establish a panel and work with other appropriate agencies to review the government's involvement in establishing network requirements and standards with domestic and international partners. That review, with input from the private sector and other levels of government, will consider the role of the government in the standards process and will identify opportunities for accelerating the deployment of the NII. In October 1993, CSL named an interagency panel, the Federal Internetworking Requirements Panel (FIRP), to review open systems network requirements and to recommend policies on the use of networking standards by the federal government. FIRP addressed issues related to the Internet Protocol Suite and Open Systems Interconnection (OSI) specifications, two widely used yet incompatible protocol suites, as well as proprietary networking protocols. CSL will review its report (Report of the Federal Internetworking Requirements Panel, NIST, May 1994) and solicit widespread review and comments before any policy changes are considered. CSL will pursue broader discussion of these issues through collaboration with the Information Infrastructure Project of the Science, Technology and Public Policy Program at the John F. Kennedy School of Government, Harvard University. A workshop was held in June 1994 to explore the development of standards that are critical to the NII. The workshop drew on experts in information policy, intellectual property, economics of standards and networks, and the role of consortia standards. A book of proceedings and supplemental materials will be produced to help clarify the strategic environment for standards development, help map and evaluate options for the federal government, and lay a foundation for future policy recommendations. CSL will review the final report of FIRP and the results of the workshop conducted by the Information Infrastructure project. A panel of federal government experts, working in cooperation with the Information Infrastructure Task Force, will be convened to consider the workshop findings and to make recommendations on issues such as establishing a process for developing a high-level architecture for the NII and a process for determining standards needed for the NII, establishing effective planning processes to accelerate the development and implementation of needed standards, establishing a process for policy coordination on standards for the federal government, improving leadership of and strengthening the standards processes, and coordinating efforts of the standards developers. “The Computer Systems Laboratory's goal for standards setting should be interoperability among distributed, heterogeneous systems, rather than the current, more limited vision of open systems” (p. 218). This goal has been adopted. The panel will continue to monitor CSL progress. “The Computer Systems Laboratory should increase its collaboration with other NIST activities, particularly with NIST's Manufacturing Technology Centers” (p. 218). CSL collaborates actively with other NIST laboratories in planning for the HPCC and is involved in the
OCR for page 205
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Manufacturing Technology Centers. The panel would like a report on CSL's progress for its fiscal year 1995 assessment. “The Computer Systems Laboratory should develop, in collaboration with its government constituencies, measures for its success and/or failure in setting standards—e.g., vendor and user adoption of such standards” (p. 218). CSL continues to assess the success of its standards-setting activities; drops activities that are not productive, timely, or responsive to user requirements; and is exploring a variety of options for standards setting, including the formal voluntary standards process, the work of consortia, and the use of publicly available specifications. Many of these issues are being addressed by FIRP. Further, CSL management contributed to and will be guided by Setting Priorities and Measuring Results at the National Institute of Standards and Technology. The panel notes the need for further attention to issues of metrics as embodied in its recommendations above. “The Computer Systems Laboratory should consider alternatives to its current facilities, e.g., temporary office space on campus or leased commercial space nearby, if additional space cannot be made available within NIST facilities” (p. 220). NIST is addressing space requirements as part of its building remodernization efforts. Relocation of CSL to an off-campus site is a possibility. The panel believes that this move is critical and CSL should plan it carefully to ensure it does not erode its effectiveness. “The Computer Systems Laboratory should seek to acquire cutting-edge equipment that mirrors the scale and heterogeneity of government and commercial computing” (p. 220). CSL has initiated actions to acquire $1.5 million in new equipment during fiscal year 1994. The panel will continue to monitor progress in this area. DIVISIONAL ASSESSMENTS Information Systems Engineering Division Mission The Information Systems Engineering Division develops standards for information systems and provides technical assistance to government and industry in data administration, data management technology, computer graphics, and the validation of related software standards. Strategy The Information Systems Engineering Division conducts research; develops federal standards and guidelines; provides advisory services to federal agencies; performs technology forecasts; maintains testbed facilities in information systems technology, including data models, database systems, data administration, database design, computer graphics systems, programming languages, and advanced applications development facilities; and develops tools and techniques for testing the functions and performance of information systems software, including programming languages, computer graphics, database software, and data dictionary software both within systems and in computer networking environments. The Information Systems Engineering
OCR for page 206
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Division uses three strategic methods to meet its mission. First, it obtains direction and support from industry (primarily information technology developers, integrators, and users), academia, and government through workshops, meetings, and cooperative agreements. Second, it delivers products and services such as Federal Information Processing Standards, profiles, and guidelines; prototypes; demonstrations; workshops; participation in the development of U.S. national standards and international standards; certification that software products conform to standards; and direct assistance to industry and federal, state, and local agencies. Last, it measures progress in terms of products and services delivered and the degree of acceptance in the market. Resources The Information Systems Engineering Division has a staff of 45, which includes full-time positions, guests, and research assistants. Fiscal year 1994 funding is estimated to be $2.1 million from STRS, $0.2 million from NIST Competence Building Program funds, and $3.6 million from OA sources. The division maintains a broad selection of hardware ranging from desktop systems to personal computers (PCs). Current staff and hardware seem adequate, and future hardware acquisition plans will improve the computing power and range of equipment. Assessment of Strategy The Information Systems Engineering Division has a good grasp of its customers and their requirements. Conformance testing consumes the majority of effort within this division. Although such testing provides a valuable service to other agencies and industry, it leaves little time to explore emerging technologies and ad hoc standards that will influence future acquisition policies. Assessment of Technical Programs Both the Information Systems Engineering and the Systems and Software Technology Divisions are acquiring equipment and developing expertise in the area of virtual reality. Although this is commendable and directly supports the NII, there is little evidence to suggest that the two divisions are establishing a unified, collaborative approach to this area. The Graphics Software Group performs research, develops standards and guidelines, and provides technical advisory services for the Graphical Kernel System (GKS), Programmers Hierarchical Interactive Graphics System (PHIGS), and Computer Graphics Metafile (CGM) and develops or licenses conformance tests to determine whether graphics products correctly implement graphics standards. The Database and Graphics Group provides design, specification, acquisition, installation, integration, training, and maintenance support for all laboratories, administrative computing, and networking for the division. In addition, this group is participating in a joint project with the
OCR for page 207
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Manufacturing Engineering Laboratory (MEL) and CSL's Advanced Systems Division to support the NII System Integration for Manufacturing Applications Program. The Data Administration Group performs valuable work in the next-generation International Organization for Standardization (ISO) Information Resource Dictionary System draft and supporting test suite. It is correctly maintaining close collaboration with outside organizations, in particular the Data Administration Management Association (DAMA), which provides an appropriate and important forum for the group to contact “real-world” data administration problems and to transfer information on standards and research activities to the DAMA membership. The Graphics Software Group provides an excellent service in the validation of GKS, PHIGS, and CGM. Although this service has a wide audience, it tends to serve users of high-performance computer equipment rather than of desktop workstations and PCs. There is some concern that the majority of graphics applications that are being developed today are neither GKS nor PHIGS compliant but rather are written using the graphics facilities provided by X-Windows, MS Windows, or OS/2. This group should take the lead in determining the relationships between existing graphics standards and the facilities provided by these window systems. With a large proportion of this group's time spent certifying software for various standards, there seems to be a need for a study of testing techniques and automatic testing methods to provide more efficient testing and more accurate compliance measurements. The Software Standards Validation Group is responsible for a wide range of standards activities and conformance testing. The group is responsible for validating several computer languages and for maintaining the CSL Validated Products List, which has a large influence in government acquisition activities. Two of the groups are involved in some aspects of software reuse: the Data Administration Group's support of an Internal Revenue Service reuse effort, and the Software Standards Group's involvement in domain modeling. The area of reuse is important enough at this point in time for NIST to take an active role in emerging reuse technologies. The Database Languages Group is primarily responsible for the standardization and conformance testing of Standard Query Language in its many forms. The group has an excellent grasp of its customer base and their requirements and performs a valuable service for NIST and its customers. Recommendations The following are the panel's recommendations for the Information Systems Engineering Division. The Information Systems Engineering Division needs to find ways to off-load conformance testing to outside sources. The Graphic Software Group in the Information Systems Engineering Division should expand its scope to include PC workstation graphical user interfaces. The Information Systems Engineering Division must clarify and better coordinate its joint program on virtual reality with the Systems and Software Technology Division.
OCR for page 216
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 The panel notes an absence of work on distributed authentication (verification of identity and access authorization across networks) using Kerberos-like functions. The use of token-based profiles for distributed authentication is widely discussed in information security forums, and at least three products are commercially available to fill the need. The security community would benefit from the division 's guidance in distributed authentication. Although the division is a logical candidate to be the administration 's key escrow agent for public-key computer security systems, the magnitude of the division's key escrow-related work has already resulted in a shift of resources away from areas (e.g., firewalls) that would have been beneficial to both government and commercial users, and it has delayed related deliverables. The panel notes that, whereas “1 percent of [NIST's] $520 million budget (is devoted) to computer and communications security” (Computerworld, May 9, 1994), standard corporate programs allocate about 10 percent of their computer systems budget to computer and communications security. Recommendations The following are the panel's recommendations for the Computer Security Division. The Computer Security Division should develop a crisper and more complete strategy statement to support the Computer Security Division 's mission, which should also be consistently stated in all strategic planning documents and presentations. The Computer Security Division should use program plans to track progress and deliverables. The division should dedicate increased resources to virus prevention and detection and distributed authentication. The division should consider assigning and increasing resources to programs having broad-based applicability, such as distributed authentication and Internet firewalls. Computer Security Division Responses to Fiscal Year 1993 Recommendations Given below are the panel's fiscal year 1993 recommendations for the Computer Security Division (quoted from the fiscal year 1993 assessment), with the division 's responses and panel comments. “The Computer Security Division should develop criteria for determining how much of a networked system must be trusted. Much of this work would tie into the security needs of the National Information Infrastructure initiative” (p. 230). The division responded that it is looking into this issue and will report to the panel. Since the fiscal year 1993 assessment, the division has worked on four items that it believed could lead to the criteria: firewalls, identification and authorization, public-key infrastructure for digital signature, and Defense Information Systems
OCR for page 217
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Agency separation kernel technology. It is not clear to the panel how the division's work on elements of the above four items will be combined to develop the criteria. “Since several research projects on detection of intrusion are well under way outside NIST, the division should rely on external research rather than start an independent project” (p. 230). The division is not doing independent research in this area but is investigating and applying available techniques. The purpose is to provide guidance and transfer technology, and support the security needs of the NII. The panel finds CSL's approach reasonable. “The Computer Security Division should continue to consult, develop guidelines, and provide training programs, especially in computer security risk management and analysis, and should complete or upgrade its computer security handbook” (p. 230). Although not reviewed by the panel, the Computer Security Handbook is complete. A training program on computer security is being developed. “The Computer Security Division should consider expanding its activities in ensuring the security of medical information” (p. 230). The Computer Security Division responded that medical information is a critical application area but that resources are too limited to pursue specific solutions for protecting medical information, and many of the generic techniques developed for protecting sensitive information of all kinds will be applicable to the protection of medical information as well. Health care is a focus area of the ATP, and CSL division managers have established a formal relationship with ATP. The panel will assess the Computer Security Division's progress at the fiscal year 1995 program reviews. “The division should take a much more active role in planning for research on ensuring overall National Information Infrastructure security requirements” (p. 230). The division participates in a CSL team to develop plans for a high-level prototype interface to the NII. Division staff members are developing and executing the detailed plans for HPCC projects, and division personnel are encouraged to work on joint projects with other groups, division, and laboratories. The Computer Security Division adjusted its focus this year to high-impact areas, most of which support the NII. The panel will monitor the progress of the planned deliverables in these areas. Systems and Network Architecture Division Mission The Systems and Network Architecture Division promotes agreements and technical solutions that facilitate electronic information transfer. Strategy The activities that are within the scope of work for the Systems and Network Architecture Division include activities that support and strengthen the network infrastructure (e.g., routing and forwarding protocols, management protocols, security protocols); result in useful applications or groups of applications (e.g., electronic mail, directory services, electronic commerce); and are
OCR for page 218
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 needed to support previously listed activities that are not pursued sufficiently elsewhere (e.g., protocol testing and validation, performance studies, software reliability). The division's customers are the user community in the government and commercial sectors that rely on networking technology; vendors of networking technology also are some of its most intensive users. Activities are focused on enabling new and better economies of scale and lower prices and, through standardization, curtailing duplication and thus reducing development costs and increasing interoperability. The division's strategic thrusts are to participate in setting voluntary standards, developing and proposing prototype standards, integrating prototypes of standardized components to demonstrate electronic commerce applications, and technology transfer. Resources The Systems and Network Architecture Division's current available resources include 30 computer scientists and 6 guest scientists, $230,000 of in-kind equipment from industry, and equipment consisting of a mix of Sun workstations and PCs connected to a 10-Mb Ethernet and a LAN linked to a fiber-distributed data interface (FDDI) ring and through it to the network service provider, SURANET. Fiscal year 1994 funding is estimated at $2.4 million from STRS and $3.1 million from OA sources. Resources appear adequate, except for a lack of space to hold demonstrations and conferences. A planned hookup to high-speed networks is jeopardized by the impending move of the division to off-site quarters. The laboratory facility should be planned to avoid disruption of the pending network connection, which might necessitate not moving the division laboratory. Assessment of Strategy The mission statement does not suggest a focus for the Systems and Network Architecture Division strategy. Excellent work is being done in the development of plans and the selection of specific projects; however, the big picture is missing. As a result, it is not clear to the panel how the division can measure its effectiveness. For example, participating in standards committee work and gaining competence in specific technological areas central to networking and network architectures are positive and useful efforts but are not clearly linked to a short- and long-term vision of the division's objectives. The division has developed a detailed 5-year strategic plan that goes into current and planned projects and the criteria for their selection. The NII and Electronic Commerce initiatives, however, should have a significant impact on these plans and specific projects in the near future. The division should document how its networking efforts are linked to the above initiatives and measure progress based on achieving specific objectives related to usage-oriented metrics. If each enabling technology were more clearly related to these more visible programs, support might be more forthcoming. In particular, the objective of making electronic commerce accessible to small business should be specified among goals and objectives.
OCR for page 219
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Assessment of Technical Programs For the Systems and Network Architecture Division, the panel supports relaxation of efforts on GOSIP adherence, development of Internet Protocol Suite protocols, and recognition of Internet Engineering Task Force standards as recommended in the FIRP report. The division's research appears to be state of the art as evidenced by its contributions to various technical committees and organizations. The work on “ software correctness,” however, appears to be out of scope for the network focus of the division's mission. It seems that existing staff skills result in an interest in an important area, but not one that matches the networking focus of the division. The relation of the division work on secure electronic mail to privacy-enhanced mail, the Massachusetts Institute of Technology effort, and other state-of-the-art research should be documented and clarified so that the Systems and Network Architecture Division's efforts and goals can be compared with other research efforts. Recommendations The following are the panel's recommendations for the Systems and Network Architecture Division. The Systems and Network Architecture Division's planning document should be recast in terms of the NII and electronic commerce so that the role and responsibility of the division are clear and its projects can be clearly identified with these national initiatives. The division's role in support of the ATP should also be specified in its strategic plan. Division projects on the NII and electronic commerce should focus on interoperability testing, perhaps using the Electronic Commerce Integration Facility or some similar user-perspective basis on which to test the adequacy and usability of standards and prototypes. The investigation into secure electronic mail needs to sharpen its goals and objectives. The division's mission statement needs to be analyzed, consolidated, and made more visible. The current mission statement is not useful in focusing the strategy. Laboratory facilities must be planned so as not to disrupt the division 's pending network connection, even if this means keeping the division on the main campus while other division move off-campus. Systems and Network Architecture Division Responses to Fiscal Year 1993 Recommendations In general, the response to the fiscal year 1993 recommendations was excellent. For example, the short- and long-term planning was improved significantly. Given below are the
OCR for page 220
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 panel's fiscal year 1993 recommendations for the Systems and Network Architecture Division (quoted from the fiscal year 1993 assessment), with the division's responses and panel comments. “A division-wide skills assessment is in order if the Systems and Network Architecture Division is to address a broader network mission ” (p. 233). The CSL assesses skills continuously on an informal basis in assigning staff members to projects and developing plans for initiatives. The panel had intended that a skills survey would be conducted and available versus needed skills summarized. This summary would then be used to guide hiring and transfer within the CSL to address efforts that cannot be currently pursued because of unavailability of skills. “In the area of electronic commerce, the Systems and Network Architecture Division should focus on only a few, well-defined pilot projects, because of the large scope of electronic commerce” (p. 233). Under the fiscal year 1994 initiative, CSL will collaborate with industry and other agencies to develop or apply technologies that enable electronic commerce in general and particularly as it applies to manufacturing electronic and mechanical components and subsystems. Fiscal year 1994 objectives include developing an electronic commerce integration and demonstration facility at NIST and soliciting industry participation and demonstration of products and investigating basic interoperability and integration issues related to use of value-added networks and electronic mail. “The division should develop a program that focuses on automated standards testing, explores the use of formal definitions, and supports the development of tools such as ESTELLE and commercial automated test generators” (p. 233). The division responded that it will be focusing on addressing integration and interoperability issues related to electronic commerce, using commercial off-the-shelf products where available and developing prototype components when no commercial products are available. Evidence of this type of automated testing development was not presented to the panel during its fiscal year 1994 assessment. Test generation was presented by CSL as a general need, but no plan or organized effort toward this end was presented. “The Systems and Network Architecture Division's exemplary strategic plan should be revised to keep pace with NIST 's expanding mission and requirements related to its role in developing the National Research and Education Network” (p. 233). The division will continue to update the plan. “The Systems and Network Architecture Division should move beyond viewing its mission as bounded by open (i.e., nonproprietary) protocols. The division should retain its current internal Open Systems Interconnections (OSI) focus but should also pioneer in the promotion of OSI for use in the United States. The division should provide broad support for its customers in defining, operating, and evolving heterogeneous network systems. This approach should include refinement of the open system network concepts and continued development of associated supporting concepts, such as distributed systems networking, specific frameworks for addressing user requirements, and standards to ensure the interoperability of heterogeneous network systems” (pp. 233-234). The draft report of FIRP addresses many of these issues and the issues raised in the 1993 recommendation immediately following. CSL will review the FIRP recommendations and conduct other strategic studies to identify future actions that will support heterogeneous network systems. The panel anxiously awaits the development of a broader mission for the division.
OCR for page 221
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 “The Systems Network Architecture Division should address problems that users experience with proprietary (or partially proprietary) systems and networks. . . . The Systems and Network Architecture Division should help ensure that tomorrow's open networks provide access to these major systems from workstations that span networks using open and de facto standards”. . . . (p. 234). The role of proprietary and de facto standards in achieving open networks is being reviewed. The panel awaits the review conclusions. “The division should form a task force with counterparts in the Manufacturing Engineering Laboratory in order to define their separate and joint responsibilities for computer systems and networks—especially where real-time processes are involved” (p. 234). CSL and MEL are working together in support of the HPCC. Systems assurance, reliability, and security are being addressed in various aspects of the HPCC programs. Advanced Systems Divisions Mission The Advanced Systems Division develops commercial enablers for key HPCC technologies likely to lead to large new markets. These enablers are cooperatively agreed on reference materials, tests, and instruments that accelerate the commercialization of otherwise “unfocused” academic and industrial research by providing a user (application)-specific focus, “broker” two or more industry segments whose cooperation via interfaces is essential for promoting new markets, and provide the measurement and instrumentation methods necessary to “bridge” research prototypes into a commercial paradigm or demonstrate the commercial merits of two or more competing research paradigms. Strategy The Advanced Systems Division has divided its programs into (1) high-performance computer systems, (2) digital communications technology, and (3) human-machine interfaces. The primary approach to selecting projects has been bottom-up. Consultations with researchers, industrial representatives, and, in particular, ARPA principal investigators suggest important topics that either receive insufficient attention or involve issues of metrology in which the Advanced Systems Division has a comparative advantage. The Advanced Systems Division research is designed to fill the gap. Resources Total estimated fiscal year 1994 funding is estimated at $4.0 million from STRS, $0.7 million from NIST Competence Building Program funds, $0.5 million from NIST Information Technology Services funds, and $5.3 million from OA sources. In-kind equipment resources total $300,000. This funding supports roughly 90 FTE employees, both in-house and contractor.
OCR for page 222
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Assessment of Strategy The Advanced Systems Division must develop a comprehensive division strategy from which individual group and program objectives can be developed. The group's current mission, strategy, and objectives often lack focus. As a result, priorities and resource allocations are not always obvious. The issue of whether any research organization generates its work from a top-down or bottom-up perspective is eternal. On the one hand, CSL and the Advanced Systems Division can never be more than pygmies among giants in terms of resources; therefore, the division must employ a niche strategy. On the other hand, CSL and the Advanced Systems Division (together with ARPA) have unique functions (as the primary technical representative for government requirements and perhaps the best honest broker for the nation's standards effort) that dictate a broader view. The balance between the two functions has to be affected by all the activity coming under the rubric of the NII. These critical functions, in turn, should influence CSL's research agenda, particularly at the cutting edge of the technology, where the Advanced Systems Division is positioned. A top-down approach would serve both as a basis for assessing the adequacy of supporting commercial research and as a guide for the organizational location of the most urgent standards-setting activity. A top-down approach would also provide a context for evaluating user perspective when resources are allocated to competing areas of metrology. The division's desire to better support U.S. industry is increasing. As one might expect, the objectives for the Advanced Systems Division groups are considerably broader than resources allow. The challenge for division and CSL management is to prioritize the objectives, document the opportunities selected (the “best” based on mission, strategy, and priorities), and proceed with the programs selected, using available resources. High-speed networks research is a critical part of the division's work. The impending move to off-site quarters is threatening to disrupt plans for connection to the high-speed network. Even if the laboratory stays, moving the staff will also adversely affect the experimentation with the network. Careful planning is needed. The Advanced Systems Division typically works with diverse applied research topics (e.g., various forms of man-machine interfaces). A beginning-to-end testing process based on user scenarios could help prevent oversights resulting from testing a technology without a user community perspective. Such a process should be more productive and less costly than the exhaustive test procedures currently employed. Projects are dropped when they are deemed ready for maintenance (e.g., the small computer systems interface [SCSI] and FDDI). The division 's transition process should be better defined and documented. Assessment of Technical Programs The Advanced Systems Division's technical programs are conducted in a competent manner. Papers, patents, and peer interaction with other institutions such as commercial laboratories or ARPA point to the high level of technical expertise and relevance accorded Advanced Systems Division work.
OCR for page 223
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 In the Parallel Processing Group, efforts are focused in the right direction, if one accepts Multiple Instruction, Multiple Data (MIMD) as the best opportunity for advancing in high-performance computing. Given the size and history of this group, the panel expected more progress since the fiscal year 1993 assessment. The division has initiated work with parallel computer architectures (e.g., MultiKron work with MIMD). With MIMD and Single Instruction, Multiple Data as well as other new architectures on the horizon, the Advanced Systems Division must extend its experimentation with these new machine architectures. The development of compiler languages for controlling the new machine architectures needs careful attention. Such languages must not be sequentially oriented but designed to take advantage of the new architecture. Object-oriented concepts provide an alternative means of programming that is not sequentially based. Fiscal year 1993 accomplishments for the Data Storage Group were not obvious; however, the plans outlined for the panel's review seem appropriate. Again, it was difficult to assess the level of achievement of the Distributed Systems Group; however, the group is doing much to help rationalize the speech recognition industry and considerable productivity was indicated in materials the division provided for the panel. The Speech Recognition Group's continued production of new speech corpora CD-ROMs and their cooperation with researchers outside NIST, both to share software and promote benchmarking, ensure that commercial speech recognition products are being developed within a framework that is universally understood. The Image Recognition Group's benchmarking activities are appropriate; however, if the different research communities are to be better linked, there is a need to continually assess the value of developing new classification systems. Advanced Communications Group activities in the area of asynchronous transfer mode are encouraging. There is considerable value in this group's obtaining experience by actually using commercially available leading-edge technologies. The use of MultiKron technology from the Parallel Processing Group provides a good example of cross-communication between CSL groups. The Advanced Systems Division should consider several interoperability areas for additional work, such as developing a methodology for modularizing and staging complex standards (e.g., OSI, Integrated Services Digital Network [ISDN], and Ada suffer from being large and complex standards that must be implemented all at once); revisiting ISDN standards that permit a message stream (e.g., at 16 bits per cycle) over the same line to be dynamically allocated to various message types (e.g., voice, video, text, graphics); and determining technological prerequisites for proto-standards for query languages that support logical inferences and context-based (as opposed to keyword-based or index-based) text searches. Recommendations The following are the panel's recommendations for the Advanced Systems Division. The Advanced Systems Division needs a baseline architecture or comprehensive model on which to relate the various division projects. In this way, plans can clearly indicate the role and priority of each project. Also, a comprehensive model could help with staffing decisions.
OCR for page 224
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 Even though the division's metrology work appears to be highly cost-effective, the panel believes more work over the next few years should be oriented toward interoperability issues and toward developing standards derived from them. The division should carefully plan to minimize disruption by the planned move to off-campus space. The division should consider development of computer languages for controlling new machine architectures. The division should consider more beginning-to-end testing processes based on user scenarios to avoid oversights resulting from testing a technology without a user community perspective. Projects are dropped when they are deemed ready for maintenance (e.g., SCSI, FDDI). The Advanced Systems Division's transition process should be better defined and documented. The Advanced Systems Division should consider additional work in several interoperability areas as discussed above. Advanced Systems Division Responses to Fiscal Year 1993 Recommendations Given below are some of the panel's fiscal year 1993 recommendations for the Advanced Systems Division (quoted from the fiscal year 1993 assessment), with the division 's responses and panel comments. “The Advanced Systems Division should collaborate with other divisions within the Computer Systems Laboratory in developing program priorities ” (p. 237). CSL responded that planning is being driven from the CSL level, particularly through the development of HPCC plans. The CSL division chiefs meet weekly, and the CSL group managers meet at least quarterly, to discuss joint projects and concerns. Although work is under way in virtual reality, the panel thinks it needs to be defined as a CSL program, with a clear leading division, and the complementary programs in other divisions held to milestones and deliverables. “The Advanced Systems Division should collect and maintain standard databases for developing and evaluating speech recognition systems ” (p. 237). With new and increased funding in fiscal year 1994, the division will initiate additional research and development of spoken-language-understanding technology in a new applications domain by developing a prototype that provides information about library catalog holdings in response to spoken natural language queries. Data (e.g., corpora) collected in this effort will be shared with the research community. The Advanced Systems Division appears to be collecting considerable data on speech recognition systems. “The Advanced Systems Division should sponsor open competition in the development of speech and handwriting recognition technology, and the division should make better use of its special strengths and expertise to advance speech recognition research in general rather than serve
OCR for page 225
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 only a small part of the research community” (p. 237). ARPA contractors and others participate in CSL's annual benchmark tests for speech recognition, spoken language understanding, and, since the fiscal year 1993 assessment, language identification technologies. State-of-the-art work that cites these results frequently appears in publications such as those resulting from the Institute of Electrical and Electronics Engineers International Conferences on Acoustics, Speech, and Signal Processing. Corpora that NIST processes and produces are available to the research community through the Linguistic Data Consortium and in some cases also through the National Technical Information Service. A major conference focusing on information retrieval techniques, TREC-3, was planned for November 1994. At the time of this assessment, 55 groups (up from the 31 groups participating in TREC-2) were signed up to participate and to run the full benchmarking tests. Among these groups are 25 commercial firms, including IBM, Xerox, Bellcore, NEC, and TRW; most of the major information retrieval companies, including Verity, BRS, Mead Data Central, West Publishing, and Oracle; and several small businesses that have benefited immensely from TREC as a validation of effective retrieval, including Conquest Software and CLARIT Corp. Fifteen of the participants are U.S. universities, including all the major research groups in information retrieval. Fifteen are foreign universities. The division is running the competition on the Second Census Optical Character Recognition Systems Conference and may do some future work in this area with NIST funds. Data exist for a third conference, and the UNIPEN project (a unique pen-based benchmarking activity for optical character recognition) would like the division to run a competition to evaluate on-line handwriting recognition. The division is discussing a similar approach with the Federal Bureau of Investigation on face recognition system evaluation. The Advanced Systems Division appears to be well involved in speech recognition. “The division should expand its efforts to inform others about the MultiKron chip through publications, reports, and other mechanisms, e.g., professional contacts. The division should explore additional uses of the chip” (p. 237). The introduction of the virtual memory extension and Sbus interface boards will provide many new opportunities for users to devise their own uses. With the supplied software, MultiKron can be installed and then used as desired. Software links are easily within the reach of the programming community, whereas new hardware wiring is not. MultiKron, version I, showed that a number of features could be improved. The need for a collection network is not attractive, and so the new ARPA MultiKron II effort focuses on not needing one. Second, the 16 register/clocks were so popular among system developers at Intel that more were needed. MultiKron II will have virtual registers that number in the thousands; this feature allows users to access these attractive items and yet still allows the system to do fast context switches. In May 1993, ARPA began another 3-year funding cycle for this work. There are fresh avenues to be pursued in refining the device. The Advanced Systems Division does not appear to be devoting resources to publicizing and finding new uses for the MultiKron chip. The panel believes that this function may be best done by its manufacturer(s). “The division's classical optical disk metrology should be further promoted” (p. 237). Earlier work on the testing methodology to predict optical disk life expectancy values is the subject of NIST special publication (SP) 500-200, which has been widely distributed and has been the basis for subsequent standards work.
OCR for page 226
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY PROGRAMS: Fiscal Year 1994 The current data integrity (care and handling) study for the National Archives and Records Administration (NARA) will be the subject of a report to be published in the Fall 1994. NARA will be disseminating this report to the archivist community, and it will be widely available to industry and government agencies, including the optical disk standards community. (SP 500-101 on the care and handling of computer magnetic storage media was very widely distributed for 10 years and is still the subject of inquiries.) Optical disk error monitoring and reporting work is the subject of American National Standards Institute/Association for Information and Image Management (AIIM) standard MS59-199X, which was on the AIIM ballot at the time of this assessment. The division played a key role in developing the ISO/IEC version of this standard. This work is jointly sponsored by NARA and two other federal agencies. Several manufacturers have indicated an intention to produce compliant products. “. . . The division reports annually to the panel on what it can do given its expertise, funding sources, and time limitations. The division should also specify what it is not working on and why. . . .” (p. 237). CSL responded to the panel with a listing of programs it has chosen not to work on. However, the Advanced Systems Division does not appear to be systematically comparing its work agenda against its total range of choices. The panel reiterates the importance of this activity as part of the division's planning process.
Representative terms from entire chapter: