National Academies Press: OpenBook
« Previous: 9 BUILDING AND FIRE RESEARCH LABORATORY
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

10

Computer Systems Laboratory

PANEL MEMBERS

John F. Sheeran, The Boeing Company, Chair

Herbert D. Benington, Paramax Systems Corporation

Dorothy E. Denning, Georgetown University

Clarence G. Feldmann, SofTech, Inc.

James George, Mesa Graphics

Robert E. Kahn, Corporation for National Research Initiatives

Sandra M. Lambert, Citibank

Roger R. A. Morton, Eastman Kodak Company

O. R. Pardo, Bechtel/Parsons Brinckerhoff

Lawrence R. Rabiner, AT&T Bell Laboratories

Michael B. Spring, University of Pittsburgh

Raymond T. Yeh, International Software Systems, Inc.

Submitted for the panel by its Chair, John F. Sheeran, this assessment of the fiscal year 1993 activities of the Computer Systems Laboratory is based on site visits by individual panel members, a formal meeting of the panel on June 13-15, 1993, in the Gaithersburg, Maryland, facilities of the National Institute of Standards and Technology (NIST), and on the annual report of the laboratory.

LABORATORY OVERVIEW
Mission

The Computer Systems Laboratory (CSL) at the National Institute of Standards and Technology helps government and U.S. industry to increase their productive use of computer and related telecommunications systems and to improve the security management and technology of these systems. CSL works closely with the users and producers of computer and telecommunications systems to improve the competitive posture of the U.S. computer-related industry. CSL's programs are mandated by the Federal Property and Administrative Services Act of 1949 as amended by the Computer Security Act of 1987 (Public Law 100-235).

Strategy

The CSL, which includes five divisions (Figure 10.1), improves the productive and reliable use of computers by developing standards and providing advice on the planning, deployment, and use of new information technologies; improves security management and technology by developing timely and innovative solutions to security problems; and contributes to

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

FIGURE 10.1 Structure and organization of the Computer Systems Laboratory.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

U.S. industrial competitiveness by developing generic methods and techniques that facilitate technology improvement and by transferring technology to government and industry organizations.

Activities

The CSL's array of activities includes:

  • Developing standards, guidelines, profiles, tests and test methods, measurement techniques, reference materials, and prototypes;

  • Developing technical, management, physical, and administrative standards and guidelines for the cost-effective protection of security and the privacy of sensitive information in federal computer systems;

  • Providing technical advice and assistance to federal agencies in implementing standards and guidelines;

  • Collaborating with industrial users in setting standards and implementation protocol;

  • Conducting research to support technical activities; and

  • Participating in planning within NIST and across government agencies for new technology-related initiatives.

Resources

The CSL's resources include $11.2 million from congressionally appropriated funds, $18.9 million from reimbursements for services provided for other agencies, $1 million from competence programs, and $1.2 million from Information Technology Services, for a total of $31.3 million.

The division's staff consists of 237 full-time permanent employees, 17 part-time or intermittent employees, and 24 guest scientists and research associates.

CSL-WIDE FINDINGS AND RECOMMENDATIONS--FISCAL YEAR 1993

Presented below are the panel's fiscal year 1993 findings and recommendations on CSL's (1) strategic planning, (2) role in standards setting, (3) role in federal initiatives, (4) facilities, and (5) equipment.

The panel notes that CSL followed up on the panel's fiscal year 1992 recommendations regarding higher-powered computers (p. 251), improved standards-setting (p. 252), integrated services digital networks testing (p. 266), and the framework for open systems (p. 257).

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Strategic Planning
Findings
  • Some of CSL's divisions have begun to focus in their plans on the demands of external “drivers” for their work. Also, collaboration between CSL divisions is increasing, and there is enthusiasm for initiating a CSL program in electronic commerce, a relatively new technology for the use of digital technology in business transactions.

    The CSL's 1991 strategic plan does not appear to have been updated, despite the panel's fiscal year 1992 recommendation. The criteria used by most divisions for selecting projects are not clearly based on computer and telecommunications technological opportunities.

  • The panel continues to be concerned about CSL's overdependence on other-agency funding, i.e., about CSL's lack of core funding for development of standards and technology in direct response to CSL's congressional assignments. Other-agency funding provides a market test of the value of CSL's efforts to government and a window on real-world applications but cannot be relied on for long-term planning.

Recommendations on Strategic Planning
  • The Computer Systems Laboratory should adopt the strategic planning framework outlined in VCAT Annual Report 1993 (Figure 2, p. 5; Visiting Committee on Advanced Technology, National Institute of Standards and Technology, Gaithersburg, Maryland, January 1994).

  • As a prelude to strategic planning, the Computer Systems Laboratory should categorize current and planned projects as either fundamental research or applied research to determine the balance between these two categories, whatever their source of funding. If the desired ratio of fundamental to applied research is proper, the sources of funding are a secondary consideration.

CSL's Role in Setting Standards
Findings

The CSL's role in setting standards for the government's use of computers and in providing standards and leadership in computing technology for the commercial sector is increasingly constrained by external factors:

  • A few major vendors no longer dominate computing technology and standards.

  • Participants in the joint national and international process for setting standards for computing increasingly regard

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

the process as “broken”; i.e., there are no schedules, commitments, accountability, or overall priority setting or planning for setting computing standards.

  • Vendors and active users have formed consortia over the past several years to accelerate the development of standards, partially because of the inadequacy of the existing formal process.

  • Such consortia in turn are often bypassed by single-topic groups, which seek to define a standard and adopt a position quickly (often to protect the group's market position). The technology involved is moving rapidly, and many leading-edge (and some trailing-edge) issues need resolution.

  • The CSL's role in setting national and international standards for computing is as a participant in “conventions” in which participants are expected to abide by agreed-upon standards. This role is much different from NIST' s role in setting national and international standards for the measurement of physical properties in terms of fundamental constants and basic units of measure.

  • The CSL sets standards for use by the federal government for computing primarily by “rule making,” whereas industry sets computing standards by consensus in formal and consortia forums or by de facto adoption in the marketplace.

  • Despite efforts by industry and CSL to provide for open systems (interoperability) through development and adoption of joint national and international standards (open specifications openly arrived at), de facto standards still dominate the marketplace.

  • Government is now, as distinct from the 1960s and 1970s, too small a portion of the market to dictate standards for computing products and services. In fact, government-dictated standards now run the risk of being bypassed by innovations in the much larger commercial market.

  • It is relevant to note that the government clients for which CSL sets standards do not participate in evaluating CSL's performance.

The above constraints limit CSL's impact on setting standards and dictate that CSL should change its approach to influencing the setting of the computing standards government and industrial use.

Recommendations on Standards Setting
  • The Computer Systems Laboratory should determine specific needs of U.S. industry, civilian agencies, and Department of Defense agencies for its services. The market for CSL's products and services can be determined by conducting surveys, public workshops, and management interviews; otherwise, setting priorities and developing performance metrics will continue to be difficult.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
  • To minimize internal inefficiencies and maximize external value, CSL could emulate industry's increasingly customer-driven approach by inviting its government clients to participate in annual assessments of CSL's performance.

  • The Computer Systems Laboratory should arrange for increased collaboration among the major users and promulgators of computer-related standards in prioritizing, setting, and adopting standards. Such collaboration offers the best hope for achieving standards that will be in the best interest of users and will lead to a better focus on the concrete problems affecting interoperability of CSL's commercial and government clients.

  • The Computer Systems Laboratory's goal for standards setting should be interoperability among distributed, heterogeneous systems, rather than the current, more limited vision of open systems. The former goal better addresses the current mix of proprietary, de facto, and formal standards. It also offers incremental value, rather than the postponed perfection of open systems, and thus will be more readily supported by CSL's constituents. This goal has already been adopted by a few of CSL's divisions.

  • The Computer Systems Laboratory should increase its collaboration with other NIST activities, particularly with NIST's Manufacturing Technology Centers. Such collaboration would provide real-world testbeds of CSL's work and channels for disseminating working standards.

  • The Computer Systems Laboratory should develop, in collaboration with its government constituencies, measures for its success and/or failure in setting standards--e.g., vendor and user adoption of such standards.

CSL's Role in Federal Initiatives

The CSL proposes a mix of technical leadership in communications infrastructure and parallel processing, and in second-level issues such as electronic commerce and distributed multimedia in the federal National Information Infrastructure (NII) initiative. Other federal efforts that involve CSL are the High Performance Computing and Communications (HPCC) initiative and the National Research and Education Network (NREN) program.

Even in the context of uncertainty of funding, the CSL's plans lack the comprehensive assessment and aggressive tactics necessary to address factors that will surely limit the Clinton administration 's national initiatives, such as:

  • Standards for connectivity. The administration's initiatives involve “networks of networks.” The standards and rules for connectivity will have major technical, economic, and policy impacts.

  • Security across networks. Standards, procedures, and policies will be needed to provide confidentiality and integrity

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

of data, to control access, to authenticate sources, and to ensure auditability and traceability.

  • Network and system management, including user and system directories. The CSL provided leadership with its government standard for network management, but the enormous complexity of the systems and networks to be interconnected will demand major improvements in techniques, standards, and software for system and network management.

At this time, the CSL's plans and resources are not of the scale to address adequately the above limiting factors.

Conclusions on Role in Federal Initiatives
  • The panel endorses findings reported in the section “Information Technology,” pp. 19-20, VCAT Annual Report 1992 (National Institute of Standards and Technology, Gaithersburg, Maryland, January 1993).

. . . We believe that networking standards and hardware/software interoperability, largely the responsibility of the Computer Systems Laboratory at NIST, are crucial to the implementation of the great “information highways” of the future. . . . However, a clear picture of how the many disparate activities fit together is lacking. In short, we see a crucial leadership vacuum in information technology. . . . NIST has an obvious leadership function, particularly in the areas of networking standards, equipment and software interoperability and performance standards, and government computer systems specifications. . . . Unfortunately, complete fulfillment of these tasks demands considerably more effort than the Computer Systems Laboratory can muster at its current size. Consequently, we do not think NIST can credibly represent the U.S. government' s position and interests in information technology standards and networking under the current circumstances. NIST's authority and resources are simply not on a par with its natural responsibilities. We believe that this situation is one aspect of our national policy in information technology that requires attention by the Administration and the Congress.

Facilities

Current constraints on both office and laboratory space limit the CSL's plans for acquisition of staff and will inhibit the laboratory 's recruitment (funding permitting). Staff acquisition and retention are central to the laboratory's ability to accomplish its mission.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Recommendation on Facilities

The Computer Systems Laboratory should consider alternatives to its current facilities, e.g., temporary office space on campus or leased commercial space nearby, if additional space cannot be made available within NIST facilities.

Equipment

CSL's continuing lack of cutting-edge equipment and leading-edge applications is cited in several of the divisions' reviews. Equipment problems were addressed in the panel's fiscal year 1992 report (p. 251): “CSL does not have the high-computing-power, graphics-intensive personal computers and workstations that are expected to be commonplace in a 2- to 5-year time frame.”

Recommendation on Equipment

The Computer Systems Laboratory should seek to acquire cutting-edge equipment that mirrors the scale and heterogeneity of government and commercial computing.

ASSESSMENT OF DIVISION PROGRAMS
Information Systems Engineering Division
Mission and Resources

The Information Systems Engineering Division develops standards for information systems and provides technical assistance to government and industry in data administration, data management technology, computer graphics, and the validation of software standards.

In fiscal year 1993, the Information Systems Engineering Division had 38 full-time, 3 part-time permanent, and 8 intermittent employees. Funding for Scientific and Technical Research and Services (STRS) from NIST was $1.3 million, reimbursement for services to other agencies was $5.6 million, and NIST's director provided $204,000 for building technical competence. Industry provided $205,000 worth of hardware and software.

The division's broad collection of hardware consists predominantly of desktop systems. The division has a significant shortage of modern, powerful workstations, i.e., the leading-edge hardware necessary to be a leader in applying information systems technology.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Panel's Findings--Fiscal Year 1993

Personnel in the Information Systems Engineering Division were dedicated, highly motivated, and discussed their activities with candor. Their short-term tactical planning is commendable; however, some of the staff are not aware of the division's strategic plans. Researchers are self-motivated, pursue professional interests, gain support for their ideas from the division management or other agencies, and achieve worthwhile results that they publish.

Despite a nationwide preponderance of desktop computing equipment, the division ignores the needs of desktop computing users, choosing rather to address needs related to the few major government purchases. For example, the division develops standards and provides technical assistance for Sun, SGI, and VAX but develops few if any services for personal computers. Desktop systems are growing in power (if not in cost) and will become the laboratory and networked computers of tomorrow. With the division's current focus, the division-generated Federal Information Processing Standards could become irrelevant to a majority of future government computing equipment.

About 80 percent of the division's funding comes from other agencies. As a result, the division focuses on other-agency conformance testing, which consumes valuable staff time that, if core funding were available, could be assigned to CSL 's primary statutory mission and NIST's long-term goals. The division has proposed that alternative sources be used to conduct conformance testing; however, the division would still be needed as a proactive catalyst for promoting the development and practice of conformance testing.

The panel commends the Information Systems Engineering Division on its excellent support of the next-generation Information Resource Dictionary System (IRDS2). The next stage in the expansion of data administration and management involves object-oriented technology, which directly affects both data administration and management and software engineering.

Panel's Recommendations--Fiscal Year 1993
  • In the next iteration of its strategic planning, the Information Systems Engineering Division should methodically identify its markets and customers, prioritize services to be provided, appraise the division's capacity to meet customer needs, schedule delivery dates for services, and plan for the dissemination and delivery of services. Services that have decreasing or marginal utility should be identified for possible elimination or reduction. Also, the division should explore the gains from planning for joint programs with CSL's Systems and Software Technology Division.

  • The Information Systems Engineering Division should influence the entire spectrum of computing in government,

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

including desktop computing, rather than only the computing of a few large machines purchased by government. The division should become more involved in developing standards and practices for desktop computers.

  • The Information Systems Engineering Division should (1) improve the timeliness of its delivery of conformance tests (conformance tests have more impact during the ramping-up phase than at the end of the product cycle), (2) evaluate the utility of the rewritten Graphical Kernal Standard conformance test as a function of timeliness in the product cycle, (3) apply formal description languages to assist in the development of conformance tests, (4) host industry leaders in a product area in need of testing, to discuss industrial needs for conformance testing and arrange for a collaboration in developing the conformance testing methodology and tools, and (5) search for alternative sources for the development, validation, and performance of conformance tests.

  • The Information Systems Engineering Division should reevaluate its apparent strategy of seeking support for ongoing rather than new activities.

  • The Information Systems Engineering Division should develop an IRDS2 testbed.

Systems and Software Technology Division
Mission and Strategy

The Systems and Software Technology Division provides technical assistance to users of information systems and software and to the corresponding industry through promotion and development of concepts and technology for open systems. This mission and the division's seven specific tactical objectives provide much better guidance than was evident at the time of the fiscal year 1992 assessment.

The division proposes (1) to align its efforts with the national HPCC, NREN, NII, and Information Infrastructure and Technology Applications initiatives; (2) to respond to U.S. user requirements in government and industry; and (3) to provide technical support to technology development and standards making in support of U.S. industry.

The divisions's seven tactical objectives are to

  • Identify and nurture emerging information technologies;

  • Promote the convergence of standards that underpin information systems and software;

  • Build consensus among developers, vendors, and users;

  • Act as a catalyst and accelerate the development and use of information technology;

  • Maintain national and international perspectives and participation in selecting and implementing projects;

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
  • Serve as an unbiased (third-party) technical authority in the research and user communities and in the marketplace; and

  • Seize opportunities to make a difference through the development of technology and setting of standards leading to open systems.

Resources

The division's total funding for fiscal year 1993 was $4.8 million, of which $1.1 million was directly appropriated by Congress, $205,000 was from the NIST's director's competence building fund, $3.3 million (70 percent) was from other federal agencies for services rendered, and $100,000 was from the Department of Defense as compensation for conformance tests.

In fiscal year 1993, 38 of the staff were full-time permanent employees (a net increase of 7 over fiscal year 1992). Five university faculty members, five students, one guest worker, and two research associates supplemented the staff's laboratory research.

Responses to Fiscal Year 1992 Recommendations

The Systems and Software Technology Division responded well to the panel's fiscal year 1992 recommendations. The panel believes that the division is now much better organized and coordinated, although the division considered that the panel's criticism regarding “lack of coordination with other NIST groups” reflected a misunderstanding rather than a true picture. In any case, the division's interaction and coordination with other groups within NIST now include regular biweekly technical interchanges as well as cooperative technical efforts.

The panel endorses the division's strategy of using specific tactical objectives and aligning its efforts with the four national initiatives mentioned above. The emphasis on joint proposals with other CSL divisions is also commendable. However, the panel was concerned that the division did not seem to follow up on its good strategy with a process for selecting which standards and technology to work on. The needs and interests of the user community, not the talents of division staff, should be paramount in selecting the division's projects. Existing staff should be reassigned and recruited to mirror users' needs.

Long-range technology forecasting seemed to be lacking in the division, which has, in general, a short-term focus. The impression is that the division considers potential areas of investigation “bottom-up ” rather than “top-down.” That is, individual technical topics are considered and selected without evidence that the staff has given due consideration to the broad picture of the technical area. For example, the areas of multimedia and virtual reality do not seem to be as critical as

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

other areas of investigation, given the user community needs at this time.

With regard to international competitiveness, the panel believes that the four national initiatives selected by the division to guide future direction will be helpful.

Panel's Findings--Fiscal Year 1993

The Systems and Software Technology Division was reorganized after the fiscal year 1992 assessment and at the time of the fiscal year 1993 assessment had a new division chief and major groups for (1) Software Engineering, (2) Distributed Systems Engineering, and (3) Office Systems Engineering. The reorganization created (1) better communication within the division and among CSL's divisions, (2) recognition of the blurring of historical organizational boundaries, and (3) good team spirit.

Software Engineering. The Software Engineering Group improves software quality and productivity throughout the software life cycle. Rather than focus on the life cycle of a single product, the group adopts a level of abstraction that includes functional areas of different life cycles. Specifically, the group works on integrated software engineering environment, high-integrity software, and software assurance technology for telecommunications--areas that generally lack standards. The group plans a long-term effort in developing the needed definitions and standards.

The group should pay as much attention to the significant developments of the U.S. software industry as it does to those in the European Community and the Department of Defense.

Distributed Systems Engineering. The Distributed Systems Engineering Group identifies and defines sets of specifications, methods, and concepts necessary and sufficient to establish and operate large-scale, heterogeneous, distributed systems environments. The initial goal is to define standard architecture for distributed systems. Although developed for open systems, the original framework for distributed systems developed by the division can be used, revised, and expanded to describe both proprietary and open systems and the necessary interconnections. When the architecture becomes stabilized, the group plans to address the harder problem of large and dispersed “legacy” systems that must be upgraded and reengineered to accommodate newer technology. The portability and interoperability of applications in the environment of the reengineered distributed system must then be addressed.

The group must take particular care to collaborate with other groups within CSL and with U.S. industry. Also, the group

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

should remain cognizant of the changing environment for systems architecture. Of particular note are the current increased interest in open system interconnection as a model for application-layer standards and an increased reliance on standards such as those in the TCP/IP suite. Similarly, there is increased use of proprietary standards rather than internationally accepted standards.

Finally, higher priority should be placed on data processing in distributed information systems. The group's first two releases focused on portability of applications.

Office Systems Engineering. The Office Systems Engineering Group improves the quality and productivity of technologies that enable better communication of information through documents. Here, the term “documents” denotes a variety of forms including text, graphics, images, indexes, equations, tables, glossaries, signatures, and so on for storing knowledge and communicating. (Although this definition is convenient for internal purposes, the panel cautions that such a nonstandard meaning of “document” could cause major misunderstanding of the group's efforts by its customers and fellow professionals.)

The group faces the dilemma of working simultaneously on mundane but critical problems, such as document interchange, and on the more challenging and innovative technologies involved in electronic data processing. The dilemma points to the importance of a clear vision and strategic plan in helping group managers to assess relative merits of their various research, measurement, and standardization projects. The group's initial priorities in electronic documentation are multimedia and hypermedia, virtual reality, and image quality.

This group has exceptionally strong technical expertise but lacks a comprehensive strategic plan. Also, the specific issues to be investigated are dictated largely by the availability of other-agency funding rather than by critical national user needs. Standardization of office systems will be critical over the next decade; therefore, it is important that the limited resources available for work in this area be carefully allocated to achieve the greatest return on investment.

Division-wide Issues

  • The high level of other-agency funding clearly indicates the worth of the division's services to other federal agencies; however, division managers must exert continual vigilance to ensure that their services for other agencies also contribute to the division's, CSL's, and NIST's mission and long-term goals.

  • The North American Portable Common Tool Environment (PCTE) standard was developed under the European Computer Manufacturers Association (ECMA) Technical Committee 33 for

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

software engineering. The division's PCTE initiative runs the following risks:

  1. Backing a perspective on software engineering that does not reflect current technology;

  2. Lacking prior review and support by U.S. industry, in particular providers of software tools, who had no opportunity to vote on the ECMA standard; and

  3. Giving market advantage to European software tool providers.

  • A more aggressive plan is needed for expanding the division's staff.

  • The division continued its series of Integrated Software Engineering Environments (ISEE) workshops to develop a reference model and tool interface specifications for fully integrated software engineering environments that support software products and processes throughout the software life cycle. In the opinion of the panel, the ISEE, as currently envisioned, has the following shortcomings:

  1. It focuses too much on PCTE, a software engineering standard from ECMA.

  2. It lacks guidelines for definition, simulation, and enactment; process language; and process integration technology. Certification issues with the Capability Maturity Model and International Organization for Standardization (ISO) 9000 remain.

  3. It relies on PCTE, even though the common object request broker architecture (CORBA) is much more likely to be commercially successful. (CORBA already proposes to handle objects across heterogeneous distributed systems.)

  4. It overlooks the fact that PCTE is less important to the generic data integration issue than is Common Schema.

  5. It does not develop such ISEE needs as control and integration of data integration, process control, and guidelines.

  6. Data repository issues do not appear to be sorted out between the Information Resource Dictionary System, PCTE, and other standards.

  • Also missing from the division's plans was total quality management.

Panel's Recommendations--Fiscal Year 1993
  • The Systems and Software Technology Division should take a more active role in developing the U.S. and international standards definitions of the neutral (meta) data formats (e.g., CASE Data Interchange Format, the conceptual schema/modeling facility, and STEP/Express), all addressing the problem of transferring and storing (in IRDS2) data and software definitions.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
  • The Systems and Software Technology Division should play a more proactive role in the ISO 9000 and Software Engineering Institute's Capability Maturity Model assessment, rather than merely following the progress of these two important standards-setting activities.

  • The division should provide a significant, positive leadership role in mapping the Capability Maturity Model to the division's understanding of the Generic Life Cycle Model.

  • The Systems and Software Technology Division should investigate the use of fellows from industry, contractors, and part-time retired NIST staff. The division lacks critical core competencies needed for staffing national initiatives being discussed.

  • The division should monitor the adequacy of its equipment continuously, even though its current facilities are adequate.

Computer Security Division
Mission

As mandated by the Computer Security Act of 1987, the Computer Security Division (CSD) primarily serves the federal government by providing standards, guidelines, training, and technical assistance as needed to protect sensitive but unclassified federal information. The act does not charge NIST to serve the private sector, except to provide assistance in using and applying the results of NIST's programs.

However, because NIST is charged by the Omnibus Trade and Competitiveness Act of 1988 to enhance the competitiveness of U.S. industry and to assist private-sector initiatives in capitalizing on advanced technology, serving commercial interests is becoming a major function of NIST; therefore, the CSD's efforts to respond to the needs of industry as well as government are in direct support of NIST's expanding function.

Strategy

Each group within the CSD has a strategic plan with varying degrees of detail. The divisional strategic plans and reviews of objectives, milestones, schedules, and status ensure forward thinking, satisfy management's stewardship responsibility, and document significant achievements.

The CSD is developing cryptography-based security technology; federal and international security criteria; a computer security handbook; security incidence prevention and response; network security, network risk analysis, and network management; and computer virus control.

Projects, including those requiring technical research, are selected based on the needs of federal agencies.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

The staff collaborates extensively with people in other agencies and the private sector to obtain leverage from other work; to develop prototypes, standards, and guidelines that are responsive to government needs; and to affect the development of national and international standards that satisfy U.S. government requirements. Collaboration with government users in the development of prototypes, standards, and guidelines ensures that the division's results are relevant.

Resources

The Computer Security Division is severely understaffed and underfunded given its statutory security responsibilities, the growing national recognition of the need to protect unclassified but sensitive information, and the unique role the division can play in fostering security in commercial architectures, hardware, and software.

The division's computing hardware appears to be adequate for existing projects. However, limited space would make it difficult to hire new staff if funding were to be increased.

Panel's Findings--Fiscal Year 1993

The CSD has prime responsibility for NIST's computer security activities; however, there is also a small program managed by the CSL's associate director for computer security as well as security-related activities in other CSL divisions.

The Computer Security Division makes full use of work of others, establishes cost-sharing partnerships, transfers routine work externally, and influences others to respond to the division's objectives.

Noteworthy achievements of the division in cryptographic-based security technology since the fiscal year 1992 assessment include the following:

  • The Digital Signature Standard will be released as a Federal Information Processing Standard (FIPS) pending a patent resolution;

  • The Secure Hash Algorithm was published as a Federal Information Processing Standard (FIPS 180);

  • Security requirements for cryptographic modules (FIPS 140-1) were drafted;

  • Security guidelines for local area networks were drafted and will soon be released for public comment;

  • Prototype “smart” cards for authentication and digital signatures were developed;

  • An advanced smart card access control system was developed under sponsorship of the Advanced Research Projects Agency (ARPA); and

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
  • The division sponsored extensive participation in national and international standards activities.

The smart cards are an advanced technology and are in demand by government agencies. The Computer Security Division received four awards for development of these cards.

Another noteworthy achievement is completion of draft federal criteria for computer security. This was followed by initiation of a major collaborative effort with other countries to develop common criteria. The common criteria will pull together and supersede the federal criteria, European international criteria, and Canadian criteria and will give vendors the advantage of having to meet only one set of security criteria. The federal criteria, together with comments received on them, will be a major source of input for developing the common criteria.

Although the impact of the federal and common criteria on bringing secure products to market has yet to be determined, NIST's contributions to developing the criteria have received considerable recognition. Many participants are optimistic that the new criteria will allow for a shorter assessment cycle, especially for products requiring lower levels of assurance, than do the existing federal criteria.

The division participated, as secretariat, in the Forum of Incident Response and Security Teams, an effort aimed at helping government and nongovernment organizations to deal effectively with computer break-ins, viruses, and other threats.

Considerable progress was made toward publishing a computer security handbook and framework that will enable agencies to identify, assess, and control secure computer applications and communications.

Significant progress was made in support of developing security services and interfaces in all areas of open system environments.

NIST makes its work products available electronically through a dial-up bulletin board and the Internet. These products provide a valuable service to the public as well as to other federal agencies.

Future Directions, with Panel Comments

In the cryptography area, the Computer Security Division would like to develop standard services and interfaces for authentication, single log-on, distributed certification authority systems, and standards and guidelines for validating cryptographic systems and environments. All of these areas build on current work and are appropriate for the division's to pursue.

In the area of federal criteria and common criteria, the division wants to develop (1) a “trust technology” assessment program for evaluating computer security and products, (2) an

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

evaluation methodology for composite or integrated systems, and (3) infrastructure and guidelines for in situ accrediting systems.

The panel endorses a research project being considered by the division to develop guidance for securely handling data and software in distributed systems. The research would tie in with that of CSL's Systems and Software Technology Division and the Information Systems Engineering Division. The CSD also proposes research on using audit analysis for assessing intrusion detection systems.

The Computer Security Division is currently counseling the White House on security for medical information as part of the White House 's development of a national health care program.

Panel's Recommendations--Fiscal Year 1993
  • The Computer Security Division should develop criteria for determining how much of a networked system must be trusted. Much of this work would tie into the security needs of the National Information Infrastructure initiative.

  • Since several research projects on detection of intrusion are well under way outside NIST, the division should rely on external research rather than start an independent project.

  • The Computer Security Division should continue to consult, develop guidelines, and provide training programs, especially in computer security risk management and analysis, and should complete or upgrade its computer security handbook.

  • The Computer Security Division should consider expanding its activities in ensuring the security of medical information.

  • The division should take a much more active role in planning for research on ensuring overall National Information Infrastructure security requirements.

Systems and Network Architecture Division
Mission, Strategy, and Resources, with Panel Comments

The Systems and Network Architecture Division develops Open Systems Interconnection (OSI) technology and standards, develops and applies automated protocol methods, and develops technology for integrated, interoperable network management. The division views its mission as bounded by protocols for open systems interconnections.

Even though the division's comprehensive draft strategic plan for October 1993 through 1998 was well structured, was informative, and provided an excellent basis for operation, the plan is being outdated by NIST's expanding mission. The division's emphasis on “getting closer to the customer” was a positive step in fiscal year 1993 that should be

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

institutionalized, especially in the development of interoperability across heterogeneous networks.

During fiscal year 1993, the division received $1.1 million in directly appropriated funds from Congress, $3.0 million in reimbursements from other agencies, and $170,105 in equipment loans from industry. The staff consists of 46 employees.

The division has a competent staff and--because of its involvement in testing for interoperability--has, for the most part, state-of-the-art equipment. The division works primarily (73 percent) for other agencies. The division could benefit from more varied and flexible arrangements for cooperation, e.g., cooperative arrangements such as those used in the Technology Reinvestment Program of the U.S. Department of Defense, in addition to Cooperative Research and Development Agreements. The panel endorses the division's plan for inviting guest researchers.

Response to Panel's Fiscal Year 1992 Recommendations

The panel's fiscal year 1992 report (pp. 256-257) recommended by inference that (1) the CSL (i.e., the Systems and Network and Architecture Division) should evaluate the relative merits of industry's de facto standards and protocol for independent interfaces as well as network interoperability, thereby providing vendor-neutral input needed to best pursue coexistence and transition (i.e., issues related to OSI, TCP/IP, systems network architecture, and other major networking architectures); (2) the division should plan for defining standards for the new NREN, for a major upgrade in national network capability, and for developing new mechanisms for collaboration with other major participants in the NREN; and (3) the division should provide needed leadership for OSI network management in the United States through increasing the resources of its Common Management Information Protocol Network Management Laboratory.

Recommendation (1) was partially addressed through participation in the Internet Society's development of standards and specifications and in the Open Management Roadmap partnership for developing network management specifications (Open Management Network Interoperability Points) to satisfy user priorities for products that vendors agree to supply. The protocol, standards, and specifications need to be developed at a faster pace. With reference to recommendation (2), the definition of standards for the NREN was deferred until fiscal year 1994, pending the allocation of funds. With reference to recommendation (3), the division decided that the current resources of its Common Management Information Protocol Network Management Laboratory were adequate given the division's involvement in workshops, consortia, and the Network Management Forum.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Panel's Findings--Fiscal Year 1993

The Systems and Network Architecture Division has good national and international credibility. Its customers are correctly identified as users and vendors of networking technologies, and the division is active in both the OSI and Internet arenas. Within the scope of nonproprietary networks with open systems interconnections, the division has correctly identified areas for future activity. The division maintains contact with its customers, helps develop consensus among users and vendors, aids the standards-setting process, defines missing elements in the development and use of the technology, and provides the test definition and support called for in its mission.

Highlights of the Systems and Network Architecture Division's achievements during fiscal year 1993 were as follows:

  • The division completed the draft Industry/Government Open Systems Specification, which harmonizes the profile specifications of the Government Open Systems Interconnection Profile (GOSIP), Manufacturing Automation Protocol and Technical and Office Protocol, Unified Communications Architecture, and Canadian GOSIP.

  • Division staff joined vendors and users in an Open Management Roadmap partnership to develop network management specifications called Open Management Network Interoperability Points (OMNI Points) that freeze specifications as prioritized by the users and according to which vendors agree to build products. Such OMNI Points tend to minimize the risks to vendors and users. Two major network management protocols--the Systems Network Management Protocol and the Common Management Information Protocol--have grown out of the Open Management Roadmap partnership.

  • The division sponsored well-attended industrywide laboratory tests for OSI dynamic routing and frame relay.

  • The division provided staff support for the electronic mail and directory services interface standards of the Institute for Electrical and Electronics Engineers.

Other notable achievements reported by the division for fiscal year 1992 included the continued publication of guidelines for use of OSI applications, support of the GOSIP testing program, work on helping the Internet Society to resolve the TCP/IP addressing problem, and the use of ESTELLE (a formal specification language) to evaluate the Network Security Protocol Specification.

The division plans to emphasize developing a model for open communications protocols that draws on both the TCP/IP and OSI standards; assisting federal agencies in the use of electronic document interchange and promoting its integration into open systems; promoting standards for standardizing electronic transactions; developing models for electronic commerce--starting

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

with electronic data interchange; analyzing the necessary infrastructure standards, services, and products; and working with other agencies to develop model applications.

Based on its experience in GOSIP certification and other forms of testing against standards, the division is considering initiating a program in automated testing that uses ESTELLE as a tool for generating code for implementing protocols from formal definitions of requirements. Currently, much money and time are spent developing manual test suites and manually running them.

Other opportunities for the division include working with federal agencies to deploy large X.400 and X.500 (electronic data transmission protocols) applications; collaborating with industry in defining the barriers to and interface requirements for interoperations involving de facto standards and proprietary networks; and defining the protocol requirements for NREN and NII networks in the near term, medium term (5 to 10 years), and long term (10 to 20 years). Also, the division is well suited to become a demonstration site of OSI functionality among its own workstations.

In its close associations with industry, the division is a full partner, as is illustrated by the division's linkage with industry and users through the Network Management Forum.

Panel's Recommendations--Fiscal Year 1993
  • A division-wide skills assessment is in order if the Systems and Network Architecture Division is to address a broader network mission.

  • In the area of electronic commerce, the Systems and Network Architecture Division should focus on only a few, well-defined pilot projects, because of the large scope of electronic commerce.

  • The division should develop a program that focuses on automated standards testing, explores the use of formal definitions, and supports the development of tools such as ESTELLE and commercial automated test generators.

  • The Systems and Network Architecture Division's exemplary strategic plan should be revised to keep pace with NIST's expanding mission and requirements related to its role in developing the National Research and Education Network.

  • The Systems and Network Architecture Division should move beyond viewing its mission as bounded by open (i.e., nonproprietary) protocols. The division should retain its current internal Open Systems Interconnection (OSI) focus but should also pioneer in the promotion of OSI for use in the United States. The division should provide broad support for its customers in defining, operating, and evolving heterogeneous network systems. This approach should include refinement of open system network concepts and continued development of associated supporting concepts, such as distributed systems networking, specific frameworks for addressing user requirements, and

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

standards to ensure the interoperability of heterogeneous network systems.

  • The Systems and Network Architecture Division should address problems that users experience with proprietary (or partially proprietary) systems and networks. Many of the major government systems are legacy systems developed on proprietary communications architectures. The Systems and Network Architecture Division should help ensure that tomorrow's open networks provide access to these major systems from workstations that span networks using open and de facto standards. These same problems confront U.S. industry and academia, and the solutions will dramatically affect overall national productivity.

  • The division should form a joint task force with counterparts in the Manufacturing Engineering Laboratory in order to define their separate and joint responsibilities for computer systems and networks--especially where real-time processes are involved.

Advanced Systems Division
Mission, Strategy, and Resources, with Panel Comments

The Advanced Systems Division conducts research and provides technical assistance to federal agencies and industry organizations in advanced communications such as integrated services digital networks, distributed systems, automated recognition, data storage technologies, and parallel processing. The division's goals are (1) to accelerate the commercialization of otherwise unfocused academic or industrial research, (2) to broker essential cooperation among industrial segments, and (3) to provide measurement and instrumentation methods needed for commercialization.

The Advanced Systems Division's plans anticipate major participation in the multiagency HPCC program as described in the High-Performance Computing Act of 1991 and the Information Infrastructure and Technology Act of 1992 introduced by then-U.S. Senator Albert Gore.

Given that the division's three general goals are appropriate and clearly within the scope of CSL's mission, the division's performance should be appraised in the context of those goals. For example, significant efforts were made to develop handwriting recognition technology through the development of a corpora for testing. As a first major test of the corpora, the work of one participating vendor, much of which was not made public because of its proprietary nature, showed a recognition success rate of 98.5 percent of corpora input. No other systems successfully recognized more than 97 percent, and NIST systems scored no higher than 95 percent. In view of documented technical achievements prior to the division's initiative, the panel 's concern is whether the Advanced Systems

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×

Division's effort adequately addresses the division's goal of accelerating the commercialization of otherwise unfocused academic or industrial research. Even so, development of a corpora for testing handwriting technology provides reliable, baseline information.

The division's stated strategy is to focus its contributions on the convergence of computation, communication, and display technology. However, it is not clear to the panel what this focus contributes to selecting particular projects.

The division's research in handwriting and speech recognition can easily be related to human-computer interfaces; however, if this is the purpose, it is less clear why gesture input, “eyephones,” and a myriad of other human-computer input and output technologies are not also examined. Also, CSL's virtual reality research is housed in the Systems and Software Technology Division, but there is much overlap with the mission of the Advanced Systems Division.

The panel had difficulty in assessing the relevance and relatedness of the Advanced Systems Division's various projects as described in the final draft of Implementation Plan for High Performance Computing and Communications/Information Infrastructure Budget Initiative for Fiscal Year 1994 (National Institute of Standards and Technology, April 16, 1993). In contrast, the division's projects as outlined in a 1991 joint planning document of the Advanced Systems Division and the Computing and Applied Mathematics Laboratory were coherent and relevant. The division's strategic planning process is plagued by often delayed or uncertain funding and a budget that is more than 60 percent dependent on other-agency funding.

During fiscal year 1993, the Advanced Systems Division received $2 million in STRS funding, $3.7 million from other agencies, $639,000 from the NIST's director's competence building fund, and reimbursements of $1 million for technical services to support a staff of 74.

Resources available to the division in terms of equipment and personnel are appropriate. The division has a reasonable array of equipment, which may be characterized as being on the cutting edge. More importantly, state-of-the-art work is being carried out on the equipment. The division has benefited over the last several years from a demonstration personnel project authorized by Congress that allows NIST to more easily recruit and retain well-qualified personnel.

Response to Fiscal Year 1992 Recommendations

In its fiscal year 1992 report, the panel made no specific recommendations regarding the Advanced Systems Division.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Panel's Findings--Fiscal Year 1993

The Advanced Systems Division is having a positive impact on the marketplace by encouraging new technologies as well as by developing and disseminating test metrics and methodology. The division's scholarly reports provide significant impetus to its counterpart research communities.

The panel reviewed the following: (1) parallel processing and performance measurement, (2) corpora for speech and handwriting, (3) corpora for text retrieval, (4) optical disk work, and (5) work on integrated services digital networks and distributed systems.

  1. The parallel processing and performance measurement project has been focused over the last few years on developing advanced measurement devices (e.g., MULTIKRON). While MULTIKRON began as a means for measuring the performance of parallel processing systems, it appears to be applicable for unobtrusive in situ measurements for industrial process control. MULTIKRON's design is simple, elegant, easily extended, and suited for multiple applications. The potential scope of MULTIKRON 'S application may far exceed the applications proposed or imagined by the division staff.

  2. The Advanced Systems Division provides a clearinghouse for speech recognition databases obtained either through contracts with industry or from ARPA programs. These databases are distributed on CD-ROMs, for which the division contracts production. Currently, only ARPA contractors and a newly formed private corporation, the Linguistic Data Consortium, have access to the division's databases. Unfortunately, since the division's Speech Group is funded primarily by ARPA, there is little incentive to satisfy or examine the needs of speech researchers outside ARPA.

  3. The division's corpora and competitions in text retrieval address a mature technology. While the development of testbeds for a 40-year-old technology would not be easily justified, there is evidence that new retrieval techniques are being developed. Other retrieval issues beg for attention; therefore, testbed development requires a strong justification. For example, image retrieval and intelligent electronic text conversion are not being addressed by the division, are as important and difficult as current projects within the division, and have equally challenging technical problems.

  4. Data preservation is of concern to every community working in information technology. The division's efforts in optical technology, which has been heralded as a significant step in developing stable, long-term storage media, are of strategic importance to data preservation; however, the division's efforts to quantify and measure the stability of other emerging storage media, e.g., magnetic and electrostatic media, are of equal importance.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
  1. Deployment of integrated services digital networks (ISDNs) is relatively recent; however, the related standard has existed in one form or another for many years. The Advanced Systems Division is correctly shifting its efforts toward higher-bandwidth and more robust protocols (e.g., broadband-ISDN, asynchronous transfer mode). Its examination of wireless communication needs more planning. Exploration of the integration of the base technologies and their impact on available bandwidth is encompassed by the charge to the Advanced Systems Division.

Similarly, understanding distributed systems is critical to the success of the HPCC initiative. The Advanced Systems Division is only one of the CSL divisions addressing the HPCC initiative.

Panel's Recommendations--Fiscal Year 1993
  • The Advanced Systems Division should not overemphasize its involvement in the High Performance Computing and Communications initiative but rather should concentrate on its mission.

  • The Advanced Systems Division should collaborate with other divisions within the Computer Systems Laboratory in developing program priorities.

  • The Advanced Systems Division should collect and maintain standard databases for developing and evaluating speech recognition systems.

  • The Advanced Systems Division should sponsor open competition in the development of speech and handwriting recognition technology, and the division should make better use of its special strengths and expertise to advance speech recognition research in general rather than serve only a small part of the research community.

  • The division should expand its efforts to inform others about the MULTIKRON chip through publications, reports, and other mechanisms, e.g., professional contacts. The division should explore additional uses of the chip.

  • The division's classical optical disk metrology should be further promoted.

  • The panel's biggest concern about the Advanced Systems Division's efforts is what it cannot do rather than what it is doing. The division reports annually to the panel on what it can do given its expertise, funding sources, and time limitations. The division should also specify what it is not working on and why. The division management began to address these questions for fiscal year 1993.

Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
This page in the original is blank.
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 213
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 214
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 215
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 216
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 217
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 218
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 219
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 220
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 221
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 222
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 223
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 224
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 225
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 226
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 227
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 228
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 229
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 230
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 231
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 232
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 233
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 234
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 235
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 236
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 237
Suggested Citation:"10 COMPUTER SYSTEMS LABORATORY." National Research Council. 1994. An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993. Washington, DC: The National Academies Press. doi: 10.17226/9192.
×
Page 238
Next: 11 COMPUTING AND APPLIED MATHEMATICS LABORATORY »
An Assessment of the National Institute of Standards and Technology Programs: Fiscal Year 1993 Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!