At the request of a consortium of federal government agencies, the Committee on Virtual Reality Research and Development was established to provide guidance and direction on the allocation of resources for a coordinated federal program in the area of virtual reality. In responding to this charge, the committee has included both virtual environments and teleoperation in its assessment of the field.
This report includes recommendations and extensive background material concerning systems popularly referred to by such terms as virtual reality,1cyberspace, virtual environments, teleoperation, telerobotics, augmented reality, and synthetic environments. In all such systems, the basic components are a human operator, a machine, and a human-machine interface linking the human operator to the machine.
In a teleoperator system, the machine is an electromechanical tool containing sensors and actuators (i.e., a telerobot) that effectively extend the operator's sensorimotor system and thereby allow him or her to sense and manipulate the real environment in new ways. In a virtual environment (VE) system, the machine is an appropriately programmed computer
that generates or synthesizes virtual worlds with which the operator can interact. Whereas the purpose of a teleoperator system is to sense and transform the real world (as in removal of hazardous waste by teleoperation), the purpose of a virtual environment system is to alter the state of the human operator or the computer (as in the use of virtual environment systems for training, designing, marketing, or scientific modeling). In many systems, such as teleoperator systems that make use of virtual environment systems to help plan future actions, teleoperator systems and virtual environment systems are combined. In an augmented-reality system, the operator's interaction with the real world (either directly or via a teleoperator system) is enhanced by overlaying the associated real-world information with information stored in the computer (generated from models, derived previously from other sensing systems, etc.). In general, we refer to all systems of the types just described as synthetic environment (SE) systems.
Virtual environment systems differ from traditional simulator systems in that they rely much less on physical mock-ups for simulating objects within reach of the operator and are much more flexible and reconfigurable. Virtual environment systems differ from other previously developed computer-centered systems in the extent to which real-time interaction is facilitated, the perceived visual space is three-dimensional rather than two-dimensional, the human-machine interface is multimodal, and the operator is immersed in the computer-generated environment.
In recent years, synthetic environment systems, particularly virtual environment systems, have generated both great excitement and great confusion. These factors are evident in the extensive material published in the popular press; in the unrealistic expectations on the part of the public; in the inadequate terminology being used; in the deluge of conferences, articles, books, and demonstrations occurring; in the difficulties being experienced in communicating across disciplinary boundaries even by individuals whose professional work lies within the domain of synthetic environment systems; and in the frenetic pace at which most of the individuals concerned with synthetic environments are working.
In this book, we attempt to describe the current state of research and technology that is relevant to the development of synthetic environment systems, provide a summary of the application domains in which such systems are likely to make major contributions, and outline a series of recommendations that we believe are crucial to rational and systematic development of the synthetic environment field. Inasmuch as the ''bottom line" of the committee's work is our recommendations (presented in the final section of the overview), the remainder of this executive summary focuses on these recommendations. They are summarized under
In discussing these areas, it should be noted that the recommendations have not been prioritized in any detailed manner. This is due primarily to our judgment that successful development and application of SE systems depends on an entire matrix of interrelated factors. We nevertheless feel that it is important to stress the crucial need for improved hardware technologies to enable development of improved interface devices and improved computer generation of multimodal images. Unlike the situation in the area of teleoperation, in the area of VEs there are relatively few individuals who have primary interests or backgrounds in hardware; most individuals in the VE area are involved primarily in the software end of computer science, in communication or entertainment media, and in human perception and performance. Thus, the importance of adequate hardware, without which the VE field will never come close to realizing its potential, tends to be underplayed by the VE community. A somewhat similar comment concerns the issue of user comfort. To date, a very large fraction of VE usage has occurred in the context of short demonstrations, a context in which the degree of comfort is relatively unimportant. However, if the comfort of VE systems (particularly head-mounted displays) cannot be radically improved, the practical usage of these systems will be limited to emergency situations or to very short time periods. In other words, adequate comfort, as well as technically adequate hardware, are essential to realizing the potential of the SE field.
Finally, it should also be noted that our thoughts about government policy and infrastructure are stated as comments and suggestions rather than as recommendations. They are based solely on the experience and judgment of the committee members.
Significant research and development is taking place in a wide variety of application domains, and in some cases the results of this work are beginning to be applied on an experimental basis. Although it is not yet clear which tasks will eventually gain the most from the use of SE systems, the committee has identified four application domains that show particular promise: (1) design, manufacturing, and marketing; (2) medicine and health care; (3) hazardous operations; and (4) training.
Other important application domains that are assigned lower priority are education, information visualization, and telecommunications and teletravel. The application domain of education is of critical concern;
however, in our judgment, the overwhelming issues in this domain are social, political, and economic rather than technological. Because the domains of information visualization and telecommunications and teletravel cut across all others, we expect them to receive substantial attention in connection with work that is primarily addressed to the other domains. Training has the same cross-cutting property; however, it was judged to be so important and so well matched to the technology that it was nevertheless given high priority.
Because human beings constitute an essential component of all SE systems, there are very few areas of knowledge about human behavior that are not relevant to the design, use, and evaluation of these systems.
The committee recommends that work in this area be organized around the following objectives: (1) development of a comprehensive review of theory and data on human performance characteristics from the viewpoint of synthetic environment systems; (2) development of a theory that facilitates quantitative predictions of human responses to alterations in sensorimotor loops; (3) development of cognitive models that facilitate effective design of VE systems for purposes of education, training, and information visualization; and (4) development of improved understanding of possible deleterious effects of spending substantial portions of time in synthetic environments.
Despite the enthusiasm and the "hype" surrounding the SE field, there is a substantial gap between the technology that is available and the technology that is needed to realize the potential of SE systems envisioned in the various application domains. Two partial exceptions are the virtual environment technology used in the entertainment industry and the teleoperator technology used for hazardous operations such as waste removal. For most applications to be truly successful, however, the development of substantially improved technology is a major requirement. In our review, we divide the relevant technology into four general categories: human-machine interfaces, the computer generation of virtual environments, telerobotics, and networks.
Human-machine interfaces for SE systems include all the devices used to present information to human users or to sense the human actions or
responses that control the machine in question. In this book we examine the needs for human research studies and technology development for the visual, auditory, and haptic channels; for whole body motion and locomotion displays; for position tracking; and for speech communication, physiological, olfactory, and gustatory interfaces. After careful analysis we have determined that the most important research and development needs in the interface area concern the visual channel, the haptic channel, locomotion displays, and position tracking, and we make specific recommendations on these topics accordingly.
Computer Hardware and Software
It is computer hardware and software that produce virtual environments. Technology should be capable of generating such environments in a way that makes them appear convincingly real to human users and that allows them to interact with the environments in real time. With available technology, however, there is a major trade-off between realistic images and realistic interactivity.
Hardware requirements for virtual environments include very large physical memories, multiple high-performance scalar processors, high-bandwidth mass storage devices, and high-speed interface ports for various input and output peripherals. In the committee's judgment, commercial market forces, if they continue to grow at the current rate, will probably be sufficient to support the needed development. Therefore, the committee recommends no aggressive federal involvement in computer hardware development in the SE area at this time. Rather we conclude that hardware development remain largely a private-sector activity. Should serious lags in development occur, the government might then consider strategies for leveraging private-sector development efforts.
Software requirements are such that a major unified research program, focusing on the generation, implementation, and application of virtual environments, should be undertaken. The basic topics that need to be considered in such a program include: (1) multimodal human-computer interactions, (2) rapid specification and rendering of visual, auditory, and haptic images, (3) models and tools for representing and interacting with physical objects under multimodal conditions (including automated model acquisition from real data), (4) simulation frameworks (5) a new time-critical, real-time operating system suitable for virtual environments with relatively simple input/output requirements, (6) registration of real and virtual images in augmented reality applications, (7) navigational cues in virtual space, (8) the behavior of autonomous actors, and (9) computer generation of auditory and haptic images. Because the natural tendency of computer scientists to concentrate on graphics will
focus sufficient attention on the visual channel, special attention must be given to the modeling and generation of auditory and haptic images and to the needs associated with integrating the different modalities in virtual environment systems.
In many ways, research issues in teleoperation are similar to those in virtual environments. Independent of the purpose for which a system is being designed (e.g., to train an operator, to remove hazardous waste) and independent of whether the relevant environment is real or virtual, both are concerned with the design, construction, and application of multimodal, immersive systems that enable the operator to interact usefully with some structured environment. Because of these similarities and the relatively long history of research in the teleoperation area, results in teleoperation can be usefully exploited in the virtual environment area. Concerns unique to teleoperator systems relate to the design and performance of the complex electromechanical systems referred to as telerobots and the unavoidable time delays that arise in communicating between the human-machine interface and the telerobot when these subsystems are separated by large distances. Such communication delays can result in degraded or unstable teleoperator performance.
Improved teleoperator systems require improved control algorithms and methods for constructing and using predictive displays and for realizing effective supervisory control techniques as strategies for combatting communication time delays. Hardware requirements include: (1) multiaxis, high-resolution tactile sensors to provide telerobots with an adequate sense of touch; (2) robot proximity sensors for local guidance prior to grasping, (3) multiaxis force sensors to measure net force and torque exerted on end effectors, (4) improved actuator and transmission designs for high-performance joints, and (5) real-time computational architectures. Also, since many new problems arise when a human is interfaced to a microtelerobot, research is needed to capitalize on the advances now being made in the field of microelectromechanics. Similarly, research is needed to explore possibilities and problems associated with the development and application of distributed telerobots (macro and micro).
Communication networks have the potential to transform virtual environments into shared worlds in which individuals, objects, and processes interact without regard to their location. In the future, such networks
will allow us to use virtual environments for such purposes as distance learning, group entertainment, distributed training, and communication among telerobots in diverse locations. Although the technology is becoming able to support the development of distributed virtual environments, it is currently insufficient to support multiple users and multiple modes of input in real time. Other problems to be resolved are network host-interface slowdowns caused by the multiple layers of operating system software and the high cost of purchasing time on high-speed, wide-area networks.
We anticipate that in the future most virtual environment applications will rely heavily on network hardware and software. Because several forces in the federal government and in the private sector are driving the major advances in hardware, we do not advise additional investment in network hardware development at this time. We do propose, however, that the federal government provide funding for a program (to be conducted with industry and academia in collaboration) aimed at developing network standards that support the requirements for implementing distributed virtual environments on a large scale. Furthermore, we propose funding of an open VE network that can be used by researchers, at a reasonable cost, to experiment with various VE network software developments and applications.
In general, SE technology and SE systems are not being adequately evaluated. Admittedly, the evaluation task is complex: it involves considerations of many disciplines, both whole systems and individual components, a wide variety of component technologies, and many different types of evaluation goals. Nevertheless, we believe that if the SE field is to progress beyond the stage of demonstrations, serious evaluations are crucial. They are needed not only for estimating overall cost-effectiveness, but also for analyzing performance in terms of the contributions made by different component features and thereby guiding the directions of future research and development.
Developers of SE systems should conduct evaluations using a variety of approaches throughout the development process, and the federal government should therefore encourage the developers whose work it supports to include a comprehensive evaluation plan in the design stages of their research and development projects. The federal government should also help coordinate the development of standardized testing procedures for use across studies, systems, and laboratories, particularly in areas in which the private sector has not been active.
Government Policy and Infrastructure
Because the field of synthetic environments is in its very early stages, the federal government has a rare and important opportunity to foster careful planning for its research and development. In this section, we discuss some mechanisms that we think federal agencies should consider as part of their strategic planning for the research and development agenda in the area of synthetic environments.
First, a national information system that provides comprehensive coverage of research activities and results on synthetic environments in a user-friendly way to a wide variety of users could be a useful tool for promoting cross-fertilization and integration of the research and development efforts. Such a system could serve as a repository of text, data, computational models, and software and could include effective subsystems for both retrieval and dissemination.
Second, federal agencies might fruitfully consider establishing a small number of national research and development teams, each focusing on a specific application. These teams could involve government, industry, and academia, as well as the various disciplines relevant to the given application. Funding could be provided jointly by both the federal government and the private sector.
Third, it might be useful for some federal agencies and offices to explore the use of synthetic environment technology to meet their own administrative and program needs. In addition to the application of synthetic environments to the defense and space programs already under way, other application domains, such as training, telecommunication and teletravel, and information visualization, are relevant to the activities of many agencies. One way for the government to facilitate the development of the SE field would be to select a few agencies to serve as test beds for synthetic environment technology in these general domains.
Fourth, although it is probably too early in the development of synthetic environment systems to establish standards and regulations, it is not too early for the federal government to begin to evaluate the work already under way in connection with the telecommunications and entertainment industries. Problems that are already of concern and are likely to increase as the field develops relate to technological compatibility issues, enforcement and control issues, and social and ethical issues.
Finally, in developing a funding strategy for specific research and development projects, it is essential that federal planners be aware of existing market forces, which are as likely to be shaped by the results of research and development as they are to shape the research and development that is performed. With strategic planning, it would be possible for the federal government to use its investment in research to leverage developments
in the private sector. Similarly, it is essential that federal planners take account of the societal implications of the technology. As with most other technologies, the effects of the advances in synthetic environments are likely to be mixed: some effects will be positive and others negative. It cannot be assumed that all technological advances, even those that are likely to have substantial practical applications, will necessarily be beneficial.
Overall, the committee believes that synthetic environment systems have great potential for helping to satisfy various societal needs and stimulating advances in some important areas of science and technology. In pursuing the committee's recommendations for research and development, the federal government has an opportunity to make important contributions to the development of this exciting new field.