I

OVERVIEW



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
Virtual Reality: Scientific and Technological Challenges I OVERVIEW

OCR for page 11
Virtual Reality: Scientific and Technological Challenges This page in the original is blank.

OCR for page 11
Virtual Reality: Scientific and Technological Challenges At the request of a consortium of federal government agencies, the Committee on Virtual Reality Research and Development was established to provide guidance and direction on the allocation of resources for a coordinated federal program in the area of virtual reality. In responding to this charge, the committee has included both virtual environments and teleoperation in its assessment of the field. Such an extension is required not only for logical and scientific reasons, but also because many of the examples cited in the charge feature the use of teleoperator systems. In a synthetic environment (SE) system, the human operator is transported into a new interactive environment by means of devices that display signals to the operator's sense organs and devices that sense various actions of the operator. In teleoperator systems, the human operator is connected by means of such displays and controls to a telerobot that can sense, travel through, and manipulate the real world. In virtual reality (VR) or virtual environment (VE) systems, the human operator is connected to a computer that can simulate a wide variety of worlds, both real and imaginary. Simple remote manipulators are an example of the first type of system; video games of the second type. Teleoperator systems effectively provide the operator with a transformed sensorimotor system that enables him or her to perform new types of actions in the real world. Virtual environment systems effectively provide the operator with controllable methods for generating new types of experiences. Using both teleoperator and virtual environment systems, one can (or will be able to) explore the ocean floor and outer space, visit Samarkand while staying in Elmira, try out products not yet manufactured, dig up a 10-ton container of hazardous waste, take a canoe trip through the human circulatory system, and have one's hair trimmed by a barber in Seville. SCOPE OF THE SYNTHETIC ENVIRONMENT FIELD The research and development required to realize the potential of SE systems is extremely challenging. The systems are complicated because they involve both complex artificial devices and a complex biological system (the human operator). There is a crucial need for cooperation among many disciplines, including computer science, electrical and mechanical engineering, sensorimotor psychophysics, cognitive psychology, and human factors. Also, the range of possible applications is exceedingly broad. Overall, the committee believes that the SE field has great potential, that the research and development required to realize this potential is just beginning, and that work in this area should be vigorously pursued by a wide variety of specialists in a wide variety of institutions.

OCR for page 11
Virtual Reality: Scientific and Technological Challenges There is currently a great deal of excitement, a great deal of ''hype," and a great deal of confusion associated with the SE area. A major source of the confusion is the combination of rapid acceleration of interest in the area and the coming together of individuals from widely varying disciplines. In some cases, individuals are coming together because the problem to be solved requires expertise in diverse areas. In other cases, they are coming together because it has suddenly become apparent that essentially the same problems are being addressed by individuals in different fields who have never had the benefit of communicating with each other about them. Associated with this interdisciplinary feature of the SE field is confusion over terminology: each discipline brings to the field its own language and its own biases. For example, whereas computer scientists naturally use the terms input and output in reference to the computer, psychologists use these terms in reference to the human user. Thus, in a virtual environment system, what is output to the psychologist is input to the computer scientist. Similar confusions often arise with the term interface. Whereas computer scientists frequently use this term to designate a component internal to the computer's hardware or software, many others use the term as a shorthand for human-computer interface devices external to the computer. Also, of course, in addition to the communication difficulties associated with the interdisciplinary nature of the field, there are communication difficulties associated with the tendency of different individuals, institutions, and countries to compete rather than to cooperate. Another source of confusion results from political and public relations considerations. Virtual reality and virtual environment (two terms that we regard as equivalent) are such "hot" terms that many people tend to use them even when their use is logically inappropriate. Thus, for example, these terms are often used in a manner that implies that teleoperator systems are a special case of virtual reality systems. At the same time, however, when describing the origins of virtual reality systems, the history of teleoperator systems (in particular, the use of head-mounted displays in these systems) is entirely ignored. Similar distortions often occur in connection with simulator systems. Although simulator systems, like teleoperator systems, are closely related to virtual environment systems and have a long and distinguished history, past accomplishments in the simulation area are often inappropriately downplayed. Further discussion of the basic concepts and terminology is presented in the next section. Generally speaking, virtual reality currently has an extremely high "talk-to-work" or "excitement-to-accomplishment" ratio. Between 1992 and 1994, roughly 12 new books have been published, 4 new journals or magazines have been started, and 200 new articles have been published

OCR for page 11
Virtual Reality: Scientific and Technological Challenges on the topic of virtual reality. Major professional meetings and trade shows are occurring at a rate of roughly one per month. Over 10 government agencies have held conferences or written reports on VE during the same two-year period. And practically everyone in the field is spending substantial time traveling to other laboratories that are working on VE and providing demonstrations of their own facilities in their own laboratories. Despite this high talk-to-work or excitement-to-accomplishment ratio, substantial efforts are, in fact, under way in various research and development areas and in various application domains. Significant research and development programs, as well as applications of currently available technology, are being pursued in government, in academia, and in industry. Also, some attempts are being made to develop adequate course material for educational programs in the SE area; however, it is likely to be some time before most academic departments recognize SE as a legitimate field of specialization (e.g., one in which faculty can achieve tenure). Current research and development efforts directly relevant to the creation of useful SE technology are concerned with (1) computer generation of virtual environments, (2) design of telerobots, (3) improvement of human-machine interfaces, (4) study of relevant aspects of human behavior, and (5) development of communication systems that are adequate to support networking of SE systems. Items (3) and (4) are relevant to all the kinds of systems considered, item (1) to VE systems, item (2) to teleoperation systems, and item (5) to networked systems. An additional item of importance when augmented-reality systems are considered is (6) merging of computer-generated images with images derived directly from the real world. The "SE Challenge" is related to the High Performance Computing and Communications (HPCC) Grand Challenge program initiated by the federal government through both the computer generation of VEs and networked systems. For many applications, adequate computer generation of the associated virtual worlds is going to require very high-performance computing. Similarly, the networking of SE systems is going to require very high-performance communications. In general, SE systems will provide both a major application area for HPCC and an important source of constraints for the design of HPCC systems. Currently, the main commercial driving force for the development of VE systems is the entertainment application. There is no equivalent commercial driving force for the development of teleoperators or augmented-reality systems at this time. Programs on SE technologies and applications are under way in almost every developed country (Thompson, 1993). Major players are the

OCR for page 11
Virtual Reality: Scientific and Technological Challenges United States, Japan, and the European Economic Community; other players include South Korea, Singapore, the Netherlands, and Sweden. Although each of these regions is engaged in a full range of research, development, and commercial activities, the work in each region bears the marks of its distinctive culture. Today, more than 25 universities, at least 15 federal agencies, and more than 100 large and small companies throughout the United States are contributing to the growth of research and development in the SE field. In industry, research and development directed toward defense, space, scientific visualization, and medicine are more prominent in the United States than elsewhere. The European Economic Community and Japan have regional or national initiatives on SE, but such initiatives are still being debated in this country. Although the recession of the early 1990s in Europe has slowed down investment, a variety of SE projects are under way in industry and, to a lesser extent, in universities. Interests in the United Kingdom are similar to those in the United States but place more emphasis on education, training, and entertainment. The United Kingdom may well be the world leader in SE entertainment systems. On the continent, work on SE applications is being conducted at the European Space Research Center in Noordwijk, the Netherlands. Research on computer-aided architectural software and a virtual railroad environment are also being supported in the Netherlands. In France, the university at Metz is developing an autonomous motor vehicle for people with disabilities that uses SE technology. In Lille, the University of Technology is exploring the use of teleoperation in surgery. At the University of Paderborn in Germany, a new method for walk-through animation in three-dimensional scenes is under way. In Italy, the University of Genoa is developing a knowledge-based simulation for production engineers. Japan entered the VE part of the SE world later than the United States and Europe. Recently, however, that country has realized that VE, as well as teleoperation, is a logical extension of its strong national interest and background in robotics, automation, and high-definition television. Concern with haptic interfaces and force-feedback sensor display systems is also intense. As a consequence, Japan has established 10 national consortia for research and development in the SE area that, taken together, provide more funds per year than all SE investment in the United States (Larinaji, 1994). In 1992, the Japan Technology Transfer Association formed an Artificial Reality and Tele-Existence Research Committee of 90 participating companies from the SE industry. Knowledge and technology sharing among companies—generally a boon to Japanese industry—are extensive. These indicators, together with its typical long-range financial horizon, large targeted investments, and a national technology

OCR for page 11
Virtual Reality: Scientific and Technological Challenges agenda, could give Japan a major competitive advantage in SE. The extent to which this advantage is actually realized will depend, at least in part, on the extent to which Japan can become a leader in the relevant computer software areas. This overview begins by presenting some basic concepts and terminology that are important in talking about virtual environments and teleoperator systems. We then present some visions of where we think the technology may be leading. The visions section differs from the rest of this report in the speculative nature of the material and in the incorporation of societal issues into the scenarios. The overview then goes on to summarize the current state of the synthetic environment (SE) field, covering application domains, knowledge about human behavior and performance, technology issues, and evaluation issues. The committee's assessment of needs and priorities completes the overview. In making these recommendations, we include consideration of the extent to which various research goals are likely to be realized without special government funding efforts or are likely to require such efforts. Similarly, we consider issues related to the infrastructure required to carry out various research and development programs. BASIC CONCEPTS AND TERMINOLOGY There are currently no precise and generally accepted definitions of the terms being used in our area of interest. This is due in part, as already discussed, to the interdisciplinary nature of the field and to public relations matters. It is also due to fundamental problems of the type usually encountered in efforts to create language that faithfully reflects the structures and processes to which the language refers. For example, whereas language is fundamentally discrete, the evolutionary process by which virtual environment systems have developed from antecedent systems (such as desktop computing systems, simulators, teleoperator systems, etc.) is effectively continuous. Thus, either the definition of virtual environment systems must remain rather fuzzy, or one must set arbitrary thresholds on the complex, continuous evolutionary process. Here, we outline some of the principal defining ideas and indicate how the terms virtual environment, teleoperator, and augmented reality are related to each other and to other closely related terms such as simulator, telerobot, and robot. Our purpose is to provide background on the meaning of the terms we use in order to permit readers to understand later sections of the report. The process of creating and defining terms in this area will of course continue for many years. A teleoperator system consists of a human operator, a human-machine interface, and a telerobot (Figure 1). Environmental signals are sensed by

OCR for page 11
Virtual Reality: Scientific and Technological Challenges sensors (cameras, microphones, etc.) located in the telerobot, transmitted to the human-machine interface, and presented to the human by means of display devices (e.g., cathode ray tubes, earphones) in the interface. Human responses, usually motor actions, are sensed by the interface and used to control the actions of the telerobot. Thus, a teleoperator system can be viewed as a system for extending the sensorimotor system of the human organism. The purpose of such a system is to facilitate the human operator's ability to sense, maneuver in, and manipulate the environment. Teleoperator systems vary along many dimensions, including the structure of the human-machine interface and the telerobot and the nature of the control algorithms. Teleoperator systems have been used to conduct work in outer space and under the ocean; to perform a variety of tasks in connection with security, firefighting, nuclear plants, and hazardous waste removal; to assist in various types of military operations; to perform microsurgery; and to aid in the rehabilitation of individuals with severe physical disabilities. In some teleoperator systems, the human operator has direct and detailed control of all the telerobots actions. In other systems, the human's control occurs only at a supervisory level and many of the telerobots detailed actions are controlled locally and automatically. In the extreme, there is no human control, all actions of the telerobot are automatic and autonomous, and the telerobot is called simply a robot. A virtual environment system (also illustrated in Figure 1) consists of a human operator, a human-machine interface, and a computer. The computer and the displays and controls in the interface are configured to immerse the operator in an environment containing three-dimensional objects with three-dimensional locations and orientations in three-dimensional space. Each virtual object has a location and orientation in the surrounding space that is independent of the operator's viewpoint, and the operator can interact with these objects in real time using a variety of motor output channels to manipulate them. The extent to which a virtual environment is designed to simulate a real environment depends on the specific application in mind. As illustrated in Figure 1, teleoperator and virtual environment systems are similar in that they both involve human operators and elaborate human-machine interfaces. They differ however, with respect to what takes place on the nonhuman side of the interface. Whereas in a teleoperator system the interface is connected to a telerobot that operates in a master-slave or supervisory control mode in a real-world environment, in a VE system the interface is connected to a computer. Consistent with this difference in structure is the difference in purpose between the two types of systems: whereas the purpose of a teleoperator system is to sense, manipulate, and transform the state of the

OCR for page 11
Virtual Reality: Scientific and Technological Challenges FIGURE 1 Schematic outline comparing a teleoperator system, a virtual environment system, and an unmediated (normal) system. real-world environment, the purpose of a VE system is to sense, manipulate, and transform the state of the human operator (as in training or in scientific visualization) or to modify the state of the information stored in the computer (e.g., the virtual environment or some theoretical model represented in the computer software). Virtual environment systems are being used in the areas of telecommunication, information visualization, health care, education and training, product design, manufacturing, marketing, and entertainment. In the near future, such systems are likely to find further applications in various areas of psychology, including basic psychophysical research, biofeedback, and psychotherapy. Many systems are now being developed that are mixtures or blends of teleoperator and virtual environment systems. Thus, for example, VE systems are now being introduced as subsystems of teleoperator systems in order to assist the human operator in controlling the telerobot. In particular, when the telerobot is sufficiently far removed from the human operator to cause significant time delays in the transmission of information between the telerobot and the human operator, virtual environments can be used to present computer-generated information derived from predictive models in the computer. People are also designing systems in which virtual and real environments are combined (Figure 2). The use of such augmented-reality systems

OCR for page 11
Virtual Reality: Scientific and Technological Challenges FIGURE 2 Schematic outline of some augmented reality systems. One kind of augmented-reality system combines images obtained from direct sensing of objects in the environment with images generated by a computer using see-through (or hear-through or feel-through) displays. A second kind combines images obtained by means of a telerobot with those generated by a computer. In principle, systems that combine all three channels of input information could be combined. Also, of course, and as mentioned in the text, it is possible to consider systems that merge output (control) information rather than, or as well as, input (display) information. is being explored in medical applications, manufacturing applications, and driving applications (both airplanes and cars). In many such cases, information from the real environment is sensed directly by means of a see-through display, and the supplementary information from the virtual environment is overlaid on this display. In other cases, the real-environment information to be combined with the virtual-environment information is derived by means of a teleoperator system. Although currently receiving less attention in the SE community, it is also possible of course to consider augmented-reality systems in which, instead of combining input channels, output channels are combined. For example, speech sounds or commands uttered by the human operator might be combined with those uttered by an automatic speech-synthesis system, or physical objects in the environment might be manipulated by systems that include both the hand of the human operator and a telerobotic hand controlled by the operator. There are certainly many tasks in which it would be extremely useful to have a third hand (with special features perhaps) that could work cooperatively with one's own two hands and be controlled,

OCR for page 11
Virtual Reality: Scientific and Technological Challenges perhaps, by simple speech commands. A further way of picturing some possible relations between teleoperator and virtual environment systems within an SE system is illustrated in Figure 3. In all of these systems, the human operator is projected into a new interactive environment that is mediated by artificial electronic and electromechanical devices, and in all of these systems, the operator's performance and subjective experience in the new environments depend strongly on the human-machine interface and the associated environmental (real or virtual) interactions. In general, we refer to all of these systems (teleoperator systems, virtual environment systems, augmented-reality systems, etc.) as synthetic environment (SE) systems. In considering these different kinds of systems, it should be noted that many of the problems now facing designers of VE systems have been studied previously in the field of telerobotics. This is the case, for example, in the area of human-machine interfaces. Although the constraints FIGURE 3 Schematic outline of a further configuration of SE system components. In this configuration, there are two computers, one on each side of a communication link. If all the components on the left side of this link are deleted and the computer on the right is used to control the telerobot, the system reduces to an autonomous robot. If the components to the right of the link are deleted and the computer on the left is used to generate a virtual environment, the system reduces to a pure virtual environment system. If all the components are included, but the computer on the left is not used to generate a virtual environment, then the system reduces to a pure teleoperator system (which would have supervisory control if local, low-level actions were controlled by one of the computers). If all the components are included, and the computer on the left is used to generate a virtual environment, then the system becomes an augmented-reality system of the second type described in the caption to Figure 2.

OCR for page 11
Virtual Reality: Scientific and Technological Challenges excessive, generally lead to perceivable lags and distortions. In the case of force-reflecting haptic interfaces, they may also cause mechanical instabilities. Therefore, rapid rendering software that minimizes time delay while retaining optimal display resolution is critical for each of the modalities. Under multimodal conditions, the additional condition of synchronization of the modality-specific displays needs to be satisfied. We recommend support for development of software tools for rapidly driving visual and auditory display devices, together with fast, real-time control of haptic interfaces. Such software can have both device-dependent and device-independent components. There is a major need to develop more powerful methods for acquiring and representing realistic models of physical objects and for realistic simulation of the physical behavior of these objects. To construct realistic models of truly complex environments is all but impossible with current computer-assisted design tools. Creation of such complex models will require a combination of automated model acquisition from real data and automated model synthesis based on concise descriptions. Such models will ultimately need to capture not only the object characteristics relevant to the visual channel, but also all of the physical properties that must be specified to realistically simulate objects' appearance and behavior in the broad multimodal sense. Simulating the mechanics of the everyday world will be of central importance in giving virtual environments a sense of solidity and allowing users to effectively manipulate virtual objects through haptic interfaces. The problems that arise in generalizing standard batch-simulation methods to handle interactive VEs are analogous to those that arise in the extension of static rendering techniques. Research into the development of environments in which object behavior as well as object appearance can rapidly be specified is an area that needs further work. We call this area simulation frameworks. Such a framework makes no assumptions about the actual behavior (just as graphics systems currently make no assumptions about the appearance of graphical objects). A good term for what a simulation framework is trying to accomplish is meta-modeling. Such frameworks would facilitate the sharing of objects between environments and allow the establishment of object libraries. Issues to be researched include the representation of object behavior and how different behaviors are to be integrated into a single system. Because most current operating systems are built on commercial versions of UNIX, which is not designed to meet real-time performance requirements, the committee recommends that approaches to a new operating system suitable for VEs be studied. In principle this could be achieved by creating a new operating system architecture or providing upgrades or enhancements to existing operating systems (e.g., UNIX, Windows NT).

OCR for page 11
Virtual Reality: Scientific and Technological Challenges The operating system capabilities required for VE include support of very large numbers of lightweight processors communicating by means of shared memory, support of automatic or transparent distribution of tasks to multiple computing resources, support of time-critical computation and rendering, and very high resolution time-slicing and guaranteed execution for high-priority processes (to within 0.001 s resolution). Although not specifically addressing all of these concerns, the efforts of the IEEE Posix standards committee are starting to bring real-time capabilities to the open-system workstation environment.1 Supporting these capabilities in the operating system will significantly facilitate the development of many VE applications, especially larger, more ambitious efforts). The commercial sector cannot be expected to perform the necessary research and development in this area without incentives from the federal government. Specifically, we recommend that the government participate with industry in funding the upgrades and enhancements needed to provide an operating system that will meet the performance requirements for implementing VEs. Moreover, these joint funding efforts should be accompanied by a plan to move the new or upgraded systems to commercial adoption. To ensure that VE systems are written using an appropriate operating system, a financially sound transition plan must be formulated, funded, and executed. Another important area for development is registration of real and synthetic images for augmented-reality applications. To create the illusion that synthetic and real objects exist in the same world requires highly accurate registration. For example, to make a synthetic object appear to rest on a real table, the object must move with the table as the observer moves, and accurate registration requires both a good geometric model of the scene and good measurements of observer motion. In addition to the purely geometric aspects of registration, illumination effects (casting synthetic shadows onto real objects) must be handled. Note also that significant misregistration may be disastrous in certain applications (e.g., surgery). In addition to the general areas discussed above, the committee recommends that research and development on the following topics be supported: navigational cues in virtual space, the behavior of autonomous agents, and the computer generation of both auditory and haptic images for VE. Navigational cues are important because there is a great tendency in current VE systems for users to lose their way during virtual travel (or even simply during rotations of the head). Work on autonomous agents 1   For VEs with relatively simple input/output, the real-time requirements are different from those of the telerobotics community. The fundamental difference is that the massive input/output requirements for complicated haptic and other human interfaces associated with telerobotics cannot be handled by an ordinary workstation-based architecture.

OCR for page 11
Virtual Reality: Scientific and Technological Challenges is important because many future applications are likely to require such agents, and the task of designing appropriate psychological and physical models for ''driving" these agents is an extremely difficult one. With regard to computer generation of auditory images, spatialization, synthesis of environmental sounds, and auditory scene analysis are judged to be the most critical; in the haptic channel, because so few results are currently available, a wide array of research projects should be supported. Although certain components of some of these problems relate primarily to design of human-machine interface devices, others relate primarily to software. Telerobotics Recommendations in the area of telerobotics that are not already included elsewhere concern: (1) the effects of communication time delays on teleoperator performance, (2) telerobotics hardware (structures, actuators, and sensors), (3) microtelerobotics, (4) distributed telerobotics, and (5) real-time computational architectures. RECOMMENDATION: The committee recommends that support be given to improving control algorithms, improving methods for constructing and using predictive displays, and improving methods for realizing effective supervisory control strategies. Unless communication delays are properly handled, teleoperator performance will be severely degraded and may, under certain circumstances, become unstable. In order to combat the effects of such delays, continued efforts should be directed toward the development of improved control algorithms that ensure stability and yet, to the extent possible, provide reasonable gains. At the same time, continued effort should be directed toward the development of improved methods for constructing and using predictive displays and for realizing effective supervisory control strategies. Advances in combatting the delay problem are required not only in connection with hazardous operations, but also in connection with certain components of telemedicine (particularly telesurgery). RECOMMENDATION: The committee recommends work in four areas of hardware development: (1) multiaxis, high-resolution tactile sensors, (2) robot proximity sensors for local guidance prior to grasping, (3) multiaxis force sensors, and (4) improved actuator and transmission designs. Multiaxis high-resolution tactile sensors are needed to provide the telerobot with an effective sense of touch. Robot proximity sensors are required to provide local guidance prior to grasping. Such guidance

OCR for page 11
Virtual Reality: Scientific and Technological Challenges would greatly facilitate the development of adequate supervisory control. Multiaxis force sensors are needed to measure the net force and torque exerted on end effectors. For example, miniature force sensors of this type could be mounted on finger segments to accurately control fingertip force. Improved actuator and transmission designs are required to provide high-performance joints and improved performance of telerobotics limbs. RECOMMENDATION: The committee recommends that research be conducted on issues that arise when microtelerobots are used in teleoperation. As the field of microelectromechanics evolves, and smaller and smaller telerobots can be constructed, the need for both basic and applied research in this area will steadily increase. For example, it will be necessary to address problems associated with the scaling of movements and forces. Because the mechanical behavior of objects in the micro domain are radically different than in the macro domain, such scaling will require the development of new types of telerobotic controllers. RECOMMENDATION: The committee recommends that consideration be given to the development and application of distributed telerobotic systems. Relatively little attention has been given to teleoperator systems in which the human operator is interfaced to a distributed set of telerobots. Because many functions require sensing or acting over a region that is large relative to the size of an individual telerobot (e.g., patrolling land or structures for security reasons), such systems, if appropriately designed and developed, would have many important applications. Issues that need to be addressed in this area include the careful selection of specific applications, the design of the communication system for transmitting information among the telerobots and between the set of telerobots and the human operator, and the design of human-machine interfaces that are well matched to human sensory and control capabilities in situations involving multiple telerobots. RECOMMENDATION: The committee recommends the establishment of intercommunication standards for point-to-point connections in coarse-grained parallel computational architectures. However, for applications with demanding input/output operations, the committee does not recommend new real-time development systems or operating systems. The most demanding VE system will require powerful real-time input/output capabilities to handle haptic interfaces, trackers, visual displays,

OCR for page 11
Virtual Reality: Scientific and Technological Challenges and auditory displays. In robotics and telerobotics, in which the requirements are similar, there is a general movement toward coarse-grained systems based on point-to-point communications, such as transputers and C40 systems. Commercial development environments and real-time operating systems are adequate for such systems. However, there has not yet emerged a high-speed intercommunication standard for point-to-point computational architectures, which would offer users great flexibility in mixing and matching components across vendors and different processor types. Networks The committee anticipates that in the future most VE applications will rely heavily on network hardware and software. Although networks are now becoming fast enough for distributed VE applications, development is needed to provide the enormous bandwidth required to support multiple users, video, audio, haptics, and possibly the exchange of three-dimensional interaction primitives and models in real time. Moreover, handling the mix of data over network links will require new applications level protocols and techniques. Because of the central nature of network technology to the implementation of VEs, the committee sees network hardware and software development as critical to advancing the science of VE and its applications. However, we believe that the hardware necessary to support VE applications will be developed without intervention from the VE research community. In other words, there are forces in both the federal government and the private sector that are driving major advances in hardware. As a result, we do not recommend additional investment in network hardware development at this time. Nevertheless, it is important to acknowledge the existence of significant infrastructure problems that could impede the use of networks for VE applications. For these problems, specific effort should be provided in support of VE requirements. One infrastructure issue is the high cost of research on large-scale networked VEs. A very limited number of universities can afford to have dedicated T-1 lines (with installation expenses of $40,000 and operating costs of $140,000 per year, as for the Defense Simulation Internet currently) needed to support these activities. Various approaches, such as an open VE network and the necessary VE applications protocol, should be considered for providing research universities with access to the needed facilities. Unless costs are significantly reduced, it will not be possible to initiate a concerted effort to develop software solutions for networked VE. Perhaps our greatest infrastructure concern is the need for the development of network standards that will be compatible with the long-range

OCR for page 11
Virtual Reality: Scientific and Technological Challenges needs of distributed VEs. One danger is that the entertainment industry, with its interest in interactive games for the home, will set the networking protocol standards at the low end, and the military community will set the standards at the high end. Therefore: RECOMMENDATION: The committee recommends that the federal government provide funding for a program (to be conducted with industry and academia in collaboration) aimed at developing network standards that support the requirements for implementing distributed VEs on a large scale. Furthermore, we recommend funding of an open VE network that can be used by researchers, at a reasonable cost, to experiment with various VE network software developments and applications. Evaluation of SE Systems RECOMMENDATION: The committee recommends that the federal government encourage the SE system developers it supports to include a comprehensive evaluation plan in the early design stages of their research projects. It also recommends that the federal government help coordinate the development of standardized testing procedures for use across studies, systems, and laboratories, particularly in those areas in which the private sector has not acted. SE technology is in the early stages of development, is growing rapidly, and is the subject of highly optimistic projections about its usefulness. In contrast, the extent to which its usefulness has actually been seriously evaluated is vanishingly small. In general, evaluations are required not only to compare overall cost effectiveness of SE approaches with other approaches addressed to the same goals, but also to provide insights to guide modifications and new design directions. To be optimally effective, such evaluations must take place both at the overall system level, at the component level, and at all stages of the development process. Although many of the specific questions to be addressed in an evaluation effort are likely to depend to some degree on the structure and purpose of the system or component in question, it should be possible to determine a common framework for a substantial portion of the evaluation needs. In order to help ensure adequate SE evaluation, the federal government should encourage individuals involved in federal supported research and development to include serious evaluation plans in the design of their projects. Such plans should address questions about engineering performance, user needs and acceptance, dependence of human performance and safety on various system or component features, and costs of

OCR for page 11
Virtual Reality: Scientific and Technological Challenges development, implementation, and marketing. Furthermore, in order to facilitate consistency across SE projects, the federal government should help coordinate the development of standardized testing procedures for use across studies, systems, and laboratories, particularly in those areas in which the private sector has not acted. These procedures should include methods for identifying key system dimensions that affect task performance, developing special metrics uniquely suited to evaluating SEs, and comparing SE system performance to performance of other systems intended to meet the same or similar goals. Suggestions for Government Policy and Infrastructure The magnitude, quality, and effect of the SE-oriented research and development that is accomplished will clearly depend on the role played by the federal government. The current status of the SE field is sufficiently embryonic, compared with what is likely to develop over the next 10 years, that the federal government now has a rare opportunity to foster coherent planning in this area. Furthermore, the recently established National Science and Technology Council at the White House would appear to be an appropriate organization to provide oversight for such a planning effort. Also, in conducting such a planning effort, substantial benefits would be gained by attending carefully to the developments that are already taking place in the other areas of the administration's planning effort—for example, the Advanced Technology Program of the National Institute of Standards and Technology, the High Performance Computing and Communications program, and the programs associated with defense conversion. In this section, we discuss a number of mechanisms that illustrate the kind of leadership role that the government could play. We see that role as both informing and complementing the federal agencies' strategic planning for their support of research and development programs. Establish an Effective Information Infrastructure A national information system that provides comprehensive coverage of research activities and results in the SE field in a user-friendly way to a wide variety of users could be a useful tool for promoting cross-fertilization and integration of the research and development efforts. The free flow of ideas and information among researchers, users, and individuals in government, academia, and industry who require information for SE planning and decision making is crucial to the development of this new field. Also, in order to diminish the increasing threat of a major societal division between the technologically advantaged and the technologically deprived (as well as to counter the current hype about virtual

OCR for page 11
Virtual Reality: Scientific and Technological Challenges reality), the public should have information of the appropriate type in an easily available form. Although information by itself cannot prevent such a division, it is a necessary ingredient of any program that could. We suggest that the federal government consider establishing a national information system in order to promote these vital communication goals. To reduce costs and to realize potential benefits as soon as possible, consideration could be given to integrating the SE information system with other public information systems currently being developed. For example, such a system might be an ideal component of the national digital library based on high-speed networking envisioned in the National Information Infrastructure (NII) initiative of the Clinton administration. Issues of ownership and control, as well as technological issues, will be important to consider in the design of an SE information system. To some extent, the technology, procedures, and ideas being developed within the SE field itself could be usefully exploited in the design of the SE information system. Such a system might eventually have uses well beyond those initially envisioned; for example, it might include a library of computational models. Although for many years there has been a tendency for scientists to express their understanding of various systems and phenomena in terms of computational models, this tendency is clearly being accelerated by the role such models play in the generation of VEs. Indeed, it seems possible that, in the near future, computational models will constitute one of the society's primary forms of knowledge representation. Thus, for example, reading a book about Newtonian mechanics is likely to be augmented by interacting with a virtual world based on a computational model that includes Newtonian mechanics and then, perhaps, "reading" the computational model. The same kind of evolution is, of course, occurring with fiction and imaginary worlds; independent of whether a structure or a series of events is real or imaginary, much of the relevant information can be stored in the form of a computational model. In order to make such computational models available to society, the federal government might consider establishing a national system for standardizing, collecting, storing, and disseminating such models. In view of society's current concerns with health care, initial efforts in this area might be focused on computational models related to the structure and function of the human body and modifications of the human body associated with injury, trauma, disease, aging, and medical and surgical treatments. Encourage Appropriate Organizational Structures and Behaviors Two major factors that could inhibit advances in SE involve the ability of researchers to communicate and cooperate across disciplines and

OCR for page 11
Virtual Reality: Scientific and Technological Challenges across organizations. Because the creation of effective SE systems requires contributions from many different disciplines (with many different associated cultures), special efforts are required to ensure adequate communication and cooperation across disciplines. Similarly, because of the high value placed on competition within our society, special efforts are needed to ensure adequate communication and cooperation across government agencies, military branches, industrial firms, and academic institutions. At present, the organizational barrier appears to be more debilitating than the disciplinary (or cultural) barrier. In fact, the lack of cooperation among competing organizational entities (for example, competing companies) probably constitutes the main obstacle to achieving a truly satisfactory solution to the information infrastructure problem discussed above. Consideration of explicit incentives for cooperative behavior might be very useful. In order to reduce these problems, the committee suggests that the federal government consider establishing a small number of national research and development teams, each of which would focus on a specific application area. These teams could involve government, industry, and academia, as well as the various disciplines relevant to the given application area. Funding could be provided jointly by the federal government and the private sector. The work to be performed by each national research and development team would include basic research, technology development, functional prototypes, technology evaluation, and technology transfer to industry. Despite the emphasis on applications that is implied by how the teams are defined, the work could be directed toward long-term as well as short-term goals, and the basic research needed to achieve these goals would then be a priority for support. Also, to the extent feasible, it might make sense to connect these collaborative teams or applications consortia not only to already existing federal activities (as has been the case, for example, with the textile partnership AMTEX that is being managed by the Department of Energy), but also to already existing professional societies. In setting up these teams, the choice of leadership for the activity will be crucial. In some cases, federal leadership may be appropriate; in others, industrial; in still others, academic. Finally, each of the envisioned teams might well find it appropriate to develop a powerful networked communication system among its members to ensure true collaboration at the working level. Use SE Systems Within the Government It might be useful for some federal agencies and offices to explore the use of SE to meet their own administrative and program needs. In addition

OCR for page 11
Virtual Reality: Scientific and Technological Challenges to the application of SEs to the defense and space programs already under way, other application domains, such as training, telecommunication and teletravel, and information visualization, are relevant to the activities of many agencies. One way for the government to facilitate development of the SE field would be to select a few agencies to serve as test beds for synthetic environment technology in these general domains. There are a number of reasons for suggesting that the government make use of SE systems in conducting its own activities. Government agencies (local as well as federal) are natural early users of new technology: they could help spearhead development efforts and provide feedback to the developers. Also, such use could increase the cost-effectiveness of government activities. In addition, such use could create a market for SE systems and thus stimulate private industry to become involved in the design and production of SE systems. At present, uses within the government that are receiving the most attention are those associated with the Department of Defense; however, other entities, such as NASA and the Department of Energy, are also involved. Although military applications trail behind those associated with the entertainment industry as an economic driving force, they nevertheless constitute a force that is significant. This significance is derived not only from the overall magnitude of the associated economic activity, but also from the special role played by defense agencies in stimulating the development of relatively high-quality systems for military applications. Also of interest in this connection are the current efforts to explore the use of SE systems in the Department of Defense for education and training. If the results of these studies are positive, they could play a significant role in stimulating the use of SE systems for education and training throughout the nation as a whole. The use of SE systems in NASA appears to hold great potential not only with respect to training people for operations in hostile environments, but also with respect to performing the operations themselves. The enormous expense associated with manned space flights and space stations may well serve as a strong stimulus to the use of teleoperation in space activities. Developing National Standards and Regulations Although it is probably too early in the development of SE systems to establish national standards and regulations, it is not too early to begin to evaluate the work already under way in connection with the formulation of standards and regulations for the telecommunications and entertainment industries. Problems that are already of concern but are likely to become of even greater concern as the SE field develops relate to technological

OCR for page 11
Virtual Reality: Scientific and Technological Challenges compatibility and interoperability issues, enforcement and control issues, and social and ethical issues. For example, in the technological area, problems related to the timing of information flow in SE networks merit special consideration. Similarly, in the social and ethical area, the potential of SE for providing participants with powerful emotional experiences (including those related to sex and violence) needs to be addressed. In general, it appears that SE, because of its mass entertainment potential, is likely to become one of the largest uses of high-speed communication networks, and its use should have an early and continuing part in the development of standards, regulatory principles, and tariff-setting models for such networks. The recent congressional attention that has been given to the kinds of material that are appropriate for the media to present is but a mild precursor to the public debate that is likely to arise when advanced VE technology becomes widely available. It will be critical for the federal government to consider VEs in the formulation of national standards and regulations. Studies could be undertaken to illuminate issues related to technological compatibility and interoperability, enforcement and control, and social and ethical problems raised by the use of VEs in society. Analyze and Evaluate Market Forces and Societal Impact The extent to which government funds will be directed toward specific SE research depends, at least in part, on the likelihood that such projects will be funded independently, i.e., by industry. Estimating this likelihood requires not only an analysis of current market forces, but also predictions of how market forces will evolve in the future. Although such predictions are notoriously difficult to make with accuracy, and market forces are as likely to be shaped by the results of the research and development as they are to shape the research and development that is performed, failure to consider market forces in making funding decisions is likely to seriously reduce the extent to which the funding is effective in advancing the field. For these reasons, it would be prudent for the federal government to monitor market forces as part of developing its strategic plan for the allocation of scarce resources. As with most other technologies, the effects of the advances in SE are likely to be mixed; some effects will be positive and others negative. And as with the predictions of market forces, although accurate predictions of societal impact are difficult to derive, serious attempts to consider such factors would be decidedly worthwhile. It cannot be assumed that all technological advances, even those that are likely to have substantial practical applications, will necessarily be beneficial.