Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 205
Virtual Reality: Scientific and Technological Challenges 6 Whole-Body Motion, Motion Sickness, and Locomotion Interfaces Many virtual environments (VEs) will give the user a sense of motion either in the form of passive transport in a "vehicle" or through active locomotor movements on a supporting surface. The latter experience couples the voluntary locomotor activity with visual and auditory sensations. In both situations, the support of the user's body by the physical contact surfaces constitutes a haptic interface. The term haptic interface is being used here in a more general sense than in Chapter 4. Here we use this term to refer to any place at which mechanical energy interchanges take place between the body and the environment. In the case of passive transport in a vehicle, haptic interfaces with the vehicle are the places at which the vehicle applies accelerative forces to the individual's body. The haptic stimulation associated with these forces contributes to the perception of body motion. Similarly, during voluntary locomotion, such as walking, the energy interchange between the feet and the support surface produces haptic cues that provide information about surface characteristics (e.g., compliance, slope, texture) and body displacement. Both passive and active displacements of the entire body (and of parts such as the torso during rotary movements of the torso on the hips) reorient sensory receptors vis-à-vis the environment. The resulting changes in receptor activity normally are taken into account, so that perceptual stability of the surroundings is maintained during active or passive self-movement. In VEs, head-trackers and other forms of position monitors are used to update visual and auditory displays to compensate for changes in body position. To the extent that there are time delays beyond 20 or 30
OCR for page 206
Virtual Reality: Scientific and Technological Challenges ms or gain changes between the body movement and the stimulus update, performance problems can be anticipated. Similarly, if the optical verticals in a VE do not correspond with the gravitational vertical of the real environment, orientation and movement difficulties may be experienced by the user. Whole-body movements and locomotion raise a large set of issues concerning the forms of compensation that take place during self-movement, the perception of forces on the surface of the body during movement, the perception of self-displacement through space, and of one's voluntary actions. The way in which adaptive compensations for unusual environments and for maintenance of accurate sensorimotor calibration are achieved is also crucial. All of these issues are critical to understanding how people will adapt to VEs involving locomotory movements, passive transport, and head, arm, and torso movements. However, it is important to recognize that our knowledge of these areas is incomplete. Motion sickness is a factor that is certainly going to affect performance in VEs involving locomotion and experienced self-motion. It is necessary to be aware of the wide range of factors that contribute to motion sickness, the variety of the symptoms, and its sometimes subtle characteristics. STATUS OF THE RELEVANT HUMAN RESEARCH Sensorimotor Stability During Self-Motion Under normal circumstances, an individual accurately perceives his or her voluntary movements and perceives the surroundings to be stable when they are actually stable. This stability that we take for granted is the result of complex sensorimotor adaptations to the 1G force field of earth. The existence of these adaptations becomes apparent during exposure to non-1G force levels or conditions of sensory rearrangement (Lackner and Graybiel, 1981). One aspect of these adaptations includes not perceiving veridically the forces acting on the body surface when they are due to voluntary activity or passive support of the body against the force of gravity. For example, the forces on the bottom of the feet feel roughly comparable when one is standing on one or both feet even though in the former case the force on the stance foot is twice as large (Lackner and Graybiel, 1984c). Similarly, in running, the forces on the feet can vary from 0 to 3G, yet these huge changes are not perceptually registered. When these same force levels are passively applied, the sensation is very strong. Body movements are accompanied by various sensorimotor consequences. In the case of object manipulation, the pattern of sensory stimulation
OCR for page 207
Virtual Reality: Scientific and Technological Challenges of the hands in relation to the patterns of motor activity controlling the hands and arms and other parts of the body allows identification of properties of the object such as its shape, texture, and weight. In the case of locomotion, similar types of information along with vestibular cues allow—even in the absence of vision—quite accurate judgments of the distance and direction moved and properties of the support surface, such as its inclination and surface properties. Manipulation of virtual objects and locomotion in VEs will disrupt many of the sensorimotor regularities that are present in the normal environment. This is a significant issue because we have limited experience with violation of these constraints, and most probably only a relatively small set of the relevant ones have yet been identified. Consequently, movements made in VEs that violate terrestrial sensorimotor constraints may initially cause performance decrements and elicit symptoms of motion sickness (Lackner et al., 1991). In fact, reports of motion sickness in simulated environments are becoming commonplace. As with other unusual sensorimotor environments, continued exposure is likely to lead to adaptive compensations that restore accurate performance and alleviate symptoms of motion sickness. However, on return to the normal environment, there may be persistence of adaptation to the virtual situation leading to performance decrements and motion sickness in the normal environment. It may be possible, however, to create states of dual simultaneous adaptation such that accurate performance is possible in the normal as well as in one or more VEs. The possibility of such dual adaptations has been shown for a number of unusual force situations, such as rotating and nonrotating environments (Lackner, 1990), as well as for other types of situations (e.g., Welch et al., 1993). Head Movements Head movements made during exposure to increased or decreased gravitoinertial force background levels tend to bring on symptoms of motion sickness because the normal patterning of sensorimotor control of the head and patterns of sensory feedback are disrupted (Lackner and Graybiel, 1984a, 1985, 1987; Lackner et al., 1991). Virtually any alteration in the normal patterning of control can be provocative (Lackner and DiZio, 1989). For example, wearing a neck brace requires head movements to be achieved by motion of the torso; this alteration in motor control can be quite provocative for many individuals. Passive exposure to various types of motion is provocative because of the labyrinthine stimulation involved (e.g., on shipboard), but such exposure is even more provocative if the head is not passively supported but rather actively controlled (Lackner et al., 1991).
OCR for page 208
Virtual Reality: Scientific and Technological Challenges Head movements made during everyday activities are not provocative. By contrast, the same head movements made during passive rotation of the body are extremely stressful and rapidly evoke disorientation and symptoms of motion sickness. Movements of the head during body rotation create unusual patterns of activation of the semicircular canals of the inner ear and generate Coriolis forces on the head (Lackner and Graybiel, 1984b, 1986). These Coriolis forces are proportional to the velocity of rotation and the velocity of the radial motion of the head (or other body part such as the hand) and act in the direction opposite that of the rotation. They are absent prior to and at the end of a movement. Coriolis forces are present when head movements are made during passive and voluntary body rotations. The fact that they are disorienting and produce motion sickness during passive and not active turning movements means that we automatically take them into account in controlling our voluntary movements. A sense of presence may be a key factor in determining the human's responses to Coriolis stimulation and to expected Coriolis stimulation in VEs. Visual stimulation, especially whole-field visual motion, can elicit sensations of self-motion. If exclusive self-motion is being experienced in an apparently stationary visual field, then head movements can rapidly elicit symptoms of motion sickness and disorientation (Dichgans and Brandt, 1978). By contrast, if visual motion is being experienced and little self-motion, then head movements tend to suppress the sensation of self-motion and less motion sickness will be experienced than from simply looking passively without moving the head (Lackner and Teixeira, 1977; Teixeira and Lackner, 1979). Accordingly, VE displays that induce apparent whole-body motion and that require head movements are likely to elicit greater levels of sickness the greater the fidelity of the experienced self-motion and sense of presence. Altering the sensorimotor control of the head with a neck brace can be mildly provocative. However, altering the weight of the head and its effective moment of inertia can be extremely provocative, even when the pattern of vestibular input is normal for the actual motion of the head. For example, wearing a helmet that increases the effective weight of the head by 50 percent greatly increases susceptibility to motion sickness during exposure to constant patterns of angular acceleration and also causes natural voluntary head movements to be provocative (Lackner and DiZio, 1989). Head-mounted displays (HMDs) are an integral component of many VE systems and affect the sensorimotor control of the head. This altered inertia of the head may be provocative if the displays are worn for more than several minutes and head movements are made. That is, simply wearing an HMD can be provocative in itself, regardless of the scenes displayed. To the extent that the display induces apparent self-motion,
OCR for page 209
Virtual Reality: Scientific and Technological Challenges head movements may be extremely provocative. In addition, the time delays in HMDs associated with updating the visual scene to compensate for head movements disrupt the normal patterning of the vestibulo-ocular reflex. This is especially nauseogenic in wide-field HMDs in which large-gaze shifts occur as individuals turn their head and eyes to acquire targets in the peripheral visual field (Lackner and DiZio, unpublished observations). The point to be taken is that there is a spectrum of factors that can be expected to evoke sickness. Arm Movements Arm movements made to objects or to control devices are accompanied by patterns of sensory feedback related to the control of the arm and the object manipulated. Object characteristics include inertia, mass, weight, texture, compliance, etc. Expectancies about the properties of objects are exceedingly important. For example, when visually similar objects of different physical sizes but identical masses are hefted in succession, the larger sized one will be perceived as being considerably lighter. The motor plans for hefting the objects probably include compensation for a greater expected mass of the larger object. This effect, known as the size-weight (or Charpentier's) illusion, persists, however, even when the objects are known to be of comparable weight. Insofar as object manipulation in VEs violates cognitive expectancies about object behavior, performance decrements will occur, adaptation will be required, and after-effects can be anticipated on return to the normal environment. During passive transport of the body in vehicles as well as during voluntary locomotion and turning of the body, arm movements are usually quite accurate. In a familiar vehicle, even one not being self-controlled, compensation can be made for the ongoing and expected motion of the vehicle (e.g., on shipboard, the rolling and scending motions are periodic and can be anticipated). During exposure to linear and angular acceleration, adjustments of the entire body as well as arm movement control may be required. For example, the driver of a vehicle will physically lean into a turn (Fukuda, 1975). Arm movements made during exposure to passive rotation of the body generate Coriolis forces that deviate the arm from its intended trajectory and target. With repeated reaches, even without visual feedback about movement accuracy, accuracy will be rapidly regained if the hand makes physical contact at the end of the reaching movement. However, in the absence of vision, if terminal contact of the hand is not present at the end of the movement, adaptation does not take place or is greatly slowed (Lackner and DiZio, 1994). On cessation of body rotation, pointing movements made to targets show error patterns that are mirror images
OCR for page 210
Virtual Reality: Scientific and Technological Challenges of the initial per-rotary reaches. In other words, adaptation to the rotating environment persists and readaptation to the stationary environment is necessary. These findings are relevant to movement control in VEs for several reasons. First, in everyday behavior, reaching movements made during voluntary body movements of the trunk or whole body generate Coriolis forces far in excess of those used in experimental studies, yet movements are made accurately. This means that motor compensation is made for expected Coriolis forces associated with self-movement. Second, during illusory (e.g., visually induced) self-rotation, compensation may be made during reaching movements for Coriolis forces that would be present during actual body rotation; consequently, reaching errors result. Thus, the perception of self-motion per se may be adequate to induce motor compensations for Coriolis forces expected with actual rotation. VEs that induce apparent motion of the body may lead to compensatory motor adjustments during arm and body movements for expected forces that are actually absent. Third, with prolonged exposure, individuals adapt their movements. Such adaptation can persist on return to the normal environment so that compensation is not initially made during body motion for Coriolis forces that are actually present. Posture and movement control take into account not only Coriolis forces but also a variety of other forces associated with experienced linear and angular accelerations and angular velocity, such as centrifugal forces (Lackner, 1993). To the extent that VEs cause experienced body motion, compensation for expected forces may be generated. The extent to which an individual feels present in a VE may determine the extent to which compensation for expected inertial forces occurs. Moreover, the significance of these compensations may differ for different VEs. For example, if simulation of a high-performance aircraft is being used for training purposes and a high degree of presence is achieved for dynamic flight conditions, then there potentially could be negative transfer to actual flight situations. For example, compensation might not be made initially for actual inertial forces generated in flight because this compensation has been trained out in the simulator. To deal with such issues, it will be necessary to explore systematically the possibility of adapting to multiple environments simultaneously. It is notable that motion sickness in flight simulators is more common in experienced pilots than in flight trainees (Kennedy et al., 1990). The pilots expect patterns of forces that are absent in the simulator; for example, in a flight simulator, backward tilt is often used to simulate forward acceleration; by contrast, during actual forward acceleration the gravitoinertial resultant force on the individual rotates and changes in magnitude.
OCR for page 211
Virtual Reality: Scientific and Technological Challenges Locomotion During normal walking, the pattern of visual flow is determined by step frequency and stride length. Muscles of the neck and torso are activated cyclically to keep the head and torso from being rotated backward by the reaction forces generated. The walking individual appropriately perceives the visual surroundings as stationary, the walking surface as stationary, and correctly appreciates his or her actual stride frequency and length. These relationships are dependent on dynamic sensorimotor calibrations that are tuned over time to accommodate for changes in body dimensions, for example, in leg length and body weight. When the relationship between visual flow and locomotory movements is disrupted (e.g., by having a subject walk in a circular treadmill within a large optokinetic drum) so that the visual flow is inappropriate in speed or in direction for the stepping movements being made, a variety of perceptual remappings occur (Bles, 1981; Lackner and DiZio, 1988). These remappings are such as to normalize the experienced body displacement through space and the apparent frequency, direction, and length of stepping movements (Lackner and DiZio, 1988, 1992, 1993). If visual flow is double normal for the stepping movements being made, the individual will perceive either an increase in stepping rate or that the stepping leg lengthens to give the extra displacement to the body. By contrast, if the visual flow is inappropriate in direction, an individual will feel he or she is voluntarily stepping backward when actually making forward-stepping movements, or feel that forward steps paradoxically propel them backward. If the person is holding a stationary bar, the bar will seem to take on a life of its own. With double normal visual flow, the bar will seem to pull the person forward, causing him or her to go faster; with reverse visual flow, the bar seems to move backward, forcing the individual to go backward. This means that the perception of causality and of the forces on the body are being perceptually remapped along with the individual's perception of volitional activity and body configuration. When individuals are exposed to visual velocity increases, such as would be associated with body acceleration, they have difficulty controlling their stepping movements and may have to hold on for support. Interestingly, when individuals walk at a constant speed on the treadmill during conditions of constant visual flow (regardless of whether the flow is appropriate in direction or magnitude for the actual stepping movements being made), they experience relatively few motion sickness symptoms. By contrast, individuals who are seated and exposed to the same visual flow patterns will report symptoms within a few minutes. It is likely that individuals walking on a treadmill who are exposed to variations in visual flow velocity will experience many more symptoms.
OCR for page 212
Virtual Reality: Scientific and Technological Challenges VE systems that embody mismatches between patterns of visual flow and activity associated with locomotion can be expected to distort the perception of body displacement and of voluntary activity and even of body dimensions. However, for VE displays involving locomotion, the degree to which motion sickness is evoked may be relatively minor when visual motion is coupled to voluntary activity. Body Orientation Under terrestrial conditions, the force of gravity provides a constant reference for orientation, determining unequivocally the direction of up and down (Howard and Templeton, 1966; Howard, 1982). Our environment, both natural and artificial, also embodies an orientational polarization. The trunks of trees are near the ground (down) and their tops are up. Rooms have floors and ceilings. Within this environment, only certain body orientations and configurations are possible. Locomotion can only take place on the floor of a room, not the walls or ceiling. This means that only certain perspectives of the room are naturally possible. For example, one cannot view the floor of the room or the walls from the perspective of being physically located at the ceiling. Similarly, one cannot (without being artificially suspended) see one's feet spatially separated from the floor when no other part of one's body is in contact with the floor of the room. These considerations raise the possibility that if one explores a VE by means of a locomotory walking interface until one has mastered the geographical layout of the environment, then this same environment may seem unfamiliar if one traverses it in a different way, such as using a control stick allowing one to ''fly" through it. Experience in weightless environments in which terrestrial constraints on orientation can be violated shows that our prior cognitive experience with 1G environments limits the perceptual patterns that are experienced (Lackner, 1992a, 1992b, 1992c, 1993). In particular, patterns of orientation cues that would not be possible on earth are interpreted in such a fashion as to create a perceived terrestrial orientation. For example, an individual floating free in an aircraft that is flying parabolic maneuvers so as to create weightless conditions may not correctly perceive his or her true orientation. The person may experience an orientation that would be possible on earth. In other words, a cognitive map of terrestrial possibilities influences the perceived orientation. Vertigo may also be a problem with VE systems depending on the perspectives generated. Height vertigo—sensations of fright and instability usually accompanied by increased body sway—occur when the viewing perspective is elevated and there are no nearby objects visible (Bles et al., 1980). VE systems will have the clear potential to create such circumstances.
OCR for page 213
Virtual Reality: Scientific and Technological Challenges On earth, sensations of falling are experienced during actual falling. However, there may be a strong cognitive component to the elicitation of these sensations. During actual free fall (e.g., in parabolic flight or in orbital flight), individuals do not generally experience sensations of falling; they feel stationary (Lackner, 1992a, 1992b). Consequently, visual motion and quick loss of support and knowledge that one is falling, for example, from a ladder, may trigger the sensations under terrestrial conditions. However, VE systems have the possibility of creating situations, for example, of a person going out a window and moving downward, that could well evoke feelings of falling and defensive reactions. One key in designing environments will be to evaluate how different motions through—and positions within—the environment can be expected to affect orientation and movement control. Illusions of Self-Motion The perception of ongoing body orientation is influenced by multisensory and motor factors. There are a variety of techniques that can be used in VE systems to provide compelling sensations of motion. However, as we note below, often when individuals experiencing such apparent motion make voluntary head movements, or their heads are passively moved, they begin to experience symptoms of motion sickness or lose their sense of self-motion. Visual influences on apparent self-motion are often subsumed under the terms circular vection and linear vection (Dichgans and Brandt, 1978). Individuals exposed to constant-velocity visual motion in a large rotating drum soon feel themselves rotating at constant velocity (in the direction opposite the physical rotation of the drum) and see the drum as stationary. When the illusion of self-motion (vection) is highly compelling, tilting head movements can elicit disorientation (pseudo-Coriolis effects partly analogous to those that occur when head movements are made during actual body rotation). This illusion can also cause motion sickness (Dichgans and Brandt, 1978). By contrast, if vection is not compelling or it is just starting to be experienced, head movements can suppress it (Lackner and Teixeira, 1977). Peripheral visual field stimulation is especially effective in eliciting vection, but even small central fields can have an effect (Dichgans and Brandt, 1978). Linear vection can be induced by exposure to constant-velocity linear visual motion (Berthoz et al., 1975; Howard, 1982). Its time course and other characteristics are similar to those of circular vection. Depending on the direction of visual flow, horizontal or vertical linear vection can be induced. Rotation of the visual field about the optical axis can elicit sensations of body tilt and rotation, but the exact pattern experienced also
OCR for page 214
Virtual Reality: Scientific and Technological Challenges depends on head orientation vis-à-vis the body and the gravitational force vector (Held et al., 1975). Whole-field visual motion is commonly used in flight simulators to induce apparent body motion. Rotating sound fields that have strong spatial volume distribution can elicit apparent motion as well as eye movements compensatory for the apparent motion in subjects in the dark (Lackner, 1977). Head movements strongly suppress the induction of apparent self-motion by sound fields. However, it is likely that moving sound fields in conjunction with visual motion would make the vection illusions more compelling. Illusions of self-motion can also be induced in seated individuals by having them pedal a platform moving under their feet or asking them to move their hands to turn a circular railing (Lackner and DiZio, 1984). Such illusions can be extraordinarily compelling and, if tilting head movements are made, disorientation and motion sickness can be evoked. The pedaling illusions work for angular motion and undoubtedly will for linear as well. Tactile stimulation of the soles of a seated subject's feet by a moving surface can also induce apparent self-motion, as can stimulation of the palms of the hands. In fact, in situations involving physical motion of the body, somatosensory stimulation can be as important as, and sometimes more important than, vestibular stimulation in determining apparent body orientation (Lackner and DiZio, 1988, 1993). The spindle receptors within skeletal muscles are important elements in the determination of the overall apparent configuration (i.e., body schema or position sense) of the body. The signals from these receptors are interpreted in conjunction with the motor signals controlling the physical length of the muscles. Mechanically vibrating a muscle excites its spindle receptors, causing it to contract, an effect known as tonic vibration reflex. Resisting the motion of the limb controlled by the muscle will cause illusory motion of the stationary limb. For example, vibration of the biceps muscle of the upper arm will cause illusory extension of the restrained forearm (Goodwin et al., 1972). Such illusory motion is in the direction that would be associated with physical lengthening of the vibrated muscle. It is possible by vibrating the appropriate muscles to elicit apparent rotation about a vertical axis in standing subjects, tilt of the body, or tilt of the head (Lackner, 1988; Lackner and Levine, 1979). Such apparent motion is generally accompanied by eye movements compensatory for the experienced motion. Individuals who are standing and walking in place on a treadmill in darkness or with synergistic patterns of visual flow come to feel that they are locomoting over a stable stationary surface. Such apparent motion through space is totally compelling (Bles and Kapteyn, 1977; Lackner and DiZio, 1988). The precise patterns experienced depend crucially on the relationship between the visual flow patterns and stepping movements
OCR for page 215
Virtual Reality: Scientific and Technological Challenges (see the discussion of locomotion above). Presenting visual flow that is inappropriate in direction for the stepping movements being made can, for example, make forward stepping individuals feel that they are walking backward over a stable surface. Intersensory effects provide ways of creating various patterns of compelling apparent self-motion and changes in apparent body orientation in stationary individuals. Such experienced motion is usually accompanied by compensatory eye movements and other compensatory postural changes. It should be recognized that physical motion of the body that elicits eye movements also influences visual and auditory localization. For example, a person in darkness exposed to leftward angular acceleration will experience leftward turning motion, and his eyes will exhibit a compensatory nystagmus with slow phase right. If the person is given a target light to fixate that is stationary in relation to his or her body midline, he or she will perceive it to displace leftward off the midline. This phenomenon is known as the oculogyral illusion (Graybiel and Hupp, 1946): there is also an analogous audiogyral illusion. Similarly, if the gravitoinertial force vector is increased in magnitude and rotated by exposing an individual to centrifugal rotation, various types of oculogravic (Graybiel, 1952) and audiogravic illusions are elicited. Analogous kinds of mislocalizations of auditory and visual signals occur during experienced as opposed to actual body motion. Motion Sickness Motion sickness is likely to be a highly significant problem in VEs that involve HMDs and that involve voluntary locomotion or passive displacement of the body. HMDs that significantly influence the effective inertia of the head disrupt the normal sensorimotor control of the head. This in itself can elicit motion sickness and disorientation (Lackner and DiZio, 1989). Likewise locomotion and passive displacement through simulated environments will be unaccompanied by the normal patterns of forces and accelerations associated with such motion through the real environment. The absence of the normally occurring patterns of forces will render many VEs highly provocative in eliciting motion sickness. Motion sickness is often equated with the nausea and vomiting that can occur during exposure to air and sea travel and, more recently, space travel. In fact, it is a much more complex syndrome and at times can be difficult to identify as such (Graybiel et al., 1968; Lackner, 1989). Motion sickness is rarely experienced during everyday activities except under conditions of passive transport. Individuals without functioning vestibular systems seem to be immune to motion sickness; by contrast, all normal individuals are susceptible to varying extents, although there are tremendous
OCR for page 220
Virtual Reality: Scientific and Technological Challenges VEs, a more severe problem after elements of the nausea syndrome have abated is almost certain to be elements of the sopite syndrome (Graybiel and Knepton, 1976). This refers to the chronic fatigue, lack of initiative, drowsiness, lethargy, apathy, and irritability that may persist even with very prolonged exposure periods. Its elicitation would limit the ability to use virtual reality systems on a regular basis. A further disadvantage of the sopite syndrome is that it can persist for prolonged periods even after exposure to the unusual environment is over. WHOLE-BODY MOTION AND LOCOMOTION INTERFACES The control and perception of real whole-body movement and locomotion involves interaction with the world through nearly every possible sensory and motor channel. In the following discussion we sketch out the range of possible levels of technology involvement in whole-body motion and locomotion interfaces relevant to teleoperator and VE systems. Such interfaces have the potential for extending the capabilities of humans at the expense of producing undesired side effects. Exact specification of the cost-benefit ratio for a given system is not possible because we lack a theory of the sensorimotor regularities to which humans are normally attuned or to which they can become attuned. In the following discussion, we consider a variety of interface systems or system components relevant to whole-body motion for teleoperator and VE systems. The discussion is subdivided into three subsections: inertial displays, locomotion displays, and noninertial displays. Inertial Displays These systems induce a sense of passive whole-body movement by exposing subjects to a different, highly constrained, body movement through the use of moving-base simulators. In these systems, a constrained motion base generates, in whole or in part, the pattern of force vectors that would be present in the situation being simulated. These systems are extremely expensive and are usually built either for research or for very specific applications. Full Inertial Displays The goal of these systems is to simulate continuously the pattern of forces that would be present in the real situation. A high-fidelity example is the Dynamic Flight Simulator at the Naval Air Development Center in Warminster, Pennsylvania. This is a large 16,000 HP gimballed centrifuge that can simulate the angular accelerations and G-loading encountered in
OCR for page 221
Virtual Reality: Scientific and Technological Challenges F/4A, F/A-18, and other aircraft. This device can convey the feel of actually flying with no external visual display. These types of devices represent the technology to be replaced by VEs because they are very expensive and are inflexible in terms of simulating multiple aircraft. One new item in this class that deserves mention is the VEDA Corporation Model 2400 Mark IV Vertifuge. It has two components: one is a short arm (2.5 m) centrifuge with a double gimballed capsule capable of accepting multiple cockpit simulators or a virtual cockpit; the other component is software that maps the pattern of force vectors that would be associated with the manipulation of cockpit controls in an actual aircraft onto the distribution of torques in the degrees of freedom of the simulator that would produce the same patterns of forces. The software attempts to minimize spurious Coriolis affects that are usually associated with short-radius centrifuges by combining the devices' centrifugal accelerations with the primary, secondary, and tertiary Coriolis forces. It also takes into account aspects of vestibular control of perceived orientation in trying to enhance the simulation. All the systems in this class can reproduce the specified patterns of force vectors imposed on an individual or other object that remains stationary within the simulator. However, all of them fail to simulate both the mechanical dynamics of any voluntary movements made within the simulator and the vestibular stimulation that would be generated during head movements. Partial Inertial Displays The goal of these systems is to provide a subset of the forces that would occur in the situation being simulated. There are devices, such as tilting platforms, that can provide the initial cues compatible with inertial body acceleration but cannot simulate sustained acceleration. Complementary devices often provide some sort of cue that sustains the experience of self-motion (e.g., see the discussion of G-seats below). A six-legged synchronized motion base for a commercial flight simulator is an example of a system in which a subject is seated in position and exposed to a low-fidelity simulation of the forces that would be present in a high-performance aircraft. The base can pitch back and translate forward slightly to partially simulate a forward inertial acceleration. The motion is usually used to enhance the sense of self-motion driven mainly by a visual display. Such systems are widely available commercially, so no specific example is mentioned. G-seats are used to complement the partial inertial simulations provided by motion bases. A G-seat is itself stationary, but the seat pan and back can be deflated to allow the user to sink into the seat as would be the
OCR for page 222
Virtual Reality: Scientific and Technological Challenges case during forward and upward accelerations. The cushions can be inflated to simulate the reverse. Some G-seats have an active harness system that controls the pressure applied to the front of the body. Increasing the tension in the harness and simultaneously thrusting the user forward with the seat cushions simulates deceleration. Some G-seats also have the capacity for providing vibration cues. Most vehicles vibrate in a way that is correlated with their acceleration through space. For experienced users, providing vibration cues enhances perceived self-motion, even when the proper accelerative forces are lacking. G-seats may be considered haptic interfaces for the whole body but are listed here because of their close functional association with moving base simulators and the intimate theoretical link between haptic and inertial cues. An example of a G-seat that has all the capabilities just listed is the ALCOGS device at Wright-Patterson Air Force Base. Variable Gravity Displays The Graybiel Laboratory slow rotation room at Brandeis University was designed in part to study artificial gravity. This rotating room is an extremely flexible research tool that enables a subject to be exposed to a noninertial, non-1G force background while moving about freely and being monitored by complex on-board equipment. Examples of the room's flexibility as a research tool is the ability it gives to investigate all of the fundamental problems described in the first section of this chapter, including the nature of automatic load compensation during arm and head movements, the control of eye movements, and sensory localization during body movement. It can also be used to study the control of locomotion in normal and moving environments. Such studies are virtually impossible to conduct in small chambers in which locomotion is limited and there is no room to set up complex equipment for three-dimensional movement analysis. Another way of simulating non-1G environments is by placing the body in a weighted or counterweighted sling and adjusting the angle of the body in order to increase the effective contact forces between the support surface and the soles of the feet along the body's long axis. The National Aeronautics and Space Administration (NASA) funded a facility at Langley Field in the 1960s that used suspension but still allowed locomotion around a track that was banked such that the component of gravitational force acting along the body's long axis was equivalent to the moon's full gravitational force. Results of studies with this device helped to ascertain what patterns of gait would be most energy efficient and be easiest to control during exploration of the moon's surface.
OCR for page 223
Virtual Reality: Scientific and Technological Challenges Locomotion Displays Ideally, a ground surface interface (which might be called a haptic interface for the feet) is a device that permits the user to experience active sensations of walking, running, climbing, etc., while remaining within a constrained volume of space. Such a device must allow the user to move his or her limbs in a natural fashion and provide feedback to the user that is matched to the space-time characteristics of the simulated surface (e.g., an inclined plane or a heaving ship deck) and must sense the behavior (positions, movements, forces) of the user in order to both control the actions of the device and to provide appropriate information to the other display systems in the synthetic environment (SE). Locomotion interfaces can provide the experience of moving about in a large space while actually being confined to a small space. To the extent that such devices reduce the workspace volume of the synthetic environment (SE) system, they reduce the requirements on the other system components. For example, both the subsystem for general monitoring of position and the subsystem for grounding of the haptic interface (for the hands) need not cover large spatial regions. More significant, however, it extends the range of applicability of the SE system to situations in which inclusion of locomotion and/or controlled surface conditions are important, i.e., situations in which conditions underfoot are abnormal, in which visual information is severely reduced, or in which several individuals must coordinate their movements. Treadmill-Type Displays The term treadmill originated to describe a form of prison punishment: walking on an endless belt driven by rollers has risen socially to a form of voluntary exercise, and the technology has advanced mainly to serve the commercial demands of health clubs. A typical high-performance treadmill (for example, one made by the Quinton Corp.) has some features that make it suitable for conversion to a research or SE tool. The motor and belt have a combined stiffness that allows the horizontal ground contact forces usually applied in walking to be manipulated. It is microprocessor-controlled so that users can preprogram their preferred exercise regimens, but real-time control must be achieved by a custom interface. The entire base can be inclined under the control of a motorized drive system, which could also be controlled in real time by an external custom interface. A major limitation of commercial treadmills is their one-dimensional nature. A first step in expanding their capabilities would be the development of systems with a belt for each foot. A split-belt system would be
OCR for page 224
Virtual Reality: Scientific and Technological Challenges useful for simulating turning. The laboratory of Esther Thelen at Indiana University has a custom-built device suitable for such research with infants. A useful addition to treadmill systems is being developed by the Kistler Corporation. This system has a force-plate under the belt that can sense the vertical ground reaction forces produced during locomotion. This could potentially provide information that could be fed back to the belt through a real-time control algorithm to simulate slippery ground surfaces, for example. Haptic Interfaces for Individual Feet Treadmills will be limited to use in SEs that involve locomotion on uniform (but not necessarily horizontal) ground surfaces. Stair climbing exercise machines are just vertical treadmills, in this sense. It may be desirable to generate SEs with arbitrary footing conditions, such as sandy, muddy, and icy conditions, discrete obstacles, stairs, wobbly platforms. A general approach to realizing these conditions would be to develop individual six-degrees-of-freedom platforms or shoes for each foot. Such a system would be analogous to force-feedback haptic devices for the hands. Challenges to development include deciding whether anthropomorphic linkages are necessary, identifying drive systems with sufficient power and bandwidth, and learning more about the role of impedance matching in human locomotion. Programs that have led to the development of improved bicycles and human-powered aircraft have faced some of these problems (especially impedance matching) and may provide an initial strategy and some data for the present application. Noninertial Displays Noninertial displays induce a sense of whole-body movement in stationary individuals although they can also be used in conjunction with moving bases. Many simulators currently work by presenting stationary individuals with stimuli that are normally associated with body motion so as to enhance a sense of self-motion. This section concentrates on new approaches in this area. Visual Displays A variety of devices are being used at present for stimulating individuals with representations of scenes that change in a way that is consistent with body motion through the environment in a vehicle. The major characteristic of visual displays to be used in SEs involving whole-body motion and locomotion is a large stereoscopic field of view. HMDs require
OCR for page 225
Virtual Reality: Scientific and Technological Challenges hardware and programming to adjust the display to compensate for changes in viewpoint brought about by head movements. A high-resolution display is not critical for visual induction of illusory self-motion. However, if a low-resolution display is used, the visual scene must contain low spatial frequencies and strong pictorial depth cues. Some real visual scenes contain only relatively high spatial frequency textures. If a high-fidelity facsimile of such a scene is moved, an observer experiences self-motion; however, presentation through a low-resolution display unit does not result in perceived self-motion. Obviously, the need for wide (and high) fields of view, the preference for high resolution, and changes in the visual scene to accommodate head movements will generate a high demand for computational and rendering speeds. Although there is no clear psychophysical data on required frame rates, occasional skips in a display that is being generated at an average rate of 30 frames per second can reduce the sense of virtual body motion (assuming there is no moving base or any other supplementary stimulus). Research that may help guide the integration of visual displays with other SE subsystems for simulating human locomotion is being carried out in several university laboratories (Lackner and DiZio, 1984, 1988; Warren, 1993). One device used for such research includes an independently controllable circular treadmill and visual surround developed by Lackner and DiZio at Graybiel Laboratory, Brandeis University. The circular treadmill is advantageous because combinations of overground and treadmill walking can be combined in the same apparatus. The subject can move through space without walking away from the treadmill and visual display. In this device it is possible to have a subject walk in place at a set pace on the treadmill while the visual display presents a scene whose movement varies in speed and direction. In this situation, the subject perceives whole-body movement with a range of speeds and directions and varying patterns of stepping consistent with it (Lackner and DiZio, 1988). Warren and his colleagues at Brown University have shown that altering the normal ebb and flow of visual feedback (expansion-contraction, vertical oscillation) associated with the walking cycle can alter the pattern of locomotion and the perception of self-motion. This points out the need for developing physical models of the visual feedback from voluntary movement and incorporating it into visual simulations of whole-body movement. Auditory Displays Two techniques have been used in this area: one involves presenting a binaural auditory stimulus that exhibits the same spatial and temporal pattern as it would during movement through a natural environment; the
OCR for page 226
Virtual Reality: Scientific and Technological Challenges other involves cognitive cuing. Induction of illusory self-rotation by simulating a sound field rotating around the head of a stationary subject is an example of the former (Lackner, 1977), and augmentation of visually induced self-motion by virtual wind or engine noise in an aircraft simulator is an example of the latter (Previc et al., 1991). Technology for realizing the first approach is discussed in Chapter 3. There are no adequate psychophysical data to specify the exact characteristics of auditory displays that are critical for inducing a sense of self-motion. However, neither the number of possible sound sources nor the fidelity with which these sources are simulated is critical for inducing a sense of self-motion. Of more importance is creating the impression of a virtual terrain through which self-motion can occur and, toward this end, the ability to simulate a rich set of sound reflecting surfaces. Technology for simulation of echos and reverberation is also discussed in Chapter 3. Vestibular Displays A sense of body motion in a stationary subject can also be elicited artificially by stimulating the semicircular canals of the vestibular system. One method consists of directing streams of cool air or water into one auditory canal and warm into the other. A few seconds of such caloric irrigation can lead to several minutes of perceived self-motion and nystagmus. There are at least two mechanisms governing this response. Altering the temperature of the temporal bone around the external auditory canal (1) locally alters the temperature of the membranous labyrinth of the lateral semicircular canal encased within the bone, and thereby causes a convection current that mechanically stimulates the hair cells of the crista (Barany, 1906) and (2) alters the temperature of the hair cells and primary afferent fibers by direct conduction, leading to modulation of their activity levels (Coats and Smith, 1967). The convection component accounts for about 75 percent of the response (Minor and Goldberg, 1990). This is a standard clinical technique for testing vestibular function that could, in principle, be adapted to SE technology. Its drawbacks for application to SEs are that the latency to onset is about 15 s, the effect is limited to the yaw plane of the head, and it can be nauseogenic. Galvanic stimulation achieved by applying a current between the mastoid bones can also be used to elicit apparent self-motion by directly exciting the vestibular end organs. Proprioceptive/Kinesthetic Displays Even in darkness, a sense of moving through the environment, as well as compensatory postural and oculomotor reactions, arises when
OCR for page 227
Virtual Reality: Scientific and Technological Challenges someone walks in place on a treadmill (Bles and Kapteyn, 1977; Lackner and DiZio, 1988) or manually pushes a revolving railing around (Brandt et al., 1977; Lackner and DiZio, 1984). These results demonstrate that if individuals produce the locomotory movements that normally propel them through space (but without actually displacing), the associated muscular, joint, and tactile feedback, as well as efferent signals, lead to an experience of self-motion. Even if no locomotor movements are made, patterns of cutaneous or muscular afference that are normally associated with movement through the environment can induce apparent self-motion. One method for achieving this is passing a moving surface against the soles of the feet or the palms of the hands; individuals then report a sense of body motion in the opposite direction. Presenting differential surface speeds to the hands and feet can lead to the feeling that the torso is twisting (Lackner and DiZio, 1984). Another method of eliciting whole-body motion utilizes muscle vibration. If a standard physiotherapy vibrator oscillating at 120 Hz is applied to the body surface overlying a muscle tendon, such as the Achilles tendon, the spindle receptors of the associated muscle are stretched relative to the extrafusal force-generating fibers because of differential viscoelastic properties. This increases the output of muscle spindle Ia and probably Ib fibers relative to the level appropriate for maintenance of the desired posture. The muscle contracts reflexively to relax the spindle and restore its output to the original level, and the limb controlled by the muscle moves—for example, the ankle extends. This is called a tonic vibration reflex (TVR). If a limb is prevented from moving under the influence of a TVR, the spindle activity remains high, and a limb movement will be experienced that is consistent with the muscle's being stretched, for example, flexion of the ankle (Goodwin et al., 1972). When standing on the ground, ankle flexion would ordinarily mean that either the ground is tilted up or the body is tilted forward. Vibrating the Achilles tendons of a subject who is restrained in a standing posture thus elicits an experience of falling forward (Lackner and Levine, 1979). Lackner (1988) has shown that virtually any apparent movement of the body can be elicited by vibration of the proper postural muscles. The movements experienced can be supranormal in the sense that anatomically impossible apparent body configurations are generated—for example, hyperextension of limbs. RESEARCH NEEDS The research and development efforts on whole-body motion displays that are needed for development of the SE field, beyond those directed
OCR for page 228
Virtual Reality: Scientific and Technological Challenges toward achieving more computational power, are summarized briefly in the following paragraphs. Inertial displays Relatively simple, partial inertial displays need to be developed to replace complex, full-inertial displays because of the cost factor. Currently, however, there is little technology available for reproducing critical subparts of the accelerations that are present in real situations involving active or passive transport through a large volume. Development of such systems should take advantage of the intimate relationship that normally exists between whole-body movement in inertial space and the contact forces that must be applied to the body in order to accelerate it. For example, when a subject is accelerated, the vestibular system senses the inertial motion and cutaneous receptors respond to the contact force propelling the body. Research needs to be focused on this area of haptic-vestibular interactions. An area of technology development that could assist in exploring the roles of these two factors is new padding materials that can allow experimenters to systematically manipulate the distribution of contact forces on the body surface during accelerations in an inertial display. Electrorheological materials hold some promise for research applications and eventually, perhaps, for SE displays. Another area that could benefit from attention is how active movements affect the perception of whole-body motion induced by noninertial whole-body motion displays. Achieving the same body-referenced limb or head motions requires different muscle forces in stationary and accelerative environments and also leads to different sensory feedback because of noncontact inertial forces on the limb. The lack of expected feedback for the state of body motion being experienced can inhibit the perception of self-motion and lead to a perceptual mapping that better fits all the current sensors. Making head movements during visually induced illusory self-motion can suppress or enhance the sense of self-motion, depending on whether those movements are in the plane of motion and begin when visual stimulation begins or are out of the motion plane and begin after its onset. Basic research may help determine methods for preventing the inhibition of perceived whole-body motion when the head or arms move or enhancing a weak sense of self-motion. A crucial issue here concerns the extent to which methods can be found for inducing people to perceive contact cues provided by means of haptic VE displays as noncontact inertial perturbations of their limbs. Locomotion displays Current locomotion displays consist of constant-speed linear treadmills that can provide an individual confined to a small volume with the pattern of visual, auditory, and tactile cues that
OCR for page 229
Virtual Reality: Scientific and Technological Challenges would be present if they were locomoting through a larger space. However, the only situation that can physically be mimicked by such a device is constant-velocity, linear locomotion. There are several routes to expanding on these capabilities. First, visual, auditory, and other displays may be used to enhance simple treadmills. To accomplish this, psychophysical work is needed to determine the degree to which perceived acceleration and deceleration (including rotation) can be elicited by such displays, even when it is absent in the mechanical stimulus. Another direction is to improve the treadmills. Linear acceleration can be simulated if hardware and software interfaces are developed that allow control of treadmill acceleration and deceleration when propulsive step forces are generated by the subject. Another advance would be treadmills with a belt for each foot. This would be the simplest version of individual haptic interfaces for each foot and would better allow simulation of changes in direction, i.e., turning. Finally, research should be performed with a view toward providing a system that has separate multidegree-of-freedom platforms for each foot with appropriate sensors and feedback subsystems that can mimic the conditions of walking on level or inclined ground, climbing stairs, and navigating around and over obstacles. The padding materials mentioned above for inertial displays designed for passive whole-body motion might also be useful here for simulating different ground conditions. Visual displays Requirements on visual displays imposed by consideration of passive whole-body motion or active locomotion are similar to those previously mentioned in other chapters: the best displays would be an HMD that is inexpensive, lightweight, and comfortable; has high resolution and a wide field of view (both horizontally and vertically); and includes both full-color images and refined stereopsis. Of all these characteristics, color is probably the least important. Auditory displays The most important needs in this area concern those features of the synthesized acoustic field relevant to the illusion of moving through the field. Aside from simulating changes in the direction of sound sources, changes in the apparent distance of the sources and changes in the apparent location of the individual within the reflecting environment are important. Thus, one of the main special needs associated with passive whole-body motion and active locomotion in this area concerns the inclusion of a rich array of reflecting surfaces in the acoustic simulation. Motion sickness Over the years, motion sickness has arisen as a significant problem with all new modes of passive transport of the body (Guignard and McCauley, 1990). Clearly it will be a problem in SEs as well, especially those involving virtual acceleration and motion of the body (Biocca, 1992). Reports of sickness in SEs are already common
OCR for page 230
Virtual Reality: Scientific and Technological Challenges (McCauley, 1984; McCauley and Sharkey, 1992). Research should be directed toward identifying the factors that determine which SEs are especially provocative and how to minimize this while preserving the efficacy of the system. Mechanical factors, such as altered inertial loading of the head by HMDs, as well as sensory factors, need to be considered. Also, attention must be given to elements of the sopite syndrome that are more subtle then those usually associated with motion sickness. Sensorimotor loops Many SE systems introduce distortions, time delays, gain changes, and statistical variability (noise) between voluntary movements and associated patterns of sensory feedback. Systematic research is necessary to determine the extent to which these factors degrade performance and the subjective state of the user. Acceptable tolerances should be determined for these factors, as well as for the extent to which sensory feedback across different modalities must be in temporal synchrony. Multisensory and motor influences on orientation This is a critical research area for designing effective VEs that involve locomotion and haptic exploration. Very little is known at present about these influences, except that they are highly complex and pervasive. They are difficult to identify as such because so much of what we take for granted in our everyday activities, such as the perceptual stability of our environment and our bodies during movement, is due to their action.
Representative terms from entire chapter: