12
Specific Applications of SE Systems

DESIGN, MANUFACTURING, AND MARKETING

Computer technology generally, and synthetic environment (SE) technology more specifically, are potentially important drivers for future developments in design, manufacturing, and product marketing. Trends already under way suggest a movement toward the development of manufacturing systems in which production processes are integrated with all elements of the product life-cycle from concept through sales, including quality, cost, schedule, and the determination of user requirements. This process, known as concurrent engineering, provides for parallel development across product life-cycle activities through the use of technologies such as computer-aided design/computer-aided manufacturing (CAD/CAM) and computer-integrated manufacturing (CIM). Using shared databases, customers, designers, and production managers can simultaneously evaluate a proposed product design. As a result, the design of the product, as it evolves, can incorporate the requirements of the user, special needs for marketing, and any limitations of the production process (Krishnaswamy and Elshennawy, 1992). Once developed, advanced visualization technologies such as virtual environments (VE) may provide valuable extensions to current practices.

In April 1993, manufacturing was named as one of six national initiatives to be administered by the Federal Coordinating Council for Science, Engineering, and Technology (FCCSET). The primary focus of the FCCSET mission on manufacturing is to assess special opportunities for



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 387
Virtual Reality: Scientific and Technological Challenges 12 Specific Applications of SE Systems DESIGN, MANUFACTURING, AND MARKETING Computer technology generally, and synthetic environment (SE) technology more specifically, are potentially important drivers for future developments in design, manufacturing, and product marketing. Trends already under way suggest a movement toward the development of manufacturing systems in which production processes are integrated with all elements of the product life-cycle from concept through sales, including quality, cost, schedule, and the determination of user requirements. This process, known as concurrent engineering, provides for parallel development across product life-cycle activities through the use of technologies such as computer-aided design/computer-aided manufacturing (CAD/CAM) and computer-integrated manufacturing (CIM). Using shared databases, customers, designers, and production managers can simultaneously evaluate a proposed product design. As a result, the design of the product, as it evolves, can incorporate the requirements of the user, special needs for marketing, and any limitations of the production process (Krishnaswamy and Elshennawy, 1992). Once developed, advanced visualization technologies such as virtual environments (VE) may provide valuable extensions to current practices. In April 1993, manufacturing was named as one of six national initiatives to be administered by the Federal Coordinating Council for Science, Engineering, and Technology (FCCSET). The primary focus of the FCCSET mission on manufacturing is to assess special opportunities for

OCR for page 387
Virtual Reality: Scientific and Technological Challenges technology to change the way manufacturing will be carried out in the next century. Rationale Although the ultimate goal of all manufacturing is to produce a tangible object or component, information—in the form of plans, specifications, and processes—plays a most important role. Thus, it should be expected that information technology, including VE, could have a meaningful role in manufacturing by enabling people to generate and manage such information more effectively. Consider the following points in the life-cycle of a manufactured product: Developing design requirements. VE could be used as a medium in which a customer's mental image of a product can be fashioned into a virtual image of the product. That image could be subsequently manipulated or even used as the basis for production specifications. Examples: architectural walkthroughs of spatial designs, such as proposed buildings, rooms, and aircraft interiors. Undertaking detailed design. VE could provide designers with the ability to reach inside the design and move elements around, to test for accessibility, and to try out planned maintenance procedures. The designer could thus have a comprehensive view of how changes made in the design or placement of one component could affect the design of other system components. Producing the artifact. Virtual pilot lines could simulate both human and machine processes on the production line. Such a virtual pilot line could be used to predict performance and to diagnose the source of faults or failures. Plant management could be improved as engineers are given the capability of reviewing and modifying various plant layouts in virtual space. Marketing the artifact. By providing potential customers with the ability to visualize various uses of an artifact, VE could be used for marketing an array of completed product designs to customers prior to their production. Specific Manufacturing Applications Building Prototypes Electronically Building prototypes electronically provides a number of advantages, including the opportunity for sharing data across manufacturing functions and the ability to modify designs with greater ease than in a physical mock-up. A further advantage is the ability to incorporate stress and

OCR for page 387
Virtual Reality: Scientific and Technological Challenges durability test data into the design process without physically performing each test. VE promises to enhance the value of prototyping electronically by offering customers, sales staff, and engineers the ability to walk around the product and manipulate it in virtual space, much the same way as they would explore a physical mock-up in real space. A long-range goal is to create VE systems that can be extended to provide groups of individuals in different locations with the capability to work together in a shared virtual space. Researchers at the University of North Carolina (Airey et. al., 1990) have worked on the development of software for creating interactive virtual building environments. This software can be used to present architectural walkthroughs of buildings that have not yet been constructed. In touring a virtual building, an individual will be provided with changing views and lighting that are consistent with his or her position relative to the building space. Such software can be useful for design of any interior spaces, including industrial buildings, hospital operating rooms, churches, homes, and aircraft passenger compartments, to name a few. Electronic Configuration and Management of Production Lines Another potentially important area for the application of VE technology is in the design and testing of the processing, fabrication, and assembly lines. Virtual pilot lines might be developed instead of real pilot lines to simulate human and machine tasks and make predictions about potential problems for human performance and safety as well as estimating the probability of failure and the line's expected operating efficiency. The promise is that virtual pilot lines will be far easier to modify in response to diagnosed problems than a physical pilot line, and they will provide the opportunity to introduce information on manufacturing efficiency early in the product design process. In addition, a virtual line could be run in parallel with an operating line for purposes of diagnosing failures, retooling for new products, or changing human-machine interface designs or procedures at points in the process at which errors or problems are occurring. Although VE technology provides more of a promise than an existing capability for industry, several forces within various government and manufacturing enterprises will push for its development and use. From the industry perspective, VE technology has the potential to make the manufacturing process (from planning through sales) more flexible and economical. The aerospace, automobile, and textile industries are pursuing VE technology as a means for speeding development and making product modification easier. Chrysler, Ford, and General Motors have formed a VE consortium with the U.S. Army vehicle center, the automotive

OCR for page 387
Virtual Reality: Scientific and Technological Challenges division of United Technologies, the University of Michigan, and several small companies. In a recent proposal to the Advanced Research Projects Agency (ARPA), the consortium predicted that VE technology would lead to improved product design, a better market response, and reductions in time or cost (Adam, 1993). Exemplar Industries In the following sections we provide a discussion of the potential for VE in the textile and aerospace industries. The selection of these industries was not based on an exhaustive or systematic search of industries and applications; however, both industries offer some interesting illustrations of VE technology that transfer broadly to other industries. Textiles VE may have very important applications in the marketing and manufacture of clothing. The concept is that customers could shop for apparel in a VE in which they would see virtual clothes on virtual images of their own bodies and feel how the clothes would fit. On the basis of this experience, customers would select and order outfits that would be fabricated on demand and sent out to them within a short time period. The result would be to significantly reduce financial losses associated with fabric waste during apparel production and with product markdown and liquidation. Moreover, the customer would be provided with a greater range of choice and an improved made-to-measure fit. This approach appears to be a natural extension of the current market trends of increased shopping through catalogs and home shopping networks and the accompanying decrease in retail outlet shopping. Industry Efforts VE technology has captured the interest of the textile industry (Steward, 1993). In 1993, a collaborative research and development program, the American Textile Partnership (AMTEX), was initiated between the Department of Energy (DOE), the DOE national laboratories, and the fibers, textiles, and apparel industry to improve the competitiveness of the U.S. textile industry through the application of technology. The national laboratories plan to work together and coordinate with industry through major industry-supported research and technology transfer facilities. Matching funds for the partnership are to be provided by government and industry. The first joint project between the national laboratories and the industry will involve the creation of an industry model for integrating hardware and software in a system to provide Demand Activated Manufacturing Architecture (DAMA). One aspect of this

OCR for page 387
Virtual Reality: Scientific and Technological Challenges effort will involve research on the uses of VE technology (Hall and Walsh, 1993). The U.S. textile industry, which includes fiber producers, textile weavers, apparel makers, and retailers, employs over 1 million workers (10 percent of the manufacturing work force in the United States) and includes 26,000 companies. It is the largest producer of nondurable goods, experiences annual consumer sales of approximately $200 billion, and contributes $53 billion to the U.S. gross national product (Hall and Walsh, 1993; Steward, 1993). Each year the industry fails to realize revenues of approximately $25 billion due to inventory markdowns and liquidation. Most companies are small, with profit margins of 2 percent or less, and so are not in a position to conduct or support research. Almost all the research in the industry is conducted by five large research centers based in universities and jointly funded by industry and government. One of these centers, the Apparel CIM Center, was established in 1988 with the goals of removing barriers to adopting proven CIM technology, establishing CIM standards, providing assistance to state industry, and conducting broad-based research and development to keep the industry competitive. The primary charge of the Apparel CIM Center is to investigate applications of VE to clothing as seen, examined, and purchased by the retail customer. A second charge is to apply VE technology to represent the internal view of a textile manufacturing plant, including the position of machines, the air conditioning, the noise level, and the lighting. The goal of the project is to facilitate the reorganization of a manufacturing plant by providing engineers and factory workers with the ability to walk through a virtual plant; to move machines around on the basis of requirements to produce new lines of apparel (seasonal changes); to examine spacing, lighting, and noise to ensure good human factors practices; and to assess the effects of various equipment configurations on work flow. Technology Requirements The technology required for implementing the internal plant layout includes: (1) building an object database of all equipment needed in the plant, (2) creating the capability to determine light and ventilation, (3) providing noise levels based on the combination and spacing of machines, (4) matching lighting requirements and noise levels against federal requirements, (5) integrating new software capabilities with existing simulations of work flow through the plant for manufacturing different products, and (6) developing an interface for engineers that is easy to use and acceptable. According to Steward (1993), all of these activities are under way. There are many technologies, including VE, contributing to this application—some existing and some in development. As these activities evolve, there will be a need for VE technology to rely on and interface with other developing information technologies.

OCR for page 387
Virtual Reality: Scientific and Technological Challenges Developing the technology required to fully implement a VE system for marketing clothing is a long-term effort. One area for development is body measurement technology. Currently the concept is to have the customer don a body stocking and be electronically scanned. The linear and volumetric dimensions from the scan would be stored on a card that the customer would use when entering the virtual shopping space. When a customer's dimensions change, he or she could be scanned again. Cyber-ware has built and demonstrated effective full body scanners. However, this technology is produced on an individual basis and is expensive to acquire. A second area of development is the technology for accurately representing material draping. A critical factor in deciding to purchase a garment is appearance: how the jacket hangs, how the folds appear, how the fabric moves when the individual wearing it moves, etc. Thus, the draping of virtual clothes on a virtual customer must appear real. Other areas requiring technology development include providing accurate colors in the virtual world, giving customers the opportunity to ''feel" fabrics, and providing customers with a sense of how the garment "fits." All of these factors are important to customers in selecting clothing. Colors must be accurate so that different parts of an outfit can be matched; feel and fit are critical to comfort and style. Of all the research and technology development issues identified above, the most complex and long range will be developing the tactile feedback needed to create a sense of fit. Aerospace The aerospace industry is expected to be a major user of VE technology in the future. Companies such as Boeing and Rockwell International have long-range plans to develop VE systems that will provide all interested parties with the ability to view and interact with three-dimensional images of prototype parts or assemblies of prototype parts. Currently, both companies are using CAD tools to create electronic prototypes of parts in lieu of physical mock-ups. Industry Efforts Staff at Rockwell International, through its Virtual Reality Laboratory (Tinker, 1993), are working on virtual prototypes and mock-ups; virtual world human factors assessment for proposed task environments; and training for manual factory workers, maintenance personnel, and equipment operators. These efforts are in the early stages of implementation. Proprietary software has been developed to read CAD data into a virtual reality database. The long-range goal is to provide the ability for multiple participants to work together in a shared virtual space interacting with high-resolution CAD data in real time.

OCR for page 387
Virtual Reality: Scientific and Technological Challenges At Boeing, the design of the 777 aircraft is being accomplished without a physical mock-up; all of the 6.5 million parts are being prototyped electronically using CAD tools. As a result, designers, engineers, and possibly customers see only models of parts or assemblies of parts on a computer screen. Although this approach provides for shared databases among design, manufacturing, and sales components and adds significant flexibility to the design process, it takes away the ability to walk around, explore, and manipulate parts. As a result, Boeing is working toward the development of a VE system that would give these capabilities to designers, engineers, customers, and marketing personnel. According to Mizell (1993), the plans for Boeing's VE project include: (1) giving designers the ability to reach in and move parts assemblies around, (2) conducting human factors tests in virtual space, using human models, to determine whether maintenance can be accomplished and control operations can be easily performed, and (3) providing customers with a variety of customized aircraft interiors to walk through and make modifications in real time. All of these applications feed directly back into the design process. Cabin layout modifications made by customers influence the placement of wiring, ventilation, windows, seats, etc. Mizell believes that Boeing would consider the work in virtual reality a success if its only use was to provide customers with the ability to walk through and experience various configurations of aircraft interiors. Implementation of the project is in its initial stages. But Boeing already has software that reads CAD data into a VE preprocessed database. A limiting factor at this point is the computing and graphics power needed to represent the CAD database in a three-dimensional virtual space so that real-time interaction and a feeling of presence can be facilitated. It is anticipated that VE technology will begin to contribute to Boeing's productivity in the next two years, but development will probably need to be continued over a 15-year time period (see the more detailed discussion of implementation issues in the section on technology requirements). A second major project area at Boeing is the application of augmented reality to various parts of the manufacturing process. This project seeks to eliminate the need for complex assembly instructions or manually manipulated templates by creating a system in which computer-produced diagrams are superimposed onto work pieces. The technology to accomplish this goal involves a head-mounted see-through display and a head-position-sensing and real-world registration system. The augmented-reality project is being developed to assist factory workers in performing many complex, manual, skill-based tasks that rely heavily on human perception and decision making and therefore are not easily automated. Currently, guidelines are presented to workers in the form of overlays, templates, or written instructions for each step in the process. When parts

OCR for page 387
Virtual Reality: Scientific and Technological Challenges or processes are modified by designers in the CAD system, a substantial amount of time may be required to reflect the appropriate changes in the manufacturing documentation. The augmented-reality system would link manufacturing instructions with the CAD system and superimpose these instructions in the form of diagrams on work pieces. The diagrams would appear to be painted on. For each step, a new diagram is projected. The link with the CAD system would make it possible to show changes in design or procedures to the worker immediately. A more detailed description of this project is provided by Caudell and Mizell (1992). Technology Requirements In Boeing's augmented-reality project, a prototype system has been developed and tested. The primary technology needs are for a comfortable head-mounted color display with a field of view wider than 30 degrees. Another goal is a position-tracking system that will leave the worker untethered. In order to implement Boeing's vision for using VE, several areas require technology development. One critical problem is the lack of graphics and computing power. The CAD database for the 777 aircraft contains between 5 and 10 billion polygons. Even though only a fraction of the database may be needed at any one time, the existing graphics hardware limits the ability to create a scene that is interactive in real time, particularly because of the complexity of the geometry in the CAD database. The problems created by the size of the database, the inadequate hardware, and the requirement for a VE that looks real and behaves in predictable ways underscore the need for research on real-time scheduling, assigning reduced workload areas, and developing heuristics to accomplish graceful degradation. Another goal requiring technology development is providing engineers with the ability to interact with objects in virtual space. Currently, Boeing is working with a mannequin developed by Norman Badler at the University of Pennsylvania (Badler et al., 1993) that can be put inside the CAD geometry and changed in size or shape. Similar technology has been used by the automobile industry for several years. The next major step is to develop the capability for an engineer to inhabit the mannequin in a virtual space, to move around inside the CAD geometry, perform maintenance checks, and in general, feel present inside the scene while others monitor from a third person. Particularly important development areas include the need for collision detection and the requirement to give the individual in the virtual space some sense of force feedback, especially when testing the difficulty of performing various maintenance operations. Developments in this area, particularly those involving haptic feedback, are at least 10 to 15 years in the future. Creating architectural walkthroughs of customized aircraft interiors

OCR for page 387
Virtual Reality: Scientific and Technological Challenges is another important area for development. These models would provide customers with the opportunity to see and experience the aircraft they are purchasing before it is actually built. MEDICINE AND HEALTH CARE Rationale The knowledge base of medicine has exploded in the past 30 years, and it continues to expand at a staggering rate. As a result, medical practitioners have difficulty in keeping pace with changes in practice, and medical students and residents have difficulty in assimilating the information presented in their medical educations. As in other information-intensive disciplines, computer and communications technologies have important roles to play in reducing the cognitive demands on medical practitioners and students by helping to manage, filter, and process multiple sources of information. The following subset of medical knowledge and skill is well-suited to management and handling by VE, augmented reality, and teleoperator systems: Anatomical relations of various organs and systems. Knowing that a particular organ is located underneath another organ is an essential part of anatomical knowledge. The ability to "walk through" the body and to see anatomy in its natural state with all of the interrelations of various organs and systems would greatly facilitate the acquisition of certain important pieces of anatomical knowledge. Development of manipulative skills involving precise motor control and hand-eye coordination. Surgical trainers can be particularly useful in acquiring needed skills. Image interpretation. Although various imaging devices are common in medicine, their effective use depends on the skill of the viewer to identify often small differences between normal and abnormal images. Telemedicine through teleoperation. Medical expertise is often unavailable in remote areas. Telemedicine—whether through consultation or through remote manipulators that enable teleoperation—offers some potential to place medical expertise in locations that might not otherwise have access to such expertise. Teleoperation also enables one to effectively transform the sensorimotor system of the physician (diagnostician or surgeon) to better match the task. Specific Medical Applications The following discussion focuses on six applications of VE technology to medicine: medical education, accreditation, surgical planning,

OCR for page 387
Virtual Reality: Scientific and Technological Challenges telepresence, telesurgery, and rehabilitation. Each subsection addresses a long-range vision of how VE might assist in these applications and a description of possible near-term demonstrations. Preservice and Continuing Medical Education Medical education has changed little in the last 30 years, despite enormous advances in knowledge. Most medical schools emphasize learning facts by rote. Information is provided in a lecture format, and students study outlines for endless hours in the library. Little effort is expended to place the information into a context or framework that might help to structure and organize seemingly disparate facts. As a result, students must use their own, perhaps incomplete experience to begin assimilating the data and creating a logical, integrated framework of anatomy, physiology, biochemistry, genetics, and the myriad springs of subspecialized knowledge from contemporary medical research. Teaching Anatomy The teaching of anatomy is illustrative, and the application of VE and augmented reality to such teaching has great potential. The static, transparent, two-dimensional overlays typical of anatomy textbooks could someday be replaced by a virtual human. Indeed, today the National Institutes of Health is funding the Visible Human, a project to develop a complete static digital representation of an adult human. Once the data are collected, a student would be able to operate a VE system for anatomy that would illustrate the spatial interrelationships of all body organs relative to each other, selectively enabling or suppressing the display of selected body subsystems (e.g., displaying only the digestive system, viewing the complete image without the circulatory system). A much more sophisticated version of the Visible Human would be a dynamic model that could illustrate how various organs and systems move during normal or diseased states, or how they respond to various externally applied forces (e.g., the touch of a scalpel). Thus, a student could view the heart in normal and diseased states pumping blood, or observe how the stomach wall moved while cutting it. Today, several virtual worlds have been developed to demonstrate basic anatomy and as rudimentary models of training simulators. One is a model of the optic nerve created by VPL (VPL, Inc., 1991). This model illustrates, in three dimensions, the path of the optic nerve from the retina to the optic cortex. By pointing a finger one can fly along this path, looking to either side at adjacent structures. In this way, less effort is

OCR for page 387
Virtual Reality: Scientific and Technological Challenges expended in constructing a three-dimensional image in the individual's mind and more effort is channeled into learning the anatomical relationships. A second model is a rudimentary simulator for the abdomen created by Satava (1993b). With this simulator, one can travel from the esophagus throughout the intestine, taking side trips through the biliary system and the pancreas. It is a unique instructional tool that describes anatomy from the inside of the intestines rather than from the outside. It is of considerable benefit in training individuals to perform colonoscopy and esophago-gastroduodenoscopy, as well as teaching students the true anatomic relationships of intraabdominal structures. Basically, one is able to fly around various organs and experience their actual relationships—the model provides the learner with the ability to interdigitate between organs and behind them without destroying their relationships to one another in the process. Another educational tool is an augmented-reality system that allows the user to see virtual information superimposed over real structures. See-through displays provide the user with a view of the surrounding environment, along with an image displayed on goggles. Investigators in Boston and at the University of North Carolina (UNC) have created see-through displays using computer-assisted tomography (CT) scan, magnetic resonance imaging (MRI), or ultrasound technology as imaging techniques (Bajura et al., 1992). The work on augmented reality at UNC (Bajura et al., 1992) is based on the images from an ultrasound that delineates abdominal structures in three dimensions. Specifically, the investigators created a graphic of a three-dimensional model and projected it through the head-mounted display (HMD), as an overlay onto the user's view of the abdomen. This program, used on a pregnant woman, allows the operator to "open a window" into the abdomen and view the fetus in a three-dimensional manner without incising the skin. Although the application of such programs to view a developing fetus is limited, the technology raises the possibility of visualizing other intraabdominal structures. See-through models can be used to teach surgeons where an organ is located and show its relation to surrounding tissues. Novice surgeons often have difficulty visualizing the location of the gallbladder and the cystic duct in relation to the common bile duct. Despite extensive anatomy instruction, the first few operations are difficult because structures in a living body appear different from the illustrations in an anatomical atlas. A see-through display gives surgeons in training an opportunity to develop their own internal three-dimensional map of living organs, rather than having to operate without one.

OCR for page 387
Virtual Reality: Scientific and Technological Challenges area of required work is the development of computational software to manage large and diverse databases in ways that allow users to explore alternatives and make discoveries. This issue is discussed in more detail in Chapter 8, on computer generation of virtual environments and in the section below on scientific visualization. A second area of required work is the determination of what information should be provided, how it should be formatted, and how we expect the user to interact with it. Researchers have worked for many years to create dynamic two-dimensional information displays for such tasks as monitoring the status of a nuclear power plant, flying a jet aircraft, controlling aircraft traffic, and analyzing complex data. This work has provided some insight into how much information can be absorbed at one time, how it should be organized on the screen, and how frequently it can be updated before the limits of human information processing are exceeded (Ellis, 1993). In addition, cognitive scientists have been exploring the relationship between types of tasks and the most appropriate types of information to support those tasks (Palmiter and Elkerton, 1993). Although some of the knowledge about human information processing, learning, and problem solving gained when using two-dimensional displays will be of value in designing information displays in three-dimensional environments, we will need to mount a substantial research effort to determine how to use the capabilities of three-dimensional environments effectively. An integral part of these research efforts will be a determination of the most user-friendly and efficient interaction techniques. Other chapters of this book provide a discussion of these issues. Of particular importance will be research into the use of sensory modalities other than vision in increasing or modifying the comprehension of information. Currently, little progress has been made in the use of virtual or augmented reality for the purposes of information visualization. However, some investigators have begun to explore various aspects of visualization for scientific purposes. A brief description of results in this area are reported below. Many of the problems raised will be pertinent to the design of information presentation for other types of activities. Scientific Visualization Scientific visualization (McCormick et al., 1987) is the use of computer graphics to create visual images that aid in the understanding of complex, often massive, numerical representations of scientific concepts or results. Such numerical representations, or datasets, may be the output of numerical simulations as in computational fluid dynamics and or molecular modeling, recorded data as in geological and astronomical applications,

OCR for page 387
Virtual Reality: Scientific and Technological Challenges or constructed shapes as in visualizing topological arguments. These simulations may contain high-dimensional data in a three-dimensional volume, and they often vary in time. Different locations in such datasets can exhibit strikingly and interestingly different features, and difficulty in specifying locations will impede exploration. Scientific insight into complex phenomena depends in part on our ability to develop meaningful three-dimensional displays. Rationale Traditionally, scientific visualization has been based on static or animated two-dimensional images that have generally required a significant investment in time and expertise to produce.6 As a result, severe limits have been placed on the number of ways in which a dataset can be explored. That is, an explorer does not know a priori what images are unimportant, but when the effort to produce a visualization is large, there will understandably be a hesitation to produce a picture that is likely to be discarded. Other problems arise with traditional scientific visualization techniques because they are not well suited to the computational datasets associated with modern engineering simulations. These datasets may be inherently complex, consisting of a time series of three-dimensional volumes, with many parameters at each point. Also, scientists are often interested in behavior induced by these data (i.e., streamlines in a vector field) rather than the data values themselves. Under these circumstances, real-time interactive visualization is likely to pay off, due to the complexity of phenomena that can occur in a three-dimensional volume. VE technology is a natural match for the analysis of complex, time-varying datasets. Scientific visualization requires the informative display of abstract quantities and concepts, rather than the realistic representation of objects in the real world. Thus, the graphics demands of scientific visualization can be oriented toward accurate, as opposed to realistic, representations. Furthermore, as the phenomena being represented are abstract, a researcher can perform investigations in VE that are impossible or meaningless in the real world. The real-time interactive capabilities 6   In the early days of visualization, it was rather difficult for the researcher to produce visualizations beyond conventional drawings and plots. Familiarity with computer graphics programming was required to do more sophisticated visualization, a need that was addressed through the creation of ''visualization shops," in which a visualization was produced to order. A researcher provided data to a visualization programmer, who then produced a high-quality image or animation. Thus, there was a significant investment involved in the production of visualization. This served the purpose of visualization as a presentation medium, but it hindered the use of visualization as an exploratory medium.

OCR for page 387
Virtual Reality: Scientific and Technological Challenges promised by VE can be expected to make a significant difference in these investigations, with the potential to provide: the ability to quickly sample a datasets volume without cluttering the visualization; no penalty for investigating regions that are not expected to be of interest; and the ability to see the relationship between data nearby in space or time without cluttering up the visualization. In short, real-time interaction should encourage exploration. Just as important, a natural, anthropomorphic three-dimensional VE-based interface can aid the unambiguous display of these structures by providing a rich set of spatial and depth cues. VE input interfaces allow the rapid and intuitive exploration of the volume containing the data, enabling the various phenomena at various places in that volume to be explored, as well as providing simple control of the visualization environment through controls integrated into the environment. A properly constructed VE-based interface will require very little of the user's attention; it would be used naturally, using pointing and speech commands and directions rather than command-line text input. Someone using such an interface would see an unambiguous three-dimensional display. This would contrast with the current interaction paradigm in scientific visualization, which is based on text or two-dimensional input via graphical user interfaces and two-dimensional projections of three-dimensional scenes. Specific Applications VE systems for scientific visualization are in many ways like software packages for graphing: tools for displaying and facilitating the interpretation of large datasets. But it is too early to describe a single general-purpose VE system for scientific visualization. At the same time, a number of projects have demonstrated that VR does have significant application potential. Aeronautical Engineering: The virtual wind tunnel (Bryson and Levit, 1992; Bryson and Gerald-Yamasaki, 1992) uses virtual reality to facilitate the understanding of precomputed simulated flow fields resulting from computational fluid dynamics calculations. The visualization of these computations may be useful to the designers of modern high-performance aircraft. The virtual wind tunnel is expected to be used by aircraft researchers in 1994 and provides a variety of visualization techniques in both single-user and remotely located multiple-user environments. General Relativity: Virtual Spacetime (Bryson, 1992) is an extension of the virtual wind tunnel in which curved space-times, which are solutions

OCR for page 387
Virtual Reality: Scientific and Technological Challenges to Einstein's field equations of gravitation, are visualized using article paths in virtual reality. Molecular modeling: Molecular docking studies using a VE that included a force-reflecting manipulation device have been performed with the GROPE system at the University of North Carolina at Chapel Hill (Brooks et al., 1990). Investigators employed a head-tracked stereo display in conjunction with the force-feedback arm to investigate how various molecules dock together. These studies have implications for the design of pharmaceuticals. Scanning Tunneling Microscopy: A VE coupled with a telerobot for the control and display of results from a scanning tunneling microscope called the Nanomanipulator has been developed at the University of North Carolina at Chapel Hill (Taylor et al., 1993). This system uses a head-tracked stereo display in conjunction with a force-feedback arm to display a surface with molecular resolution via graphics and force reflection based on data obtained in near-real time from a scanning tunneling microscope. In addition, there is the ability to deposit very small amounts of material on the surface via direct manipulation by the user. Medical visualization: Medical visualization systems using augmented reality (e.g., Bajura et al., 1992) have been developed at several sites. The primary difficulty with medical visualization at this time involves the very large amounts of graphic data being displayed. Bajura et al. are designing a system that will map ultrasound imagery in real time onto the physician's view of the real patient, allowing the location of the features shown in the ultrasound imagery to be quickly and intuitively located in the patient. Astrophysics: A system to investigate cosmic structure formation has been implemented at the National Center for Supercomputing Applications (Song and Norman, 1993). This system visualizes structure arising from simulations of the formation of galaxies in the early universe. Circuit Design: The Electronic Visualization Laboratory at the University of Illinois, Chicago, has implemented several scientific visualization applications in a virtual environment setting. For descriptions of the individual projects, see Cruz-Neira et al. (1993a, 1993b). Issues to be Addressed Experience with VE-based scientific visualizations has shown that in order to sustain usable interaction and to make the user feel that a series of pictures integrates into an insightful animation, a number of criteria must be met. First, the system must provide interactive response times to the user of approximately 0.1 s or less. Interactive response time is a measure of the speed with which the user sees the results of actions; if the

OCR for page 387
Virtual Reality: Scientific and Technological Challenges interactive response time is too slow, the user will experience difficulty in precisely placing visualization tools (Sheridan and Ferrill, 1974). Second, effective systems for scientific visualization must have animation rates of at least 10 frames/s. Animation rate is a measure of how fast images are presented to the user; this rate is particularly relevant with respect to viewing control and for time-varying datasets. If the rate is too slow, the images will be perceived as a series of still pictures rather than a continuous evolution or movement. These two parameters are psychologically and perceptually related, albeit computationally distinct. Some VE systems may separate the computation and visualization processes, so that they run asynchronously. We are at the beginning of understanding the potential of this technology for scientists. Research is needed to answer such questions as when are continuous images more useful than discrete images for scientific insight. Scientific visualization also makes particular demands on virtual reality displays. The phenomenon to be displayed in a scientific visualization application often involves a delicate and detailed structure, requiring high-quality, high-resolution, and full-color displays. Experience has shown that displays with 1,000 × 1,000 pixel resolution are sufficient for many applications. In addition, a wide field of view is often desirable, as it allows the researcher to view how detailed structures are related to larger, more global phenomena. Lastly, user acceptance criteria suggest that few researchers would be willing to invest the time required to don and doff head-mounted displays available at the time of this writing. Furthermore, many researchers have expressed distaste for donning helmets or strapping displays onto their heads. TELECOMMUNICATIONS AND TELETRAVEL Rationale As facilitators of distributed collaboration, the applications of telecommunications and teletravel cut across all of the other applications discussed in this chapter. For manufacturing activities of the future, it is anticipated that virtual images of products will be simultaneously shared by geographically dispersed design engineers, sales personnel, and potential customers, thus providing the means for joint discussion and product modification. In health care, there are several examples of distributed collaboration, including remote surgical practice and remote diagnostic consultation among patients, primary care physicians, and specialists who may all be viewing common data or three-dimensional images. An example of the latter is the development of a telemedicine system in Georgia,

OCR for page 387
Virtual Reality: Scientific and Technological Challenges in which medical center expertise is shared over a network with rural doctors and their patients. For education and training, there are many instances in which distributed collaboration may be useful. One example is the use of a shared virtual battlefield for mission planning, rehearsal, and training. Another potential use is offering students from several schools around the country the opportunity to come together through network technology to share a common virtual world—such as a reconstruction of a historic site that no longer exists. Finally, in hazardous operations, distributed collaboration is a central feature of humans and telerobots working together is the same remote environment. The discussion in this section focuses on the increasingly collaborative nature of modern business and the potential contribution of SE technology to facilitating this collaboration in a cost-effective manner. Specifically, we discuss telecommunication and teletravel. Both of these processes use technology to reduce unproductive travel time to and from meeting sites or regular work sites. Today, many of those who are knowledge workers already use technology to avoid travel. The nature of a knowledge worker's job is such that by working at home or in satellite locations using personal computers, modems, and the telephone network, these telecommuting workers can perform their jobs reasonably well. Greater understanding and acceptance of this phenomenon in the workplace is illustrated by the response of the Los Angeles work force to the earthquake of January 1994. In the aftermath of the disaster and the consequent disruption of customary commuter routes, telecommuting increased dramatically. However, most workers in the United States do not have jobs that can be performed using only a screen, a keyboard, and a mouse. Most sales people must interact face-to-face, and others, such as craftspeople, work on solid objects with their eyes and hands. Even telecommuting knowledge workers need face-to-face meetings for discussions involving more than two people, job interviews, and many other work situations in which gestures, facial expression, and eye contact are critical components of the interchange. These added task requirements open the door to the next step in the use of VE technology. Background The historical evolution of distributed collaboration provides a useful context for this discussion. Study of the paths followed in the earlier efforts in both research and system development can reveal the current robust status of telecommunication facilities, as well as the potential consequences of expanding such facilities to include VE capabilities.

OCR for page 387
Virtual Reality: Scientific and Technological Challenges Distributed collaboration first emerged as a product of advanced technology in the form of multiperson telephone conversations. Such services were provided by AT&T during the 1950s with economic benefits for both users and providers. Routine use of this technological capability came in the 1960s, along with expansion of the concept to include secure conferences between remote participants. A separate and special technology, telecommunication, came to be institutionalized as a consequence. Several major systems were developed at this time to serve the needs of the federal government. Among such systems were the first models of AUTOSEVOCOM, a secure network that could support remote conferences between top-level military commanders from their stations around the world (Sinaiko and Belden, 1965). This system also cemented the integration of computers into the multistation communication network. In these early instances, the computer was used only as a switching device and a tool for signal encryption. However, its presence in the system was a definite harbinger of things to come. In addition, the design effort for military systems set off programs of research intended to explore the potentials, both positive and negative, of computer-mediated communication. For example, studies were initiated to determine the feasibility of using a computer-mediated network to link North Atlantic Treaty Organization member heads of state for purposes of joint crisis resolution. Problems ranging from how to implement rules of diplomatic protocol to overcoming language barriers were explored. The outcomes of these research programs revealed some of the limitations on communication effectiveness imposed by an absence of visual information to augment the direct voice transmissions. Digitization, packet switching, and optical fibers, among other technological advances, began to open more vistas in the 1970s. It was then that the first instances of teletravel began to appear (Fordyce, 1974; Craig, 1980) after having been forecast several years previously. Entire new areas of economic advantage began to be apparent, such as the possible savings in gasoline use and pollution reduction. Negatives such as lowered productivity due to a lack of supervisory presence were played down by early enthusiasts but have come to be treated as significant matters in present-day application instances. In any case, telecommuting, as a form of distributed collaboration, has become an accepted option for some workers in some organizations (Shirazi, 1991). The other critical ingredient in the evolution of distributed collaboration was the rapid adoption of small but powerful computers by workers in many different occupations. Computerized networks began to become widespread in the 1980s. In a sense, the computer became an actual participant in multiperson collaborations that were performed on the network. The computer provided an information storage and retrieval capability

OCR for page 387
Virtual Reality: Scientific and Technological Challenges that far exceeded what the humans could contribute. Also, the computer could provide a dynamic color graphics capability that is not available by any other means. Object-oriented collaborations, such as designing electronic circuitry, have been quite successful when these technologies have been employed (Sheridan, 1993a; Fanning and Raphael, 1986). In summary, distributed collaboration is not a particularly new idea. Working in this manner has gradually expanded over the past four decades, as people have accustomed themselves to the concept and its ramifications and as the technology has progressed to a point at which it supports new modes of activity at affordable costs. Now the question becomes: What can or will VE add to the process? Will VE provide the means to take a few more incremental steps in the further expansion of distributed collaboration—or will VE provide the basis for major change in how the concept is actualized? Teletravel and Virtual Environments VE offers the possibility for one to participate in a meeting in which all the other attendees were present in the form of virtual images. Each participant in the virtual meeting would see and hear the other participants through lightweight, see-through VE goggles that resemble eyeglasses, while his or her own appearance was captured by a video camera for broadcast to all the others. Different communications channels would support both group communications and communications to selected individuals—the equivalent of whispering to someone. The social feasibility of virtual meetings goes beyond the technology. The enormous difficulty of scheduling conference phone calls for more than four busy people suggests that a new set of social norms would be needed before virtual meetings could be called routinely. For example, people would have to feel that is was unacceptable to remain in a virtual meeting and attend to other business simultaneously. Shared workspaces refer to the real or virtual gathering of people at a specific location for the purpose of interacting with an artifact or an object. With the appropriate technology for automatic model generation, any physical space—and the relevant artifacts and objects—could be turned into a virtual meeting place. Thus a group could travel to any location where sensors existed and meet while observing events taking place in that real-world location. VE also offers the possibility that one could be part of collaborative communities at a distance. For example, Xerox PARC is currently experimenting with technologies that virtually bring together people working in different offices. Although the current effort is limited to small-screen

OCR for page 387
Virtual Reality: Scientific and Technological Challenges video and audio, VE-based collaborative communities could offer illusions of physical presence so real that offices of collaborators and colleagues could be geographically dispersed much more than they are today. To facilitate the illusion of shared office spaces, office workers might wear glasses that could give them the feeling that their offices were part of a much larger common area in which many other participants were present. Each participant would see the other participants sitting at desks in their own offices. In situations in which a traveler is unable to physically go to the target site, virtual travel may be useful. An example would be to enable inmates of correctional facilities to hold jobs or be trained in the outside world while still being controlled and monitored physically. Police might travel virtually to the middle of a jail riot to gather further intelligence. Researchers, regulators, and site planners might virtually meet in a hazardous environment, such as a nuclear dumping site, to examine conditions and plan operations for the future. It is conceivable that close observation and review of site conditions could provide superior input to planning based on video or still pictures. Teleoperation and Remote Access Teleoperated systems controlled through a VE interface would enable an individual or a group to go beyond mere passive observation. For example, a group on a virtual visit to a nuclear power plant could be authorized to open and shut valves and make other changes to the physical status of the plant. Since the valves and other actuators in such a plant are electronically controlled from a control panel, it is not too far-fetched to imagine the sensor data and control of the operation of the plant being part of the virtual world inhabited by the plant's human operators. The operators might be more aware of the status of the plant in emergencies if they could virtually travel through its radioactive corridors. Any device that was connected to the communications grid could be controlled by a virtually present human. A home security system, for example, could summon home its owner when an alarm went off. (For an example based in telecommunications prior to VE technology, see Taylor, 1980.) With appropriate cameras and sensors, the owner could travel through the house (virtually) to see if intruders were present. With certain actuators present in the home and linked to the network, the owner might either drive off to escape confrontation or try to capture the intruders in the home. A very general means of making changes to the world would be for a distant person to occupy a telerobot to perform some task. Discussions of

OCR for page 387
Virtual Reality: Scientific and Technological Challenges telerobotics tend to treat the link between human operator and telerobot as semipermanent. But if telerobots were common, they might be treated more like telephones—that is, known locations into which one could project one's eyes and hands. Research Needs A key aspect of a virtual meeting is the ability to see body movements and facial expressions, a feature difficult to achieve with current video conferencing systems. In real meetings in real places, the participants perceive themselves to be in a place, surrounded by its walls. They are able to observe the positions of other participants within that space and hear their voices coming from specific directions. These perceptual possibilities are not available from video telecommunication systems. Therefore, the use of directional sound in virtual meetings will be especially important. With as few as two pairs of people having simultaneous independent conversations, the conversations will be disrupted without the ability of the hearers to filter out the unwanted sounds. This is done in real life (the "cocktail party effect") by the human ability to selectively filter sounds based on their directionality, and the fact that sounds from more distant sources are less loud. Both of these properties of real sounds can be supported in shared virtual worlds. To capture an image of a user's facial expression while allowing the user to view the shared virtual world, several display methods are available. One way to do it is for the user to wear a see-through HMD, resembling eyeglasses. A video camera could be trained on the user's face. The see-through capability of the HMD in this case is primarily useful for allowing the camera to see though to the user's face, rather than for allowing the user to see the real surroundings. Another viewing method is to put the user in an immersive viewing station, similar to the CAVE, a room whose walls, ceiling, and floor surround a viewer with projected images (Cruz-Neira et al., 1992). Since the CAVE uses polarized glasses for stereo, a camera is needed that can see through to collect images of both eyes of the user through the polarization. Both of these viewing setups are adequate to allow a group of people, each at a viewing station, to see and be seen by the other members of a group of people occupying a virtual meeting space. The two-dimensional image of face and body that would be collected by a normal video camera is adequate but not optimal. The two-dimensional face texture could be mapped onto a virtual mannequin representing the person in the virtual world, and the same could be done for the two-dimensional body image. This would provide a very flat person to the virtual world, but it would still have advantages over video telecommunication,

OCR for page 387
Virtual Reality: Scientific and Technological Challenges which does not show the locations of the different participants very effectively. A more elaborate body image capture method would be to use range-imaging techniques to capture a three-dimensional model of the body and face. Such automatic three-dimensional model acquisition is needed by other branches of the SE field, and various prototype systems exist for range imaging. A three-dimensional image of the body of each participant in a virtual meeting begins to make such meetings sound like they could actually come close to duplicating the perceptual feel of being physically present at a real meeting. These technologies, though still immature, offer the possibility of electronically projecting oneself, as easily as one currently makes telephone calls, into virtual worlds inhabited by other distant human users, with whom one can have face-to-face interactions both one-on-one and in groups. These shared multiperson virtual worlds create a shared space, in which each human participant has a position, a body image resembling his or her own real appearance, and a viewpoint from which to observe the behaviors and facial expressions of the other people engaged in the transaction.