Modeling and Simulation: Linking Entertainment and Defense (1997)

Page 115

D—
Position Papers

Prior to the Computer Science and Telecommunications Board's October 1996 workshop on modeling and simulation, participants were asked to submit a one- to three-page position paper that responded to three questions:

1. How do you see your particular industry segment evolving over the next decade (i.e., how will markets and products evolve)?

2. What technological advances are necessary to enable the progress outlined in your answer to question 1? What are the primary research challenges?

3. Are you aware of complementary efforts in the entertainment or defense sectors that might be applicable to your interests? If so, please describe them.

This appendix reproduces a number of these position papers. The papers examine technologies of interest to the entertainment industry and the U.S. Department of Defense, as well as some of the barriers to collaboration. Several of the papers are cited in the body of the report; substantial portions of some have also been incorporated there.

Page 116

Brian Blau—
VRML: Future of the Collaborative 3D Internet

Introduction

VRML (virtual reality modeling language) is the three-dimensional computer graphics interchange file specification that has become the standard for Internet-based simulations. It is being used in many industries, and the momentum of the standard and industry acceptance continues to grow at a fast pace. Most of the major software and hardware corporations are now starting serious efforts to build core VRML technologies directly into business applications, scientific and engineering tools, software development tools, and entertainment applications.

One of the most significant developments in the history of VRML was its adoption by Silicon Graphics Inc. (SGI), Netscape, and Microsoft during 1995-1996. This broad level of industry acceptance continues to challenge the VRML community to provide an official international standard so that wide adoption will be possible. Given that creation of VRML came from a unique and open consensus-based process, its future depends on continued innovation in the directions of true distributed simulations as well as efforts to keep the standards process moving forward.

Historical Development of VRML

Over the past two years the development of a standard for distributing 3D computer graphics and simulations over the Internet has taken the quick path from idea to reality. In 1994 a few San Francisco cyberspace artisans (Mark Pesce, Tony Parisi, and Gavin Bell) combined their efforts to start the VRML effort. Their intention was to create a standard that would enable artists and designers to deliver a new kind of content to the browsable Internet.

In mid-1995 VRML version 1.0 emerged as the first attempt at this standard. After an open Internet vote, VRML 1.0 was to be based on Silicon Graphics' popular Open Inventor technology. VRML was widely evaluated as unique and progressive but still not useable. At this point broad industry support for VRML was coalescing in an effort to kick-start a new industry. Complimentary efforts were also underway to deliver both audio and video over the Internet. The general feeling was that soon the broad acceptance of distributed multimedia on the Internet was a real possibility and that VRML would emerge as the 3D standard.

After completion of the VRML 1.0 standard, the VRML Architecture Group (VAG) was established at SIGGRAPH 1995 and consisted of eight

Page 117

Internet and 3D simulation experts. In early 1996 VAG issued a request for proposals on the second round of VRML development. The call was answered by six industry leaders. Through an open vote SGI emerged as the winner with its Moving Worlds proposal. By this time over 100 companies had publicly endorsed VRML, and many of them were working on core technologies, browsers, authoring tools, and content. At SIGGRAPH 1996 VAG issued the final VRML 2.0 specification and made a number of other significant announcements.

To help maintain VRML as a standard, VAG made several concrete moves. First, it started the process of creating the VRML Consortium, a nonprofit organization devoted to VRML standard development, conformance, and education. Second, VAG announced that the International Standards Organization (ISO) would adopt VRML and the consensus-based standardization process as its starting place for an international 3D metafile format.

Distributed and Multiuser Simulations Using VRML

Based on the current state of technology, it is now obvious that distributed 3D simulations are clearly possible for a wide audience. Distributed simulation is the broad term that defines 3D applications that communicate by standards-based communications protocols. Military training, collaborative design, and multiuser chat are examples of such applications.

Widespread adoption of this technology depends on the following key technology factors: platforms, rendering, multimedia, and connectivity. Today, the most popular platforms for accessing the Internet are desktop machines—namely, Windows 95/NT and the Macintosh PowerPC family. These operating systems are running on computing platforms powerful enough to display complex 3D-rendered scenes. The tools are readily available as well, thanks to Microsoft's DirectX media integration API's and ActiveX Internet controls as well as Netscape's Live3D and LiveConnect developer platforms. These software tools, combined with powerful desktop processors, make it easy for software developers to create VRML technologies and products.

Another key aspect of development is the tight integration of multimedia into these platforms. Hollywood and the video games industry see the desktop PC as the next major platform for delivery of multimedia content. This means VRML technology development will be accessible to developers of all types of integrated Internet-based media.

The final key is development of open-protocol communications standards suited for Internet use. Currently, the military uses distributed interactive simulation (DIS) as the communications protocol for training

Page 118

applications and has been successful to date. The integration of DIS with Internet technology is key but not the entire solution. DIS was developed only for military applications. Its broader acceptance by industry is dependent on significant changes to its infrastructure, including the simulation model, numerical representation, integration with VRML, and dependence on Department of Defense initiatives.

Another complementary area of interest is multiuser VRML spaces. These applications are the next step in on-line human-to-human communication and are enabled by the Internet and VRML. Several companies have products that let individuals directly interact with others. In these on-line worlds each person views a fully interactive 3D VRML world, including moving graphical avatars that are the virtual representations of their human counterparts. Some of these applications also include real-time voice that is syncopated with movements of the avatar's eyes and mouth. It is very compelling to communicate with someone and only be able to see their virtual representation.

Several companies and organizations are now starting to collaborate on a standard for VRML-based avatars. These groups are now in the formative stages and are being published by fairly small companies. The first avatar standard will roll out later in 1996.

Future Directions

VRML technology and content development in 1996-1997 will focus on several areas. On the standards front, the VRML Consortium and ISO will continue to broaden acceptance of VRML. The VRML Consortium will have its first official meetings in late 1996. Creating the organization and filling it with technical, creative, process-oriented people will be a goal. The VAG will continue to serve as the focus for standards-based VRML work until the consortium is self-sustaining. Also during 1997, ISO will officially adopt VRML as the only international 3D metafile format for the Internet. Once the VRML Consortium is operational, the focus of activities will be on continued development of the VRML specification and the creation of working groups.

On the software and hardware development fronts many advances will be made. VRML 2.0 browsers will emerge and will integrate directly into the popular HTML-based browsers. Manufacturers of three-dimensional hardware accelerators will add features that directly support basic VRML graphics. Tool manufacturers, such as polygonal modelers and scene creation tools, will incorporate VRML read-and-write capabilities. Integration of DIS and other distributed simulation communications protocols will quickly help content authors build multiuser capabilities into

Page 119

their worlds. Finally, content developers will enjoy the flood of new modeling and programming tools.

Given all of these advances there are still three immediate technical areas that need to be addressed before VRML becomes widely adopted: common scripting language, external API, and binary file format. Currently, these areas are quite controversial, but it is clear within the VRML community that solutions to the problems are within reach.

VRML Resources on the Internet

http://vag.vrml.org—Official home of the VRML spec and the VAG

http://sdsc.vrml.org—Very comprehensive list of VRML resources

http://www.intervista.co—Popular VRML browser

http://www.microsoft.com/ie/ie3/vrml.htm—Popular VRML browser

http://www.sgi.com/cosmo—Popular VRML browser

http://home.netscape.com/eng/live3d—Popular VRML browser

http://www.blacksun.com—Multiuser 3D application

http://www.onlive.com—Multiuser application with real-time voice

http://www.dimensionx.com—Java-based VRML tools

http://www.ktx.com—VRML tools

Page 120

Mark Bolas

Introduction

If the National Aeronautics and Space Administration's VIEW laboratory marks the beginning of the virtual reality (VR) industry, the industry is just about to pass its 10-year mark. There is a rule of thumb stating that it takes about 20 years for a new technology to find its way into the mainstream economy. Applied here, this means 10 years before VR is in the mainstream economy. This prediction seems completely reasonable, or even pessimistic. Consumers can currently purchase VR headsets with integrated tracking for less than $800. A handful of automotive manufacturers and aerospace contractors use VR on an ongoing basis to solve design and engineering problems. However, early adopters are incorporating the technology into their work and lives. They face all of the frustrations and challenges typically associated with being on the cutting edge. The next 10 years will see the VR industry evolve in a straightforward and boring fashion—early adopters will have paved the way for easy use by the mainstream.

This evolution will require a fundamental shift in the way VR technology is viewed and used. The technology must cease to stand apart; it needs to become an invisible part of a user's lifestyle and work habits. This requires progress on two basic fronts: First, the technology must be integrated into the user's physical environment. Second, it must be integrated into the user's software environment.

Evolution

For mainstream users to benefit from VR technologies, the technologies must become pervasive. They must extend throughout our industries and lives. They must diligently work for their users and quietly become part of their lifestyle. The facsimile machine is an example of a technology that has accomplished this.

Walkmen, dishwashers, televisions—All these have become pervasive by thoroughly changing the way people do things. A person does not talk about using a walkman, or a dishwasher, or a television. If anything, a person discusses the content or end result as opposed to the

NOTE: The industry segment described here is defined as industries that benefit from immersive human-computer interfaces. The term virtual reality is intended to include this definition.

Page 121

actual device. "I heard a good song," "The dishes are clean," "Did you see that stupid show last night?"

There is little question that three-dimensional (3D) graphics and simulation are on the way to becoming pervasive. In industry the design process is being transformed to demand 3D models and simulations. This Christmas consumers will be choosing between the Sony or Nintendo platforms with 3D graphics capability being assumed.

However, the VR industry must evolve to provide such 3D systems with immersive interfaces that multiply the utility and effect of the 3D graphics. Currently, most 3D graphics are shown on a 2D screen and manipulated via a 2D mouse. These interfaces effectively remove much of the value present in the 3D environments. The VR industry must maintain the utility and comfort present in a user's natural ability to perceive and manipulate 3D environments and objects.

Advances

For VR to become a pervasive tool, it must become integrated into both the user's physical and software environments. Seamless integration with a user's physical environment is not simple because immersive interfaces tend to immerse—that is, they surround and envelop the user. This can easily intrude on a user's physical and mental environment. The VR industry needs to minimize this intrusion to the point where immersive interfaces are as natural to use as a telephone or mouse. It is interesting to note that both these examples are not inherently natural, but both have been integrated into users' workspaces and lifestyles.

To achieve a natural interface, paradigms that transcend the standard goggles-and-gloves paradigm need to be pursued. The fact that people collaborate, multitask, and eat while they work are down-to-earth aspects that must be considered in the design of immersive tools.

Equally challenging is the integration of these new interfaces in the software environment. Application software packages have typically been written for 2D screens and interfaces. As a result, most immersive interfaces are poor retrofits onto existing packages that were never designed to incorporate them. This lack of integration severely cripples the utility of immersive interfaces.

This integration is probably best achieved by starting with a "top down/bottom up" design approach on a number of key applications. For example, the entertainment industry could use an immersive set design and preview system, while the Defense Department would benefit from a simulation-based design and modeling system that fully utilizes a human's ability to think, design, and manipulate 3D space.

Page 122

Peter Bonanni

The U.S. armed forces have created the most advanced training systems in the world. Some segments of the armed forces, however, are facing true training shortfalls for the first time in decades. These training deficiencies are being caused by worldwide deployments. U.S. Air Force active duty and reserve squadrons, for example, have experienced a reduction in training sorties of up to 25 percent. This reduction is a direct result of deployments in support of contingency operations over Iraq and Bosnia. Pilots are most proficient and able to fight when they are first deployed to these areas. As the deployment wears on, with little or no training opportunities, pilot proficiency slips. The same problem is occurring in other combat arms as the trend to use U.S. forces in peacekeeping roles accelerates. Since conducting realistic training is impossible on most of these missions, simulators provide the only realistic training alternative. Unfortunately, most of the simulators in use today are very expensive, are limited to single-crew training, and are not deployable.

Emerging commercial simulation technology, however, may provide a near-term solution to this military training problem. Some fighter pilot skills, for example, cannot be practiced in simulation, regardless of the fidelity. The most important (and perishable) skills, however, can be honed by very-low-cost simulators. The computer game Falcon 4.0 is an example of a commercial product that is shattering the fidelity threshold and providing a model for very-low-cost simulation. There are several key components to Falcon 4.0 that allow this type of breakthrough. Falcon 4.0 features "SIMNET-like" networking protocols that create a large man-in-the-loop environment. These features of Falcon 4.0 provide the basic building blocks for producing a simulator that will be low in cost and deployable and that will provide pilots with team training opportunities. In the near term this capability will be enhanced with the development of commercial head-mounted displays and voice recognition systems.

Page 123

Defense Modeling and Simulation Office:
DOD Modeling and Simulation Overview and Opportunities for Collaboration Between the Defense and Entertainment Industries

The U.S. Department of Defense (DOD) is building a robust modeling and simulation (M&S) capability to evaluate weapons system requirements and courses of actions; to reduce the time line, risk, and cost of complex weapons system development; to conduct training; and for realistic mission rehearsal. Part One of this paper provides a description of the current and envisioned application of M&S in the training, analysis, and acquisition support functional areas. It also summarizes the plan that is in place to help achieve DOD's M&S vision. Part Two is a list of technology areas that DOD believes have a potential for collaborative development with the entertainment industry.

Part One:
DOD Modeling and Simulation Overview

The foundation for the above set of DOD M&S capabilities will be the development of a common technical framework to maximize interoperability among simulations and the reuse of simulation components. The cornerstone of the common technical framework (CTF), the High-level Architecture (HLA), has just been adopted as DOD-wide policy. Together with the other elements of the CTF, data standards, and a common understanding (or conceptual model) of the real world, the HLA will enable DOD to use and combine simulations in as-yet unimagined ways. Establishment of a commercial standard will follow as applications spread to training for natural disaster response, weather and crop forecasting, and a host of other business and social problems.

Common services and tools also will be provided to simulation developers to further reduce the cost and time required to build high-fidelity representations of real-world systems and processes. Realistic simulations, interacting with actual war-fighting systems, will enable combatants to rehearse missions and "train as we fight." Virtual prototypes developed in a collaborative design environment using the new integrated product and process development concept will be evaluated and perfected with the help of real war fighters before physical realizations are ever constructed. DOD

Page 124

will enforce recently approved policies and procedures for the verification, validation, and accreditation of models and simulations to ensure accuracy, thereby enhancing the credibility of simulation results.

The advanced M&S capability envisioned by DOD will be a rapidly configured mix of computer simulations, actual war-fighting systems, and weapons systems simulators geographically distributed and networked and involving tens of thousands of entities to support training, analysis, and acquisition. Not only is there a desire to quickly scale the size and mix of simulations, but DOD also is pursuing the capability whereby both groups and individuals can interact equally well with a synthetic environment. The major challenge in providing scalability, as well as the group and individual experience, is achieving consistency and coherence of both time and space.

Other areas of ongoing research in DOD that show promising results are the accurate representation of human behavior, systems, and the natural environment (air, space, land, sea, weather, and battle effects). DOD's efforts are focused on just-in-time generation of integrated and consistent environmental data to support realistic mission rehearsals anywhere in the world, including inaccessible or operationally dangerous locations. Investments in the rapid extraction of land and water surfaces, features existing on those surfaces, and features derived from ocean, air, and space grided fields have begun to yield results. The goal is to develop a capability to generate feature-integrated surfaces that require minimal editing and model-based software for feature extraction. Achieving this will, for example, ensure that weather fronts that bring rain or snow change the characteristics of the ground so that transit rate is affected and the associated wind patterns move trees, create waves, and alter dispersal patterns of smoke and dust. The resulting realism will add significantly to training, analysis, and acquisition. These effects, when coupled with dial-up capability to create custom correlated conditions, can provide year-round training.

Warriors of every rank will use M&S to challenge their skills at the tactical, operational, or strategic level through the use of realistic synthetic environments for a full range of missions, to include peacekeeping and providing humanitarian aide. Huge exercises, combining forces from all services in carefully planned combined operations, will engage in realistic training without risking injury, environmental damage, or costly equipment damage. Simulation will enable leaders to train at scales not possible in any arena short of full-scale combat operations, using weap-

Page 125

ons that would be unsafe on conventional live ranges. Simulation will be used to evaluate the readiness of our armed forces as well.

The active duty and reserve components of all forces will be able to operate together in synthetic environments without costly and time-consuming travel to live training grounds. In computer-based training, both the friendly and opposition forces, or computer-generated forces (CGFs), are highly aggregated into large command echelons and carry out the orders resulting from staff planning and decision making. CGFs fall into two categories: (1) semiautomated forces (SAFs), which require some direct human involvement to make tactical decisions and control the activities of the aggregated force, and (2) automated forces, which are associated with autonomous agent (AA) technology. AAs are in early development phases and will find extensive applications in M&S as the technology matures.

There is now a diverse and active interest throughout the DOD M&S community, academia, and the software industry in the development of CGFs and AAs. The Defense Advanced Research Projects Agency is sponsoring the development of Modular Semi-Automated Forces for the Synthetic Theater of War program, which includes both intelligent forces and command forces. This effort also involves development of the command and control simulation interface language. It is designed for communications between and among simulated command entities, small units, and virtual platforms. The services, more specifically the Army's Close Combat Tactical Trainer program, is now developing opposing forces and blue forces to be completed in 1997. The British Ministry of Defence also is developing similar capabilities using command agent technology in a program called Command Agent Support for Unit Movement Facility. Academic and industrial interest in this technology has led to the First International Conference on Autonomous Agents, which will take place in Marina del Rey, California, on February 5-8, 1997.

M&S will provide DOD with a powerful set of tools to systematically analyze alternative force structures. Analysts and planners will design virtual joint forces, fight an imaginary foe, reconfigure the mix of forces, and fight the battle numerous times in order to learn how best to shape future task forces. Not only will simulation shape future force structure, but it will also be used to evaluate and optimize the course of action in response to events that occur worldwide.

M&S representations will enable planners to design the most effective logistics pipelines to supply the warriors of the future, whether they are facing conventional combat missions or operations other than war.

Page 126

Operating in the same virtual environments, virtual prototypes will enable acquisition executives to determine the right mix of system capability and affordability prior to entering production. Fighting synthetic battles repeatedly while inserting new systems or different components will help determine the right investment and modernization strategy for our future armed forces. Models and simulations will reduce the time, resources, and risks of the acquisition process and will increase the quality of the systems produced.

M&S will allow testers to create realistic test and evaluation procedures without the expense and danger of live exercises. "Dry runs" of live operational tests will minimize the risks to people, machines, and testing ranges.

M&S will enhance information sharing among designers, manufacturers, logisticians, testers, and end users, shortening the system development cycle and improving the Integrated Product Team development processes.

The DOD M&S Master Plan (MSMP) is a corporate plan to achieve DOD's vision. Its first objective is the establishment of a common technical framework, anchored by the HLA. The HLA has been defined and adopted as the standard simulation architecture for all DOD simulations. Development continues on the other elements of the CTF, and DOD's investment strategy for M&S is focused on achieving the vision.

The second objective of the MSMP is to provide timely and authoritative representations of the natural environment. To this end, Executive Agents (EAs) have been established to coordinate development in their respective areas of oceans, aerospace, and terrain. EAs have also begun to explore potential commercial marketplaces for their databases.

The remaining objectives address representation of systems, human behavior, and establishing a robust infrastructure to meet the needs of simulation developers and end users. The infrastructure will include resource repositories—virtual libraries—and a help desk for users.

The final objective of the plan is to share the benefits of M&S. DOD must educate potential users about the benefits of employing M&S. To that end, an extensive study is under way to quantify objective data on the cost-effectiveness and efficiency of M&S in training, analysis, and acquisition applications throughout DOD. Extensive anecdotal data exist, but no concerted effort to demonstrate the return on investment has been done.

Page 127

Part Two:
M&S Technology Areas

Although the vision for M&S described previously is focused on meeting the needs of the military, implementing the vision requires the development and exploitation of technologies that also have application to the entertainment industry. The following partial list of technologies was identified by members of the DOD M&S community as areas where cooperative development with the entertainment industry will have the greatest benefit to both communities.

Virtual presence is the subjective sense of being physically present in one environment when actually present in another environment (Sheridan, 1992). Researchers in virtual reality (VR) have hypothesized the importance of inducing a feeling of presence in individuals experiencing virtual environments if they are to perform their intended tasks effectively. Creating this sense of presence is not well understood at this time, but among its potential benefits may be (1) providing the specific cues required for task performance, (2) motivation to perform to the best of one's abilities, and (3) providing an overall experience similar enough to the real world that it effectively allows suspension of disbelief while at the same time the synthetic environment elicits the conditioned or desired response while in the real world.

This is the primary means to foster presence in most of today's simulators. However, because of an insufficient consideration of the impact of granularity, texture, and style in graphics rendering, the inherent capability of the available hardware is not utilized to the greatest effect. One potential area of collaboration could be to investigate the concepts of visual stimulus requirements and the various design approaches to improve graphics-rendering devices to satisfy these requirements.

DOD has initiated numerous efforts to improve the production of 3D sound techniques, but it has not yet been effectively used in military simulations. Providing more realistic sound to a synthetic environment can have two potential benefits for training: (1) providing more realistic

Page 128

sound cues and (2) providing a more realistic aural environment that enhances realism.

Smell can contribute to task performance in certain situations and can contribute to the full sense of presence in the synthetic environment. There are certain distinctive smells that serve as cues for task initiation. A smoldering electrical fire can be used to trigger certain concerns by individuals participating in a training simulator. In addition, smells such as hydraulic fluid can enhance the synthetic environment to the extent of creating a sense of enhanced danger.

Another sense that can be involved to create an enhanced synthetic environment is touch and feel. Current simulator design has concentrated on moving the entire training platform while often ignoring the importance of surface temperature and vibration in creating a realistic environment.

One area that has not received much research is the required coherent application of the above-listed stimulations to create an enhanced synthetic environment. Although each stimulation may be valid in isolation, the real challenge is the correct level and intensity of combined stimulations.

This area includes technologies that emphasize the representation of individuals and the interactions among virtual and live participants in an individual or group experience.

While methods are evolving for creating computer-generated representations of human figures that are anthropometrically valid, in general these methods are computationally complex while at the same time stylishly rigid. Approaches for automated modeling of human figures that result in more natural representations that are more computationally efficient is a topic of great interest in a number of disciplines, including medicine. The need is to determine the minimum essential information required to pro-

Page 129

vide a representation of human actions that are sufficiently realistic for both communities. Animation of human figures, including speech, running, and facial expressions, still requires significant development.

Research has begun on methods for tracking and capturing the motion of humans that support real-time interaction with both virtual and constructive simulations.

Creation of a full virtual environment requires generating the natural and/or cultural features of the background in which the interaction takes place. Specific areas of research include automating the production of background environments and efficient representations in scalable databases.

Most of the research in virtual presence has been single person oriented (e.g., head-mounted displays and tracking systems, hand and foot controls). DOD has a direct interest and experience in developing the group or team training experience, which is also of interest to the entertainment industry. DOD would like to enhance its capability for an entire group to interact with a virtual environment and each other without the need for unique individual hardware devices.

Summary

The DOD vision is to apply M&S to the full range of military applications, including training, analysis, and acquisition. The vision can only be met if the technology defined above is readily available, of low cost, and operationally valid. It is the desire of DOD to explore technologies with the entertainment industry that are relevant to modeling and simulation. These technologies may include animation, graphical imaging, data communication and storage, architectures, and human immersion. DOD believes research in collaboration with the entertainment industry will provide mutual benefit to both communities.

Reference

Sheridan, T.B. 1992. Telerobotics, Automation, and Human Supervisory Control. MIT Press, Cambridge, Mass.

Page 130

James Foran

Nintendo 64, the first truly interactive three-dimensional video game machine, provides a level of experiences that has not generally been available outside the traditional simulation and training community. It does so at a price point that allows virtually every household to own one. The implications of the technology embedded in the machine for all types of training and simulation are tremendous. Not only does it provide a low-cost ubiquitous platform, but it also portends a future where even more powerful and realistic machines will be pervasive.

Silicon Graphics, relying on 15 years of experience, was able to utilize state-of-the-art semiconductor technology to achieve a low-cost, high-performance, high-volume product for Nintendo. The chips utilized were among the first logic chips to be produced using 0.35-micron technology. This represents a fundamental change in the way technology is driven. In the past, advanced technologies were first used to produce low-volume, high-cost systems principally for military use. These seed applications provided the opportunity to make the technology viable economically. Over time the technology moved down in product price point until eventually it appeared in consumer devices.

All of this has now changed. Today, with fabrication facilities costing over $1 billion, large-scale markets must exist to justify the expense of construction. Although DRAM [dynamic random access memory] has long been the primary justification for new fabrications, the cyclical nature of demand requires that other applications need to exist to balance capacity utilization. Video games are the largest market for consumption of advanced semiconductor technologies; their public acceptance is orders of magnitude higher than that of traditional computer products. In its first six months, 2.7 million units are expected to be sold, increasing to a total of 5 million within the first nine months.

In the video game market it is possible to get an advanced product like this out at a price point that is acceptable to the consumer only because it is possible to subsidize hardware with software. The hardware is brought to market with a very low margin throughout the chain from manufacture to distribution. Much like a CD player, the box has no intrinsic value to the consumer; it is simply a necessary expense in order to enjoy the game. Over the product life it is typical for each platform to average 10 games. This provides the return on investment to the manufacturer as well as a living for the content providers.

This is a great development for kids who want to play games, but what implications does it have for other markets? It is instructive to look at some of the similarities to the requirements that are traditionally asso-

Page 131

ciated with the military market. Typical military programs have stressed advanced technology. After all, competitiveness is the cornerstone of any military development. The video game business is a war for the consumer pocketbook. Because of the requirement for competitive advantage, both applications are up-front and capital intensive. Long-term product stability also is important for both markets. In this respect, video games are unusual for a consumer product. Each hardware unit in the field must play every game cartridge the same as every other machine. Maintenance of the design for a 10-year period is accepted.

So we can see that there are many characteristics of video game hardware that match up with typical military requirements. How could this type of hardware be put to use? In the field of training and simulation the military has long led the way. With increasing sophistication of weaponry and the political sensitivities associated with the type of actions encountered in today's world, military preparedness is more necessary than ever. Simulation also provides the cost-effectiveness required by today's budget realities. Nevertheless, practical training equipment, although decreasing in price, has not yet become ubiquitous. This type of video game platform now makes that possible.

The question before this group is, How can the military take advantage of this commercially developed technology? One immediate answer is that training cartridges can be developed for the actual home game platform. This requires the setting up of some sort of development rights with the game platform manufacturer. This is actually a very practical method for training applications where the home game hardware is sufficient to achieve the training objective. In the case where input devices must be similar to actual operational hardware or where systems must be embedded into operational equipment, one must go beyond the box available at the toy store.

Some of these requirements can be met by physically reconfiguring the hardware and developing the appropriate accessories. In other cases, where requirements may exceed the capabilities of the home game box, more powerful systems can be built utilizing the same components.

Generally, a semiconductor process yields a speed range of parts that can vary in horsepower by 50 percent or more. In the case of a product like Nintendo64, because of the requirement for high volume and low cost, all devices must work in the target system. This means that through speed grading much more powerful components can be obtained. By using these components and more robust system configurations it is possible to satisfy more demanding requirements. Since the semiconductor process used to manufacture these state-of-the-art devices is itself quite new, it is a natural that as the process matures a shift of yield to higher-speed parts will result. This is a quite common phenomenon in the DRAM business.

Page 132

So what is the issue that prevents this type of technology from being utilized by the military? The military can accomplish tremendous projects during times of war or national emergency, but during peace time the design and procurement cycle moves at a snail's pace. I recently talked to a customer—a military system integrator—about designing a graphic function for use in a new vehicle. He was concerned that he might prototype with something that would not be cost-effective in implementation. I asked him: "Well, how far out is production?" The answer was that production would start in six to seven years. I told him there wasn't anything on Earth with regard to electronics that would not be cost-effective in six years if it exists today.

How can the military deal with this situation?

1. It can think long term. We have to have a vision of what kind of technology we will want to be using 5 to 10 years from now. We have to be practical. There are far too many "futurists" on the speaking circuit whose ideas are either too far out in time or lack any understanding of the infrastructure required to provide a whole solution. Nevertheless, a long-term vision is a necessity.

2. It can focus. Focus on specific objectives of large scale. This provides volume, which is necessary to entice companies to commit resources. However, don't get trapped into trying to define a universal device that meets all needs for everyone. The desired product must be simple to describe and easily understood by everyone involved.

3. It can make commitments. No company with shareholders is going to make a major investment of its resources for something that might happen. The military must realize that despite everyone's good intentions some of these commitments will result in failure. Monitoring of progress and conditions is required, but this needs to be at arm's length.

4. It can cut the red tape. Companies need to be able to do what is business as usual to them. Requirements need to be clearly communicated and feasibility agreed. After that, get out of the way and focus on results rather than procedures.

5. It can put the infrastructure in place. No solution will succeed unless the proper infrastructure is put in place on the customer side. There needs to be honesty in the assessment of how fast infrastructure changes can be accomplished, and these must mesh with the time scale of the project.

In closing, there is no limit to where technology can go. The limits lie in our ability to apply technology. The reason that entertainment markets are able to apply technology successfully is that major commitments are made for specific focused objectives of large magnitude. This provides lucrative opportunities for technology companies to provide new solutions.

Page 133

John Geddes—
Collaborative Challenges: Understanding Cultural Differences

There is a broad cultural chasm between the U.S. Department of Defense (DOD) and the entertainment/computer industry. This chasm can present a serious obstacle to successful collaboration. Processes and attitudes will have to be created or modified if collaboration is going to succeed between the two groups.

In the context of modeling and simulation, DOD can be characterized by varied and often competing interests, funding that is renewed annually, and extremely hierarchical and time-consuming approval and review processes.

• Varied and competing interests. Three domains of simulation competing for funding (ACR, RDA, TEMO); uncertainty and competition between DARPA, RDECs, STRICOM, and major commands like SSDC for primacy in development and program management of new simulation activities.

• Funding uncertainty. Annual budget processes, effects of changing military and civilian leadership on priorities; for example, Army Modeling and Simulation organizations (MISMA, AMSO, DUSA/OR, M&S GOWG) and National Rotorcraft Technology Center funding profile.

• Need for coordination across commands and agencies to get approval and requirements for periodic reviews at multiple levels; examples in ACTD processes, Soldier System development. Long duration of projects—one year to get consensus, two years until funding; examples in Louisiana Maneuvers, Battlefield Visualization.

In comparison, the commercial entertainment/computer industry can be characterized by short project horizons, more stable funding, relatively flat hierarchies for approval, and more informal and spontaneous review processes.

• Product horizons are one to three years from concept to product on the shelf; an example is the Nintendo 64.

• Once a company approves a project for development and production, funding is programmed and maintained generally for the duration of the effort and is not subject to the whims of elected representatives.

• Flatter hierarchies and more informal reviews, resulting from total quality management or reengineering and closer scrutiny of value-added functions; less internal regulation.

Page 134

Recommendations for successful collaboration:

1. Create an advisory board with power to sponsor and recommend collaborative and cooperative efforts. Publish annual report with positive results and with opportunities that were neglected. Include lessons learned about positive and negative collaborative results.

2. Exchange liaisons. Create positions that are geographically proximate for providing effective coordination and for seeking opportunities-a few that work for advisory board, more that work for specific participants, both DOD and non-DOD.

3. Allow decisions at the lowest levels. Minimize hierarchical reviews. Nonproductive time for most participants. Use advisory board liaisons.

4. Understand and use existing cooperative mechanisms—cooperative research and development agreements, cooperative agreements, and other transactions. Involve a congressional staff in advisory panel to help shape future mechanisms.

Page 135

Martin Gundersen

Advanced modeling and simulations for games, entertainment, manufacturing, education, the U.S. Department of Defense, finance, and other applications will grow with the development of integrated media systems incorporating software and hardware development at many levels. Integrated media systems will powerfully impact all fields of inquiry and technology. Integrated media systems of the future will seamlessly combine digital video, digital audio, computer animation, text, and graphics into common displays in such a way as to allow for mixed media creation, dissemination, and interactive access in real time. Prodigious national and international resources are currently being marshaled for integrated media technologies' research, development, infrastructure installation, product creation and commercialization, public performance, and training. According to a recent projection, multimedia and creative technologies will represent a new total market of $40 billion by the year 2000 and $65 billion by the year 2010.

The beckoning opportunity is to accelerate progress in this new discipline by revolutionizing our access to information sources, easing the effort required to author original works, and transforming our capacity to augment and enhance the productivity of human creative endeavors. The corresponding challenge is to first recognize the impact of these dramatic changes on the very nature of our teaching tools, manufacturing methods, defense systems, health care systems, and entertainment/art forms and to then exert sufficient positive leadership to assure maximum benefit. At the University of Southern California we are pursuing a large-scale program that is relevant to the goals of utilizing entertainment-oriented technology. We have established a Center for Integrated Media Systems, which is directed by Max Nikias, for research, development, and teaching in advanced systems for multimedia applications, including entertainment. This research has recently received one of four Engineering Research Center awards this year from the National Science Foundation, the proposal ranking first out of 117 proposals.

There are three major areas of importance with opportunities for research and development: interfaces, communications, and databases. These are discussed below. The next generation of integrated media systems in the augmented reality, interactive multimedia, heterogeneous computing, distributed database, wireless communication, and high-speed network environments will impact every facet of our lives. Access to a wealth of diverse and distributed information resources will be possible from within an individualized "information framework." Interactive media will enable new paradigms for education, training, manufac-

Page 136

turing, and entertainment that provide worlds—real, augmented, and fantasy—for people to experience, learn through, and interact with. Design-based industries will develop products through virtual design systems that integrate software applications and manage both the design process and the design data, as well as incorporate input from intended consumers, designers, production engineers, quality assurance and quality control specialists, cost analysts, and manufacturing engineers.

Creator-Computer-Consumer Interfaces

Computer interfaces are unidirectional and inefficient. A significant bottleneck has emerged at the creator-computer and computer-consumer interfaces owing to an increasing mismatch between computational and display power, on the one hand, and human-computer input/output (I/ O) on the other. Simply put, highly visually and aurally oriented human beings are constrained to interact with an assistant that cannot see, hear, or speak. The human-computer interface has evolved over four decades from plug boards, lights, punch cards, and text printers to postscript laser printers, mouse-based window systems, and primitive head-mounted displays. The trend has clearly been toward interaction modes that are more intuitive, enabling people to communicate more effectively to and through computer systems. Today, enhanced video and audio capabilities fuel the explosive success of both multimedia-equipped studio-grade workstations (the creator-computer interface) and personal computers (the computer-consumer interface), as particularly evidenced by the trend toward truly interactive media applications.

Technological advances in the area of human-computer interfaces are necessary to achieve a new level of even richer and more perceptive interfaces that are characterized by the immersion of the user/participant in highly communicative multisensory interactions. These advances must span both visual and aural interface technologies. Input to the computer can be enhanced by means of smart cameras for environmental awareness and expression recognition and with robust speech recognition for extended natural language interactions. Immersivision methods for panoramic scene reprojection and novel approaches to three-dimensional (3D) displays enrich the presentation of graphic output. The computer's sense of the environment is enhanced through smart-camera-based tracking technology, which in turn is pivotal for both augmented reality applications and the synthesis of an accurate 3D aural environment through immersive sound reproduction. Furthermore, the coupling of these technologies with advances in wireless networks and distributed databases will allow the integration of mobile workstations (personal data assistants) with tracked

Page 137

head-up displays for application in augmented office, classroom, factory, and cockpit environments.

Media Interconnection and Delivery Fabrics

Real-time distribution and storage of multimedia information is expensive. Even with compression, which can only be employed in certain applications, digital video and audio can consume large portions of database storage and network bandwidth. Access to even currently available network bandwidth is limited by workstation I/O design bottlenecks. A need therefore exists for both high-bandwidth interconnections and interfaces and real-time artifact-free compression and decompression algorithms.

Over the past decade, user demands on networks and databases have escalated from the bandwidth and storage requirements characteristic of text to those characteristic of both images and real-time production-quality video and audio. As integrated media systems evolve to incorporate the advanced interfaces described above, they will impose even greater demands on high-speed wired and wireless communications networks. These enhanced visual and aural interfaces, as well as real-time digital video servers, integrated media databases, and distributed processing systems will require the effective and efficient image and data compression methods, multi-gigabit-per-second (Gbps) fiber-optic networks, and high-bandwidth wireless networks developed in this thrust. Two cases illustrate how the need for such delivery fabrics arises depending on the number of connected users. In today's manufacturing environments with hundreds of untethered workers, or in video-on-demand networks with thousands of consumers, each person requires on the order of 20 Mbps of bandwidth over wireless or wired networks to receive compressed video and graphics. On the other hand, in today's video production environment with dozens of users, each requires about 270 Mbps for D1 digital video. A shared network is an efficient means for distributing data in both of these cases. One challenge for such a system with multi-Gbps (2 to 50 Gbps) aggregate throughput is to seamlessly support multiple data types such as D1, MPEG, text, and graphics. In addition, the interconnection and delivery fabric must be capable of satisfying future standards, such as video quality that is significantly superior to that of D1 or high-definition television. The research challenge in this area is focused on the development of technologies for shared integrated media networks.

Distributed Multimedia Information Management

An effective methodology for managing large integrated media databases does not exist. Integrated media databases of the future will con-

Page 138

tain terabytes of information. Information relevant to a given need will likely reside in a collection of interconnected heterogeneous and distributed knowledge bases. Techniques for locally organizing, browsing, discovering, and querying such integrated media repositories are needed. Furthermore, many applications demand seamless synchronous access to multiple audio and video threads from distributed digital databases, a capability that does not currently exist.

Advanced human-computer interfaces and enhanced wired and wireless media interconnection and delivery networks cannot function effectively without access to dramatically scaled-up databases that can seamlessly manage multiple media types. Hence, the central integrated media-systems-related issue that must be addressed during the next decade is the storage, indexing, structuring, manipulating, and "discovery" of integrated multimedia information units (MIUs) that include structured data values (strings and numbers), text, images, audio, and video. The key research focus in this area centers on managing multimedia information units in the context of a highly distributed and interconnected network of information collections and repositories. Current data and knowledge management technology that addresses collections of formatted data and text is inadequate to meet the needs of video and audio information, as well as the mixture of modalities in MIUs. Furthermore, the highly distributed and interconnected nature of the emerging information superhighway accentuates the need for techniques that enable multimedia information sharing. The research challenge in this area involves the development of mechanisms that address four critical aspects of distributed multimedia information management: (1) multimedia information content representation and extraction; (2) multimedia database networking: discovery, filtering, query, sharing; (3) storage of and access to continuous media data types; and (4) visual presentation of information across cultures.

We are developing collaborations with other efforts, including related research activity at Howard University and the University of South Carolina. The South Carolina program has initiated development of a "virtual testbed," which is a top-down, mission-oriented approach emphasizing simulation of complete electrical systems on U.S. Navy ships using advanced visualization techniques. This program is under the direction of Professor Roger Dougal.

At the University of Southern California we have developed an industrial partnership with over 50 companies that are literally a cross-section of industry working to develop and apply the new technology. In entertainment we have formed a panel of entertainment professionals who will foster collaboration with the Hollywood industry that will be strongly impacted by multimedia simulations and

Page 139

modeling. The professionals are a cross-section of the industry, including actors, directors, film editors, audio engineers, computer network experts, writers, and others, including investors. Over the next few years we will be working to provide an academic venue for this technology to be researched, viewed, and understood, with emphasis on entertainment applications. The panel on entertainment applications will be meeting with industrial partners of the center at USC in a review that will occur in November of 1996.

Page 140

Will Harvey—
The Future of Internet Games

Latency is a major barrier to fast-action Internet games. Game developers can either hope the problem goes away or adopt new game architectures that work around it. There is compelling evidence that the problem will never go away and that the hardware will never improve to the point that developers can afford to treat the Internet like a local area network (LAN). Sandcastle offers an alternative, a software solution that enables fast-action Internet games.

High latency is incompatible with the client/server and lockstep designs that current LAN games use. A response time of 33 milliseconds (ms) has been the industry standard for over 20 years, and even with premium on-line services, Internet performance is nowhere near that level. In fact, it cannot reach that level. In fiber, light takes 54 ms to travel roundtrip between New York and San Francisco. Networking experts agree that the Internet's latency will plateau between 100 and 130 ms cross country (Figure D.1).

Fast-action client/server and lockstep games are no fun at this speed. A player trying to dodge a bullet will feel either frustrated, because the response time is too slow for him to dodge, or cheated, because the program displays his character such that it appears he has dodged when he has not. Punches a player could land will miss; opponents a player could tackle will evade. Without responsiveness, fast-action games are not fun.

The Shift to Distributed Processing

The solution is to move to a distributed architecture. In a distributed game, each player controls a character on his local machine, so it responds to his actions instantly, with no latency. The new challenge is then to synchronize the game state on all the machines and to coordinate interactions among objects that different players control.

In Figure D.2 the big circle is a server or multicast router in a building. The small circles are machines in people's homes. X, Y, and Z represent objects controlled by users from their own homes. Proxies not shown in these figures display the objects on every machine. The X, Y, and Z letters represent the point of control of each object.

In the lockstep architecture, each machine broadcasts its user input to the other machines and advances one simulation cycle when it has received a complete set of user input from all participating machines. Since advancing a cycle requires complete exchange of user input, the responsiveness is limited by the speed of the worst communications la-

Page 141

FIGURE D.1

FIGURE D.2

tency of any machine. In the client/server architecture, each machine independently sends its user input or action request to the server in order to perform an action in the simulation. Controlling an object from a client machine still entails a roundtrip delay, but the responsiveness of any individual client machine is not affected by the communications speed of the other machines. In a distributed architecture, machines control objects locally and broadcast the results of actions to other machines, which receive the information with some time delay. Each machine has immediate responsiveness controlling its own objects but must synchronize interactions between its own objects and objects controlled by remote machines.

In both the lockstep and client/server architectures, responsiveness is limited by the roundtrip communication latency to the server, which will always be too long for fast-action games. Controlling objects locally and synchronizing interactions between them is the only solution. The shift from a central architecture to a distributed architecture transforms the latency problem into a synchronization problem.

Page 142

Solving the Synchronization Problem

Methods of solving the latency and synchronization problems fall into three categories, represented in Table D.1.

At the lowest level, the first approach attempts to improve the speed of the network to reduce the latency problem by brute force, instead of adopting a distributed architecture. This approach will always have slow reactions because of the speed of light and network overhead, so it will be limited to domains like Quake, where players don't have a true opportunity to dodge bullets.

The second approach follows a software technology called distributed interactive simulation developed for military simulations. This approach accommodates the delay in which information is received from other participants by "dead-reckoning" or predicting the actions of the other participants to bring all objects displayed on a machine into the same time frame. Because predicting only works for predictable and continuously moving objects, such as planes and tanks, it does not apply to domains of rich human interaction like playing Nintendo's Mario 64 or playing catch with a ball over the Internet.

The third approach, synchronization, leverages off of the other two technologies, but more importantly it picks up where the other technologies reach their fundamental limitations. Information from remote machines will always be received with a time delay, and many actions cannot be predicted. Thus, remote objects must be shown in time delay. If a user has no interactions with remote objects, he cannot tell that he is seeing those objects "in the past"; but if he does interact with them, those inter-

Page 143

actions must accommodate the time difference. Synchronization technologies are a set of software networking components that enable interactions between objects in different time frames.

Sandcastle's Direction

Sandcastle is developing synchronization technologies that give users the impression that the network has zero latency, or immediate responsiveness. Specifically, the technologies address the problems of interacting with shared objects, like throwing a football between users, and interacting directly with objects controlled by remote machines, as in a fighting game or a race.

Our view is that the latency problems of central processing are fundamental. Over time, the demands for high responsiveness will drive an inevitable shift in programming paradigms from central processing to distributed processing. As this shift occurs, the technologies and tools that address the critical problems of real-time distributed applications will become increasingly important. We believe that these contributions are the beginnings of a foundation not just for games and chat environments but for all of twenty-first century interactive entertainment.

Page 144

Robert Jacobs

As the Computer Science and Telecommunications Board (CSTB) of the National Research Council assesses research priorities for defense and entertainment simulation, it must be mindful of the significant differences in objectives, risk and reward environment, and business traditions and customs, especially with respect to proprietary intellectual property, that characterize these two simulation industries.

Defense simulation programs focus on the solution of problems, the production of operational skills through training, the support of combat development test and evaluation, or the resolution of complex engineering optimization questions as a part of design and development. How well a defense simulation achieves its mission is usually determined by how its designers tailor the technology to address the problem of interest. Entertainment simulations, on the other hand, are a medium for the delivery of recreational experiences; the measure of success is not a matter of problem solution or production of information or skill but rather is determined largely by how exciting and enjoyable the experience is for the paying customer. The "fun quotient" of an entertainment simulation is predominantly a matter of art rather than technology; the technical side of the system must be capable of presenting the "story," but the perceived value of the experience hinges largely on the quality of the creative element.

Defense simulations are developed to a specification that defines the nature of the virtual world and the expectations of the customer for behaviors to be executed within it. As long as the product meets the specification, the development is deemed a success. Developers are compensated on the basis of their development cost plus a modest margin whose magnitude is negotiated in accordance with guidelines reflecting whether the customer or the developer takes on the development risk. There is no end-user specification to be met for an entertainment simulation. The developer must identify a market need, formulate a creative concept that addresses that need, and then back his intuition by investing his own money to field the concept. Maybe the marketplace will accept the concept; maybe it will reject it. For the most part, the market has been disappointed to date. If the simulation sells, it is priced in accordance with what the market will bear in view of competition, useful economic life, perceived value, return on investment, and so forth. What the market will bear may or may not be enough to recover development costs and to realize an attractive margin.

When the government contracts for research and development, both the client and the contractor generally acknowledge that the product of the effort belongs at least in part to the client. In the best case (from the

Page 145

contractor's point of view), the developer may share in the right to future exploitation of what is produced; however, the government belongs to all of us, and the government's equity in the ideas and technology is part of the public domain. In the entertainment world, proprietary intellectual property is the principal stock in trade, and ownership of the right to future exploitation is the primary asset resulting from the investment in a project. The customer buys the right to exhibit the product but never the right to the underlying proprietary intellectual property. Technological and creative innovations are important contributors to the asset value of the enterprise. To the extent that they can be protected, they will not be willingly given away.

Where is It All Going?

These contrasts, particularly the last point, create an interesting challenge for the CSTB in its quest to encourage open collaboration between defense and entertainment simulation developers. There is such a difference in the operational norms between these two industry segments that the resulting cultural barrier has been successfully breeched in only very few instances. One might expect that there also is a divergence of views as to how the industry and its technology will evolve in the coming decade.

On the defense side, the next few years will see continuing efforts to develop and disseminate technologies for more effective application of simulations to military and civil problems. These will include:

• Increased emphasis on large-scale simulations of military activity at the joint and coalition levels.

• Increased dependence on simulation technology to offset cuts in OpTempo, to conduct distributed planning and rehearsal, and to provide visualization for distributed command and control.

• Increased ubiquity of simulation, so that players will be able to join distributed virtual activities from any place and at any time.

• Increased capability for scalability—from combat theater to foxhole—with appropriate level of detail to support activity at either extreme.

• Improved ability to represent the behaviors of forces by computer-driven virtual entities to include complex concept formulation, planning, and reasoning activities in addition to simple drills.

• Increased availability of communication bandwidth to accommodate more simultaneous players, accommodate demand for more tightly coupled and reactive simulation processes, to realistically stress players, and to realistically simulate "fast" processes.

• Increased availability of tools for economical "rapid prototyping."

Page 146

In the entertainment simulation world, return on investment is a key consideration. Research and development will focus on achieving value in the perception of the end customer. The need to impress end customers whose experience base is grounded in television and the real world will focus the competition at the highest levels of fidelity consistent with economic pricing. Pressure will continue to increase the performance and reduce the cost of leading-edge technologies so that each new generation of a product stimulates new demand and creates a competitive edge over its predecessor.

A conflict can be expected to develop between advocates of open standards and guardians of proprietary intellectual property. The substantial barrier to entry represented by development investment and the reduction of same that common standards promote will be cited by both groups as justification for promoting or avoiding the adoption of technologies common to competitive development teams. Ultimately, competition will refocus on the creative aspects of entertainment simulations, as developers realize that economy and speed in bringing an idea to market are greater factors in economic success than proprietary technology.

Entertainment developers suffer an approach-avoidance conflict over the accelerating pace of technological innovations, both because of the diminishing economic half-life of a development investment and the chaos in the competitive environment that the continuing avalanche of new capabilities will create. Even savvy buyers will become dizzy and indecisive as great products are eclipsed by pending spectacular ones.

Enabling Technologies

What are some of the research priorities that will fuel the evolution suggested here? (1) Continuing geometric advancement in computing power, especially in the special-purpose hardware that creates imagery, with an accompanying dramatic reduction in price per performance. We can look ahead to the availability of photorealistic interactive systems at a price affordable by every household—e.g., the cost of a television set. (2) Dramatic improvements in the capability to display virtual environments to human senses: very-high-resolution visual displays; true spatial sound; and tactile displays that communicate surface qualities (friction), resilience, and thermal characteristics (heat capacity).

Page 147

Traci A. Jones

There has recently been an increased focus on simulating and modeling the individual soldier within the synthetic, or virtual, battlefield. The U.S. Department of Defense (DOD) has approved a Defense Technology Objective (DTO) for Individual Combatant Simulation (ICS). The ICS DTO is currently supported by an Army Science and Technology Objective (STO) for ICS. This is a joint STO between the Simulation, Training and Instrumentation Command (STRICOM) and the Army Research Laboratory, coordinated with the Natick Research and Engineering Directorate. The program intends to procure and demonstrate technologies for creating real-time simulations to immerse the individual soldier and allow for interaction in a synthetic environment. The cost-effectiveness of networked virtual reality devices will be determined using a multisite distributed laboratory consistent with DOD's High-level Architecture. The STRICOM Engineering Directorate is working closely with the Project Manager for Distributed Interactive Simulation (DIS) on the Dismounted Warrior Network project, which will take advantage of several technology-based efforts to provide an engineering proof of principle for immersing an individual into a synthetic environment.

The products that will evolve within DOD include the definition of a systems architecture to support the requirements for ICS as well as platforms and simulations that will support low-cost capabilities for mission rehearsal, materiel development, and training of individual soldiers and marines. There also is potential application of these technologies to training and rehearsal for the Federal Bureau of Investigation and the law enforcement industry.

The technological advances required and the technological challenges include low-cost solutions for:

• Visualization of human articulation in real-time networked environments,

• High-fidelity fully immersive systems,

• Interoperability between different fidelity simulators,

• Expansion of computer-generated forces for intelligent individual soldier interaction and decision making,

• Integration of high-resolution terrain databases with immersive simulations instrumentation of the individual for high-precision engagement data collection capability within buildings,

• Rapidly generated terrain databases to support mission planning and rehearsal while en route to a conflict, and

• Accurate simulation of weapons systems in real-time computer-generated environments.

Page 148

Regarding complementary efforts in the entertainment or defense sectors that might be applicable to my own interests, I am aware of the motion-capture techniques used by the entertainment industry, primarily for game development and motion picture special effects. One such product is being used for the STRICOM Dismounted Soldier Simulation (DSS) system, under contract to Veda Inc. DSS uses a wireless optical tracking system developed by the Biomechanics Corporation for Acclaim Entertainment. The technology has been integrated into a real-time DIS environment. The untethered soldier, outfitted with a set of optical markers and wireless helmet-mounted display, moves about freely in a real-world motion-capture area, while position and orientation data are gathered and sent to a DIS network via tracking cameras and image-processing computers. Fully articulated human motion rotations and translations are sent out to the DIS network using entity state and data protocol data units. Issues such as network bandwidth limitation and system latency have been analyzed.

Other potential products being developed by STRICOM have application to the entertainment industry. The Omni-Directional Treadmill is an example of a locomotion simulator that allows an individual to walk and run in a virtual world. As the user moves on the treadmill, his view of the computer-generated world changes, immersing him into the virtual environment. The Army may use this technology, for example, to rehearse for a mission by walking through a hostile environment beforehand. It is anticipated that additional technologies developed by the entertainment industry can be leveraged to support DOD requirements for individual combatant simulation.

Page 149

Eugenia M. Kolasinski—
Predicted Evolution of Virtual Reality

As this report indicates, virtual reality (VR) technology has many promising applications in both the simulation and entertainment arenas. VR technology is already being used for simulation, and, as the cost decreases, its many potential applications will likely lead to widespread use of VR, especially in the home for entertainment.

Necessary Technological Advances and Primary Research Challenges

Unfortunately, a phenomenon exists that may pose a threat to the ultimate usability of this new technology. That phenomenon is referred to as "simulator sickness" and it is a well-documented effect of simulator exposure (Reason and Brand, 1975; Kennedy and Frank, 1983; Kennedy et al., 1989; Casali, 1986). Simulator sickness is similar to motion sickness but can occur without actual physical motion. The cardinal signs resemble those of motion sickness: vomiting, nausea, pallor, and cold sweating. Other symptoms include drowsiness, confusion, difficulty concentrating, fullness of head, blurred vision, and eye strain. Along with the potential discomfort to the individual, there are several operational consequences of simulator sickness: decreased simulator use, compromised training, and ground and flight safety (Crowley, 1987). There are additional effects of simulator exposure: delayed flashbacks and aftereffects (a sudden onset of symptoms) (Baltzley et al., 1989); shifts in dark focus (the physiological resting position of accommodation) (Fowlkes et al., 1993); eye strain (Mon-Williams et al., 1993); and performance changes (Kennedy et al., 1993).

One potentially critical effect of simulator exposure is postural disequilibrium, referred to as ataxia. Baltzley et al. (1989) suggested that unsteadiness and ataxia are the greatest threats to safety because there have been reports of such posteffects lasting longer than 6 hours and, in some cases, longer than 12 hours. Clearly, occurrence of ataxia has the potential for disastrous consequences.

Recent research (Kolasinski, 1996; Knerr et al., 1993; Regan, 1993) has documented that simulator sickness can also occur in conjunction with VR exposure. The potential consequences of such sickness—particularly with widespread use of VR technology—raise important safety and legal issues for both manufacturers and users alike. Thus, simulator sickness (including effects such as ataxia) as it occurs with VR exposure must be understood if the technology is to make its predicted progress over the

Page 150

next decade. To meet this goal, the primary research challenges will be to thoroughly investigate the phenomenon.

Fortunately, simulator sickness in a virtual environment (VE)—or "cybersickness," as it is called—need not be regarded as an entirely new phenomenon. As already noted, simulator sickness is related to motion sickness, a phenomenon for which a body of literature exists (Reason and Brand, 1975). In addition, a body of literature exists for simulator sickness occurring in military flight simulators and, to a lesser degree, other simulators such as driving simulators (Crampton, 1990). Thus, VR researchers need not entirely reinvent the wheel but can and should draw on the existing literature, at least in the initial stages of investigation.

Much of the sickness literature that may be applicable to VEs is reviewed by Kolasinski (1995). In this report, three major categories of factors that may be related to simulator sickness as it occurs in a VE were identified: factors related to the individual using the system, factors related to the task performed in the VE, and factors related to the VR system itself. Although simulator sickness is not a new phenomenon, a VE may differ in several important respects from the typical simulator. For example, depending on how a VE is defined, such a system is likely to involve some form of direct sensory input, probably through a head-mounted display (HMD), at least. Such devices may pose unique concerns, and current research efforts (Mon-Williams et al., 1993) are examining the effects of HMD use on the visual system. Thus, although research into sickness occurring in VEs can draw on previous simulator sickness research, new research must be conducted specifically in VEs in order to address sickness issues unique to the VR setting. Very little research exists on sickness as it occurs in conjunction with VR exposure. Furthermore, with few exceptions (Regan and Price, 1994), the majority of VR studies currently reported in the literature were not designed to specifically investigate sickness. Instead, most studies investigated the use of VR systems, with sickness examined only as an aside.

Kolasinski (1996) represents one of the first experimental investigations of simulator sickness as it occurs in VEs. The primary focus was to investigate the prediction of sickness based on characteristics associated with an individual using a VR system, but the occurrence of ataxia following exposure also was investigated. This research established that sickness did, in fact, occur. In some cases it was severe—one participant vomited—and/or involved lingering or delayed effects. Ataxia, however, was not found.

This latter finding—that ataxia did not occur even though sickness did—supports findings presented by Kennedy et al. (1995), who found that, with repeated exposure to a simulator, sickness decreases over time but ataxia increases. Although their finding has implications for repeat-

Page 151

ed use of VR technology, the finding of Kolasinski (1996) raises some specific issues of importance to the future application of VR technology. Ataxia is a well-documented effect of simulator exposure (Kellogg and Gillingham, 1986; Kennedy et al., 1993), and previous research has suggested that ataxia may also occur in conjunction with VR exposure. Rolland et al. (1995) found degradation in hand-eye coordination and errors in pointing accuracy following the wearing of a see-through HMD-results that demonstrate that negative aftereffects are indeed possible. There have also been anecdotal observations of individuals demonstrating significant ataxia following a 30-minute VR exposure (K.M. Stanney, personal communication, April 9, 1996). Finally, recent research (Kennedy et al., 1996) has concretely established the occurrence of ataxia following VR exposure.

The VE used in conjunction with the anecdotal observations referred to above was a maze, the traversal of which involved both forward and left/right-represented movements. On the other hand, the task employed in Kolasinski (1996)—the computer game Ascent—involved represented movements primarily in the forward direction only. This suggests that the kinematics of the task performed in the VE may have an important effect on the occurrence of ataxia. For example, VR applications involving limited represented movement—such as teleoperation or simple games—may pose limited risks of ataxia, whereas applications involving a high degree of represented movement—such as highly dynamic games—may pose greater risks of ataxia. Clearly, this unresolved issue is a critical one that must be investigated further.

Research on simulator sickness in VEs should also look at one area that has been neglected in the military simulator environment. Although studies indicate that sickness can occur, little—if any—research has investigated whether such sickness has an impact on training effectiveness. Given the great emphasis often afforded to the use of VR technology for training and education, investigation of the effects of sickness on training effectiveness is an important research issue whose time has come.

Applicable Complementary Efforts

As is clear from the above discussion and the references therein, a plethora of complementary efforts—both past and present research—exist in the area of simulator sickness. Most of these efforts are directed toward military simulators. Leaders in such research include the Systems Effectiveness Division of Essex Corporation and the Spatial Orientation Systems Department at the Naval Aeromedical Research Laboratory (http://www.accel.namrl.navy.mil).

However, as noted, research specific to VEs also must be conducted

Page 152

to address the phenomenon specifically as it occurs in VR systems. VR research is being conducted in many laboratories around the globe, several of which are also interested in the investigation of simulator sickness. Such laboratories include the Human Interface Technology Laboratory at the University of Washington (http://www.hitl.washington.edu) and the Ashton Graybiel Spatial Orientation Laboratory at Brandeis University (http://www.bio.brandeis.edu/pages/faculty/dizio.html). There are also many laboratories in the United Kingdom conducting VR research. The major VR researchers there have established a group known as the UK Virtual Reality Special Interest Group (http://www.crg.cs.nott.ac.uk/ukvrsig/), made up of representatives from both industry and academia, which aims to provide a communications network for all VR researchers and users in the United Kingdom. Some of the member laboratories, such as the Virtual Environment Laboratory at the University of Edinburgh (http://hagg.psy.ed.ac.uk/), also are interested in investigation of the effects of VR exposure.

A final major contributor to the investigation of simulator sickness in VEs is the Simulator Systems Research Unit (SSRU) of the U.S. Army Research Institute (http://www.ari.fed.us/ssru.htm). SSRU is investigating the use of VEs for the training of dismounted infantry (Lampton et al., 1994a) for the ultimate goal of integrating the dismounted soldier into large-scale networked simulations. As part of its research effort, SSRU also is dedicated to investigation of the occurrence of sickness in VEs (Lampton et al., 1994b).

References

Baltzley, D.R., R.S. Kennedy, K.S. Berbaum, M.G. Lilienthal, and D.W. Gower. 1989. "The Time Course of Postflight Simulator Sickness Symptoms," Aviation, Space, and Environmental Medicine 60(11):1043-1048.

Casali, J.G. 1986. Vehicular Simulation-induced Sickness, Volume 1: An Overview. IEOR Technical Report No. 8501, NTSC TR 86-010. Naval Training Systems Center, Orlando, Fla.

Crampton, G. (ed.). 1990. Motion and Space Sickness. CRC Press, Boca Raton, Fla.

Crowley, J.S. 1987. "Simulator Sickness: A Problem for Army Aviation," Aviation, Space, and Environmental Medicine 58(4):355-357.

Fowlkes, J.E., R.S. Kennedy, L.J. Hettinger, and D.L. Harm. 1993. "Changes in the Dark Focus of Accommodation Associated with Simulator Sickness," Aviation, Space, and Environmental Medicine 64(7):612-618.

Kellogg, R.S., and K.K. Gillingham. 1986. "United States Air Force Experience with Simulator Sickness, Research and Training," in Proceedings of the 30th Annual Meeting of the Human Factors Society 1:427-429.

Kennedy, R.S., and L.H. Frank. 1983. "A Review of Motion Sickness with Special Reference to Simulator Sickness," paper presented at the National Research Council Committee on Human Factors Workshop on Simulator Sickness, September 26-28, Naval Post-Graduate School, Monterey, Calif.

Page 153

Kennedy, R.S., J.E. Fowlkes, and M.G. Lilienthal. 1993. "Postural and Performance Changes Following Exposures to Flight Simulators," Aviation, Space, and Environmental Medicine 64(10):912-920.

Kennedy, R.S., M.B. Jones, K.M. Stanney, A.D. Ritter, and J.M. Drexler. 1996. "Human Factors Safety Testing for Virtual Environment Mission-operation Training," Contract No. NAS9-19482.

Kennedy, R.S., D.S. Lanham, J.M. Drexler, and M.G. Lilienthal. 1995. "A Method for Certification That Aftereffects of Virtual Reality Exposures Have Dissipated: Preliminary Findings," pp. 263-270 in A.C. Bittner and P.C. Champney (eds.), Advances in Industrial Ergonomics Safety VII. Taylor and Francis, London.

Kennedy, R.S., M.G. Lilienthal, K.S. Berbaum, D.R. Baltzley, and M.E. McCauley. 1989. "Simulator Sickness in U.S. Navy Flight Simulators," Aviation, Space, and Environmental Medicine 60(1):10-16.

Knerr, B.W., D.R. Lampton, J.P. Bliss, J.M. Moshell, and B.S. Blau. 1993. "Human Performance in Virtual Environments: Initial Experiments," Proceedings of the 29th International Applied Military Psychology Symposium. Wolfson College, Cambridge, U.K.

Kolasinski, E.M. 1995. "Simulator Sickness in Virtual Environments," ARI Technical Report 1027. U.S. Army Research Institute for the Behavioral and Social Sciences, Alexandria, Va.; available on-line at http://www.ari.fed.us/ssru.htm. Also available by anonymous ftp at ftp.hitl.washington.edu/pub/scivw/publications/SimSick.rtf.

Kolasinski, E.M. 1996. "Prediction of Simulator Sickness in a Virtual Environment," Dissertation Abstracts International, 57-03. University Microfilms No. 96-21485. Available online at http://www.hitl.washington.edu/projects/vestibular/kolasinski/.

Lampton, D.R., B.W. Knerr, S.L. Goldberg, J.P. Bliss, J.M. Moshell, and B.S. Blau. 1994a. "The Virtual Environment Performance Assessment Battery (VEPAB): Development and Evaluation," Presence 3(2):145-157.

Lampton, D.R., E.M. Kolasinski, B.W. Knerr, J.P. Bliss, J.H. Bailey, and B.G. Witmer. 1994b. "Side Effects and Aftereffects of Immersion in Virtual Environments," Proceedings of the 38st Annual Meeting of the Human Factors and Ergonomics Society, Vol. 2, pp. 11541157.

Mon-Williams, M.A., J.P. Wann, S.K. Rushton, and R. Ackerley. 1993. "Real Problems with Virtual Worlds," Ophthalmic and Physiological Optics 13:435-436.

Reason, J.T., and J.J. Brand. 1975. Motion Sickness. Academic Press, London.

Regan, E.C. 1993. "Side-effects of Immersion Virtual Reality," paper presented at the International Applied Military Psychology Symposium, July 26-29.

Regan, E.C., and K.R. Price. 1994. "The Frequency of Occurrence and Severity of Side-effects of Immersion Virtual Reality," Aviation, Space, and Environmental Medicine 65(6):527-530.

Rolland, J.P., F.A. Biocca, T. Barlow, and A. Kancherla. 1995. "Quantification of Adaptation to Virtual-eye Location in See-thru Head-mounted Displays," Proceedings of the Virtual Reality Annual International Symposium 1995. IEEE Computer Society Press, Los Alamitos, Calif., pp. 56-66.

Page 154

John N. Latta—
Flights of Fantasy: An Oxymoron—Defense and Entertainment

The lure is engrossing—incredible defense technology being converted to the best entertainment this side of watching war on CNN. Visions of long lines of want-to-be war fighters can be seen making entertainment operators salivate at the thought of bulging bank accounts based on skyrocketing cash flow per square foot. Fantasy or a potential winner? Just a dream. Entertainment is a business, and war fighting is about execution in combat. There is no congruence in commercial business models and military mission statements. Out-of-home entertainment is a social experience, while winning on the battlefield is about doctrine, planning, leadership, and team effectiveness. Defense is also about leveraging technology to superior advantage in war. Yet in entertainment, technology is a lever to increase play rates and draw in the context of social environment. The often-heard chorus is that defense technology has applications in many sectors and the entertainment industry may be one. Yet, for example, in three-dimensional technology the conversion has largely taken place and the fuel of innovation is not Department of Defense (DOD) reuse but entrepreneurs seeking to get rich as they spend venture capitalists' money in new start-ups. DOD can help the entertainment industry by having more movie theaters on military bases.

Page 155

William K. McQuay

Advances in software and computer technology are making possible complex simulations based on affordable and reusable modeling components. Businesses will soon be able to realize increases in productivity through the widespread employment of simulations as aids for decision making and training. As a result, the commercial marketplace will increase for generic simulation techniques, simulation infrastructure, and off-the-shelf components for applications in financial industries, manufacturing, industrial process control, biotechnology, health care, communication and information systems, and entertainment.

DOD Technology for Industry

The entertainment industry has brought simulation technology and synthetic environments into the media mainstream. However, development of the software to enable such simulations is a manpower-intensive endeavor and thus is costly. Industry has the opportunity to exploit current U.S. Department of Defense (DOD) research and simulation technologies to bring products to market faster and at lower cost. Industry can leverage DOD joint standards and modeling and simulation (M&S) initiatives such as the DOD High-level Architecture, distributed interactive simulation (DIS), joint simulation system (JSIMS), the joint warfare simulation (JWARS), and the joint modeling and simulation system (JMASS). The joint M&S standards provide execution frameworks and emphasize models based on interoperability, reuse, portability, distributed operation, scalability, broad applicability, technological evolvability, and maximum feasible use of commercial off-the-shelf software. A potential high-payoff defense simulation technology is desktop M&S—simulation brought to the personal computer on the desktop of the engineer, analyst, and decision maker. Desktop M&S technology could be the basis for future video games, Internet games, or location-based attractions.

As entertainment simulations become increasing complex, the industry will face some of the same challenges faced by DOD in military simulations. As a result, DOD and industry could benefit from technology sharing in such areas as:

• Extensible architectural frameworks for tools and models that support a "plug-and-play" concept;

• The ability to geographically distribute simulations across a heterogeneous computer network;

Page 156

• Simulation development tools to support creation of model components that comply with architectural standards;

• Multiple language support: a user can specify the target source language (C, C++, Objective C, Ada, Java, etc.) to ease the transition to Internet-based entertainment; and

• Object-based technologies to allow component reuse in different products and on different platforms.

Commercial Technology for DOD

It is current DOD policy to use commercial off-the-shelf software whenever it meets DOD requirements. The DOD joint standards are designed as open systems architectures that support commercial off-the-shelf software and tools. The commercial sector has been very successful in developing two- and three-dimensional visualization software and in creating virtual reality applications. Such tools are more affordably and efficiently created by industry and can be maintained at low cost by a broad customer base.

Under a collaborative M&S marketplace concept, industry could build commercial and entertainment simulations based on DOD frameworks and reusable components and supplement them with advanced visualization technology and animation. DOD could employ these commercial products as needed to meet individual organizational requirements. Broad DOD and military service requirements could be satisfied by core joint M&S and supplemented by multiple commercial tools and capabilities from the collaborative M&S marketplace. DOD has insufficient resources to purchase DOD-wide licenses for the multiplicity of unique and individual products required for all DOD and service organizations. Instead, the collaborative M&S marketplace becomes a new outlet for commercial application developers where the DOD field organizations buy the exact product they need. Companies will have a new arena for sales of commercial products (tools and eventually even model parts) compatible with DOD joint standards. The best of DOD and commercial technology would be available to both sectors.

Page 157

Jacquelyn Ford Morie—
The Military and Entertainment: Historical Approaches and Common Ground

The military and the entertainment industries have come to their respective uses of technology from very different directions and motivations. The military has typically started with an existing need: for training people how to fly an airplane, for example, or for better communications. The military has then been extremely successful in creating the technology that will meet those needs—thus producing the better-trained, or better-informed, individual. The creation of a technology is driven by need. The entertainment industry, on the other hand, has typically started with existing technology but has been very good at creating a need within the audience that will bring the people in—to the arcade, the theme park, or other venue. The need, be it for an experience that continues the story of a popular film or a way to move people around a park, comes after the technology that supports it.

It is immediately apparent that there is a great deal of common ground in these two approaches. The military and entertainment industries have been complementary for longer than one might realize. There is a sign on an airplane simulator invented by Edwin Link in 1930 at the U.S. Air Force Armament Museum in Pensacola, Florida, that states that it was originally designed as an entertainment device. This "Blue Box" was sold to amusement parks until 1934, when Link, a pilot himself, met with the Army Air Corp to sell the Corp on the concept of pilot training with his device. The rest is history. The key here is that people enjoy interesting and satisfying experiences, whether for job enhancement or personal enrichment. For the military, the experiences provided by the technology were directly applicable to better performance in the mission of the job. They worked because they were interesting and pleasurable, as well as realistic, ways of learning the task at hand. For entertainment audiences, the motivation is more self-centered and aimed at enhancing one's personal time. These "civilian" experiences are motivated by several desires: thrill seeking, escape from one's everyday world, social interaction, or self-betterment (physical or mental).

What the military did in accepting Edwin Link's idea to use his entertainment device as a trainer has been echoed in more recent times by the appropriation of military technology by the entertainment industry. The common ground is an invention or an idea that lends itself to multiple uses. Where do these ideas comes from? Many of them come from a fertile environment for thinking and creating. For years the military has

Page 158

utilized just this kind of environment within the academic walls of university research labs to help develop some of its more cutting-edge ideas. By investing in these groups, the military has allowed ideas to ferment in diverse locations with heterogeneous teams of people. Over the decades, it has received a very nice return on its investments. Until recently, however, very few entertainment companies had taken advantage of the potential of these same research settings.

In 1991 I proposed to my research laboratory, the Institute for Simulation and Training (IST) at the University of Central Florida, a new initiative designed to bring together entertainment companies with what I saw as the related research we were doing for the government in virtual reality technology. Working with a theme park design professional, Chris Stapleton, as my partner to determine areas of common interest, we set about to bring the entertainment industry to a working familiarity with the latest in digital-research, in a project we dubbed "Operation Entertainment." Dozens of entertainment professionals came to IST over the next three years; we brought them in for endless demonstrations of what we were doing and intense discussions of how the work could apply to their profession. While we were never able to convince them to invest money in our laboratory, there were many seeds planted and several successes. One was when we advised Doug Trumbull on computer technology and connected him with an Orlando business from which he purchased the equipment to start up his company to produce the Luxor project. The second was in the creation of "Toy Scouts," which are discussed in the following paragraph. Many of the ideas developed through the history of Operation Entertainment have pointed the way to where the entertainment industry might go if it were to invest in the research labs that are already out there. Japanese companies have been doing so with the largest labs and the entertainment giants are starting to follow suit. There are many more labs out there as well that could prove extremely useful as the technology develops, and since many of them are already involved in military research as well, there is a great potential to maximize this research so that it benefits both groups. This is truly the best and most promising common ground. But exactly how can this type of collaboration be accomplished? Entertainment companies certainly don't have the dollars to invest in research the way the government does. This is, for the most part, true, but there are new ways we can think about collaboration and mutual discoveries.

One example is embodied in my work with a group at the Institute for Simulation and Training's Visual Research Laboratory that we called the "Toy Scouts." This was a group of undergraduate computer science and art students who met on Friday nights to see what they could do with the treasure chest of military "toys" that existed in our research

Page 159

laboratory. Guided by volunteer researchers in the lab, and with the outside advice of some local entertainment experts who would periodically visit, the students developed truly innovative full-body immersive games using virtual reality technology. One of the games was called "Nose Ball." In Nose Ball you used your nose as the paddle that controlled the ball in a three-dimensional breakout game. Because it was in the center of your stereoscopic vision, it was a perfect aiming device. Nose Ball was also a full-body physical workout. In the four years of the Scout activity, approximately a dozen new full-body immersive games were developed, with many clever and innovative ways to interface with the technology. These students, with their raw energy and fresh approaches, came up with ideas that might not have occurred to the more seasoned professional. The students benefited educationally from the expertise of the researchers they worked next to, and the researchers were often able to look at things with fresh eyes because of their close proximity to the Scouts. The entertainment industry was able to get new ideas from this work, and it became a wonderfully synergistic approach and experience to all involved.

The military has long partnered with the academic research community as an integral part of the discovery and implementation process for bringing new technology and techniques to a state of usefulness. The above example of the Toy Scouts is only one suggestion of how the military and entertainment industries can find common ground in academic research laboratories. The entertainment industry could sponsor such groups around the country at military research laboratories, and both groups could reap the rewards. No doubt there are many more ways that can be imagined; if only a fraction of them are implemented, the benefits might amaze us all.

Page 160

Jacquelyn Ford Morie—
The Evolution of Entertainment: Who's in Charge?

In the entertainment realm the audience is starting to become more and more sophisticated. Reversing a decades-old decline that has continually devolved an audience into ever-more-passive beings, today's audiences are eager and hungry for more direct participation. Fueled partly by home video games, and partly by the Internet, participants want more and more control over the experiences they are being offered. Video games appeal because the player is in control; one achieves a sense of satisfaction by reaching ever higher levels at one's own pace. The Internet is engaging in large part because it empowers the user to be a producer as well as a consumer. The entertainment industry, by contrast, driven as it is by economics of throughput and ticket prices, wants neither producers or controllers as its perfect audience. A passive audience allows for the most control over the numbers and timing of the attractions. However, the result of this is boredom: while the attractions grow ever-more grandiose and able to accommodate ever-larger crowds, the audience tires quickly and does not come back for repeated plays. The people do not feel themselves an active part of the experience. The audience has the ultimate control—it speaks with its time and its wallet. The entertainment industry will find it more difficult to continue in the old proven formulas of canned events that an audience is driven, flown, walked, or bumped through.

The next decade will see a trend toward what audiences demand—more control and empowerment. This will happen in several ways. The first is through more individual and unique play experiences, the second through more team play experiences, and the third through more spectator experiences. A few words on each are in order.

Individual Play Experiences

Individual play experiences appeal to our need for a self-directed experience, even if done in a social setting. They need to progress beyond individual home or arcade video games and extend the level of interactivity far beyond simple repetitious button punching.

This area was one I worked in for several years at the Institute for Simulation and Training's Visual Research Laboratory with a group we called the "Toy Scouts." This was a group of undergraduate computer science and art students who met on Friday nights to see what they could do with the treasure chest of military "toys" that existed in our research laboratory. Guided by volunteer researchers at the lab and with the outside

Page 161

advice of some local entertainment experts, these students developed truly innovative full-body immersive games using virtual reality technology. One example was a game called "Nose Ball." In Nose Ball you used your nose as the paddle that controlled the ball in a three-dimensional breakout game. Because it was in the center of your stereoscopic vision, it was a perfect aiming device. Nose Ball was also a full-body physical workout. In the four years of the Scout activity, approximately a dozen new full-body immersive games were developed. It was far cry from the couch potato mentality we might have expected from the video game and TV generation. In fact, this is an innovative way to combine sports and simulated experiences—a wonderful athletic hybrid. Think of going to some future digital gym for a Nose Ball workout!

While immensely popular with the audiences who experienced them, the drawback to these games for the entertainment industry is economics. The games were so enjoyable that the typical experience was 10 to 15 minutes long. Add to that the suiting up time and lead-in of how to play, and there just couldn't be enough return on an investment to make a profit. For this to evolve, the technology needs to be cheaper and easier to use, but it also requires a new way of thinking about technology as something active, vibrant, and participatory, with innovative interfaces that extend interactivity far beyond simple button pushing.

Team Play Experiences

A second big challenge for entertainment companies today is how to make computer interactivity play to a group larger than just a few people at a time. The military solved this problem years ago with SIMNET. As the grandfather of this area, SIMNET provided not so much prescribed scenarios but a common ground for participants to work together toward a goal. We have seen only a handful of successes in the entertainment community so far, and these involve fairly small-sized audiences—typically 12 to perhaps 100 people.

There is definite need to continue to develop experiences in this realm. These types of activities fulfill our need as social beings to work together and communicate with one another in a group situation. This is one of the reasons why Internet chat groups are so popular. The best and most successful group experience to date, especially in terms of the larger audience, is Loren Carpenter's 1991 interactive piece shown at SIGGRAPH in Las Vegas (and again at SIGGRAPH 1994 in Orlando).

Loren's "game" not only allowed for 3,000 to 5,000 simultaneous players to control a "pong" game or a flight simulator, but it did so while building a level of group excitement and involvement that has rarely been seen in our current digital entertainments. A surprising outcome of

Page 162

this game was that the audience as a whole did not perform at an average level, as might be expected, but at a much higher collective performance level. What heightened the level of the collective fervor was that the individual audience members could immediately sense their influence on the outcome. More work needs to be done at this level of team play.

Spectator Experiences

An obvious extension to the realm of team play is that of spectator play. Not everyone involved with digital entertainment will want to be a direct participant. Sometimes people enjoy themselves when they are engaged as a spectator. Being a spectator is not necessarily about being passive; it is about being a participant with anonymity within a crowd. This provides some people a less threatening forum in which to express themselves. Look at football or other team sports as the best example: only a small percentage of the participants actually play. The bulk of the industry (as well as the money to pay the players) is built around the fans. There is a potentially huge market to be developed for providing a substantial and rewarding spectator experience in the digital entertainment realm. So far no one is exploring this avenue.

These types of experiences require a new collaboration of entertainment with its audience. The military, in this respect, has been most responsive to its audience—not only the individual player but also the group dynamics that it served to train or connect. The thing to remember is that technology itself will not sell anything beyond a momentary novelty. It is the larger experience that will spell success or failure, and it is in giving the audience what it desires that the most successes will be found. It is up to us to find the ways to do this.

Page 163

Jeffrey Potter

Our segment of the computer graphics market extends from the plug-in card for the home personal computer all the way up to the high-powered workstation graphics accelerator for engineering industrial use. We expect to see the natural increase in renderer horsepower and on-line storage capabilities that the computer industry has become accustomed to. Every 12 to 18 months, the processing sees about a twofold increase in performance, with storage capacities moving at nearly the same pace. Simultaneously, we expect to see features once reserved only for the expensive workstation market to gradually filter down and become available to the home computer user. These features include high-quality antialiasing, acceleration of both geometry and display processing, and advanced texturing capabilities. Simultaneously, we expect to see new exotic ways in which three-dimensional (3D) computer graphics can be applied to the common tasks done in a 2D world today. Remember, not too long ago we were using 24-line, 80-character, alphanumeric-only displays to do our word processing and spreadsheets. With the advent of inexpensive 3D graphics, ordinary 2D graphics might seem quaint and backwards in just a few more years.

Like any product that undergoes evolutionary change, computer graphics products will react to developers' needs. Operations that become the most commonly used routines performed by the host central processing unit (CPU) in software will eventually migrate to hardware. The host CPU is then able to control rendering at a higher level, and developers can start thinking up the next big processor-intensive algorithm. We do not see a fixed set of features being used to separate the personal computer (PC) market from the workstation market. The line between personal computer graphics and workstation graphics will be more rooted in price points, not capabilities. That is to say, what we consider to be workstation-quality graphics today will be on every PC owner's desktop in a couple of years. Of course, what will be on the workstation at that time will be limited only by our imagination today.

The enabling technological advances are primarily what has driven the computing industry so far:

• Semiconductor process and geometry—the push to fit ever more gates onto reasonably priced pieces of silicon while keeping thermal and mechanical problems under control. This matters to both the "number crunching" hardware and the random access memory.

• Memory bandwidth—developing newer higher-bandwidth memory architectures that adapt readily to the 3D graphics paradigm.

Page 164

• Interface standards—such as the advanced graphics port, allowing the processors and custom-rendering hardware the capability to take advantage of new higher-bandwidth memory.

• New algorithm development—especially in areas such as image compression to further enhance the apparent processing speed of a system.

The research challenges are to invent the next "big thing" in computer graphics. Our Compu-Scene IV product practically stole the market in high-end military flight simulation and training in 1984 when we introduced photographic-quality texturing to real-time graphics. Research and development must strike a happy medium between finding the next gee-whiz feature that engineering can dream up and the marketable improvements that translate into increased sales.

In our experience, one market drives the other, and occasionally developments and feature sets come full circle. U.S. Department of Defense (DOD) applications concentrate on real-world accuracy and training effectiveness. Entertainment applications want the "look and feel" of the high-powered military simulations but at consumer price points. So the products for the entertainment market are designed with carefully chosen compromises based on engineering/marketing research and user feedback. These commercial products then sometimes catch the interest of military customers, who realize that some lower-fidelity systems (such as part-task trainers) can deliver effective training with these compromises.

The drive to create interactive entertainment over the Internet is a prime example of complementary efforts. The lessons learned by the defense industry suppliers involved in the Distributed Interactive Simulation standard can be put to good use by the entertainment community.

We have had a close working relationship with Sega Enterprises, Ltd., developing the graphics hardware systems for the Model 2 and Model 3 arcade systems. This drove us to miniaturize our image generator architecture and to develop new algorithms for such features as antialiasing. We have used this cross-pollination of ideas to enhance our product line, most notably the R3D/100 chop set and R3D/PRO-1000 system. The R3D/PRO-1000 system is then able to serve markets that previously required expensive workstation-based systems at lower cost.

Page 165

David R. Pratt—
Military Entertainment?

The Joint Simulation System (JSIMS) is the flagship program of the next generation of constructive models. JSIMS is a single, seamlessly integrated simulation environment that includes a core infrastructure and mission space objects, both maintained in a common repository. These can be composed to create a simulation capability to support joint or service training, rehearsal, or education objectives. JSIMS must facilitate Joint Service training, significantly reduce exercise support resources, and allow user interactions via real-world command, control, communication, computing, and intelligence (C4I) systems. The final system will support the ability to resolve down to the platform level the development of doctrine and tactics, mission rehearsal, linkages with other models (e.g., analytical, live, virtual), and a wide range of military operations.

As outlined above, the modeling and simulation (M&S) goals of JSIMS are undoubtedly bold and ambitious. Early on, service- and agency-specific programs were identified to be part of the overall JSIMS program. Based on the three pillars of the Defense Modeling and Simulation Office's common technical framework (conceptual model of the mission space (CMMS), High-level Architecture (HLA), and data standards) along with technology infusion provided by Defense Advanced Research Projects Agency programs (such as the Synthetic Theater of War and Advanced Simulation Technology Thrust), JSIMS represents the first true U.S. Department of Defense (DOD) community-wide M&S developmental effort. The question is whether JSIMS can possibly leverage off of M&S efforts from outside DOD, in particular those from the entertainment industry. Foremost, the goals for a successful military simulation and an entertainment simulation are markedly different. In entertainment the driving factors are excitement and fun. Users must want to spend their money to use it again and again (either at home or at an entertainment center) and hopefully be willing to tell others about it. Unrealistically dangerous situations, exaggerated hazardous environments, and multiple lives and heroics are acceptable, even desirable, to increase the thrill factor. On the other hand, defense simulations overwhelmingly stress realistic environments and engagement situations. The interactions are quite serious in nature, can crucially depend on terrain features or other environmental phenomena, and generally rely on the ability to coordinate jointly with other players. The value of these defense simulations is measured in terms of training and insights revealed. A successful military simulator could be deemed boring and therefore useless in terms of entertainment. Similarly, a successful enter-

Page 166

tainment simulator could be deemed unrealistic and therefore useless in terms of military training. However, I believe there exists a potential for DOD and the entertainment industry to leverage off each other's M&S efforts provided there is an understanding of how the two fundamentally differ and what each strives to do best.

From an operational point of view, there are three hard technological challenges facing JSIMS: synthetic environment (SE), computer-generated forces (CGFs), and resource reduction. To gain a level of confidence in the outcome of defense models, the models must realistically and consistently represent all of the battlespace in the SE. Tactically significant interactions with the SE, such as rain affecting mobility and line of sight, cross-environment interactions so that objects from the air domain can engage objects from the land domain seamlessly, must be simulated realistically across multiple types of platforms with different underlying terrain representations. CGF behaviors of entities in the simulation need to be flexible and rapidly configurable by end users, and the generated behaviors must continue to evolve through the experience gained as part of the exercises much like humans do in battle. Resources (in terms of time, equipment, and personnel) that currently drive training schedules must be reduced from their current levels. It simply takes too much to set up a simulation exercise. The goal is 96 hours versus the current six months.

The large-scale joint service nature and complexity of JSIMS generally preclude it from taking advantage of using much of the SE framework developed by the entertainment industry so far. However, efforts in the development of user interfaces, use of avatars, and artificial intelligence are of potential interest. User interface development is largely driven by the entertainment industry already as it is the primary means by which its customers experience the desired thrills. The defense training community could benefit from immersive user interfaces that permit more realistic interactions with the SE. Also of interest are more natural interfaces to effectively manipulate large numbers of CGFs or some aspect of the SE, as are the use of avatars to convey information. M&S-driven computer technology advancements that result in the availability of cheaper hardware to do complex computations efficiently, increased personnel expertise, and improved user interfaces could contribute to a significant reduction of resources required to conduct a simulation exercise. Artificial intelligence in CGFs used to populate environments of both defense and entertainment simulations can likely be leveraged provided that they can be flexibly programmed to carry out a variety of tasks and can exhibit advanced behaviors such as the capability to learn. This is the current challenge facing the CGF community within DOD today, and I pose it to the entertainment industry as well in hopes that we may be able to work together on this difficult problem. I have not been able to

Page 167

find a technical reason why the defense and the entertainment M&S communities cannot leverage off each other's efforts. A cross-pollination of ideas between the two appears fruitful provided that their differing M&S goals are not adversely compromised. In general, negative military training, which could result from lack of simulation fidelity or ambiguity in a user interface, is considered to be worse than no training at all.

Page 168

Alex Seiden—
Electronic Storytelling and Human Immersion

The past half-decade has seen a renaissance in digital effects in motion pictures. Correspondingly, the use of certain "traditional" effects technologies, such as compositing with optical printers, has diminished greatly. Writers and directors have been given a new and powerful set of tools to realize their visions. New techniques have made the impossible possible and the prohibitively expensive more affordable. Additionally, a tremendous amount of effects work is in the "invisible" category: wire and rig removals, sky and background enhancements, and so on.

Box office success fuels much in the world of filmmaking. (I am not so cynical as to say it is the only force in operation.) The tremendous returns on Terminator 2: Judgment Day and Jurassic Park exploded studio interest in visual effects and the facilities that create them. Many studios have made substantial investments in their own effects units. Currently, films such as Twister and Independence Day reinforce this trend. The demand for visual effects has never been as high as it is today, and it will continue to grow for the next few years. After leveling off, there will doubtless be a shakeout in the number of facilities providing these services; in some ways this may have already begun.

Visual effects and simulation computer graphics in the visual effects world have very little to do with "simulation," as the term is commonly used in the computing community. The broadest definition of a simulation is any synthetic or counterfeit creation. However, most in this audience would consider a simulation as being a mathematical or algorithmic model, combined with a set of initial conditions, that allows prediction and visualization as time unfolds. This generalized model allows easy manipulation of the initial conditions and parameters, such that many possibilities can be explored.

Computer graphics in visual effects has more to do with what is euphemistically called "hybrid" or "empirical" techniques, and more candidly called "grotesque hacks." The dinosaurs of Jurassic Park were not "simulated" any more than pre-World War II Los Angeles was "simulated" for Chinatown. Certainly, any shot in the latter film has a rich sense of place and time: the office of private investigator Jake Gittes is powerfully evoked, and the audience imagines the rest of that world. But turn the camera 30 degrees and you'll see a bunch of C-stands and some grips. Changing views requires substantial time—though the illusion of a complete and continuous world still exists. Similarly, changing the action of a synthetic creature, such as a dinosaur, requires extensive rework. No single generalized conceptual model exists for those dinosaurs and their

Page 169

important visual properties, such as their gait cycles, the movement of flesh over bones, or the texture of their saurian hides. Often, a precise simulation would not only be more complicated but would also be aesthetically undesirable; for example, the scale of dinosaurs in Jurassic Park changes dramatically from shot to shot and sequence to sequence.

Technological Advances and Research Challenges

Turnkey animation systems have advanced greatly in the past several years. Such advancements include flexible inverse kinematics that make creature animation practical, particle system front ends that allow complicated dynamic effects, and the big strides in software on Macintosh and personal computer-compatible systems. Nevertheless, software is the single largest area where attention should be focused. Animators and technical directors face daunting challenges as shots become more and more complicated. Techniques must be developed that allow more facile management of this increasing complexity. Visual programming, to name just one possible solution, has been proposed as a method of allowing people to work at higher levels of abstraction without sacrificing precise control; other ideas will hopefully emerge as well. Ask the production manager of any effects studio and he or she will tell you the biggest problem is the shortage of skilled animators and technical directors. As such, training and education will be a huge issue for the next several years.

Current renderers lack good simulation of area light sources and diffuse-diffuse interactions. This is critical for matching live-action photography, especially for daylight exteriors. Current solutions rely on difficult, poor-looking substitutes. I expect to see radiosity-based techniques become more common. Motion blur and programmable shading are necessary. Some researchers and industry luminaries have talked with great excitement about the potential for photo-realistic artificial actors. This will certainly see some application, but the interest in a revived Marilyn Monroe or Elvis is perhaps overrated. Hardware will continue its inevitable increase in performance and will be matched step for step by growing computing demands. In the long run, frame-rendering times never fall: my average frame time in 1985 was 20 minutes per TV-res frame; now it's around 30 minutes. The vast amounts of data contained in film-res images place great strain on networks; increasing speed and reliability will help speed production. The continuing development of lossless compression techniques also helps. Hopefully, display technologies will develop that can achieve the quality of film, especially more dynamic range and wider color gamuts. Affordable high-resolution real-time playback devices, such as high-definition television, may be part of the solution.

Page 170

Linear and Nonlinear Storytelling

It is important not to lose sight of the goal of movies or any entertainment: to expand and enrich our lives, to extend our experiences beyond those we have lived through ourselves, to illuminate and explain the experiences that we have lived, and to do all of this in a way that is engaging and compelling. Any art, particularly film, succeeds when the audience forgets itself and is transported into another world. Visual effects must always serve the story. I say this even though I am fully aware that recent effects-laden box office extravaganzas have been disturbingly lacking in this area. Nevertheless, I don't think many people outside the industry would pay $7.50 to sit through the effects reels (an effects reel is an edited piece of film showing only the shots that have visual effects and omitting the rest of the film) of any of these movies, stunning technical and creative achievements though they might be. In any event, many effects-oriented movies fail both commercially and aesthetically.

The Web, CD-ROM, virtual reality, and other multimedia techniques have been touted as ways to transcend the limits of linear storytelling, to give the audience a richer participation in an imaginary world. I will risk alienating many at this workshop by staking out what I expect to be some unpopular ground: I've never seen a CD-ROM that moved me the way a powerful film has. I've never visited a Web page with great emotional impact. I contend that linear narrative is the fundamental art form of humankind: the novel, the play, the film, even the orally told joke or story—these are the forms that define our cultural experience. Although "interactive" and "nonlinear" forms exist, they have never been paramount in our consciousness. Remember when, around the campfire during summer camp, one person would begin a story, tell it for a few minutes, and then another person would take over, developing the story in their own direction? Similar forms exist in improvisational theater games. And although these techniques can be amusing in small doses, I would argue that they are the exception that proves the rule: nonlinear storytelling forms can exist, but they will never supplant the complete control of the linear storyteller. Now, any new medium—film, radio, and television are good examples—has its curmudgeonly detractors. The infant film industry received similar criticism from those who argued that theater would always be the dominant medium. The critics lacked the vision of what film could and would become, as the language of film developed. Perhaps I, too, lack such vision and in 10 or 20 years will be eating my words. Will the world of interactive nonlinear entertainment grow? Yes, certainly it will. Will exciting and emotionally powerful images be created in these formats? Yes, there are too many talented people and too much money being sent in that direction for nothing to happen. And

Page 171

certainly the Web will become the preeminent forum for the exchange of commercial and scientific information; its significance will exceed that of the cellular phone, the automated teller machine, the fax machine, and the Home Shopping Network combined. This is not a trivial development. Whether storytelling itself will be fundamentally changed depends on a paradigm shift that I would contend is much larger than for other emerging media. To fully evaluate the likelihood and meaning of such as shift requires a careful distinction between what we think of now as a "story" and what we consider a "game" or "environment." A full appraisal of the differences between the cognitive processes involved is beyond the scope of this paper and is an excellent subject for further research.

Page 172

Steven Seidensticker—
Distributed Simulation: A View from the Future

The battle date is August 17, 1943. I am the ball turret gunner of Luscious Lady, a brand new B-17F of the 427th squadron, 303rd Bombardment Group, of the Eighth Air Force. Our takeoff from Molesworth was without incident, but as soon as we were off the ground the pilot asked me to check the wheels. He had an indication that the left main gear had not retracted fully. I hopped into the ball and spun it until I had a good view of the wheel. It looked OK. We chalked it up to a bad indicator in the cockpit. Although the ball with its twin 50s is primarily intended to protect a B-17 from enemy fighters approaching from below, the view from beneath the aircraft comes in handy for other chores. We climb out and begin a long lazy circle. I keep tabs on and report other squadron aircraft as they join our formation.

We are on our second mission and our first over Germany. Our first mission was to bomb a Luftwaffe airfield near Paris. The target was partly obscured by weather. Opposition was light. A few Me-109s came up to meet us. They were not particularly aggressive or well coordinated. Nevertheless, we lost one of our squadron. I saw Old Ironsides get most of her rudder shot off. The pilot was obviously losing control and chose to abandon his ship. I saw 10 good chutes. The debriefing team called the mission a "milk run." The missions would become much tougher as we gained more experience. We were happy to get this far.

My pilot and copilot are in Milwaukee. The navigator/bombardier is in Montreal. Other crew members are in Seattle, San Jose, Denver, and Green Bay. We cannot see or touch each other, but we communicate via what appears to be a B-17's standard intercom. In fact, we are part of a wide-area high-speed data network that connects all crew stations of all aircraft, both friendly and hostile. I don't know the total number of nodes on this network, but it must be in the thousands. The number of spectators who can tap into the net is in the millions. In addition to our voices, this network carries all the data that our individual crew station simulators need to show other aircraft the terrain over which we fly, the weather, and other elements of our environment. To participate in these missions each of us simply dials into the network at the time scheduled for the mission, gets the standard crew briefing on our screens, and waits for our turn to take off. The pilots, bombardiers, and navigators get a detailed briefing on the target and expected weather. The rest of the crew gets briefed on expected opposition. The briefings are, of course, the same as (or as close as possible to) the original briefings given to the original crews. Like in the original briefings, we can ask questions and get answers.

Page 173

Not all the crew stations on Luscious Lady are manned by humans. The waist gunners and the radio operator are computer-generated entities. They do their jobs reasonably well. They even respond to us when we talk to them over the intercom. However, if the conversation strays from simple orders or reports they quickly become confused and start spouting gibberish. Some of the other friendly aircraft on the mission and some of the opposing Luftwaffe fighters have no human crews at all. But it's getting harder to tell who is human and who is computer-generated, because the programmers keep tweaking their behavior algorithms. But my personal feeling is that they will never get to the point where these simulations are totally indistinguishable from real people. I hope they don't.

Over the Channel the pilot gives us the order to test our guns. This is a ritual that ensures that the guns are working and marks the real beginning of the mission for us gunners. From here we are in harm's way. I cock both guns, point to a clear area, and let loose with a short burst. The tracers arc away gracefully. I have managed not to hit anyone else in the formation. To do so is considered very bad form. It also requires the hapless shooter to buy dinner for the shootee's crew at our next annual convention. Of course, the computers that run this whole operation keep track of everything, so there is no arguing or hiding. The target today is the Me-109 plant in Regensburg. We know that the Luftwaffe was out in force that day. The Eighth Air Force lost 24 B-17s out of a force of 147. Shortly after we cross the French coast the nose gunner shouts "four 109s at 12 o'clock low." The control yoke feels comfortable in my hands as I spin the turret forward. They are coming at our formation four abreast from dead ahead. The winking lights on the leading edge of their wings show that they are firing. I mash the right pedal hard to tell the lead computing gun sight to use maximum range. The left pedal goes to the third notch to input the wing span of an Me-109. I line the sight's pipper on the number two plane and fire short bursts, trying to adjust the range as they close. My shots appear low. Just about everyone in our formation is firing. A puff of smoke bursts from the number three fighter. It continues to smoke as their formation passes right through ours.

This line abreast head-on attack was developed by the Luftwaffe in early 1943. It took a lot of courage and discipline on the part of the German pilots, but it was very effective. The idea was not only to get the best shots possible but also to intimidate the bomber pilots and break up the formation. It was probably the greatest game of chicken ever and it frequently ended in collision. The right waist gunner reports another formation at the four o'clock level. But they are out of our range and overtaking us on a parallel course, no doubt moving up for another head-on pass through the bomber stream. I can see their yellow cowlings and

Page 174

know that they belong to JG 26, the "Abbeville Kids," one of the best Luftwaffe fighter wings.

The attacks continue sporadically until we are about 30 miles from the target. At that point we start seeing the dreaded flak. The small black clouds bloom innocently in the distance, but we know that as the ground gunners adjust the aim of their 88s, the bursts will be right around us. There is little evasive action that a formation of B-17s can take. We are near the IP (initial point) that the pilot must fly over if we are to get our bombs anywhere near the target. At that point, the bombardier takes over and actually flies the plane to the bomb release point, using autopilot controls on the famous Norden bomb sight, probably one of the most famous but overrated technical developments of World War II. The flak rounds get closer.

The concussion from one of them is louder than the fifties going off next to my ears. The pilot reports that number four engine is starting to vibrate and that the manifold pressure is dropping. Bad news. If it fails we will have to drop out of the formation. Like the weak separated from the herd, we will be on our own. We may have to fight packs of fighters as we try for the coast and the protection of friendly Spitfires. Most who have been through this say that it can be the most exciting part of an afternoon of simulation, but the B-17 seldom survives. Those that do get an award at the next convention and, of course, their battle with the fighters is replayed on the large screen.

We finally reach the target, the bombardier hits the pickle switch, and I watch the bombs fall away. I loose sight of them after a few seconds, but shortly thereafter see a string of explosions on the ground. The bombs land in a rail yard just east of the target complex. But that's closer than the original crew came in 1943.

The flight back was challenging. For two hours we endured more flak and almost constant fighter harassment. Our pilot managed to coax enough power out of the number four engine to maintain our position in the formation. The rest of the formation was not so lucky. Stric Nine took an 88-mm round in the right wing root and the whole wing came off. There were no chutes. Wallaroo lost an engine and had to drop back, but we were close to the coast and a flight of P-47s escorted her back. Once we got over the Channel I turned over my role to an automatic ball turret simulation and had a quick dinner in the kitchen with my wife. I doubt that the rest of my crew even noticed I was gone. I rejoined the simulation for the debriefing. The colonel told us that we had done reasonably well for a second mission crew.

My ball turret is a medium-priced model from RealSim Inc., one of the rising companies in this field. It provides a lot of fidelity for the price and has a lot of update options. I'm very happy with it. The ball spins

Page 175

and rotates vertically much the way the original did and takes up less than half of my garage. The visual scenes are presented on panels built right into the ball. Sound and vibration are provided by some large but ordinary speakers. RealSim sells the basic turret dirt cheap but knows how sim-heads get hooked on fidelity, and so they offer a large range of add-ons that can become real expensive. Some of my colleagues have mounted their units on electrically driven motion platforms. I don't know if that is worth the extra cost. Maybe next year. Many other simulated crew stations are built around virtual reality goggles. Those are a lot less expensive but work quite well. One enthusiastic crew has built a whole B-17 fuselage in a warehouse.

As in most simulations, visual scenes provide the dominant cues. The simulation industry long ago reached its holy grail of creating visual images that are indistinguishable from the real thing. The processing power needed to create them is so cheap that the image generators are no longer a cost factor in most simulators. Databases that represent the terrain of any portion of the earth are readily available at any resolution desired. Specialty "period" databases (Dunkirk or Waterloo for instance) for groundpounders are becoming available but are very expensive.

The key factor that made this kind of group simulation possible was the development of the DIS (distributed interactive simulation) standards about 25 years ago. Once these standards were in place, the designers and builders of simulator components didn't have to spend any more time thinking about linking them together than does the designer of a railroad car need to worry about how to couple his car to a train. The DIS standards allowed the simulation industry to concentrate on functionality, performance, and cost reduction.

My wife used to ask me why I spend so much time and money on this. There are a number of reasons. I, like most middle-aged guys, have often fantasized about going into battle to test my wits and skill with a comparably equipped enemy. In this fantasy I support my comrades and in turn depend on their support. I yearn to experience the heat of battle, victory over my adversary, or a narrow escape from the reach of his weapons. However, I have no desire to shed any of my blood.

I also love history, great battles in particular. I know of no greater battle than that between the U.S. Eighth Air Force and the German Luftwaffe in 1943 and 1944. The leaders of the American forces felt that they could win the war with heavy bombing of German military and industrial targets. To be accurate this had to be done in daylight. Escort fighters of the day did not have sufficient range to cover the bombers. The bombers had to depend on their own defensive weapons.

Participation in these re-created battles is available at a number of levels. I started as a spectator. The magic carpet mode of my computer

Page 176

let me observe operations from any point in space. It also let me attach myself to any aircraft in the battle and listen to the radio and intercom traffic for that aircraft. Running commentary is available from experts. Previews and schedules of upcoming battles are carried by the major sports pages. Reports of completed battles also are carried. These tend to dwell on the personalities involved and the shoot-em-up aspects. How close the reenactment came to the original battle seems to be getting lost.

After watching several of the major raids, I was hooked and wanted to play an active role. My first desire was to be a Luftwaffe pilot, but the requirement for fluency in German eliminated that. Rumors are that an English-speaking Luftwaffe wing is forming. My second choice was to sit in the cockpit of a B-17. But, like the original aircrews, I needed training. The training course for all pilot positions is long and demanding. I opted for the less ambitious role of gunner. Fortunately, the simulator technology that I own trains me more efficiently and quickly than did similar training programs in 1943. After a few intense weekends, I passed the qualification tests and was assigned to my present crew. We are not the most proficient crew on today's raid, but neither were the new crews in 1943.

As I become more serious in this avocation, I wonder where it is going. Some social commentators are starting to decry the "glorification of war." Others counter with statements about "harmless outlets of male aggression," despite the fact that at last year's convention the Best B-17 Crew Award went to an all-female crew. Some critics are worried that the super-realistic simulation available today is going to replace drugs as the national addiction. Who knows! The raid on the ball-bearing factories in Schweinfurt is scheduled for next week. It was the bloodiest for the Eighth Air Force. I think my crew and I are good enough and lucky enough to survive. I can hardly wait to find out.

Page 177

Jack Thorpe—
Research Needs for Synthetic Environments

Purpose

This paper introduces one approach for thinking about the technical challenges of constructing synthetic environments and some of the related research issues. The paper is designed to stimulate discussion, not to be a comprehensive treatise on the topic.

Discussion

Simulation, virtual reality, gaming, and film share the common objective of creating a believable artificial world for participants. In this context, believability is less about the specific content of the environment and more about the perception that there exists a world that participants can port themselves into and be active in—that is, exert behavior of some sort.

In film, this is vicarious. In simulation, virtual reality, and gaming it tends to be active, even allowing participants to choose the form for porting into the environment: either as an occupant of a vehicle moving through the environment, as a puppet (proxy) of him or herself that he or she controls from an observation station, or as a fully immersed human. The iconic representation or avatar can assume whatever form is appropriate for the environment.

When the participant is an audience member in a single venue and is neither required to interact overtly with other audience members in the same venue or other connected venues, the issues of large-scale interactivity and distributed locations are minimal. On the other hand, when tens or hundreds of remotely located participants are ported into the same world and begin to interact freely (and unpredictably), as demonstrated in recent advances in distributed interactive simulation, not only are the environments more interesting but the technical challenges are also more difficult. It is likely that these will also be the next-generation commercial application for this technology, and so addressing technical issues is timely.

To design and build these more complex worlds, the following major tasks have been found to be useful classifications of the work needed to be done and the tools required to perform this work, thus leading to the research and development needed to construct the tools. For each of these tasks a few of the research issues are identified, but this is far from a comprehensive treatment:

Page 178

• Efficient fabrication of the synthetic environment;

• Design and manufacture of affordable porting devices that allow humans to enter and/or interface with these environments;

• Design and management of a worldwide simulation Internet to connect these porting devices in real time;

• Development of computational proxies (algorithms) that accurately mimic the behavior of humans unable to be present;

• Staffing, organization, and management of realistic, validated sentient opponents (or other agents), networked based, for augmenting the world; and

• Development of innovative applications and methodologies for exploiting this unique capability.

Efficient Fabrication of the Synthetic Environment

Artificial worlds are usually three-dimensional spaces whose features are sensed by the participants in multiple modes, almost always visual but possibly auditory, tactile, whole-body motion, infrared, radar, or via a full range of notional sensor or information cues. For each of these modes of interaction, the attributes can be specified in a prebuilt database ahead of time, or calculated in real time, or both.

The challenge is to construct interesting three-dimensional environments efficiently. Cost rises as a function of the size of the space (in some military simulations it can be thousands of square miles of topography), resolution, detail (precision cues needed for interaction), dynamic features (objects that can interact with participants, like doors that can open or buildings that can be razed), and several other factors. As a general observation, the tools needed to efficiently construct large complex environments are lacking, a particularly serious shortfall when fine-tuning environments for specific goals of a simulation or game. Toolsets are quirky and primitive, require substantial training to master, and often prohibit the environment architect from including all of the attributes desired. This is a serious problem, one that seems to get relatively little attention. It is an area that needs continual research and development focus.

Design and Manufacture of Affordable Porting Devices that Allow Humans to Enter and/or Interface with These Environments

The manner in which the human enters the synthetic environment continues to undergo rapid change. Flight simulators are a good example. Twenty years ago a sophisticated flight simulator cost $20 million to

Page 179

$40 million. Ten years ago technology allowed costs to drop by a factor of 100. Today there has been another one or two orders of magnitude decrease. Further, each new generation is more capable than its more costly predecessor. This drop in cost, with an increase in the richness of the participant's ability to interact with the environment and other people and agents similarly ported there, is especially important as large-scale simulations are constructed—that is, those that might have 50 or more participants (some military simulations have thousands of participants). The cost per participant (cost per seat) can be a limiting factor no matter how rich the interface.

The research issues include the design methodology that leads to good functional specifications for the simulation or game (the work on selective fidelity by Bob Jacobs at Illusion Inc. is relevant), the design and fabrication approaches for full-enclosure simulators (vehicles) and caves (individuals), the porting facade at the desktop workstation (partly manifested by the graphical user interface), and other means of entering the environment, such as while mobile via a wireless personal digital assistant.

Design and Management of a Worldwide Simulation Internet to Connect These Porting Devices in Real Time

Small-scale as well as large-scale distributed interactive environments have baseline requirements for latency, which is compounded when a requirement to worldwide entry into environments is added. Latency is influenced by the type of interaction a participant is involved with in the specific synthetic environment. The requirement is that the perception of "real timeness" is not violated, that is, that participants do not perceive a rift in the time domain (a stutter, momentary freeze, or unnatural delay in consequence of some action that should be a seamless interaction). Because this is a perceptual issue, it is dependent on the nature of the interaction and the participant's expectations.

This becomes a technology issue as the number of independently behaving participants grows, the number of remote sites increases, and the diversity of the types of interactions coming from these sites and participants grows. It has been demonstrated that unfiltered broadcasting of interaction messages ("I am here doing this to you") quickly saturates the ability of every participant to sort through all the incoming messages, the majority of which are irrelevant to a specific participant. The functionality needed in this type of large interactive network is akin to dynamically reconfigurable multicasting, as yet unavailable as a network service.

It could turn out that as the Internet expands it will provide the ded-

Page 180

icated protected speed and addressing for these types of interactions, but this is not the case to date, and dedicated networks have had to be installed to support large exercises. Further, it is conceivable that the appetite of the simulation or game designer for more complex and interactive environments will outpace the near-term flexibility and capacity of network providers. Networks are going to have to be smarter, a continuing research issue.

Development of Computational Proxies (Algorithms) That Accurately Mimic the Behavior of Humans Unable to Be Present

Late 1980s experimentation with distributed interactive simulations resulted in the constant pressure to grow the environments in the numbers of participants, but there were never enough porting devices or people to man them to satisfy this growth. Since these environments began as behaviorally rich human-on-human/force-on-force experiences, players demanded that any additional agents brought on via computer algorithm have all the characteristic behaviors of intelligent beings, that is, that they passed the Turing test and would be indistinguishable from real humans—a tall order.

This resulted in a series of developments of semiautomated and fully automated forces capable of behaving as humans and interacting alongside or against other humans ported into the simulation. These developments have met with mixed success. In some cases computer algorithms have been constructed that are excellent mimics of actual individuals and teams, particularly in vehicles, but in other cases the problem is more difficult, especially in areas of mimicking cognition as in decision making. Nonetheless, the commercial application as well as the defense application of large-scale interactive environments will require large-scale synthetic forces behaving correctly. Given that understanding, predicting, and "generating" human behavior transcends simulation and gaming, this will continue to be a major research area.

Staffing, Organization, and Management of Realistic, Validated Sentient Opponents (or Other Agents), Networked Based, for Augmenting the World

Where environments require teams of people acting in concert to augment the synthetic environment for participants, for example, teams of well-trained and commanded competitors, the opportunity presents itself for the establishment of network-based teams. These could be widely remoted themselves, even though they would be perceived as being

Page 181

ported into the synthetic environment at a single location. The challenge of establishing these teams is less technical and more organizational, typical of military operations, except in the case where these teams are required to faithfully portray forces of different backgrounds, languages, and value systems. Technology can assist with real-time language generation and translation. Behaving as someone from a different culture is more difficult.

Development of Innovative Applications and Methodologies for Exploiting This Unique Capability

The capabilities created through the design and instantiation of a synthetic environment can be unprecedented, making conventional applications and methodologies obsolete. This task recognizes that research is needed on how to characterize these new capabilities and systematically exploit them.