Modeling and simulation technology has become increasingly important to both the entertainment industry and the U.S. Department of Defense (DOD). In the entertainment industry, such technology lies at the heart of video games, theme park attractions and entertainment centers, and special effects for film production. For DOD, modeling and simulation technology provides a low-cost means of conducting joint training exercises, evaluating new doctrine and tactics, and studying the effectiveness of new weapons systems. Both the entertainment industry and DOD are aggressively pursuing development of distributed simulation systems that can support Internet-based games and large-scale training exercises. These common interests suggest that the entertainment industry and DOD may be able to more efficiently achieve their individual goals by working together to advance the technology base for modeling and simulation. Such cooperation could take many forms, including collaborative research and development projects, sharing research results, or coordinating ongoing research programs to avoid unnecessary duplication of effort.
This report summarizes the results of a workshop, convened by the National Research Council's Computer Science and Telecommunications Board, that brought together members of the entertainment and defense industries to discuss research interests in modeling and simulation. The workshop revealed several areas in which the entertainment industry and DOD have common interests (see Box ES.1). This report examines the research challenges in these areas with an eye toward identifying
Technologies for Immersion
• Image generationgraphics
computers capable of generating complex visual images.
• Higher-bandwidth networksto
allow faster communication of greater amounts of information among
Standards for Interoperability
• Virtual reality transfer
protocolto facilitate large scale networking of
distributed virtual environments
• Adaptabilitydevelopment of
computer-generated characters that can modify their behavior
automatically over time
Aggregation/disaggregationthe capability to aggregate
smaller units into larger ones and to disaggregate them back into
smaller ones without sacrificing the fidelity of a simulation or
frustrating attempts at interoperability.
Tools for Creating Simulated Environments
• Database generation and
manipulationtools for managing and storing information in
large databases, to allow rapid retrieval of information, feature
extraction, creation, and simplification.
areas for possible cooperation. The report does not attempt to provide answers to existing research questions, nor does it necessarily recommend that cooperative efforts be initiated in the areas discussed. Such decisions need to be made on a case-by-case basis by the individual organizations that might participate.
Cooperative endeavors between DOD and the entertainment industry will face many obstacles. As the workshop revealed, numerous cultural barriers divide the entertainment industry and DOD, and few mechanisms exist to facilitate the sharing of information. If such obstacles can be overcome, these differences could be a source of strength, ensuring a complementarity of interests, capabilities, and approaches that might benefit both communities. Already, the U.S. Marine Corps is evaluating commercial computer games for training purposes, the Army is considering use of game machines as personal training units, and members of the Air National Guard are evaluating the use of commercial flight simulator programs to supplement standard training regimens. Such initiatives suggest that cooperation is possible but only begin to hint at the kinds of benefits that might be achieved through greater collaboration in research, which is the main subject of this report.
Other tasks will also need to be undertaken jointly in order to ensure
the future strength of both communities. DOD and the entertainment industry will need to foster the establishment and expansion of education programs to train students in the technical and nontechnical underpinnings of modeling, simulation, and virtual environments. They also will need to ensure the viability of the university research base, which not only will produce these students but will also generate many of the technical advances upon which future entertainment and defense systems will be built.
Toward a Research Agenda
Workshop discussions revealed several research areas that are of interest to both the entertainment industry and the defense modeling and simulation community: technologies for immersion, networked simulation, interoperability, computer-generated characters, and hardware and software tools for creating synthetic environments. Each of these areas demonstrates sufficient overlap in interest by DOD and the entertainment industry to suggest that common work may be possible, although additional study may be required to fully detail the scale and scope of such work. While both DOD and the entertainment industry have similar interests at the research and technology levels, the applications and end products into which research results will be incorporated may differ in fundamental ways, reinforcing the notion that the most effective forms of cooperation will derive more from early stages of research than from sharing products. Emphasizing cooperation in research over sharing of products also helps avoid many of the concerns about intellectual property and proprietary interests that could impede collaboration between the entertainment industry and DOD.
Technologies for Immersion
Both the entertainment industry and DOD are interested in developing immersive systems that allow participants (whether game players or soldiers) to enter and navigate simulated environments. For DOD such systems can be used to train groups of combatants or, increasingly, individual combatants for particular missions when access to the actual location is either hazardous or just not possible. They can also be used to create virtual prototypes of military systems that designers can walk through and visualize. For the entertainment industry such systems are the basis for virtual reality (VR) experiences being incorporated into location-based entertainment centers, theme parks, and video game centers. Immersive technologies are also finding their way into home applications, prompted by the greater availability of three-dimensional (3D)
graphics on personal computers and reductions in the cost of peripheral devices, such as joysticks and head-mounted displays.
Immersive environments benefit from a wide range of technologies that provide the sensory cues necessary for participants to perceive their environments. The most basic of these are image generators that create 3D visual displays of the environment itself. Other technologies include locomotion platforms and unobtrusive bodysuits that allow participants to walk through virtual environments and track their movements and interactions. Such bodysuits are also used to build keyframes for animated characters in film and video game productions.1 DOD is pursuing these applications as part of its dismounted infantry program, and the Defense Modeling and Simulation Office recently funded relevant work at the U.S. Army's Simulation, Training, and Instrumentation Command. It is also funding work on generating other sensory stimuli in virtual environments: auditory, olfactory, and tactile. Such work is also the focus of the commercial VR industry and parts of the entertainment industry.
The entertainment industry and DOD may also be able to benefit from their complementary approaches to selective fidelity. Both communities have learned how to boost the fidelity (or accuracy) of some parts of a simulated environment and limit the fidelity of others while creating an engaging simulation experience. Whereas DOD has tended to emphasize the fidelity of interactions between objects in a simulated environment (using science-based models), the entertainment industry has tended to promote visual fidelity and uses principles of good storytelling to help participants suspend their disbelief about the reality of a synthetic experience (whether a VR attraction or a film). Additional work in these areas, and sharing of approaches, may allow both communities to create more engaging simulated experiences while minimizing the technical demands placed on the system itself.
The entertainment industry and DOD face similar challenges in creating networking technologies capable of supporting distributed simulations. DOD has already demonstrated the capability to link thousands of participants into a single training exercise and is working on systems that would engage tens of thousands of participants. Internet-based game companies have recently begun to move fast-action video games onto the Internet and are looking for ways to increase the number of simultaneous players from 10 or 20 to hundreds or perhaps thousands. As the number of participants in Internet-based games and the military's joint training exercises grows, improvements to simulation networks will be needed to
ensure that communications between participants can be transmitted in a reliable and timely fashion.
Several technologies can support such requirements. First, DOD and the entertainment industry can pursue ways of expanding the bandwidth of simulation networks to allow more information to be transmitted more quickly. Many of the technologies for doing so will derive from technical advances made by the communications and networking industries, on which both DOD and the entertainment industry rely. Other approaches also are being pursued by DOD and the Internet community, including multicast and software-based area-of-interest managers. These techniques can be used to minimize message traffic across the network by directing copies of a single message to only those recipients who have an interest in seeing it. DOD and the games industry are also developing ways of compensating for the latency of distributed networks through algorithms for predicting the future location and state of other objects and for synchronizing events among different participants.
Standards for Interoperability
Both DOD and the entertainment industry are developing architectures and protocols for linking distributed simulation systems. DOD has promoted the development of standards for distributed interactive simulation (DIS) that specify the protocols such systems should follow. It has also developed a high-level architecture for military simulation systems that allows different simulator platforms to interoperate and enables reuse of existing simulation programs. Commercial industry, in contrast, has developed standards for enabling different types of computer systems to play the same game. Rather than adopt a common set of protocols for allowing games to work with one another, game companies have each tended to develop their own proprietary protocols that allow copies of their own games to play against each other but do not allow them to work with another company's games. Some of these protocols derive from DIS standards but are modified to boost the performance of a particular game. It is not clear that DOD and the entertainment industry will adopt common standards on a wide scale as long as proprietary interests continue to dominate protocol decisions.
Nevertheless, both DOD and the entertainment industry will need to solve common problems in developing their network architectures and protocols; common research into interoperability standards might be beneficial. A careful and considered joint research program is needed that studies the issues involved in designing a common scalable network software architecture capable of supporting large numbers of players across wide-area networks.2 Components of this re-
search effort include Web-based interoperability standards that would allow the linking of distributed virtual environments, architectures for "plug-and-play" interoperability that allow different simulation systems to interoperate, and network software architectures for maintaining persistent universessimulated worlds that continue to exist and evolve even after an individual participant leaves the simulation. Little fundamental research is being conducted in these areas by either DOD or the entertainment industry; rather, both communities are concentrating on developing networked simulation systems without addressing the basic issues of the network software architecture.
The term computer-generated characters refers to the broad range of entities in a simulated environment (people, tanks, aircraft, etc.) whose behaviors are controlled wholly or in part by a computer.3 They include the computerized opponents in computer chess games, digital actors that appear in films and television, and simulated enemy forces in military training exercises. Computer-generated characters are a part of virtually every major DOD simulation and all video games in which players compete against the computer instead of, or in addition to, other players. They attempt to reproduce realistic intelligent human behavior that provides participants with a compelling simulated experience.
Additional research would benefit computer-generated characters in both entertainment and defense applications. Gilman Louie, of Spectrum HoloByte Inc., estimates that games companies allocate up to two-thirds of their development efforts to programming autonomous characters that cannot be reused in other games. DOD, while creating more advanced computer-generated characters, tends to program behaviors into entities, such as tanks crews, that operate according to strict rules of engagement derived from military doctrine. These entities cannot be easily modified or reprogrammed; nor can they accurately portray the behaviors of individual soldiers on a battlefield. Both DOD and the entertainment industry would like to develop computer-generated characters that have adaptable behaviors and can learn from experience. Some research is ongoing, under DOD sponsorship, to apply techniques of artificial intelligence and genetic algorithms to computer-generated characters so that they can achieve these capabilities. Other work is needed to develop adequate models of individual human behaviors and realistic representations of humans in virtual environments. Significant work is needed to develop the capability to define computer-generated characters at a high level of abstraction that will facilitate their reuse.
Tools for Creating Simulated Environments
Workshop participants identified low-cost, easy-to-use hardware and software tools for creating simulated environments as a critical need for entertainment and defense applications of modeling and simulation.4 Such tools are needed to enable the rapid creation and manipulation of large databases of information describing terrain, buildings, 3D objects, and dynamic features of virtual environments, and to facilitate the compositing of such disparate imagery into a unified simulated world. Although the entertainment industry purchases a wide variety of graphics hardware and software from established vendors, it spends little on improving these tools and instead concentrates on short-term solutions to devise advanced special effects. Many existing systems are expensive and difficult to learn. Additional research is needed to create more interactive tools that allow designers to develop simulated environments using input devices other than keyboard and mouse. In one system described at the workshop, an immersive VR system was developed to allow filmmakers to manipulate computer imagery in real time. Such techniques may have broader applicability in entertainment and defense.
Toward Greater Cooperation
Promoting cooperation between DOD and the entertainment industry in modeling and simulation will require both communities to overcome cultural barriers that have, to date, isolated them from one another and limited the flows of information between them. Differences in business models will need to be overcome if joint research is to be achieved. At the same time, DOD and the entertainment industry will need to ensure that they take the necessary actions, both individually and jointly, to ensure a continued supply of good people and good ideas for future modeling and simulation efforts. Educational programs are needed to train students in the technical and nontechnical skills that are important to creating effective simulated environments. Fundamental research programs are needed to generate ideas and explore new technologies that are broadly relevant to modeling and simulation. DOD and the entertainment industry will need to solicit additional input from the academic research community to better understand how to accomplish these tasks.
Information Sharing and Technology Transfer
The workshop conducted as part of this study was unique in that it brought together two communities that traditionally have shared little information and transferred little technology between them. Its success
attests to the vision of DOD in identifying a potential basis for cooperation and the efforts of the committee to create an atmosphere in which mutually beneficial exchanges of information could occur. For the most part, DOD and the entertainment industry are two different cultures, with different languages and separate communities of researchers and managers. Few opportunities exist for promoting information exchanges between the two communities. The ones that do existmostly government efforts to promote commercialization of technologies developed by federal laboratorieshave been relatively unsuccessful in creating bridges to the entertainment industry.
Workshop participants suggested that additional mechanisms are necessary for promoting information exchanges on modeling and simulation technology that would benefit both DOD and the entertainment industryeven if they do nothing more than identify research problems that have already been solved. These could take the form of formal collaborative arrangements between entertainment companies and DOD, efforts by individual firms to supply modeling and simulation technology to both communities, or joint research endeavors mediated by a university research center. Experiments are needed to test the viability and effectiveness of these different arrangements. Less formal mechanisms also could be effective. Conferences are the primary mechanism for information exchanges today, but DOD and entertainment industry representatives tend to attend separate conferences. Some progress could be made by encouraging cross-attendance at major conferences within each community or by cross-fertilizing boards of relevant technical and planning groups and establishing a separate symposium to specifically explore topics of interest to both communities. Greater use of the Internet and World Wide Web also might facilitate greater communication.
At the workshop, representatives of the entertainment industry and DOD noted an apparent shortage of talented people with the broad range of skills needed to develop models and simulations. Both communities increasingly have trouble finding programmers with experience in content development and the technical problems associated with multiplayer/multiprocessor games and simulations. Both DOD and the entertainment industry are seeking people who are visually literate: people who are skilled in generating economical, high-quality graphics displays and have a good understanding of human perception so that they can create worlds that have the desired effect on those who experience them.
Additional efforts will be needed to enhance educational programs for visual literacy. At present, only a small number of U.S. universities
offer interdisciplinary programs that combine technical and artistic studies. Expansion of such programs will require the creation of interdisciplinary degree programs and reward systems that encourage faculty members to pursue such endeavors. The research community will need to articulate a research agenda that incorporates the perspectives of technical and nontechnical disciplines. Workshop participants believed that DOD and the entertainment industry could use existing funding mechanisms as a means for encouraging the creation of such programs without incurring additional costs.
Preserving the Research Base
Ensuring an adequate supply of new ideas and technologies for future modeling and simulation efforts requires continued support for relevant fundamental research. University research is especially important because it concentrates more heavily on basic than applied research and has the added benefit of educating students, who then disseminate new knowledge throughout the research community and industry when they graduate.
Workshop participants concurred that over the past 20 years the nature of research funding in such fields as computer graphics and networking has changed. University researchers have less freedom to select and pursue research areas they deem interesting. In part because of growing demands for accountability, government-funded projects are often more results-oriented than they used to be, and government agencies are under greater pressure to demand specific project goals and delivery dates for each task. The implication is not only a change in the kind of research many investigators perform, but also a reduction in the quantity of research conducted. Most university researchers reported that they now spend less time on research and more time filling out grant applications and seeking funding. Determining the most suitable method for government support of university research is beyond the scope of this study,5 but members of the university research community present at the workshop expressed considerable concern about the current trend in government funding.
To date, industry funding has not compensated for changes in federal research funding. Although industry contributions to university research have grown over the past decade, they are still small. Moreover, as in many other industries, entertainment companies tend not to conduct long-term basic research, largely because of short planning horizons and the inability to fully appropriate the results of fundamental research.6 Entertainment companies tend to obligate most of their research and development expenditures to technical problems related to a particular film
or game release rather than to longer-term issues related to future needs. Industry-sponsored research is more closely tied to particular product needs, potentially limiting the scope of inquiry and raising concerns that valuable new information will not be widely disseminated.
Other forms of industry support for university research also appear to be changing. Several workshop participants observed that computer firms do not donate as much equipment to university laboratories as in the past. University representatives noted that they rely heavily on such donations to acquire state-of-the-art equipment for research projects and education. Their inability to attract such donations affects not only the quality of research but also the training of students.
Recent trends in federal and industry funding for university research in modeling and simulation mirror those of other scientific and technical fields. National support for research and development (R&D) is undergoing a period of transformation. Interest in reducing the federal budget deficit and in realigning defense needs to match the challenges of the post-Cold War environment will continue to put pressure on federal funding for R&D. Increased competition seems to be changing the nature and structure of industrial R&D. Such issues must be addressed at the national level to ensure the continued viability of the technology base for modeling and simulation.
1. The need for research into lightweight tracking technology is fully described in another National Research Council report. See Virtual Reality: Scientific and Technological Challenges, Nathaniel I. Durlach and Anne S. Mavor, eds., National Academy Press, Washington, D.C., 1995.
2. The term network software architecture encompasses both network architecture and software architecture to indicate that the problems of network bandwidth and limited processor cycles must be solved together to achieve scalability.
3. The terms computer-generated forces, autonomous forces, semiautonomous forces, and autonomous agents all refer to computer-generated characters. The first three terms are widely used throughout the defense community; the term autonomous agents refers to a broader class of entities used for seeking relevant information on computer networks as well as generating computerized opponents for game players.
4. The need for such tools is also described in Virtual Reality: Scientific and Technological Challenges, note 1 above.
5. The National Research Council's Computer Science and Telecommunications Board is conducting two other studies that may more fully investigate this topic. The first will look retrospectively at the role of government, industry, and universities in key innovations in information technology. The second will look prospectively at institutional arrangements for ensuring the continued leadership of the U.S. information technology industry.
6. The difficulty of appropriating profits from investments in basic research has been a long-standing issue in economics and management. For greater discussion of this topic, see Teece, David J., 1988, "Profiting from Technological Innovation: Implications for Integra-
tion, Collaboration, Licensing, and Public Policy," in Readings in the Management of Innovation, Michael L. Tushman and William L. Moore, eds., Ballinger Publishing Company, Cambridge, Mass., pp. 621-647; and Levin, Richard et al., 1987, "Appropriating the Returns from Industrial Research and Development," Brookings Papers on Economic Activity, No. 3, pp. 783-831.