NATIONAL ACADEMY PRESS
2101 Constitution Ave., N.W. Washington, DC 20418
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
The work of the Committee on Virtual Reality Research and Development is supported by Department of the Army Contract No. DAAD05-92-C-0087 issued by the U.S. Aberdeen Proving Ground Support Activity. The views and opinions, and findings contained in this report are those of the author(s) and should not be construed as an official Department of the Army position, policy, or decision, unless so designated by other official documentation.
Library of Congress Cataloging-in-Publication Data
Virtual reality : scientific and technological challenges / Nathaniel I. Durlach and Anne S. Mavor, editors.
"Committee on Virtual Reality Research and Development, Commission on Behavioral and Social Sciences and Education, Commission on Physical Sciences, Mathematics, and Applications, National Research Council."
Includes bibliographical references and index.
1. Human-computer interaction. 2. Virtual reality. I. Durlach, Nathaniel I. II. Mavor, Anne S. III. National Research Council (U.S.). Committee on Virtual Reality Research and Development.
Copyright 1995 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
COMMITTEE ON VIRTUAL REALITY RESEARCH AND DEVELOPMENT
NATHANIEL DURLACH (Chair),
Department of Electrical Engineering and Computer Science, Research Laboratory of Electronics, Massachusetts Institute of Technology
NASA-Ames Research Center, Moffett Field, California
Robert A. Welch Foundation, Houston, Texas
JOHN N. HOLLERBACH,
Department of Biomedical Engineering, McGill University
JAMES R. LACKNER,
Ashton Graybiel Spatial Orientation Lab, Brandeis University
J. MICHAEL MOSHELL,
Institute for Simulation and Training, The University of Central Florida
Department of Computer Science, University of Virginia
RICHARD W. PEW,
BBN Laboratories, Inc., Cambridge, Massachusetts
Virtual Reality Games, Inc., Chapel Hill, North Carolina
Dartmouth Hitchcock Medical Center, Lebanon, New Hampshire
MANDAYAM A. SRINIVASAN,
Department of Mechanical Engineering, Research Laboratory of Electronics, Massachusetts Institute of Technology
JAMES J. THOMAS,
Battelle Pacific Northwest Laboratory, Richland, Washington
ANDRIES VAN DAM,
Computer Science Department, Brown University
NASA-Ames Research Center, Moffett Field, California
School of Computer Science, Carnegie Mellon University
Department of Electrical Engineering and Computer Science, University of California, Berkeley
Department of Computer Science, Naval Postgraduate School
ANNE S. MAVOR, Study Director
HERBERT S. LIN, Senior Staff Officer,
Computer Science and Telecommunications Board
CINDY S. PRINCE, Project Assistant
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council.
The Committee on Virtual Reality Research and Development was established by the National Research Council in 1992 at the request of a consortium of federal government agencies: the Advanced Research Projects Agency, the Air Force Office of Scientific Research, the Human Research and Engineering Directorate of the Army Research Laboratory, the Crew Systems Directorate and the Human Resources Directorate of the Armstrong Laboratory, the Army Natick RD&E Center, the National Aeronautics and Space Administration, the National Science Foundation, the National Security Agency, and the Sandia National Laboratory. As a group, these agencies sought guidance and direction regarding the federal investment in research and development in the area of virtual reality.
This report constitutes the committee's response to the charge to "recommend a national research and development agenda in the area of virtual reality" to guide government research and development over the next generation. Although the charge refers only to virtual reality systems (systems in which the world the human operator interacts with is generated by a computer), the committee has also considered teleoperator systems. Such an extension is required not only for logical and scientific reasons, but also because many of the examples cited in the charge are teleoperator systems.
The purpose of the agenda is to provide a technical rationale to federal agencies for the allocation of their resources in support of research and technology development—and thereby to help define and shape the field. In keeping with this purpose, the recommendations in this report
have been generated by a committee consisting primarily of computer scientists, engineers, and psychologists, some of whom have extensive experience in federal science and technology policy. The committee, in addition to drawing on the expertise of its own members, has made substantial use of advisers and consultants. Overall, this group's expertise is well matched to its task of surveying the scientific and technological state of the field, the potential of current and future technology for improving performance in various application areas, and the types of research and development required to realize this potential. In addition, it is important to note that most members of the committee have a vested interest in seeing the field of virtual reality and synthetic environments flourish because much of their professional work is related to it. Indeed, serious involvement in and knowledge about the field were among the main criteria used to form the committee.
Ultimately, however, the federal research and development agenda must also take account of societal needs, the potential of various research and development accomplishments for fulfilling these needs, and the kinds and magnitudes of attention and support various programs are likely to receive without special efforts. Furthermore, given a research and development program that receives high ratings according to these criteria, it is important to specify the kinds of infrastructure that will facilitate the carrying out of this program. We recognize that the full range of expertise required to determine and judge many of these factors goes well beyond the competencies represented on the committee. In particular, both the formulation and evaluation of societal needs and the specification of appropriate societal infrastructure to ensure efficient performance in carrying out federal research and development programs require inputs from individuals with expertise not only in computer science, engineering, and psychology, but also in sociology, economics, business, social policy, and government.
We have attempted, to the extent possible, to avoid basing our recommendations on assumptions concerning the relative importance of various societal needs. Also, to the extent possible, we have avoided basing our recommendations on assumptions concerning the extent to which and the manner in which given technological advances will influence society—the history of such predictions is extremely humbling. However, given the importance of assigning some sort of priorities to different possible research and development programs (a laundry list of all possible programs would be of little use to anyone), these issues cannot be avoided entirely. What we have tried to do is to make our assumptions explicit. We also suggest that individuals with additional expertise be asked to follow up this work. Finally, with regard to the problem of ensuring appropriate infrastructures to carry out the recommended research
and development programs—a problem that is perhaps more difficult to solve than the problem of selecting the programs themselves—we point out some of the factors that we believe need to be considered in the follow-up work.
This research and development agenda for the federal investment in virtual environments and teleoperation is rooted in a careful review and analysis of the current state of research and technology and of the steps required to reach a point at which significant applications can be fully realized. Our perspective on infrastructure takes cognizance of the need for additional studies of issues related to market forces and societal design as this agenda is further shaped and implemented. In the committee's view, our scientific and technical assessment represents only the first step in a multistage process that would appropriately lead to a more fully developed federal research agenda based on considerations of societal as well as scientific and technical issues.
Although our report is directed primarily to the federal agencies who requested it, we have attempted to make it useful to a wide variety of readers, including students and professionals working in academia and industry. The material is structured in the following way. The executive summary provides a brief synopsis of the main points that can easily be read in a few minutes. The overview is intended to provide a general understanding of the field and of our recommendations; it can be read in a few hours. The remainder of the report provides the detailed analysis as well as numerous bibliographical citations.
As in most reports prepared by large committees, there has been a struggle to find an appropriate balance between coherence and uniformity of style on one hand and free and full expression by individual experts on the other. As is evident when reading the report, different chapters have been prepared by different people. In other words, with the exception of the executive summary and the overview, we have leaned toward full and free expression rather than uniformity. It should also be noted that the differences among the chapters reflect not only the different writing styles of the various contributors but also the different cultures associated with the topics being discussed. For example, even within the single domain of human-machine interfaces, discussions of the visual, auditory, and haptic channels would require rather disparate treatment even if written by the same individual.
Many individuals have made a significant contribution to the committee's thinking and to various sections of this report by serving as presenters, consultants, and advisers. A complete list of contributors and their affiliations is presented in Appendix B. Although all of these individuals provided us with valuable information, a few played a more direct role in the preparation of this manuscript and deserve special mention.
We extend our gratitude to Kurt Akeley for his contribution on computer architecture; to Walter Aviles for his work on visual technology and remote, teleoperated vehicles; to Norman Badler for his contributions on modeling virtual worlds and virtual actors; to Paul DiZio for his work on full-body motion interfaces; to Blake Hannaford and Thomas Sheridan for their discussions of teleoperator systems; to Michael Macedonia for his work on networking; to John Makhoul and Kenneth Stevens for their contributions on speech communication; to Barbara Shinn-Cunningham for her work on the auditory channel; to Thomas Wiegand and Richard Held for their discussions of the status of research on the human visual system; to David Zeltzer for his work on visual technology and computer modeling; and to George Zweig for his sustained interest in our study and his extremely valuable critical comments on a wide variety of crucial issues.
In the course of preparing this report, each member of the committee took an active role in drafting sections of chapters, leading discussions, and reading and commenting on successive drafts. Even so, John Hollerbach and Michael Zyda deserve special acknowledgment for their heroic efforts throughout the study. John Hollerbach assumed major responsibility for the work on telerobotics as well as contributing extensively to several other sections of the report. Michael Zyda took responsibility for obtaining, summarizing, and integrating material on the computer generation of virtual environments and networking. Other committee members contributing work on the computer generation of virtual environments include Steve Bryson for scientific visualization, Randy Pausch for techniques for interacting in three-dimensional environments, Andrew Witkin for modeling software, and Andries van Dam for his overall review and insightful comments. Committee members who contributed their expertise to the human-machine interface sections of the report were James Lackner for whole-body motion and illusions; Elizabeth Wenzel for the auditory channel; and Mandayam Srinivasan for the haptic channel and for the discussion of hardware and software requirements for multimodal interfaces. Those who provided expertise in the various application domains of virtual environments include Norman Hackerman, Michael Moshell, Richard Pew, Warren Robinett, Joseph Rosen, James Thomas, and Eugene Wong.
Staff of the National Research Council made important contributions to our work in many ways. We would like to extend a special note of thanks to Harold Van Cott and Herbert Lin. Harold Van Cott was instrumental in the original thinking about this project and the formation of the committee; he has been of immeasurable assistance throughout the process in providing advice and draft materials. Herbert Lin, a senior staff officer to the committee from the Computer Science and Telecommunications
Board, contributed extensively to the applications section of the report. Further, we would like to express our appreciation to Marjory Blumenthal, director of the board, for her contributions in the early stages of project development, and to Alexandra Wigdor, director of the Division on Education, Labor, and Human Performance, for her valuable insight and support. Thanks are also due to Cindy Prince, the committee's administrative assistant, who was indispensable in organizing meetings, arranging travel, and compiling agenda materials. We are also indebted to Carolyn Sax and Nora Luongo for their work on the manuscript, and to Christine McShane, who edited and significantly improved the report.
To our sponsors we are most grateful for their interest in the topic of this report and for their many useful contributions to the committee's deliberations. We are particularly appreciative of the early efforts of Bernard Corona, of the Human Research and Engineering Directorate, U.S. Army Research Laboratory, and James Jenkins, of the National Aeronautics and Space Administration, which led to the formation of the committee.
Finally, it is a deep pleasure to acknowledge the remarkable feelings that appear to have been generated by this cooperative, interdisciplinary study. These feelings can be best summarized by noting the following two ideas expressed at the last meeting:
Whatever the reception of the report, the field of synthetic environments was significantly advanced by the education of the committee members, along with their consultants and advisers, that took place during the study;
Many of the committee members intend to continue meeting on a regular basis to continue the spirited interdisciplinary dialogue that was initiated during the study.
Nathaniel I. Durlach, Chair
Anne S. Mavor, Study Director
Committee on Virtual Reality Research and Development