National Academies Press: OpenBook

Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium (2017)

Chapter: Frontiers in Virtual Reality Headsets - Warren Hunt

« Previous: Computational Near-Eye Displays: Engineering the Interface to the Digital World - Gordon Wetzstein
Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×

Frontiers in Virtual Reality Headsets

WARREN HUNT
Oculus Research

Technological advances are transforming virtual and augmented reality from science fiction to consumer products. When widely deployed, these technologies have the potential for major impact on entertainment, culture, and commerce. This article provides a basic overview of virtual (VR) and augmented (AR) realities, describes some potential high-impact applications, discusses the effort required to achieve these technologies, and explains an aspect of the human visual system that presents challenges for augmented and virtual reality.

WHAT IS VIRTUAL REALITY?

Virtual reality is any simulation created by a computer, presented to a person, and perceivable as real. Current VR devices consist of a head-mounted display, often with headphones. These devices block out a person’s hearing and vision of the real world; ultimately, VR technology will encompass more senses.

Although it’s not yet possible to display real-time VR content that’s indistinguishable from reality, it is already possible to produce an experience referred to as “presence”—the sense that a person has, in fact, been transported somewhere else. This transportation isn’t always conscious: a person experiencing presence may logically know they’re wearing a headset, but have measurable reactions to virtual objects or threats, such as fear in response to virtual heights.

A variety of companies sell commercial head-mounted displays. Several, such as Sony, have announced future products, and a number of startups are producing prototypes.

Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×

GOING BEYOND: AUGMENTED REALITY

While virtual reality aims to completely override human senses, augmented reality systems aim to combine both real-world and virtual stimuli. Devices such as Microsoft’s HoloLens use a see-through display to overlay a virtual world onto the real world.

The blending of real and virtual content can augment everything from daily life, such as virtual name tags or line-item reviews superimposed on a restaurant menu, to complex specialized tasks, such as the overlay of MRI data directly onto a patient in an operating room.

WHY VIRTUAL AND AUGMENTED REALITY?

Virtual and augmented reality have a large range of potential uses. The most obvious and immediate are currently commercially available: entertainment, movies, 360 video, and games. These technologies also have numerous additional applications, including social and business communication, journalism, e-commerce, and education.

Virtual reality has been a futuristic technology for a long time now, and many factors suggest that it is now ready to succeed in the main stream. Moore’s law has enabled powerful graphics hardware to render high-definition resolutions at frame rates sufficient for a commercially viable visual experience. Moreover, the rise of smartphones has made very high density organic LED (OLED) display panels and low-latency accelerometers, two key components in high-quality VR, inexpensive and widely available.

WHAT DOES IT TAKE TO MAKE VR/AR?

Building a virtual or augmented reality system is a massive multidisciplinary effort. At the heart of this effort are perceptual scientists: they define the requirements for matching and driving the human perceptual system in order to make VR believable and prevent users from experiencing motion sickness or any other form of discomfort from the VR experience. Such requirements include audio/visual fidelity, latency limits, and tracking accuracy, among others.

Building a head-mounted display requires optical, electrical, and mechanical engineering, understanding of displays and tracking technology, and software expertise in graphics, sound, computer vision, and user interaction. These components must be implemented with a great degree of care and coordination.

Furthermore, achieving high quality in graphical and computer vision systems requires extreme amounts of computing power. Virtual and augmented reality systems currently produce a rather crude representation of the world and a resolution far inferior to what humans can perceive. Achieving a virtual system

Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×

that is indistinguishable from reality could consume many orders of magnitude more computing power.

A STABLE VIRTUAL WORLD

To achieve a compelling VR experience—and one that minimizes the risk of motion sickness—the virtual world must consistently appear stable to the user. While traditional displays, e.g., a TV or desktop monitor, tend to stay in one place, VR head-mounted displays are worn on a user’s head and often move very quickly. This rapid display movement can cause artifacts that can break the sense of immersion or, worse, make the user physically uncomfortable. As with most of the requirements for VR headsets, stability requirements are driven by the human perceptual system.

Vestibulo-Ocular Reflex

The human visual system has one of the fastest reflexes in the human body. The vestibulo-ocular reflex (VOR) stabilizes human vision during head motion, and does so with a latency of 3 neurons (about 10 ms). This reflex is responsible for turning the eyes to compensate for head motion and provide a stable retinal image during typical head movement.

Because of this reflex, the human visual system expects a stable, crisp image during head rotation even when the screen (attached to the head) is moving at 300 degrees per second. The reflex actively destabilizes the view presented on a head-mounted display, causing it to slide across the retina and blur as VOR makes the eye counterrotate.

Visual Artifacts from Displays

Judder

Judder is the blurring effect caused by the VOR and a static head-mounted display. Most displays illuminate pixels for the duration of a frame (about 17 ms at 60Hz or 11 ms at 90Hz). When a user’s head turns at 300 degrees/second, 11 ms corresponds to 3.3 degrees, and during this movement the pixel becomes smeared across that angle. To address this, VR displays use “low persistence” mode and are activated for only 1–2 ms out of each frame, displaying black the remainder of the time. This prevents the smearing artifact, but leads to a dimmer display and, under certain conditions, a strobe effect.

Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×

Latency

Display latency can cause severe artifacts. Even at a modest 100 degrees per second, a system latency of 30 ms would cause the world to lag by a full 3 degrees behind a viewer’s gaze. The constant lag causes a noticeable “swimming” artifact that can be disturbing and lead to motion sickness. Better algorithms, advances in graphics hardware, and careful orchestration between applications and display hardware can reduce these effects.

Rolling

A rolling display illuminates pixels as they arrive over the wire, rather than all at the same time. Old CRT TVs and most OLED phones have rolling displays: they start by illuminating the top row of the screen, then the next, and so on, until the whole screen has been illuminated. Alternatively, a global display reads the entire frame before displaying every pixel simultaneously.

Each approach has pros and cons. The global display adds significant latency: rather than displaying pixels immediately, all pixels wait until the last one to arrive before illuminating. With rolling displays, if users move their eyes during the display update, the image appears to distort or shear depending on the direction of movement. Compensation for this artifact requires the integration of high-quality eye tracking.

Achieving the latency reduction allowed by rolling displays while minimizing artifacts is an open research problem.

CONCLUSION

Virtual and augmented reality are nascent technologies, but have the promise of dramatic worldwide impact. Continued improvements to displays, graphics, and tracking, coupled with enhanced understanding of the human perceptual system, will enable the realization of more applications. A variety of companies are investing heavily in this space in anticipation of its potential impact.

A proper mix of technology and funding is shaping up to make for a very exciting future for virtual reality!

Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×
Page 13
Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×
Page 14
Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×
Page 15
Suggested Citation:"Frontiers in Virtual Reality Headsets - Warren Hunt." National Academy of Engineering. 2017. Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium. Washington, DC: The National Academies Press. doi: 10.17226/23659.
×
Page 16
Next: First-Person Computational Vision - Kristen Grauman »
Frontiers of Engineering: Reports on Leading-Edge Engineering from the 2016 Symposium Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume presents papers on the topics covered at the National Academy of Engineering's 2016 US Frontiers of Engineering Symposium. Every year the symposium brings together 100 outstanding young leaders in engineering to share their cutting-edge research and innovations in selected areas. The 2016 symposium was held September 19-21 at the Arnold and Mabel Beckman Center in Irvine, California. The intent of this book is to convey the excitement of this unique meeting and to highlight innovative developments in engineering research and technical work.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!