Frontiers in Virtual Reality Headsets

WARREN HUNT
Oculus Research

Technological advances are transforming virtual and augmented reality from science fiction to consumer products. When widely deployed, these technologies have the potential for major impact on entertainment, culture, and commerce. This article provides a basic overview of virtual (VR) and augmented (AR) realities, describes some potential high-impact applications, discusses the effort required to achieve these technologies, and explains an aspect of the human visual system that presents challenges for augmented and virtual reality.

WHAT IS VIRTUAL REALITY?

Virtual reality is any simulation created by a computer, presented to a person, and perceivable as real. Current VR devices consist of a head-mounted display, often with headphones. These devices block out a person’s hearing and vision of the real world; ultimately, VR technology will encompass more senses.

Although it’s not yet possible to display real-time VR content that’s indistinguishable from reality, it is already possible to produce an experience referred to as “presence”—the sense that a person has, in fact, been transported somewhere else. This transportation isn’t always conscious: a person experiencing presence may logically know they’re wearing a headset, but have measurable reactions to virtual objects or threats, such as fear in response to virtual heights.

A variety of companies sell commercial head-mounted displays. Several, such as Sony, have announced future products, and a number of startups are producing prototypes.

GOING BEYOND: AUGMENTED REALITY

While virtual reality aims to completely override human senses, augmented reality systems aim to combine both real-world and virtual stimuli. Devices such as Microsoft’s HoloLens use a see-through display to overlay a virtual world onto the real world.

The blending of real and virtual content can augment everything from daily life, such as virtual name tags or line-item reviews superimposed on a restaurant menu, to complex specialized tasks, such as the overlay of MRI data directly onto a patient in an operating room.

WHY VIRTUAL AND AUGMENTED REALITY?

Virtual and augmented reality have a large range of potential uses. The most obvious and immediate are currently commercially available: entertainment, movies, 360 video, and games. These technologies also have numerous additional applications, including social and business communication, journalism, e-commerce, and education.

Virtual reality has been a futuristic technology for a long time now, and many factors suggest that it is now ready to succeed in the main stream. Moore’s law has enabled powerful graphics hardware to render high-definition resolutions at frame rates sufficient for a commercially viable visual experience. Moreover, the rise of smartphones has made very high density organic LED (OLED) display panels and low-latency accelerometers, two key components in high-quality VR, inexpensive and widely available.

WHAT DOES IT TAKE TO MAKE VR/AR?

Building a virtual or augmented reality system is a massive multidisciplinary effort. At the heart of this effort are perceptual scientists: they define the requirements for matching and driving the human perceptual system in order to make VR believable and prevent users from experiencing motion sickness or any other form of discomfort from the VR experience. Such requirements include audio/visual fidelity, latency limits, and tracking accuracy, among others.

Building a head-mounted display requires optical, electrical, and mechanical engineering, understanding of displays and tracking technology, and software expertise in graphics, sound, computer vision, and user interaction. These components must be implemented with a great degree of care and coordination.

Furthermore, achieving high quality in graphical and computer vision systems requires extreme amounts of computing power. Virtual and augmented reality systems currently produce a rather crude representation of the world and a resolution far inferior to what humans can perceive. Achieving a virtual system

that is indistinguishable from reality could consume many orders of magnitude more computing power.

A STABLE VIRTUAL WORLD

To achieve a compelling VR experience—and one that minimizes the risk of motion sickness—the virtual world must consistently appear stable to the user. While traditional displays, e.g., a TV or desktop monitor, tend to stay in one place, VR head-mounted displays are worn on a user’s head and often move very quickly. This rapid display movement can cause artifacts that can break the sense of immersion or, worse, make the user physically uncomfortable. As with most of the requirements for VR headsets, stability requirements are driven by the human perceptual system.

Vestibulo-Ocular Reflex

The human visual system has one of the fastest reflexes in the human body. The vestibulo-ocular reflex (VOR) stabilizes human vision during head motion, and does so with a latency of 3 neurons (about 10 ms). This reflex is responsible for turning the eyes to compensate for head motion and provide a stable retinal image during typical head movement.

Because of this reflex, the human visual system expects a stable, crisp image during head rotation even when the screen (attached to the head) is moving at 300 degrees per second. The reflex actively destabilizes the view presented on a head-mounted display, causing it to slide across the retina and blur as VOR makes the eye counterrotate.

Visual Artifacts from Displays

Judder

Judder is the blurring effect caused by the VOR and a static head-mounted display. Most displays illuminate pixels for the duration of a frame (about 17 ms at 60Hz or 11 ms at 90Hz). When a user’s head turns at 300 degrees/second, 11 ms corresponds to 3.3 degrees, and during this movement the pixel becomes smeared across that angle. To address this, VR displays use “low persistence” mode and are activated for only 1–2 ms out of each frame, displaying black the remainder of the time. This prevents the smearing artifact, but leads to a dimmer display and, under certain conditions, a strobe effect.

Latency

Display latency can cause severe artifacts. Even at a modest 100 degrees per second, a system latency of 30 ms would cause the world to lag by a full 3 degrees behind a viewer’s gaze. The constant lag causes a noticeable “swimming” artifact that can be disturbing and lead to motion sickness. Better algorithms, advances in graphics hardware, and careful orchestration between applications and display hardware can reduce these effects.

Rolling

A rolling display illuminates pixels as they arrive over the wire, rather than all at the same time. Old CRT TVs and most OLED phones have rolling displays: they start by illuminating the top row of the screen, then the next, and so on, until the whole screen has been illuminated. Alternatively, a global display reads the entire frame before displaying every pixel simultaneously.

Each approach has pros and cons. The global display adds significant latency: rather than displaying pixels immediately, all pixels wait until the last one to arrive before illuminating. With rolling displays, if users move their eyes during the display update, the image appears to distort or shear depending on the direction of movement. Compensation for this artifact requires the integration of high-quality eye tracking.

Achieving the latency reduction allowed by rolling displays while minimizing artifacts is an open research problem.

CONCLUSION

Virtual and augmented reality are nascent technologies, but have the promise of dramatic worldwide impact. Continued improvements to displays, graphics, and tracking, coupled with enhanced understanding of the human perceptual system, will enable the realization of more applications. A variety of companies are investing heavily in this space in anticipation of its potential impact.

A proper mix of technology and funding is shaping up to make for a very exciting future for virtual reality!