Augmented & Virtual Reality: Past, Present and Future

The recent SPIE AR VR MR 2020 Conference provided an entertaining look back at the history of virtual and augmented reality headsets and devices, and an exploration of the industry’s future in the many informative technical talks, panel discussions, industry presentations, and demos.

Looking Back: The Museum of AR/VR/MR

Attendees at the San Francisco-based conference could view an extensive “Museum of AR/VR.” On exhibit showcased examples of a Stereoscope from the mid-1800s, the first wired VR gaming headsets of the 1990s, Iron Man’s helmet from the Marvel movie series, and a full array of AR/VR devices from the last 30 years. 

Some “old time” VR systems on the top row, left to right: Stereoscope images ca. 1850, the NASCAR Racer video game headset (1991), and the Toy Quest VR World headset (1997). On the second row, some later systems, left to right: the MyVu (2008)—a first generation iPod connector, an ODG-R6 from now-defunct Osterhaut Design Group (2014), and Ripple AR’s SeeBright (2015), which operates similarly to Google Cardboard. Bottom row, left to right: Samsung Gear (2017), the ACER / Microsoft VR headset (2017), Daqri’s Smart Helmet (2017), and the Iron Man helmet (2018).

The broad array of technologies, optical systems, form factors, and display types on view was testament to the explosion of research and development, experimentation, innovation, and creative problem solving over past decades that has brought the industry to this point. For example, a selection of the display and optics technologies on display included (in no particular order):

This AR/VR development continues today with ongoing exploration of waveguide technologies, holographic 2D and 3D imaging techniques, and display types ranging across OLED, LCOS, projection, and (maybe soon?) microLED. 

Today the AR/VR/MR Industry is approaching an important phase in its development and maturation, which Bernard Kress of Microsoft characterized as the “Age of Reason.”  He perceives that the industry is beginning to converge on technologies that offer the most promising solutions, to rationalize its approaches, and that the center of gravity is moving beyond the early/emerging research and experimentation phase toward some consensus on what it will take to be commercially viable. 

Organizations such as NIST have already started the process of defining standards for near-eye display (NED) devices and the FDA has been involved with assessing potential safety considerations. According to some experts, it may still be premature to lock in a fixed body of standards while so many elements of device hardware, optics, and their application are still undergoing rapid development. Yet we can expect to see the industry move in this direction in the coming years, paralleling the evolutionary path taken by displays, lasers, and other technologies when their markets matured.

Trending Application: Enhancing Vision

One of the interesting trends that emerged from this year’s conference was using augmented reality technologies to help enhance human visual capabilities. Augmented reality means, quite literally, to augment a user’s view of reality by overlaying information (such as text and graphics) onto the background real-world environment. Some innovative new companies are taking augmented reality a step further using it not only to present information over the user’s existing view of reality, but to actually improve that view—enhancing the user’s visual capabilities to better see reality.

For people with visual impairments, the capabilities of AR/MR can be a boon. Many AR/MR devices include systems to “map” the visual environment. Instead of using these maps to integrate displayed information, the information about the real environment can then be presented back to the wearer to help them better “see” the world around them.  

Roughly 75% percent of U.S. adults use some sort of vision correction (eye glasses or contact lenses)¹ for mild to severe visual issues including myopia or hyperopia (near- of far-sightedness), astigmatism, presbyopia (age-related visual degeneration), poor peripheral vision, light sensitivity, loss of night vision, and other conditions and diseases of the eye.

A recent study from the University of Southern California Keck School of Medicine determined that patients with retinitis pigmentosa—an inherited degenerative eye disease characterized by poor peripheral vision and difficulty seeing in low light conditions—could benefit from using adapted AR glasses. The researchers provided test patients with AR glasses that overlaid “objects within a 6-foot wireframe with four bright, distinct colors…providing visual color cues”² that helped the subjects interpret complex or/or dimly lit environments to avoid obstacles. To do this, they used a process called simultaneous location and mapping that allowed the AR glasses to render the 3D structure of a room in real time.  

Meanwhile, coming at the problem from the other side, a team from Stanford University has helped develop a computational display technology that can correct for any near- or far-sightedness (refractive errors) in your vision when looking at a display such as your phone. Instead of having to wear glasses or contact lenses, the display itself would have vision-correcting algorithms to enable a user to view it clearly.
 

Gordon Wetzstein, head of Stanford University’s Computational Imaging Group, delivers a TED talk about display screens that can correct your vision for you.

Prescription AR Glasses

For the many people who already wear some kind of vision correction such as prescription glasses or contact lenses, many smart glasses makers will incorporate adjustments or add-in elements to match a user’s unique prescription. For example, smart glasses makers Vuzix and North both offer customized prescription versions of their eyewear, and Nvidia has announced “Prescription AR” glasses as well.

Vuzix (left) offers to customize it’s Blade^® smart glasses to your prescription (or you can buy a replacement set of prescription lenses to pop into an existing pair of glasses), and North (right) touts that its Focals lenses are custom-built to your prescription.

Replacing Human Sight?

But can these systems go beyond just enhancing people’s existing, if limited, visual capabilities to actually replace vision for people who are blind? At California Institute of Technology, scientists have been working on an approach to help those with visual impairments by using AR and audio cues. In their system “a wearable computer captures video and other data, extracts important scene knowledge, and conveys that to the user in compact form… each object in the environment has a voice and communicates with the user on command.”³ Their solution uses a Microsoft Hololens headset and software they’ve developed called Cognitive Augmented Reality Assistant (CARA).

The Caltech visual-assistive AR solution uses a Microsoft Hololens and CARA software.

Future Trend: MicroLED

Another trend spotlighted at the SPIE AR VR MR Conference was the emergence of microLED technology for AR/VR applications. Despite the challenges still being worked out for cost-effective, consistent microLED fabrication and assembly, may leaders in the AR/VR industry continue to be excited about the potential of this display technology. In fact, Srinivasa Banna, Vice President at Lumileds, went so far as to predict that in the future there will be just two sources of light: the sun, and LEDs.⁴ 

MicroLEDs are seen as particularly useful for AR and MR displays, where their high brightness is an advantage for the contrast needed to view projected images on a transparent medium with the ambient environment still visible in the background. Companies and organizations in the microLED segment including Lumileds, Lumiode, glō, Jasper Display, Mojo Vision, Industrial Technology Research Institute (ITRI), and eLux were on site to share their approaches and latest developments in tackling the issues such as backplanes, mass transfer, uniformity, color, cost, and power efficiency that lie on the path toward mass commercialization. 

Ensuring the Quality of AR/VR/MR Display Devices

For makers of NED devices, ensuring correct function and appearance of displays is critical, and the quality of the user experience is paramount. Color and brightness (luminance) uniformity, pixel defects, image contrast, distortion, and stray light effects can all interfere with device visual performance, functionality, and safety. The Radiant Vision Systems AR/VR Lens has a unique optical design specially engineered for measuring NEDs such as those integrated into AR/VR/MR headsets. 

The lens design simulates the size, position, and field of view of the human eye. The aperture of the AR/VR Lens is located on the front of the lens, enabling positioning of the imaging system’s entrance pupil within NED headsets at the same location as the human eye to capture the full field of view of head-mounted displays (HMDs) (up to 120° horiztonal, 80° vertical). The aperture size of 3.6 mm also matches the size of a human entrance pupil, allowing displays to be measured under the same conditions as they are viewed by a human observer. This ensures that all aspects of the display visible to the user are comprehensively measured to ensure the intended experience.

Paired with a Radiant ProMetric^® Imaging Photometer or Colorimeter, the AR/VR Lens captures high-resolution images of displays for pixel and subpixel measurements to detect defects and anomalies in LED, LCD, OLED, LCOS, DLP and projection displays. Radiant TrueTest™ Software provides the leading display test algorithms, with the capability to sequence tests for rapid evaluation of all relevant display characteristics in a matter of seconds, applying selected tests to a single image or multiple images driven to the headset by the software. Standard tests provided in our TrueTest family of software include display luminance, chromaticity, contrast (clarity), uniformity, mura (blemishes), pixel and line defects, and more. Unique tests for NED display analysis in our TT-ARVR™ test module include:

• Modulation transfer function (MTF) to evaluate image clarity, including line pair analysis and slant-edge testing based on Slant Edge Contrast ISO 12233
• Image Distortion (for distortion characterization)
• Stray light and ghosting effects
• Image Retention/Sticking
• Factory distortion calibration of the lens solution to normalize wide field-of-view images for testing
• Spatial x,y position reported in degrees (°)

With this complete Radiant solution configured for exact application needs, manufacturers can implement an efficient AR/VR measurement method using a single provider with end-to-end support.

A Radiant ProMetric^® Imaging Colorimeter with the AR/VR Lens and TT-ARVR sSoftware from our TrueTest family provides an effective solution for NED device testing.

CITATIONS

1. Vision Council of America
2. Anastasios Nikolas Angelopoulos, Hossein Ameri, Debbie Mitra, Mark Humayun. “Enhanced Depth Navigation Through Augmented Reality Depth Mapping in Patients with Low Vision”, Scientific Reports, 2019; 9 (1) DOI: 10.1038/s41598-019-47397-w
3. Liu, Y., Stiles, N., and Meister, M., “Augmented Reality Powers a Cognitive Assistant for the Blind”, eLife, Vol. 7, Art. No. e37841. DOI: 10.7554/eLife.37841, 2018
4. Banna, Srinivasa, during his presentation “MicroLED Technology for AR/VR Displays” at SPIE Augmented, Virtual, and Mixed Reality Conference, February 3, 2020, San Francisco, CA.