Take Your Glasses Off and See

Can we create a display that adapts itself to improve one’s eyesight? Top figure compares the view of a 2.5-diopter farsighted individual in regular and tailored displays. We use currently available inexpensive technologies to warp light fields to compensate for refractive errors and scattering sites in the eye.

Our new display uses measurements of refractive errors (see alsoNETRA) and cataract maps (see also CATRA) to free the viewer from needing wearable optical corrections when looking at displays. It supports nearsightedness, farsightedness, astigmatism, presbyopia(reading glasses), coma, keratoconus, other higher-order aberrationsand any type of cataracts. Knowledge of the eye conditions allows traditional displays go beyond an individual’s visual acuity, presenting images that are in focus even without wearing corrective eyeglasses. The hardware is the same of glasses-free 3D displays (dual stack of LCDs), but in higher resolution. We propose their use in daily tasks where using eyeglasses are unfeasible or inconvenient (e.g., on head-mounted displays, e-readers, as well as for games); when a multi-focus function is required but undoable (e.g., driving for farsighted individuals, checking a portable device while doing physical activities for presbyopic); or for correcting the visual distortions produced by high-order aberrations that eyeglasses are not able to.

Videos

Download the video here (60MB).

Publications

• Vitor F. Pamplona
  Interactive Measurements and Tailored Displays for Optical Aberrations of the Human Eye 
  PhD Thesis 2012. 
   Draft Version.
• Vitor F. Pamplona, Manuel M. Oliveira, Daniel G. Aliaga, Ramesh Raskar.
  Tailored Displays to Compensate for Visual Aberrations.
  ACM SIGGRAPH 2012. ACM Trans. on Graph. 31(4). Pages 87:1-12

High Resolution (16MB)

Low Resolution (1.4M)

•  Vitor F. Pamplona, Manuel M. Oliveira.
  Method and Aparatus for Displays that Compensate for Visual Aberrations.
  BR / INPI.

SIGGRAPH Slide Set

Brief Technical Description

The letter G (a) on the retina of a 2D-coma, +5D-myopia (c), and cataract affected (d) subject. The holography G is virtualized at many depths (16cm < j < 25cm) from the eye. (b) shows how the image reaches the subject’s retina. (e) and (f) shows our computations for the light-field display. Giant ray circle reflects a 5mm-diameter pupil. Notice how the letter is deformed behind each section. Greenish circles in (e) and (f) highlight the cataract-avoiding feature of our algorithm.

We introduce tailored displays that enhance visual acuity by decomposing virtual objects and placing the resulting anisotropic pieces into the subject’s focal range. Our tailoring process uses aberration andscattering maps to account for refractive errors and cataracts. It splits an object’s light field into multiple instances that are each in-focus for a given eye subaperture. Their integration onto the retina leads to a quality improvement of perceived images when observing the display with naked eyes. The use of multiple depths to render each point of focus on the retina creates multi-focus, multi-depth displays. User evaluations and validation with modified camera optics are performed.

We call tailoring the process of adapting a light field to compensate for an individual’s inability to focus. We pre-warp the light field to counter act the distortion on the subject’s eyes. It is performed in two main steps: (i) pairing light-field rays and retinal positions (R) to associate a raw intensity to each ray; and (ii) normalizing retinal “pixels”. Given as input an expected image to be received by the retina and wavefront andcataract maps of the subject’s eye, the method produces a light field to be shown on a specified display. Our approach can be described as the projection of depth-dependent anisotropic patterns according to the spatially-distributed optical aberrations of the eye. The depth-dependent patterns are “anisotropic images” virtually placed at the right point in focus for a given optical power in a section for the cornea. Because these images are placed at multiple depths to create a single image in focus, we say our system has a multi-depth feature. Our method makes sure the depth-dependent patchs are seen only through eye sub-apertures with given refractive powers. Light paths that go through opacities or unpredictable scattering sites, such as cataracts, are avoided. The final result is a light field to be displayed at a given distance from the eye.

Acuity enhancement options range from simple eyeglasses to optical replacements and expensive relays for adaptive optics. They are grouped according to the correction bearer. Eyeglasses with simple, bifocal, and multi-focal lenses, contact lenses, LASIK and cataract surgeries can be used to enhance visual acuity. However, all of them require wearing prosthesis or making incisions in the eye. Multi-focus displays and holograms can enhance visual acuity by projecting images on the subject’s range of accommodation. These techniques, however, do not account for the subject’s individual variability. This paper proposes a new type of display that adjusts itself to compensate the subject’s eye refractive errors and avoids light scattering materials on the optical path, such as cataracts. The display is to some extent similar to a still-under-research adaptive-optics-based contact lenses, but applied to the device instead of the eye.

Potential Impact

Cataracts and refractive errors are the main causes of loss of visual acuity and leading causes of avoidable blindness worldwide. Approximately two billion people need eyeglasses (WHO – Vision 2020 Report) and the prevalence of both conditions are expected to grow with the increasing longevity and heavy near work, such as the use of electronic gadgets. For instance, in urban areas of east Asia, 80-90% of children completing high school are now myopic. People with absence of refractive errors and cataracts – i.e., normal vision – have visual acuity close to 1 arc minute (the smallest discernible feature of a typical eye). Theoretical limits on the human foveal vision are, however, found to be between 0.6 and 0.25 arc minutes. By increasing the optical quality of the image reaching the retina, tailored displays could potentially double an emmetropic’s visual performance.

Input images (a) and how they are perceived on a tailored display by a farsighted subject (c). Column (b) shows how the subject sees the respective image on (a), with same size, at the same distance in a standard monitor.

We believe the evolution of this technology will lead to new tailorable user experiences. Our work is inspired by consumer light-field cameras that have realized that exceeding sensor resolution is redundant and can instead be exploited for refocusing. We realize that current trend in display resolution will also create redundant angular resolution for human consumption and it can instead be used to provide new functionality (e.g., vision corrected displays). There are several immediate usages for our work on today’s products, including head-mounted displays, digital wrist devices, tracking and monitoring tools, small-screen cell phones, and music players. All these can be efficiently built with the inexpensive LCD panels used in our prototypes. We propose its usage in all daily tasks where eyewear is not convenient. Checking the time or speed while running is hard for farsighted individuals, who do not carry reading glasses during these activities. When applied to phones and wrist watches, tailored displays could remove the need to carry reading glasses and the annoying act of putting them on and off just to check why the phone is beeping. Head-mounted displays could provide improved vision and create multi-focus environments inside the subject’s accommodation range without moving parts. Tailoring displays have potential to affect several activities where a multi-focus function is required but undoable, such as in tailoring the dashboard of the car to avoid reading glasses for farsighted individuals. Finally, for people with high-order aberrations or cataracts (which cannot be corrected for with eyeglasses), tailoring displays are an interesting alternative to surgeries (which may not be recommended due to patients heart conditions, etc).

We hope the proposed method contributes for future multi-focal techniques that avoid the mismatch between convergence and accommodation on parallax barriers. Tailored displays could diminish eye strain problems of current displays. Other conditions, such as color vision deficiencies (recoloring), retinal displacement and detachment (sensor warping) could be included in the tailoring pipeline. Eye tracking systems and physiologically based models for the eye can provide real-time data, such as the distance eye-display, eye angle and pupil size for the best experience on using our displays. Additional research is required for applying tailored displays in multi-user interactive environments (e.g., several eyes looking at a public tailored display) and the use of multiple points of view to maximize the search for the best viewing angle. We hope the future of glasses-free 3D displays also accounts for user’s eyeglasses. Color calibration for the wavelength profiles of red, green and blue channels would enhance the image being displayed. Interactive measuring techniques for refractive error can be improved by applying tailoring to distort their displayed patterns on the fly. A worldwide used cloud-based medical record system that keeps tracking of users’ eye optical properties could potentially make tailored displays ubiquitous. A wavelength-dependent tailoring process could create new insights on the eye accommodation behavior for natural scenes when refractive variations among wavelengths are close to null. Convergence-based 3D displays with multi-focus and tailoring features can lead to a new ultra-resolution vision-enhanced 3D technology.

The ultimate tailored display would equalize a designed user experience among an audience. Each individual intakes and interprets sensory stimuli in slightly different ways. The computer’s ability to enhance an individual’s focusing dexterity could also be applied to hearing, taste, smell, touch, temperature, and time. The individually-enhanced stimuli would compensate for variations in one’s ability to accurate sense them. Imagine how valuable medical record systems will be when every product experience becomes tailored for their users.

Related Projects

• Vitor’s PhD. Thesis
• This Paper (SIGGRAPH’12): Tailored Displays to Compensate for Visual Aberrations
• CATRA (SIGGRAPH’11): Cataract Assessment Maps using Light-field Displays
• NETRA (SIGGRAPH’10): Refractive Tests using Light-field Displays
• Pupil and Iris Light Reflex Models (SIGGRAPH’10)

Contact

Technical Details
Vitor Pamplona, M.Sc., EyeNetra
vitor (at) vitorpamplona.com

Acknowledgments

Vitor and Manuel acknowledge CNPq-Brazil fellowships 142563/2008-0, 308936/2010-8, 480485/2010-0, and 308936/2010-8. Daniel is supported by NSF CNS 0913875 and Ramesh by Alfred P Sloan Foundation fellowship, DARPA Young Faculty Award and Media Lab consortium sponsors. iPad photo by Di B’edard, and video voiceover by Cody Sumter.