Home   |   News   |   Join Us   |   People   |   Projects   |   Publications   |   Talks   |   Courses

HR3D: High-Rank 3D Display
using Content Adaptive Parallax Barriers

Figure 1: HR3D display with content-adaptive parallax barriers. We show that light field display using dual-stacked LCDs can be cast as a matrix approximation problem, leading to a new set of content-adaptive parallax barriers. (Left, Top) A 4D light field, represented as a 2D array of oblique projections. (Left, Bottom) A dual-stacked LCD displays the light field using content-adaptive parallax barriers, confirming both vertical and horizontal parallax. (Middle and Right) A pair of content-adaptive parallax barriers, drawn from a rank-9 decomposition of the reshaped 4D light field matrix. Compared to conventional parallax barriers, with heuristically-determined arrays of slits or pinholes, content adaptation allows increased display brightness and refresh rate while preserving the fidelity of projected images.

Photo Credit: MIT Media Lab, Camera Culture Group

Abstract

Today's 3D display are not only light deficient, but rank deficient. We have developed a 3D display that eliminates the need for special glasses, while solving both light and rank deficiency. Until now, the commercial potential of glasses-free 3D displays, particularly those based on liquid crystal displays (LCDs), has been primarily limited by decreased image resolution and brightness compared to systems employing special eyewear.

In the Camera Culture group at the MIT Media Lab, we have found a way to increase the brightness and resolution of LCD-based, glasses-free 3D displays using a method they call Content-Adaptive Parallax Barriers. We call our new display technology High-Rank 3D or HR3D, since our display is capable of displaying a full-resolution light field.

Figure 2: Increasing display brigtness and refresh rate. The HR3D display, using content-adaptive barriers, is compared to time-shifted barriers with the exposure normalized so the relative image brigtness is consistent with observation. The input 3x3 light field is compressed by our display for T<9, where T is the number of time-multiplexed sub-frames to be shown (i.e. the rank of the displayed light field). Experimental photographs (fourth column) are compared to predicted images. All images correspond to the central view of the light field (looking straight at the screen) for the light field shown in Figure 1.

Photo Credit: MIT Media Lab, Camera Culture Group

Current commercially-available, LCD-based 3D displays use a simple concept, first proposed by Frederic Ives in 1903; Ives achieved the illusion of depth by introducing the notion of a parallax barrier. In his design an array of slits is placed slightly in front of a normal 2D display. The slits ensure that each eye sees different regions of the underlying display and, therefore, different images. This method is still widely-employed today, despite its limitations. In particular, the slits required by Ives's parallax barriers function by blocking rays of light and, as a result, significantly reduce the resolution and brightness of the underlying display.

Content-adaptive parallax barriers improve upon Ives's method by allowing the spacing and orientation of slits to be optimized to transmit as much light as possible, while retaining the fidelity of the projected 3D images. The key insight is to realize that the pattern of slits must be changed depending on the 3D scene being projected. By eliminating the earlier fixed, heuristic design, content-adaptive parallax barriers can significantly increase the brightness of glasses-free 3D displays based on LCD technology. Furthermore, by using recent high-speed LCDs, the researchers have shown that full-resolution 3D images can be achieved. Content-adaptive parallax barriers are well-suited to mobile devices, optimizing the brightness of displays without reducing battery life.

Figure 3: (Left) 3D display prototype constructed using off-the-shelf parts. Two LCD panels are disassembled and stacked on top of one another, creating a "dual-stacked LCD". This device can use either conventional or content-adaptive parallax barriers to achieve glasses-free 3D display. Visible are the (a) rear LCD, (b) spacer, (c) front LCD, and (d) polarizing film.
(Right) A side view of the prototype, showing the two LCD panels. A simple mechanical design allows the gap between the panels to be adjusted, controlling the field of view in which a viewer can be located and perceive the illusion of depth.
Photo Credit: MIT Media Lab, Camera Culture Group

Please read the FAQ for more information.

Paper

• High Resolution (18MB)
  
• Low Resolution (1.4MB)

Citation

Douglas Lanman, Matthew Hirsch, Yunhee Kim, Ramesh Raskar. Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization. Proc. of SIGGRAPH Asia 2010 (ACM Transactions on Graphics 29, 6), 2010.

Bibtex

@article{Lanman_et_al_2010,
  author = {Lanman, Douglas. and Hirsch, Matthew and Kim, Yunhee and Raskar, Ramesh},
  title = {Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization},
  journal = {ACM Trans. Graph.},
  volume = {29},
  number = {6},
  year = {2010},
  issn = {0730-0301},
  pages = {163:1--163:10},
  doi = {http://dx.doi.org/10.1145/1882261.1866164},
  publisher = {ACM},
  address = {New York, NY, USA}
}

Video

Slide Set

We will be giving a technical paper talk on this project at SIGGRAPH Asia 2010 in Seoul, South Korea, on Friday, 17 December 2010, 4:15PM local time. The talk, in the Imaging Hardware section, is in the COEX convention center, room E1-E4.

After the talk, we will post our slides here.

Brief Technical Description

How can we create a 3D display that does not require viewers to wear special glasses?

Thin displays that present an illusion of depth have become a driving force in the electronics and entertainment industries. Binocular depth cues are achieved by presenting different images to each eye. Current-generation 3D displays require special eyewear, such as LCD shutters or polarizing filters. glasses can be eliminated by modifying the optical design of existing displays. Historically, glasses-free 3D displays require adding optically-attenuating masks, known as parallax barriers, or refracting lens arrays to a 2D display. We optimize the performance of 3D displays built by stacking a pair of modified LCD panels. We introduce adaptive masks, optimized for each multi-view video frame that increase brightness and frame rate compared to conventional parallax barriers. These patterns can extend the battery life of next-generation mobile 3D displays and enable full motion parallax—allowing a viewer to tilt his head and still perceive the illusion of depth.

Read our PDF Overview handout.

Figure 4: Dual-stacked LCDs are considered as general spatial light modulators that act in concert to recreate a target light field by attenuating rays emitted by the backlight. (Left) A conventional parallax barrier configuration for glasses-free 3D display and (Right) a content-adaptive parallax barrier display that allows increased image resolution and brightness.

Illustration Credit: MIT Media Lab, Camera Culture Group

Figure 5: Rank constraints for parallax barriers. (Left) Conventional parallax barriers approximate the light field matrix (center) as the outer product of mask vectors (above and to the left). The resulting rank-1 approximation accurately reproduces the circled elements (corresponding to the central views in Figure 4). Note that most columns are not reconstructed, reducing display resolution and brightness. Periodic replicas of the central views are created outside the circled regions. (Middle Left) Time-shifted parallax barriers achieve higher-rank reconstructions by integrating a series of rank-1 approximations, each created by a single translated mask pair. (Middle Right) Content-adaptive parallax barriers increase display brightness by allowing both masks to exhibit non-binary opacities. Here a rank-1 approximation is demonstrated using a single mask pair. (Right) Rank-T approximations are achieved using temporal multiplexing of T content-adaptive parallax barriers. In practice, the light field will be full rank without enforcing periodic replication (as created by conventional parallax barriers). As a result, we do not constrain rays (shown in red) outside the central view in our optimization to find the masks.

Illustration Credit: MIT Media Lab, Camera Culture Group

Figure 6: A thin, dual-layer display (e.g., a dual-stacked LCD) allows depth perception without special eyewear. Multi-view content is rendered or photographed and represented as a 4D light field. Content-adaptive parallax barriers are obtained by applying non-negative matrix factorization to the input light field. the resulting set of content-adaptive parallax barriers are displayed using the dual-layer display, leading to increased brightness and frame rate compared to conventional parallax barriers.

Illustration Credit: MIT Media Lab, Camera Culture Group

Our prototype HR3D display relies on an optimization technique to arrive at the optimal set of images and masks to display for a desired output light field. The content creator decides what image quality, brightness, and framerate to target, and the optimization will try to generate a set of mask pairs to achieve this result. Because there is no way to produce negative light, the masks generated by our optimization must be non-negative. Therefore, we use a Non-negative Matrix Factorization (or NMF) in our optimization.

Figure 7: Approximation error as a function of NMF iteration. The average PSNR of the reconstruction is plotted for a rank-9 decomposition of the light field shown in Figure 1.

Illustration Credit: MIT Media Lab, Camera Culture Group

One of the primary benefits of our work is the ability to make an automultiscopic barrier display with full horizontal and vertical parallax brighter. The HR3D display can trade brightness for image fidelity better than pinholes. Here we characterize that trade.

Figure 8: Approximation error as a function of gain in brightness. The average PSNR of the reconstruction is plotted for a rank-9 decomposition of the light fields shown in Figure 1. For time-shifted parallax barriers, transmission can be increased either by enlarging slits/pinholes or by brightening the rear LCD. The latter is considered here, however simulations of the former also confirm time-shifted parallax barriers cannot achieve a PSNR greater than 15 dB when increasing brightness by a factor greater than two.

Illustration Credit: MIT Media Lab, Camera Culture Group

Another benefit of the HR3D display is that content-adaptive barriers allow the rank of the displayed light field to vary. This could be achieved with a scanned pinhole by simply removing some of the view positions, but the effect of this has more serious consequences for reconstruction PSNR than our reduced rank reconstruction. HEre, we characterize the how reconstruction PSNR changes with decomposition rank.

Figure 9: Approximation error as a function of decomposition rank. The average PSNR of the reconstruction is plotted for a rank-T decomposition of the light fields in Figure 1. For 3×3 views, a theoretical PSNR of infinity is achieved with 9 time-shifted conventional parallax barriers. In comparison, content-adaptive barriers achieve higher PSNR than conventional barriers when fewer frames are used. Experimental and predicted images with varying degrees of compression are shown in Figure 2.

Illustration Credit: MIT Media Lab, Camera Culture Group

Existing Techniques

Existing 3D displays come in many varieties. Displays are broadly classified as either glasses-bound, those which rely on the viewer wearing a pair of special glasses, or unencumbered, those which allow the viewer to walk up and perceive a 3D image without any extra equipment.

Glasses-bound displays tend to also be stereoscopic, meaning that they provide only two distinct views – one for each eye. Unencumbered displays, on the other hand, tend to be multiscopic, meaning that many images are displayed at once. (Think of stereo- vs. multi-channel audio). A stereo 3D display will produce strange, non-physical distortion when the viewer moves his head from side to side. In contrast, a multiview, or multiscopic display, will allow the viewer to look around objects when moving his head, in closer agreement to the way we see the real world. The figure below, taken from our course, Build Your Own 3D Display, shows how many popular technologies fit into the taxonomy.

Figure 10: A taxonomy of 3D display devices, taken from our course, Build Your Own 3D Display.

Illustration Credit: Matthew Hirsch, Douglas Lanman. BYO3D Course, SIGGRAPH 2010

Please see our course, Build Your Own 3D Display for further details about the current state-of-the-art in 3D display. In particular, check out the first half of Section 2, Representation and Display, in which we expand upon the above taxonomy.

To build our prototype HR3D display, we disassembled two Viewsonic VX2265wm 120Hz LCD panels. The front panel was completely disassembled, removing both the front diffusing polarizing layer and rear transparent polarizing layer. The images below show the sequence of steps to disassemble the display. The most difficult step is the final step of removing the polarizing films from the LCD glass. After carefully pulling up the films, we used a pencil eraser and acetone to remove the adhesive.

Acknowledgements

We thank the SIGGRAPH Asia 2010 technical paper reviewers for insightful feedback, the Camera Culture and Information Ecology groups at the MIT Media Lab for their support, and Samsung Electronics for its sponsorship. Szymon Jakubczak contributed to an earlier version of this work presented at SIGGRAPH 2010. Thomas Baran and Gabriel Taubin contributed useful discussions. Douglas Lanman is supported by NSF Grant CCF-0729126, Yunhee Kim by NRF of Korea Grant 2009-352-D00232, and Ramesh Raskar by an Alfred P. Sloan Research Fellowship.

Contact

Technical Details
Douglas Lanman, Postdoctoral Associate, MIT Media Lab
dlanman (at) media.mit.edu

Press
Alexandra Kahn, Senior Press Liaison, MIT Media Lab
akahn (at) media.mit.edu or 617/253.0365

Recent projects in the Camera Culture group

Cameras	6D Display Lighting and Viewpt aware displays	Bokode Long Distance Barcodes	BiDi Screen Touch+3D Hover on Thin LCD	NETRA Cellphone based Eye Test

Home   |   News   |   Join Us   |   People   |   Projects   |   Publications   |   Talks   |   Courses