Ashok Veeraraghavan¹, Ramesh Raskar², and Douglas Lanman²

¹Rice University                        ²MIT Media Lab

NSF IIS-1116718                   NSF IIS-1116452

1.     Summary

Adaptive methods influenced by control theory and information theory can dramatically improve and enlarge today’s imaging and display platforms, enabling capabilities that cannot be achieved with today’s passive approaches. The transition from film to digital has so far been about convenience and programmability, but the remarkable optical and computational flexibility available today means that real-time adaptation is now within reach. Imaging and display devices that rapidly adapt to scene, motion, geometry, viewer, or illumination conditions will result both in performance improvements and in new and novel capabilities hiatherto unexplored. There is an immediate need for new theory that: (1) unifies existing, non-adaptive coded imaging and displays (CID), (2) characterizes the benefits of such adaptation, and (3) derives fundamental information-theoretic limits of adaptive coded imaging and displays (AdaCID). In addition to methods from signal processing (Fourier analysis) and applied mathematics (linear algebra, compressive sensing), this also requires modern methods from control theory (feedback control) and information theory (mutual information).

Intellectual Merit: We aim to develop unified theory and practical designs for Adaptive Coded Imaging and Displays (AdaCID). We will pursue this research agenda with four intellectual thrusts. First, we will establish a theoretical framework for AdaCID using signal and device representations that enable analysis and design. Second, we will demonstrate adaptive coded imaging systems that adapt to scene geometry, motion, and illumination to maximize information throughput. Third, we will demonstrate adaptive and interactive coded 2D/3D displays that adapt in real-time to content, viewer position, and the human visual system to enhance visual appearance and allow intuitive 3D interaction. Finally, we will demonstrate coded feedback projector-camera systems for rapid acquisition of range and material characteristics. Given our promising preliminary results, we believe there is significant potential to develop a broad theoretical framework that will eventually lead to orders of magnitude performance improvements and new abilities in imaging and display systems.

Broader Impact: Progress in AdaCID will have far-reaching impact to diverse applications spanning consumer imaging and displays, machine vision and automation, scientific/medical imaging and displays, robotic surgery, and surveillance and remote sensing. The PIs have several ongoing collaborations in these areas and will work to increase the immediate impact of the research outcomes to these areas. The PIs have a total of about 50 patents to their credit in the area of computational imaging and they have a strong history of technology transfer. Since AdaCID and the broader field of computational imaging and displays is increasingly important, the PIs will (and already do) integrate their research into various vision and imaging courses offered at Rice University and MIT. The PIs will exploit the free, open-licensed Connexions program (at RiceUniversity) and OpenCourseWare (at MIT) and participate in public-domain museum initiatives (at the MIT Museum). This will enable a wider audience than those reached with traditional courses and textbooks. Applications of AdaCID to problems of national interest will also be explored through ongoing collaboration with MIT Lincoln Laboratory.

Keywords: Adaptive, Computational Imaging, 3D-Displays, Coding, Feedback.

2. Publications from this Project

2012

F.-C. Huang, D. Lanman, B. Barsky, and R. Raskar. Correcting Optical Aberrations using Multilayer Displays. ACM SIGGRAPH Asia, December 2012. 

G. Wetzstein, D. Lanman, M. Hirsch, W. Heidrich, and R. Raskar. Compressive Light Field Displays. IEEE Computer Graphics and Applications, September 2012. 

G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar. Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting. ACM Transactions on Graphics (SIGGRAPH 2012), August 2012.

D. Lanman, G. Wetzstein, M. Hirsch, W. Heidrich, and R. Raskar. Polarization Fields: Dynamic Light Field Display using Multi-Layer LCDs. ACM Transactions on Graphics (SIGGRAPH Asia 2011), December 2011.

A. Velten, T.Willwacher, O.Gupta, A.Veeraraghavan, M.G. Bawendi, R. Raskar, “Recovering three- dimensional shape around a corner using ultrafast time-of-flight imaging”, Nature Communications,  2012.

O.Gupta, A.Velten, T. Willwacher, A. Veeraraghavan, R. Raskar, “Reconstruction of hidden 3D shapes using diffuse reflections”, Optics Express,  2012.

D. Lanman, G. Wetzstain, M. Hirsch, and R. Raskar. Depth of Field Analysis for Multilayer Automultiscopic Displays. In Proc. of the OSA International Symposium on Display Holography (ISDH 2012), June 2012.

G. Wezstein, D. Lanman, M. Hirsch, and R. Raskar. Real-Time Image Generation for Compressive Light Field Displays. In Proc. of the OSA International Symposium on Display Holography (ISDH 2012), June 2012.

M. Hirsch, D. Lanman, G. Wezstein, and R. Raskar. Construction and Calibration of Optically Efficient LCD-based Multi-Layer Light Field Displays. In Proc. of the OSA International Symposium on Display Holography (ISDH 2012), June 2012.

D. Lanman, G. Wetzstein, M. Hirsch, W. Heidrich, and R. Raskar. Beyond Parallax Barriers: Applying Formal Optimization Methods to Multi-Layer Automultiscopic Displays. In Proc. of SPIE Stereoscopic Displays and Applications XXIII, January 2012.

A. Veeraraghavan, A. C. Sankaranarayanan, and R. G. Baraniuk, “Compressive sensing for video applications,”  E-Reference Signal Processing, 2012.

J. Holloway, A. C. Sankaranarayanan, A. Veeraraghavan, and S. Tambe, “Flutter shutter video camera for compressive sensing of videos,” in IEEE Intl. Conf. Computational Photography, 2012.

K. Mitra and A. Veeraraghavan, “Light Field Denoising, Light Field Superresolution and Stereo Camera Based Refocussing using a GMM Light Field Patch Prior”, IEEE International Conference on Computer Vision and Pattern Recognition – Workshop on Computational Cameras and Displays, 2012.

N. Shroff, A. Veeraraghavan, Y. Taguchi, O. Tuzel, A. Agrawal, R. Chellappa, “Variable Focus Video: Reconstructing Depth and Video for Dynamic Scenes”, in IEEE Intl. Conf. Computational Photography, 2012.

K. Mitra, A. Veeraraghavan, A. Sankaranarayanan and R. G. Baraniuk, “Towards Compressive Camera Networks”, IEEE Computer Magazine, May 2012.

A. Ito, S. Tambe, K. Mitra, A. Sankaranarayanan, A. Veeraraghavan,“Compressive Epsilon Photography for Post-Capture Control in Digital Imaging”, SIGGRAPH (ACM Transactions on Graphics), 2014.

K. Mitra, O. Cossairt and A. Veeraraghavan, “A Framework for Analysis of Computational Imaging Systems: Role of Signal Prior, Sensor Noise and Multiplexing”, IEEE Transactions on Pattern Analysis and Machine Learning (TPAMI), 2014.

S. Tambe, High resolution light field capture using GMM prior and sparse coding, Master of Science Thesis, ECE, Rice University, 2014. 

V. Boominathan, K. Mitra and A. Veeraraghavan, “Improving Resolution and Depth-of-Field of Light Field Cameras Using a Hybrid Imaging System”, IEEE International Conference on Computational Photography (ICCP), 2014.

K. Mitra, O. Cossairt and A. Veeraraghavan, “Can we Beat Hadamard Multiplexing? Data- driven Design and Analysis for Computational Imaging Systems”, IEEE International Conference on Computational Photography (ICCP), 2014.

K. Mitra, O. Cossairt and A. Veeraraghavan, “To Denoise or Deblur: Parameter Optimization for Imaging Systems”, SPIE Electronic Imaging, 2014.

A. Samaniego, V. Boominathan, A. Sabharwal and A. Veeraraghavan, “mobileVision: A Face-mounted, Voice-activated, Non-mydriatic “Lucky” Ophthalmoscope”, Wireless Health, 2014.

Chen R, Maimone A, Fuchs H, Raskar R, Wetzstein G (2014). Wide field of view compressive light field display using a multilayer architecture and tracked viewers.. Journal of the Society for Information Display.

Ye G, Jolly S, Bove Jr VM, Dai Q, Raskar R, Wetzstein G. (2014). Toward BxDF display using multilayer diffraction. ACM Transactions on Graphics (TOG).

2015

Hang Zhao, Boxin Shi, Christy Fernandez­Cull, Sai­Kit Yeung, Ramesh Raskar (2015). Unbounded High Dynamic Range Photography using a Modulo Camera. ICCP.

Suren Jayasuriya, Adithya Pediredla, Sriram Sivaramakrishnan, Alyosha Molnar, Ashok Veeraraghavan (2015). Depth Fields: Extending Light Field Techniques to Time-of-Flight Imaging.International Conference on 3D Vision.

Ryuichi Tadano, Adithya Pediredla, Ashok Veeraraghavan (2015). Depth Selective Camera: A Direct, On-chip, Programmable Technique for Depth Selectivity in Photography. International Conference on Computer Vision (ICCV).

Huaijin Chen, M. Salman Asif, Aswin C. Sankaranarayanan, Ashok Veeraraghavan (2015). FPA-CS: Focal Plane Array-based Compressive Imaging in Short-wave Infrared. IEEE International Conference on Computer Vision and Pattern Recognition.

Richard Latimer, Jason Holloway, Ashok Veeraraghavan, Ashutosh Sabharwal (2015). SocialSync: Sub-Frame Synchronization in a Smartphone Camera Network. European Conference on Computer Vision.

3. Broader Dissemination

3.1 Short Courses and Tutorials: 

1. Short-course on Compressive Video Sensing: PI Veeraraghavan co-organized a short-course on compressive sensing and its broader implications for sensing visual signal at CVPR 2012.

(http://www.ece.rice.edu/~as48/VideoCS2012Tutorial/)

2. PI Veeraraghavan was involved in organizing a short course on manifold theoretic methods for computer vision. The short-course was taught at CVPR 2012.

(http://stat.fsu.edu/~anuj/CVPR_Tutorial/ShortCourse.htm)

3.  Short-course on “Computational Plenoptic Imaging”: PI Raskar co-organized a short course on computational plenoptic imaging and its broader implications for advancing the state-of-the-art in consumer and scientific photography at ACM SIGGRAPH 2012. 

(http://web.media.mit.edu/~gordonw/courses/ComputationalPlenopticImaging/)

4. Short-course on “Computational Displays”: The team members co-organized a short course on computational displays and their broader implications for advancing the state-of-the-art in consumer and scientific display technology at ACM SIGGRAPH 2012. 

(http://web.media.mit.edu/~gordonw/courses/ComputationalDisplays/)

5. Workshop on “Computational Cameras and Displays”: PI Raskar co-organized a one-day workshop on computational imaging and display systems and their broader implications for the scientific imaging community. The workshop was co-located with IEEE CVPR 2012. This workshop brought attention to the emerging topic of adaptive, coded imaging systems within the academic imaging community.

(http://computationalcamerasanddisplays.media.mit.edu/)

3.2 Surveys:

PI Veeraraghavan is involved in a novel e-reference article on compressive sensing applied to video applications. The goal of this resource is to create a tutorial with rich nontextual data to guide researchers interested in the general topic.