Ashok Veeraraghavan1, Ramesh Raskar2, and Douglas Lanman2
1Rice University 2MIT Media Lab
Adaptive methods influenced by control theory and information theory can dramatically improve and enlarge today’s imaging and display platforms, enabling capabilities that cannot be achieved with today’s passive approaches. The transition from film to digital has so far been about convenience and programmability, but the remarkable optical and computational flexibility available today means that real-time adaptation is now within reach. Imaging and display devices that rapidly adapt to scene, motion, geometry, viewer, or illumination conditions will result both in performance improvements and in new and novel capabilities hiatherto unexplored. There is an immediate need for new theory that: (1) unifies existing, non-adaptive coded imaging and displays (CID), (2) characterizes the benefits of such adaptation, and (3) derives fundamental information-theoretic limits of adaptive coded imaging and displays (AdaCID). In addition to methods from signal processing (Fourier analysis) and applied mathematics (linear algebra, compressive sensing), this also requires modern methods from control theory (feedback control) and information theory (mutual information).
Intellectual Merit: We aim to develop unified theory and practical designs for Adaptive Coded Imaging and Displays (AdaCID). We will pursue this research agenda with four intellectual thrusts. First, we will establish a theoretical framework for AdaCID using signal and device representations that enable analysis and design. Second, we will demonstrate adaptive coded imaging systems that adapt to scene geometry, motion, and illumination to maximize information throughput. Third, we will demonstrate adaptive and interactive coded 2D/3D displays that adapt in real-time to content, viewer position, and the human visual system to enhance visual appearance and allow intuitive 3D interaction. Finally, we will demonstrate coded feedback projector-camera systems for rapid acquisition of range and material characteristics. Given our promising preliminary results, we believe there is significant potential to develop a broad theoretical framework that will eventually lead to orders of magnitude performance improvements and new abilities in imaging and display systems.
Broader Impact: Progress in AdaCID will have far-reaching impact to diverse applications spanning consumer imaging and displays, machine vision and automation, scientific/medical imaging and displays, robotic surgery, and surveillance and remote sensing. The PIs have several ongoing collaborations in these areas and will work to increase the immediate impact of the research outcomes to these areas. The PIs have a total of about 50 patents to their credit in the area of computational imaging and they have a strong history of technology transfer. Since AdaCID and the broader field of computational imaging and displays is increasingly important, the PIs will (and already do) integrate their research into various vision and imaging courses offered at Rice University and MIT. The PIs will exploit the free, open-licensed Connexions program (at RiceUniversity) and OpenCourseWare (at MIT) and participate in public-domain museum initiatives (at the MIT Museum). This will enable a wider audience than those reached with traditional courses and textbooks. Applications of AdaCID to problems of national interest will also be explored through ongoing collaboration with MIT Lincoln Laboratory.
Keywords: Adaptive, Computational Imaging, 3D-Displays, Coding, Feedback.
2. Publications from this Project
F.-C. Huang, D. Lanman, B. Barsky, and R. Raskar. Correcting Optical Aberrations using Multilayer Displays. ACM SIGGRAPH Asia, December 2012.
G. Wetzstein, D. Lanman, M. Hirsch, W. Heidrich, and R. Raskar. Compressive Light Field Displays. IEEE Computer Graphics and Applications, September 2012.
G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar. Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting. ACM Transactions on Graphics (SIGGRAPH 2012), August 2012.
D. Lanman, G. Wetzstein, M. Hirsch, W. Heidrich, and R. Raskar. Polarization Fields: Dynamic Light Field Display using Multi-Layer LCDs. ACM Transactions on Graphics (SIGGRAPH Asia 2011), December 2011.
A. Velten, T.Willwacher, O.Gupta, A.Veeraraghavan, M.G. Bawendi, R. Raskar, “Recovering three- dimensional shape around a corner using ultrafast time-of-flight imaging”, Nature Communications, 2012.
O.Gupta, A.Velten, T. Willwacher, A. Veeraraghavan, R. Raskar, “Reconstruction of hidden 3D shapes using diffuse reflections”, Optics Express, 2012.
D. Lanman, G. Wetzstain, M. Hirsch, and R. Raskar. Depth of Field Analysis for Multilayer Automultiscopic Displays. In Proc. of the OSA International Symposium on Display Holography (ISDH 2012), June 2012.
G. Wezstein, D. Lanman, M. Hirsch, and R. Raskar. Real-Time Image Generation for Compressive Light Field Displays. In Proc. of the OSA International Symposium on Display Holography (ISDH 2012), June 2012.
M. Hirsch, D. Lanman, G. Wezstein, and R. Raskar. Construction and Calibration of Optically Efficient LCD-based Multi-Layer Light Field Displays. In Proc. of the OSA International Symposium on Display Holography (ISDH 2012), June 2012.
D. Lanman, G. Wetzstein, M. Hirsch, W. Heidrich, and R. Raskar. Beyond Parallax Barriers: Applying Formal Optimization Methods to Multi-Layer Automultiscopic Displays. In Proc. of SPIE Stereoscopic Displays and Applications XXIII, January 2012.
A. Veeraraghavan, A. C. Sankaranarayanan, and R. G. Baraniuk, “Compressive sensing for video applications,” E-Reference Signal Processing, 2012.
J. Holloway, A. C. Sankaranarayanan, A. Veeraraghavan, and S. Tambe, “Flutter shutter video camera for compressive sensing of videos,” in IEEE Intl. Conf. Computational Photography, 2012.
K. Mitra and A. Veeraraghavan, “Light Field Denoising, Light Field Superresolution and Stereo Camera Based Refocussing using a GMM Light Field Patch Prior”, IEEE International Conference on Computer Vision and Pattern Recognition – Workshop on Computational Cameras and Displays, 2012.
N. Shroff, A. Veeraraghavan, Y. Taguchi, O. Tuzel, A. Agrawal, R. Chellappa, “Variable Focus Video: Reconstructing Depth and Video for Dynamic Scenes”, in IEEE Intl. Conf. Computational Photography, 2012.
K. Mitra, A. Veeraraghavan, A. Sankaranarayanan and R. G. Baraniuk, “Towards Compressive Camera Networks”, IEEE Computer Magazine, May 2012.
A Kadambi, R Whyte, A Bhandari, L Streeter, C Barsi, A Dorrington, R. Raskar (2013). Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. 32. (6). ACM Transactions on Graphics (TOG), 32. 167.
A Kadambi, R Whyte, A Bhandari, L Streeter, C Barsi, A Dorrington, R. Raskar (2013). Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. 32. (6). ACM Transaction on Graphics (TOG), 32. 167.
A Mainmone, G Wetzstein, D Lanman, M Hirsch, H Fuchs, R Raskar (2013). Focus 3D: Computational Accommodation Display. NA. ACM Transactions on Graphics, NA. 12.
B Masia, G Wetzstein, C Aliaga, R Raskar, D Gutierrez (2013). Display adaptive 3D content remapping. 37. (8). Computers & Graphics, 37. 983.
S. Tambe, A. Agrawal, A. Veeraraghavan. Towards Motion-Aware Light Field Video for Dynamic Scenes, submitted to IEEE International Conference on Computer Vision (ICCV), 2013.
M. Gupta, A. Agrawal, A. Veeraraghavan, S. G. Narasimhan. A Practicle Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus, Internation Journal of Computer Vision (IJCV), 2013.
A.Veeraraghavan. A Compressive Video Sensing Approach to Tackle Motion Blur, Motion Deblurring: Algorithms and Systems, Cambridge University Press, 2013.
A. Veeraraghavan, A. Sankaranarayanan and R. G. Baraniuk, “Compressive Sensing for Video Applications” a book chapter in book “e-Reference on Signal Processing”, Elsevier, 2013.
M. Hirsch, S. Jayasuriya, S. Sivaranakrishnan, A. Wang, A. Molnar, R. Raskar, G. Wetzstein (2014). A Switchable Light Field Camera Architecture with Angle Sensitive Pixels and Dictionary-based Sparse Coding. IEEE International Conference on Computational Photography, 2014.
A. Maimone, R. Chen, H. Fuchs, R. Raskar, G. Wetzstein (2014).Wide-Field-of-View Compressive Light-Field Display Using a Multilayered Architecture and Viewer Tracking. SID Display Week, 2014.
Bhandari, A., Kadambi, A., Whyte, R., Barsi, C., Feigin, M., Dorrington, A., & Raskar, R. (2014). Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. 39. (6). Optics letters, 39. 1705.
F. Heide, J. Gregson, G. Wetzstein, R. Raskar, W. Heidrich (2014). Compressive multi-mode superresolution display. 22. (12). OSA Optics Express, 22. 14981.
F.C. Huang, G. Wetzstein, B. Barsky, R. Raskar (2014). Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays. 33. (4). ACM Transaction on Graphics (SIGGRPAH), 33. 59.
M. Hirsch, G. Wetzstein, R. Raskar (2014). A Compressive Light Field Projection System. 33. (4). ACM Transactions on Graphics (SIGGRAPH), 33. 58.
J. Holloway, K. Mitra, S. Koppal, A. Veeraraghavan, “Generalized Assorted Camera Arrays: Robust Cross-channel Registration and Applications”, under review in IEEE Transactions on Image Processing (TIP), April 2014.
A. Ito, S. Tambe, K. Mitra, A. Sankaranarayanan, A. Veeraraghavan,“Compressive Epsilon Photography for Post-Capture Control in Digital Imaging”, SIGGRAPH (ACM Transactions on Graphics), 2014.
K. Mitra, O. Cossairt and A. Veeraraghavan, “A Framework for Analysis of Computational Imaging Systems: Role of Signal Prior, Sensor Noise and Multiplexing”, IEEE Transactions on Pattern Analysis and Machine Learning (TPAMI), 2014.
S. Tambe, High resolution light field capture using GMM prior and sparse coding, Master of Science Thesis, ECE, Rice University, 2014.
V. Boominathan, K. Mitra and A. Veeraraghavan, “Improving Resolution and Depth-of-Field of Light Field Cameras Using a Hybrid Imaging System”, IEEE International Conference on Computational Photography (ICCP), 2014.
K. Mitra, O. Cossairt and A. Veeraraghavan, “Can we Beat Hadamard Multiplexing? Data- driven Design and Analysis for Computational Imaging Systems”, IEEE International Conference on Computational Photography (ICCP), 2014.
K. Mitra, O. Cossairt and A. Veeraraghavan, “To Denoise or Deblur: Parameter Optimization for Imaging Systems”, SPIE Electronic Imaging, 2014.
A. Samaniego, V. Boominathan, A. Sabharwal and A. Veeraraghavan, “mobileVision: A Face-mounted, Voice-activated, Non-mydriatic “Lucky” Ophthalmoscope”, Wireless Health, 2014.
Chen R, Maimone A, Fuchs H, Raskar R, Wetzstein G (2014). Wide field of view compressive light field display using a multilayer architecture and tracked viewers.. Journal of the Society for Information Display.
Ye G, Jolly S, Bove Jr VM, Dai Q, Raskar R, Wetzstein G. (2014). Toward BxDF display using multilayer diffraction. ACM Transactions on Graphics (TOG).
Hang Zhao, Boxin Shi, Christy FernandezCull, SaiKit Yeung, Ramesh Raskar (2015). Unbounded High Dynamic Range Photography using a Modulo Camera. ICCP.
Suren Jayasuriya, Adithya Pediredla, Sriram Sivaramakrishnan, Alyosha Molnar, Ashok Veeraraghavan (2015). Depth Fields: Extending Light Field Techniques to Time-of-Flight Imaging.International Conference on 3D Vision.
Ryuichi Tadano, Adithya Pediredla, Ashok Veeraraghavan (2015). Depth Selective Camera: A Direct, On-chip, Programmable Technique for Depth Selectivity in Photography. International Conference on Computer Vision (ICCV).
Huaijin Chen, M. Salman Asif, Aswin C. Sankaranarayanan, Ashok Veeraraghavan (2015). FPA-CS: Focal Plane Array-based Compressive Imaging in Short-wave Infrared. IEEE International Conference on Computer Vision and Pattern Recognition.
Richard Latimer, Jason Holloway, Ashok Veeraraghavan, Ashutosh Sabharwal (2015). SocialSync: Sub-Frame Synchronization in a Smartphone Camera Network. European Conference on Computer Vision.
3. Broader Dissemination
3.1 Short Courses and Tutorials:
PI Veeraraghavan is involved in a novel e-reference article on compressive sensing applied to video applications. The goal of this resource is to create a tutorial with rich nontextual data to guide researchers interested in the general topic.