Fall 2008: MAS 963(H) Computational Camera and Photography

Class Times:	Fridays 1:30-4:30pm
Units:	6-3-3
Location:	E15-283A (Roth Room)
Instructor:	Ramesh Raskar (raskar(at)media.mit)
Office Hours:	By appointment (Room #324)

Course description

Keywords: Signal processing, Applied optics, Computer graphics and vision, Electronics, Art, and Online photo collections

A computational camera attempts to digitally capture the essence of visual information by exploiting the synergistic combination of task-specific optics, illumination, sensors and processing. We will discuss and play with thermal cameras, multi-spectral cameras, high-speed, and 3D range-sensing cameras and camera arrays. We will learn about opportunities in scientific and medical imaging, mobile-phone based photography, camera for HCI and sensors mimicking animal eyes.

We will learn about the complete camera pipeline. In several hands-on projects we will build several physical imaging prototypes and understand how each stage of the imaging process can be manipulated.

We will learn about modern methods for capturing and sharing visual information. If novel cameras can be designed to sample light in radically new ways, then rich and useful forms of visual information may be recorded -- beyond those present in traditional protographs. Furthermore, if computational process can be made aware of these novel imaging models, them the scene can be analyzed in higher dimensions and novel aesthetic renderings of the visual information can be synthesized.

In this couse we will study this emerging multi-disciplinary field -- one which is at the intersection of signal processing, applied optics, computer graphics and vision, electronics, art, and online sharing through social networks. We will examine whether such innovative camera-like sensors can overcome the tough problems in scene understanding and generate insightful awareness. In addition, we will develop new algorithms to exploit unusual optics, programmable wavelength control, and femto-second accurate photon counting to decompose the sensed values into perceptually critical elements.

Format

The course will consist of lectures, 5 project assignments, one mid-term exam and a final project. The emphasis will be on hardware as well as software hands-on projects and we will progressively build the camera pipeline. We will use several camera elements such as optical elements (lenses, prisms, apertures, masks), light sources (programmable LEDs and projectors), sensors (high speed, thermal, multispectral, range-sensing) in our projects. However, this is not an optics class. The goal is to learn and build novel imaging devices. We will have a few guest talks by experts in the field.

Participants

The course is intended for students with interest in algorithmic and technical aspects of imaging and photography. Successful research in imaging requires a solid understanding in algorithms as well as technologies.

Prerequisites

Familiarity with imaging, camera techniques, applied optics, linear algebra and signal processing will be helpful but not necessary. We try to keep the mathematical prerequisites to a minimum, but we will introduce material from broad areas at a fast pace.

Grading

The credit weights are as follows: projects 40%, final project 40% and mid-term exam 20%. To receive credit, you must attend regularly, complete the project assignment and develop a software or hardware prototype for final project.

Schedule

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.

For Class notes and details, please follow the Stellar site. It will be updated regularly.

	Date	Topic (tentative)	Notes	Reading and Assignments	Links
1	Sept 5th	Introduction and Fast forward preview of all topics	[PPT]
2	Sept 12th	Modern Optics and Lenses, Ray-matrix operations
3	Sept 19th	Virtual Optical Bench, Lightfield Photography, Fourier Optics, Wavefront Coding
4	Sept 26th	Digital Illumination, Hadamard Coded and Multispectral Illumination
5	Oct 3rd	Emerging Sensors: High speed imaging, 3D range sensors, Femto-second concepts, Front/back illumination, Diffraction issues
6	Oct 10th	Beyond Visible Spectrum: Multispectral imaging and Thermal sensors, Fluorescent imaging, 'Audio camera'
7	Oct 17th	Image Reconstruction Techniques, Deconvolution, Motion and Defocus Deblurring, Tomography, Heterodyned Photography, Compressive Sensing
8	Oct 24th	Cameras for Human Computer Interaction (HCI): 0-D and 1-D sensors, Spatio-temporal coding, Frustrated TIR, Camera-display fusion
9	Oct 31st	Useful techniques in Scientific and Medical Imaging: CT-scans, Strobing, Endoscopes, Astronomy and Long range imaging
10	Nov 7th	Mid-term Exam
11	Nov 14th	Optics and Sensing in Animal Eyes. What can we learn from successful biological vision systems? Mobile Photography, Video Blogging, Life logs and Online Photo collections
12	Nov 21st	No Class
13	Nov 28th	Thanksgiving Holiday (No Class)
14	Dec 5th	Final Projects

Thanks

The course material has been prepared with slides, discussions and other contributions from many people. Only some of them are listed below and my apologies. Thank you all !
Jack Tumblin, Northwestern University, Shree Nayar, Columbia University, Amit Agrawal, MERL, Marc Levoy, Bennet Wilburn, Stanford U., Alyosha Efros, CMU, Steve Seitz, UW, Irfan Essa, Georgia Tech, Fredo Durand, MIT, Jingyi Yu, Delaware, Aseem Agrawala, U of Washington, Paul Debevec, USC, Todor Georgiev, Adobe, Hendrik Lensch, MPI.

I also want to thank many who have donated additional equipment, cameras, sensors, optics, software etc.: Fatih Porikli, Amit Agrawal, MERL, James Davis, UC of Santa Cruz, 3DV, Eddy Talvala and Andrew Adams, Stanford.

Reading List

A list of suggested readings will be provided for each class.

Final Projects

Projects ideas.

Computational Photography links

Siggraph Course Notes, Raskar and Tumblin [Additional Material]

Computational Photography: Mastering New Techniques for Lenses, Lighting, and Sensors: 2008, A K Peters, Publishers

Symposium on Computational Photography and Video Cambridge, MA (May 2005)
Fredo Durand's list of useful links and Computational Photography course