Computational Photography

Combining smart optics, sensors and processing to enable novel imaging applications.

With advances in sensors and computational power, cameras coupled with computation offer us new possibilities which were not possible with traditional film based cameras. Computational photography is emerging as a new field combining computer vision, graphics, optics and imaging to overcome the limitations of current cameras.

A key aspect is to design novel sensors for specific applications that go beyond the traditional capturing of the scene as a regular grid of pixel intensities. Such sensors, for example, include coding and modulation strategies along dimensions of space, time, angle and/or wavelength. The imaging group at MERL has been at the forefront of developing fundamental technologies for computational photography. The flutter shutter (coded exposure) camera and mask-based heterodyne light field camera are few of the key inventions of our group.

  • Researchers

  • News & Events

    •  NEWS   MERL presenting 9 papers at ICASSP 2018
      Date: April 15, 2018 - April 20, 2018
      Where: Calgary, AB
      MERL Contacts: Petros Boufounos; Takaaki Hori; Toshiaki Koike-Akino; Jonathan Le Roux; Dehong Liu; Hassan Mansour; Philip Orlik; Pu (Perry) Wang
      Research Areas: Signal Processing, Speech & Audio, Computational Sensing, Digital Video
      Brief
      • MERL researchers are presenting 9 papers at the IEEE International Conference on Acoustics, Speech & Signal Processing (ICASSP), which is being held in Calgary from April 15-20, 2018. Topics to be presented include recent advances in speech recognition, audio processing, and computational sensing. MERL is also a sponsor of the conference.

        ICASSP is the flagship conference of the IEEE Signal Processing Society, and the world's largest and most comprehensive technical conference focused on the research advances and latest technological development in signal and information processing. The event attracts more than 2000 participants each year.
    •  
    •  TALK   Theory and Applications of Sparse Model-Based Recurrent Neural Networks
      Date & Time: Tuesday, March 6, 2018; 12:00 PM
      Speaker: Scott Wisdom, Affectiva
      MERL Host: Jonathan Le Roux
      Research Areas: Multimedia, Speech & Audio
      Brief
      • Recurrent neural networks (RNNs) are effective, data-driven models for sequential data, such as audio and speech signals. However, like many deep networks, RNNs are essentially black boxes; though they are effective, their weights and architecture are not directly interpretable by practitioners. A major component of my dissertation research is explaining the success of RNNs and constructing new RNN architectures through the process of "deep unfolding," which can construct and explain deep network architectures using an equivalence to inference in statistical models. Deep unfolding yields principled initializations for training deep networks, provides insight into their effectiveness, and assists with interpretation of what these networks learn.

        In particular, I will show how RNNs with rectified linear units and residual connections are a particular deep unfolding of a sequential version of the iterative shrinkage-thresholding algorithm (ISTA), a simple and classic algorithm for solving L1-regularized least-squares. This equivalence allows interpretation of state-of-the-art unitary RNNs (uRNNs) as an unfolded sparse coding algorithm. I will also describe a new type of RNN architecture called deep recurrent nonnegative matrix factorization (DR-NMF). DR-NMF is an unfolding of a sparse NMF model of nonnegative spectrograms for audio source separation. Both of these networks outperform conventional LSTM networks while also providing interpretability for practitioners.
    •  

    See All News & Events for Computational Photography
  • Recent Publications

    •  Liu, M.-Y., Tuzel, O., "Coupled Generative Adversarial Nets", arXiv, June 2016.
      BibTeX Download PDFRead TR2016-070
      • @techreport{MERL_TR2016-070,
      • author = {Liu, M.-Y. and Tuzel, O.},
      • title = {Coupled Generative Adversarial Nets},
      • institution = {MERL - Mitsubishi Electric Research Laboratories},
      • address = {Cambridge, MA 02139},
      • number = {TR2016-070},
      • month = jun,
      • year = 2016,
      • url = {http://www.merl.com/publications/TR2016-070/}
      • }
    •  Kadambi, A.; Boufounos, P.T., "Coded Aperture Compressive 3-D LIDAR", IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), DOI: 10.1109/ICASSP.2015.7178153, April 2015, pp. 1166-1170.
      BibTeX Download PDFRead TR2015-028
      • @inproceedings{Kadambi2015apr,
      • author = {Kadambi, A. and Boufounos, P.T.},
      • title = {Coded Aperture Compressive 3-D LIDAR},
      • booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
      • year = 2015,
      • pages = {1166--1170},
      • month = apr,
      • publisher = {IEEE},
      • doi = {10.1109/ICASSP.2015.7178153},
      • url = {http://www.merl.com/publications/TR2015-028}
      • }
    •  Ataer-Cansizoglu, E.; Taguchi, Y.; Ramalingam, S.; Miki, Y., "Calibration of Non-overlapping Cameras Using an External SLAM System", International Conference on 3D Vision (3DV), DOI: 10*1109/eDV.2014.106, December 2014, pp. 509-516.
      BibTeX Download PDFRead TR2014-106
      • @inproceedings{Cansizoglu2014dec,
      • author = {Ataer-Cansizoglu, E. and Taguchi, Y. and Ramalingam, S. and Miki, Y.},
      • title = {Calibration of Non-overlapping Cameras Using an External SLAM System},
      • booktitle = {International Conference on 3D Vision (3DV)},
      • year = 2014,
      • pages = {509--516},
      • month = dec,
      • publisher = {IEEE},
      • doi = {10*1109/eDV.2014.106},
      • url = {http://www.merl.com/publications/TR2014-106}
      • }
    •  Taguchi, Y., "Rainbow Flash Camera: Depth Edge Extraction Using Complementary Colors", International Journal of Computer Vision, DOI: 10.1007/s11263-014-0726-4, ISSN: 1573-1405, ISBN: 0920-5691, Vol. 110, No. 2, pp. 156-171, November 2014.
      BibTeX Download PDFRead TR2014-059
      • @article{Taguchi2014may,
      • author = {Taguchi, Y.},
      • title = {Rainbow Flash Camera: Depth Edge Extraction Using Complementary Colors},
      • journal = {International Journal of Computer Vision},
      • year = 2014,
      • volume = 110,
      • number = 2,
      • pages = {156--171},
      • month = nov,
      • doi = {10.1007/s11263-014-0726-4},
      • issn = {1573-1405},
      • isbn = {0920-5691},
      • url = {http://www.merl.com/publications/TR2014-059}
      • }
    •  Agrawal, A.; Ramalingam, S., "Single Image Calibration of Multi-Axial Imaging Systems", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), DOI: 10.1109/CVPR.2013.184, ISSN: 1063-6919, June 2013, pp. 1399-1406.
      BibTeX Download PDFRead TR2013-041
      • @inproceedings{Agrawal2013jun1,
      • author = {Agrawal, A. and Ramalingam, S.},
      • title = {Single Image Calibration of Multi-Axial Imaging Systems},
      • booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      • year = 2013,
      • pages = {1399--1406},
      • month = jun,
      • doi = {10.1109/CVPR.2013.184},
      • issn = {1063-6919},
      • url = {http://www.merl.com/publications/TR2013-041}
      • }
    •  Liu, M.-Y.; Tuzel, O.; Taguchi, Y., "Joint Geodesic Upsampling of Depth Images", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), DOI: 10.1109/CVPR.2013.29, ISSN: 1063-6919, June 2013, pp. 169-176.
      BibTeX Download PDFRead TR2013-042
      • @inproceedings{Liu2013jun,
      • author = {Liu, M.-Y. and Tuzel, O. and Taguchi, Y.},
      • title = {Joint Geodesic Upsampling of Depth Images},
      • booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      • year = 2013,
      • pages = {169--176},
      • month = jun,
      • doi = {10.1109/CVPR.2013.29},
      • issn = {1063-6919},
      • url = {http://www.merl.com/publications/TR2013-042}
      • }
    •  Agrawal, A.; Ramalingam, S.; Taguchi, Y.; Chari, V., "A Theory of Multi-Layer Flat Refractive Geometry", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2012, pp. 3346-3353.
      BibTeX Download PDFRead TR2012-047
      • @inproceedings{Agrawal2012jun,
      • author = {Agrawal, A. and Ramalingam, S. and Taguchi, Y. and Chari, V.},
      • title = {A Theory of Multi-Layer Flat Refractive Geometry},
      • booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      • year = 2012,
      • pages = {3346--3353},
      • month = jun,
      • url = {http://www.merl.com/publications/TR2012-047}
      • }
    •  Wu, D.; O'Toole, M.; Velten, A.; Agrawal, A.; Raskar, R., "Decomposing Global Light Transport using Time of Flight Imaging", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), DOI: 10.1109/CVPR.2012.6247697, June 2012, pp. 366-373.
      BibTeX Download PDFRead TR2012-042
      • @inproceedings{Wu2012jun,
      • author = {Wu, D. and O'Toole, M. and Velten, A. and Agrawal, A. and Raskar, R.},
      • title = {Decomposing Global Light Transport using Time of Flight Imaging},
      • booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      • year = 2012,
      • pages = {366--373},
      • month = jun,
      • doi = {10.1109/CVPR.2012.6247697},
      • url = {http://www.merl.com/publications/TR2012-042}
      • }
    See All Publications for Computational Photography
  • Free Downloads