Computer Vision

Researchers

Awards

AWARD Best Paper - Honorable Mention Award at WACV 2021
Date: January 6, 2021
Awarded to: Rushil Anirudh, Suhas Lohit, Pavan Turaga
MERL Contact: Suhas Lohit
Research Areas: Computational Sensing, Computer Vision, Machine Learning
Brief
- A team of researchers from Mitsubishi Electric Research Laboratories (MERL), Lawrence Livermore National Laboratory (LLNL) and Arizona State University (ASU) received the Best Paper Honorable Mention Award at WACV 2021 for their paper "Generative Patch Priors for Practical Compressive Image Recovery".
  
  The paper proposes a novel model of natural images as a composition of small patches which are obtained from a deep generative network. This is unlike prior approaches where the networks attempt to model image-level distributions and are unable to generalize outside training distributions. The key idea in this paper is that learning patch-level statistics is far easier. As the authors demonstrate, this model can then be used to efficiently solve challenging inverse problems in imaging such as compressive image recovery and inpainting even from very few measurements for diverse natural scenes.
AWARD MERL Researchers win Best Paper Award at ICCV 2019 Workshop on Statistical Deep Learning in Computer Vision
Date: October 27, 2019
Awarded to: Abhinav Kumar, Tim K. Marks, Wenxuan Mou, Chen Feng, Xiaoming Liu
MERL Contact: Tim K. Marks
Research Areas: Artificial Intelligence, Computer Vision, Machine Learning
Brief
- MERL researcher Tim Marks, former MERL interns Abhinav Kumar and Wenxuan Mou, and MERL consultants Professor Chen Feng (NYU) and Professor Xiaoming Liu (MSU) received the Best Oral Paper Award at the IEEE/CVF International Conference on Computer Vision (ICCV) 2019 Workshop on Statistical Deep Learning in Computer Vision (SDL-CV) held in Seoul, Korea. Their paper, entitled "UGLLI Face Alignment: Estimating Uncertainty with Gaussian Log-Likelihood Loss," describes a method which, given an image of a face, estimates not only the locations of facial landmarks but also the uncertainty of each landmark location estimate.
AWARD R&D100 award for Deep Learning-based Water Detector
Date: November 16, 2018
Awarded to: Ziming Zhang, Alan Sullivan, Hideaki Maehara, Kenji Taira, Kazuo Sugimoto
MERL Contact: Alan Sullivan
Research Areas: Artificial Intelligence, Computer Vision, Machine Learning
Brief
- Researchers and developers from MERL, Mitsubishi Electric and Mitsubishi Electric Engineering (MEE) have been recognized with an R&D100 award for the development of a deep learning-based water detector. Automatic detection of water levels in rivers and streams is critical for early warning of flash flooding. Existing systems require a height gauge be placed in the river or stream, something that is costly and sometimes impossible. The new deep learning-based water detector uses only images from a video camera along with 3D measurements of the river valley to determine water levels and warn of potential flooding. The system is robust to lighting and weather conditions working well during the night as well as during fog or rain. Deep learning is a relatively new technique that uses neural networks and AI that are trained from real data to perform human-level recognition tasks. This work is powered by Mitsubishi Electric's Maisart AI technology.

See All Awards for Computer Vision

News & Events

EVENT Prof. Melanie Zeilinger of ETH to give keynote at MERL's Virtual Open House
Date & Time: Thursday, December 9, 2021; 1:00pm - 5:30pm EST
Speaker: Prof. Melanie Zeilinger, ETH
Location: Virtual Event
Research Areas: Applied Physics, Artificial Intelligence, Communications, Computational Sensing, Computer Vision, Control, Data Analytics, Dynamical Systems, Electric Systems, Electronic and Photonic Devices, Machine Learning, Multi-Physical Modeling, Optimization, Robotics, Signal Processing, Speech & Audio, Digital Video, Human-Computer Interaction, Information Security
Brief
- MERL is excited to announce the second keynote speaker for our Virtual Open House 2021:
  Prof. Melanie Zeilinger from ETH .
  
  Our virtual open house will take place on December 9, 2021, 1:00pm - 5:30pm (EST).
  
  Join us to learn more about who we are, what we do, and discuss our internship and employment opportunities. Prof. Zeilinger's talk is scheduled for 3:15pm - 3:45pm (EST).
  
  Registration: https://mailchi.mp/merl/merlvoh2021
  
  Keynote Title: Control Meets Learning - On Performance, Safety and User Interaction
  
  Abstract: With increasing sensing and communication capabilities, physical systems today are becoming one of the largest generators of data, making learning a central component of autonomous control systems. While this paradigm shift offers tremendous opportunities to address new levels of system complexity, variability and user interaction, it also raises fundamental questions of learning in a closed-loop dynamical control system. In this talk, I will present some of our recent results showing how even safety-critical systems can leverage the potential of data. I will first briefly present concepts for using learning for automatic controller design and for a new safety framework that can equip any learning-based controller with safety guarantees. The second part will then discuss how expert and user information can be utilized to optimize system performance, where I will particularly highlight an approach developed together with MERL for personalizing the motion planning in autonomous driving to the individual driving style of a passenger.
EVENT Prof. Ashok Veeraraghavan of Rice University to give keynote at MERL's Virtual Open House
Date & Time: Thursday, December 9, 2021; 1:00pm - 5:30pm EST
Speaker: Prof. Ashok Veeraraghavan, Rice University
Location: Virtual Event
Research Areas: Applied Physics, Artificial Intelligence, Communications, Computational Sensing, Computer Vision, Control, Data Analytics, Dynamical Systems, Electric Systems, Electronic and Photonic Devices, Machine Learning, Multi-Physical Modeling, Optimization, Robotics, Signal Processing, Speech & Audio, Digital Video, Human-Computer Interaction, Information Security
Brief
- MERL is excited to announce the first keynote speaker for our Virtual Open House 2021:
  Prof. Ashok Veeraraghavan from Rice University.
  
  Our virtual open house will take place on December 9, 2021, 1:00pm - 5:30pm (EST).
  
  Join us to learn more about who we are, what we do, and discuss our internship and employment opportunities. Prof. Veeraraghavan's talk is scheduled for 1:15pm - 1:45pm (EST).
  
  Registration: https://mailchi.mp/merl/merlvoh2021
  
  Keynote Title: Computational Imaging: Beyond the limits imposed by lenses.
  
  Abstract: The lens has long been a central element of cameras, since its early use in the mid-nineteenth century by Niepce, Talbot, and Daguerre. The role of the lens, from the Daguerrotype to modern digital cameras, is to refract light to achieve a one-to-one mapping between a point in the scene and a point on the sensor. This effect enables the sensor to compute a particular two-dimensional (2D) integral of the incident 4D light-field. We propose a radical departure from this practice and the many limitations it imposes. In the talk we focus on two inter-related research projects that attempt to go beyond lens-based imaging.
  
  First, we discuss our lab’s recent efforts to build flat, extremely thin imaging devices by replacing the lens in a conventional camera with an amplitude mask and computational reconstruction algorithms. These lensless cameras, called FlatCams can be less than a millimeter in thickness and enable applications where size, weight, thickness or cost are the driving factors. Second, we discuss high-resolution, long-distance imaging using Fourier Ptychography, where the need for a large aperture aberration corrected lens is replaced by a camera array and associated phase retrieval algorithms resulting again in order of magnitude reductions in size, weight and cost. Finally, I will spend a few minutes discussing how the wholistic computational imaging approach can be used to create ultra-high-resolution wavefront sensors.

See All News & Events for Computer Vision

Research Highlights

Internships

CV1721: Deep Learning for Robotic Grasping

MERL is looking for a highly motivated and qualified intern to work on computer vision for robotic grasping. The ideal candidate would be a Ph.D. student with a strong background in deep learning and robotics. There are several available topics for consideration including goal-driven grasping of novel objects, grasping for bin picking, and grasping in clutter. The project requires development of novel algorithms which can be implemented and evaluated on a robotic platform. Experience in working with a physics engine simulator like PyBullet, Mujoco, or Gazebo is preferred. Proficiency in Python programming is necessary and experience with ROS is a plus. Successful candidate will collaborate with MERL researchers and publication of the relevant results is expected. Start date is flexible and expected duration of the internship is 3-4 months. Interested candidates are encouraged to apply with their recent CV and list of publications in related topics. This internship is preferred to be onsite at MERL, but may be done remotely where you live if the COVID pandemic makes it necessary.
CV1712: Symmetries, equivariance and invariance in deep learning

MERL is seeking an intern to conduct research in the areas of learning symmetries from data and equivariant neural networks for applications in computer vision. The ideal candidate is a PhD student with experience in deep learning and computer vision and a good publication record at top-tier venues. Prior knowledge and experience with group theory/geometry and equivariant neural networks are a big plus. Good Python and Pytorch/Tensorflow skills are required. Publication of results in conference proceedings and journals is expected. The expected duration of the internship is 3 months and the start date is flexible. This internship is preferred to be onsite at MERL, but may be done remotely where you live if the COVID pandemic makes it necessary.
CV1720: Computer Vision for Robotic Manipulation

MERL is looking for a highly motivated and qualified intern to work on computer vision for robotic manipulation. The ideal candidate would be a Ph.D. student with a strong background in computer vision, deep learning, and/or robotics. There are several available topics for consideration including task and motion planning (TAMP), learning for object manipulation, and pose estimation. The project requires development of novel algorithms which can be implemented and evaluated on a robotic platform. Experience in working with a physics engine simulator like PyBullet, Mujoco, or Gazebo is preferred. Proficiency in Python programming is necessary and experience with ROS is a plus. Successful candidate will collaborate with MERL researchers and publication of the relevant results is expected. Start date is flexible and expected duration of the internship is 3-4 months. Interested candidates are encouraged to apply with their recent CV and list of publications in related topics. This internship is preferred to be onsite at MERL, but may be done remotely where you live if the COVID pandemic makes it necessary.

See All Internships for Computer Vision

Openings

CV1585: Research Scientist / Principal Research Scientist - Computer Vision

See All Openings at MERL

Recent Publications

Chatterjee, M., Ahuja, N., Cherian, A., "A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction", IEEE International Conference on Computer Vision (ICCV), October 2021.
BibTeX TR2021-096 PDF
- @inproceedings{Chatterjee2021oct2,
- author = {Chatterjee, Moitreya and Ahuja, Narendra and Cherian, Anoop},
- title = {A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction},
- booktitle = {IEEE International Conference on Computer Vision (ICCV)},
- year = 2021,
- month = oct,
- url = {https://www.merl.com/publications/TR2021-096}
- }
Chatterjee, M., Le Roux, J., Ahuja, N., Cherian, A., "Visual Scene Graphs for Audio Source Separation", IEEE International Conference on Computer Vision (ICCV), October 2021.
BibTeX TR2021-095 PDF
- @inproceedings{Chatterjee2021oct,
- author = {Chatterjee, Moitreya and Le Roux, Jonathan and Ahuja, Narendra and Cherian, Anoop},
- title = {Visual Scene Graphs for Audio Source Separation},
- booktitle = {IEEE International Conference on Computer Vision (ICCV)},
- year = 2021,
- month = oct,
- url = {https://www.merl.com/publications/TR2021-095}
- }
Cherian, A., Pais, G., Jain, S., Marks, T.K., Sullivan, A., "InSeGAN: A Generative Approach to Segmenting Identical Instances in Depth Images", IEEE International Conference on Computer Vision (ICCV), October 2021.
BibTeX TR2021-097 PDF
- @inproceedings{Cherian2021oct,
- author = {Cherian, Anoop and Pais, Goncalo and Jain, Siddarth and Marks, Tim K. and Sullivan, Alan},
- title = {InSeGAN: A Generative Approach to Segmenting Identical Instances in Depth Images},
- booktitle = {IEEE International Conference on Computer Vision (ICCV)},
- year = 2021,
- month = oct,
- url = {https://www.merl.com/publications/TR2021-097}
- }
Hu, W., Pang, J., Liu, X., Tian, D., Lin, C.-W., Vetro, A., "Graph Signal Processing for Geometric Data and Beyond: Theory and Applications", IEEE Transactions on Multimedia, October 2021.
BibTeX TR2021-121 PDF
- @article{Hu2021oct,
- author = {Hu, Wei and Pang, Jiahao and Liu, Xianming and Tian, Dong and Lin, Chia-Wen and Vetro, Anthony},
- title = {Graph Signal Processing for Geometric Data and Beyond: Theory and Applications},
- journal = {IEEE Transactions on Multimedia},
- year = 2021,
- month = oct,
- url = {https://www.merl.com/publications/TR2021-121}
- }
Kannapiran, S., van Baar, J., Berman, S., "A Visual Inertial Odometry Framework for 3D Points, Lines and Planes", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 2021.
BibTeX TR2021-131 PDF
- @inproceedings{Kannapiran2021sep,
- author = {Kannapiran, Shenbagaraj and van Baar, Jeroen and Berman, Spring},
- title = {A Visual Inertial Odometry Framework for 3D Points, Lines and Planes},
- booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
- year = 2021,
- month = sep,
- url = {https://www.merl.com/publications/TR2021-131}
- }
Comas, A., Marks, T.K., Mansour, H., Lohit, S., Ma, Y., Liu, X., "TURNIP: Time-series U-NET with Recurrence for NIR Imaging PPG", IEEE International Conference on Image Processing (ICIP), September 2021.
BibTeX TR2021-099 PDF
- @inproceedings{Comas2021sep,
- author = {Comas, Armand and Marks, Tim K. and Mansour, Hassan and Lohit, Suhas and Ma, Yechi and Liu, Xiaoming},
- title = {TURNIP: Time-series U-NET with Recurrence for NIR Imaging PPG},
- booktitle = {IEEE International Conference on Image Processing (ICIP)},
- year = 2021,
- month = sep,
- url = {https://www.merl.com/publications/TR2021-099}
- }
Shi, L., Liu, D., Thornton, J.E., "Robust Camera Pose Estimation for Image Stitching", IEEE International Conference on Image Processing (ICIP), DOI: 10.1109/ICIP42928.2021.9506680, September 2021.
BibTeX TR2021-113 PDF
- @inproceedings{Shi2021sep,
- author = {Shi, Laixi and Liu, Dehong and Thornton, Jay E.},
- title = {Robust Camera Pose Estimation for Image Stitching},
- booktitle = {IEEE International Conference on Image Processing (ICIP)},
- year = 2021,
- month = sep,
- publisher = {IEEE},
- doi = {10.1109/ICIP42928.2021.9506680},
- isbn = {978-1-6654-4115-5},
- url = {https://www.merl.com/publications/TR2021-113}
- }
Hori, C., Hori, T., Le Roux, J., "Optimizing Latency for Online Video Captioning Using Audio-VisualTransformers", Annual Conference of the International Speech Communication Association (Interspeech), DOI: 10.21437/Interspeech.2021-1975, August 2021, pp. 586–590.
BibTeX TR2021-093 PDF
- @inproceedings{Hori2021aug2,
- author = {Hori, Chiori and Hori, Takaaki and Le Roux, Jonathan},
- title = {Optimizing Latency for Online Video Captioning Using Audio-VisualTransformers},
- booktitle = {Annual Conference of the International Speech Communication Association (Interspeech)},
- year = 2021,
- pages = {586–590},
- month = aug,
- publisher = {ISCA},
- doi = {10.21437/Interspeech.2021-1975},
- url = {https://www.merl.com/publications/TR2021-093}
- }

See All Publications for Computer Vision

Quick Links

Tim K.Marks

Michael J.Jones

AnoopCherian

AlanSullivan

ChioriHori

Matthew E.Brand

HassanMansour

TakaakiHori

Ronald N.Perry

JayThornton

RaduCorcodel

Devesh K.Jha

AnthonyVetro

Petros T.Boufounos

JonathanLe Roux

DehongLiu

SuhasLohit

Daniel N.Nikovski

DiegoRomeres

ArvindRaghunathan

YeWang

SiddarthJain

GordonWichern

William S.Yerazunis

ToshiakiKoike-Akino

Philip V.Orlik

HuifangSun

YebinWang

VarunHaritsa

Kuan-ChuanPeng

Robust Machine Learning

Video Anomaly Detection

MERL Shopping Dataset

Point-Plane SLAM

CV1721: Deep Learning for Robotic Grasping

CV1712: Symmetries, equivariance and invariance in deep learning

CV1720: Computer Vision for Robotic Manipulation

CV1585: Research Scientist / Principal Research Scientist - Computer Vision

[MERL Seminar Series 2021] Learning to See by Moving: Self-supervising 3D scene representations for perception, control, and visual reasoning

[MERL Seminar Series 2021] Look and Listen: From Semantic to Spatial Audio-Visual Perception

Towards Human-Level Learning of Complex Physical Puzzles

3D Object Discovery and Modeling Using Single RGB-D Images Containing Multiple Object Instances

Joint 3D Reconstruction of a Static Scene and Moving Objects

Direct Multichannel Tracking

FoldingNet: Interpretable Unsupervised Learning on 3D Point Clouds

FasTFit: A fast T-spline fitting algorithm

CASENet: Deep Category-Aware Semantic Edge Detection

Object Detection and Tracking in RGB-D SLAM via Hierarchical Feature Grouping

Pinpoint SLAM: A Hybrid of 2D and 3D Simultaneous Localization and Mapping for RGB-D Sensors

Action Detection Using A Deep Recurrent Neural Network

Robot Bin Picking

MERL Research on Autonomous Vehicles

3D Reconstruction

Sapphire - High Accuracy NC Milling Simulation

Dose optimization for particle beam therapy

Obstacle Detection

Semantic Scene Labeling

Saffron - Digital Type System

Deep Hierarchical Parsing for Semantic Segmentation

Global Local Face Upsampling Network

Gaussian Conditional Random Field Network for Semantic Segmentation

Fast Graspability Evaluation on Single Depth Maps for Bin Picking with General Grippers

Point-Plane SLAM for Hand-Held 3D Sensors

Tracking an RGB-D Camera Using Points and Planes

Fast Plane Extraction in Organized Point Clouds Using Agglomerative Hierarchical Clustering

Calibration of Non-Overlapping Cameras Using an External SLAM System

Voting-Based Pose Estimation for Robotic Assembly Using a 3D Sensor

Fast Object Localization and Pose Estimation in Heavy Clutter for Robotic Bin Picking

Learning to rank 3D features

Generating Visual Dynamics from Sound and Context

Adversarially-Contrastive Optimal Transport

MotionNet

Contact-Implicit Trajectory Optimization

FoldingNet++

Landmarks’ Location, Uncertainty, and Visibility Likelihood

Gradient-based Nikaido-Isoda

Circular Maze Environment

Discriminative Subspace Pooling

Kernel Correlation Network

Tim K.
Marks

Michael J.
Jones

Anoop
Cherian

Alan
Sullivan

Chiori
Hori

Matthew E.
Brand

Hassan
Mansour

Takaaki
Hori

Ronald N.
Perry

Jay
Thornton

Radu
Corcodel

Devesh K.
Jha

Anthony
Vetro

Petros T.
Boufounos

Jonathan
Le Roux

Dehong
Liu

Suhas
Lohit

Daniel N.
Nikovski

Diego
Romeres

Arvind
Raghunathan

Ye
Wang

Siddarth
Jain

Gordon
Wichern

William S.
Yerazunis

Toshiaki
Koike-Akino

Philip V.
Orlik

Huifang
Sun

Yebin
Wang

Varun
Haritsa

Kuan-Chuan
Peng