Robotics
Where hardware, software and machine intelligence come together.
Our research is interdisciplinary and focuses on sensing, planning, reasoning, and control of single and multi-agent systems, including both manipulation and mobile robots. We strive to develop algorithms and methods for factory automation, smart building and transportation applications using machine learning, computer vision, RF/optical sensing, wireless communications, control theory and signal processing. Key research themes include bin picking and object manipulation, sensing and mapping of indoor areas, coordinated control of robot swarms, as well as robot learning and simulation.
Quick Links
-
Researchers
Devesh K.
Jha
Diego
Romeres
Daniel N.
Nikovski
Mouhacine
Benosman
Arvind
Raghunathan
Stefano
Di Cairano
Yebin
Wang
Toshiaki
Koike-Akino
William S.
Yerazunis
Karl
Berntorp
Scott A.
Bortoff
Radu
Corcodel
Tim K.
Marks
Siddarth
Jain
Matthew E.
Brand
Alan
Sullivan
Bingnan
Wang
Ye
Wang
Avishai
Weiss
Jianlin
Guo
Jonathan
Le Roux
Hassan
Mansour
Philip V.
Orlik
Ronald N.
Perry
Rien
Quirynen
Koon Hoo
Teo
Anthony
Vetro
Pedro
Miraldo
-
Awards
-
AWARD MERL Researcher Devesh Jha Wins the Rudolf Kalman Best Paper Award 2019 Date: October 10, 2019
Awarded to: Devesh Jha, Nurali Virani, Zhenyuan Yuan, Ishana Shekhawat and Asok Ray
MERL Contact: Devesh K. Jha
Research Areas: Artificial Intelligence, Control, Data Analytics, Machine Learning, RoboticsBrief- MERL researcher Devesh Jha has won the Rudolf Kalman Best Paper Award 2019 for the paper entitled "Imitation of Demonstrations Using Bayesian Filtering With Nonparametric Data-Driven Models". This paper, published in a Special Commemorative Issue for Rudolf E. Kalman in the ASME JDSMC in March 2018, uses Bayesian filtering for imitation learning in Hidden Mode Hybrid Systems. This award is given annually by the Dynamic Systems and Control Division of ASME to the authors of the best paper published in the ASME Journal of Dynamic Systems Measurement and Control during the preceding year.
See All Awards for MERL -
-
News & Events
-
TALK [MERL Seminar Series 2022] Prof. Michael Posa presents talk titled Hybrid robotics and implicit learning Date & Time: Tuesday, May 3, 2022; 1:00 PM
Speaker: Michael Posa, University of Pennsylvania
MERL Host: Devesh K. Jha
Research Areas: Control, Optimization, RoboticsAbstract
Machine learning has shown incredible promise in robotics, with some notable recent demonstrations in manipulation and sim2real transfer. These results, however, require either an accurate a priori model (for simulation) or a large amount of data. In contrast, my lab is focused on enabling robots to enter novel environments and then, with minimal time to gather information, accomplish complex tasks. In this talk, I will argue that the hybrid or contact-driven nature of real-world robotics, where a robot must safely and quickly interact with objects, drives this high data requirement. In particular, the inductive biases inherent in standard learning methods fundamentally clash with the non-differentiable physics of contact-rich robotics. Focusing on model learning, or system identification, I will show both empirical and theoretical results which demonstrate that contact stiffness leads to poor training and generalization, leading to some healthy skepticism of simulation experiments trained on artificially soft environments. Fortunately, implicit learning formulations, which embed convex optimization problems, can dramatically reshape the optimization landscape for these stiff problems. By carefully reasoning about the roles of stiffness and discontinuity, and integrating non-smooth structures, we demonstrate dramatically improved learning performance. Within this family of approaches, ContactNets accurately identifies the geometry and dynamics of a six-sided cube bouncing, sliding, and rolling across a surface from only a handful of sample trajectories. Similarly, a piecewise-affine hybrid system with thousands of modes can be identified purely from state transitions. Time permitting, I'll discuss how these learned models can be deployed for control via recent results in real-time, multi-contact MPC.
-
NEWS Radu Corcodel to present invited seminar at NYU on Robot Vision Date: May 4, 2022
MERL Contact: Radu Corcodel
Research Areas: Computer Vision, RoboticsBrief- Radu Corcodel, a Principal Research Scientist in MERL's Computer Vision Group, will present an overview of the Robot Perception research published by MERL for advanced manipulation. The talk will mainly cover topics pertaining to robotic manipulation in unstructured environments such as machine vision, tactile sensing and autonomous grasping. The seminar will also cover specific perception problems in non-prehensile interactions such as Contact-Implicit Trajectory Optimization and Tactile Classification, and is intended for a broader audience.
See All News & Events for Robotics -
-
Internships
-
CV1738: Robot autonomous grasping using tactile sensing
The Computer Vision group is offering an internship opportunity in robot autonomous grasping using tactile sensing. The internship is open to highly skilled graduate students on a PhD track. Candidates should have a solid understanding of reinforcement learning, contact mechanics, simulating contacts, grasping, pose estimation and point cloud processing. The policies will be deployed on physical robots and the sensing is provided by various types of tactile sensing arrays. Strong programming skills are required, including MuJoCo, ROS, C++ and Python. Duration and start dates are flexible.
-
CV1703: Software development in ROS for robotic manipulation
MERL is offering an internship position for non-research software development for robotic manipulation. The scope of the internship is to develop robust ROS packages by refactoring existing experimental code. The position is open to prospective candidates with very strong programming skills in ROS (Robot Operating System) using C++ primarily and Python respectively. The selected intern will have a software engineering role rather than research oriented. The position is open to both senior undergraduate students and master students. Flexible start and end dates.
-
CA1728: Safe data-driven control of dynamical systems under uncertainty
MERL is looking for a highly motivated individual to work on safe control of data-driven, uncertain, dynamical systems. The research will develop novel optimization and learning-based control algorithms to guarantee safety and performance in various industrial applications, including autonomous driving. The ideal candidate should have experience in either one or multiple of the following topics: optimal control under uncertainty, (robust and stochastic) model predictive control, (convex and non-convex) optimization, and (reinforcement and statistical) learning. Ph.D. students in engineering or mathematics with a focus on control, optimization, and learning are encouraged to apply. A successful internship will result in submission of relevant results to peer-reviewed conference proceedings and journals, and development of well-documented (Python/MATLAB) code for MERL. The expected duration of the internship is 3-6 months, and the start date is Summer 2022.
See All Internships for Robotics -
-
Recent Publications
- , "Robust Pivoting: Exploiting Frictional Stability Using Bilevel Optimization", IEEE International Conference on Robotics and Automation (ICRA) 2022, May 2022.BibTeX TR2022-045 PDF
- @inproceedings{Shirai2022may,
- author = {Shirai, Yuki and Jha, Devesh K. and Raghunathan, Arvind and Romeres, Diego},
- title = {Robust Pivoting: Exploiting Frictional Stability Using Bilevel Optimization},
- booktitle = {IEEE International Conference on Robotics and Automation (ICRA) 2022},
- year = 2022,
- month = may,
- url = {https://www.merl.com/publications/TR2022-045}
- }
- , "Learning to Synthesize Volumetric Meshes from Vision-based Tactile Imprints", IEEE International Conference on Robotics and Automation (ICRA) 2022, March 2022.BibTeX TR2022-035 PDF
- @article{Zhu2022mar,
- author = {Zhu, Xinghao and Jain, Siddarth and Tomizuka, Masayoshi and van Baar, Jeroen},
- title = {Learning to Synthesize Volumetric Meshes from Vision-based Tactile Imprints},
- journal = {IEEE International Conference on Robotics and Automation (ICRA) 2022},
- year = 2022,
- month = mar,
- url = {https://www.merl.com/publications/TR2022-035}
- }
- , "Learning robot motor skills with mixed reality", International Workshop on Virtual, Augmented, and Mixed-Reality for Human-Robot Interactions ACM/IEEEE International Conference on Human-Robot Collaboration 2022, March 2022.BibTeX TR2022-032 PDF
- @inproceedings{Rosen2022mar,
- author = {Rosen, Eric and Rammohan, Sreehari and Jha, Devesh K.},
- title = {Learning robot motor skills with mixed reality},
- booktitle = {International Workshop on Virtual, Augmented, and Mixed-Reality for Human-Robot Interactions ACM/IEEEE International Conference on Human-Robot Collaboration 2022},
- year = 2022,
- month = mar,
- url = {https://www.merl.com/publications/TR2022-032}
- }
- , "Model-Based Reinforcement Learning Using Monte Carlo Gradient Estimation", Automatica.it, September 2021.BibTeX TR2021-108 PDF
- @inproceedings{Amadio2021sep,
- author = {Amadio, Fabio and Dalla Libera, Alberto and Carli, Ruggero and Romeres, Diego},
- title = {Model-Based Reinforcement Learning Using Monte Carlo Gradient Estimation},
- booktitle = {Automatica.it},
- year = 2021,
- month = sep,
- url = {https://www.merl.com/publications/TR2021-108}
- }
- , "Visual 3D Perception for Interactive Robotic Tactile Data Acquisition", IEEE International Conference on Automation Science and Engineering (CASE 2021), August 2021.BibTeX TR2021-092 PDF
- @inproceedings{Jain2021aug,
- author = {Jain, Siddarth and Corcodel, Radu and van Baar, Jeroen},
- title = {Visual 3D Perception for Interactive Robotic Tactile Data Acquisition},
- booktitle = {2021 IEEE International Conference on Automation Science and Engineering (CASE)},
- year = 2021,
- month = aug,
- url = {https://www.merl.com/publications/TR2021-092}
- }
- , "Robotic Applications of Mechanical Metamaterials Produced Using SLA 3D Printing: Cthulhu-Morphic Grippers", Solid Freeform Fabrication Symposium, DOI: 10.26153/tsw/17539, August 2021.BibTeX TR2021-088 PDF
- @inproceedings{Solomon2021aug,
- author = {Solomon, E. and Yerazunis, William S.},
- title = {Robotic Applications of Mechanical Metamaterials Produced Using SLA 3D Printing: Cthulhu-Morphic Grippers},
- booktitle = {Solid Freeform Fabrication Symposium},
- year = 2021,
- month = aug,
- doi = {10.26153/tsw/17539},
- url = {https://www.merl.com/publications/TR2021-088}
- }
- , "Control of Mechanical Systems via Feedback Linearization Based on Black-Box Gaussian Process Models", European Control Conference (ECC), DOI: 10.23919/ECC54610.2021.9654429, June 2021.BibTeX TR2021-068 PDF
- @inproceedings{DallaLibera2021jun,
- author = {Dalla Libera, Alberto and Amadio, Fabio and Nikovski, Daniel N. and Carli, Ruggero and Romeres, Diego},
- title = {Control of Mechanical Systems via Feedback Linearization Based on Black-Box Gaussian Process Models},
- booktitle = {European Control Conference (ECC)},
- year = 2021,
- month = jun,
- publisher = {IEEE},
- doi = {10.23919/ECC54610.2021.9654429},
- isbn = {978-9-4638-4236-5},
- url = {https://www.merl.com/publications/TR2021-068}
- }
- , "Robust Pivoting: Exploiting Frictional Stability Using Bilevel Optimization", IEEE International Conference on Robotics and Automation (ICRA) 2022, May 2022.
-
Videos
-
[MERL Seminar Series Spring 2022] Hybrid robotics and implicit learning
-
[MERL Seminar Series Spring 2022] Exact Structural Analysis of Multimode Modelica Models
-
[MERL Seminar Series Spring 2022] Self-Supervised Scene Representation Learning
-
[MERL Seminar Series 2021] Learning to See by Moving: Self-supervising 3D scene representations for perception, control, and visual reasoning
-
Robotic Research at MERL
-
Control of Mechanical Systems via Feedback Linearization Based on Black-Box Gaussian Process Models
-
Modelica-Based Modeling and Control of a Delta Robot
-
Towards Human-Level Learning of Complex Physical Puzzles
-
Assembly of Belt Drive Units
-
Examples of Robotic Manipulation
-
Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry
-
Cooperating Modular Goal Selection and Motion Planning for Autonomous Driving
-
Deep Reactive Planning in Dynamic Environments
-
Monte Carlo Probabilistic Inference for Learning Control
-
Experimental Validation of Reachability-based Decision Making for Autonomous Driving
-
-
Software Downloads
