Point-Plane SLAM for RGB-D Sensors and Its Applications

Real-time 3D reconstruction using an RGB-D sensor on a tablet.

MERL Researchers: Yuichi Taguchi, Jeroen van Baar, Alan Sullivan, Esra Cansizoglu, Chen Feng (Computer Vision).

Search MERL publications by keyword: Computer Vision, SLAM, 3D reconstruction, registration, plane extraction,


We present a real-time 3D reconstruction system using an RGB-D sensor on a hand-held tablet. The main novelty of the system is a simultaneous localization and mapping (SLAM) algorithm that uses both point and plane features as primitives. Planes are the most common structures in man-made indoor and outdoor scenes.

Our mobile SLAM system using an RGB-D sensor on a tablet

In contrast to existing SLAM algorithms that use only point features, our algorithm that also exploits plane features has the following advantages:

  • (1) it enables faster correspondence search and registration, since the number of planes is typically much smaller than the number of points;
  • (2) planes generated by many points are more robust to noise than individual points, leading to more accurate registration; and
  • (3) it produces plane-based 3D models that are more compact than point-based ones.

Plane-based 3D model
Point-based 3D model

As the core of the algorithm, we show that it is possible to register 3D data in two different coordinate systems using any combination of three point/plane features (3 planes, 2 planes and 1 point, 1 plane and 2 points, and 3 points). We use the minimal set of features in a RANSAC framework to robustly compute correspondences and estimate the camera pose. We also use the point and plane features in bundle adjustment to jointly refine all the camera poses and point/plane parameters.

Our system enables large-scale indoor 3D reconstruction and generates plane-based and point-based 3D models. The 3D models can be used for several applications, including robot collision avoidance (demonstrated at iREX 2013 and MODEX 2014), 2D camera localization for augmented reality (demonstrated at 3DV 2014 and MVA 2015), and calibration of non-overlapping cameras (presented at 3DV 2014).

Videos

Point-Plane SLAM for Hand-Held Sensors.
Tracking an RGB-D Camea Using Points and Planes.
Fast Plane Extraction in Organized Point Clouds Using Agglomerative Hierarchical Clustering.
Calibration of Non-Overlapping Cameras Using an External SLAM System.

Software Download

Plane Extraction using Agglomerative Clustering (PEAC) is available for non-commercial use.

News & Events

References

  •  Taguchi, Y.; Jian, Y.-D.; Ramalingam, S.; Feng, C., "SLAM Using Both Points and Planes for Hand-Held 3D Sensors", International Symposium on Mixed and Augmented Reality (ISMAR), November 2012.
    BibTeX Download PDFRead TR2012-091
    • @inproceedings{Taguchi2012nov,
    • author = {Taguchi, Y. and Jian, Y.-D. and Ramalingam, S. and Feng, C.},
    • title = {SLAM Using Both Points and Planes for Hand-Held 3D Sensors},
    • booktitle = {International Symposium on Mixed and Augmented Reality (ISMAR)},
    • year = 2012,
    • month = nov,
    • url = {http://www.merl.com/publications/TR2012-091}
    • }
  •  Taguchi, Y.; Jian, Y.-D.; Ramalingam, S.; Feng, C., "Point-Plane SLAM for Hand-Held 3D Sensors", IEEE International Conference on Robotics and Automation (ICRA), DOI: 10.1109/ICRA.2013.6631318, ISSN: 1050-4729, ISBN: 978-1-4673-5641-1, May 2013, pp. 5182-5189.
    BibTeX Download PDFRead TR2013-031
    • @inproceedings{Taguchi2013may,
    • author = {Taguchi, Y. and Jian, Y.-D. and Ramalingam, S. and Feng, C.},
    • title = {Point-Plane SLAM for Hand-Held 3D Sensors},
    • booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
    • year = 2013,
    • pages = {5182--5189},
    • month = may,
    • doi = {10.1109/ICRA.2013.6631318},
    • issn = {1050-4729},
    • isbn = {978-1-4673-5641-1},
    • url = {http://www.merl.com/publications/TR2013-031}
    • }
  •  Ataer-Cansizoglu, E.; Taguchi, Y.; Ramalingam, S.; Garaas, T., "Tracking an RGB-D Camera Using Points and Planes", IEEE Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV), December 2013.
    BibTeX Download PDFRead TR2013-106
    • @inproceedings{Ataer-Cansizoglu2013dec,
    • author = {Ataer-Cansizoglu, E. and Taguchi, Y. and Ramalingam, S. and Garaas, T.},
    • title = {Tracking an RGB-D Camera Using Points and Planes},
    • booktitle = {IEEE Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV)},
    • year = 2013,
    • month = dec,
    • url = {http://www.merl.com/publications/TR2013-106}
    • }
  •  Feng, C.; Taguchi, Y.; Kamat, V., "Fast Plane Extraction in Organized Point Clouds Using Agglomerative Hierarchical Clustering", IEEE International Conference on Robotics and Automation (ICRA), DOI: 10.1109/ICRA.2014.6907776, May 2014, pp. 6218-6225.
    BibTeX Download PDFRead TR2014-066
    • @inproceedings{Feng2014may,
    • author = {Feng, C. and Taguchi, Y. and Kamat, V.},
    • title = {Fast Plane Extraction in Organized Point Clouds Using Agglomerative Hierarchical Clustering},
    • booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
    • year = 2014,
    • pages = {6218--6225},
    • month = may,
    • publisher = {IEEE},
    • doi = {10.1109/ICRA.2014.6907776},
    • url = {http://www.merl.com/publications/TR2014-066}
    • }
  •  Ataer-Cansizoglu, E.; Taguchi, Y.; Ramalingam, S.; Miki, Y., "Calibration of Non-overlapping Cameras Using an External SLAM System", International Conference on 3D Vision (3DV), DOI: 10*1109/eDV.2014.106, December 2014, pp. 509-516.
    BibTeX Download PDFRead TR2014-106
    • @inproceedings{Cansizoglu2014dec,
    • author = {Ataer-Cansizoglu, E. and Taguchi, Y. and Ramalingam, S. and Miki, Y.},
    • title = {Calibration of Non-overlapping Cameras Using an External SLAM System},
    • booktitle = {International Conference on 3D Vision (3DV)},
    • year = 2014,
    • pages = {509--516},
    • month = dec,
    • publisher = {IEEE},
    • doi = {10*1109/eDV.2014.106},
    • url = {http://www.merl.com/publications/TR2014-106}
    • }