Co-Fusion: Real-time Segmentation,
Tracking and Fusion of Multiple Objects
and
University College London
Accepted to ICRA2017
In this paper we introduce a new dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects, using either motion or semantic cues, while simultaneously tracking and reconstructing their 3D shape in real time.
Crucially, we use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers that are of no interest to the robot, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system has the benefit to enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.
Code
BibTeX
@inproceedings{ruenz2017icra,
title={Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects},
author={Martin R{\"u}nz and Lourdes Agapito},
booktitle={2017 IEEE International Conference on Robotics and Automation (ICRA)},
year={2017},
month={May},
pages = {4471-4478},
keywords={visual SLAM;multi-object;RGBD;dynamic;fusion;tracking;mapping;segmentation;CRF;real-time}
}
Acknowledgements
This work has been supported by the SeconHands project, funded from the EU Horizon 2020 Research and Innovation programme under grant agreement No 643950.