Abstract:

The paper presents a method for learning state representations of articulated mechanisms consisting of multiple links connected by joints but not equipped with positional encoders, from short sequences of visual observations of the mechanism collected by stationary RGB-D cameras.
The method leverages recent advances in algorithms based on deep neural networks for long-term keypoint tracking in image sequences to extract tracks of matching keypoints over time, even when some of them are temporarily occluded.
Because keypoint tracking is not perfectly reliable, we propose a novel method for robust clustering to discover how many rigid bodies (links) the mechanism consists of by analyzing which points move together and thus must belong to the same rigid body. Experiments with recently proposed longterm keypoint trackers demonstrate successful state estimation of the kinematic structure of articulated mechanisms entirely from camera images, effectively allowing computation of inverse kinematics and visual servocontrol for mechanisms such as robot arms, cranes, etc. even without positional encoders.