• 
• 
• 
• 
• 
• 

Visual Tracking of Flexible 3D Surfaces

The flexible tracker is a single-camera computer vision system that estimates the 3D translation, rotation, and deformations of nonrigid surfaces such as faces. The input is low-resolution low-quality video and the output is a just a handful of parameters encoding the motion and deformation. This is ideal for animation and for highly compressed video transmission.

Background & Objective:  Reliable tracking of faces and other nonrigid 3D objects will be an important enabling technology for entertainment (animation and special effects), communication (low bit-rate video coding and related MPEG-4) and surveillance (people tracking, biometrics, and resolution enhancement). Our goal is to produce a tracker that works with a single low-resolution low-quality camera and yet produces high-quality 3D tracking information, including precise coding of surface deformations such as the motion of an eyebrow.

Technical Discussion:  We developed a linear framework for model-based tracking of nonrigid 3D objects and for acquiring such models from video. 3D motions and deformations are calculated directly from image intensities without information-lossy intermediate results. Measurement uncertainty is quantified and fully propagated through the inverse model to yield posterior mean and/or mode pose estimates. A Bayesian framework manages uncertainty, accommodates priors, and gives confidence measures. We obtained highly accurate and robust closed-form motion estimators by minimizing information loss from non-reversible (inner-product and least-squares) operations, and, when unavoidable, performing such operations with the appropriate error norm. For model acquisition, we can refine a crude or generic model to fit the video subject. Demonstrated uses are 3D tracking, 3D model refinement, and super-resolution texture lifting from low-quality low-resolution video.

Technology Areas:
Computer Vision
Graphics

Modification Date:  July 18, 2002