3D from Video
This technology extracts 3D models of nonrigid surfaces such as faces directly from intensity variations in a single stream of video. The resulting model is morphable, meaning that it contains information about the natural ways in which the object changes shape, for example, how the facial tissues move as one speaks. The recovered model can be used for animation, special effects, low bit-rate video coding, biometrics, and CAD. The image at left shows 3 of the 148 original video frames and corresponding profile views that were synthesized from the recovered 3D information.
Background & Objective: Automatic acquisition of 3D shape and texture is an intensely studied problem due to the growing demand for 3D models in industries ranging from entertainment to medicine to engineering. Current methods only handle specially constrained cases, for example, rigid objects, multiple calibrated cameras, specially marked surfaces, etc. We give the first general solution for nonrigid low-texture surfaces viewed in ordinary uncalibrated video.
Technical Discussion: There are two main results: (1) A method for tracking nonrigid low-texture surfaces in video that ensures that the 2D motion is consistent with a low-dimensional space of nonrigid 3D motions. (2) A solution for factoring the image intensity variations associated with these 2D motions into 3D shape, rotations, and deformations. By discarding the deformations, one obtains the average 3D shape of the video. Or by altering the deformations, one can produce new realistic performances in 3D. These methods are intended to work with low-quality low-resolution sources such as low-end consumer video, so there is a formal emphasis on robustness.
Publications:
Technology Areas:
Computer Vision
Audio Video Processing
Graphics
Modification Date: July 18, 2002