Shadow Puppetry
The shadow puppet infers full 3D body pose and orientation from a sequence of silhouettes. It provides a low-cost form of motion capture that could be used for video games, film special effects, and surveillance. Simple and robust computer vision algorithms compute a sequence of silhouettes from video input. The shadow puppet infers a 3D motion sequence that is most consistent with the silhouettes and with prior knowledge about how the body moves. Everything is learned from data. If the puppet is trained on stylistic motion, it will infer motion in that style, e.g., if trained on a good dancer, when observing a poor dancer it will synthesize more graceful motion.
Background & Objective: Motion capture data is the basis for a wide variety of film and video special effects, as well as for synthetic characters if videogames and virtual spaces. It is also extremely expensive: Full-body motion capture systems cost $50K and up, and often produce "dirty" data that requires days of cleaning by highly skilled technicians. Our goal is to produce high-quality data using a PC with low-cost consumer-quality video input.
Technical Discussion: We learn the natural dynamics of the whole body by modeling existing motion capture data with an entropically estimated hidden Markov model. Entropic estimation produces a compact, sparse, and minimally ambiguous state machine, essentially discovering key body poses, dynamics, and timing. This model captures the kinematics, dynamics, and style of the original performers. The model is made to observe both the 3D data and simple statistics of its 2D silhouettes. These silhouettes can be reliably extracted from video if the background is stable. Given new video data, the system calculates a trajectory through 3D configuration space that is maximally compatible with the learned body dynamics and with the newly observed silhouettes, essentially synthesizing motion capture data.
Technology Areas:
Computer Vision
Artificial Intelligence
Graphics
Modification Date: June 26, 2001