Learning to track 3D human motion from silhouettes

ABSTRACT

We describe a sparse Bayesian regression method for recovering 3D human body motion directly from silhouettes extracted from monocular video sequences. No detailed body shape model is needed, and realism is ensured by training on real human motion capture data. The tracker estimates 3D body pose by using Relevance Vector Machine regression to combine a learned autoregressive dynamical model with robust shape descriptors extracted automatically from image silhouettes. We studied several different combination methods, the most effective being to learn a nonlinear observation-update correction based on joint regression with respect to the predicted state and the observations. We demonstrate the method on a 54-parameter full body pose model, both quantitatively using motion capture based test sequences, and qualitatively on a test video sequence.

REFERENCES

• 1: Agarwal, A., & Triggs, B. (2004a). 3D Human Pose from Silhouettes by Relevance Vector Regression. Int. Conf. Computer Vision & Pattern Recognition.
• 2: Agarwal, A., & Triggs, B. (2004b). Tracking Articulated Motion with Piecewise Learned Dynamical Models. European Conf. Computer Vision.
• 3: Athitsos, V., & Sclaroff, S. (2000). Inferring Body Pose without Tracking Body Parts. Int. Conf. Computer Vision & Pattern Recognition.
• 4: Athitsos, V., & Sclaroff, S. (2003). Estimating 3D Hand Pose From a Cluttered Image. Int. Conf. Computer Vision.
• 5: S. Belongie , J. Malik , J. Puzicha, Shape Matching and Object Recognition Using Shape Contexts, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.4, p.509-522, April 2002 [doi>10.1109/34.993558 ]
• 6: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 7: Matthew Brand, Shadow Puppetry, Proceedings of the International Conference on Computer Vision-Volume 2, p.1237, September 20-25, 1999
• 8: C. Bregler , J. Malik, Tracking People with Twists and Exponential Maps, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.8, June 23-25, 1998
• 9: D'Souza, A., Vijayakumar, S., & Schaal, S. (2001). Learning Inverse Kinematics. Int. Conf. on Intelligent Robots and Systems.
• 10: Kristen Grauman , Gregory Shakhnarovich , Trevor Darrell, Inferring 3D Structure with a Statistical Image-Based Shape Model, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.641, October 13-16, 2003
• 11: Howe, N., Leventon, M., & Freeman, W. (1999). Bayesian Reconstruction of 3D Human Motion from Single-Camera Video. Neural Information Processing Systems.
• 12: Frédéric Jurie , Michel Dhome, Hyperplane Approximation for Template Matching, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.7, p.996-1000, July 2002 [doi>10.1109/TPAMI.2002.1017625 ]
• 13: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 14: Greg Mori , Jitendra Malik, Estimating Human Body Configurations Using Shape Context Matching, Proceedings of the 7th European Conference on Computer Vision-Part III, p.666-680, May 28-31, 2002
• 15: Ormoneit, D., Sidenbladh, H., Black, M., & Hastie, T. (2000). Learning and Tracking Cyclic Human Motion. Neural Information Processing Systems (pp. 894--900).
• 16: Pavlovic, V., Rehg, J., & MacCormick, J. (2000). Learning Switching Linear Models of Human Motion. Neural Information Processing Systems (pp. 981--987).
• 17: Yossi Rubner , Carlo Tomasi , Leonidas J. Guibas, A Metric for Distributions with Applications to Image Databases, Proceedings of the Sixth International Conference on Computer Vision, p.59, January 04-07, 1998
• 18: Gregory Shakhnarovich , Paul Viola , Trevor Darrell, Fast Pose Estimation with Parameter-Sensitive Hashing, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.750, October 13-16, 2003
• 19: Hedvig Sidenbladh , Michael J. Black , Leonid Sigal, Implicit Probabilistic Models of Human Motion for Synthesis and Tracking, Proceedings of the 7th European Conference on Computer Vision-Part I, p.784-800, May 28-31, 2002
• 20: Sminchisescu, C., & Triggs, B. (2003). Kinematic Jump Processes For Monocular 3D Human Tracking. Int. Conf. Computer Vision & Pattern Recognition.
• 21: B. Stenger , A. Thayananthan , P. H. S. Torr , R. Cipolla, Filtering Using a Tree-Based Estimator, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1063, October 13-16, 2003
• 22: Taylor, C. (2000). Reconstruction of Articulated Objects from Point Correspondances in a Single Uncalibrated Image. Int. Conf. Computer Vision & Pattern Recognition.
• 23: Tipping, M. (2000). The Relevance Vector Machine. Neural Information Processing Systems.
• 24: Michael E. Tipping, Sparse bayesian learning and the relevance vector machine, The Journal of Machine Learning Research, 1, p.211-244, 9/1/2001
• 25: Oliver Williams , Andrew Blake , Roberto Cipolla, A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.353, October 13-16, 2003