Sphere-meshes for real-time hand modeling and tracking

Abstract

Modern systems for real-time hand tracking rely on a combination of discriminative and generative approaches to robustly recover hand poses. Generative approaches require the specification of a geometric model. In this paper, we propose a the use of sphere-meshes as a novel geometric representation for real-time generative hand tracking. How tightly this model fits a specific user heavily affects tracking precision. We derive an optimization to non-rigidly deform a template model to fit the user data in a number of poses. This optimization jointly captures the user's static and dynamic hand geometry, thus facilitating high-precision registration. At the same time, the limited number of primitives in the tracking template allows us to retain excellent computational performance. We confirm this by embedding our models in an open source real-time registration algorithm to obtain a tracker steadily running at 60Hz. We demonstrate the effectiveness of our solution by qualitatively and quantitatively evaluating tracking precision on a variety of complex motions. We show that the improved tracking accuracy at high frame-rate enables stable tracking of extended and complex motion sequences without the need for per-frame re-initialization. To enable further research in the area of high-precision hand tracking, we publicly release source code and evaluation datasets.

References

1. Albrecht, I., Haber, J., and Seidel, H.-P. 2003. Construction and animation of anatomically based human hand models. In Proc. of the Symposium on Computer Animation (SCA).
2. Ballan, L., Taneja, A., Gall, J., Van Gool, L., and Pollefeys, M. 2012. Motion capture of hands in action using discriminative salient points. In Proc. of the European Conf. on Computer Vision (ECCV).
3. Bloomenthal, J., and Shoemake, K. 1991. Convolution surfaces. In Computer Graphics (Proc. SIGGRAPH).
4. Bloomenthal, J., Bajaj, C., Blinn, J., Cani, M.-P., Rockwood, A., Wyvill, B., and Wyvill, G. 1997. Introduction to Implicit Surfaces. Morgan Kaufmann.
5. Botsch, M., Kobbelt, L., Pauly, M., Alliez, P., and Lévy, B. 2010. Polygon Mesh Processing. A. K. Peters.
6. Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Transactions on Graphics (Proc. of SIGGRAPH).
7. Buss, S. R. 2004. Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. IEEE Journal of Robotics and Automation.
8. de La Gorce, M., Fleet, D. J., and Paragios, N. 2011. Model-based 3D hand pose estimation from monocular video. Pattern Analysis and Machine Intelligence (PAMI).
9. Dipietro, L., Sabatini, A. M., and Dario, P. 2008. A survey of glove-based systems and their applications. IEEE Transactions on Systems, Man, and Cybernetics.
10. Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., and Twombly, X. 2007. Vision Based Hand Pose Estimation: A Review. Computer Vision Image Understanding.
11. Fleishman, S., Kliger, M., Lerner, A., and Kutliroff, G. 2015. Icpik: Inverse kinematics based articulated-icp. In Proc. of the IEEE CVPR Workshops (HANDS).
12. Innmann, M., Zollhöfer, M., Niessner, M., Theobalt, C., and Stamminger, M. 2016. Volumedeform: Real-time volumetric non-rigid reconstruction. In Proc. of the European Conf. on Computer Vision (ECCV).
13. Keskin, C., Kiraç, F., Kara, Y. E., and Akarun, L. 2012. Hand pose estimation and hand shape classification using multilayered randomized decision forests. In Proc. of the European Conf. on Computer Vision (ECCV).
14. Khamis, S., Taylor, J., Shotton, J., Keskin, C., Izadi, S., and Fitzgibbon, A. 2015. Learning an efficient model of hand shape variation from depth images. Proc. of Comp. Vision and Pattern Recog. (CVPR).
15. Krupka, E., Vinnikov, A., Klein, B., Bar Hillel, A., Freedman, D., and Stachniak, S. 2014. Discriminative ferns ensemble for hand pose recognition. In Proc. of Comp. Vision and Pattern Recog. (CVPR).
16. Loper, M. M., and Black, M. J. 2014. OpenDR: An approximate differentiable renderer. In Proc. of the European Conf. on Computer Vision (ECCV).
17. Makris, A., and Argyros, A. A. 2015. Model-based 3d hand tracking with on-line shape adaptation. In Proc. of British Machine Vision Conference (BMVC).
18. Melax, S., Keselman, L., and Orsten, S. 2013. Dynamics based 3d skeletal hand tracking. In Proc. of Graphics Interface.
19. Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohli, P., Shotton, J., Hodges, S., and Fitzgibbon, A. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR).
20. Newcombe, R. A., Fox, D., and Seitz, S. M. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. Proc. of Comp. Vision and Pattern Recog. (CVPR).
21. Oberweger, M., Wohlhart, P., and Lepetit, V. 2015. Training a Feedback Loop for Hand Pose Estimation. In Proc. of the Intern. Conf. on Computer Vision (ICCV).
22. Oikonomidis, I., Kyriazis, N., and Argyros, A. A. 2011. Efficient model-based 3D tracking of hand articulation using kinect. In Proc. of British Machine Vision Conference (BMVC).
23. Olson, M., and Zhang, H. 2006. Silhouette extraction in hough space. Computer Graphics Forum (Proc. of EuroGraphics).
24. Plankers, R., and Fua, P. 2003. Articulated soft objects for multi-view shape and motion capture. Pattern Analysis and Machine Intelligence (PAMI).
25. Qian, C., Sun, X., Wei, Y., Tang, X., and Sun, J. 2014. Realtime and robust hand tracking from depth. In Proc. of Comp. Vision and Pattern Recog. (CVPR).
26. Rehg, J. M., and Kanade, T. 1994. Visual tracking of high dof articulated structures: An application to human hand tracking. In Proc. of the European Conf. on Computer Vision (ECCV).
27. Rhee, T., Neumann, U., and Lewis, J. P. 2006. Human hand modeling from surface anatomy. In Proc. ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (i3D).
28. Schroder, M., Maycock, J., Ritter, H., and Botsch, M. 2014. Real-time hand tracking using synergistic inverse kinematics. In Proc. of the Intern. Conf. on Robotics and Automation (ICRA).
29. Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., Wei, A. V. Y., Krupka, D. F. P. K. E., Fitzgibbon, A., and Izadi, S. 2015. Accurate, robust, and flexible real-time hand tracking. In Proc. of ACM Special Interest Group on Computer-Human Interaction (CHI).
30. Sridhar, S., Oulasvirta, A., and Theobalt, C. 2013. Interactive markerless articulated hand motion tracking using RGB and depth data. In Proc. of the Intern. Conf. on Computer Vision (ICCV).
31. Sridhar, S., Rhodin, H., Seidel, H.-P., Oulasvirta, A., and Theobalt, C. 2014. Real-time hand tracking using a sum of anisotropic gaussians model. In Proc. International Conference on 3D Vision (3DV).
32. Sridhar, S., Mueller, F., Oulasvirta, A., and Theobalt, C. 2015. Fast and robust hand tracking using detection-guided optimization. In Proc. of Comp. Vision and Pattern Recog. (CVPR).
33. Sun, X., Wei, Y., Liang, S., Tang, X., and Sun, J. 2015. Cascaded hand pose regression. In Proc. of Comp. Vision and Pattern Recog. (CVPR).
34. Tagliasacchi, A., and Li, H. 2016. Modern techniques and applications for real-time non-rigid registration. Proc. SIGGRAPH Asia (Technical Courses).
35. Tagliasacchi, A., Schroeder, M., Tkach, A., Bouaziz, S., Botsch, M., and Pauly, M. 2015. Robust articulated-icp for real-time hand tracking. Computer Graphics Forum (Proc. Symposium on Geometry Processing, SGP).
36. Tagliasacchi, A., Delame, T., Spagnuolo, M., Amenta, N., and Telea, A. 2016. 3d skeletons: A state-of-the-art report. Computer Graphics Forum (Proc. of EuroGraphics).
37. Tan, D. J., Cashman, T., Taylor, J., Fitzgibbon, A., Tarlow, D., Khamis, S., Izadi, S., and Shotton, J. 2016. Fits like a glove: Rapid and reliable hand shape personalization. In Proc. of Comp. Vision and Pattern Recog. (CVPR).
38. Tang, D., Yu, T.-H., and Kim, T.-K. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proc. of the Intern. Conf. on Computer Vision (ICCV).
39. Taylor, J., Stebbing, R., Ramakrishna, V., Keskin, C., Shotton, J., Izadi, S., Hertzmann, A., and Fitzgibbon, A. 2014. User-specific hand modeling from monocular depth sequences. In Proc. of Comp. Vision and Pattern Recog. (CVPR).
40. Taylor, J., Bordeaux, L., Cashman, T., Corish, B., Keskin, C., Soto, E., Sweeney, D., Valentin, J., Luff, B., Topalian, A., Wood, E., Khamis, S., Kohli, P., Sharp, T., Izadi, S., Banks, R., Fitzgibbon, A., and Shotton, J. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Transactions on Graphics (Proc. of SIGGRAPH).
41. Tejani, A., Tang, D., Kouskouridas, R., and Kim, T.-K. 2014. Latent-class hough forests for 3d object detection and pose estimation. In Proc. of the European Conf. on Computer Vision (ECCV).
42. Thiery, J.-M., Guy, E., and Boubekeur, T. 2013. Spheremeshes: Shape approximation using spherical quadric error metrics. ACM Transactions on Graphics (Proc. of SIGGRAPH).
43. Thiery, J.-M., Guy, E., Boubekeur, T., and Eisemann, E. 2016. Animated mesh approximation with sphere-meshes. ACM Transactions on Graphics (TOG).
44. Tompson, J., Stein, M., Lecun, Y., and Perlin, K. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (TOG).
45. Vaillant, R., Barthe, L., Guennebaud, G., Cani, M.-P., Rohmer, D., Wyvill, B., Gourmel, O., and Paulin, M. 2013. Implicit skinning: Real-time skin deformation with contact modeling. ACM Transactions on Graphics (Proc. of SIGGRAPH).
46. Vaillant, R., Guennebaud, G., Barthe, L., Wyvill, B., and Cani, M.-P. 2014. Robust iso-surface tracking for interactive character skinning. ACM Transactions on Graphics (TOG).
47. Wang, R. Y., and Popovic, J. 2009. Real time hand tracking with a colored glove. ACM Transactions on Graphics (Proc. of SIGGRAPH).
48. Welch, G., and Foxlin, E. 2002. Motion tracking: No silver bullet, but a respectable arsenal. IEEE Comput. Graph. Appl..

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Tracking

      • Computer vision representations

        • Shape representations

      • Image and video acquisition

        • Motion capture

  • Computer graphics

    • Shape modeling

      • Volumetric models