skip to main content
10.1145/1180495.1180526acmconferencesArticle/Chapter ViewAbstractPublication PagesvrstConference Proceedingsconference-collections
Article

From motion capture to action capture: a review of imitation learning techniques and their application to VR-based character animation

Published: 01 November 2006 Publication History

Abstract

We present a novel method for virtual character animation that we call action capture. In this approach, virtual characters learn to imitate the actions of Virtual Reality (VR) users by tracking not only the users' movements but also their interactions with scene objects.Action capture builds on conventional motion capture but differs from it in that higher-level action representations are transferred rather than low-level motion data. As an advantage, the learned actions can often be naturally applied to varying situations, thus avoiding retargetting problems of motion capture. The idea of action capture is inspired by human imitation learning; related methods have been investigated for a longer time in robotics. The paper reviews the relevant literature in these areas before framing the concept of action capture in the context of VR-based character animation. We also present an example in which the actions of a VR user are transferred to a virtual worked.

References

[1]
J. Aleotti, S. Caselli, and M. Reggiani. Leveraging on a Virtual Environment for Robot Programming by Demonstration. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems IROS 2003, Workshop on Robot Programming by Demonstration, Las Vegas (USA), 2003.
[2]
A. Alissandrakis, C. L. Nehaniv, and K. Dautenhahn. Imitation with ALICE: Learning to Imitate Corresponding Actions Across Dissimilar Embodiments. IEEE Trans. Systems, Man and Cybernetics, 32(4):482--296, 2002.
[3]
M. A. Arbib. The mirror system, imitation, and the evolution of language. In Dautenhahn and Nehaniv
[4]
P. Bakker and Y. Kuniyoshi. Robot see, Robot do: An Overview of Robot Imitation. In AISB96 Workshop: Learning in Robots and Animals, pages 3--11, 1996.
[5]
A. Billard and R. Siegwart, editors. Special Issue on Robot Learning from Demonstration, volume 47 of Robotics and Autonomous Systems, 2004.
[6]
D. Buchsbaum and B. Blumberg. Imitation as a First Step to Social Learning In Synthetic Characters: A Graph-based Approach. In SCA '05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pages 9--18, New York, NY, USA, 2005. ACM Press.
[7]
K. Dautenhahn and C. Nehaniv, editors. Imitation in Animals and Artifacts. MIT Press, 2002.
[8]
R. Dillmann, M. Ehrenmann, P. Steinhaus, O. Rogalla, and R. Zöllner. Human Friendly Programming of Humanoid Robots: The German Collaborative Research Center. In IARP 2002, 2002.
[9]
M. Gleicher. Retargetting Motion to New Characters. In SIGGRAPH'98 Conference Proceedings, Computer Graphics Annual Conference Series, pages 33--42. ACM, 1998.
[10]
G. Heumer, H. Ben Amor, M. Weber, and B. Jung. Calibration-free Recognition of Grasp Types - A Comparison of Classification Methods. In Proceedings Dritter Workshop Virtuelle und Erweiterte Realität der GI-Fachgruppe VR/AR, 2006.
[11]
A. Ijspeert, J. Nakanishi, and S. Schaal. Movement Imitation with Nonlinear Dynamical Systems in Humanoid Robots. In IEEE International Conference on Robotics and Automation, 2002.
[12]
S. Kopp and O. Graeser. Imitation Learning and Response Facilitation in Embodied Agents. In Intelligent Virtual Agents 2006, LNAI, pages 28--41. Springer-Verlag, Berlin, 2006.
[13]
L. Kovar, M. Gleicher, and F. Pighin. Motion Graphs. ACM Transactions on Graphics, 21(3), 2002.
[14]
M. Mataric. Getting Humanoids to Move and Imitate. IEEE Intelligent Systems, pages 18--24, July/August 2000.
[15]
M. Mataric. Sensory-motor primitives as a basis for learning by imitation: Linking perception to action and biology to robotics. In Dautenhahn and Nehaniv {7}.
[16]
A. N. Meltzoff. The Human Infant as Imitative Generalist: A 20-year Progress Report on Infant Imitation with Implications for Comparative Psychology. In Social Learning in Animals: The Roots of Culture, pages 347--370, 1996.
[17]
J. Nakanishi, J. Morimoto, G. Endo, G. Cheng, S. Schaal, and M. Kawato. Learning from Demonstration and Adaptation of Biped Locomotion. Robotics and Autonomous Systems, 47(2-3):79--91, 2004.
[18]
C. Nehaniv and K. Dautenhahn. The Correspondence Problem. In Dautenhahn and Nehaniv {7}, pages 41--61.
[19]
E. Oztop and M. Arbib. Schema design and implementation of the grasp-related mirror neuron system. Biological Cybernetics, 87:116--140, 2002.
[20]
J. Piaget. Play, Dreams and Imitation in Childhood. New York: W. W. Norton, 1962.
[21]
R. Rao, A. P. Shon, and A. N. Meltzoff. A Bayesian Model of Imitation in Infants and Robots. In Imitation and Social Learning in Robots, Humans and Animals: Behavioural, Social and Communicative Dimensions, 2004.
[22]
C. Reynolds. Flocks, Herds and Schools: A Distributed Behavioural Model. Computer Graphics, 21(4):25--34, 1987.
[23]
G. Rizzolatti, L. Fogassi, and V. Gallese. Neurophysiological Mechanisms Underlying the Understanding and Imitation of Action. Nature Reviews Neuroscience, pages 661--770, September 2001.
[24]
G. Schlesinger. Der Mechanische Aufbau der Künstlichen Glieder. In M. Borchardt et al., editors, Ersatzglieder und Arbeitshilfen für Kriegsbeschädigte und Unfallverletzte, pages 321--661. Springer-Verlag: Berlin, Germany, 1919.
[25]
C. Taylor and R. Schwarz. The Anatomy and Mechanics of the Human Hand. Artificial Limbs, 2:22--35, 1955.
[26]
E. L. Thorndike. Animal Intelligence: An Experimental Study of the Associative Processes in Animals. Psychological Review Monographs, 8, 1898.
[27]
B. Tomlinson. From Linear to Interactive Animation: How Autonomous Characters Change the Process and Product of Animating. ACM Computers In Entertainment, 3(1), 2005.
[28]
A. Witkin and M. Kass. Spacetime Constraints. In SIGGRAPH'88 Conference Proceedings, volume 22, pages 159--168. ACM, 1988.
[29]
V. B. Zordan, A. Majkowska, B. Chiu, and M. Fast. Dynamic Response for Motion Capture Animation. ACM Transactions on Graphics, 24(3):697--701, 2005.

Cited By

View all
  • (2023)Record, Review, Edit, Apply: A Motion Data Pipeline for Virtual Reality Development & DesignProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3587191(1-4)Online publication date: 12-Apr-2023
  • (2020)Feasible Stylized MotionProceedings of the 7th International Conference on Movement and Computing10.1145/3401956.3404188(1-8)Online publication date: 15-Jul-2020
  • (2020)Virtual Observations: a software tool for contextual observation and assessment of user’s actions in virtual realityVirtual Reality10.1007/s10055-020-00463-5Online publication date: 19-Aug-2020
  • Show More Cited By

Index Terms

  1. From motion capture to action capture: a review of imitation learning techniques and their application to VR-based character animation

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        VRST '06: Proceedings of the ACM symposium on Virtual reality software and technology
        November 2006
        400 pages
        ISBN:1595933212
        DOI:10.1145/1180495
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 01 November 2006

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. action capture
        2. character animation
        3. imitation learning
        4. motion capture
        5. virtual reality

        Qualifiers

        • Article

        Conference

        VRST06

        Acceptance Rates

        Overall Acceptance Rate 66 of 254 submissions, 26%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)29
        • Downloads (Last 6 weeks)11
        Reflects downloads up to 18 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2023)Record, Review, Edit, Apply: A Motion Data Pipeline for Virtual Reality Development & DesignProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3587191(1-4)Online publication date: 12-Apr-2023
        • (2020)Feasible Stylized MotionProceedings of the 7th International Conference on Movement and Computing10.1145/3401956.3404188(1-8)Online publication date: 15-Jul-2020
        • (2020)Virtual Observations: a software tool for contextual observation and assessment of user’s actions in virtual realityVirtual Reality10.1007/s10055-020-00463-5Online publication date: 19-Aug-2020
        • (2019)Virtual Observation of Virtual Reality SimulationsExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3312836(1-6)Online publication date: 2-May-2019
        • (2015)3-D models merging based on cost optimizationProceedings of the 7th International Conference on Internet Multimedia Computing and Service10.1145/2808492.2808553(1-4)Online publication date: 19-Aug-2015
        • (2014)State Classification and Motion Description for the Lower Extremity Exoskeleton SJTU-EXJournal of Bionic Engineering10.1016/S1672-6529(14)60034-211:2(249-258)Online publication date: Apr-2014
        • (2010)Manual Intelligence as a Rosetta Stone for Robot CognitionRobotics Research10.1007/978-3-642-14743-2_12(135-146)Online publication date: 2010
        • (2007)Grasp Recognition with Uncalibrated Data Gloves - A Comparison of Classification Methods2007 IEEE Virtual Reality Conference10.1109/VR.2007.352459(19-26)Online publication date: Mar-2007
        • (2006)An animation system for imitation of object grasping in virtual realityProceedings of the 16th international conference on Advances in Artificial Reality and Tele-Existence10.1007/11941354_8(65-76)Online publication date: 29-Nov-2006

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media