Learning to dress: synthesizing human dressing motion via deep reinforcement learning

Abstract

Creating animation of a character putting on clothing is challenging due to the complex interactions between the character and the simulated garment. We take a model-free deep reinforcement learning (deepRL) approach to automatically discovering robust dressing control policies represented by neural networks. While deepRL has demonstrated several successes in learning complex motor skills, the data-demanding nature of the learning algorithms is at odds with the computationally costly cloth simulation required by the dressing task. This paper is the first to demonstrate that, with an appropriately designed input state space and a reward function, it is possible to incorporate cloth simulation in the deepRL framework to learn a robust dressing control policy. We introduce a salient representation of haptic information to guide the dressing process and utilize it in the reward function to provide learning signals during training. In order to learn a prolonged sequence of motion involving a diverse set of manipulation skills, such as grasping the edge of the shirt or pulling on a sleeve, we find it necessary to separate the dressing task into several subtasks and learn a control policy for each subtask. We introduce a policy sequencing algorithm that matches the distribution of output states from one task to the input distribution for the next task in the sequence. We have used this approach to produce character controllers for several dressing tasks: putting on a t-shirt, putting on a jacket, and robot-assisted dressing of a sleeve.

References

1. Yunfei Bai, Wenhao Yu, and C Karen Liu. 2016. Dexterous manipulation of cloth. In Computer Graphics Forum, Vol. 35.
2. Benjamin Balaguer and Stefano Carpin. 2010. Motion planning for cooperative manipulators folding flexible planar objects. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on. IEEE, 3842--3847.
3. Dmitry Berenson. 2013. Manipulation of deformable objects without modeling and simulating deformation. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 4525--4532.
4. Alexander Clegg, Jie Tan, Greg Turk, and C. Karen Liu. 2015. Animating Human Dressing. ACM Trans. Graph. 34, 4, Article 116 (July 2015), 9 pages.
5. Alexander Clegg, Wenhao Yu, Zackory M. Erickson, C. Karen Liu, and Greg Turk. 2017. Learning to Navigate Cloth using Haptics. CoRR abs/1703.06905 (2017). http://arxiv.org/abs/1703.06905
6. Martin de Lasa, Igor Mordatch, and Aaron Hertzmann. 2010. Feature-based Locomotion Controllers. ACM Trans. Graph. 29, 4, Article 131 (July 2010), 10 pages.
7. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. 2016. Benchmarking Deep Reinforcement Learning for Continuous Control. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML'16). JMLR.org, 1329--1338. http://dl.acm.org/citation.cfm?id=3045390.3045531
8. Tom Erez, Yuval Tassa, and Emanuel Todorov. 2015. Simulation tools for model-based robotics: Comparison of bullet, havok, mujoco, ode and physx. In Robotics and Automation (ICRA), 2015 IEEE International Conference on. IEEE, 4397--4404.
9. Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. 2013. Flexible Muscle-based Locomotion for Bipedal Creatures. ACM Trans. Graph. 32, 6, Article 206 (Nov. 2013), 11 pages.
10. Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).
11. Edmond SL Ho and Taku Komura. 2009. Character motion synthesis by topology coordinates. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 299--308.
12. Jessica K. Hodgins, Wayne L. Wooten, David C. Brogan, and James F. O'Brien. 1995. Animating Human Athletics. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, NY, USA, 71--78.
13. Sumit Jain, Yuting Ye, and C. Karen Liu. 2009. Optimization-Based Interactive Motion Synthesis. ACM Transaction on Graphics 28, 1 (2009), 1--10.
14. Yifeng Jiang and C. Karen Liu. 2017. Data-Driven Approach to Simulating Realistic Human Joint Constraints. CoRR abs/1709.08685 (2017). arXiv:1709.08685 http://arxiv.org/abs/1709.08685
15. Jeongseok Lee, Michael Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha Srinivasa, Mike Stilman, and C Karen Liu. 2018. DART: Dynamic Animation and Robotics Toolkit. 3 (02 2018), 500.
16. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
17. Libin Liu and Jessica Hodgins. 2017. Learning to schedule control fragments for physics-based characters using deep q-learning. ACM Transactions on Graphics (TOG) 36, 3 (2017), 29.
18. Libin Liu, Michiel Van De Panne, and Kangkang Yin. 2016. Guided Learning of Control Graphs for Physics-Based Characters. ACM Trans. Graph. 35, 3, Article 29 (May 2016), 14 pages.
19. Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae-Yong Kim. 2014. Unified Particle Physics for Real-time Applications. ACM Trans. Graph. 33, 4, Article 153 (July 2014), 12 pages.
20. Eder Miguel, Andrew Feng, Yuyu Xu, Ari Shapiro, Rasmus Tamstorf, Derek Bradley, Sara C Schvartzman, Bernhard Thomaszewsky, Bernd Bickel, Wojciech Matusik, et al. 2014. Towards cloth-manipulating characters. In Computer Animation and Social Agents, Vol. 3.
21. Stephen Miller, Jur Van Den Berg, Mario Fritz, Trevor Darrell, Ken Goldberg, and Pieter Abbeel. 2012. A geometric approach to robotic laundry folding. The International Journal of Robotics Research 31, 2 (2012), 249--267.
22. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.
23. Kourosh Naderi, Joose Rajamäki, and Perttu Hämäläinen. 2017. Discovering and Synthesizing Humanoid Climbing Movements. ACM Trans. Graph. 36, 4, Article 43 (July 2017), 11 pages.
24. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. ACM Transactions on Graphics (Proc. SIGGRAPH 2018 - to appear) 37, 4 (2018).
25. John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, and Pieter Abbeel. 2015a. Trust region policy optimization. CoRR, abs/1502.05477 (2015).
26. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015b. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 (2015).
27. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature 529, 7587 (2016), 484--489.
28. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. Nature 550, 7676 (2017), 354.
29. Tomoya Tamei, Takamitsu Matsubara, Akshara Rai, and Tomohiro Shibata. 2011. Reinforcement learning of clothing assistance with a dual-arm robot. In Humanoid Robots (Humanoids), 2011 11th IEEE-RAS International Conference on. IEEE, 733--738.
30. Jur Van Den Berg, Stephen Miller, Ken Goldberg, and Pieter Abbeel. 2010. Gravity-based robotic cloth folding. In Algorithmic Foundations of Robotics IX. Springer, 409--424.
31. He Wang, Kirill A Sidorov, Peter Sandilands, and Taku Komura. 2013. Harmonic parameterization by electrostatics. ACM Transactions on Graphics (TOG) 32, 5 (2013), 155.
32. KangKang Yin, Kevin Loken, and Michiel van de Panne. 2007. SIMBICON: simple biped locomotion control. ACM Trans. on Graphics (SIGGRAPH) 26, 3 (2007), 105.
33. Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-energy Locomotion. ACM Transactions on Graphics (Proc. SIGGRAPH 2018 - to appear) 37, 4 (2018).

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

  • Machine learning

    • Learning paradigms

      • Reinforcement learning

    • Machine learning approaches

      • Neural networks