End-User Programming of Manipulator Robots in Situated Tangible Programming Paradigm

ABSTRACT

While the cost of creating robots is declining, deploying them in industry remains expensive. Widespread use of robots, particularly in smaller industries, is more easily realized if robot programming is accessible to non-programmers. Our research explores techniques to lower the barrier to robot programming. In one such attempt, we propose situated tangible robot programming to program a robot by placing specially designed tangible blocks in its workspace. These blocks are used for annotating objects, locations, or regions, and specifying actions and their ordering. The robot compiles a program by detecting blocks and objects in the environment and grouping them into instructions by solving constraints. We designed a preliminary tangible language and blocks and evaluated the intuitiveness and learnability of the approach. Our user studies provide evidence for the promise of situated tangible programming and identify the challenges to address. In addition to improving the block design and extending the language, we are planning to integrate tangible programming into a holistic ecosystem of a programming environment in future.

References

1. Universal robots user manual. Universal robots website. https://universal-robots.com/media/8704/ur5_user_manual_gb.pdf, 2014.
2. B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz. Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, pages 391--398, 2012.
3. S. Alexamdrova, M. Cakmak, K. Hsaio, and L. Takayama. Robot programming by demonstration with interactive action visualizations. In Robotics: Science and Systems (RSS), 2014.
4. S. Alexandrova, Z. Tatlock, and M. Cakmak. Roboflow: A flow-based visual programming language for mobile manipulation tasks. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5537--5544. IEEE, 2015.
5. D. Anderson, J. L. Frankel, J. Marks, D. Leigh, E. Sullivan, J. Yedidia, and K. Ryall. Building virtual structures with physical blocks. In Proceedings of the 12th annual ACM symposium on User interface software and technology, pages 71--72. ACM, 1999.
6. M. Burnett. Visual programming. Wiley Encyclopedia of Electrical and Electronics Engineering, 1999.
7. R. Cantrell, P. Schermerhorn, and M. Scheutz. Learning actions from human-robot dialogues. In 2011 RO-MAN, pages 125--130. IEEE, 2011.
8. R. Fang, M. Doering, and J. Y. Chai. Embodied collaborative referring expression generation in situated human-robot interaction. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 271--278. ACM, 2015.
9. K. Fischer, F. Kirstein, L. C. Jensen, N. Kr, K. Kukli, M. V. aus der Wieschen, T. R. Savarimuthu, et al. A comparison of types of robot control for programming by demonstration. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 213--220. IEEE, 2016.
10. D. Gallardo, C. F. Julià, and S. Jorda. Turtan: A tangible programming language for creative exploration. In Tabletop, pages 89--92. Citeseer, 2008.
11. D. F. Glas, T. Kanda, and H. Ishiguro. Human-robot interaction design using interaction composer: Eight years of lessons learned. In The Eleventh ACM/IEEE International Conference on Human Robot Interation, pages 303--310. IEEE Press, 2016.
12. M. S. Horn and R. J. Jacob. Designing tangible programming languages for classroom use. In Proceedings of the 1st international conference on Tangible and embedded interaction, pages 159--162. ACM, 2007.
13. J. Huang, T. Lau, and M. Cakmak. Design and evaluation of a rapid programming system for service robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 295--302. IEEE, 2016.
14. T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on, pages 259--266. IEEE, 2010.
15. S. Lauria, G. Bugmann, T. Kyriacou, and E. Klein. Mobile robot programming using natural language. Robotics and Autonomous Systems, 38(3):171--181, 2002.
16. T. Lozano-Perez. Robot programming. Proceedings of the IEEE, 71(7):821--841, 1983.
17. M. Luria, G. Hoffman, B. Megidish, O. Zuckerman, and S. Park. Designing vyo, a robotic smart home assistant: Bridging the gap between device and social agent. In Robot and Human Interactive Communication (RO-MAN), 2016 25th IEEE International Symposium on, pages 1019--1025. IEEE, 2016.
18. C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In AAAI, pages 2556--2563, 2014.
19. C. Matuszek, D. Fox, and K. Koscher. Following directions using statistical machine translation. In Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction, pages 251--258. IEEE Press, 2010.
20. S. Mohn and J. Laird. Learning goal-oriented hierarchical tasks from situated interactive instruction. In Proceedings of the The Twenty-eighth National Conference on Artificial Intelligence (AAAI), 2014.
21. A. Mohseni-Kabir, C. Rich, S. Chernova, C. L. Sidner, and D. Miller. Interactive hierarchical task learning from a single demonstration. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 205--212. ACM, 2015.
22. H. Newton-Dunn, H. Nakano, and J. Gibson. Block jam: a tangible interface for interactive music. In Proceedings of the 2003 conference on New interfaces for musical expression, pages 170--177. National University of Singapore, 2003.
23. S. Niekum, S. Chitta, A. Barto, B. Marthi, and S. Osentoski. Incremental semantically grounded learning from demonstration. In Robotics: Science and Systems, volume 9, 2013.
24. S. Niekum, S. Osentoski, G. Konidaris, and A. G. Barto. Learning and generalization of complex tasks from unstructured demonstrations. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5239--5246, 2012.
25. T. Sapounidis, S. Demetriadis, and I. Stamelos. Evaluating children performance with graphical and tangible robot programming tools. Personal and Ubiquitous Computing, 19(1):225--237, 2015.
26. E. Schweikardt and M. D. Gross. roblocks: a robotic construction kit for mathematics and science education. In Proceedings of the 8th international conference on Multimodal interfaces, pages 72--75. ACM, 2006.
27. L. She, Y. Cheng, J. Y. Chai, Y. Jia, S. Yang, and N. Xi. Teaching robots new actions through natural language instructions. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 868--873. IEEE, 2014.
28. A. Sipitakiat and N. Nusen. Robo-blocks: designing debugging abilities in a tangible programming system for early primary school children. In Proceedings of the 11th International Conference on Interaction Design and Children, pages 98--105. ACM, 2012.
29. M. Tykal, A. Montebelli, and V. Kyrki. Incrementally assisted kinesthetic teaching for programming by demonstration. In The Eleventh ACM/IEEE International Conference on Human Robot Interation, pages 205--212. IEEE Press, 2016.
30. S. Zhao, K. Nakamura, K. Ishii, and T. Igarashi. Magic cards: a paper tag interface for implicit robot control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 173--182. ACM, 2009.

Index Terms

• Human-centered computing

  • Interaction design

    • Interaction design process and methods

      • User centered design