A Survey on Multi-touch Gesture Recognition and Multi-touch Frameworks

ABSTRACT

The multi-touch gesture recognition problem has drawn great attention from the human-computer interaction (HCI) community, mainly since multi-touch surfaces and other touch-capable devices reached the mainstream market. In the past decade, several multi-touch gesture recognition techniques and multi-touch frameworks were proposed. When we started our research on touch-based gestures recognition, we identified some surveys focused on computer vision or accelerometers. However, in several multi-touch surface devices, the multi-touch sensor is the only input method available. We here present a survey on touch-based gestures recognition techniques and frameworks, and propose an extended set of requirements such techniques and frameworks should meet in order to provide better support to multi-touch surface applications.

References

1. Anthony, L., and Wobbrock, J. O. A lightweight multistroke recognizer for user interface prototypes. In Proceedings of Graphics Interface 2010, GI '10, Canadian Information Processing Society (Toronto, Ont., Canada, Canada, 2010), 245--252.
2. Anthony, L., and Wobbrock, J. O. $n-protractor: a fast and accurate multistroke recognizer. In Proceedings of Graphics Interface 2012, GI '12, Canadian Information Processing Society (Toronto, Ont., Canada, Canada, 2012), 117--120.
3. Chen, F.-S., Fu, C.-M., and Huang, C.-L. Hand gesture recognition using a real-time tracking method and hidden markov models. Image and Vision Computing 21, 8 (2003), 745--758.
4. Chen, L., Wang, F., Deng, H., and Ji, K. A survey on hand gesture recognition. In Proceedings of the 2013 International Conference on Computer Sciences and Applications, CSA '13, IEEE Computer Society (Washington, DC, USA, 2013), 313--316.
5. Clayphan, A., Collins, A., Ackad, C., Kummerfeld, B., and Kay, J. Firestorm: a brainstorming application for collaborative group work at tabletops. ITS '11, ACM (New York, NY, USA, 2011), 162--171.
6. Codeplex. MIRIA SDK, 2011.
7. Dietz, P., and Leigh, D. DiamondTouch: a multi-user touch technology. In Proceedings of the 14th annual ACM symposium on User interface software and technology, UIST '01, ACM (New York, NY, USA, 2001), 219--226.
8. Echtler, F., and Butz, A. GISpL: gestures made easy. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction, TEI '12, ACM (New York, NY, USA, 2012), 233--240.
9. Echtler, F., and Klinker, G. A multitouch software architecture. In Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges, NordiCHI '08, ACM (New York, NY, USA, 2008), 463--466.
10. Echtler, F., Klinker, G., and Butz, A. Towards a unified gesture description language. In Proceedings of the 13th International Conference on Humans and Computers, HC '10, University of Aizu Press (Fukushima-ken, Japan, Japan, 2010), 177--182.
11. Govoni, D. Java Application Frameworks, 1 ed. Wiley, June 1999.
12. Hinrichs, U., and Carpendale, S. Gestures in the wild: studying multi-touch gesture sequences on interactive tabletop exhibits. In Proceedings of the 2011 annual conference on Human factors in computing systems, CHI '11, ACM (New York, NY, USA, 2011), 3023--3032.
13. Hoste, L. Software engineering abstractions for the multi-touch revolution. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, vol. 2, IEEE (May 2010), 509--510.
14. Hoste, L., De Rooms, B., and Signer, B. Declarative gesture spotting using inferred and refined control points. In Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods (ICPRAM 2013) (Barcelona, Spain, 2013), 1--6.
15. Hoste, L., Dumas, B., and Signer, B. Mudra: A unified multimodal interaction framework. In Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI '11, ACM (New York, NY, USA, 2011), 97--104.
16. Institute, F. MT4j - multitouch for java, 2011.
17. Jord, S., Geiger, G., Alonso, M., and Kaltenbrunner, M. The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In Proceedings of the 1st international conference on Tangible and embedded interaction, TEI '07, ACM (New York, NY, USA, 2007), 139--146.
18. Kaltenbrunner, M. reacTIVision and TUIO: a tangible tabletop toolkit. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ITS '09, ACM (New York, NY, USA, 2009), 9--16.
19. Kaltenbrunner, M., Bovermann, T., Bencina, R., and Costanza, E. TUIO: A protocol for table-top tangible user interfaces. In 6th International Gesture Workshop (2005).
20. Kammer, D., Henkens, D., and Groh, R. GeForMTjs: A javascript library based on a domain specific language for multi-touch gestures. In Proceedings of the 12th International Conference on Web Engineering, ICWE'12, Springer-Verlag (Berlin, Heidelberg, 2012), 444--447.
21. Kammer, D., Keck, M., Freitag, G., and Wacker, M. Taxonomy and overview of multi-touch frameworks: Architecture, scope and features. In Workshop on Engineering Patterns for Multitouch Interfaces (2010).
22. Kammer, D., Wojdziak, J., Keck, M., Groh, R., and Taranko, S. Towards a formalization of multi-touch gestures. ITS '10, ACM (New York, NY, USA, 2010), 49--58.
23. Khandkar, S. H., and Maurer, F. A domain specific language to define gestures for multi-touch applications. In Proceedings of the 10th Workshop on Domain-Specific Modeling, DSM '10, ACM (New York, NY, USA, 2010), 2:1--2:6.
24. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M. Proton++: a customizable declarative multitouch framework. In Proceedings of the 25th annual ACM symposium on User interface software and technology, UIST '12, ACM (New York, NY, USA, 2012), 477--486.
25. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M. Proton: multitouch gestures as regular expressions. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, ACM (New York, NY, USA, 2012), 2885--2894.
26. Kruger, R., Carpendale, S., Scott, S. D., and Greenberg, S. How people use orientation on tables: comprehension, coordination and communication. In Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work, GROUP '03, ACM (New York, NY, USA, 2003), 369--378.
27. L, H., and Li, Y. Gesture coder: a tool for programming multi-touch gestures by demonstration. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, ACM (New York, NY, USA, 2012), 2875--2884.
28. LaViola, Jr., J. J. A survey of hand posture and gesture recognition techniques and technology. Tech. rep., Brown University, Providence, RI, USA, 1999.
29. Li, Y. Protractor: a fast and accurate gesture recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '10, ACM (New York, NY, USA, 2010), 2169--2172.
30. Mindstorm. Breezemultitouch, 2010.
31. Mitra, S., and Acharya, T. Gesture recognition: A survey. Trans. Sys. Man Cyber Part C 37, 3 (May 2007), 311--324.
32. Morris, M. R., Huang, A., Paepcke, A., and Winograd, T. Cooperative gestures: multi-user gestural interactions for co-located groupware. In Proceedings of the SIGCHI conference on Human Factors in computing systems, CHI '06, ACM (New York, NY, USA, 2006), 1201--1210.
33. NUIGroup, T. Gesture definition markup language (GDML), 2009.
34. Nygard, E. S., and Thomassen, A. Multi-touch Interaction with Gesture Recognition. PhD thesis, 2010.
35. Renaux, T., Hoste, L., Marr, S., and De Meuter, W. Parallel gesture recognition with soft real-time guarantees. In Proceedings of the 2Nd Edition on Programming Systems, Languages and Applications Based on Actors, Agents, and Decentralized Control Abstractions, AGERE! '12, ACM (New York, NY, USA, 2012), 35--46.
36. Rhyne, J. R., and Wolf, C. G. Gestural interfaces for information processing applications. Tech. Rep. RC12179, IBM Corporation, Yorktown Heights, NY, 1986.
37. Rubine, D. Specifying gestures by example. SIGGRAPH Comput. Graph. 25, 4 (July 1991), 329--337.
38. Scholliers, C., Hoste, L., Signer, B., and De Meuter, W. Midas: a declarative multi-touch interaction framework. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction, TEI '11, ACM (New York, NY, USA, 2011), 49--56.
39. Scott, S. D., Sheelagh, M., Carpendale, T., and Inkpen, K. M. Territoriality in collaborative tabletop workspaces. In Proceedings of the 2004 ACM conference on Computer supported cooperative work, CSCW '04, ACM (New York, NY, USA, 2004), 294--303.
40. Shen, C., Vernier, F. D., Forlines, C., and Ringel, M. DiamondSpin: an extensible toolkit for around-the-table interaction. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI '04, ACM (New York, NY, USA, 2004), 167--174.
41. Spano, L. D., Cisternino, A., and Patern, F. A compositional model for gesture definition. In Proceedings of the 4th International Conference on Human-Centered Software Engineering, HCSE'12, Springer-Verlag (Berlin, Heidelberg, 2012), 34--52.
42. Strothoff, S., Valkov, D., and Hinrichs, K. Triangle cursor: interactions with objects above the tabletop. ITS '11, ACM (New York, NY, USA, 2011), 111--119.
43. Vatavu, R.-D., Anthony, L., and Wobbrock, J. O. Gestures as point clouds: a $p recognizer for user interface prototypes. In Proceedings of the 14th ACM international conference on Multimodal interaction, ICMI '12, ACM (New York, NY, USA, 2012), 273--280.
44. Wobbrock, J. O., Morris, M. R., and Wilson, A. D. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, ACM (New York, NY, USA, 2009), 1083--1092.
45. Wobbrock, J. O., Wilson, A. D., and Li, Y. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th annual ACM symposium on User interface software and technology, UIST '07, ACM (New York, NY, USA, 2007), 159--168.
46. Yoon, H.-S., Soh, J., Bae, Y. J., and Seung Yang, H. Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognition 34, 7 (2001), 1491--1501.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

    • Interaction paradigms