GATO: predicting human performance with multistroke and multitouch gesture input

ABSTRACT

We introduce GATO, a human performance analysis technique grounded in the Kinematic Theory that delivers accurate predictions for the expected user production time of stroke gestures of all kinds: unistrokes, multistrokes, multitouch, or combinations thereof. Our experimental results obtained on several public datasets (82 distinct gesture types, 123 participants, ≈36k gesture samples) show that GATO predicts user-independent gesture production times that correlate r_s > .9 with groundtruth, while delivering an average relative error of less than 10% with respect to actual measured times. With its accurate estimations of users' a priori time performance with stroke gesture input, GATO will help researchers to understand better users' gesture articulation patterns on touchscreen devices of all kinds. GATO will also benefit practitioners to inform highly effective gesture set designs.

References

1. Johnny Accot and Shumin Zhai. 1997. Beyond Fitts' law: Models for trajectory-based HCI tasks. In Proc. CHI '97.
2. Jr. Allan C. Long, James A. Landay, Lawrence A. Rowe, and Joseph Michiels. 2000. Visual similarity of pen gestures. In Proc. CHI '00.
3. Abdullah Almaksour, Eric Anquetil, Réjean Plamondon, and Christian O'Reilly. 2011. Synthetic handwritten gesture generation using Sigma-Lognormal model for evolving handwriting classifiers. In Proc. IGS '11.
4. Lisa Anthony, Radu-Daniel Vatavu, and Jacob O. Wobbrock. 2013. Understanding the consistency of users' pen and finger stroke gesture articulation. In Proc. GI '13.
5. Lisa Anthony and Jacob O. Wobbrock. 2012. $N-protractor: a fast and accurate multistroke recognizer. In Proc. GI '12.
6. Caroline Appert and Shumin Zhai. 2009. Using strokes as command shortcuts: Cognitive benefits and toolkit support. In Proc. CHI '09.
7. Ilhan Aslan, Ida Buchwald, Philipp Koytek, and Elisabeth André. 2016. Pen + mid-air: An exploration of mid-air gestures to complement pen input on tablets. In Proc. NordiCHI '16.
8. Shiri Azenkot, Kyle Rector, Richard Ladner, and Jacob Wobbrock. 2012. PassChords: Secure multi-touch authentication for blind people. In Proc. ASSETS '12.
9. Gilles Bailly, Jörg Müller, and Eric Lecolinet. 2012. Design and evaluation of finger-count interaction: Combining multitouch gestures and menus. Int. J. Hum.-Comput. Stud. 70(10).
10. Xiaojun Bi, Yang Li, and Shumin Zhai. 2013. Ffitts law: Modeling finger touch with Fitts' law. In Proc. CHI '13.
11. Rachel Blagojevic, Samuel Hsiao-Heng Chang, and Beryl Plimmer. 2010. The power of automatic feature selection: Rubine on steroids. In Proc. SBIM '10.
12. Xiang Cao and Shumin Zhai. 2007. Modeling human performance of pen stroke gestures. In Proc. CHI '07.
13. Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The keystroke-level model for user performance time with interactive systems. Commun. ACM 23(1).
14. Steven J. Castellucci and I. Scott MacKenzie. 2008. Graffiti vs. Unistrokes: An empirical comparison. In Proc. CHI '08.
15. Xiang Anthony Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+touch: Interweaving touch & in-air gestures. In Proc. UIST '14.
16. Moussa Djioua and Réjean Plamondon. 2009. Studying the variability of handwriting patterns using the Kinematic Theory. Hum. Mov. Sci. 28(5).
17. Paul M. Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47(6).
18. Tamar Flash and Neville Hogan. 1985. The coordination of arm movements: an experimentally confirmed mathematical model. J. Neurosci. 5(7).
19. Markus Funk, Alireza Sahami, Niels Henze, and Albrecht Schmidt. 2014. Using a touch-sensitive wristband for text entry on smart watches. In Proc. CHI '14.
20. Javier Galbally, Réjean Plamondon, Julián Fierrez, and Javier Ortega-García. 2012. Synthetic on-line signature generation. Part II: Experimental validation. Pattern Recogn. 45(7).
21. Emilien Ghomi, Stéphane Huot, Olivier Bau, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2013. Arpège: Learning multitouch chord gestures vocabularies. In Proc. ITS '13.
22. Tovi Grossman, Xiang Anthony Chen, and George Fitzmaurice. 2015. Typing on glasses: Adapting text entry to smart eyewear. In Proc. MobileHCI '15.
23. Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. Tapsense: Enhancing finger interaction on touch surfaces. In Proc. UIST '11.
24. Florian Heller, Stefan Ivanov, Chat Wacharamanotham, and Jan Borchers. 2014. Fabritouch: Exploring flexible touch input on textiles. In Proc. ISWC '14.
25. Uta Hinrichs and Sheelagh Carpendale. 2011. Gestures in the wild: Studying multi-touch gesture sequences on interactive tabletop exhibits. In Proc. CHI '11.
26. Poika Isokoski. 2001. Model for unistroke writing time. In Proc. CHI '01.
27. Shaun K. Kane, Jacob O. Wobbrock, and Richard E. Ladner. 2011. Usable gestures for blind people: Understanding preference and performance. In Proc. CHI '11.
28. Per Ola Kristensson and Shumin Zhai. 2004. SHARK²: A large vocabulary shorthand writing system for pen-based computers. In Proc. UIST '04.
29. Luis A. Leiva. 2017. Large-scale user perception of synthetic stroke gestures. In Proc. DIS '17.
30. Luis A. Leiva, Daniel Martín-Albo, and Réjean Plamondon. 2016. Gestures à Go Go: Authoring synthetic human-like stroke gestures using the kinematic theory of rapid movements. ACM T. Intel. Syst. Tec. 7(2).
31. Luis A. Leiva, Daniel Martín-Albo, and Réjean Plamondon. 2017a. The Kinematic Theory produces human-like stroke gestures. Interact. Comput. 29(4).
32. Luis A. Leiva, Daniel Martín-Albo, Réjean Plamondon, and Radu-Daniel Vatavu. 2018. KeyTime: Super-accurate prediction of stroke gesture production times. In Proc. CHI '18.
33. Luis A. Leiva, Daniel Martín-Albo, and Radu-Daniel Vatavu. 2017b. Synthesizing stroke gestures across user populations: A case for users with visual impairments. In Proc. CHI '17.
34. Yang Li. 2010. Gesture Search: A tool for fast mobile data access. In Proc. UIST '10.
35. Hao Lü and Yang Li. 2011. Gesture avatar: A technique for operating mobile user interfaces using gestures. In Proc. CHI '11.
36. Yuexing Luo and Daniel Vogel. 2015. Pin-and-cross: A unimanual multitouch technique combining static touches with crossing selection. In Proc. UIST '15.
37. Daniel Martín-Albo and Luis A. Leiva. 2016. G3: bootstrapping stroke gestures design with synthetic samples and built-in recognizers. In Proc. MobileHCI '16.
38. Daniel Martín-Albo, Réjean Plamondon, and Enrique Vidal. 2014. Training of on-line handwriting text recognizers with synthetic text generated using the Kinematic Theory of rapid human movements. In Proc. ICFHR '14.
39. Daniel Martín-Albo, Réjean Plamondon, and Enrique Vidal. 2015. Improving sigma-lognormal parameter extraction. In Proc. ICDAR '15.
40. Fabrice Matulic, Daniel Vogel, and Raimund Dachselt. 2017. Hand contact shape recognition for posture-based tabletop widgets and interaction. In Proc. ISS '17. 3--11.
41. Meredith Ringel Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jeffrey P. Bigham, and Shaun K. Kane. 2016. "with most of it being pictures now, i rarely use it": Understanding twitter's evolving accessibility to blind users. In Proc. CHI '16.
42. Jörg Müller, Antti Oulasvirta, and Roderick Murray-Smith. 2017. Control theoretic models of pointing. ACM Trans. Comput.-Hum. Interact. 24(4).
43. Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of pre-designed and user-defined gesture sets. In Proc. CHI '13.
44. Réjean Plamondon. 1995a. A kinematic theory of rapid human movements. Part I: Movement representation and control. Biol. Cybern. 72(4).
45. Réjean Plamondon. 1995b. A kinematic theory of rapid human movements. Part II: Movement time and control. Biol. Cybern. 72(4).
46. Réjean Plamondon, Adel M. Alimi, Pierre Yergeau, and Franck Leclerc. 1993. Modelling velocity profiles of rapid movements: a comparative study. Biol. Cybern. 69(1).
47. Réjean Plamondon and Moussa Djioua. 2006. A multi-level representation paradigm for handwriting stroke generation. Hum. Mov. Sci. 25(4--5).
48. Benjamin Poppinga, Alireza Sahami Shirazi, Niels Henze, Wilko Heuten, and Susanne Boll. 2014. Understanding shortcut gestures on mobile touch devices. In Proc. MobileHCI '14.
49. Philip Quinn and Shumin Zhai. 2018. Modeling gesture-typing movements. Hum.-Comput. Interact. 33(2).
50. Yosra Rekik, Laurent Grisoni, and Nicolas Roussel. 2013. Towards many gestures to one command: A user study for tabletops. In Proc. INTERACT '13.
51. Yosra Rekik, Radu-Daniel Vatavu, and Laurent Grisoni. 2014a. Match-up & conquer: A two-step technique for recognizing unconstrained bimanual and multi-finger touch input. In Proc. AVI '14.
52. Yosra Rekik, Radu-Daniel Vatavu, and Laurent Grisoni. 2014b. Understanding users' perceived difficulty of multi-touch gesture articulation. In Proc. ICMI '14.
53. Yosra Rekik, Radu-Daniel Vatavu, and Laurent Grisoni. 2016. Spontaneous Gesture Production Patterns on Multi-touch Interactive Surfaces. Springer, Cham.
54. Quentin Roy, Sylvain Malacria, Yves Guiard, Eric Lecolinet, and James Eagan. 2013. Augmented letters: Mnemonic gesture-based shortcuts. In Proc. CHI '13.
55. Huawei Tu, Xiangshi Ren, and Shumin Zhai. 2012. A comparative evaluation of finger and pen stroke gestures. In Proc. CHI '12.
56. Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu, Luis A. Leiva, and Réjean Plamondon. 2018. Gesture input for users with motor impairments on touchscreens: Empirical results based on the kinematic theory. In Proc. CHI EA '18. Article LBW537, 6 pages.
57. Radu-Daniel Vatavu. 2017. Improving gesture recognition accuracy on touch screens for users with low vision. In Proc. CHI '17.
58. Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2013. Relative accuracy measures for stroke gestures. In Proc. ICMI '13.
59. Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2014. Gesture heatmaps: Understanding gesture performance with colorful visualizations. In Proc. ICMI '14.
60. Radu-Daniel Vatavu, Gabriel Cramariuc, and Doina Maria Schipor. 2015. Touch interaction for children aged 3 to 6 years: Experimental findings and relationship to motor skills. Int. J. Hum.-Comput. Stud. 74(1).
61. Radu-Daniel Vatavu, Daniel Vogel, Géry Casiez, and Laurent Grisoni. 2011. Estimating the perceived difficulty of pen gestures. In Proc. INTERACT '11.
62. Paolo Viviani and Tamar Flash. 1995. Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. J. Exp. Psychol. 21(1).
63. Paolo Viviani and Carlo Terzuolo. 1982. Trajectory determines movement dynamics. Neuroscience 7(2).
64. Rand Wilcox. 2012. Modern Statistics for the Social and Behavioral Sciences. Taylor & Francis Group, LLC, Boca Raton, FL, USA.
65. D. Willems, R. Niels, M. van Gerven, and L. Vuurpijl. 2009. Iconic and multi-stroke gesture recognition. Pattern Recogn. 42(12).
66. Jacob O. Wobbrock, Edward Cutrell, Susumu Harada, and I. Scott MacKenzie. 2008. An error model for pointing based on fitts' law. In Proc. CHI '08.
67. Jacob O. Wobbrock, Brad A. Myers, and John A. Kembel. 2003. Edgewrite: A stylus-based text entry method designed for high accuracy and stability of motion. In Proc. UIST '03.
68. Shaomei Wu and Lada A. Adamic. 2014. Visually impaired users on an online social network. In Proc. CHI '14. 10.
69. Shumin Zhai and Per Ola Kristensson. 2012. The word-gesture keyboard: Reimagining keyboard interaction. Commun. ACM 55(9).
70. Shumi Zhai, Per Ola Kristensson, Caroline Appert, Tue H. Anderson, and Xiang Cao. 2012. Foundational issues in touch-surface stroke gesture design --- an integrative review. In Foundations and Trends in Human-Computer Interaction. Vol. 5.
71. Chi Zhang, Nan Jiang, and Feng Tian. 2016. Accessing mobile apps with user defined gesture shortcuts: An exploratory study. In Proc. ISS '16.

• Human-centered computing