BeyondTouch: Extending the Input Language with Built-in Sensors on Commodity Smartphones

ABSTRACT

While most smartphones today have a rich set of sensors that could be used to infer input (.e.g., accelerometer, gyroscope, microphone), the primary mode of interaction is still limited to the front-facing touchscreen and several physical buttons on the case. To investigate the potential opportunities for interactions supported by built-in sensors, we present the implementation and evaluation of BeyondTouch, a family of interactions to extend and enrich the input experience of a smartphone. Using only existing sensing capabilities on a commodity smartphone, we offer the user a wide variety of additional tapping and sliding inputs on the case of and the surface adjacent to the smartphone. We outline the implementation of these interaction techniques and demonstrate empirical evidence of their effectiveness and usability. We also discuss the practicality of BeyondTouch for a variety of application scenarios.

References

1. Baudisch, P., & Chu, G. (2009, April). Back-of-device interaction allows creating very small touch devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1923--1932). ACM.
2. Cai, L., & Chen, H. (2011, August). TouchLogger: Inferring Keystrokes on Touch Screen from Smartphone Motion. In HotSec.
3. Chen, K. Y., Ashbrook, D., Goel, M., Lee, S. H., & Patel, S. (2014, September). AirLink: sharing files between multiple devices using in-air gestures. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 565--569). ACM.
4. Goel, M., Lee, B., Islam Aumi, M. T., Patel, S., Borriello, G., Hibino, S., & Begole, B. (2014, April). SurfaceLink: using inertial and acoustic sensing to enable multi-device interaction on a surface. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (pp. 1387--1396). ACM.
5. Goel, M., Wobbrock, J., & Patel, S. (2012, October). GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones. In Proceedings of the 25th annual ACM symposium on User interface software and technology (pp. 545--554). ACM.
6. Harrison, C., & Hudson, S. E. (2008, October). Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology (pp. 205--208). ACM.
7. Harrison, C., Schwarz, J., & Hudson, S. E. (2011, October). TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology (pp. 627--636). ACM.
8. Heo, S., & Lee, G. (2011, August). Forcetap: extending the input vocabulary of mobile touch screens by adding tap gestures. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (pp. 113--122). ACM.
9. Hinckley, K., Pierce, J., Sinclair, M., & Horvitz, E. (2000, November). Sensing techniques for mobile interaction. In Proceedings of the 13th annual ACM symposium on User interface software and technology (pp. 91--100). ACM.
10. Hinckley, K., & Song, H. (2011, May). Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 801--810). ACM.
11. Hiraoka, S., Miyamoto, I., & Tomimatsu, K. (2003). Behind Touch, a Text Input Method for Mobile Phones by The Back and Tactile Sense Interface. Information Processing Society of Japan, Interaction 2003, 131--138.
12. Hudson, S. E., Harrison, C., Harrison, B. L., & LaMarca, A. (2010, January). Whack gestures: inexact and inattentive interaction with mobile devices. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (pp. 109--112). ACM.
13. JTransform: https://sites.google.com/site/piotrwendykier/software/jtransforms
14. Li, K. A., Baudisch, P., & Hinckley, K. (2008, April). Blindsight: eyes-free access to mobile phones. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1389--1398). ACM.
15. McGrath, W., & Li, Y. (2014, October). Detecting tapping motion on the side of mobile devices by probabilistically combining hand postures. In Proceedings of the 27th annual ACM symposium on User interface software and technology (pp. 215--219). ACM.
16. Miluzzo, E., Varshavsky, A., Balakrishnan, S., & Choudhury, R. R. (2012, June). Tapprints: your finger taps have fingerprints. In Proceedings of the 10th international conference on Mobile systems, applications, and services (pp. 323--336). ACM.
17. Norman, D. A. (2002). The design of everyday things. Basic books.
18. Owusu, E., Han, J., Das, S., Perrig, A., & Zhang, J. (2012, February). Accessory: password inference using accelerometers on smartphones. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications (p. 9). ACM.
19. Paradiso, J. A., Leo, C. K., Checka, N., & Hsiao, K. (2002, April). Passive acoustic knock tracking for interactive windows. In CHI'02 Extended Abstracts on Human Factors in Computing Systems (pp. 732--733). ACM.
20. Robinson, S., Rajput, N., Jones, M., Jain, A., Sahay, S., & Nanavati, A. (2011, May). TapBack: towards richer mobile interfaces in impoverished contexts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2733--2736). ACM.
21. Ronkainen, S., Häkkilä, J., Kaleva, S., Colley, A., & Linjama, J. (2007, February). Tap input as an embedded interaction method for mobile devices. In Proceedings of the 1st international conference on Tangible and embedded interaction (pp. 263--270). ACM.
22. Saponas, T. S., Harrison, C., & Benko, H. (2011, October). PocketTouch: through-fabric capacitive touch input. In Proceedings of the 24th annual ACM symposium on User interface software and technology (pp. 303--308). ACM.
23. Schmieder, P., Hosking, J., Luxton-Reilly, A., & Plimmer, B. (2013). Thumbs Up: 3D Gesture Input on Mobile Phones Using the Front Facing Camera. In Human-Computer Interaction - INTERACT 2013 (pp. 318--336). Springer Berlin Heidelberg.
24. Schwesig, C., Poupyrev, I., & Mori, E. (2004, April). Gummi: a bendable computer. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 263--270). ACM.
25. Scott, J., Izadi, S., Rezai, L. S., Ruszkowski, D., Bi, X., & Balakrishnan, R. (2010, September). RearType: text entry using keys on the back of a device. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services (pp. 171--180). ACM.
26. Sugimoto, M., & Hiroki, K. (2006, September). HybridTouch: an intuitive manipulation technique for PDAs using their front and rear surfaces. In Proceedings of the 8th conference on Human-computer interaction with mobile devices and services (pp. 137--140). ACM.
27. Weka. DOI: http://www.cs.waikato.ac.nz/ml/weka/
28. Wiese, J., Saponas, T. S., & Brush, A. J. (2013, April). Phoneprioception: enabling mobile phones to infer where they are kept. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2157--2166). ACM.
29. Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., & Shen, C. (2007, October). Lucid touch: a see-through mobile device. In Proceedings of the 20th annual ACM symposium on User interface software and technology (pp. 269--278). ACM.
30. Wobbrock, J. O., Myers, B. A., & Kembel, J. A. (2003, November). EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion. In Proceedings of the 16th annual ACM symposium on User interface software and technology (pp. 61--70). ACM.
31. Xu, Z., Bai, K., & Zhu, S. (2012, April). Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors. In Proceedings of the fifth ACM conference on Security and Privacy in Wireless and Mobile Networks (pp. 113--124). ACM.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)