PointPose: finger pose estimation for touch input on mobile devices using a depth sensor

ABSTRACT

The expressiveness of touch input can be increased by detecting additional finger pose information at the point of touch such as finger rotation and tilt. PointPose is a prototype that performs finger pose estimation at the location of touch using a short-range depth sensor viewing the touch screen of a mobile device. We present an algorithm that extracts finger rotation and tilt from a point cloud generated by a depth sensor oriented towards the device's touchscreen. The results of two user studies we conducted show that finger pose information can be extracted reliably using our proposed method. We show this for controlling rotation and tilt axes separately and also for combined input tasks using both axes. With the exception of the depth sensor, which is mounted directly on the mobile device, our approach does not require complex external tracking hardware, and, furthermore, external computation is unnecessary as the finger pose extraction algorithm can run directly on the mobile device. This makes PointPose ideal for prototyping and developing novel mobile user interfaces that use finger pose estimation.

References

1. Bailly, G., Müller, J., Rohs, M., Wigdor, D., and Kratz, S. Shoesense: a new perspective on gestural interaction and wearable applications. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, ACM (2012), 1239--1248.
2. Boring, S., Ledo, D., Chen, X., Marquardt, N., Tang, A., and Greenberg, S. The fat thumb: using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services, ACM (2012), 39--48.
3. Buxton, W. A three-state model of graphical input. In Proc. of the IFIP TC13 Third Intl. Conference on Human-Computer Interaction, North-Holland Publishing Co. (1990), 449--456.
4. Buxton, W. Multi-Touch Systems that I have Known and Loved.hboxhttp://www.billbuxton.com/multitouchOverview.html, June 2013.
5. Casiez, G., Roussel, N., and Vogel, D. 1 euro filter: a simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, ACM (2012), 2527--2530.
6. Enrico Guizzo, IEEE Robotics Blog. Hands-On With the Next Generation Kinect: PrimeSense Capri. http://spectrum.ieee.org/automaton/robotics/robotics-hardware/handson-with-the-next-generation-kinect-primesense-capri, May 2013.
7. Fischler, M. A., and Bolles, R. C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24, 6 (1981), 381--395.
8. GSM Arena. HTC Evo 3D (Smartphone with Stereo Camera Pair). http://www.gsmarena.com/htc_evo_3d-3901.php, June 2013.
9. Harrison, C., Benko, H., and Wilson, A. D. Omnitouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM (2011), 441--450.
10. Hasan, K., Ahlström, D., and Irani, P. Ad-binning: leveraging around device space for storing, browsing and retrieving mobile device content. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 899--908.
11. Holz, C., and Baudisch, P. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of the 28th international conference on Human factors in computing systems, ACM (2010), 581--590.
12. Holz, C., and Baudisch, P. Understanding touch. In Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011), 2501--2510.
13. Intel. Intel Perceptual Computing SDK. http://software.intel.com/en-us/vcsource/tools/perceptual-computing-sdk, June 2013.
14. Ketabdar, H., Yüksel, K. A., Jahnbekam, A., Roshandel, M., and Skirpo, D. Magisign: User identification/authentication based on 3d around device magnetic signatures. In UBICOMM 2010, The Fourth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (2010), 31--34.
15. Kratz, S., Rohs, M., Guse, D., Müller, J., Bailly, G., and Nischt, M. Palmspace: continuous around-device gestures vs. multitouch for 3d rotation tasks on mobile devices. In Proceedings of the International Working Conference on Advanced Visual Interfaces, ACM (2012), 181--188.
16. Liang, R.-H., Cheng, K.-Y., Su, C.-H., Weng, C.-T., Chen, B.-Y., and Yang, D.-N. Gausssense: Attachable stylus sensing using magnetic sensor grid. In Proceedings of the 25th annual ACM symposium on User interface software and technology, ACM (2012), 319--326.
17. Moscovich, T., and Hughes, J. F. Navigating documents with the virtual scroll ring. In Proceedings of the 17th annual ACM symposium on User interface software and technology, ACM (2004), 57--60.
18. Olwal, A., Feiner, S., and Heyman, S. Rubbing and tapping for precise and rapid selection on touch-screen displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2008), 295--304.
19. pygame.org. Pygame game programming library. http://www.pygame.org, June 2013.
20. Rogers, S., Williamson, J., Stewart, C., and Murray-Smith, R. Anglepose: robust, precise capacitive touch tracking via 3d orientation estimation. In Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011), 2575--2584.
21. Roudaut, A., Huot, S., and Lecolinet, E. Taptap and magstick: improving one-handed target acquisition on small touch-screens. In Proceedings of the working conference on Advanced visual interfaces, ACM (2008), 146--153.
22. Roudaut, A., Lecolinet, E., and Guiard, Y. Microrolls: expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2009), 927--936.
23. Rusu, R. B., and Cousins, S. 3d is here: Point cloud library (pcl). In 2011 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2011), 1--4.
24. Stewart, C., Rohs, M., Kratz, S., and Essl, G. Characteristics of pressure-based input for mobile devices. In Proceedings of the 28th international conference on Human factors in computing systems, ACM (2010), 801--810.
25. Takeoka, Y., Miyaki, T., and Rekimoto, J. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces, ACM (2010), 91--94.
26. von Zezschwitz, E., Koslow, A., De Luca, A., and Hussmann, H. Making graphic-based authentication secure against smudge attacks. In Proceedings of the 2013 international conference on Intelligent user interfaces, ACM (2013), 277--286.
27. Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., and Shen, C. Lucid touch: a see-through mobile device. In Proceedings of the 20th annual ACM symposium on User interface software and technology, ACM (2007), 269--278.
28. Williams, L. SmartQuill. British Telecom Labs (1997).

Index Terms

• Human-centered computing