3D gesture interaction for handheld augmented reality

ABSTRACT

In this paper, we present a prototype for exploring natural gesture interaction with Handheld Augmented Reality (HAR) applications, using visual tracking based AR and freehand gesture based interaction detected by a depth camera. We evaluated this prototype in a user study comparing 3D gesture input methods with traditional touch-based techniques, using canonical manipulation tasks that are common in AR scenarios. We collected task performance data and user feedback via a usability questionnaire. The 3D gesture input methods were found to be slower, but the majority of the participants preferred them and gave them higher usability ratings. Being intuitive and natural was the most common feedback about the 3D freehand interface. We discuss implications of this research and directions for further work.

References

1. Bai, H., Gao, L., El-sana, J., and Billinghurst, M. 2013. Work-in-progress: Markerless 3D Gesture-based Interaction for Handheld Augmented Reality Interfaces. In Proceedings of the 16th IEEE International Symposium on Mixed and Augmented Reality, IEEE Computer Society, Washington, DC, USA, ISMAR '13, 1--6.
2. Bai, H., Lee, G. A., and Billinghurst, M. 2014. Using 3D Hand Gestures and Touch Input for Wearable AR Interaction. In Extended Abstracts on ACM CHI 2014 Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, CHI EA '14.
3. Baldauf, M., Zambanini, S., Fröhlich, P., and Reichl, P. 2011. Markerless Visual Fingertip Detection for Natural Mobile Device Interaction. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, ACM, New York, NY, USA, MobileHCI '11, 539--544.
4. Chen, L.-H., Yu, C.-J., and Hsu, S.-C. 2008. A Remote Chinese Chess Game Using Mobile Phone Augmented Reality. In Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, ACM, New York, NY, USA, ACE '08, 284--287.
5. Chun, W. H., and Höllerer, T. 2013. Real-time Hand Interaction for Augmented Reality on Mobile Phones. In Proceedings of the 18th International Conference on Intelligent User Interfaces, ACM, New York, NY, USA, IUI '13, 307--314.
6. Henrysson, A., Marshall, J., And Billinghurst, M. 2007. Experiments in 3D Interaction for Mobile Phone AR. In Proceedings of the 5th International Conference on Computer Graphics and Interactive Techniques in Australia and Southeast Asia, ACM, New York, NY, USA, GRAPHITE '07, 187--194.
7. Hürst, W., and Wezel, C. V. 2013. Gesture-based Interaction via Finger Tracking for Mobile Augmented Reality. Multimedia Tools and Applications 62, 1 (January), 233--258.
8. Kim, H., Reitmayr, G., and Woo, W. 2011. Interactive annotation on mobile phones for real and virtual space registration. In Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, IEEE Computer Society, Washington, DC, USA, ISMAR '11, 265--266.
9. Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, ACM, New York, NY, USA, UIST '12, 167--176.
10. Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer, C., Reitmayr, G., and Schmalstieg, D. 2012. Sketching Up the World: In Situ Authoring for Mobile Augmented Reality. Personal Ubiquitous Computer 16, 6, 623--630.
11. Mossel, A., Venditti, B., and Kaufmann, H. 2013. 3DTouch and HOMER-S: Intuitive Manipulation Techniques for One-Handed Handheld Augmented Reality. In Proceedings of the 15th International Conference of Virtual Technologies, ACM, New York, NY, USA, VRIC '13, 12:1--12:10.
12. Seo, B.-K., Choi, J., Han, J.-H., Park, H., and Park, J.-I. 2008. One-handed Interaction with Augmented Virtual Objects on Mobile Devices. In Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, ACM, New York, NY, USA, VRCAI '08, 8:1--8:6.
13. Wang, R., Paris, S., and Popović, J. 2011. 6D hands: Markerless Hand-tracking for Computer Aided Design. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ACM, New York, NY, USA, UIST '11, 549--558.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Mixed / augmented reality

      • Virtual reality