Mark Billinghurst
ABSTRACT
Augmented Reality (AR) is technology that allows virtual imagery to be seamlessly integrated into the real world. Although first developed in the 1960's it has only been recently that AR has become widely available, through platforms such as the web and mobile phones. However most AR interfaces have very simple interaction, such as using touch on phone screens or camera tracking from real images. New depth sensing and gesture tracking technologies such as Microsoft Kinect or Leap Motion have made is easier than ever before to track hands in space. Combined with speech recognition and AR tracking and viewing software it is possible to create interfaces that allow users to manipulate 3D graphics in space through a natural combination of speech and gesture. In this paper I will review previous research in multimodal AR interfaces and give an overview of the significant research questions that need to be addressed before speech and gesture interaction can become commonplace.
- Azuma, R. 1997. A Survey of Augmented Reality. Presence, 6 (4), 355--385.Google Scholar
Digital Library
- Bai, H., Lee, G. A., and Billinghurst, M. 2012. Freeze view touch and finger gesture based interaction methods for handheld augmented reality interfaces. In Proceedings of the 27th Conference on Image and Vision Computing New Zealand (pp. 126--131). ACM. Google ScholarDigital Library
- Billinghurst, M. and Lee, M (2012). Multimodal Interfaces for Augmented Reality. In Dill, John, Earnshaw, Rae, Kasik, David et al (editors), Expanding the Frontiers of Visual Analytics and Visualization. London : Springer.Google Scholar
- Chu CP, Dani TH, and Gadh R. 1997. Multimodal Interface for a virtual reality based computer aided design system. Proceedings of 1997 IEEE International Conference on Robotics and Automation 2: 1329--1334Google Scholar
- Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C., and Sullivan, J. W. 1989. Synergistic use of direct manipulation and natural language. In ACM SIGCHI Bulletin (Vol. 20, No. SI, pp. 227--233). ACM. Google ScholarDigital Library
- Irawati, S., Green, S., Billinghurst, M., Duenser, A., and Ko, H. 2006. An evaluation of an augmented reality multimodal interface using speech and paddle gestures. In Advances in Artificial Reality and Tele-Existence (pp. 272--283). Springer Berlin Heidelberg. Google ScholarDigital Library
- Microsoft Kinect Website: http://www.xbox.com/KINECTGoogle Scholar
- Krum, D. M., Omoteso, O., Ribarsky, W., Starner, T., and Hodges, L. F. 2002. Speech and gesture multimodal control of a whole Earth 3D visualization environment. In Proceedings of the symposium on Data Visualisation 2002 (pp. 195--200). Eurographics Association. Google ScholarDigital Library
- Leap Motion Website: https://www.leapmotion.com/Google Scholar
- Olwal, A., Benko, H., and Feiner, S. 2003. SenseShapes: Using statistical geometry for object selection in a multimodal augmented reality system. In Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality (p. 300). IEEE Computer Society. Google ScholarDigital Library
Index Terms
- Hands and speech in space: multimodal interaction with augmented reality interfaces
Recommendations
Multimodal Human Machine Interactions in Virtual and Augmented Reality
Multimodal Signals: Cognitive and Algorithmic IssuesVirtual worlds are developing rapidly over the Internet. They are visited by avatars and staffed with Embodied Conversational Agents (ECAs). An avatar is a representation of a physical person. Each person controls one or several avatars and usually ...
The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments
MuC '22: Proceedings of Mensch und Computer 2022Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. ...
Using Hand Gesture and Speech in a Multimodal Augmented Reality Environment
Gesture-Based Human-Computer Interaction and SimulationIn this work we describe a 3D authoring tool which takes advantage of multimodal interfaces such as gestures and speech. This tool allows real-time Augmented Reality aimed to aid the tasks of interior architects and designers. This approach intends to ...
Comments