Article

Towards the integration of shape-related information in 3-D gestures and speech

Author:

Timo SowaAuthors Info & Claims

ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

Pages 92 - 99

https://doi.org/10.1145/1180995.1181014

Published: 02 November 2006 Publication History

Abstract

This paper presents a model for the unified semantic representation of shape conveyed by speech and coverbal 3-D gestures. The representation is tailored to capture the semantic contributions of both modalities during free descriptions of objects. It is shown how the semantic content of shape-related adjectives, nouns, and iconic gestures can be modeled and combined when they occur together in multimodal utterances like "a longish bar" + iconic gesture. The model has been applied for the development of a prototype system for gesture recognition and integration with speech.

References

[1]

I. Biederman. Recognition-by-components: A theory of human image understanding. Psychological Review, 94(2):115--147, 1987.

[2]

M. Johnston and S. Bangalore. Finite-state methods for multimodal parsing and integration. In Proceedings of the ESSLLI Summer School on Logic, Language, and Information, Helsinki, Finland, 2001.

[3]

M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith. Unification-based multimodal integration. In Proc. of the 35th Annual Meeting of the Association for Computational Linguistics, Madrid, pages 281--288, 1997.

Digital Library

[4]

D. B. Koons, C. J. Sparrell, and K. R. Thorisson. Integrating simultaneous input from speech, gaze and hand gestures. In M. T. Maybury, editor, Intelligent Multimedia Interfaces, chapter 11, pages 257--276. MIT Press, Cambridge, MA, 1993.

Digital Library

[5]

S. Kopp, P. Tepper, and J. Cassell. Towards integrated microplanning of language and iconic gesture for multimodal output. In Proceedings of the 6th International Conference on Multimodal Interfaces, pages 97--104, New York, 2004. ACM Press.

Digital Library

[6]

E. Lang. The semantics of dimensional designation of spatial objects. In M. Bierwisch and E. Lang, editors, Dimensional adjectives: Grammatical structure and conceptual interpretation, pages 263--417. Springer, Berlin, 1989.

[7]

M. E. Latoschik. A user interface framework for multimodal VR interactions. In Proceedings of the 7th International Conference on Multimodal Interfaces, pages 76--83, New York, 2005. ACM Press.

Digital Library

[8]

D. Marr and H. Nishihara. Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Society, Series B, 200:269--294, 1978.

[9]

D. McNeill. Gesture & Thought. The University of Chicago Press, Chicago, 2005.

[10]

L. Nigay and J. Coutaz. A generic platform for addressing the multimodal challenge. In I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and N. Jakob, editors, Human Factors In Computing Systems: CHI '95 Conference Proceedings, pages 98--105, New York, 1995. ACM Press.

Digital Library

[11]

S. Oviatt. Multimodal interfaces. In J. Jacko and A. Sears, editors, The Human-Computer Interaction Handbook, pages 286--304. Lawrence Erlbaum, Mahwah, NJ, 2003.

Digital Library

[12]

T. Sowa. Understanding Coverbal Iconic Gestures in Shape Descriptions. DISKI 294. Akademische Verlagsgesellschaft Aka, Berlin, 2006.

Digital Library

[13]

T. Sowa and I. Wachsmuth. A model for the representation and processing of shape in coverbal iconic gestures. In K. Opwis and I. Penner, editors, Proceedings of KogWis05. The German Cognitive Science Conference 2005., pages 183--188, Basel, 2005. Schwabe.

[14]

C. J. Sparrell and D. B. Koons. Interpretation of coverbal depictive gestures. In Proceedings of Intelligent Multi-Modal Multi-Media Interface Systems, AAAI Spring Symposium Series, pages 8--12. Stanford University, March 1994.

[15]

A. Waibel, M. Tue Vo, P. Duchnowski, and S. Manke. Multimodal interfaces. Artificial Intelligence Review, 10:299--319, 1996.

Digital Library

Cited By

Sadeghipour AKopp S(2012)Gesture processing as grounded motor cognition: Towards a computational modelProcedia - Social and Behavioral Sciences10.1016/j.sbspro.2012.01.03232(213-223)Online publication date: 2012
https://doi.org/10.1016/j.sbspro.2012.01.032
Fröhlich CBiermann PLatoschik MWachsmuth I(2009)Processing Iconic Gestures in a Multimodal Virtual Construction EnvironmentGesture-Based Human-Computer Interaction and Simulation10.1007/978-3-540-92865-2_20(187-192)Online publication date: 14-Jan-2009
https://dl.acm.org/doi/10.1007/978-3-540-92865-2_20

Index Terms

Towards the integration of shape-related information in 3-D gestures and speech
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Exploring the effects of restraining the use of gestures on narrative speech
Abstract
Research on gesture production has emphasized the strong relationship between speech and gesture. Studies have explored whether the inability to gesture is detrimental to speech at different levels. However, findings are still ...
Highlights
- The study investigates whether restraining speakers from gesturing affects their narrative speech in terms of fluency and prosodic features.
Facilitating multiparty dialog with gaze, gesture, and speech
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

We study how synchronized gaze, gesture and speech rendered by an embodied conversational agent can influence the flow of conversations in multiparty settings. We begin by reviewing a computational framework for turn-taking that provides the foundation ...
Spontaneous spoken dialogues with the furhat human-like robot head
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction

Furhat [1] is a robot head that deploys a back-projected animated face that is realistic and human-like in anatomy. Furhat relies on a state-of-the-art facial animation architecture allowing accurate synchronized lip movements with speech, and the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

November 2006

404 pages

ISBN:159593541X

DOI:10.1145/1180995

General Chairs:
Francis Quek
Virginia Tech, USA
,
Jie Yang
Carnegie Mellon University, USA
,
Program Chairs:
Dominic Massaro
University of California, Santa Cruz, USA
,
Abeer Alwan
University of California, Los Angeles, USA
,
Timothy J. Hazen
Massachusetts Institute of Technology, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI06

Sponsor:

ICMI06: 8th International Conference on Multimodal Interfaces 2006

November 2 - 4, 2006

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sadeghipour AKopp S(2012)Gesture processing as grounded motor cognition: Towards a computational modelProcedia - Social and Behavioral Sciences10.1016/j.sbspro.2012.01.03232(213-223)Online publication date: 2012
https://doi.org/10.1016/j.sbspro.2012.01.032
Fröhlich CBiermann PLatoschik MWachsmuth I(2009)Processing Iconic Gestures in a Multimodal Virtual Construction EnvironmentGesture-Based Human-Computer Interaction and Simulation10.1007/978-3-540-92865-2_20(187-192)Online publication date: 14-Jan-2009
https://dl.acm.org/doi/10.1007/978-3-540-92865-2_20

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten