skip to main content
10.1145/1180995.1181022acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
Article

A contextual multimodal integrator

Published: 02 November 2006 Publication History

Abstract

Multimodal Integration addresses the problem of combining various user inputs into a single semantic representation that can be used in deciding the next step of system action(s). The method presented in this paper uses a statistical framework to implement the integration mechanism and includes contextual information additionally to the actual user input. The underlying assumption is that the more information sources are taken into account, the better picture can be drawn about the actual intention of the user in the given context of the interaction. The paper presents the latest results with a Maximum Entropy classifier, with special emphasis on the use of contextual information (type of gesture movements and type of objects selected). Instead of explaining the design and implementation process in details (a longer paper to be published later will do that), only a short description is provided here about the demonstration implementation that produces above 91% accuracy for the 1st best and higher than 96% for the accumulated five N-bests results.

References

[1]
Adam Berger, Stephen Della Pietra and Vincent Della Pietra. 1996. "A Maximum Entropy Approach to Natural Language Processing." Computational Linguistics, March 1996, (vol. 22, no. 1), pp.39--71.
[2]
Péter Pál Boda and Edward Filisko. 2004. "Virtual Modality: a Framework for Testing and Building Multimodal Applications." HLT-NAACL 2004 Workshop on Spoken Language Understanding for Conversational Systems, Boston, Massachusetts, USA, May 7, 2004.
[3]
Boda, P. P. 2004. "Multimodal Integration in a Wider Sense." COLING 2004 Satellite Workshop on Robust and Adaptive Information Processing for Mobile Speech Interfaces. Geneva, Switzerland, August 28-29, 2004.
[4]
Michael H. Coen. 2001. "Multimodal Integration A Biological View." 17th International Joint Conference on Artificial Intelligence, IJCAI 2001, Seattle, Washington, USA, August 4-10, pp. 1417--1424.
[5]
J. Glass, G. Flammia, D. Goodine, M. Phillips, J. Polifroni, S. Sakai, S. Seneff, and V. Zue. 1995. "Multilingual Spoken-Language Understanding in the MIT Voyager System," Speech Communication, 17(1-2):1--18.

Cited By

View all
  • (2022)Enhancing interaction of people with quadriplegiaProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3529218(223-229)Online publication date: 29-Jun-2022
  • (2010)Physical Activity Recognition with Mobile Phones: Challenges, Methods, and ApplicationsMultimedia Interaction and Intelligent User Interfaces10.1007/978-1-84996-507-1_8(185-213)Online publication date: 2010
  • (2009)Poster abstractProceedings of the 2009 International Conference on Information Processing in Sensor Networks10.5555/1602165.1602203(371-372)Online publication date: 13-Apr-2009

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces
November 2006
404 pages
ISBN:159593541X
DOI:10.1145/1180995
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. context
  2. data fusion
  3. machine learning
  4. maximum entropy
  5. multimodal database
  6. multimodal integration
  7. virtual modality

Qualifiers

  • Article

Conference

ICMI06
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Enhancing interaction of people with quadriplegiaProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3529218(223-229)Online publication date: 29-Jun-2022
  • (2010)Physical Activity Recognition with Mobile Phones: Challenges, Methods, and ApplicationsMultimedia Interaction and Intelligent User Interfaces10.1007/978-1-84996-507-1_8(185-213)Online publication date: 2010
  • (2009)Poster abstractProceedings of the 2009 International Conference on Information Processing in Sensor Networks10.5555/1602165.1602203(371-372)Online publication date: 13-Apr-2009

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media