skip to main content
10.1145/1027933.1027996acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
Article

A maximum entropy based approach for multimodal integration

Published: 13 October 2004 Publication History

Abstract

Integration of various user input channels for a multimodal interface is not just an engineering problem. To fully understand users in the context of an application and the current session, solutions are sought that process information from different intentional, i.e. user-originated, as well as from passively available sources in a uniform manner. As a first step towards this goal, the work demonstrated here investigates how intentional user input (e.g. speech, gesture) can be seamlessly combined to provide a single semantic interpretation of the user input. For this classical Multimodal Integration problem the Maximum Entropy approach is demonstrated with 76.52% integration accuracy for the 1st and 86.77% accuracy for the top 3-best candidates. The paper also exhibits the process that generates multimodal data for training the statistical integrator, using transcribed speech from MIT's Voyager application. The quality of the generated data is assessed by comparing to real inputs to the multimodal version of Voyager.

References

[1]
Berger, A., Della Pietra, S. and Della Pietra, V. A maximum entropy approach to natural language processing. Computational Linguistics, March 1996, (vol. 22, no. 1), pp.39--71.
[2]
Boda, P. P. and Filisko, E. Virtual Modality: a Framework for Testing and Building Multimodal Applications. HLT-NAACL 2004 Workshop on Spoken Language Understanding for Conversational Systems, Boston, Massachusetts, USA, May 7, 2004.
[3]
Boda, P. P. Multimodal Integration in a Wider Sense. COLING 2004 Satellite Workshop on Robust and Adaptive Information Processing for Mobile Speech Interfaces. Geneva, Switzerland, August 28 -- 29, 2004
[4]
Glass, J., Flammia, G., Goodine, D., Phillips, M., Polifroni, J., Sakai, S., Seneff, S. and Zue, V. Multilingual Spoken-Language Understanding in the MIT Voyager System. Speech Communication, 17(1--2):1--18, 1995.
[5]
Wang, S. B. A Multimodal Galaxy-based Geographic System. S.M. thesis, MIT Department of Electrical Engineering and Computer Science. 2003.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces
October 2004
368 pages
ISBN:1581139950
DOI:10.1145/1027933
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2004

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. machine learning
  2. maximum entropy
  3. multimodal database
  4. multimodal integration
  5. virtual modality

Qualifiers

  • Article

Conference

ICMI04
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 258
    Total Downloads
  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media