skip to main content
10.1145/1647314.1647345acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Benchmarking fusion engines of multimodal interactive systems

Published: 02 November 2009 Publication History

Abstract

This article proposes an evaluation framework to benchmark the performance of multimodal fusion engines. The paper first introduces different concepts and techniques associated with multimodal fusion engines and further surveys recent implementations. It then discusses the importance of evaluation as a mean to assess fusion engines, not only from the user perspective, but also at a performance level. The article further proposes a benchmark and a formalism to build testbeds for assessing multimodal fusion engines. In its last section, our current fusion engine and the associated system HephaisTK are evaluated thanks to the evaluation framework proposed in this article. The article concludes with a discussion on the proposed quantitative evaluation, suggestions to build useful testbeds, and proposes some future improvements.

References

[1]
Bertoncini, M. Cavazza, M. Emotional Multimodal Interfaces for Digital Media: The CALLAS Challenge. In Proceedings of HCI International 2007, Beijing (2007).
[2]
Bolt, R.A. Put-that-there: voice and gesture at the graphics interface. Computer Graphics, 14(3), pp. 262--270 (1980).
[3]
Bouchet, J., Nigay, L., Ganille, T.: ICARE Software Components for Rapidly Developing Multimodal Interfaces. In: Conference Proceedings of ICMI'2004, State College, Pennsylvania, USA, Oct. 2004, ACM Press, pp. 251--258.
[4]
Bourguet, M. L.: A Toolkit for Creating and Testing Multimodal Interface Designs. In: companion proceedings of UIST'02, Paris, Oct. 2002, pp. 29--30 (2002).
[5]
Bui T.H.: Multimodal Dialogue Management -- State of the Art. CTIT Technical Report series No. 06-01, University of Twente (UT), Enschede, The Netherlands (2006).
[6]
Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM international Conference on Multimedia, Seattle, USA, pp. 31--40, (1997).
[7]
Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J. Young, R.: Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE properties. In: Proceedings of INTERACT'95, Lillehammer, Norway, June 1995, pp. 115--120, Chapman&Hall Publ. (1995).
[8]
Dumas, B., Lalanne, D., Ingold, R.: Prototyping Multimodal Interfaces with SMUIML Modeling Language. In: CHI 2008 Workshop on User Interface Description Languages for Next Generation User Interfaces, CHI 2008, Firenze, Italy, pp. 63--66 (2008).
[9]
Dumas, B., Lalanne, D., Oviatt, S. "Multimodal Interfaces: A Survey of Principles, Models and Frameworks". In Denis Lalanne, Jürg Kohlas eds. Human Machine Interaction, LNCS 5440, Springer-Verlag, pp. 3--26 (2009)
[10]
Dumas, B., Lalanne, D., Guinard, D., Ingold, R., Koenig, R.: Strengths and Weaknesses of Software Architectures for the Rapid Creation of Tangible and Multimodal Interfaces. In: Proceedings of 2nd international conference on Tangible and Embedded Interaction (TEI 2008), Bonn (Germany), February 19 -- 21 2008, pp. 47--54 (2008).
[11]
EMMA: Extensible MultiModal Annotation markup language: W3C recommandation. http://www.w3.org/TR/emma/
[12]
Flippo, F., Krebs, A. Marsic I.: A Framework for Rapid Development of Multimodal Interfaces. In: Proceedings of ICMI'03, Vancouver, BC, Nov. 5--7, pp. 109--116 (2003).
[13]
Garofolo, J. Overcoming Barriers to Progress in Multimodal Fusion Research. In AAAI Fall 2008 Symposium Proceedings (2008).
[14]
Holzapfel, H., Nickel, K., and Stiefelhagen, R. Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures. In Proceedings of ICMI '04 (State College, PA, USA, October 13--15, 2004). ACM, New York, NY, pp. 175--182 (2004).
[15]
Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L., Pittman, J. A., Smith, I.: Unification-based multimodal integration. In: Proceedings of the Eighth Conference on European Chapter of the Association For Computational Linguistics (Madrid, Spain, July 07--12, 1997), pp. 281--288.
[16]
Krahnstoever, N., Kettebekov, S., Yeasin, M. Sharma, R.: A real-time framework for natural multimodal interaction with large screen displays. In: ICMI'02, Pittsburgh, USA, Oct. 2002 (2002).
[17]
Lawson, J--Y., Al--Akkad, A., Vanderdonckt, J. and Macq, B. An Open Source Workbench for Prototyping Multimodal Interactions Based on Off-The-Shelf Heterogeneous Components. Proceedings of the First ACM SIGCHI Symposium on Engineering Interactive Computing Systems, ACM Press, USA, July 14--17, 2009.
[18]
Nigay, L., Coutaz, J.A.: Design space for multimodal systems: concurrent processing and data fusion. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (Amsterdam, The Netherlands, April 24 -- 29, 1993). ACM, New York, NY, pp. 172--178 (1993).
[19]
Oviatt, S.L.: Ten myths of multimodal interaction. In: Communications of the ACM, 42(11), New York: ACM Press, pp. 74--81 (1999).
[20]
Oviatt, S. L., Cohen, P. R., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. In: Human Computer Interaction, 2000, vol. 15, no. 4, pp. 263--322 {Reprinted in Human--Computer Interaction in the New Millennium (ed. J. Carroll), Addison-Wesley Press, Reading, MA, 2001; chapter 19, pp. 421--456} (2000).
[21]
Oviatt, S.L., Coulston, R., Tomko, S., Xiao, B., Lunsford, R., Wesson, M., Carmichael, L.: Toward a theory of organized multimodal integration patterns during human-computer interaction. In: Proceedings of ICMI 2003, ACM Press, pp. 44--51 (2003).
[22]
Sharma, R., Pavlovic, V. I., Huang, T.S.: Toward multimodal human-computer interface. In: Proceedings IEEE, 86(5) {Special issue on Multimedia Signal Processing}, pp. 853--860 (1998).

Cited By

View all
  • (2020)Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS)10.1109/MASS50613.2020.00046(309-317)Online publication date: Dec-2020
  • (2019)The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart EnvironmentsSensors10.3390/s1911258719:11(2587)Online publication date: 6-Jun-2019
  • (2018)A Multimodal Interaction Design Guidelines for VR Foot Reflexology Therapy ApplicationInternational Journal of Operations Research and Information Systems10.4018/IJORIS.20160701057:3(74-91)Online publication date: 13-Dec-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces
November 2009
374 pages
ISBN:9781605587721
DOI:10.1145/1647314
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2009

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. fusion engines evaluation
  2. multimodal fusion
  3. multimodal interfaces
  4. multimodal toolkit

Qualifiers

  • Research-article

Conference

ICMI-MLMI '09
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2020)Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS)10.1109/MASS50613.2020.00046(309-317)Online publication date: Dec-2020
  • (2019)The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart EnvironmentsSensors10.3390/s1911258719:11(2587)Online publication date: 6-Jun-2019
  • (2018)A Multimodal Interaction Design Guidelines for VR Foot Reflexology Therapy ApplicationInternational Journal of Operations Research and Information Systems10.4018/IJORIS.20160701057:3(74-91)Online publication date: 13-Dec-2018
  • (2017)Modelling fusion of modalities in multimodal interactive systems with MMMMProceedings of the 19th ACM International Conference on Multimodal Interaction10.1145/3136755.3136768(288-296)Online publication date: 3-Nov-2017
  • (2016)In-Depth Analysis of Multimodal InteractionProceedings, Part II, of the 18th International Conference on Human-Computer Interaction. Interaction Platforms and Techniques - Volume 973210.1007/978-3-319-39516-6_22(233-240)Online publication date: 17-Jul-2016
  • (2015)Evaluation of multimodal fusion methods using integration patterns modeling2015 38th International Conference on Telecommunications and Signal Processing (TSP)10.1109/TSP.2015.7296287(377-381)Online publication date: Jul-2015
  • (2014)The Automated Interplay of Multimodal Fission and Fusion in Adaptive HCIProceedings of the 2014 International Conference on Intelligent Environments10.1109/IE.2014.32(170-177)Online publication date: 30-Jun-2014
  • (2014)Graphical Toolkits for Rapid Prototyping of Multimodal Systems: A SurveyInteracting with Computers10.1093/iwc/iwu00327:4(470-488)Online publication date: 30-Mar-2014
  • (2014)A graphical editor for the SMUIML multimodal user interaction description languageScience of Computer Programming10.1016/j.scico.2013.04.00386(30-42)Online publication date: Jun-2014
  • (2014)A Proposal for Processing and Fusioning Multiple Information Sources in Multimodal Dialog SystemsHighlights of Practical Applications of Heterogeneous Multi-Agent Systems. The PAAMS Collection10.1007/978-3-319-07767-3_16(167-178)Online publication date: 2014
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media