research-article

Dynamic modality weighting for multi-stream hmms inaudio-visual speech recognition

Authors:

Mihai Gurban,

Jean-Philippe Thiran,

Thomas Drugman,

Thierry DutoitAuthors Info & Claims

ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces

Pages 237 - 240

https://doi.org/10.1145/1452392.1452442

Published: 20 October 2008 Publication History

Get Access

Abstract

Merging decisions from different modalities is a crucial problem in Audio-Visual Speech Recognition. To solve this, state synchronous multi-stream HMMs have been proposed for their important advantage of incorporating stream reliability in their fusion scheme. This paper focuses on stream weight adaptation based on modality confidence estimators. We assume different and time-varying environment noise, as can be encountered in realistic applications, and, for this, adaptive methods are best suited. Stream reliability is assessed directly through classifier outputs since they are not specific to either noise type or level. The influence of constraining the weights to sum to one is also discussed.

References

[1]

H. Bourlard and S. Dupont. A new ASR approach based on independent processing and recombination of partial frequency bands. Proc. International Conference on Spoken Language Processing, pages 426--429, 1996.

Google Scholar

[2]

S. Dupont and J. Luettin. Audio-visual speech modeling for continuous speech recognition. In IEEE Transactions on Multimedia, volume 2, pages 141--151, Sept. 2000.

Digital Library

Google Scholar

[3]

H. Glotin, D. Vergyri, C. Neti, G. Potamianos, and J. Luettin. Weighting schemes for audio-visual fusion in speech recognition. In ICASSP01, Salt Lake City, USA, volume 1, pages 173--176, May 2001.

Google Scholar

[4]

G. Gravier, S. Axelrod, G. Potamianos, and C. Neti. Maximum entropy and MCE based HMM stream weight estimation for audio-visual ASR. Proc. Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2002.

Google Scholar

[5]

H. Misra. Multi-stream processing for noise robust speech recognition. PhD dissertation, EPFL, Lausanne, May 2006.

Google Scholar

[6]

E. K. Patterson, S. Gurbuz, Z. Tufekci, and J. N. Gowdy. Moving-talker, speaker--independent feature study and baseline results using the CUAVE multimodal speech corpus. EURASIP JASP, 2002(11):1189--1201, 2002.

Digital Library

Google Scholar

[7]

G. Potamianos and C. Neti. Stream confidence estimation for audio-visual speech recognition. In ICSLP00, Beijing, China, volume 3, pages 746--749, 2000.

Google Scholar

[8]

E. Sanchez-Soto, A. Potamianos, and K. Daoudi. Unsupervised stream weight computation using anti-models. In ICASSP07, Hawaii, USA, April 2007.

Google Scholar

[9]

P. Scanlon and G. Potamianos. Exploiting lower face symmetry in appearance-based automatic speechreading. Proc. Works. Audio-Visual Speech Process. (AVSP), pages 79--84, 2005.

Google Scholar

[10]

R. Seymour, D. Stewart, and J. Ming. Audio-visual integration for robust speech recognition using maximum weighted stream posteriors. Proc. INTERSPEECH, 2007.

Google Scholar

[11]

S. Tamura, K. Iwano, and S. Furui. A stream-weight optimization method for multi-stream HMMs based on likelihood value normalization. In ICASSP05, Philadelphia, USA, pages 468--472, March 2005.

Crossref

Google Scholar

[12]

S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland. The HTK Book. Cambridge, Entropic Ltd., 1999.

Google Scholar

Cited By

View all

Fu JTan JYin WPashami SBjörkman M(2023)Component attention network for multimodal dance improvisation recognitionProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614114(114-118)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614114
Yu WZeiler SKolossa D(2022)Reliability-Based Large-Vocabulary Audio-Visual Speech RecognitionSensors10.3390/s2215550122:15(5501)Online publication date: 23-Jul-2022
https://doi.org/10.3390/s22155501
Lyu HLuo JMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Understanding Political Polarization via Jointly Modeling Users, Connections and Multimodal Contents on Heterogeneous GraphsProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547898(4072-4082)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547898
Show More Cited By

Index Terms

Dynamic modality weighting for multi-stream hmms inaudio-visual speech recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
2. Hardware
  1. Communication hardware, interfaces and storage
    1. Signal processing systems

Recommendations

Audio-visual speech recognition integrating 3D lip information obtained from the Kinect

Audio-visual speech recognition (AVSR) has shown impressive improvements over audio-only speech recognition in the presence of acoustic noise. However, the problems of region-of-interest detection and feature extraction may influence the recognition ...
Audio-visual speech recognition using deep learning

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for ...
Audio-visual speech recognition using MPEG-4 compliant visual features

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio conditions. The system utilizes facial ...

Comments

Information & Contributors

Information

Published In

ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces

October 2008

322 pages

ISBN:9781605581989

DOI:10.1145/1452392

General Chairs:
Vassilis Digalakis
TU Crete, Greece
,
Alex Potamianos
TU Crete, Greece
,
Matthew Turk
UC Santa Barbara, USA
,
Program Chairs:
Roberto Pieraccini
SpeechCycle, USA
,
Yuri Ivanov
MERL Research, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '08

Sponsor:

ICMI '08: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES

October 20 - 22, 2008

Crete, Chania, Greece

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
246
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)2

Reflects downloads up to 09 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fu JTan JYin WPashami SBjörkman M(2023)Component attention network for multimodal dance improvisation recognitionProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614114(114-118)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614114
Yu WZeiler SKolossa D(2022)Reliability-Based Large-Vocabulary Audio-Visual Speech RecognitionSensors10.3390/s2215550122:15(5501)Online publication date: 23-Jul-2022
https://doi.org/10.3390/s22155501
Lyu HLuo JMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Understanding Political Polarization via Jointly Modeling Users, Connections and Multimodal Contents on Heterogeneous GraphsProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547898(4072-4082)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547898
Yang RWang SSun YZhang HLiao YGu YHou BJiao L(2022)Multimodal Fusion Remote Sensing Image–Audio RetrievalIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing10.1109/JSTARS.2022.319407615(6220-6235)Online publication date: 2022
https://doi.org/10.1109/JSTARS.2022.3194076
Radha NShahina ANayeemulla Khan AVelusami J(2022)A new multi-stream approach using acoustic and visual features for robust speech recognition systemMaterials Today: Proceedings10.1016/j.matpr.2022.03.53762(4916-4924)Online publication date: 2022
https://doi.org/10.1016/j.matpr.2022.03.537
Yu WZeiler SKolossa D(2021)Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition2020 28th European Signal Processing Conference (EUSIPCO)10.23919/Eusipco47968.2020.9287841(341-345)Online publication date: 24-Jan-2021
https://doi.org/10.23919/Eusipco47968.2020.9287841
Sajid UChen XSajid HKim TWang G(2021)Audio-Visual Transformer Based Crowd Counting2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)10.1109/ICCVW54120.2021.00254(2249-2259)Online publication date: Oct-2021
https://doi.org/10.1109/ICCVW54120.2021.00254
Mittal TBhattacharya UChandra RBera AManocha DWen Chen CCucchiara RHua XQi GRicci EZhang ZZimmermann R(2020)Emotions Don't LieProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3413570(2823-2832)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3394171.3413570
Zhu WWang XGao W(2020)Multimedia Intelligence: When Multimedia Meets Artificial IntelligenceIEEE Transactions on Multimedia10.1109/TMM.2020.296979122:7(1823-1835)Online publication date: Jul-2020
https://doi.org/10.1109/TMM.2020.2969791
Zhuang Y(2020)Emotional Analysis of Sentences Based on Machine LearningBig Data Analytics for Cyber-Physical System in Smart City10.1007/978-981-15-2568-1_111(813-820)Online publication date: 12-Jan-2020
https://doi.org/10.1007/978-981-15-2568-1_111
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Audio-visual speech recognition integrating 3D lip information obtained from the Kinect

Audio-visual speech recognition using deep learning

Audio-visual speech recognition using MPEG-4 compliant visual features

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations