Joint visual-text modeling for automatic retrieval of multimedia documents

ABSTRACT

In this paper we describe a novel approach for jointly modeling the text and the visual components of multimedia documents for the purpose of information retrieval(IR). We propose a novel framework where individual components are developed to model different relationships between documents and queries and then combined into a joint retrieval framework. In the state-of-the-art systems, a late combination between two independent systems, one analyzing just the text part of such documents, and the other analyzing the visual part without leveraging any knowledge acquired in the text processing, is the norm. Such systems rarely exceed the performance of any single modality (i.e. text or video) in information retrieval tasks. Our experiments indicate that allowing a rich interaction between the modalities results in significant improvement in performance over any single modality. We demonstrate these results using the TRECVID03 corpus, which comprises 120 hours of broadcast news videos. Our results demonstrate over 14 % improvement in IR performance over the best reported text-only baseline and ranks amongst the best results reported on this corpus.

REFERENCES

• 1: A. Berger and J. Lafferty. The Weaver System for Document Retrieval. In Proceedings of the Eighth Text REtrieval Conference (TREC-8), pages 163--174. NIST Special Publication 500-246, 2000.
• 2: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 3: H. Cunningham, D. Maynard, K. Bontcheva, and V. Tablan. GATE: A framework and graphical development environment for robust NLP tools and applications. In Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics, pages 168--175, 2002.
• 4: J. Darroch and D. Ratcliff. Generalized Iterative Scaling for Log-Linear Models. The Annals of Mathematical Statistics, 43(5):1470--1480, 1972.
• 5: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 6: S. L. Feng,, R. Manmatha, and V. Lavrenko. Multiple bernoulli relevance models for image and video annotation. In Intl. Conf. on Computer Vision and Pattern Recognition, Washington D.C., June 2004.
• 7: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 8: Arnab Ghoshal , Pavel Ircing , Sanjeev Khudanpur, Hidden Markov models for automatic annotation and content-based retrieval of images and video, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076127]
• 9: A. Hauptmann, D. Ng, R. Baron, M. Chen, and et. al. Informedia at TRECVID 2003: Analyzing and searching broadcast news video. In Proceedings of TRECVID2003, Gaithersburg, MD, November 2003. NIST.
• 10: T. M. J. Baldridge and G. Bierner. openNLP maximum entropy modeling toolkit. http://maxent.sourceforge.net/, version 2.2.0, 2004.
• 11: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 12: D. Klakow. Log-linear interpolation of language models. In Proc. International Conference on Speech and Language Processing (ICSLP, Sydney, Australia, November 1998.
• 13: John Lafferty , Chengxiang Zhai, Document language models, query models, and risk minimization for information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.111-119, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383970]
• 14: V. Lavrenko, S. L. Feng, and R. Manmatha. Statistical models for automatic video annotation and retrieval. In Intl. Conf. On Acoust., Sp., and Sig. Proc., pages 417--420, Montreal, QC, May 2004.
• 15: C.-Y. Lin, B. Tseng, and J. R. Smith. Video Collaborative Annotation Forum: Establishing Ground-truth Labels on Large Multimedia Datasets. In Proceedings of the TRECVID2003: NIST Special Publications, Gaithersburg, MD, 2003. NIST.
• 16: NIST. Proceedings of the TREC Video Retrieval Evaluation Conference(TRECVID2003), Gaithersburg, MD, November 2003.
• 17: NIST. Proceedings of the TREC Video Retrieval Evaluation Conference(TRECVID2004), Gaithersburg, MD, November 2004.
• 18: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 19: A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In E. Brill and K. Church, editors, Proc. Conf. on Empirical Methods in Natural Language Processing, pages 133--142. Assn Comp. Ling., Somerset, New Jersey, 1996.
• 20: Jeffrey C. Reynar , Adwait Ratnaparkhi, A maximum entropy approach to identifying sentence boundaries, Proceedings of the fifth conference on Applied natural language processing, p.16-19, March 31-April 03, 1997, Washington, DC [doi>10.3115/974557.974561]
• 21: N. Tishby, F. Pereira, and W. Bialek. The information bottleneck method. In Proceedings of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368--377, 1999.
• 22: T. Westerveld and A. P. de Vries. Multimedia retrieval using multiple examples. In Proceedings of Conference on Image and Video Retrieval CIVR, Dublin, Ireland, July 2004.
• 23: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 24: Hui Yang , Lekha Chaisorn , Yunlong Zhao , Shi-Yong Neo , Tat-Seng Chua, VideoQA: question answering on news video, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957146]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Experimentation

Keywords:
TRECVID, joint visual-text models, multimedia retrieval models, retrieval models