Hybrid visual and conceptual image representation within active relevance feedback context

ABSTRACT

Many of the available image databases have keyword annotations associated with the images. In spite of the availability of good quality low-level visual features that reflect well the physical content, image retrieval based on visual features alone is subject to semantic gap. Text annotations are related to image context or semantic interpretation of the visual content and are not necessarely directly linked to the visual appearance of the images. Keywords and visual features thus provide complementary information. Using both sources of information is an advantage in many applications and recent work in this area reflects this interest. In this paper, we address the challenge of semantic gap reduction using a hybrid visual and conceptual representation of the content within an active relevance feedback context. We introduce a new feature vector, based on the keyword annotations available for the images, which makes use of conceptual information extracted from an external lexical database, information represented by a set of "core concepts". Our experiments show that the use of the proposed hybrid conceptual and visual feature vector dramatically improves the quality of the relevance feedback results.

REFERENCES

• 1: W. H. Adams, G. Iyengar, C.-Y. Lin, M. R. Naphade, C. Neti, H. J. Nock, and J. R. Smith. Semantic indexing of multimedia content using visual, audio and text cues. EURASIP Journal on Applied Signal Processing, 3(2):170--185, 2003.
• 2: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 3: R. Beckwith, C. Fellbaum, D. Gross, and G. Miller. WordNet: A lexical database organized on psycholinguistic principles. In U. Zernik, editor, Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon, pages 211--232. Erlbaum, 1991.
• 4: N. Boujemaa, J. Fauqueur, M. Ferecatu, F. Fleuret, V. Gouet, B. L. Saux, and H. Sahbi. Ikona: Interactive generic and specific image retrieval. In Proceedings of the International workshop on Multimedia Content-Based Indexing and Retrieval (MMCBIR'2001), 2001.
• 5: A. Budanitsky and G. Hirst. Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures. In Proceedings of the Workshop on WordNet and Other Lexical Resources NAACL 2001, 2001.
• 6: I. J. Cox , M. L. Miller , T. P. Minka , P. N. Yianilos, An Optimized Interaction Strategy for Bayesian Relevance Feedback, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.553, June 23-25, 1998
• 7: Alberto Del Bimbo, Visual information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 8: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 9: C. Fellbaum and G. Miller, editors. WordNet: An Electronic Lexical Database. The MIT Press, 1998.
• 10: Marin Ferecatu , Michel Crucianu , Nozha Boujemaa, Retrieval of difficult image classes using svd-based relevance feedback, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026716]
• 11: F. Fleuret and H. Sahbi. Scale-invariance of support vector machines based on the triangular kernel. In 3rd International Workshop on Statistical and Computational Theories of Vision, October 2003.
• 12: T. Gevers and A. W. M. Smeulders. Content-based image retrieval: An overview. In G. Medioni and S. B. Kang, editors, Emerging Topics in Computer Vision. Prentice Hall, 2004.
• 13: M. La Cascia , S. Sethi , S. Sclaroff, Combining Textual and Visual Cues for Content-Based Image Retrieval on the World Wide Web, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.24, June 21-21, 1998
• 14: Claudia Leacock , George A. Miller , Martin Chodorow, Using corpus statistics and WordNet relations for sense identification, Computational Linguistics, v.24 n.1, March 1998
• 15: Michael S. Lew, Principles of visual information retrieval, Springer-Verlag, London, 2000
• 16: Dekang Lin, An Information-Theoretic Definition of Similarity, Proceedings of the Fifteenth International Conference on Machine Learning, p.296-304, July 24-27, 1998
• 17: V. Mezaris, I. Kompatsiaris, and M. G. Strintzis. Region-based image retrieval using an object ontology and relevance feedback. EURASIP Journal on Applied Signal Processing, 2004(6):886--901, June 2004.
• 18: Pabitra Mitra , C. A. Murthy , Sankar K. Pal, A Probabilistic Active Support Vector Learning Algorithm, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.3, p.413-418, March 2004 [doi>10.1109/TPAMI.2004.1262340]
• 19: P. Resnik. Using information content to evaluate semantic similarity in a taxonomy. In C. S. Mellish, editor, Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 448--453, San Mateo, Aug. 20-25 1995. Morgan Kaufmann.
• 20: B. Schölkopf. The kernel trick for distances. In Advances in Neural Information Processing Systems, volume 12, pages 301--307. MIT Press, 2000.
• 21: F. Seydoux and J.-C. Chappelier. Indexation sémantique au moyen de coupes de redondance minimale dans une ontologie. In Proceedings of Traitement Automatique du Langage Naturel (TALN'05), pages 33--42, June 2005.
• 22: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972 ]
• 23: J. R. Smith, S. Basu, C.-Y. Lin, M. R. Naphade, and B. Tseng. Integrating features, models and semantics for content-based retrieval. In Proceedings of the international workshop on MultiMedia Content-Based Indexing and Retrieval (MMCBIR'01), pages 95--98, September 2001.
• 24: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 25: Zhibiao Wu , Martha Palmer, Verbs semantics and lexical selection, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.133-138, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981751]
• 26: H.-J. Zhang and Z. Su. Improving CBIR by semantic propagation and cross-mode query expansion. In Proceedings of the international workshop on MultiMedia Content-Based Indexing and Retrieval (MMCBIR'01), pages 83--86, September 2001.
• 27: Xiang Sean Zhou , Thomas S. Huang, Unifying Keywords and Visual Contents in Image Retrieval, IEEE MultiMedia, v.9 n.2, p.23-33, April 2002 [doi>10.1109/93.998050 ]
• 28: X. S. Zhou and T. S. Huang. Relevance feedback for image retrieval: a comprehensive review. Multimedia Systems, 8(6):536--544, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Algorithms, Experimentation, Performance

Keywords:
image descriptors, image retrieval, keyword annotations, relevance feedback