Image annotations by combining multiple evidence & wordNet

ABSTRACT

The development of technology generates huge amounts of non-textual information, such as images. An efficient image annotation and retrieval system is highly desired. Clustering algorithms make it possible to represent visual features of images with finite symbols. Based on this, many statistical models, which analyze correspondence between visual features and words and discover hidden semantics, have been published. These models improve the annotation and retrieval of large image databases. However, current state of the art including our previous work produces too many irrelevant keywords for images during annotation. In this paper, we propose a novel approach that augments the classical model with generic knowledge-based, WordNet. Our novel approach strives to prune irrelevant keywords by the usage of WordNet. To identify irrelevant keywords, we investigate various semantic similarity measures between keywords and finally fuse outcomes of all these measures together to make a final decision using Dempster-Shafer evidence combination. We have implemented various models to link visual tokens with keywords based on knowledge-based, WordNet and evaluated performance using precision, and recall using benchmark dataset. The results show that by augmenting knowledge-based with classical model we can improve annotation accuracy by removing irrelevant keywords.

REFERENCES

• 1: Yuksel Alp Aslandogan , Clement T. Yu, Diogenes: a web search agent for person images, Proceedings of the eighth ACM international conference on Multimedia, p.481-482, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.376393]
• 2: S. Banerjee and T. Pedersen. Extended gloss overlaps as a measure of semantic relatedness. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pages 805--810, 2003.
• 3: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 4: David M. Blei , Michael I. Jordan, Modeling annotated data, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860460]
• 5: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 6: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 7: J.Jiang and D.Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Procedeeings on International Conference on Research in Computational Linguistics, 1997.
• 8: Feng Kang , Rong Jin , Joyce Y. Chai, Regularizing translation models for better automatic image annotation, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA [doi>10.1145/1031171.1031242]
• 9: Kobus. http://www.cs.arizona.edu/people/kobus/research/data. 2002.
• 10: C. Leacock. Combining local context and wordnet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: A Lexical Reference System and its Application. MIT Press, Cambridge, MA., pages 265--283, 1998.
• 11: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 12: Dekang Lin, Using syntactic dependency as local context to resolve word sense ambiguity, Proceedings of the 35th annual meeting on Association for Computational Linguistics, p.64-71, July 07-12, 1997, Madrid, Spain
• 13: Michael Lesk, Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone, Proceedings of the 5th annual international conference on Systems documentation, p.24-26, June 1986, Toronto, Ontario, Canada [doi>10.1145/318723.318728]
• 14: Y. Mori, H. Takahashi, and R. Oka. Image-to-word transformation based on dividing and vector quantizing images with words. MISRM'99 Frist International Workshop on Multimedia Intellegent Storage and Retrieval Management, 1999.
• 15: Jia-Yu Pan , Hyung-Jeong Yang , Christos Faloutsos , Pinar Duygulu, Automatic multimedia cross-modal correlation discovery, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014135]
• 16: P. Resnik. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1995.
• 17: G. Shafer. A Mathematical Theory of Evidence. Princeton University Press, 1976.
• 18: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.731, June 17-19, 1997
• 19: R. M. V. Lavrenko and J. Jeon. A model for learning the semantics of pictures. Proceedings of the 17th Annual Conference on Neural Information Processing Systems, 2003.
• 20: L. Wang and L. Khan. Automatic image annotation and retrieval using weighted feature selection. To appear in International Journal of Multimedia Tools and Applications by Kluwer Publisher, 2005.
• 21: Y. Jin, L. Wang and L. Khan, Improving Image Annotations using WordNet, International Workshop on Multimedia Information Systems (MIS 2005), Sorrento, Italy, page: 115-130, September, 2005.Editor's Note: This last reference was added at the author's request after official publication of the proceedings.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Management

Keywords:
Dempster-Shafer rule, corel dataset, image annotation, semantic-similarity, wordNet