Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching

ABSTRACT

Automatic image annotation automatically labels image content with semantic keywords. For instance, the Relevance Model estimates the joint probability of the keyword and the image [3]. Most of the previous annotation methods assign keywords separately. Recently the correlation between annotated keywords has been used to improve image annotation. However, directly estimating the joint probability of a set of keywords and the unlabeled image is computationally prohibitive. To avoid the computation difficulty we propose a heuristic greedy iterative algorithm to estimate the probability of a keyword subset being the caption of an image. In our approach, the correlations between keywords are analyzed by "Automatic Local Analysis" of text information retrieval. In addition, a new image generation probability estimation method is proposed based on region matching. We demonstrate that our iterative annotation algorithm can incorporate the keyword correlations and the region matching approaches handily to improve the image annotation significantly. The experiments on the ECCV2002 [2] benchmark show that our method outperforms the state-of-the-art continuous feature model MBRM with recall and precision improving 21% and 11% respectively.

REFERENCES

• 1: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 2: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 3: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 4: S. L. Feng, R. Manmatha, V. Lavrenko. Multiple Bernoulli Relevance Models for Image and Video Annotation. In Proc. of IEEE CVPR'04, 2004.
• 5: V. Lavrenko, R. Manmatha, and J. Jeon. A Model for Learning the Semantics of Pictures. In Proc. of NIPS'03, 2003.
• 6: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109 ]
• 7: Ruofei Zhang , Zhongfei (Mark) Zhang, A Clustering Based Approach to Efficient Image Retrieval, Proceedings of the 14th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'02), p.339, November 04-06, 2002
• 8: H. W. Kuhn, The Hungarian Method for the Assignment Problem, Naval Research Logistics Quarterly, vol.2, p. 83--97, 1955.
• 9: Rong Jin , Joyce Y. Chai , Luo Si, Effective automatic image annotation via a coherent language model and active learning, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027732]
• 10: Yohan Jin , Latifur Khan , Lei Wang , Mamoun Awad, Image annotations by combining multiple evidence & wordNet, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101305]
• 11: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 12: Qi Zhang , Sally A. Goldman , Wei Yu , Jason Fritts, Content-Based Image Retrieval Using Multiple-Instance Learning, Proceedings of the Nineteenth International Conference on Machine Learning, p.682-689, July 08-12, 2002
• 13: Munirathnam Srikanth , Joshua Varner , Mitchell Bowden , Dan Moldovan, Exploiting ontologies for automatic image annotation, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076128]
• 14: Florent Monay , Daniel Gatica-Perez, On image auto-annotation with latent space models, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957070]
• 15: Florent Monay , Daniel Gatica-Perez, PLSA-based image auto-annotation: constraining the latent space, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027608]
• 16: Yuli Gao , Jianping Fan , Xiangyang Xue , Ramesh Jain, Automatic image annotation by incorporating feature hierarchy and boosting to scale up SVM classifiers, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180840]
• 17: Jia Li , James Z. Wang, Real-time computerized annotation of pictures, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180841]
• 18: T. Hastie, R. Tibshirani, J. Friedman. The Element of Statistical Learining: data mining, inference, and prediction. Springer-Verlag, 2001.
• 19: Fei Song , W. Bruce Croft, A general language model for information retrieval, Proceedings of the eighth international conference on Information and knowledge management, p.316-321, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320022]
• 20: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688 ]
• 21: R. Shi, T. S. Chua, C. H. lee, S. Gao. Bayesian Learning of Hierarchical Multinomial Mixture Models of Concepts for Automatic Image Annotation. In proc. of Int'l Conf. on Image and Video Retrieval, 2006.
• 22: Tao Wang , Yong Rui , Jia-Guang Sun, Constraint Based Region Matching for Image Retrieval, International Journal of Computer Vision, v.56 n.1-2, p.37-45, January-February 2004
• 23: A. Yavlinsky, E. Schofield and S. Ruger. Annotation Using Global Features and Robust Nonparametric Density Estimation. In proc. of Int'l Conf. on Image and Video Retrieval, 2005.
• 24: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 25: G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross and K. J. Miller. Introduction to WordNet: an on-line lexical database. Intl. Jour. of Lexicography, vol. 3, pp. 235--244, 1990.
• 26: Jing Liu , Mingjing Li , Wei-Ying Ma , Qingshan Liu , Hanqing Lu, An adaptive graph model for automatic image annotation, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178689]
• 27: X. Zhou, L. Chen, J. Ye, Q. Zhang, B. Shi, Automatic Image Semantic Annotation Based on Keyword-Image Document Model. In proc. of Int'l Conf. on Image and Video Retrieval, 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Performance

Keywords:
continuous feature model, image annotation, words correlation