Multi-level annotation of natural scenes using dominant image components and semantic concepts

ABSTRACT

Automatic image annotation is a promising solution to enable semantic image retrieval via keywords. In this paper, we propose a multi-level approach to annotate the semantics of <b><i>natural scenes</i></b> by using both the dominant image components (salient objects) and the relevant semantic concepts. To achieve automatic image annotation at the content level, we use salient objects as the dominant image components for image content representation and feature extraction. To support automatic image annotation at the concept level, a novel image classification technique is developed to map the images into the most relevant semantic image concepts. In addition, Support Vector Machine (SVM) classifiers are used to learn the detection functions for the pre-defined salient objects and finite mixture models are used for semantic concept interpretation and modeling. An <b><i>adaptive EM algorithm</i></b> has been proposed to determine the optimal model structure and model parameters simultaneously. We have also demonstrated that our algorithms are very effective to enable multi-level annotation of <b><i>natural scenes</i></b> in a large-scale image dataset.

REFERENCES

• 1: John R. Smith , Shih-Fu Chang, Visually Searching the Web for Content, IEEE MultiMedia, v.4 n.3, p.12-20, July 1997 [doi>10.1109/93.621578 ]
• 2: E. Chang, "Statistical learning for effective visual information retrieval", Proc. ICIP, 2003.
• 3: Xiaofei He , Wei-Ying Ma , Oliver King , Mingjing Li , Hongjiang Zhang, Learning and inferring a semantic space from user's relevance feedback for image retrieval, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641080]
• 4: John R. Smith , Chung-Sheng Li, Image classification and querying using composite region templates, Computer Vision and Image Understanding, v.75 n.1-2, p.165-174, July/Aug. 1999 [doi>10.1006/cviu.1999.0771 ]
• 5: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 6: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 7: Martin Szummer , Rosalind W. Picard, Indoor-Outdoor Image Classification, Proceedings of the 1998 International Workshop on Content-Based Access of Image and Video Databases (CAIVD '98), p.42, January 03-03, 1998
• 8: R. Schettini, A. Valsasna, C. Brambilla, M. De Ponti, "A indoor/outdoor/close-up photo classifier", Proc. Color Imaging, 2001.
• 9: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Region-Based Image Querying, Proceedings of the 1997 Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '97), p.42, June 20-20, 1997
• 10: Jing Huang , S. Ravi Kumar , Ramin Zabih, An automatic hierarchical image classification scheme, Proceedings of the sixth ACM international conference on Multimedia, p.219-228, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290774]
• 11: N. Campbell, B. Thomas, T. Troscianko, "Automatic segmentation and classification of outdoor images using neural networks", Intl. Journal of Neural Systems, vol.8, pp.137--144, 1997.
• 12: James Ze Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-sensitive Integrated Matching for Picture Libraries, Proceedings of the 4th International Conference on Advances in Visual Information Systems, p.360-371, November 02-04, 2000
• 13: A. Vailaya, M. Figueiredo, A.K. Jain, H.J. Zhang, "Image classification for content-based indexing", IEEE Trans. on Image Processing, vol.10, 2001.
• 14: A. Hartmann, R. Lienhart, "Automatic classification of images on the web", Proc. SPIE, vol.4676, 2002.
• 15: E. Chang, K. Goh, G. Sychay, G. Wu, "CBSA: Content-based annotation for multimodal image retrieval using Bayes point machines", IEEE Trans. CSVT, 2002.
• 16: Beitao Li , Kingshy Goh, Confidence-based dynamic ensemble for image annotation and semantics discovery, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957051]
• 17: A. Mojsilovic, J. Gomes, B. Rogowitz, "ISee: Perceptual features for image library navigation", Proc. SPIE, 2001.
• 18: Antonio B. Torralba , Aude Oliva, Semantic Organization of Scenes Using Discriminant Structural Templates, Proceedings of the International Conference on Computer Vision-Volume 2, p.1253, September 20-25, 1999
• 19: J.R. Smith and S.-F. Chang, "Multi-stage classification of images from features and related text", Proc. DELOS, 1997.
• 20: Florent Monay , Daniel Gatica-Perez, On image auto-annotation with latent space models, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957070]
• 21: J. Luo and S. Etz, "A physical model-based approach to detecting sky in photographic images", IEEE Trans. on Image Processing, vol.11, 2002.
• 22: S.F. Chang, W. Chen, H. Sundaram, "Semantic visual template: Linking visual features to semantics", Proc. ICIP, 1998.
• 23: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 24: Cha Zhang , Tsuhan Chen, Indexing and retrieval of 3D models aided by active learning, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500261]
• 25: Dorin Comaniciu , Peter Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.603-619, May 2002 [doi>10.1109/34.1000236 ]
• 26: Y. Wu, Q. Tian, T.S. Huang, "Discriminant-EM algorithm with application to image retrieval", Proc. CVPR, pp.222--227, 2000.
• 27: J. Lin, "Divergence measures based on the Shannon entropy", IEEE Trans. on IT, vol.37, no.1, 1991.
• 28: A.B. Benitez, J.R. Smith and S.-F. Chang, "MediaNet: A multimedia information network for knowledge representation", Proc. SPIE, vol.4210, 2000.
• 29: Hayit Greenspan , Jacob Goldberger , Arnaldo Mayer, Probabilistic Space-Time Video Modeling via Piecewise GMM, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.3, p.384-396, March 2004 [doi>10.1109/TPAMI.2004.1262334]
• 30: K. Barnard and D. Forsyth, "Learning the semantics of words and pictures", Proc. ICCV, pp.408--415, 2001.
• 31: M.R. Naphade, X. Zhou, and T.S. Huang, "Image classification using a set of labeled and unlabeled images", Proc. SPIE, 2000.
• 32: M.R. Naphade and T.S. Huang, "A probabilistic framework for semantic video indexing, filtering, and retrival", IEEE Trans. on Multimedia, vol.3, pp.141--151, 2001.
• 33: R. Oami, A. Benitez, S.-F. Chang, N. Dimitrova, "Understanding and modeling user interests in consumer videos", ICME, 2004.
• 34: Naonori Ueda , Ryohei Nakano , Zoubin Ghahramani , Geoffrey E. Hinton, SMEM algorithm for mixture models, Proceedings of the 1998 conference on Advances in neural information processing systems II, p.599-605, July 1999
• 35: B. Zhang, C. Zhang, X. Yi, "Competitive EM algorithm for finite mixture models", Pattern Recognition, vol.37, pp.131--144, 2004.
• 36: Mario A. T. Figueiredo , Anil K. Jain, Unsupervised Learning of Finite Mixture Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.3, p.381-396, March 2002 [doi>10.1109/34.990138 ]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Object recognition

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
adaptive EM algorithm, automatic image annotation, salient objects