Incorporating concept ontology to enable probabilistic concept reasoning for multi-level image annotation

ABSTRACT

To enable automatic multi-level image annotation, we have addressed two inter-related important issues:(1)more effective framework for image content representation and feature extraction to characterize the middle-level semantics of image contents;(2)new framework for hierarchical probabilistic image concept reasoning and detection. To address the first issue salient objects are used as the semantic building blocks to characterize the middle-level semantics of image contents effectively while reducing the image analysis cost significantly. We have proposed three approaches to designing the detection functions for automatic salient object detection,and automatic function selection is also supported to find the "right "assumptions of the principal visual properties for the corresponding salient object classes. To address the second issue wehaveproposed a novel framework to incorporate the concept ontology to achieve hierarchical probabilistic image concept reasoning for multi-level image annotation. The concept ontology for a large-scale public image database called Label Me is semi-automatically derived from the available image labels by using WordNet The image concepts at the first level of the concept ontology are used to characterize the most specific semantics of image contents with the smallest variations, and their correspondences with the semantic building blocks (i.e.,salient objects)are well-de fined and can be modeled accurately by using Bayesian networks. In addition,the predictions of the appearances of the higher-level image concepts with large variations are adopted by the underlying concept ontology or by combining the available predictions of the appearances of their children concepts through hierarchical Bayesian networks.Our experiments on a large public dataset have shown that our framework for hierarchical probabilistic image concept reasoning is scalable to diverse image contents (i.e.,large amount of salient object classes)with large within-category variations.

References

1. Y. Rui, T. S. Huang, and S.-F. Chang, "Image Retrieval:Current Techniques,Promising Directions and Open Issues", Journal of Visual Communication and Image Representation Vol.10, pp.39--62, 1999.
2. F. Monay, D. Gatica-Perez,"On image auto-annotation with latent space models", ACM Multimedia, pp.275--278, 2003.
3. A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain,"Content-based image retrieval at the end of the early years", IEEE Trans. on PAMI vol. 22, pp.1349--1380, 2000.
4. R. Zhao, W. I. Grosky, "Negotiating the semantic gap: from feature maps to semantic landscapes", Pattern Recognition vol.35, no.3, pp.593--600, 2002.
5. X. He, W.-Y. Ma, O. King, M. Li and H. J. Zhang, "Learning and inferring a semantic space from user 's relevance feedback", ACM Multimedia,2002.
6. R. Lienhart and A. Hartmann," Classifying images on the web automatically", Journal of Electronic Imaging vol. 11, no.4, pp. 445--454, 2002.
7. C. Carson, S. Belongie, H. Greenspan, J. Malik, "Blobworld: Image segmentation using expectation-maximization and its application to image querying ", IEEE Trans. PAMI 2002.
8. Y. Gong," Advancing Content-Based Image Retrieval by Exploiting Image Color and Region Features ", Multimedia Systems vol.7, no.6, pp.449--457, 1999.
9. K. Vu, K. A. Hua, W. Tavanapong, "Image Retrieval Basedon Regions of Interest", IEEE Trans. TKDE vol.15, no.4, pp. 1045--1049, 2003.
10. J. Z. Wang, J. Li and G. Wiederhold, "SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries", IEEE Trans. on PAMI vol.23, no.9, pp. 947--963, 2001.
11. J. R. Smith and C.-S. Li,"Image classification and querying using composite region template ", Journal of CVIU 1999.
12. J. Fan, Y. Gao, H. Luo, "Multi-level annotation of natural scenes using dominant image components and semantic image concepts", ACM Multimedia, 2004.
13. A. B. Benitez, S.-F. Chang, "Image classi fication using multimedia knowledge networks", ICIP, pp.613--616, 2003.
14. A. B. Benitez, J. R. Smith, S.-F. Chang,"MediaNet: A multimedia information network for knowledge representation", SPIE, vol. 4210, 2000.
15. S.-F. Chang, J. R. Smith, M. Beigi, A. B. Benitez, "Visual information retrieval from large distributed on-line repositories", Comm. of the ACM vol.40, no. 12, pp.63--71, 1997.
16. Y. A. Aslandogan, C. T. Yu, "Evaluating strategies and systems for content based indexing of person images on the Web", ACM Multimedia, 2000.
17. J. Huang, S. Kumar, R. Zabih, "An automatic hierarchical image classi fication scheme", ACM multimedia, 1998.
18. A. G. Hauptmann,"Towards a large scale concept ontology for broadcast video", CIVR, 2004.
19. A. Natsev, M. R. Naphade, J. R. Smith, "Semantic representation: search and mining of multimedia content", KDD, pp.641--646, 2004.
20. J. Li and J. Z. Wang,"Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach", IEEE Trans. on PAMI vol.25, no.9, pp. 1075--1088, 2003.
21. K. Barnard and D. Forsyth,"Learning the semantics of words and pictures", Proc. ICCV, pp.408--415, 2001.
22. N. Vasconcelos, "Image indexing with mixture hierarchies", IEEE CVPR, 2001.
23. J. Fan, H. Luo, Y. Gao, M.-S. Hacid, "Mining image databases on semantics via statistical learning", ACM SIGKDD, 2005.
24. F. Monay, D. Gatica-Perez, "PLSA-based image auto-annotation:constraining the latent space", ACM Multimedia, pp. 348--351, 2004.
25. N. Serrano, A. E. Savakis, J. Luo, "Improved scene classification using efficient low-level features and semantic cues ",Pattern Recognition vol.37, no.9, pp.1773--1784, 2004.
26. R. Jin, A. G. Hauptmann, "Using a probabilistic source model for comparing images", ICIP, pp.941--944, 2002.
27. A. Vailaya, M. Figueiredo, A. K. Jain, H. J. Zhang, "Image classification for content-based indexing ", IEEE Trans. on Image Processing vol.10, pp. 117--130, 2001.
28. C. Fellbaum, WordNet: An Electronic Lexical Database MIT Press, 1998.
29. D. Lowe,"Distinctive image features fromscale-invariant keypoints ", International Journal of Computer Vision 2004.
30. L. Fei-Fei, R. Fergus, P. Perona, "A Bayesian approach to unsupervised One-Shot learning of Object categories", IEEE ICCV, 2003.
31. M. Sanderson, B. Croft, "Deriving concept hierarchies from text ", ACM SIGIR, 1999.
32. D. J. Lawrie, B. Croft, "Generating hierarchical summaries for web searches", ACM SIGIR, 2003.
33. K. Toutanova, F. Chen, K. Popat, T. Hofmann, "Text Classification in a Hierarchical Mixture Model for Small Training Sets", ACM CIKM, 2001.
34. S. Dumais, H. Chen, "Hierarchical classification of Web content", ACM SIGIR, 2000.
35. D. Comaniciu, Peter Meer, "Mean Shift: A robust approach toward feature space analysis", IEEE Trans. on PAMI vol.24, no.5, 2002.
36. Y. Freund, R. E. Schapire, "Experiments with a new boosting algorithm", Proc. ICML, pp. 148--156, 1996.
37. A. Torralba, K. Murphy, W. Freeman, "Sharing features: effcient boosting procedures for multiclass object detection", CVPR, 2004.
38. P. Viola, M. Jones, "Robust real-time face detection", Intl. J. ComputerVision vol. 57, no. 2, 2004.
39. J. C. Platt, "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods", in Adavances in Large Margin Classifiers MIT Press, 1999.
40. Y. Gao, J. Fan, "Semantic Image Classification with Hierarchical Feature Subset Selection", ACM SIGMM International Workshop on Multimedia Information Retrieval, November 10--11, 2005, Singapore.
41. Y. Gao, J. Fan, H. Luo, X. Xue, R. Jain, "Automatic image annotation by incorporating feature hierarchy and boosting to scale up SVM classi fiers", ACM Multimedia, 2006.
42. D. Heckerman, D. Geiger, D. Chickering, "Learning Bayesian networks: The combination of knowkedge and statistical data", Machine Learning vol.20, 1995.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Object recognition

• Information systems

  • Information systems applications

    • Multimedia information systems

      • Multimedia databases