Incorporating concept ontology to enable probabilistic concept reasoning for multi-level image annotation

ABSTRACT

To enable automatic multi-level image annotation, we have addressed two inter-related important issues:(1)more effective framework for image content representation and feature extraction to characterize the middle-level semantics of image contents;(2)new framework for hierarchical probabilistic image concept reasoning and detection. To address the first issue salient objects are used as the semantic building blocks to characterize the middle-level semantics of image contents effectively while reducing the image analysis cost significantly. We have proposed three approaches to designing the detection functions for automatic salient object detection,and automatic function selection is also supported to find the "right "assumptions of the principal visual properties for the corresponding salient object classes. To address the second issue wehaveproposed a novel framework to incorporate the concept ontology to achieve hierarchical probabilistic image concept reasoning for multi-level image annotation. The concept ontology for a large-scale public image database called Label Me is semi-automatically derived from the available image labels by using WordNet The image concepts at the first level of the concept ontology are used to characterize the most specific semantics of image contents with the smallest variations, and their correspondences with the semantic building blocks (i.e.,salient objects)are well-de fined and can be modeled accurately by using Bayesian networks. In addition,the predictions of the appearances of the higher-level image concepts with large variations are adopted by the underlying concept ontology or by combining the available predictions of the appearances of their children concepts through hierarchical Bayesian networks.Our experiments on a large public dataset have shown that our framework for hierarchical probabilistic image concept reasoning is scalable to diverse image contents (i.e.,large amount of salient object classes)with large within-category variations.

REFERENCES

• 1: Y. Rui, T. S. Huang, and S.-F. Chang, "Image Retrieval:Current Techniques,Promising Directions and Open Issues", Journal of Visual Communication and Image Representation Vol.10, pp.39--62, 1999.
• 2: Florent Monay , Daniel Gatica-Perez, On image auto-annotation with latent space models, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957070]
• 3: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972 ]
• 4: R. Zhao, W. I. Grosky, "Negotiating the semantic gap: from feature maps to semantic landscapes", Pattern Recognition vol.35, no.3, pp.593--600, 2002.
• 5: Xiaofei He , Wei-Ying Ma , Oliver King , Mingjing Li , Hongjiang Zhang, Learning and inferring a semantic space from user's relevance feedback for image retrieval, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641080]
• 6: R. Lienhart and A. Hartmann," Classifying images on the web automatically", Journal of Electronic Imaging vol. 11, no.4, pp. 445--454, 2002.
• 7: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800 ]
• 8: Yihong Gong, Advancing content-based image retrieval by exploiting image color and region features, Multimedia Systems, v.7 n.6, p.449-457, Nov. 1999 [doi>10.1007/s005300050145 ]
• 9: K. Vu, K. A. Hua, W. Tavanapong, "Image Retrieval Basedon Regions of Interest", IEEE Trans. TKDE vol.15, no.4, pp. 1045--1049, 2003.
• 10: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109 ]
• 11: John R. Smith , Chung-Sheng Li, Image classification and querying using composite region templates, Computer Vision and Image Understanding, v.75 n.1-2, p.165-174, July/Aug. 1999 [doi>10.1006/cviu.1999.0771 ]
• 12: Jianping Fan , Yuli Gao , Hangzai Luo, Multi-level annotation of natural scenes using dominant image components and semantic concepts, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027660]
• 13: A. B. Benitez, S.-F. Chang, "Image classi fication using multimedia knowledge networks", ICIP, pp.613--616, 2003.
• 14: A. B. Benitez, J. R. Smith, S.-F. Chang,"MediaNet: A multimedia information network for knowledge representation", SPIE, vol. 4210, 2000.
• 15: Shih-Fu Chang , John R. Smith , Mandis Beigi , Ana Benitez, Visual information retrieval from large distributed online repositories, Communications of the ACM, v.40 n.12, p.63-71, Dec. 1997 [doi>10.1145/265563.265573]
• 16: Yuksel Alp Aslandogan , Clement T. Yu, Evaluating strategies and systems for content based indexing of person images on the Web, Proceedings of the eighth ACM international conference on Multimedia, p.313-321, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354514]
• 17: Jing Huang , S. Ravi Kumar , Ramin Zabih, An automatic hierarchical image classification scheme, Proceedings of the sixth ACM international conference on Multimedia, p.219-228, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290774]
• 18: A. G. Hauptmann,"Towards a large scale concept ontology for broadcast video", CIVR, 2004.
• 19: Apostol (Paul) Natsev , Milind R. Naphade , John R. Smith, Semantic representation: search and mining of multimedia content, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014133]
• 20: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 21: K. Barnard and D. Forsyth,"Learning the semantics of words and pictures", Proc. ICCV, pp.408--415, 2001.
• 22: N. Vasconcelos, "Image indexing with mixture hierarchies", IEEE CVPR, 2001.
• 23: Jianping Fan , Hangzai Luo , Mohand-Said Hacid, Mining images on semantics via statistical learning, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081877]
• 24: Florent Monay , Daniel Gatica-Perez, PLSA-based image auto-annotation: constraining the latent space, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027608]
• 25: N. Serrano, A. E. Savakis, J. Luo, "Improved scene classification using efficient low-level features and semantic cues ",Pattern Recognition vol.37, no.9, pp.1773--1784, 2004.
• 26: R. Jin, A. G. Hauptmann, "Using a probabilistic source model for comparing images", ICIP, pp.941--944, 2002.
• 27: A. Vailaya, M. Figueiredo, A. K. Jain, H. J. Zhang, "Image classification for content-based indexing ", IEEE Trans. on Image Processing vol.10, pp. 117--130, 2001.
• 28: C. Fellbaum, WordNet: An Electronic Lexical Database MIT Press, 1998.
• 29: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 30: Li Fei-Fei , Rob Fergus , Pietro Perona, A Bayesian Approach to Unsupervised One-Shot Learning of Object Categories, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1134, October 13-16, 2003
• 31: Mark Sanderson , Bruce Croft, Deriving concept hierarchies from text, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.206-213, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312679]
• 32: Dawn J. Lawrie , W. Bruce Croft, Generating hierarchical summaries for web searches, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860549]
• 33: Kristina Toutanova , Francine Chen , Kris Popat , Thomas Hofmann, Text classification in a hierarchical mixture model for small training sets, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502604]
• 34: Susan Dumais , Hao Chen, Hierarchical classification of Web content, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.256-263, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345593]
• 35: Dorin Comaniciu , Peter Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.603-619, May 2002 [doi>10.1109/34.1000236 ]
• 36: Y. Freund, R. E. Schapire, "Experiments with a new boosting algorithm", Proc. ICML, pp. 148--156, 1996.
• 37: A. Torralba, K. Murphy, W. Freeman, "Sharing features: effcient boosting procedures for multiclass object detection", CVPR, 2004.
• 38: Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
• 39: J. C. Platt, "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods", in Adavances in Large Margin Classifiers MIT Press, 1999.
• 40: Yuli Gao , Jianping Fan, Semantic image classification with hierarchical feature subset selection, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101850]
• 41: Y. Gao, J. Fan, H. Luo, X. Xue, R. Jain, "Automatic image annotation by incorporating feature hierarchy and boosting to scale up SVM classi fiers", ACM Multimedia, 2006.
• 42: David Heckerman , Dan Geiger , David M. Chickering, Learning Bayesian Networks: The Combination of Knowledge and Statistical Data, Machine Learning, v.20 n.3, p.197-243, Sept. 1995 [doi>10.1023/A:1022623210503]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Object recognition

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
bayesian network, concept ontology, hierarchical probabilistic image concept reasoning, multi-level image annotation