A bootstrapping framework for annotating and retrieving WWW images

ABSTRACT

Most current image retrieval systems and commercial search engines use mainly text annotations to index and retrieve WWW images. This research explores the use of machine learning approaches to automatically annotate WWW images based on a predefined list of concepts by fusing evidences from image contents and their associated HTML text. One major practical limitation of employing supervised machine learning approaches is that for effective learning, a large set of labeled training samples is needed. This is tedious and severely impedes the practical development of effective search techniques for WWW images, which are dynamic and fast-changing. As web-based images possess both intrinsic visual contents and text annotations, they provide a strong basis to bootstrap the learning process by adopting a co-training approach involving classifiers based on two orthogonal set of features -- visual and text. The idea of co-training is to start from a small set of labeled training samples, and successively annotate a larger set of unlabeled samples using the two orthogonal classifiers. We carry out experiments using a set of over 5,000 images acquired from the Web. We explore the use of different combinations of HTML text and visual representations. We find that our bootstrapping approach can achieve a performance comparable to that of the supervised learning approach with an F1 measure of over 54%. At the same time, it offers the added advantage of requiring only a small initial set of training samples.

REFERENCES

• 1: Y. Alp Aslandogan , Clement T. Yu, Multiple evidence combination in image retrieval: Diogenes searches for people on the Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.88-95, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345553]
• 2: K. Barnard & D.A. Forsyth. Learning the semantics of words and pictures. IEEE International Conference on Computer Vision II, 408--415 (2001).
• 3: F. Bergeaud , S. Mallat, Matching pursuit of images, Proceedings of the 1995 International Conference on Image Processing (Vol. 1)-Volume 1, p.53, October 23-26, 1995
• 4: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 5: Chad Carson , Megan Thomas , Serge Belongie , Joseph M. Hellerstein , Jitendra Malik, Blobworld: A System for Region-Based Image Indexing and Retrieval, Proceedings of the Third International Conference on Visual Information and Information Systems, p.509-516, June 02-04, 1999
• 6: Edward Chang, Kingshy Goh, Gerard Sychay & Gang Wu. CBSA: content-based soft annotation for multimodal image retrieval using Bayes Point Machines. IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on Conceptual and Dynamical Aspects of Multimedia Content Description 13, 26--38 (2003).
• 7: T. S. Chua, Y. Zhao, L. Chaisorn, C.-K. Koh, H. Yang, H. Xu and Q. Tian. TREC 2003 Video Retrieval and Story Segmentation Task at NUS PRIS. 2003. http://www-nlpir.nist.gov/projects/tv.pubs.org
• 8: M. Collins & Y. Singer. Unsupervised models for name entity classification. Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural language Processing and Very Large Corpora. 1999.
• 9: HuaMin Feng , Tat-Seng Chua, A bootstrapping approach to annotating large image collection, Proceedings of the 5th ACM SIGMM international workshop on Multimedia information retrieval, November 07-07, 2003, Berkeley, California [doi>10.1145/973264.973274]
• 10: A. Hauptman, R.V. Baron, M.-Y. Chen, M. Christel, P. Duygulu, C. Huang, R. Jin, W.-H Lin, T. Ng, N. Moraveji, N. Papernick, C.G.M. Snoek, G. Tzanetakis, J. Yang, & H.D Wactlar. Informedia at TRECVID 2003: analyzing and searching broadcast news video, 2003, http://www-nlpir.nist.gov/projects/tv.pubs.org
• 11: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 12: Rainer Lienhart & Alex Hartmann. "Classifiying images ont eh web automatically". Joutrnal of electronic imaging. 11(4), Oct 2002. 1--10.
• 13: S.G.. Mallat & Z.F. Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41(12), 3397--3415, 1993.
• 14: B. S. Manjunath , W. Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.837-842, August 1996 [doi>10.1109/34.531803 ]
• 15: Y. Mori, H. Takahashi & R. Oka. Image-to-word transformation based on dividing and vector quantizing images with words. First International Workshop on multimedia Intelligent Storage and Retrieval Management (1999).
• 16: Kamal Nigam , Rayid Ghani, Analyzing the effectiveness and applicability of co-training, Proceedings of the ninth international conference on Information and knowledge management, p.86-93, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354805]
• 17: Lexin Pan. Image8: an image search engine for the Internet. Honours year project report. School of Computing, National University of Singapore, Apr 2003.
• 18: D. Pierce & C. Cardie. Limitations of co-training for natural language learning from large datasets. Proceeding of Empirical Methods in Natural Language Processing. 2001.
• 19: J.C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In 'Advances in Large Margin Classifiers', A.J. Smola, P. Bartlett, B. Scholkopf & D. Schuurmans (Eds). MIT Press, 1999.
• 20: H.R. Rabiee, R.L. kashyap & S.R. Safavian. Adaptive multiresolution image coding with matching and basis pursuits. IEEE ICIP '96, EPFL, Switzerland, Sept, 1996.
• 21: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 22: V. Harmandas , M. Sanderson , M. D. Dunlop, Image retrieval by hypertext links, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.296-303, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 23: Heng Tao Shen , Beng Chin Ooi , Kian-Lee Tan, Giving meanings to WWW images, Proceedings of the eighth ACM international conference on Multimedia, p.39-47, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.376098]
• 24: Michael Shneier , Mohamed Abdel-Mottaleb, Exploiting the JPEG Compression Scheme for Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.849-853, August 1996 [doi>10.1109/34.531805 ]
• 25: A. Smeanton, W. Kraaij & P. Over. TRECVID 2003 - An Introduction. http://www-nlpir.nist.gov/projects/tv.pubs.org, 2003.
• 26: John R. Smith , Shih-Fu Chang, VisualSEEk: a fully automated content-based image query system, Proceedings of the fourth ACM international conference on Multimedia, p.87-98, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244151]
• 27: R. Shi, H. Feng, T.-S. Chua & C.-H. Lee. An adaptive image content representation and segmentation approach to automatic image annotation. To appear in Conference on Image and Video Retrieval (CIVR'04), Dublin, Jul 2004.
• 28: M. Unser. Texture classification and segmentation using Wavelet frames. IEEE Transactions on Image Processing, 4(11), 1549--1560, 1995.
• 29: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 30: James Z. Wang , Jia Li, Learning-based linguistic indexing of pictures with 2--d MHMMs, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641104]
• 31: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109 ]
• 32: Keiji Yanai, Generic image classification using visual knowledge on the web, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957047]
• 33: C. Zhang & T. Chen. An active learning framework for content-based information retrieval. IEEE transactions on multimedia. 4, 260--268, 2002.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Object recognition

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
WWW images, bootstrapping, co-training, image annotation