Online Multimodal Co-indexing and Retrieval of Weakly Labeled Web Image Collections

ABSTRACT

Weak supervisory information of web images, such as captions, tags, and descriptions, make it possible to better understand images at the semantic level. In this paper, we propose a novel online multimodal co-indexing algorithm based on Adaptive Resonance Theory, named OMC-ART, for the automatic co-indexing and retrieval of images using their multimodal information. Compared with existing studies, OMC-ART has several distinct characteristics. First, OMC-ART is able to perform online learning of sequential data. Second, OMC-ART builds a two-layer indexing structure, in which the first layer co-indexes the images by the key visual and textual features based on the generalized distributions of clusters they belong to; while in the second layer, images are co-indexed by their own feature distributions. Third, OMC-ART enables flexible multimodal search by using either visual features, keywords, or a combination of both. Fourth, OMC-ART employs a ranking algorithm that does not need to go through the whole indexing system when only a limited number of images need to be retrieved. Experiments on two published data sets demonstrate the efficiency and effectiveness of our proposed approach.

References

1. J. C. Caicedo, J. BenAbdallah, F. A. González, and O. Nasraoui. Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization. Neurocomputing, 76(1):50--60, 2012.
2. J. C. Caicedo, J. G. Moreno, E. A. Niño, and F. A. González. Combining visual features and text data for medical image retrieval using latent semantic kernels. In Proceedings of the international conference on Multimedia information retrieval, pages 359--366, 2010.
3. P. Chandrika and C. V. Jawahar. Multi modal semantic indexing for image retrieval. In CIVR, pages 342--349, 2010.
4. T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng. NUS-WIDE: a real-world web image database from national university of singapore. In CIVR, 2009.
5. L. De Lathauwer, B. De Moor, and J. Vandewalle. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4):1253--1278, 2000.
6. P. Duygulu, K. Barnard, J. F. de Freitas, and D. A. Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In ECCV, pages 97--112, 2002.
7. H. J. Escalante, M. Montes, and E. Sucar. Multimodal indexing based on semantic cohesion for image retrieval. Information Retrieval, 15(1):1--32, 2012.
8. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In Proceedings of the European Conference on Computer Vision (ECCV), pages 529--545, 2014.
9. M. Li, X.-B. Xue, and Z.-H. Zhou. Exploiting multi-modal interactions: A unified framework. pages 1120--1125, 2009.
10. R. Lienhart, S. Romberg, and E. Hörster. Multilayer pLSA for multimodal image retrieval. In Proceedings of the ACM International Conference on Image and Video Retrieval, 2009.
11. T. Mei, Y. Rui, S. Li, and Q. Tian. Multimedia search reranking: A literature survey. ACM Computing Surveys (CSUR), 46(3):38, 2014.
12. L. Meng and A.-H. Tan. Semi-supervised hierarchical clustering for personalized web image organization. In Proceedings of International Joint Conference on Neural Networks (IJCNN), pages 1--8, 2012.
13. L. Meng and A.-H. Tan. Community discovery in social networks via heterogeneous link association and fusion. In Proceedings of the SIAM International Conference on Data Mining (SDM), pages 803--811, 2014.
14. L. Meng, A.-H. Tan, and D. C. Wunsch. Vigilance adaptation in adaptive resonance theory. In Proceedings of International Joint Conference on Neural Networks (IJCNN), pages 1--7, 2013.
15. L. Meng, A.-H. Tan, and D. Xu. Semi-supervised heterogeneous fusion for multimedia data co-clustering. IEEE Transactions on Knowledge and Data Engineering, 26(9):2293--2306, 2014.
16. Y. Mu, J. Shen, and S. Yan. Weakly-supervised hashing in kernel space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3344--3351, 2010.
17. L. Nie, M. Wang, Y. Gao, Z.-J. Zha, and T.-S. Chua. Beyond text QA: Multimedia answer generation by harvesting web information. IEEE Transactions on Multimedia, 15(2):426--441, 2013.
18. L. Nie, M. Wang, Z.-J. Zha, G. Li, and T.-S. Chua. Multimedia answering: Enriching text QA with media information. In SIGIR, pages 695--704, 2011.
19. A. W. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12):1349--1380, 2000.
20. J.-H. Su, B.-W. Wang, T.-Y. Hsu, C.-L. Chou, and V. S. Tseng. Multi-modal image retrieval by integrating web image annotation, concept matching and fuzzy ranking techniques. International Journal of Fuzzy Systems, 12(2):136--149, 2010.
21. F. X. Yu, R. Ji, M.-H. Tsai, G. Ye, and S.-F. Chang. Weak attributes for large-scale image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2949--2956, 2012.
22. S. Zhang, M. Yang, X. Wang, Y. Lin, and Q. Tian. Semantic-aware co-indexing for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1673--1680, 2013.

Index Terms

• Information systems

  • Information retrieval

    • Document representation

    • Retrieval tasks and goals

      • Clustering and classification

      • Document filtering

      • Information extraction

    • Search engine architectures and scalability

      • Search engine indexing

  • Information systems applications

    • Data mining

      • Clustering