Social-Sensed Image Search

Abstract

Although Web search techniques have greatly facilitate users’ information seeking, there are still quite a lot of search sessions that cannot provide satisfactory results, which are more serious in Web image search scenarios. How to understand user intent from observed data is a fundamental issue and of paramount significance in improving image search performance. Previous research efforts mostly focus on discovering user intent either from clickthrough behavior in user search logs (e.g., Google), or from social data to facilitate vertical image search in a few limited social media platforms (e.g., Flickr). This article aims to combine the virtues of these two information sources to complement each other, that is, sensing and understanding users’ interests from social media platforms and transferring this knowledge to rerank the image search results in general image search engines. Toward this goal, we first propose a novel social-sensed image search framework, where both social media and search engine are jointly considered. To effectively and efficiently leverage these two kinds of platforms, we propose an example-based user interest representation and modeling method, where we construct a hybrid graph from social media and propose a hybrid random-walk algorithm to derive the user-image interest graph. Moreover, we propose a social-sensed image reranking method to integrate the user-image interest graph from social media and search results from general image search engines to rerank the images by fusing their social relevance and visual relevance. We conducted extensive experiments on real-world data from Flickr and Google image search, and the results demonstrated that the proposed methods can significantly improve the social relevance of image search results while maintaining visual relevance well.

References

1. Eugene Agichtein, Eric Brill, Susan T. Dumais, and Robert Ragno. 2006. Learning user interaction models for predicting web search result preferences. In Proceedings of SIGIR. 3--10.
2. Paul André, Edward Cutrell, Desney S. Tan, and Greg Smith. 2009. Designing novel image search interfaces by understanding unique characteristics and usage. In Proceedings of the 12th IFIPTC 13 International Conference on Human-Computer Interaction: Part I (INTERACT). 340--353.
3. Javed A. Aslam and Mark H. Montague. 2001. Models for metasearch. In Proceedings of SIGIR. 275--284.
4. Nadav Ben-Haim, Boris Babenko, and Serge Belongie 2006. Improving Web-based image search via content based clustering. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06). IEEE, 106--106.
5. Bisheng Chen, Jingdong Wang, Qinghua Huang, and Tao Mei. 2012a. Personalized video recommendation through tripartite graph propagation. In Proceedings of the 20th ACM International Conference on Multimedia. ACM, 1133--1136.
6. Lin Chen, Dong Xu, Ivor W. Tsang, and Jiebo Luo. 2012b. Tag-based image retrieval improved by augmented features and group-based refinement. IEEE Trans. Multimedia 14, 4, 1057--1067.
7. Lin Chen, Dong Xu, Ivor Wai-Hung Tsang, and Jiebo Luo. 2010. Tag-based Web photo retrieval improved by batch mode re-tagging. In Proceedings of CVPR. 3440--3446.
8. Paul-Alexandru Chirita,Wolfgang Nejdl, Raluca Paiu, and Christian Kohlschütter. 2005. Using ODP metadata to personalize search. In Proceedings of SIGIR. 178--185.
9. Youngok Choi and Edie M. Rasmussen 2003. Searching for images: The analysis of users’ queries for image retrieval in American history. J. Amer. Soc. Inf. Sci. Technol. 54, 6, 498--511.
10. Peng Cui, Fei Wang, Shaowei Liu, Mingdong Ou, Shiqiang Yang, and Lifeng Sun. 2011. Who should share what?: Item-level social influence prediction for users and posts ranking. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 185--194.
11. Ritendra Datta, Dhiraj Joshi, Jia Li, and James Ze Wang 2008. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40, 2.
12. Sotiris Diplaris, Symeon Papadopoulos, Ioannis Kompatsiaris, Ayse Goker, Andrew MacFarlane, Jochen Spangenberg, Hakim Hacid, Linas Maknavicius, and Matthias Klusch. 2012. Socialsensor: Sensing user generated input for improved media discovery and experience. In Proceedings of WWW (Companion Volume). 243--246.
13. Yue Gao, Meng Wang, Zheng-Jun Zha, Jialie Shen, Xuelong Li, and Xindong Wu. 2013. Visual-textual joint relevance learning for tag-based social image search. IEEE Trans. Image Process. 22, 1, 363--376.
14. Winston H. Hsu, Lyndon S. Kennedy, and Shih-Fu Chang. 2007. Video search reranking through random walk over document-level context graph. In Proceedings of the 15th International Conference on Multimedia. ACM, 971--980.
15. David A. Hull, Jan O. Pedersen, and Hinrich Schütze. 1996. Method combination for document filtering. In Proceedings of SIGIR. 279--287.
16. Vidit Jain and Manik Varma 2011. Learning to re-rank: Query-dependent image re-ranking using click data. In Proceedings of the 20th International Conference on World Wide Web. ACM, 277--286.
17. Bernard J. Jansen, Danielle L. Booth, and Amanda Spink. 2007. Determining the user intent of Web search engine queries. In Proceedings of WWW. 1149--1150.
18. Meng Jiang, Peng Cui, Rui Liu, Qiang Yang, Fei Wang, Wenwu Zhu, and Shiqiang Yang. 2012a. Social contextual recommendation. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. ACM, 45--54.
19. Meng Jiang, Peng Cui, Fei Wang, Qiang Yang, Wenwu Zhu, and Shiqiang Yang. 2012b. Social recommendation across multiple relational domains. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. ACM, 1422--1431.
20. Yushi Jing and Shumeet Baluja. 2008. VisualRank: Applying PageRank to large-scale image search. IEEE Trans. Patt. Anal. Mach. Intell. 30, 11, 1877--1890.
21. Martha Larson, Christoph Kofler, and Alan Hanjalic. 2011. Reading between the tags to predict real-world size-class for visually depicted objects in images. In Proceedings of the 19th ACM International Conference on Multimedia (MM). ACM, New York, NY, 273--282. DOI:http://dx.doi.org/10.1145/2072298.2072335.
22. Kristina Lerman, Anon Plangprasopchok, and Chio Wong. 2007. Personalizing image search results on Flickr. CoRR abs/0704.1676.
23. Xirong Li, Cees G. M. Snoek, and Marcel Worring. 2009. Learning social tag relevance by neighbor voting. IEEE Trans. Multimedia 11, 7, 1310--1322.
24. Marek Lipczak, Michele Trevisiol, and Alejandro Jaimes. 2013. Analyzing favorite behavior in Flickr. In Proceedings of the 19th International Conference on Advances in Multimedia Modeling. Lecture Notes in Computer Science, vol. 7732. Springer, 535--545.
25. Shaowei Liu, Peng Cui, Huanbo Luan, Wenwu Zhu, Shiqiang Yang, and Qi Tian. 2013. Social visual image ranking for Web image search. In Proceedings of the 19th International Conference on Advances in Multimedia Modeling. Lecture Notes in Computer Science, vol. 7732. Springer, 239--249.
26. Shaowei Liu, Peng Cui, Huanbo Luan, Wenwu Zhu, Shiqiang Yang, and Qi Tian. 2014. Social-oriented visual image search. Comput. Vision Image Understand. 118, 30--39.
27. Dongyuan Lu and Qiudan Li. 2011. Personalized search on Flickr based on searcher’s preference prediction. In Proceedings of WWW (Companion Volume). 81--82.
28. Einat Minkov and William W. Cohen. 2010. Improving graph-walk-based similarity with reranking: Case studies for personal information management. ACM Trans. Info. Syst. 29, 1, 4.
29. Radu Andrei Negoescu and Daniel Gatica-Perez. 2010. Modeling Flickr communities through probabilistic topic-based analysis. IEEE Trans. Multimedia 12, 5, 399--416.
30. David Nistér and Henrik Stewénius 2006. Scalable recognition with a vocabulary tree. In Proceedings of CVPR. 2161--2168.
31. Feng Qiu and Junghoo Cho. 2006. Automatic identification of user interest for personalized search. In Proceedings of WWW. 727--736.
32. Jitao Sang, Jing Liu, and Changsheng Xu. 2011. Exploiting user information for image tag refinement. In Proceedings of the ACM International Conference on Multimedia (MM). ACM, New York, NY, 1129--1132. DOI:http://dx.doi.org/10.1145/2072298.2071956.
33. Jitao Sang, Changsheng Xu, and Dongyuan Lu. 2012. Learn to personalized image search from the photo sharing websites. IEEE Trans. Multimedia 14, 4, 963--974.
34. Florian Schroff, Antonio Criminisi, and Andrew Zisserman 2011. Harvesting image databases from the Web. IEEE Trans. Patt. Anal. Mach. Intell. 33, 4, 754--766.
35. Ahu Sieg, Bamshad Mobasher, and Robin D. Burke. 2007. Web search personalization with ontological user profiles. In Proceedings of CIKM. 525--534.
36. Barry Smyth. 2007. A community-based approach to personalizing Web search. IEEE Computer 40, 8, 42--50.
37. Kazunari Sugiyama, Kenji Hatano, and Masatoshi Yoshikawa. 2004. Adaptive Web search based on user profile constructed without any effort from users. In Proceedings of WWW. 675--684.
38. Aixin Sun and Sourav S. Bhowmick. 2010. Quantifying tag representativeness of visual content of social images. In Proceedings of the ACM International Conference on Multimedia (MM). ACM, New York, NY, 471--480. DOI:http://dx.doi.org/10.1145/1873951.1874029.
39. Jian-Tao Sun, Hua-Jun Zeng, Huan Liu, Yuchang Lu, and Zheng Chen. 2005. CubeSVD: A novel approach to personalized web search. In Proceedings of WWW. 382--390.
40. Jaime Teevan, Susan T. Dumais, and Eric Horvitz 2005. Personalizing search via automated analysis of interests and activities. In Proceedings of SIGIR. 449--456.
41. Jaime Teevan, Susan T. Dumais, and Daniel J. Liebling. 2008. To personalize or not to personalize: Modeling queries with variation in user intent. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 163--170.
42. Jaime Teevan, Meredith Ringel Morris, and Steve Bush. 2009. Discovering and using groups to improve personalized search. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining. ACM, 15--24.
43. Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. 2006. Fast random walk with restart and its applications. In Proceedings of the International Conference on Data Mining (ICDM). 613--622.
44. Michele Trevisiol, Luca Chiarandini, Luca Maria Aiello, and Alejandro Jaimes 2012. Image ranking based on user browsing behavior. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 445--454.
45. Gang Wang and David Forsyth. 2008. Object image retrieval by exploiting online knowledge resources. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). IEEE, 1--8.
46. Jun Wang, Arjen P. de Vries, and Marcel J. T. Reinders. 2006. Unifying user-based and item-based collaborative filtering approaches by similarity fusion. In Proceedings of SIGIR. 501--508.
47. Zhiyu Wang, Peng Cui, Lexing Xie, Hao Chen, Wenwu Zhu, and Shiqiang Yang 2012. Analyzing social media via event facets. In Proceedings of the 20th ACM International Conference on Multimedia. ACM, 1359--1360.
48. Ingmar Weber and Alejandro Jaimes. 2011. Who uses web search for what: And how. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. ACM, 15--24.
49. Xing Xie, Hao Liu, Simon Goumaz, and Wei-Ying Ma. 2005. Learning user interest for image browsing on small-form-factor devices. In Proceedings of CHI. 671--680.
50. Hao Xu, Jingdong Wang, Xian-Sheng Hua, and Shipeng Li. 2009. Tag refinement by regularized LDA. In Proceedings of the 17th ACM International Conference on Multimedia. ACM, 573--576.
51. Hao Xu, Jingdong Wang, Zhu Li, Gang Zeng, Shipeng Li, and Nenghai Yu. 2011. Complementary hashing for approximate nearest neighbor search. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). IEEE, 1631--1638.
52. Shengliang Xu, Shenghua Bao, Ben Fei, Zhong Su, and Yong Yu 2008. Exploring folksonomy for personalized search. In Proceedings of SIGIR. 155--162.
53. Rong Yan, Alexander Hauptmann, and Rong Jin. 2003. Multimedia search with pseudo-relevance feedback. In Proceedings of the 2nd International Conference on Image and Video Retrieval. Lecture Notes in Computer Science, vol. 2728. Springer, 238--247.
54. Yun Yang, Peng Cui, Wenwu Zhu, and Shiqiang Yang. 2013. User interest and social influence based emotion prediction for individuals. In Proceedings of the 21st ACM International Conference on Multimedia. ACM, 785--788.
55. Hilmi Yildirim and Mukkai S. Krishnamoorthy. 2008. A random walk method for alleviating the sparsity problem in collaborative filtering. In Proceedings of the ACM Conference on Recommender Systems (RecSys). 131--138.
56. Shiliang Zhang, Qi Tian, Gang Hua, Qingming Huang, and Shipeng Li. 2009. Descriptive visual words and visual phrases for image applications. In Proceedings of the 17th ACM International Conference on Multimedia. ACM, 75--84.
57. Xiaofeng Zhu, Zi Huang, Hong Cheng, Jiangtao Cui, and Heng Tao Shen. 2013. Sparse hashing for fast multimedia search. ACM Trans. Inf. Syst. 31, 2, 9.
58. Hilal Zitouni, Sare Sevil, Derya Ozkan, and Pinar Duygulu. 2008. Re-ranking of Web image search results using a graph algorithm. In Proceedings of the 19th International Conference on Pattern Recognition (ICPR’08). IEEE, 1--4.

Index Terms

• Information systems

  • Information retrieval