Photo-to-search: using multimodal queries to search the web from mobile devices

ABSTRACT

Nowadays, mobile phones with the digital camera are getting more and more popular. With necessary technologies, they are possible to become a powerful tool to search the Web on the go. Most Web search engines only support text queries. Therefore, users have to convert their information needs into words. However, it is sometimes difficult to describe the needs in text and the text input is inconvenient on small devices. To solve the problem, we propose a system named Photo-to-Search which allows users to input multimodal queries. Particularly, we study queries with captured images and optional text messages in this paper. For example, the user can simply take a photo of the flower and input a few terms like "flower". Textually relevant Web images are retrieved according to the query terms. Afterwards, the snapped picture is compared with these images by the CBIR (Content Based Image Retrieval) method. According to the context of the visually similar images, related key phrases are extracted. Finally, the search results are returned in multiple forms. Our system can also search for very similar images on the Web, such as movie posters or photos of film stars, to find related information. Experimental results on the large scale data showed our system achieved satisfactory efficiency and performance.

REFERENCES

• 1: Richard C. Bodner , Fei Song, Knowledge-Based Approaches to Query Expansion in Information Retrieval, Proceedings of the 11th Biennial Conference of the Canadian Society for Computational Studies of Intelligence on Advances in Artificial Intelligence, p.146-158, May 21-24, 1996
• 2: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800 ]
• 3: E. Chang, C. Li, J. Z. Wang, et al., Searching near-replicas of images via clustering, Proc. of SPIE Multimedia Storage and Archiving System VI, vol.3846, pp.281--292, Boston, USA, Sep. 1999.
• 4: Zheng Chen , Liu Wenyin , Chunhui Hu , Mingjing Li , Hong-Jiang Zhang, iFind: a web image search engine, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.450, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384091]
• 5: Kenneth Ward Church , Patrick Hanks, Word association norms, mutual information, and lexicography, Computational Linguistics, v.16 n.1, p.22-29, March 1990
• 6: Myron Flickner , Harpreet Sawhney , Wayne Niblack , Jonathan Ashley , Qian Huang , Byron Dom , Monika Gorkani , Jim Hafner , Denis Lee , Dragutin Petkovic , David Steele , Peter Yanker, Query by Image and Video Content: The QBIC System, Computer, v.28 n.9, p.23-32, September 1995 [doi>10.1109/2.410146 ]
• 7: Google Mobile Search, http://www.google.com/xhtml
• 8: Google SMS, http://www.google.com/sms/
• 9: J. S. Hare and P. H. Lewis, Content-based image retrieval using a mobile device as a novel interface, Proc. of SPIE Storage and Retrieval Methods and Applications for Multimedia 2005, vol.5682, pp.64--75, San Jose, USA, Jan. 2005.
• 10: A. Jaimes, S.-F Chang, and A.C. Loui, Detection of non-identical duplicate consumer photographs, Proc. of the Fourth Pacific Rim Conference on Multimedia, vol.1, pp.16--20, Singapore, Dec. 2003.
• 11: Yan Ke , Rahul Sukthankar , Larry Huston, An efficient parts-based near-duplicate and sub-image retrieval system, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027729]
• 12: C. Kim, Content-based image copy detection, Signal Processing: Image Communication, vol.18, no.3, pp.169--184, Mar. 2003.
• 13: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 14: M. Noda, H. Sonobe, S. Takagi, and F. Yoshimoto, Cosmos: convenient image retrieval system of flowers for mobile computing situations, Proc. of the IASTED Conference on Information Systems and Databases 2002, pp.25--30, Tokyo, Japan, Sep. 2002.
• 15: M. F. Porter, An algorithm for suffix stripping, Program, vol.14, no.3, pp.130--137, 1980.
• 16: N. Sebe, Q. Tian, E. Loupias, M. Lew, and T. Huang, Evaluation of salient point techniques, Image and Vision Computing, vol.21, pp.1087--1095, 2003.
• 17: John R. Smith , Shih-Fu Chang, VisualSEEk: a fully automated content-based image query system, Proceedings of the fourth ACM international conference on Multimedia, p.87-98, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244151]
• 18: H. Sonobe, S. Takagi, and F. Yoshimoto, Image retrieval system of fishes using a mobile device, Proc. of International Workshop on Advanced Image Technology 2004, pp.33--37, Singapore, Jan. 2004.
• 19: Ellen M. Voorhees, Query expansion using lexical-semantic relations, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.61-69, July 03-06, 1994, Dublin, Ireland
• 20: W3C Document Object Model, http://www.w3.org/DOM/
• 21: Yahoo! Mobile, http://mobile.yahoo.com
• 22: T. Yeh, K. Tollmar, and T. Darrell, Searching the Web with mobile images for location recognition, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, vol.2, pp.76--81, Washington D.C., USA, Jun. 2004.
• 23: Tom Yeh , Kristen Grauman , Konrad Tollmar , Trevor Darrell, A picture is worth a thousand keywords: image-based object search on a mobile platform, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057083]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Algorithms, Human Factors, Performance

Keywords:
content based image retrieval, duplicate image detection, mobile search, multimodal interactions, web image search