skip to main content
10.1145/1101826.1101851acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
Article

Photo-to-search: using multimodal queries to search the web from mobile devices

Published: 10 November 2005 Publication History

Abstract

Nowadays, mobile phones with the digital camera are getting more and more popular. With necessary technologies, they are possible to become a powerful tool to search the Web on the go. Most Web search engines only support text queries. Therefore, users have to convert their information needs into words. However, it is sometimes difficult to describe the needs in text and the text input is inconvenient on small devices. To solve the problem, we propose a system named Photo-to-Search which allows users to input multimodal queries. Particularly, we study queries with captured images and optional text messages in this paper. For example, the user can simply take a photo of the flower and input a few terms like "flower". Textually relevant Web images are retrieved according to the query terms. Afterwards, the snapped picture is compared with these images by the CBIR (Content Based Image Retrieval) method. According to the context of the visually similar images, related key phrases are extracted. Finally, the search results are returned in multiple forms. Our system can also search for very similar images on the Web, such as movie posters or photos of film stars, to find related information. Experimental results on the large scale data showed our system achieved satisfactory efficiency and performance.

References

[1]
R. Bodner and F. Song, Knowledge-based approaches to query expansion in information retrieval, Advances in Artificial Intelligence, pp.146--158, Springer, 1996.
[2]
C. Carson, S. Belongie, H. Greenspan, and J. Malik, Blobworld: image segmentation using expectation-maximization and its application to image querying, IEEE Transactions on PAMI, vol.24, no.8, pp.1026--1038, 2002.
[3]
E. Chang, C. Li, J. Z. Wang, et al., Searching near-replicas of images via clustering, Proc. of SPIE Multimedia Storage and Archiving System VI, vol.3846, pp.281--292, Boston, USA, Sep. 1999.
[4]
Z. Chen, W. Liu, C. Hu, M. Li, and H.-J. Zhang, Ifind: A Web Image Search Engine, Proc. of the 24th ACM SIGIR conference on Research and development in information retrieval, pp. 450, New Orleans, USA, Sep. 2001.
[5]
K. W. Church and P. Hanks, Word association, norms, mutual information and lexicography, Computational Linguistics, vol.16, no.1, pp.22--29, 1990.
[6]
M. Flickner, H. Sawhney, W. Niblack, et al., Query by image and video content: the QBIC system, IEEE Computer Special Issue on Content-Based Retrieval, vol.28, no.9, pp.23--32, Sep. 1995.
[7]
Google Mobile Search, http://www.google.com/xhtml
[8]
Google SMS, http://www.google.com/sms/
[9]
J. S. Hare and P. H. Lewis, Content-based image retrieval using a mobile device as a novel interface, Proc. of SPIE Storage and Retrieval Methods and Applications for Multimedia 2005, vol.5682, pp.64--75, San Jose, USA, Jan. 2005.
[10]
A. Jaimes, S.-F Chang, and A.C. Loui, Detection of non-identical duplicate consumer photographs, Proc. of the Fourth Pacific Rim Conference on Multimedia, vol.1, pp.16--20, Singapore, Dec. 2003.
[11]
Y. Ke, R. Sukthankar, and L. Huston, Efficient near-duplicate and sub-image retrieval, Proc. of the 12th ACM International Conference on Multimedia, pp.869--876, New York, USA, Nov. 2004.
[12]
C. Kim, Content-based image copy detection, Signal Processing: Image Communication, vol.18, no.3, pp.169--184, Mar. 2003.
[13]
G. Miller, WordNet: A lexical database, Communication of the ACM, vol.38, no.11, pp.39--41, 1995.
[14]
M. Noda, H. Sonobe, S. Takagi, and F. Yoshimoto, Cosmos: convenient image retrieval system of flowers for mobile computing situations, Proc. of the IASTED Conference on Information Systems and Databases 2002, pp.25--30, Tokyo, Japan, Sep. 2002.
[15]
M. F. Porter, An algorithm for suffix stripping, Program, vol.14, no.3, pp.130--137, 1980.
[16]
N. Sebe, Q. Tian, E. Loupias, M. Lew, and T. Huang, Evaluation of salient point techniques, Image and Vision Computing, vol.21, pp.1087--1095, 2003.
[17]
J. R. Smith and S.-F. Chang, VisualSEEk: a fully automated content-based image query system, Proc. of the 4th ACM International Conference on Multimedia, pp.87--93, Boston, USA, Nov. 1996.
[18]
H. Sonobe, S. Takagi, and F. Yoshimoto, Image retrieval system of fishes using a mobile device, Proc. of International Workshop on Advanced Image Technology 2004, pp.33--37, Singapore, Jan. 2004.
[19]
E. M. Voorhees, Query expansion using lexical-semantic relations, Proc. of the 17th ACM SIGIR conference on Research and development in information retrieval, pp.61--69, Dublin, Ireland, Jul. 1994.
[20]
W3C Document Object Model, http://www.w3.org/DOM/
[21]
Yahoo! Mobile, http://mobile.yahoo.com
[22]
T. Yeh, K. Tollmar, and T. Darrell, Searching the Web with mobile images for location recognition, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, vol.2, pp.76--81, Washington D.C., USA, Jun. 2004.
[23]
T. Yeh, K. Tollmar, K. Grauman, and T. Darrell, A picture is worth a thousand keywords: image-based object search on a mobile platform, Proc. of the 2005 Conference on Human Factors in Computing Systems, pp.2025--2028, Portland, USA, Apr. 2005.

Cited By

View all
  • (2018)Here and NowProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176384(171-180)Online publication date: 1-Mar-2018
  • (2018)Demonstrating Reality-Based Information RetrievalExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3186493(1-4)Online publication date: 20-Apr-2018
  • (2018)Progressive Image Retrieval With Quality Guarantee Under MapReduce FrameworkIEEE Access10.1109/ACCESS.2018.28427966(44685-44697)Online publication date: 2018
  • Show More Cited By

Index Terms

  1. Photo-to-search: using multimodal queries to search the web from mobile devices

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MIR '05: Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval
    November 2005
    274 pages
    ISBN:1595932445
    DOI:10.1145/1101826
    • General Chairs:
    • Hongjiang Zhang,
    • John Smith,
    • Qi Tian
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 November 2005

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. content based image retrieval
    2. duplicate image detection
    3. mobile search
    4. multimodal interactions
    5. web image search

    Qualifiers

    • Article

    Conference

    MM&Sec '05
    MM&Sec '05: Multimedia and Security Workshop 2005
    November 10 - 11, 2005
    Hilton, Singapore

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)2
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2018)Here and NowProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176384(171-180)Online publication date: 1-Mar-2018
    • (2018)Demonstrating Reality-Based Information RetrievalExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3186493(1-4)Online publication date: 20-Apr-2018
    • (2018)Progressive Image Retrieval With Quality Guarantee Under MapReduce FrameworkIEEE Access10.1109/ACCESS.2018.28427966(44685-44697)Online publication date: 2018
    • (2017)Search by Screenshots for Universal Article Clipping in Mobile AppsACM Transactions on Information Systems10.1145/309110735:4(1-29)Online publication date: 23-Jun-2017
    • (2016)A methodology for machine translation of simple sentences from Kannada to English language2016 2nd International Conference on Contemporary Computing and Informatics (IC3I)10.1109/IC3I.2016.7917967(237-241)Online publication date: Dec-2016
    • (2016)UniClip: Leveraging Web Search for Universal Clipping of Articles on MobileData Science and Engineering10.1007/s41019-016-0012-21:2(101-113)Online publication date: 18-Jul-2016
    • (2013)Image search—from thousands to billions in 20 yearsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/24908239:1s(1-20)Online publication date: 17-Oct-2013
    • (2013)Recognition of Limited Vocabulary Kannada Words Through Structural Pattern Matching: An Experimentation on Low Resolution ImagesMultimedia Processing, Communication and Computing Applications10.1007/978-81-322-1143-3_15(181-194)Online publication date: 26-May-2013
    • (2013)Word Level Script Identification of Text in Low Resolution Images of Display Boards Using Wavelet FeaturesProceedings of International Conference on Advances in Computing10.1007/978-81-322-0740-5_26(209-220)Online publication date: 2013
    • (2012)Annotating Images by Mining Image SearchMachine Learning10.4018/978-1-60960-818-7.ch417(1066-1089)Online publication date: 2012
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media