skip to main content
10.1145/1459359.1459387acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

ContextSeer: context search and recommendation at query time for shared consumer photos

Published: 26 October 2008 Publication History

Abstract

The advent of media-sharing sites like Flickr has drastically increased the volume of community-contributed multimedia resources on the web. However, due to their magnitudes, these collections are increasingly difficult to understand, search and navigate. To tackle these issues, a novel search system, ContextSeer, is developed to improve search quality (by reranking) and recommend supplementary information (i.e., search-related tags and canonical images) by leveraging the rich context cues, including the visual content, high-level concept scores, time and location metadata. First, we propose an ordinal reranking algorithm to enhance the semantic coherence of text-based search result by mining contextual patterns in an unsupervised fashion. A novel feature selection method, wc-tf-idf is also developed to select informative context cues. Second, to represent the diversity of search result, we propose an efficient algorithm cannoG to select multiple canonical images without clustering. Finally, ContextSeer enhances the search experience by further recommending relevant tags. Besides being effective and unsupervised, the proposed methods are efficient and can be finished at query time, which is vital for practical online applications. To evaluate ContextSeer, we have collected 0.5 million consumer photos from Flickr and manually annotated a number of queries by pooling to form a new benchmark, Flickr550. Ordinal reranking achieves significant performance gains both in Flcikr550 and TRECVID search benchmarks. Through a subjective test, cannoG expresses its representativeness and excellence for recommending multiple canonical images.

References

[1]
L. Kennedy et al, "How Flickr helps us make sense of the world: Context and content in community-contributed media collections," ACM Multimedia, pp. 631--640, 2007.
[2]
M. Ames et al, "Why we tag: Motivations for annotation in mobile and online media," ACM CHI, pp. 971--980, 2007.
[3]
L. Kennedy, S.-F. Chang, and I. Kozintsev, "To search or to label?: Predicting the performance of search-based automatic image classifiers," Proc. ACM Int. workshop on Multimedia information retrieval, pp. 249--258, 2006.
[4]
A. K. Dey, "Understanding and using context," Personal and Ubiquitous Computing, vol. 5, no. 1, 2001.
[5]
M. Naphade et al, "Large-scale concept ontology for multimedia," IEEE Multimedia Magazine, vol. 13, no. 3, pp. 86--91, 2006.
[6]
K. Toyama et al, "Geographic location tags on digital images," ACM Multimedia, pp. 156--166, 2003.
[7]
W. Hsu et al, "Video search reranking via information bottleneck principle," ACM Multimedia, pp. 35--44, 2006.
[8]
W. Hsu, L. Kennedy, and S.-F. Chang, "Video search reranking through random walk over document-level context graph," Proc. ACM Multimedia, pp. 971--980, 2007.
[9]
L. Kennedy and S.-F. Chang, "A reranking approach for context-based concept fusion in video indexing and retrieval," ACM CIVR, pp. 333--340, 2007.
[10]
Y.-H. Yang and W.-H. Hsu, "Video search reranking via online ordinal reranking," IEEE ICME, 2008.
[11]
A. Natsev et al, "Semantic concept-based query expansion and re-ranking for multimedia retrieval," ACM Multimedia, pp. 991--1000, 2007.
[12]
J. Battelle, The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture, 2006.
[13]
NIST TREC Video Retrieval Evaluation. {online} http://www-nlpir.nist.gov/projects/trecvid/.
[14]
X. Li et al, "Video search in concept subspace: a text like paradigm," ACM CIVR, pp. 603--610, 2007.
[15]
A. Aizawa, "An information-theoretic perspective of tf-idf measures," Information Processing and Management, vol. 39, pp. 45--65, 2003.
[16]
S. Palmer et al, "Canonical perspective and the perception of objects," Attention and Performance IX, pp. 135--151, 1981.
[17]
L. Kennedy et al, "Generating diverse and representative image search results for landmarks," WWW, 2008.
[18]
Y. Jing et al, "Canonical image selection from the web," ACM CIVR, pp. 280--287, 2007.
[19]
R. Yan, A. Hauptmann, and R. Jin, "Multimedia search with pseudo-relevance feedback," ACM CIVR, 2003.
[20]
R. Herbrich et al, "Support vector learning for ordinal regression," IEEE ICANN, pp. 97--102, 1999.
[21]
Z. Cao et al, "Learning to rank: from pairwise approach to listwise approach," IEEE ICML, pp. 129--136, 2007.
[22]
S.-F. Chang et al, "Columbia University TRECVID-2005 video search and high-level feature extraction," NIST TRECVID workshop, 2005.
[23]
I. Simon et al, "Scene summarization for online image collections," IEEE ICCV, pp. 1--8, 2007.
[24]
S. Wang et al, "IGroup: presenting web image search results in semantic clusters," ACM CHI, 2007, pp. 377--384.
[25]
N. J. Belkin, "Helping people find what they don't know," Communication of the ACM, vol. 43, no. 8, pp. 58--61, 2000.
[26]
C.-K. Huang et al, "Relevant term suggestion in interactive web search based on contextual information in query session logs," Journal of the American Society for Information Science and Technology, vol. 54, no. 7, pp. 638--649, 2003.
[27]
R. Jones, B. Rey, O. Madani, and W. Greiner, "Generating query substitutions," ACM WWW, 2006.
[28]
J. Xu and W. Croft, "Query expansion using local and global document analysis," ACM SIGIR, pp. 4--11, 1996.
[29]
J. Sivic and A. Zisserman, "Video Google: A text retrieval approach to object matching in videos, ICCV, 2003.
[30]
K. Mikolajczyk and C. Schmid, "Scale & affine invariant interest point detectors," IJCV, vol.60, no.1, pp. 63--86, 2004.
[31]
D. Lowe, "Distinctive image features from scale-invariant keypoints," IJCV, vol. 60, no. 2, pp. 91--110, 2004.
[32]
S. Sontag, On Photography, Picador USA, 2001.
[33]
L. Page et al., "The PageRank citation ranking: Bringing order to the web," Stanford University, 1998.

Cited By

View all

Index Terms

  1. ContextSeer: context search and recommendation at query time for shared consumer photos

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '08: Proceedings of the 16th ACM international conference on Multimedia
    October 2008
    1206 pages
    ISBN:9781605583037
    DOI:10.1145/1459359
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 October 2008

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. canonical image
    2. context
    3. metadata
    4. recommending
    5. rerank
    6. search
    7. shared consumer photo
    8. tag
    9. visual word

    Qualifiers

    • Research-article

    Conference

    MM08
    Sponsor:
    MM08: ACM Multimedia Conference 2008
    October 26 - 31, 2008
    British Columbia, Vancouver, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)2
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 07 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2018)Collective Intelligence: OverviewEncyclopedia of Social Network Analysis and Mining10.1007/978-1-4939-7131-2_106(286-301)Online publication date: 12-Jun-2018
    • (2017)Semi-active learning to rank algorithms for document retrievalInternational Journal of Intelligent Information and Database Systems10.5555/3160710.316071610:3-4(289-313)Online publication date: 1-Jan-2017
    • (2017)A knowledge-based semantic approach for image collection summarizationMultimedia Tools and Applications10.1007/s11042-016-3840-176:9(11917-11939)Online publication date: 1-May-2017
    • (2017)Learning Visual Word Patterns Using BoVW Model for Image RetrievalComputational Intelligence in Data Mining10.1007/978-981-10-3874-7_46(489-498)Online publication date: 20-May-2017
    • (2017)Collective Intelligence, OverviewEncyclopedia of Social Network Analysis and Mining10.1007/978-1-4614-7163-9_106-1(1-15)Online publication date: 26-Jul-2017
    • (2015)I2RSProceedings of the VLDB Endowment10.14778/2824032.28240928:12(1884-1887)Online publication date: 1-Aug-2015
    • (2015)Enhancing sparse voice annotation for semantic retrieval of personal photos by continuous space word representations2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2015.7178991(5341-5345)Online publication date: Apr-2015
    • (2015)ImageCLEF annotation with explicit context-aware kernel mapsInternational Journal of Multimedia Information Retrieval10.1007/s13735-015-0082-34:2(113-128)Online publication date: 20-Mar-2015
    • (2014)Image Relevance Prediction Using Query-Context Bag-of-Object Retrieval ModelIEEE Transactions on Multimedia10.1109/TMM.2014.232683616:6(1700-1712)Online publication date: Oct-2014
    • (2014)Network-Dependent Image Annotation Based on Explicit Context-Dependent Kernel MapsProceedings of the 2014 22nd International Conference on Pattern Recognition10.1109/ICPR.2014.118(625-630)Online publication date: 24-Aug-2014
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media