poster

Comparing click-through data to purchase decisions for retrieval evaluation

Authors:

Katja Hofmann,

Bouke Huurnink,

Marc Bron,

Maarten de RijkeAuthors Info & Claims

SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

Pages 761 - 762

https://doi.org/10.1145/1835449.1835603

Published: 19 July 2010 Publication History

Get Access

Abstract

Traditional retrieval evaluation uses explicit relevance judgments which are expensive to collect. Relevance assessments inferred from implicit feedback such as click-through data can be collected inexpensively, but may be less reliable. We compare assessments derived from click-through data to another source of implicit feedback that we assume to be highly indicative of relevance: purchase decisions. Evaluating retrieval runs based on a log of an audio-visual archive, we find agreement between system rankings and purchase decisions to be surprisingly high.

References

[1]

S. P. Harter. Variations in relevance assessments and the measurement of retrieval effectiveness. JASIS&T, 47(1):37--49, 1996.

Digital Library

Google Scholar

[2]

B. Huurnink, L. Hollink, W. van den Heuvel, and M. de Rijke. The search behavior of media professionals at an audiovisual archive: A transaction log analysis. JASIS&T, 2010.

Crossref

Google Scholar

[3]

T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In SIGIR '05, pages 154--161, 2005. ACM.

Digital Library

Google Scholar

[4]

J. Kamps, M. Koolen, and A. Trotman. Comparative analysis of clicks and judgments for IR evaluation. In WSCD '09, pages 80--87, 2009. ACM.

Digital Library

Google Scholar

[5]

F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM '08, pages 43--52, 2008. ACM.

Digital Library

Google Scholar

[6]

I. Ruthven. Integrating approaches to relevance. New directions in cognitive information retrieval, pages 61--80, 2005. Springer.

Google Scholar

Cited By

View all

Hofmann KLi LRadlinski F(2016)Online Evaluation for Information RetrievalFoundations and Trends in Information Retrieval10.1561/150000005110:1(1-117)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1561/1500000051
Hofmann KWhiteson SRijke M(2013)Fidelity, Soundness, and Efficiency of Interleaved Comparison MethodsACM Transactions on Information Systems10.1145/2536736.253673731:4(1-43)Online publication date: 1-Nov-2013
https://dl.acm.org/doi/10.1145/2536736.2536737
Berendsen RTsagkias MWeerkamp Wde Rijke MJones GSheridan PKelly Dde Rijke MSakai T(2013)Pseudo test collections for training and tuning microblog rankersProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484063(53-62)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1145/2484028.2484063
Show More Cited By

Index Terms

Comparing click-through data to purchase decisions for retrieval evaluation
1. Information systems
  1. Information retrieval

Recommendations

Exploiting click-through data for entity retrieval
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

We present an approach for answering Entity Retrieval queries using click-through information in query log data from a commercial Web search engine. We compare results using click graphs and session graphs and present an evaluation test set making use ...
Current Status of the Evaluation of Information Retrieval

This is the second in the series of the articles on an application of the systems analytic approach to evaluation of information retrieval (IR). In the previous article a historical overview of IR was presented and existing terminological problems ...
Evaluation of phrasal query suggestions
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management

This paper evaluates the uptake and efficacy of a unified approach to phrasal query suggestions in the context of a high-precision search engine. The search engine performs ranked extended-Boolean searches with the proximity operator <scp>NEAR</scp> ...

Comments

Information & Contributors

Information

Published In

SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

July 2010

944 pages

ISBN:9781450301534

DOI:10.1145/1835449

General Chairs:
Fabio Crestani
University of Lugano, CH
,
Stéphane Marchand-Maillet
University of Geneva, CH
,
Program Chairs:
Hsin-Hsi Chen
National Taiwan University, TW
,
Efthimis N. Efthimiadis
University of Washington, USA
,
Jacques Savoy
University of Neuchatel, CH

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2010

Check for updates

Author Tags

Qualifiers

Poster

Conference

SIGIR '10

Sponsor:

SIGIR

SIGIR '10: The 33rd International ACM SIGIR conference on research and development in Information Retrieval

July 19 - 23, 2010

Geneva, Switzerland

Acceptance Rates

SIGIR '10 Paper Acceptance Rate 87 of 520 submissions, 17%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
311
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Hofmann KLi LRadlinski F(2016)Online Evaluation for Information RetrievalFoundations and Trends in Information Retrieval10.1561/150000005110:1(1-117)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1561/1500000051
Hofmann KWhiteson SRijke M(2013)Fidelity, Soundness, and Efficiency of Interleaved Comparison MethodsACM Transactions on Information Systems10.1145/2536736.253673731:4(1-43)Online publication date: 1-Nov-2013
https://dl.acm.org/doi/10.1145/2536736.2536737
Berendsen RTsagkias MWeerkamp Wde Rijke MJones GSheridan PKelly Dde Rijke MSakai T(2013)Pseudo test collections for training and tuning microblog rankersProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484063(53-62)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1145/2484028.2484063
Hofmann KWhiteson Sde Rijke MChen XLebanon GWang HZaki M(2012)Estimating interleaved comparison outcomes from historical click dataProceedings of the 21st ACM international conference on Information and knowledge management10.1145/2396761.2398516(1779-1783)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2396761.2398516
Hofmann KBehr FRadlinski FChen XLebanon GWang HZaki M(2012)On caption bias in interleaving experimentsProceedings of the 21st ACM international conference on Information and knowledge management10.1145/2396761.2396780(115-124)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2396761.2396780
Huurnink BSnoek Cde Rijke MSmeulders A(2012)Content-Based Analysis Improves Audiovisual Archive RetrievalIEEE Transactions on Multimedia10.1109/TMM.2012.219356114:4(1166-1178)Online publication date: 1-Aug-2012
https://dl.acm.org/doi/10.1109/TMM.2012.2193561
Hofmann KWhiteson Sde Rijke M(2011)A probabilistic method for inferring preferences from clicksProceedings of the 20th ACM international conference on Information and knowledge management10.1145/2063576.2063618(249-258)Online publication date: 24-Oct-2011
https://dl.acm.org/doi/10.1145/2063576.2063618
Huurnink BHofmann KDe Rijke MBron M(2010)Validating query simulatorsProceedings of the 2010 international conference on Multilingual and multimodal information access evaluation: cross-language evaluation forum10.5555/1889174.1889183(40-51)Online publication date: 20-Sep-2010
https://dl.acm.org/doi/10.5555/1889174.1889183
Huurnink BSnoek Cde Rijke MSmeulders ALi SGao XSebe N(2010)Today's and tomorrow's retrieval practice in the audiovisual archiveProceedings of the ACM International Conference on Image and Video Retrieval10.1145/1816041.1816045(18-25)Online publication date: 5-Jul-2010
https://dl.acm.org/doi/10.1145/1816041.1816045

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Exploiting click-through data for entity retrieval

Current Status of the Evaluation of Information Retrieval

Evaluation of phrasal query suggestions

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations