skip to main content
10.1145/1458082.1458314acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
poster

Estimating retrieval effectiveness using rank distributions

Published: 26 October 2008 Publication History

Abstract

In this paper, we consider the task of estimating query effectiveness, i.e., assessment of the retrieval system performance in absence of the user relevance judgments. In our approach we model the score associated with each document in the result set as a Gaussian random variable. The mean and the variance of each document score can then be used to estimate the probability that a document will be ranked above another one and thus calculate the expected rank of the document in the ranked list. We propose to measure the effectiveness of the system performance by comparing the predicted and actual ranks of the retrieved documents. In our experiments we consider two retrieval models and five document scoring methods and evaluate their impact on the proposed estimation measures. Our experiments with standardized data sets that include document relevance judgments and the task of predicting the relative query effectiveness show that the expected rank metric is robust to variations in document scoring and retrieval algorithms.

References

[1]
Cronen-Townsend, S., Zhou, Y. and Croft, W. B. Predicting query performance. Proceedings of SIGIR 2002, 299--306
[2]
Diaz, F. Performance prediction using spatial autocorrelation. Proceedings of SIGIR 2007, 583--590
[3]
Montague, M. and Aslam, J. A. Relevance score normalization for metasearch. Proceedings of CIKM 2001, 427--433
[4]
Taylor, M., Guiver, J., Robertson, S. and Minka, T. SoftRank: Optimising Non-smooth Rank Metrics. Proceedings of WSDM 2008
[5]
Yom-Tov, E., Fine, S., Carmel, D. and Darlow, A. 2005. Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. SIGIR 2005, 512--519
[6]
Zhu, X., Lafferty, J. and Ghahramani, Z. Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. Proc. of the ICML 2003 Workshop on The Continuum from Labeled to Unlabeled Data in ML and Data Mining
[7]
The Lemur Toolkit for Language Modeling and Information Retrieval, http://www.lemurproject.org

Cited By

View all
  • (2014)Document Score Distribution Models for Query Performance Inference and PredictionACM Transactions on Information Systems10.1145/255917032:1(1-28)Online publication date: 1-Jan-2014
  • (2010)Standard deviation as a query hardness estimatorProceedings of the 17th international conference on String processing and information retrieval10.5555/1928328.1928355(207-212)Online publication date: 11-Oct-2010
  • (2009)Ranking List Dispersion as a Query Performance PredictorProceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory10.1007/978-3-642-04417-5_42(371-374)Online publication date: 3-Sep-2009
  • Show More Cited By

Index Terms

  1. Estimating retrieval effectiveness using rank distributions

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge management
      October 2008
      1562 pages
      ISBN:9781595939913
      DOI:10.1145/1458082
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 26 October 2008

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tag

      1. effectiveness estimation

      Qualifiers

      • Poster

      Conference

      CIKM08
      CIKM08: Conference on Information and Knowledge Management
      October 26 - 30, 2008
      California, Napa Valley, USA

      Acceptance Rates

      Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

      Upcoming Conference

      CIKM '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)1
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 09 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2014)Document Score Distribution Models for Query Performance Inference and PredictionACM Transactions on Information Systems10.1145/255917032:1(1-28)Online publication date: 1-Jan-2014
      • (2010)Standard deviation as a query hardness estimatorProceedings of the 17th international conference on String processing and information retrieval10.5555/1928328.1928355(207-212)Online publication date: 11-Oct-2010
      • (2009)Ranking List Dispersion as a Query Performance PredictorProceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory10.1007/978-3-642-04417-5_42(371-374)Online publication date: 3-Sep-2009
      • (2009)Risk-Aware Information RetrievalProceedings of the 31th European Conference on IR Research on Advances in Information Retrieval10.1007/978-3-642-00958-7_5(17-28)Online publication date: 18-Apr-2009

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media