skip to main content
10.1145/1835449.1835489acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

On statistical analysis and optimization of information retrieval effectiveness metrics

Published: 19 July 2010 Publication History

Abstract

This paper presents a new way of thinking for IR metric optimization. It is argued that the optimal ranking problem should be factorized into two distinct yet interrelated stages: the relevance prediction stage and ranking decision stage. During retrieval the relevance of documents is not known a priori, and the joint probability of relevance is used to measure the uncertainty of documents' relevance in the collection as a whole. The resulting optimization objective function in the latter stage is, thus, the expected value of the IR metric with respect to this probability measure of relevance. Through statistically analyzing the expected values of IR metrics under such uncertainty, we discover and explain some interesting properties of IR metrics that have not been known before. Our analysis and optimization framework do not assume a particular (relevance) retrieval model and metric, making it applicable to many existing IR models and metrics. The experiments on one of resulting applications have demonstrated its significance in adapting to various IR metrics.

References

[1]
G. Amati and C. J. V. Rijsbergen. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst., 20(4):357--389, 2002.
[2]
K. Arrow. Aspects of the Theory of Risk-Bearing. Helsinki: Yrjo Hahnsson Foundation, 1965.
[3]
J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In SIGIR, 2006.
[4]
J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In SIGIR, 2005.
[5]
C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[6]
P. F. Brown, V. J. D. Pietra, S. A. D. Pietra, and R. L. Mercer. The mathematics of statistical machine translation: parameter estimation. Comput. Linguist., 1993.
[7]
C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In SIGIR, 2000.
[8]
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML '05, 2005.
[9]
J. Carbonell and J. Goldstein. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In SIGIR, 1998.
[10]
H. Chen and D. R. Karger. Less is more: probabilistic models for retrieving fewer relevant documents. In SIGIR, 2006.
[11]
G. V. Cormack and T. R. Lynam. Statistical precision of information retrieval evaluation. In SIGIR, 2006.
[12]
W. B. Croft and D. J. Harper. Using probabilistic models of document retrieval without relevance information. Document Retrieval Systems, 1988.
[13]
D. Harman. Overview of the second text retrieval conference (trec-2). In HLT '94, 1994.
[14]
K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 2002.
[15]
J. Lafferty and C. Zhai. Document language models, query models, and risk minimization for information retrieval. In SIGIR, 2001.
[16]
C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
[17]
M. E. Maron and J. L. Kuhns. On relevance, probabilistic indexing and information retrieval. J. ACM, 1960.
[18]
S. Mizzaro. Relevance: The whole history. Journal of the American Society of Information Science, 1997.
[19]
S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, pages 294--304, 1977.
[20]
S. E. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3):129--46, 1976.
[21]
S. E. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR, 1994.
[22]
A. Singhal, C. Buckley, and M. Mitra. Pivoted document length normalization. In SIGIR, pages 21--29, 1996.
[23]
M. Taylor, J. Guiver, S. Robertson, and T. Minka. Softrank: optimizing non-smooth rank metrics. In WSDM, 2008.
[24]
S. Tomlinson. Early precision measures: implications from the downside of blind feedback. In SIGIR, 2006.
[25]
C. J. van Rijsbergen. Information Retrieval. Butterworths, London, London, UK, 1979.
[26]
M. N. Volkovs and R. S. Zemel. Boltzrank: learning to maximize expected ranking gain. In ICML '09, 2009.
[27]
E. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In Information Processing and Management, pages 315--323. ACM Press, 1998.
[28]
E. M. Voorhees. The TREC-8 question answering track report. In TREC-8, pages 77--82, 1999.
[29]
J. Wang and J. Zhu. Portfolio theory of information retrieval. In SIGIR, 2009.
[30]
Y. Wang and A. Waibel. Decoding algorithm in statistical machine translation. In EACL, 1997.
[31]
E. Yilmaz, E. Kanoulas, and J. A. Aslam. A simple and efficient sampling method for estimating ap and ndcg. In SIGIR, 2008.
[32]
E. Yilmaz and S. Robertson. On the choice of effectiveness measures for learning to rank. Information Retrieval, 2009.
[33]
Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In SIGIR, 2007.
[34]
C. Zhai. Statistical language models for information retrieval a critical review. Found. Trends Inf. Retr., 2(3):137--213, 2008.
[35]
C. Zhai and J. D. Lafferty. A risk minimization framework for information retrieval. Inf. Process. Manage., 42(1):31--55, 2006.

Cited By

View all
  • (2021)Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to RankACM Transactions on Information Systems10.1145/346430340:2(1-29)Online publication date: 16-Nov-2021
  • (2020)Modeling and optimization of traffic flows in a network2020 International Conference Automatics and Informatics (ICAI)10.1109/ICAI50593.2020.9311314(1-6)Online publication date: 1-Oct-2020
  • (2018)Recommending Based on Implicit FeedbackSocial Information Access10.1007/978-3-319-90092-6_14(510-569)Online publication date: 3-May-2018
  • Show More Cited By

Index Terms

  1. On statistical analysis and optimization of information retrieval effectiveness metrics

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
      July 2010
      944 pages
      ISBN:9781450301534
      DOI:10.1145/1835449
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 July 2010

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. ir metrics
      2. learing to rank
      3. optimal ranking
      4. optimization
      5. ranking under uncertainty
      6. retrieval models

      Qualifiers

      • Research-article

      Conference

      SIGIR '10
      Sponsor:

      Acceptance Rates

      SIGIR '10 Paper Acceptance Rate 87 of 520 submissions, 17%;
      Overall Acceptance Rate 792 of 3,983 submissions, 20%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)8
      • Downloads (Last 6 weeks)1
      Reflects downloads up to 15 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2021)Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to RankACM Transactions on Information Systems10.1145/346430340:2(1-29)Online publication date: 16-Nov-2021
      • (2020)Modeling and optimization of traffic flows in a network2020 International Conference Automatics and Informatics (ICAI)10.1109/ICAI50593.2020.9311314(1-6)Online publication date: 1-Oct-2020
      • (2018)Recommending Based on Implicit FeedbackSocial Information Access10.1007/978-3-319-90092-6_14(510-569)Online publication date: 3-May-2018
      • (2017)An in-depth study on diversity evaluationInformation Processing and Management: an International Journal10.1016/j.ipm.2017.03.00153:4(799-813)Online publication date: 1-Jul-2017
      • (2016)Diverse Yet Efficient Retrieval using Locality Sensitive HashingProceedings of the 2016 ACM on International Conference on Multimedia Retrieval10.1145/2911996.2911998(189-196)Online publication date: 6-Jun-2016
      • (2015)Dynamic Information RetrievalProceedings of the 2015 International Conference on The Theory of Information Retrieval10.1145/2808194.2809457(61-70)Online publication date: 27-Sep-2015
      • (2014)Optimization of information retrieval for cross media contents in a best practice networkInternational Journal of Multimedia Information Retrieval10.1007/s13735-014-0058-83:3(147-159)Online publication date: 8-May-2014
      • (2013)To personalize or notProceedings of the 7th ACM conference on Recommender systems10.1145/2507157.2507167(229-236)Online publication date: 12-Oct-2013
      • (2013)GAPfmProceedings of the 22nd ACM international conference on Information & Knowledge Management10.1145/2505515.2505653(2261-2266)Online publication date: 27-Oct-2013
      • (2013)Interactive exploratory search for multi page search resultsProceedings of the 22nd international conference on World Wide Web10.1145/2488388.2488446(655-666)Online publication date: 13-May-2013
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media