skip to main content
10.1145/1390156.1390306acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlConference Proceedingsconference-collections
research-article

Listwise approach to learning to rank: theory and algorithm

Published: 05 July 2008 Publication History

Abstract

This paper aims to conduct a study on the listwise approach to learning to rank. The listwise approach learns a ranking function by taking individual lists as instances and minimizing a loss function defined on the predicted list and the ground-truth list. Existing work on the approach mainly focused on the development of new algorithms; methods such as RankCosine and ListNet have been proposed and good performances by them have been observed. Unfortunately, the underlying theory was not sufficiently studied so far. To amend the problem, this paper proposes conducting theoretical analysis of learning to rank algorithms through investigations on the properties of the loss functions, including consistency, soundness, continuity, differentiability, convexity, and efficiency. A sufficient condition on consistency for ranking is given, which seems to be the first such result obtained in related research. The paper then conducts analysis on three loss functions: likelihood loss, cosine loss, and cross entropy loss. The latter two were used in RankCosine and ListNet. The use of the likelihood loss leads to the development of a new listwise method called ListMLE, whose loss function offers better properties, and also leads to better experimental results.

References

[1]
Baeza-Yates, R., & Ribeiro-Neto, B. (Eds.). (1999). Modern information retrieval. Addison Wesley.
[2]
Bartlett, P. L., Jordan, M. I., & McAuliffe, J. D. (2003). Convexity, classification, and risk bounds (Technical Report 638). Statistics Department, University of California, Berkeley.
[3]
Boyd, S., & Vandenberghe, L. (Eds.). (2004). Convex optimization. Cambridge University.
[4]
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005). Learning to rank using gradient descent. Proceedings of ICML 2005 (pp. 89--96).
[5]
Cao, Z., Qin, T., Liu, T. Y., Tsai, M. F., & Li, H. (2007). Learning to rank: From pairwise approach to listwise approach. Proceedings of the 24th International Conference on Machine Learning (pp. 129--136). Corvallis, OR.
[6]
Cossock, D., & Zhang, T. (2006). Subset ranking using regression. COLT (pp. 605--619).
[7]
Freund, Y., Iyer, R., Schapire, R. E., & Singer, Y. (1998). An efficient boosting algorithm for combining preferences. Proceedings of ICML (pp. 170--178).
[8]
Hastie, T., Tibshirani, R., & Friedman, J. H. (Eds.). (2001). The elements of statistical learning: Data mining, inference and prediction. Springer.
[9]
Herbrich, R., Graepel, T., & Obermayer, K. (1999). Support vector vector learning for ordinal regression. Proceedings of ICANN (pp. 97--102).
[10]
Jarvelin, K., & Kekanainen, J. (2000). Ir evaluation methods for retrieving highly relevant documents. Proceedings of SIGIR (pp. 41--48).
[11]
Lin, Y. (2002). Support vector machines and the bayes rule in classification. Data Mining and Knowledge Discovery, 259--275.
[12]
Liu, T. Y., Qin, T., Xu, J., Xiong, W. Y., & Li, H. (2007). Letor: Benchmark dataset for research on learning to rank for information retrieval. Proceedings of SIGIR.
[13]
Marden, J. I. (Ed.). (1995). Analyzing and modeling rank data. London: Chapman and Hall.
[14]
Nallapati, R. (2004). Discriminative models for information retrieval. Proceedings of SIGIR (pp. 64--71).
[15]
Qin, T., Zhang, X.-D., Tsai, M.-F., Wang, D.-S., Liu, T.-Y., & Li, H. (2007). Query-level loss functions for information retrieval. Information processing and management.
[16]
Xu, J., & Li, H. (2007). Adarank: a boosting algorithm for information retrieval. Proceedings of SIGIR (pp. 391--398)
[17]
Yue, Y., Finley, T., Radlinski, F., & Joachims, T. (2007). A support vector method for optimization average precision. Proceedings of SIGIR (pp. 271--278).
[18]
Zhang, T. (2004). Statistical analysis of some multicategory large margin classification methods. Journal of Machine Learning Research, 5, 1225--1251.

Cited By

View all
  • (2025)Action-guided prompt tuning for video groundingInformation Fusion10.1016/j.inffus.2024.102577113(102577)Online publication date: Jan-2025
  • (2024)Enhancing Personalized Travel Recommendations: Integrating User Behavior and Content AnalysisProceedings of the 32nd International Conference on Information Systems Development10.62036/ISD.2024.49Online publication date: 2024
  • (2024)USTADProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693052(24486-24508)Online publication date: 21-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICML '08: Proceedings of the 25th international conference on Machine learning
July 2008
1310 pages
ISBN:9781605582054
DOI:10.1145/1390156
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • Pascal
  • University of Helsinki
  • Xerox
  • Federation of Finnish Learned Societies
  • Google Inc.
  • NSF
  • Machine Learning Journal/Springer
  • Microsoft Research: Microsoft Research
  • Intel: Intel
  • Yahoo!
  • Helsinki Institute for Information Technology
  • IBM: IBM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2008

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article

Conference

ICML '08
Sponsor:
  • Microsoft Research
  • Intel
  • IBM

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)385
  • Downloads (Last 6 weeks)32
Reflects downloads up to 07 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Action-guided prompt tuning for video groundingInformation Fusion10.1016/j.inffus.2024.102577113(102577)Online publication date: Jan-2025
  • (2024)Enhancing Personalized Travel Recommendations: Integrating User Behavior and Content AnalysisProceedings of the 32nd International Conference on Information Systems Development10.62036/ISD.2024.49Online publication date: 2024
  • (2024)USTADProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693052(24486-24508)Online publication date: 21-Jul-2024
  • (2024)Listwise reward estimation for offline preference-based reinforcement learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692415(8651-8671)Online publication date: 21-Jul-2024
  • (2024)Perturbation-invariant adversarial training for neural ranking modelsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28730(8832-8840)Online publication date: 20-Feb-2024
  • (2024)Learning to rank in generative retrievalProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28717(8716-8723)Online publication date: 20-Feb-2024
  • (2024)A Self-Distilled Learning to Rank Model for Ad Hoc RetrievalACM Transactions on Information Systems10.1145/368178442:6(1-28)Online publication date: 25-Jul-2024
  • (2024)Utility-Oriented Reranking with Counterfactual ContextACM Transactions on Knowledge Discovery from Data10.1145/367100418:8(1-22)Online publication date: 4-Jun-2024
  • (2024)Listwise Generative Retrieval Models via a Sequential Learning ProcessACM Transactions on Information Systems10.1145/365371242:5(1-31)Online publication date: 29-Apr-2024
  • (2024)Passage-aware Search Result DiversificationACM Transactions on Information Systems10.1145/365367242:5(1-29)Online publication date: 13-May-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media