skip to main content
10.1145/1277741.1277792acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
Article

A regression framework for learning ranking functions using relative relevance judgments

Published: 23 July 2007 Publication History

Abstract

Effective ranking functions are an essential part of commercial search engines. We focus on developing a regression framework for learning ranking functions for improving relevance of search engines serving diverse streams of user queries. We explore supervised learning methodology from machine learning, and we distinguish two types of relevance judgments used as the training data: 1) absolute relevance judgments arising from explicit labeling of search results; and 2) relative relevance judgments extracted from user click throughs of search results or converted from the absolute relevance judgments. We propose a novel optimization framework emphasizing the use of relative relevance judgments. The main contribution is the development of an algorithm based on regression that can be applied to objective functions involving preference data, i.e., data indicating that a document is more relevant than another with respect to a query. Experimental results are carried out using data sets obtained from a commercial search engine. Our results show significant improvements of our proposed methods over some existing methods.

References

[1]
R. Atterer, M. Wunk, and A. Schmidt. Knowing the user's every move: user activity tracking for website usability evaluation and implicit interaction. Proceedings of the 15th International Conference on World Wide Web 203--212,2006.
[2]
A. Berger. Statistical machine learning for information retrieval Ph.D. Thesis, School of Computer Science, Carnegie Mellon University, 2001.
[3]
D. Bertsekas. Nonlinear programming Athena Scienti?c, second edition, 1999.
[4]
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. Proceedings of international conference on Machine learning 89--96, 2005.
[5]
H. Chen. Machine Learning for information retrieval: Neural networks, symbolic learning and genetic algorithms. JASIS 46:194--216, 1995.
[6]
W. Cooper, F. Gey and A. Chen. Probabilistic retrieval in the TIPSTER collections: an application of staged logistic regression. Proceedings of TREC 73--88, 1992.
[7]
D. Cossock and T. Zhang. Subset ranking using regression. COLT 2006.
[8]
Y. Freund, R. Iyer, R. Schapire and Y. Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research 4:933--969, 2003.
[9]
J. Friedman. Greedy function approximation: a gradient boosting machine. Ann. Statist. 29:1189--1232, 2001.
[10]
N. Fuhr. Optimum polynomial retrieval functions based on probability ranking principle. ACM Transactions on Information Systems 7:183--204, 1989.
[11]
F. Gey, A. Chen, J. He and J. Meggs. Logistic regression at TREC4: probabilistic retrieval from full text document collections. Proceedings of TREC 65--72, 1995.
[12]
K. Järvelin and J.Kekäläinen.Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20:422--446, 2002.
[13]
T. Joachims. Optimizing search engines using clickthrough data. Proceedings of the ACM Conference on Knowledge Discovery and Data Mining 2002.
[14]
T. Joachims. Evaluating retrieval performance using clickthrough data. Proceedings of the SIGIR Workshop on Mathematical/Formal Methods in Information Retrieval 2002.
[15]
T. Joachims, L. Granka, B. Pang, H. Hembrooke, and G. Gay. Accurately Interpreting Clickthrough Data as Implicit Feedback. Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2005.
[16]
J. Ponte and W. Croft. A language modeling approach to information retrieval. In Proceedings of the ACM Conference on Research and Development in Information Retrieval 1998.
[17]
G. Salton. Automatic Text Processing. Addison Wesley, Reading, MA, 1989.
[18]
H. Turtle and W. B. Croft. Inference networks for document retrieval. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1-24, 1990.
[19]
H. Zha, Z. Zheng, H. Fu and G. Sun. Incorporating query difference for learning retrieval functions in worldwidewebsearch. Proceedings of the 15th ACM Conference on Information and Knowledge Management 2006.
[20]
Diane Kelly and Jaime Teevan. Implicit Feedback for Inferring User Preference: A Bibliography. SIGIR Forum 32:2, 2003.
[21]
F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), 2005.
[22]
C. Zhai and J. Lafferty. A risk minimization framework for information retrieval, Information Processing and Management 42:31--55, 2006.

Cited By

View all
  • (2024)Utility-Oriented Reranking with Counterfactual ContextACM Transactions on Knowledge Discovery from Data10.1145/367100418:8(1-22)Online publication date: 4-Jun-2024
  • (2024)Understanding the Ranking Loss for Recommendation with Sparse User FeedbackProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671565(5409-5418)Online publication date: 25-Aug-2024
  • (2024)Collaborative Scope: Encountering the Substitution Effect within the Delivery Scope in Online Food Delivery PlatformProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680029(5151-5158)Online publication date: 21-Oct-2024
  • Show More Cited By

Index Terms

  1. A regression framework for learning ranking functions using relative relevance judgments

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
      July 2007
      946 pages
      ISBN:9781595935977
      DOI:10.1145/1277741
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 23 July 2007

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. absolute relevance judgment
      2. clickthroughs
      3. functional gradient descent
      4. gradient boosting
      5. machine learning
      6. preferences
      7. ranking function
      8. regression
      9. relative relevance judgment

      Qualifiers

      • Article

      Conference

      SIGIR07
      Sponsor:
      SIGIR07: The 30th Annual International SIGIR Conference
      July 23 - 27, 2007
      Amsterdam, The Netherlands

      Acceptance Rates

      Overall Acceptance Rate 792 of 3,983 submissions, 20%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)18
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Utility-Oriented Reranking with Counterfactual ContextACM Transactions on Knowledge Discovery from Data10.1145/367100418:8(1-22)Online publication date: 4-Jun-2024
      • (2024)Understanding the Ranking Loss for Recommendation with Sparse User FeedbackProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671565(5409-5418)Online publication date: 25-Aug-2024
      • (2024)Collaborative Scope: Encountering the Substitution Effect within the Delivery Scope in Online Food Delivery PlatformProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680029(5151-5158)Online publication date: 21-Oct-2024
      • (2024)Big portfolio selection by graph-based conditional moments methodJournal of Empirical Finance10.1016/j.jempfin.2024.10153378(101533)Online publication date: Sep-2024
      • (2024)An overview of sentence ordering taskInternational Journal of Data Science and Analytics10.1007/s41060-024-00550-918:1(1-18)Online publication date: 25-Apr-2024
      • (2024)A Simple yet Effective Framework for Active Learning to RankMachine Intelligence Research10.1007/s11633-023-1422-z21:1(169-183)Online publication date: 15-Jan-2024
      • (2023)Investigating Online Art Search through Quantitative Behavioral Data and Machine Learning TechniquesAnalytics10.3390/analytics20200212:2(359-392)Online publication date: 26-Apr-2023
      • (2023)S2phere: Semi-Supervised Pre-training for Web Search over Heterogeneous Learning to Rank DataProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599935(4437-4448)Online publication date: 6-Aug-2023
      • (2023)A Preference Judgment Tool for Authoritative AssessmentProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591801(3100-3104)Online publication date: 19-Jul-2023
      • (2023) COLTR : Semi-Supervised Learning to Rank With Co-Training and Over-Parameterization for Web Search IEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327075035:12(12542-12555)Online publication date: 1-Dec-2023
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media