skip to main content
10.1145/1277741.1277790acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
Article

A support vector method for optimizing average precision

Published: 23 July 2007 Publication History

Abstract

Machine learning is commonly used to improve ranked retrieval systems. Due to computational difficulties, few learning techniques have been developed to directly optimize for mean average precision (MAP), despite its widespread use in evaluating such systems. Existing approaches optimizing MAP either do not find a globally optimal solution, or are computationally expensive. In contrast, we present a general SVM learning algorithm that efficiently finds a globally optimal solution to a straightforward relaxation of MAP. We evaluate our approach using the TREC 9 and TREC 10 Web Track corpora (WT10g), comparing against SVMs optimized for accuracy and ROCArea. In most cases we show our method to produce statistically significant improvements in MAP scores.

References

[1]
B. T. Bartell, G. W. Cottrell, and R. K. Belew. Automatic combination of multiple ranked retrieval systems. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), 1994.
[2]
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of the International Conference on Machine Learning (ICML), 2005.
[3]
C. J. C. Burges, R. Ragno, and Q. Le. Learning to rank with non-smooth cost functions. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS), 2006.
[4]
Y. Cao, J. Xu, T.-Y. Liu, H. Li, Y. Huang, and H.-W. Hon. Adapting ranking SVM to document retrieval. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), 2006.
[5]
B. Carterette and D. Petkova. Learning a ranking from pairwise preferences. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), 2006.
[6]
R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes. Ensemble selection from libraries of models. In Proceedings of the International Conference on Machine Learning (ICML), 2004.
[7]
J. Davis and M. Goadrich. The relationship between precision-recall and ROC curves. In Proceedings of the International Conference on Machine Learning (ICML), 2006.
[8]
D. Hawking. Overview of the TREC-9 web track. 2000.
[9]
D. Hawking and N. Craswell. Overview of the TREC-2001 web track. Nov. 2001.
[10]
R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. Advances in large margin classifiers, 2000.
[11]
A. Herschtal and B. Raskutti. Optimising area under the ROC curve using gradient descent. In Proceedings of the International Conference on Machine Learning (ICML), 2004.
[12]
K. Jarvelin and J. Kekalainen. Ir evaluation methods for retrieving highly relevant documents. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), 2000.
[13]
T. Joachims. A support vector method for multivariate performance measures. In Proceedings of the International Conference on Machine Learning (ICML), pages 377--384, New York, NY, USA, 2005. ACM Press.
[14]
J. Lafferty and C. Zhai. Document language models, query models, and risk minimization for information retrieval. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 111--119, 2001.
[15]
Y. Lin, Y. Lee, and G. Wahba. Support vector machines for classification in nonstandard situations. Machine Learning, 46:191--202, 2002.
[16]
D. Metzler and W. B. Croft. A markov random field model for term dependencies. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 472--479, 2005.
[17]
K. Morik, P. Brockhausen, and T. Joachims. Combining statistical learning with a knowledge-based approach. In Proceedings of the International Conference on Machine Learning, 1999.
[18]
S. Robertson. The probability ranking principle in ir. journal of documentation. Journal of Documentation, 33(4):294--304, 1977.
[19]
I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research (JMLR), pages 1453--1484, 2005.
[20]
V. Vapnik. Statistical Learning Theory. Wiley and Sons Inc., 1998.
[21]
L. Yan, R. Dodier, M. Mozer, and R. Wolniewicz. Optimizing classifier performance via approximation to the Wilcoxon-Mann-Witney statistic. In Proceedings of the International Conference on Machine Learning (ICML), 2003.

Cited By

View all
  • (2025)A Multimodal Deep Learning Model for Detecting Endoscopic Images of Near-Infrared Fluorescence CapsulesBiosensors and Bioelectronics10.1016/j.bios.2025.117251(117251)Online publication date: Feb-2025
  • (2024)Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video RetrievalProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681110(3828-3837)Online publication date: 28-Oct-2024
  • (2024)Towards Open-World Recommendation with Knowledge Augmentation from Large Language ModelsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688104(12-22)Online publication date: 8-Oct-2024
  • Show More Cited By

Index Terms

  1. A support vector method for optimizing average precision

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
    July 2007
    946 pages
    ISBN:9781595935977
    DOI:10.1145/1277741
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 23 July 2007

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. machine learning for information retrieval
    2. ranking
    3. support vector machines

    Qualifiers

    • Article

    Conference

    SIGIR07
    Sponsor:
    SIGIR07: The 30th Annual International SIGIR Conference
    July 23 - 27, 2007
    Amsterdam, The Netherlands

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)127
    • Downloads (Last 6 weeks)9
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)A Multimodal Deep Learning Model for Detecting Endoscopic Images of Near-Infrared Fluorescence CapsulesBiosensors and Bioelectronics10.1016/j.bios.2025.117251(117251)Online publication date: Feb-2025
    • (2024)Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video RetrievalProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681110(3828-3837)Online publication date: 28-Oct-2024
    • (2024)Towards Open-World Recommendation with Knowledge Augmentation from Large Language ModelsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688104(12-22)Online publication date: 8-Oct-2024
    • (2024)Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted TreesProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657918(2390-2394)Online publication date: 10-Jul-2024
    • (2024)Towards Safer Roads: A Deep Learning Based Object Detection Technique for Vehicle Safety2024 International Conference on Advances in Computing, Communication, Electrical, and Smart Systems (iCACCESS)10.1109/iCACCESS61735.2024.10499581(01-06)Online publication date: 8-Mar-2024
    • (2024)RLocator: Reinforcement Learning for Bug LocalizationIEEE Transactions on Software Engineering10.1109/TSE.2024.345259550:10(2695-2708)Online publication date: 1-Oct-2024
    • (2024)Algorithm-Dependent Generalization of AUPRC Optimization: Theory and AlgorithmIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336186146:7(5062-5079)Online publication date: Jul-2024
    • (2024)Split Attention Mechanism of Faster RCNN for PCB Defect Detection2024 International Conference on New Trends in Computational Intelligence (NTCI)10.1109/NTCI64025.2024.10776186(514-521)Online publication date: 18-Oct-2024
    • (2024)An Unsupervised and End-to-End Registration Method Using Offset Field and Pseudodata for Video SAR ImagesIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing10.1109/JSTARS.2024.346583517(18517-18534)Online publication date: 2024
    • (2024)Mass-Ratio-Average-Absolute-Deviation Based Outlier Factor for Anomaly Scoring2024 21st International Joint Conference on Computer Science and Software Engineering (JCSSE)10.1109/JCSSE61278.2024.10613697(488-493)Online publication date: 19-Jun-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media