skip to main content
10.1145/1390334.1390451acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
poster

Evaluation measures for preference judgments

Published: 20 July 2008 Publication History

Abstract

There has been recent interest in collecting user or assessor preferences, rather than absolute judgments of relevance, for the evaluation or learning of ranking algorithms. Since measures like precision, recall, and DCG are defined over absolute judgments, evaluation over preferences will require new evaluation measures that explicitly model them. We describe a class of such measures and compare absolute and preference measures over a large TREC collection.

References

[1]
B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there: Preference judgments for relevance. In Proceedings of ECIR, pages 16--27, 2008.
[2]
M. E. Rorvig. The simple scalability of documents. JASIS, 41(8):590--598, 1990.

Cited By

View all
  • (2022)Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance JudgmentsProceedings of the ACM Web Conference 202210.1145/3485447.3511960(319-327)Online publication date: 25-Apr-2022
  • (2018)Outsider PerspectivesExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3188602(1-6)Online publication date: 20-Apr-2018
  • (2017)An in-depth study on diversity evaluationInformation Processing and Management: an International Journal10.1016/j.ipm.2017.03.00153:4(799-813)Online publication date: 1-Jul-2017
  • Show More Cited By

Index Terms

  1. Evaluation measures for preference judgments

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
    July 2008
    934 pages
    ISBN:9781605581644
    DOI:10.1145/1390334
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 July 2008

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. evaluation
    2. preference judgments

    Qualifiers

    • Poster

    Conference

    SIGIR '08
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)14
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance JudgmentsProceedings of the ACM Web Conference 202210.1145/3485447.3511960(319-327)Online publication date: 25-Apr-2022
    • (2018)Outsider PerspectivesExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3188602(1-6)Online publication date: 20-Apr-2018
    • (2017)An in-depth study on diversity evaluationInformation Processing and Management: an International Journal10.1016/j.ipm.2017.03.00153:4(799-813)Online publication date: 1-Jul-2017
    • (2017)Low-Cost Preference Judgment via TiesAdvances in Information Retrieval10.1007/978-3-319-56608-5_58(626-632)Online publication date: 8-Apr-2017
    • (2016)Learning to Rank with Labeled FeaturesProceedings of the 2016 ACM International Conference on the Theory of Information Retrieval10.1145/2970398.2970435(41-44)Online publication date: 12-Sep-2016
    • (2016)Pairwise Preferences Based Matrix Factorization and Nearest Neighbor Recommendation TechniquesProceedings of the 10th ACM Conference on Recommender Systems10.1145/2959100.2959142(143-146)Online publication date: 7-Sep-2016
    • (2013)Is top-k sufficient for ranking?Proceedings of the 22nd ACM international conference on Information & Knowledge Management10.1145/2505515.2505685(1261-1270)Online publication date: 27-Oct-2013
    • (2013)SRbench--a benchmark for soundtrack recommendation systemsProceedings of the 22nd ACM international conference on Information & Knowledge Management10.1145/2505515.2505658(2285-2290)Online publication date: 27-Oct-2013
    • (2013)A mutual information-based framework for the analysis of information retrieval systemsProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484073(683-692)Online publication date: 28-Jul-2013
    • (2013)The Utility of Discourse Structure in Forum Thread RetrievalInformation Retrieval Technology10.1007/978-3-642-45068-6_25(284-295)Online publication date: 2013
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media