skip to main content
10.1145/1526709.1526727acmconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article

Learning consensus opinion: mining data from a labeling game

Published: 20 April 2009 Publication History

Abstract

We consider the problem of identifying the consensus ranking for the results of a query, given preferences among those results from a set of individual users. Once consensus rankings are identified for a set of queries, these rankings can serve for both evaluation and training of retrieval and learning systems. We present a novel approach to collecting the individual user preferences over image-search results: we use a collaborative game in which players are rewarded for agreeing on which image result is best for a query. Our approach is distinct from other labeling games because we are able to elicit directly the preferences of interest with respect to image queries extracted from query logs. As a source of relevance judgments, this data provides a useful complement to click data. Furthermore, the data is free of positional biases and is collected by the game without the risk of frustrating users with non-relevant results; this risk is prevalent in standard mechanisms for debiasing clicks. We describe data collected over 34 days from a deployed version of this game that amounts to about 18 million expressed preferences between pairs. Finally, we present several approaches to modeling this data in order to extract the consensus rankings from the preferences and better sort the search results for targeted queries.

References

[1]
R. Bradley and M. Terry. Rank analysis of incomplete block designs, i. the method of paired comparisons. Biometrika, pages 324--345, 1952.
[2]
B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there: Preference judgments for relevance. In ECIR 2008, 2008.
[3]
W. Chu and Z. Ghahramani. Preference learning with gaussian processes. In ICML 2005, 2005.
[4]
C. Cleverdon. The significance of the cranfield tests on index languages. In SIGIR '91, pages 3--12, 1991.
[5]
W. Cohen, R. Schapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243--270, 1999.
[6]
C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In WWW '10, pages 613--622, 2001.
[7]
J. Fürnkranz and E. Hüllermeier. Pairwise preference learning and ranking. In ECML 2003, 2003.
[8]
R. Herbrich and T. Graepel. Trueskill: A bayesian skill rating system. Technical Report MSR-TR-2006-80, Microsoft Research, June 2006. ftp://ftp.research.microsoft.com/pub/tr/TR-2006-80.pdf.
[9]
T. Joachims. Optimizing search engines using clickthrough data. In KDD 2002, 2002.
[10]
E. Law, L. von Ahn, R. Dannenberg, and M. Crawford. Tagatune: A game for music and sound annotation. In Proceedings of the 8th International Conference on Music Information Retrieval, 2007.
[11]
R. Luce. Individual Choice Behaviours: A Theoretical Analysis. J. Wiley, New York, 1959.
[12]
F. Radlinski and T. Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In AAAI 2005, 2005.
[13]
F. Radlinski, R. Kleinberg, and T. Joachims. Learning diverse rankings with multi-armed bandits. In ICML 2008, 2008.
[14]
S. E. Robertson. The probability ranking principle in ir. Journal of Documentation, 33:294--304, 1977. Reprinted in: K. Sparck Jones and P. Willett (eds), Readings in Information Retrieval. Morgan Kaufmann, 1997. (pp 281--286).
[15]
D. Stern, R. Herbrich, and T. Graepel. Bayesian pattern ranking for move prediction in the game of go. In ICML '06, 2006.
[16]
L. Thurstone. A law of comparative judgment. Psychological Review, pages 273--286, 1927.
[17]
L. Thurstone. Essays in Philosophy by Seventeen Doctors of Philosophy of the University of Chicago, chapter The Measurement of Psychological Value. Open Court, Chicago, 1929.
[18]
L. Thurstone. The Measurement of Values. The University of Chicago Press, Chicago, 1959.
[19]
L. von Ahn and L. Dabbish. Labeling images with a computer game. In ACM CHI, 2004.
[20]
L. von Ahn, R. Liu, and M. Blum. Peekaboom: A game for locating objects in images. In ACM CHI, 2006.

Cited By

View all
  • (2021)Signal Processing on the Permutahedron: Tight Spectral Frames for Ranked Data AnalysisJournal of Fourier Analysis and Applications10.1007/s00041-021-09878-327:4Online publication date: 9-Aug-2021
  • (2020)Big Data and Occupants’ Behavior in Built Environments: Introducing a Game-Based Data Collection MethodJournal of Urban Planning and Development10.1061/(ASCE)UP.1943-5444.0000552146:2Online publication date: Jun-2020
  • (2016)Setwise ComparisonProceedings of the 2016 CHI Conference on Human Factors in Computing Systems10.1145/2858036.2858199(261-271)Online publication date: 7-May-2016
  • Show More Cited By

Index Terms

  1. Learning consensus opinion: mining data from a labeling game

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      WWW '09: Proceedings of the 18th international conference on World wide web
      April 2009
      1280 pages
      ISBN:9781605584874
      DOI:10.1145/1526709

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 20 April 2009

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. learning preferences
      2. preference judgments

      Qualifiers

      • Research-article

      Conference

      WWW '09
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)2
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 27 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2021)Signal Processing on the Permutahedron: Tight Spectral Frames for Ranked Data AnalysisJournal of Fourier Analysis and Applications10.1007/s00041-021-09878-327:4Online publication date: 9-Aug-2021
      • (2020)Big Data and Occupants’ Behavior in Built Environments: Introducing a Game-Based Data Collection MethodJournal of Urban Planning and Development10.1061/(ASCE)UP.1943-5444.0000552146:2Online publication date: Jun-2020
      • (2016)Setwise ComparisonProceedings of the 2016 CHI Conference on Human Factors in Computing Systems10.1145/2858036.2858199(261-271)Online publication date: 7-May-2016
      • (2015)Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search LogsProceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806479(1441-1450)Online publication date: 17-Oct-2015
      • (2014)Modeling Paying Behavior in Game Social NetworksProceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management10.1145/2661829.2662024(411-420)Online publication date: 3-Nov-2014
      • (2014)On the application of game mechanics in information retrievalProceedings of the First International Workshop on Gamification for Information Retrieval10.1145/2594776.2594778(7-11)Online publication date: 13-Apr-2014
      • (2014)Structured labeling for facilitating concept evolution in machine learningProceedings of the SIGCHI Conference on Human Factors in Computing Systems10.1145/2556288.2557238(3075-3084)Online publication date: 26-Apr-2014
      • (2014)Gaming with Purpose: Heuristic Understanding of Ubiquitous Game Development and Design for Human ComputationHandbook of Digital Games10.1002/9781118796443.ch24(645-666)Online publication date: 7-Mar-2014
      • (2013)Applying human computation mechanisms to information retrievalProceedings of the American Society for Information Science and Technology10.1002/meet.1450490105049:1(1-10)Online publication date: 24-Jan-2013
      • (2011)Human ComputationSynthesis Lectures on Artificial Intelligence and Machine Learning10.2200/S00371ED1V01Y201107AIM0135:3(1-121)Online publication date: 30-Jun-2011
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media