skip to main content
10.1145/1520340.1520597acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
extended-abstract

Rethinking the ESP game

Published: 04 April 2009 Publication History

Abstract

The ESP Game was designed to harvest human intelligence to assign labels to images - a task which is still difficult for even the most advanced systems in image processing. However, the ESP Game as it is currently implemented encourages players to assign "obvious" labels, which can be easily predicted given previously assigned labels.
We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one "off-limits" term assigned to them.
We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.

References

[1]
Ames, M. and Naaman, M. Why we tag: motivations for annotation in mobile and online media. In Proc. CHI'07 (2007), pages 971--980.
[2]
Garg, N. and Weber, I. Personalized, interactive tag recommendation for flickr. In Proc. RecSys'08 (2008), pages 67--74.
[3]
Jeon, J., Lavrenko, V. and Manmatha, R. Automatic image annotation and retrieval using cross--media relevance models. In Proc. SIGIR'03 (2003), 119--126.
[4]
Sen, S., Harper, F. M., LaPitz, A. and Riedl, J. The quest for quality tags. In Proc. GROUP'07 (2007), 361--370.
[5]
Sen, S., Lam, S. K., Rashid, A. M., Cosley, D., Frankowski, D., Osterhouse, J., Harper, F. M. and Riedl, J. Tagging, communities, vocabulary, evolution. In Proc. CSCW'06 (2006), 181---190.
[6]
Sigurbjörnsson, B. and van Zwol, R. Flickr tag recommendation based on collective knowledge. In Proc. WWW'08 (2008), 327--336.
[7]
von Ahn, L. and Dabbish, L. Labeling images with a computer game. In Proc. CHI'04 (2004), 319--326.
[8]
von Ahn, L. Games with a purpose. In Computer (2006), Vol. 39, No. 6, 92--94.

Cited By

View all
  • (2022)Human ComputationundefinedOnline publication date: 11-Mar-2022
  • (2019)FaRMProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367032.3367105(506-512)Online publication date: 10-Aug-2019
  • (2019)Crowdsourcing image descriptions using gamification: a comparison between game-generated labels and professional descriptors2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)10.23919/MIPRO.2019.8756841(537-541)Online publication date: May-2019
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '09: CHI '09 Extended Abstracts on Human Factors in Computing Systems
April 2009
2470 pages
ISBN:9781605582474
DOI:10.1145/1520340
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2009

Check for updates

Author Tags

  1. esp game
  2. image labeler
  3. tagging

Qualifiers

  • Extended-abstract

Conference

CHI '09
Sponsor:

Acceptance Rates

CHI EA '09 Paper Acceptance Rate 385 of 1,130 submissions, 34%;
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)1
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Human ComputationundefinedOnline publication date: 11-Mar-2022
  • (2019)FaRMProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367032.3367105(506-512)Online publication date: 10-Aug-2019
  • (2019)Crowdsourcing image descriptions using gamification: a comparison between game-generated labels and professional descriptors2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)10.23919/MIPRO.2019.8756841(537-541)Online publication date: May-2019
  • (2018)ClueMeInProceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts10.1145/3270316.3271534(447-452)Online publication date: 23-Oct-2018
  • (2017)LoBIAG: A location-based collaborative image annotation game2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI)10.1109/KBEI.2017.8324877(0634-0640)Online publication date: Dec-2017
  • (2016)Persistent Sharing of Fitness App Status on TwitterProceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing10.1145/2818048.2819921(184-194)Online publication date: 27-Feb-2016
  • (2016)QuaCentiveThe Journal of Supercomputing10.1007/s11227-015-1395-y72:8(2924-2941)Online publication date: 1-Aug-2016
  • (2015)A Survey on Assessment and Ranking Methodologies for User-Generated Content on the WebACM Computing Surveys10.1145/281128248:3(1-49)Online publication date: 22-Dec-2015
  • (2015)Getting More for LessProceedings of the 24th International Conference on World Wide Web10.1145/2736277.2741681(592-602)Online publication date: 18-May-2015
  • (2015)Crowdsourcing StereotypesProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems10.1145/2702123.2702151(1955-1964)Online publication date: 18-Apr-2015
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media