skip to main content
10.1145/2700171.2791053acmconferencesArticle/Chapter ViewAbstractPublication PageshtConference Proceedingsconference-collections
short-paper

Breaking Bad: Understanding Behavior of Crowd Workers in Categorization Microtasks

Published: 24 August 2015 Publication History

Abstract

Crowdsourcing systems are being widely used to overcome several challenges that require human intervention. While there is an increase in the adoption of the crowdsourcing paradigm as a solution, there are no established guidelines or tangible recommendations for task design with respect to key parameters such as task length, monetary incentive and time required for task completion. In this paper, we propose the tuning of these parameters based on our findings from extensive experiments and analysis of categorization tasks. We delve into the behavior of workers that consume categorization tasks to determine measures that can make task design more effective.

References

[1]
C. Eickhoff and A. de Vries. How crowdsourcable is your task. In Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pages 11--14, 2011.
[2]
C. Eickhoff and A. P. de Vries. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, 16(2):121--137, 2013.
[3]
U. Gadiraju, R. Kawase, and S. Dietze. A taxonomy of microtasks on the web. In Proceedings of the 25th ACM conference on Hypertext and social media, pages 218--223. ACM, 2014.
[4]
U. Gadiraju, R. Kawase, S. Dietze, and G. Demartini. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of CHI'15, CHI Conference on Human Factors in Computing Systems, 2015.
[5]
P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, pages 64--67. ACM, 2010.
[6]
N. Kaufmann, T. Schulze, and D. Veit. More than fun and money. worker motivation in crowdsourcing - a study on mechanical turk. In AMCIS, 2011.
[7]
G. Kazai, J. Kamps, and N. Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1941--1944. ACM, 2011.
[8]
C. C. Marshall and F. M. Shipman. Experiences surveying the crowd: Reflections on methods, participation, and reliability. In Proceedings of the 5th Annual ACM Web Science Conference, WebSci '13, pages 234--243, New York, NY, USA, 2013. ACM.
[9]
W. Mason and D. J. Watts. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2):100--108, 2010.
[10]
D. Oleson, A. Sorokin, G. P. Laughlin, V. Hester, J. Le, and L. Biewald. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human computation, 11:11, 2011.
[11]
J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, and M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM, 2011.
[12]
J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, pages 2863--2872. ACM, 2010.

Cited By

View all
  • (2021)Using Attention Testing to Select Crowdsourced Workers and Research ParticipantsSocial Science Computer Review10.1177/089443931984872639:1(84-104)Online publication date: 1-Feb-2021
  • (2021)Remembering Both the Machine and the Crowd When Sampling Points: Active Learning for Semantic Segmentation of ALS Point CloudsPattern Recognition. ICPR International Workshops and Challenges10.1007/978-3-030-68787-8_37(505-520)Online publication date: 10-Jan-2021
  • (2020)Just the Right Mood for HIT!Web Engineering10.1007/978-3-030-50578-3_26(381-396)Online publication date: 9-Jun-2020
  • Show More Cited By

Index Terms

  1. Breaking Bad: Understanding Behavior of Crowd Workers in Categorization Microtasks

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HT '15: Proceedings of the 26th ACM Conference on Hypertext & Social Media
    August 2015
    360 pages
    ISBN:9781450333955
    DOI:10.1145/2700171
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 24 August 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. behavior
    2. categorization
    3. crowdsourcing
    4. incentives
    5. microtasks
    6. task length
    7. workers

    Qualifiers

    • Short-paper

    Funding Sources

    • European Commission

    Conference

    HT '15
    Sponsor:
    HT '15: 26th ACM Conference on Hypertext and Social Media
    September 1 - 4, 2015
    Guzelyurt, Northern Cyprus

    Acceptance Rates

    HT '15 Paper Acceptance Rate 24 of 60 submissions, 40%;
    Overall Acceptance Rate 378 of 1,158 submissions, 33%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)3
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)Using Attention Testing to Select Crowdsourced Workers and Research ParticipantsSocial Science Computer Review10.1177/089443931984872639:1(84-104)Online publication date: 1-Feb-2021
    • (2021)Remembering Both the Machine and the Crowd When Sampling Points: Active Learning for Semantic Segmentation of ALS Point CloudsPattern Recognition. ICPR International Workshops and Challenges10.1007/978-3-030-68787-8_37(505-520)Online publication date: 10-Jan-2021
    • (2020)Just the Right Mood for HIT!Web Engineering10.1007/978-3-030-50578-3_26(381-396)Online publication date: 9-Jun-2020
    • (2019)Rehumanized CrowdsourcingProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300773(1-12)Online publication date: 2-May-2019
    • (2017)Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the CrowdEvaluation in the Crowd. Crowdsourcing and Human-Centered Experiments10.1007/978-3-319-66435-4_2(6-26)Online publication date: 28-Sep-2017

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media