skip to main content
10.1145/3078714.3078715acmconferencesArticle/Chapter ViewAbstractPublication PageshtConference Proceedingsconference-collections
research-article

Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing

Published: 04 July 2017 Publication History

Abstract

Workers of microtask crowdsourcing marketplaces strive to find a balance between the need for monetary income and the need for high reputation. Such balance is often threatened by poorly formulated tasks, as workers attempt their execution despite a sub-optimal understanding of the work to be done.
In this paper we highlight the role of clarity as a characterising property of tasks in crowdsourcing. We surveyed 100 workers of the CrowdFlower platform to verify the presence of issues with task clarity in crowdsourcing marketplaces, reveal how crowd workers deal with such issues, and motivate the need for mechanisms that can predict and measure task clarity. Next, we propose a novel model for task clarity based on the goal and role clarity constructs. We sampled 7.1K tasks from the Amazon mTurk marketplace, and acquired labels for task clarity from crowd workers. We show that task clarity is coherently perceived by crowd workers, and is affected by the type of the task. We then propose a set of features to capture task clarity, and use the acquired labels to train and validate a supervised machine learning model for task clarity prediction. Finally, we perform a long-term analysis of the evolution of task clarity on Amazon mTurk, and show that clarity is not a property suitable for temporal characterisation.

References

[1]
Omar Alonso and Ricardo Baeza-Yates. 2011. Design and implementation of relevance assessments using crowdsourcing. In ECIR. Springer, 153--164.
[2]
Omar Alonso, Catherine Marshall, and Marc Najork. 2014. Crowdsourcing a subjective labeling task: a human-centered framework to ensure reliable results. Technical Report. MSR-TR-2014-91.
[3]
Janine Berg. 2016. Income security in the on-demand economy: findings and policy lessons from a survey of crowdworkers. Comparative Labor Law & Policy Journal 37, 3 (2016).
[4]
Hein Broekkamp, Bernadette HAM van Hout-Wolters, Gert Rijlaarsdam, and Huub van den Bergh. 2002. Importance in instructional text: teachers' and students' perceptions of task demands. Journal of Educational Psychology 94, 2 (2002), 260.
[5]
Francisco Cano and María Cardelle-Elawar. 2004. An integrated analysis of secondary school studentsfi conceptions and beliefs about learning. European Journal of Psychology of Education 19, 2 (2004), 167--187.
[6]
Jeanne Sternlicht Chall and Edgar Dale. 1995. Readability revisited: The new Dale-Chall readability formula. Brookline Books.
[7]
Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. ITL-International Journal of Applied Linguistics 165, 2 (2014), 97--135.
[8]
Kevyn Collins-Thompson and James P Callan. 2004. A language modeling ap- proach to predicting reading difficulty. In HLT-NAACL . 193--200.
[9]
Scott A Crossley, Kristopher Kyle, and Danielle S McNamara. 2015. The tool for the automatic analysis of text cohesion (TAACO): automatic assessment of local, global, and text cohesion. Behavior research methods (2015), 1--11.
[10]
Tove I Dahl, Margrethe Bals, and Anne Lene Turi. 2005. Are students' beliefs about knowledge and learning associated with their reported use of learning strategies? British journal of educational psychology 75, 2 (2005), 257--273.
[11]
Edgar Dale and Jeanne S Chall. 1949. The concept of readability. Elementary English 26, 1 (1949), 19--26.
[12]
Orphée De Clercq, Véronique Hoste, Bart Desmet, Philip Van Oosten, Martine De Cock, and Lieve Macken. 2014. Using the crowd for readability prediction. Natural Language Engineering 20, 03 (2014).
[13]
Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeiro- tis, and Philippe Cudré-Mauroux. 2015. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. In WWW. International World Wide Web Conferences Steering Committee, 238--247.
[14]
Ujwal Gadiraju, Ricardo Kawase, and Stefan Dietze. 2014. A taxonomy of micro- tasks on the web. In Hypertext. ACM, 218--223.
[15]
Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding malicious behavior in crowdsourcing platforms: the case of online surveys. In CHI. ACM, 1631--1640.
[16]
Catherine Grady and Matthew Lease. 2010. Crowdsourcing document relevance assessment with mechanical turk. In HLT-NAACL workshop on creating speech and language data with Amazon's mechanical turk. Association for Computation- alLinguistics, 172--179.
[17]
Arthur C Graesser, Danielle S McNamara, Max M Louwerse, and Zhiqiang Cai. 2004. Coh-Metrix: analysis of text on cohesion and language. Behavior research methods, instruments, & computers 36, 2 (2004), 193--202.
[18]
Allison Hadwin. 2006. Student task understanding. In Learning and Teaching Conference. University of Victoria, Victoria, British Columbia, Canada.
[19]
AF Hadwin, M Oshige, M Miller, and P Wild. 2009. Examining student and instructor task perceptions in a complex engineering design task. In international conference on innovation and practices in engineering design and engineering education. McMaster University, Hamilton, ON, Canada.
[20]
T Hoßfeld, Raimund Schatz, and Sebastian Egger. 2011. SOS: The MOS is not enough!. In QoMEX. IEEE, 131--136.
[21]
Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In CHI. ACM, 611--620.
[22]
Diane Lee Jamieson-Noel. 2004. Exploring task definition as a facet of self-regulated learning . Ph.D. Dissertation. Faculty of Education-Simon Fraser University.
[23]
Rohit J Kate, Xiaoqiang Luo, Siddharth Patwardhan, Martin Franz, Radu Flo- rian, Raymond J Mooney, Salim Roukos, and Chris Welty. 2010. Learning to predict readability using diverse linguistic features. In ACL. Association for Computational Linguistics, 546--554.
[24]
Shashank Khanna, Aishwarya Ratan, James Davis, and William Thies. 2010. Evaluating and improving the usability of Mechanical Turk for low-income workers in India. In DEV. ACM, 12.
[25]
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report. DTIC Document.
[26]
Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In CHI. ACM, 453--456.
[27]
Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In CSCW. ACM, 1301--1318.
[28]
Lieve Luyten, Joost Lowyck, and Francis Tuerlinckx. 2001. Task perception as a mediating variable: A contribution to the validation of instructional knowledge. British Journal of Educational Psychology 71, 2 (2001), 203--223.
[29]
David Malvern and Brian Richards. 2012. Measures of lexical richness. The Encyclopedia of Applied Linguistics (2012).
[30]
Catherine C Marshall and Frank M Shipman. 2013. Experiences surveying the crowd: Reflections on methods, participation, and reliability. In WebSci. ACM, 234--243.
[31]
Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In EMNLP. Association for Computational Linguistics, 186--195.
[32]
Presentacion Rivera-Reyes. 2015. Students' task interpretation and conceptual understanding in electronics laboratory work. (2015).
[33]
Libby O Ruch and Rae R Newton. 1977. Sex characteristics, task clarity, and authority. Sex Roles 3, 5 (1977), 479--494.
[34]
Aaron D Shaw, John J Horton, and Daniel L Chen. 2011. Designing incentives for inexpert human raters. In CSCW. ACM, 275--284.
[35]
John Sweller and Paul Chandler. 1994. Why some material is difficult to learn. Cognition and instruction 12, 3 (1994), 185--233.
[36]
Jie Yang, Claudia Hauff, Alessandro Bozzon, and Geert-Jan Houben. 2014. Asking the right question in collaborative q&a systems. In Hypertext. ACM, 179--189.
[37]
Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling task complexity in crowdsourcing. In HCOMP. AAAI, 249--258

Cited By

View all
  • (2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
  • (2024)Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657712(1952-1962)Online publication date: 10-Jul-2024
  • (2024)Exploring Collective Theory of Mind on Pedestrian Behavioral IntentionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650930(1-8)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HT '17: Proceedings of the 28th ACM Conference on Hypertext and Social Media
    July 2017
    336 pages
    ISBN:9781450347082
    DOI:10.1145/3078714
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 04 July 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. crowd workers
    2. crowdsourcing
    3. goal clarity
    4. microtasks
    5. performance
    6. prediction
    7. role clarity
    8. task clarity

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    HT'17
    Sponsor:
    HT'17: 28th Conference on Hypertext and Social Media
    July 4 - 7, 2017
    Prague, Czech Republic

    Acceptance Rates

    HT '17 Paper Acceptance Rate 19 of 69 submissions, 28%;
    Overall Acceptance Rate 378 of 1,158 submissions, 33%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)81
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
    • (2024)Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657712(1952-1962)Online publication date: 10-Jul-2024
    • (2024)Exploring Collective Theory of Mind on Pedestrian Behavioral IntentionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650930(1-8)Online publication date: 11-May-2024
    • (2024)Cognitive Biases in Fact-Checking and Their CountermeasuresInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10367261:3Online publication date: 2-Jul-2024
    • (2024)A clarity and fairness aware framework for selecting workers in competitive crowdsourcing tasksComputing10.1007/s00607-024-01316-8106:9(3005-3030)Online publication date: 6-Jul-2024
    • (2023)Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus DisagreementProceedings of the ACM on Human-Computer Interaction10.1145/36100747:CSCW2(1-26)Online publication date: 4-Oct-2023
    • (2023)Evaluating Descriptive Quality of AI-Generated Audio Using Image-SchemasProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584083(621-632)Online publication date: 27-Mar-2023
    • (2023)Supporting Requesters in Writing Clear Crowdsourcing Task Descriptions Through Computational Flaw AssessmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584039(737-749)Online publication date: 27-Mar-2023
    • (2023)Who Broke Amazon Mechanical Turk?Proceedings of the 15th ACM Web Science Conference 202310.1145/3578503.3583622(335-345)Online publication date: 30-Apr-2023
    • (2023)Combining Worker Factors for Heterogeneous Crowd Task AssignmentProceedings of the ACM Web Conference 202310.1145/3543507.3583190(3794-3805)Online publication date: 30-Apr-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media