skip to main content
10.1145/1866029.1866040acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

TurKit: human computation algorithms on mechanical turk

Published: 03 October 2010 Publication History

Abstract

Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. We present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. We also present case studies of TurKit used for real experiments across different fields.

References

[1]
}}von Ahn, L. Games With A Purpose. IEEE Computer Magazine, June 2006. Pages 96--98.
[2]
}}von Ahn, L. and Dabbish, L. Labeling Images with a Computer Game. ACM Conference on Human Factors in Computing Systems, CHI 2004.
[3]
}}von Ahn, L., Maurer, B., McMillen, C., Abraham, D. and Blum, M. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science, September 12, 2008. pp 1465--1468.
[4]
}}Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., Panovich, K. "Soylent: A Word Processor with a Crowd Inside". UIST 2010.
[5]
}}Bryant, S. L., et al. Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia. GROUP 2005.
[6]
}}Cornsweet, T. N. The Staircase-Method in Psychophysics. The American Journal of Psychology, Vol. 75, No. 3 (Sep., 1962), pp. 485--49.
[7]
}}Dai, P., Mausam, Weld, D. S. Decision-Theoretic Control of Crowd-Sourced Workflows. AAAI 2010.
[8]
}}Dia, M. A. "On Decision Making in Tandem Networks". M. Eng. Thesis. Massachusetts Institute of Technology. 2009.
[9]
}}Feldman, S. I. and Brown, C. B. IGOR: a system for program debugging via reversible execution. Proc. ACM SIGPLAN Workshop on Parallel and Distributed Debugging. 1988.
[10]
}}Heer, J., Bostock, M. Crowdsourcing Graphical Perception: Using MTurk to Assess Visualization Design. CHI 2010.
[11]
}}Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with MTurk. CHI 2008.
[12]
}}Kittur, A. and Kraut, R. E. Harnessing the wisdom of crowds in wikipedia: quality through coordination. CSCW 2008.
[13]
}}Ko, A. J. and Myers, B. A. Finding causes of program output with the Java Whyline. CHI 2009.
[14]
}}Kosorukoff A. Human based genetic algorithm. IlliGAL report no. 2001004. UIUC, 2001.
[15]
}}Little, G., Chilton, L. B., Goldman, M. and Miller, R. C. Exploring Iterative and Parallel Human Computation Processes. KDD-HCOMP 2010.
[16]
}}Mason, W. and Watts, D. J. Financial incentives and the "performance of crowds". KDD-HCOMP 2009.
[17]
}}Quinn, A. J., Bederson, B. B. A Taxonomy of Distributed Human Computation. Technical Report HCIL-2009-23. University of Maryland. 2009.
[18]
}}Russell, B., Torralba, A., Murphy, K., Freeman, W. LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, Vol. 77, No. 1 (1 May 2008), pp. 157--173.
[19]
}}Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. EMNLP 2008.
[20]
}}Sorokin, A. and D. Forsyth, "Utility data annotation with Amazon MTurk". CVPR 2008.

Cited By

View all
  • (2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024
  • (2024)Cooperation in the Gig Economy: Insights from Upwork FreelancersProceedings of the ACM on Human-Computer Interaction10.1145/36373148:CSCW1(1-20)Online publication date: 26-Apr-2024
  • (2024)AI-Assisted Causal Pathway Diagram for Human-Centered DesignProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642179(1-19)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology
October 2010
476 pages
ISBN:9781450302715
DOI:10.1145/1866029
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. human computation
  2. mturk
  3. toolkit

Qualifiers

  • Research-article

Conference

UIST '10

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25
The 38th Annual ACM Symposium on User Interface Software and Technology
September 28 - October 1, 2025
Busan , Republic of Korea

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)85
  • Downloads (Last 6 weeks)12
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024
  • (2024)Cooperation in the Gig Economy: Insights from Upwork FreelancersProceedings of the ACM on Human-Computer Interaction10.1145/36373148:CSCW1(1-20)Online publication date: 26-Apr-2024
  • (2024)AI-Assisted Causal Pathway Diagram for Human-Centered DesignProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642179(1-19)Online publication date: 11-May-2024
  • (2024)Trustworthy human computation: a surveyArtificial Intelligence Review10.1007/s10462-024-10974-157:12Online publication date: 12-Oct-2024
  • (2023)Crowdsourcing design theories : Perspectives and new challenges in Public Management2023 18th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI58278.2023.10211774(1-6)Online publication date: 20-Jun-2023
  • (2023)Supporting Requesters in Writing Clear Crowdsourcing Task Descriptions Through Computational Flaw AssessmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584039(737-749)Online publication date: 27-Mar-2023
  • (2023)It Takes (at least) Two: The Work to Make Romance WorkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580709(1-17)Online publication date: 19-Apr-2023
  • (2023)Research Approaches for Building Analytics in Social Network towards Crowdsourcing2023 IEEE 8th International Conference for Convergence in Technology (I2CT)10.1109/I2CT57861.2023.10126479(1-7)Online publication date: 7-Apr-2023
  • (2022)In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd WorkersFrontiers in Artificial Intelligence10.3389/frai.2022.8281875Online publication date: 18-May-2022
  • (2022)Revisiting Piggyback Prototyping: Examining Benefits and Tradeoffs in Extending Existing Social Computing SystemsProceedings of the ACM on Human-Computer Interaction10.1145/35555576:CSCW2(1-28)Online publication date: 11-Nov-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media