skip to main content
10.1145/1868914.1868976acmotherconferencesArticle/Chapter ViewAbstractPublication PagesnordichiConference Proceedingsconference-collections
research-article

Crowdsourcing human-based computation

Published: 16 October 2010 Publication History

Abstract

Thousands of websites have been created to crowdsource tasks. In this paper, systems that crowdsource human-based computations are organized into four distinct classes using two factors: the users' motivation for completing the task (direct or indirect) and whether task completion is competitive. These classes are described and compared. Considerations and selection criteria for systems designers are presented. This investigation also identified several opportunities for further research. For example, existing systems might benefit from the integration of methods for transforming complex tasks into many simple tasks.

References

[1]
Adamic, L. A., Zhang, J., Bakshy, E., and Ackerman, M. S. 2008. Knowledge sharing and yahoo answers: everyone knows something. In Proc. WWW 2008. ACM, New York, NY, 665--674.
[2]
Bouguessa, M., Dumoulin, B., and Wang, S. 2008. Identifying authoritative actors in question-answering forums: the case of Yahoo! Answers. In Proc. 14th ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (Las Vegas, Nevada, USA, August 24--27, 2008). KDD '08. ACM, New York, NY, 866--874
[3]
Brin, S. and Page, L. 1998. The anatomy of a large-scale hypertextual Web search engine. Comput. Netw. ISDN Syst. 30, 1--7 (Apr. 1998), 107--117.
[4]
Bryant, S. L., Forte, A., and Bruckman, A. 2005. Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia. In Proc. 2005 international ACM SIGGROUP Conference on Supporting Group Work (Sanibel Island, Florida, USA, November 06--09, 2005). GROUP '05. ACM, New York, NY, 1--10.
[5]
Casting Words. http://castingwords.com/.
[6]
Cheyne, T. L. and Ritter, F. E. 2001. Targeting audiences on the Internet. Commun. ACM 44, 4 (Apr. 2001), 94--98.
[7]
Digg Site Ranking. http://www.alexa.com/siteinfo/digg.com.
[8]
Digg. http://digg.com.
[9]
InnoCentive. http://innocentive.com.
[10]
Kittur, A., Chi, E. H., and Suh, B. 2008. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI 2008. ACM, New York, NY, 453--456.
[11]
Kosorukoff, A. (2001) Human based genetic algorithm. IEEE Transactions on Systems, Man, and Cyber-netics, SMC-2001, 3464{3459
[12]
Kuznetsov, S. 2006. Motivations of contributors to Wikipedia. SIGCAS Comput. Soc. 36, 2 (Jun. 2006), 1.
[13]
L. von Ahn, M. Blum, J. Langford, Commun. ACM 47, 56 (2004).
[14]
Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. 2009. TurKit: tools for iterative tasks on mechanical Turk. In Proc. ACM SIGKDD Workshop on Human Computation (Paris, France, June 28--28, 2009). P. Bennett, R. Chandrasekar, M. Chickering, P. Ipeirotis, E. Law, A. Mityagin, F. Provost, and L. von Ahn, Eds. HCOMP '09. ACM, New York, NY, 29--30.
[15]
Liu, Y. and Agichtein, E. 2008. On the evolution of the Yahoo! Answers QA community. In Proc. 31st Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Singapore, Singapore, July 20--24, 2008). SIGIR '08. ACM, New York, NY, 737--738.
[16]
McDaniel, S. W. and Rao, C. P. An investigation on respondent anonymity's effect on mailed questionnaire response rate and quality. Journal of the Market Research Society 23, 3 (1981), 150--160.
[17]
Mechanical Turk. http://mturk.com.
[18]
Panos Ipeirotis, Mechanical Turk: The Demographics. http://behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html.
[19]
Pitkow, J. E. and Kehoe, C. M. 1996. Emerging trends in the WWW user population. Comm. ACM 39, 6 (Jun. 1996), 106--108.
[20]
Pitkow, J. Recker, M. Using the Web as a survey tool: Results from the Second WWW User Survey 1995; http://www.igd.fhg.de/www/www95/papers/79/survey/survey_2_paper.html.
[21]
reCAPTCHA: Human-Based Character Recognition via Web Security Measures Luis von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum (12 September 2008) Science 321 (5895), 1465.
[22]
Salon, "I make $1.45 a week and I love it". http://www.salon.com/tech/feature/2006/07/24/turks/.
[23]
Slashdot. http://slashdot.com.
[24]
Smart Company, Facebook sends uSocial cease and desist warning. http://www.smartcompany.com.au/web-20/20091123-facebook-sends-usocial-cease-and-desist-warning.html.
[25]
The Guardian, Investigate your MP's expenses. http://mps-expenses.guardian.co.uk/.
[26]
The Guardian, MPs' expenses -- what you've found so far. http://www.guardian.co.uk/politics/blog/2009/jun/19/mps-expenses-what-you-ve-found.
[27]
Turker Nation Message Board System. http://turkers.proboards.com/.
[28]
Viégas, F. B., Wattenberg, M., and Dave, K. 2004. Studying cooperation and conflict between authors with history flow visualizations. In Proc. CHI 2004. CHI '04. ACM, New York, NY, 575--582
[29]
von Ahn, L. and Dabbish, L. 2004. Labeling images with a computer game. In Proc. CHI 2004. ACM, New York, NY, 319--326.
[30]
von Ahn, L., Ginosar, S., Kedia, M., Liu, R., and Blum, M. 2006. Improving accessibility of the web with a computer game. In Proc. CHI 2006. ACM, New York, NY, 79--82.
[31]
Wired, The Rise of Crowdsourcing. http://www.wired.com/wired/archive/14.06/crowds.html.
[32]
Wikipedia. http://wikipedia.org
[33]
Wired, 50,000 Volunteers Join Distributed Search for Steve Fossett. http://www.wired.com/software/webservices/news/2007/09/distributed_search.
[34]
Wired, Online Fossett Searchers Ask, Was It Worth It? http://www.wired.com/techbiz/it/news/2007/11/fossett_search.
[35]
Wu, B. and Davison, B. D. 2005. Identifying link farm spam pages. In WWW '05. ACM, New York, NY, 820--829.
[36]
Yahoo! Answers Scoring System. http://answers.yahoo.com/info/scoring_system.
[37]
Yahoo! Answers. http://answers.yahoo.com/.

Cited By

View all
  • (2023)Human–Computer Interaction and Participation in Software CrowdsourcingElectronics10.3390/electronics1204093412:4(934)Online publication date: 13-Feb-2023
  • (2021)Application of Artificial Intelligence in New Product Development: Innovative Cases of CrowdsourcingExpert Clouds and Applications10.1007/978-981-16-2126-0_6(57-63)Online publication date: 16-Jul-2021
  • (2020)Complementing Solutions to Optimization Problems via Crowdsourcing on Video Game PlaysApplied Sciences10.3390/app1023841010:23(8410)Online publication date: 26-Nov-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries
October 2010
889 pages
ISBN:9781605589343
DOI:10.1145/1868914
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • Reykjavik University
  • University of Iceland

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. crowdsourcing
  2. distributed knowledge acquisition
  3. human-based computation

Qualifiers

  • Research-article

Conference

NordiCHI '10
Sponsor:

Acceptance Rates

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Human–Computer Interaction and Participation in Software CrowdsourcingElectronics10.3390/electronics1204093412:4(934)Online publication date: 13-Feb-2023
  • (2021)Application of Artificial Intelligence in New Product Development: Innovative Cases of CrowdsourcingExpert Clouds and Applications10.1007/978-981-16-2126-0_6(57-63)Online publication date: 16-Jul-2021
  • (2020)Complementing Solutions to Optimization Problems via Crowdsourcing on Video Game PlaysApplied Sciences10.3390/app1023841010:23(8410)Online publication date: 26-Nov-2020
  • (2020)Designing a Crowdsourcing System for the Elderly: A Gamified Approach to Speech CollectionExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3382999(1-9)Online publication date: 25-Apr-2020
  • (2019)Collective hybrid intelligence: towards a conceptual frameworkInternational Journal of Crowd Science10.1108/IJCS-03-2019-00123:2(198-220)Online publication date: 16-Aug-2019
  • (2019)Greenify: A Game with the Purpose of Test Data Generation for Unit TestingFundamentals of Software Engineering10.1007/978-3-030-31517-7_6(77-92)Online publication date: 1-May-2019
  • (2018)Evaluating the Usability and the Personal and Social Acceptance of a Participatory Digital Platform for Cultural HeritageHeritage10.3390/heritage20100012:1(1-26)Online publication date: 21-Dec-2018
  • (2018)Human based Computation: A solution to Computer Vision Problems2018 International Conference on Advances in Computing, Communication Control and Networking (ICACCCN)10.1109/ICACCCN.2018.8748543(630-633)Online publication date: Oct-2018
  • (2018)Assessing Word Difficulty for Quiz-Like GameSemantic Keyword-Based Search on Structured Data Sources10.1007/978-3-319-74497-1_7(70-79)Online publication date: 8-Feb-2018
  • (2017)Game with a Purpose for Verification of Mappings Between Wikipedia and WordNetSemantic Keyword-Based Search on Structured Data Sources10.1007/978-3-319-53640-8_14(159-170)Online publication date: 15-Feb-2017
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media