Daniel W. Barowy
Charlie Curtsinger
Emery D. Berger
Andrew McGregor
Abstract
Humans can perform many tasks with ease that remain difficult or impossible for computers. Crowdsourcing platforms like Amazon's Mechanical Turk make it possible to harness human-based computational power at an unprecedented scale. However, their utility as a general-purpose computational platform remains limited. The lack of complete automation makes it difficult to orchestrate complex or interrelated tasks. Scheduling more human workers to reduce latency costs real money, and jobs must be monitored and rescheduled when workers fail to complete their tasks. Furthermore, it is often difficult to predict the length of time and payment that should be budgeted for a given task. Finally, the results of human-based computations are not necessarily reliable, both because human skills and accuracy vary widely, and because workers have a financial incentive to minimize their effort.
This paper introduces AutoMan, the first fully automatic crowdprogramming system. AutoMan integrates human-based computations into a standard programming language as ordinary function calls, which can be intermixed freely with traditional functions. This abstraction lets AutoMan programmers focus on their programming logic. An AutoMan program specifies a confidence level for the overall computation and a budget. The AutoMan runtime system then transparently manages all details necessary for scheduling, pricing, and quality control. AutoMan automatically schedules human tasks for each computation until it achieves the desired confidence level; monitors, reprices, and restarts human tasks as necessary; and maximizes parallelism across human workers while staying under budget.
- S. Ahmad, A. Battle, Z. Malkani, and S. Kamvar. The Jabberwocky Programming Environment for Structured Social Computing. In UIST, pp. 53--64, 2011. Google Scholar
Digital Library
- Amazon. Mechanical Turk. http://www.mturk.com.Google Scholar
- C. N. Anagnostopoulos. MediaLab LPR Database. http://www.medialab.ntua.gr/research/LPRdatabase.html.Google Scholar
- M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: A Word Processor with a Crowd Inside. In UIST, pp. 313--322, 2010. Google ScholarDigital Library
- H. Chafi, Z. DeVito, A. Moors, T. Rompf, A. K. Sujeeth, P. Han-rahan, M. Odersky, and K. Olukotun. Language Virtualization for Heterogeneous Parallel Computing. In Onward!, 2010. Google ScholarDigital Library
- S. Cooper, F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, A. Leaver-Fay, D. Baker, Z. Popovi, and F. Players. Predicting Protein Structures With a Multiplayer Online Game. Nature, 466(7307):756--760, 2010.Google ScholarCross Ref
- A. DasGupta. Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics. Springer, 1st edition, 2011. Google ScholarDigital Library
- J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, pp. 137--150, 2004. Google ScholarDigital Library
- J. R. Douceur. The Sybil Attack. In IPTPS, pp. 251--260, 2001. Google ScholarDigital Library
- S. Dow, A. Kulkarni, B. Bunge, T. Nguyen, S. Klemmer, and B. Hartmann. Shepherding the Crowd: Managing and Providing Feedback to Crowd Workers. In CHI, pp. 1669--1674, 2011. Google ScholarDigital Library
- S. Due, M. Ibrahim, M. Shehata, and W. Badawy. Automatic License Plate Recognition (ALPR): A State of the Art Re-view. IEEE Transactions on Circuits and Systems for Video Technology, 2012.Google Scholar
- W. Feller. An Introduction to Probability Theory and Applica-tions, volume 1. John Wiley & Sons Publishers, 3rd edition, 1968.Google Scholar
- M. Fogus. BAYSICK---a DSL for Scala implementing a subset of BASIC. https://github.com/fogus/baysick, March 2009.Google Scholar
- M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. CrowdDB: Answering Queries with Crowdsourcing. In SIGMOD, pp. 61--72, 2011. Google ScholarDigital Library
- M. Gentile. UI widget images come from the "Vector UI Kit" package. http://www.icondeposit.com/design:9, Mar 2012. Creative Commons c 2012 Matt Gentile.Google Scholar
- J. Howe. The Rise of Crowdsourcing. Wired Magazine, 14(6):176--178, 2006.Google Scholar
- P. G. Ipeirotis. Demographics of Mechanical Turk. Tech. Rep. Working Paper CeDER-10-01, NYU Center for Digital Econ-omy Research, 2010.Google Scholar
- P. G. Ipeirotis, F. Provost, and J. Wang. Quality Management on Amazon Mechanical Turk. In HCOMP, pp. 64--67, 2010. Google ScholarDigital Library
- A. Kittur, B. Smus, and R. E. Kraut. CrowdForge: Crowdsourcing Complex Work. Tech. Rep. CMU-HCII-11-100, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University, February 2011.Google Scholar
- A. P. Kulkarni, M. Can, and B. Hartmann. Turkomatic: Auto-matic Recursive Task and Workflow Design for. Mechanical Turk. In CHI, pp. 2053--2058, 2011. Google ScholarDigital Library
- K. Land, A. Slosar, C. Lintott, D. Andreescu, S. Bamford, P. Murray, R. Nichol, M. J. Raddick, K. Schawinski, A. Szalay, D. Thomas, and J. Vandenberg. Galaxy Zoo: the Large-Scale Spin Statistics of Spiral Galaxies in the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 388:1686--1692, Aug. 2008.Google ScholarCross Ref
- G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. TurKit: Human Computation Algorithms onMechanical Turk. In UIST, pp. 57--66, 2010. Google ScholarDigital Library
- M. Marge, S. Banerjee, and A. Rudnicky. Using the Amazon Mechanical Turk for Transcription of Spoken Language. In ICASSP, pp. 5270--5273, Mar 2010.Google ScholarCross Ref
- A. McCallum, K. Schultz, and S. Singh. FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs. In NIPS, 2009.Google ScholarDigital Library
- M. Odersky and M. Zenger. Scalable Component Abstractions. In OOPSLA, pp. 41--57, 2005. Google ScholarDigital Library
- D. Oleson, V. Hester, A. Sorokin, G. Laughlin, J. Le, and L. Biewald. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. In HCOMP, pp. 43--48, 2011.Google Scholar
- D. Parikh and L. Zitnick. Human-Debugging of Machines. In NIPS CSS, 2011.Google Scholar
- D. Shahaf and E. Amir. Towards a Theory of AI Completeness. In Commonsense, 2007.Google Scholar
- D. Tamir, P. Kanth, and P. Ipeirotis. Mechanical Turk: Now With 40.92% Spam., Dec 2010.Google Scholar
- L. von Ahn, B. Maurer, C. Mcmillen, D. Abraham, and M. Blum. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science, 321(5895):1465--1468, Aug 2008.Google ScholarCross Ref
- T. Yan, V. Kumar, and D. Ganesan. CrowdSearch: Exploiting Crowds for Accurate Real-Time Image Search on Mobile Phones. In MobiSys, pp. 77--90, 2010. Google ScholarDigital Library
Index Terms
- AutoMan: a platform for integrating human-based and digital computation
Recommendations
AutoMan: a platform for integrating human-based and digital computation
OOPSLA '12: Proceedings of the ACM international conference on Object oriented programming systems languages and applicationsHumans can perform many tasks with ease that remain difficult or impossible for computers. Crowdsourcing platforms like Amazon's Mechanical Turk make it possible to harness human-based computational power at an unprecedented scale. However, their ...
iCrowd: An Adaptive Crowdsourcing Framework
SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of DataCrowdsourcing is widely accepted as a means for resolving tasks that machines are not good at. Unfortunately, Crowdsourcing may yield relatively low-quality results if there is no proper quality control. Although previous studies attempt to eliminate "...
Leveraging non-expert crowdsourcing workers for improper task detection in crowdsourcing marketplaces
Controlling the quality of tasks, i.e., propriety of posted jobs, is a major challenge in crowdsourcing marketplaces. Most existing crowdsourcing services prohibit requesters from posting illegal or objectionable tasks. Operators in marketplaces have to ...
Comments