Pricing mechanisms for crowdsourcing markets

ABSTRACT

Every day millions of crowdsourcing tasks are performed in exchange for payments. Despite the important role pricing plays in crowdsourcing campaigns and the complexity of the market, most platforms do not provide requesters appropriate tools for effective pricing and allocation of tasks.

In this paper, we introduce a framework for designing mechanisms with provable guarantees in crowdsourcing markets. The framework enables automating the process of pricing and allocation of tasks for requesters in complex markets like Amazon's Mechanical Turk where workers arrive in an online fashion and requesters face budget constraints and task completion deadlines. We present constant-competitive incentive compatible mechanisms for maximizing the number of tasks under a budget, and for minimizing payments given a fixed number of tasks to complete. To demonstrate the effectiveness of this framework we created a platform that enables applying pricing mechanisms in markets like Mechanical Turk. The platform allows us to show that the mechanisms we present here work well in practice, as well as to give experimental evidence to workers' strategic behavior in absence of appropriate incentive schemes.

References

1. Amazon mturk https://www.mturk.com.
2. Clickworker. https://www.crowdflower.com.
3. CloudCrowd. https://www.cloudcrowd.com.
4. Crowdflower. https://www.crowdflower.com.
5. duolingo. http://duolingo.com/.
6. Microtask. http://microtask.com.
7. MobileWorks. https://www.mobileworks.com.
8. oDesk. https://www.odesk.com.
9. L. v. Ahn. Games with a purpose. Computer, 39(6):92--94, June 2006.
10. A. Archer and É. Tardos. Frugal path mechanisms. In SODA, pages 991--999, 2002.
11. M. Babaioff, N. Immorlica, D. Kempe, and R. Kleinberg. A knapsack secretary problem with applications. In APPROX-RANDOM, pages 16--28, 2007.
12. M. Babaioff, N. Immorlica, D. Kempe, and R. Kleinberg. Online auctions and generalized secretary problems. SIGecom Exchanges, 7(2), 2008.
13. D. F. Bacon, D. C. Parkes, Y. Chen, M. Rao, I. A. Kash, and M. Sridharan. Predicting your own effort. In AAMAS, pages 695--702, 2012.
14. A. Badanidiyuru, R. Kleinberg, and Y. Singer. Learning on a budget: Posted price mechanisms for online procurement. In ACM Conference on Electronic Commerce, pages 128--145, 2012.
15. X. Bei, N. Chen, N. Gravin, and P. Lu. Budget feasible mechanism design via random sampling. CoRR, abs/1107.2994, 2011.
16. N. Chen, N. Gravin, and P. Lu. On the approximability of budget feasible mechanisms. 2011.
17. S. Cooper, F. Khatib, I. Makedon, H. Lü, J. Barbero, D. Baker, J. Fogarty, Z. Popovic, and F. players. Analysis of social gameplay macros in the foldit cookbook. In M. Cavazza, K. Isbister, and C. Rich, editors, FDG, pages 9--14. ACM, 2011.
18. P. Dandekar, N. Fawaz, and S. Ioannidis. Privacy auctions for recommender systems. WINE, 2012.
19. S. Dobzinski, C. Papadimitriou, and Y. Singer. Mechanisms for complement-free procurement. In EC, 2011.
20. E. B. Dynkin. The optimum choice of the instant for stopping a Markov process. Soviet Math. Dokl, 4, 1963.
21. E. J. Friedman and D. C. Parkes. Pricing wifi at starbucks: Issues in online mechanism design. In ACM Conference on Electronic Commerce, pages 240--241, 2003.
22. M. T. Hajiaghayi. Online auctions with re-usable goods. In Proceedings of the 6th ACM conference on Electronic commerce, EC '05, pages 165--174, New York, NY, USA, 2005. ACM.
23. M. T. Hajiaghayi, R. D. Kleinberg, and D. C. Parkes. Adaptive limited-supply online auctions. In ACM Conference on Electronic Commerce, pages 71--80, 2004.
24. C.-J. Ho and J. W. Vaughan. Online task assignment in crowdsourcing markets. In AAAI, 2012.
25. J. J. Horton and L. B. Chilton. The labor economics of paid crowdsourcing. EC '10, pages 209--218, 2010.
26. J. J. Horton and R. J. Zeckhauser. Algorithmic wage negotiations: Applications to paid crowsourcing. CrowdConf, 2010.
27. S. Jain, Y. Chen, and D. C. Parkes. Designing incentives for online question and answer forums. In Proceedings of the 10th ACM conference on Electronic commerce, EC '09, pages 129--138, New York, NY, USA, 2009. ACM.
28. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953--1961, 2011.
29. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI '08, pages 453--456, New York, NY, USA, 2008. ACM.
30. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. J. Mach. Learn. Res., 11:1297--1322, Aug. 2010.
31. V. S. Sheng, F. J. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD, pages 614--622, 2008.
32. Y. Singer. Budget feasible mechanisms. In FOCS, pages 765--774, 2010.
33. Y. Singer. How to win friends and influence people, truthfully: Influence maximization mechanisms for social networks. In WSDM, pages 733--742, 2012.
34. L. Tran-Thanh, S. Stein, A. Rogers, and N. R. Jennings. Efficient crowdsourcing of unknown experts using multi-armed bandits. In L. D. Raedt, C. Bessière, D. Dubois, P. Doherty, P. Frasconi, F. Heintz, and P. J. F. Lucas, editors, ECAI, volume 242 of Frontiers in Artificial Intelligence and Applications, pages 768--773. IOS Press, 2012.
35. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, pages 2424--2432, 2010.
36. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems, 22:2035--2043, 2009.

Index Terms

• Information systems