Soylent: a word processor with a crowd inside

ABSTRACT

This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.

References

1. }}Bernstein, M., Marcus, A., Karger, D. R., and Miller, R. C. Enhancing Directed Content Sharing on the Web. CHI '10, ACM Press (2010).
2. }}Bernstein, M., Tan, D., Smith, G., Czerwinski, M., et al. Collabio: A Game for Annotating People within Social Networks. UIST '09, ACM Press (2009), 177--180.
3. }}Bigham, J. P., Jayant, C., Ji, H., Little, G., et al. VizWiz: Nearly Real-time Answers to Visual Questions. UIST '10, ACM Press (2010).
4. }}Clarke, J. and Lapata, M. Models for sentence compression: a comparison across domains, training requirements and evaluation measures. ACL '06, Association for Computational Linguistics (2006).
5. }}Cohn, T. and Lapata, M. Sentence compression beyond word deletion. COLING '08, (2008).
6. }}Cypher, A. Watch What I Do. MIT Press, Cambridge, MA, 1993.
7. }}Dourish, P. and Bellotti, V. Awareness and coordination in shared workspaces. CSCW '92, ACM Press (1992).
8. }}Evans, B. and Chi, E. Towards a model of understanding social search. CSCW '08, ACM Press (2008).
9. }}Hartmann, B., MacDougall, D., Brandt, J., and Klemmer, S. What Would Other Programmers Do? Suggesting Solutions to Error Messages. CHI '10, ACM Press (2010).
10. }}Heer, J. and Bostock, M. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. CHI '10, ACM Press (2010).
11. }}Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. CHI '08, ACM Press (2008).
12. }}Knight, K. and Marcu, D. Summarization beyond sentence extraction: a probabilistic approach to sentence compression. Artificial Intelligence 139, 1 (2002).
13. }}Krieger, M., Stark, E. M., and Klemmer, S. R. Coordinating tasks on the commons: designing for personal goals, expertise and serendipity. CHI '09, ACM Press (2009).
14. }}Kukich, K. Techniques for automatically correcting words in text. ACM Computing Surveys (CSUR) 24, 4 (1992).
15. }}Lieberman, H. Your Wish is My Command. Morgan Kaufmann, San Francisco, 2001.
16. }}Little, G., Chilton, L., Goldman, M., and Miller, R. C. TurKit: Human Computation Algorithms on Mechanical Turk. UIST '10, ACM Press (2010).
17. }}Little, G., Lau, T. A., Cypher, A., Lin, J., et al. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. CHI '07, (2007).
18. }}Marcu, D. The Theory and Practice of Discourse Parsing and Summarization. MIT Press, 2000.
19. }}Mason, W. and Watts, D. Financial Incentives and the Performance of Crowds. ACM SIGKDD Workshop on Human Computation, ACM Press (2009).
20. }}Miller, R. and Myers, B. Interactive simultaneous editing of multiple text regions. USENIX '01, (2001).
21. }}Quinn, A. J. and Bederson, B. B. A Taxonomy of Distributed Human Computation.
22. }}Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., et al. Who Are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. alt.chi '10, ACM Press.
23. }}Sala, M., Partridge, K., Jacobson, L., and Begole, J. An Exploration into Activity-Informed Physical Advertising Using PEST. Pervasive '07, Springer Berlin Heidelberg (2007).
24. }}Simon, I., Morris, D., and Basu, S. MySong: automatic accompaniment generation for vocal melodies. Proc. CHI '08, ACM Press (2008).
25. }}Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. ACL '08, (2008).
26. }}Sorokin, A. and Forsyth, D. Utility data annotation with Amazon Mechanical Turk. CVPR '08, (2008).
27. }}von Ahn, L. and Dabbish, L. Labeling images with a computer game. CHI '04, ACM Press (2004).

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)