Differences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data

ABSTRACT

Research on the viability of using crowdsourcing for HCI performance experiments has concluded that online results are similar to those achieved in the lab---at least for desktop interactions. However, mobile devices, the most popular form of online access today, may be more problematic due to variability in the user's posture and in movement of the device. To assess this possibility, we conducted two experiments with 30 lab-based and 303 crowdsourced participants using basic mouse and touchscreen tasks. Our findings show that: (1) separately analyzing the crowd and lab data yields different study conclusions-touchscreen input was significantly less error prone than mouse input in the lab but more error prone online; (2) age-matched crowdsourced participants were significantly faster and less accurate than their lab-based counterparts, contrasting past work; (3) variability in mobile device movement and orientation increased as experimenter control decreased--a potential factor affecting the touchscreen error differences. This study cautions against assuming that crowdsourced data for performance experiments will directly reflect lab-based data, particularly for mobile devices.

References

1. Mohammad Allahbakhsh, Boualem Benatallah, A Ignjatovic, H R Motahari-Nezhad, E Bertino, and S Dustdar. 2013. Quality control in crowdsourcing systems. IEEE Internet Computing 17, 2: 76--81.
2. Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk. Political Analysis 20, 3: 351--368.
3. Xiaojun Bi, Yang Li, and Shumin Zhai. 2013. FFitts Law: Modeling Finger Touch with Fitts' Law. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1363--1372.
4. Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6, 1: 3--5.
5. A. Cockburn, D. Ahlström, and C. Gutwin. 2012. Understanding performance in touch selections: Tap, drag and radial pointing drag with finger, stylus and mouse. International Journal of Human-Computer Studies 70, 3: 218--233.
6. Jay L Devore. 2015. Probability and Statistics for Engineering and the Sciences. Cengage Learning.
7. Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, 1013--1022.
8. Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval 16, 2: 121--137.
9. Leah Findlater, Jon E. Froehlich, Kays Fattal, Jacob O. Wobbrock, and Tanya Dastyar. 2013. Age-related differences in performance with touchscreens compared to traditional mouse input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 343--346.
10. Tovi Grossman and Ravin Balakrishnan. 2005. The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area. In Proceedings of the SIGCHI conference on Human factors in computing systems, 281--290.
11. Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 203--212.
12. Niels Henze, Enrico Rukzio, and Susanne Boll. 2011. 100,000,000 taps: analysis and improvement of touch performance in the large. In Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI'11), 133--142.
13. John J Horton, David G Rand, and Richard J Zeckhauser. 2011. The online laboratory: Conducting experiments in a real labor market. Experimental Economics 14, 3: 399--425.
14. International Organization for Standardization. 2002. Ergonomic requirements for office work with visual display terminals (VDTs) Requirements for nonkeyboard input devices.
15. Panagiotis G Ipeirotis. 2010. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17, 2: 16--21.
16. Panos Ipeirotis. 2016. MTurk Tracker. Retrieved August 28, 2016 from http://demographics.mturktracker.com/
17. Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems, 453--456.
18. Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, et al. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, 1301--1318.
19. Steven Komarov, Katharina Reinecke, and Krzysztof Z Gajos. 2013. Crowdsourcing Performance Evaluations of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 207--216.
20. I. Scott MacKenzie and Poika Isokoski. 2008. Fitts' throughput and the speed-accuracy tradeoff. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'08), 1633--1636.
21. Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1: 1--23.
22. Gabriele Paolacci and Jesse Chandler. 2014. Inside the turk understanding mechanical turk as a participant pool. Current Directions in Psychological Science 23, 3: 184--188.
23. Alexander J Quinn and Benjamin B Bederson. 2011. Human computation: a survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'11), 1403--1412.
24. Joel Ross, Lilly Irani, M Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI'10 extended abstracts on Human factors in computing systems, 2863--2872.
25. Dmitry Rudchenko, Tim Paek, and Eric Badger. 2011. Text Text Revolution: a game that improves text entry on mobile touchscreen keyboards. In Proceedings of Pervasive 2011, 206--213.
26. M. Six Silberman, Kristy Milland, Rochelle LaPlante, Joel Ross, and Lilly Irani. 2015. Stop citing Ross et al. 2010, "Who are the crowdworkers"? Retrieved August 28, 2016 from https://medium.com/@silberman/stopciting-ross-et-al-2010-who-are-the-crowdworkersb3b9b1e8d300#.hv8r2sjcy
27. R. William Soukoreff and I. Scott MacKenzie. 2004. Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts' law research in HCI. International Journal of Human-Computer Studies 61, 6: 751--789.
28. Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 143--146.
29. Jacob O Wobbrock, Kristen Shinohara, and Alex Jansen. 2011. The effects of task dimensionality, endpoint deviation, throughput calculation, and experiment design on pointing measures and models. Proceedings of the 2011 annual conference on Human factors in computing systems CHI '11: 1639.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)