Investigating Cross-Device Interaction between a Handheld Device and a Large Display

ABSTRACT

There is a growing interest in HCI research to explore cross-device interaction, giving rise to an interest in different approaches facilitating interaction between handheld devices and large displays. Contributing to this, we have investigated the use of four existing approaches combining touch and mid-air gestures, pinching, swiping, swinging and flicking. We look specifically at their relative efficiency, effectiveness and accuracy in bi-directional interaction between a smartphone and large display in a point-click context. We report findings from two user studies, which show that swiping is both most effective, fastest and most accurate, closely followed by swinging. What these two approaches have in common is the ability to keep the pointer steady on the large display, unaffected by concurrent gestures or body movements used to complete the interaction, suggesting that this is an important factor for designing effective cross-device interaction with large displays.

References

1. Michel Beaudouin-Lafon, Stephane Huot, Mathieu Nancel, Wendy Mackay, Emmanuel Pietriga, Romain Primet, Julie Wagner, Olivier Chapuis, Clement Pillias, James Eagan, Tony Gjerlufsen, Clemens Klokmose. 2012. Multisurface interaction in the wild room. Computer 45, no. 4 (2012): 48--56.
2. Hrvoje Benko and Andrew D. Wilson. 2010. Multipoint Interactions with Immersive Omnidirectional Visualizations in a Dome. In ACM International Conference on Interactive Tabletops and Surfaces (ITS'10). ACM, NY, NY, USA, 19--28.
3. Richard A. Bolt. 1980. "Put-that-there": Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques (SIGGRAPH '80). ACM, NY, NY, USA, 262--270.
4. Sebastian Boring, Marko Jurmu, and Andreas Butz. 2009. Scroll, Tilt or Move It: Using Mobile Phones to Continuously Control Pointers on Large Public Displays. In Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7 (OZCHI '09). ACM, NY, NY, USA, 161--168.
5. Andrew Bragdon, Rob DeLine, Ken Hinckley, and Meredith Ringel Morris. 2011. Code Space: Touch + Air Gesture Hybrid Interactions for Supporting Developer Meetings. In Proceedings of the International Conference on Interactive Tabletops and Surfaces (ITS '11). ACM, NY, NY, USA, 212--221.
6. Raimund Dachselt and Robert Buchholz. 2009. Natural throw and tilt interaction between mobile phones and distant displays. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, NY, NY, USA, 3253--3258.
7. L. Gallo and A. Minutolo. 2012. Design and Comparative Evaluation of Smoothed Pointing: A Velocity-oriented Remote Pointing Enhancement Technique. Int. J. Hum.-Comput. Stud. 70, 4 (April 2012), 287--300.
8. Tony Gjerlufsen, Clemens Nylandsted Klokmose, James Eagan, Clément Pillias, and Michel BeaudouinLafon. 2011. Shared substance: developing flexible multi-surface applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 3383--3392.
9. Luke Hespanhol, Martin Tomitsch, Kazjon Grace, Anthony Collins, and Judy Kay. 2012. Investigating Intuitiveness and Effectiveness of Gestures for Free Spatial Interaction with Large Displays. In Proceedings of the 2012 International Symposium on Pervasive Displays (PerDis '12). ACM, NY, NY, USA.
10. Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 1063--1072.
11. Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems CHI '15, ACM Press, 1247--1256.
12. Steven Houben, Paolo Tell, and Jakob E. Bardram. 2014. ActivitySpace: Managing Device Ecologies in an Activity-Centric Configuration Space. In Proceedings of International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, NY, NY, USA, 119--128.
13. David C. Hoaglin and Boris Iglewicz. 1987. FineTuning Some Resistant Rules for Outlier Labeling. J. Amer. Statist. Assoc. 82, 400 (1987), 1147--1149. http://www.jstor.org/stable/2289392
14. Kaori Ikematsu and Itiro Siio. 2015. Memory Stones: An Intuitive Information Transfer Technique Between Multi-touch Computers. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (HotMobile '15). ACM, NY, NY, USA, 3--8.
15. Mikkel R. Jakobsen, Yvonne Jansen, Sebastian Boring, and Kasper Hornbæk. 2015. Should I Stay or Should I Go? Selecting Between Touch and mid-air gestures for large-display interaction. In Proceedings of the 15th IFIP TC 13 International Conference on HumanComputer Interaction (INTERACT 2015), 455--473.
16. Tero Jokela, Jarno Ojala, Guido Grassel, Petri Piippo, and Thomas Olsson. 2015. A Comparison of Methods to Move Visual Objects Between Personal Mobile Devices in Different Contexts of Use. In Proceedings of the 17th International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, NY, NY, USA, 172--181.
17. Tero Jokela, Jarno Ojala, and Thomas Olsson. 2015. A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 3903--3912.
18. Andrés Lucero, Jussi Holopainen, and Tero Jokela. 2012. MobiComics: Collaborative Use of Mobile Phones and Large Displays for Public Expression. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '12). ACM, NY, NY, USA, 383--392.
19. Anders Markussen, Sebastian Boring, Mikkel R. Jakobsen, and Kasper Hornbæk. 2016. Off-Limits: Interacting Beyond the Boundaries of Large Displays. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 5862--5873.
20. Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: A Mid-air Wordgesture Keyboard. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 1073--1082.
21. Nicolai Marquardt, Ken Hinckley, and Saul Greenberg. 2012. Cross-device interaction via micro-mobility and f-formations. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, NY, NY, USA, 13--22.
22. Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (ITS '12). ACM, NY, NY, USA, 31--40.
23. Sven Mayer, Katrin Wolf, Stefan Schneegass, and Niels Henze. 2015. Modeling Distant Pointing for Compensating Systematic Displacements. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 4165--4168.
24. Meredith R. Morris, Andreea Danielescu, Steven Drucker, Danyel Fisher, Bongshin Lee, m. c. schraefel, and Jacob O. Wobbrock. 2014. Reducing legacy bias in gesture elicitation studies. interactions 21, 3 (May 2014), 40--45.
25. Brad A. Myers, Rishi Bhatnagar, Jeffrey Nichols, Choon Hong Peck, Dave Kong, Robert Miller, and A. Chris Long. 2002. Interacting at a Distance: Measuring the Performance of Laser Pointers and Other Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, NY, NY, USA, 33--40.
26. Miguel A. Nacenta, Carl Gutwin, Dzmitry Aliakseyeu, Sriram Subramanian. 2009. There and Back again: Cross-Display Object Movement in Multi-Display Environments. Journal of Human-Computer Interaction, vol. 24 no. 1, 170--229.
27. Mathieu Nancel, Olivier Chapuis, Emmanuel Pietriga, Xing-Dong Yang, Pourang P. Irani, and Michel Beaudouin-Lafon. 2013. High-precision Pointing on Large Wall Displays Using Small Handheld Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 831--840.
28. Roman Rädle, Hans-Christian Jetter, Mario Schreiner, Zhihao Lu, Harald Reiterer, and Yvonne Rogers. 2015. Spatially-aware or Spatially-agnostic? Elicitation and Evaluation of User-Defined Cross-Device Interactions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 3913--3922.
29. Roman Rädle, Hans-Christian Jetter, Nicolai Marquardt, Harald Reiterer, and Yvonne Rogers. 2014. HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, NY, NY, USA, 45--54.
30. Dimitrios Raptis, Jesper Kjeldskov, and Mikael B. Skov. 2016. Continuity in Multi-Device Interaction: An Online Study. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction: GameChanging Design (NordiCHI '16). ACM, NY, NY, USA.
31. Umar Rashid, Jarmo Kauko, Jonna Häkkilä, and Aaron Quigley. 2011. Proximal and Distal Selection of Widgets: Designing Distributed UI for Mobile Interaction with Large Display. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, NY, NY, USA, 495--498.
32. Jun Rekimoto. 1997. Pick-and-drop: A Direct Manipulation Technique for Multiple Computer Environments. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST '97). ACM, NY, NY, USA, 31--39.
33. Jürgen Scheible, Timo Ojala, and Paul Coulton. 2008. MobiToss: A Novel Gesture Based Interface for Creating and Sharing Mobile Multimedia Art on Large Public Displays. In Proceedings of the 16th ACM International Conference on Multimedia (MM '08). ACM, NY, NY, USA, 957--960.
34. Julian Seifert, Andreas Bayer, and Enrico Rukzio. 2013. PointerPhone: Using Mobile Phones for Direct Pointing Interactions with Remote Displays. In Proceedings of the 14th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT 2013). Springer Berlin Heidelberg, 18--35.
35. Mikael B. Skov, Jesper Kjeldskov, Jeni Paay, Heidi P. Jensen, and Marius P. Olsen. 2015. Investigating Cross-Device Interaction Techniques. Proceedings of the Australian Conference on Human-Computer Interaction, OzCHI '15, ACM Press, 446--454.
36. Henrik Sørensen, Dimitrios Raptis, Jesper Kjeldskov, and Mikael B. Skov. 2014. The 4C framework: principles of interaction in digital ecosystems. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, NY, NY, USA, 87--97.
37. Daniel Vogel and Ravin Balakrishnan. 2005. Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, NY, NY, USA, 33--42.
38. Robert Walter, Gilles Bailly, Nina Valkanova, and Jörg Müller. 2014. Cuenesics: using mid-air gestures to select items on interactive public displays. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14). ACM, NY, NY, USA, 299--308.
39. Paweł Wozniak, Lars Lischke, Benjamin Schmidt, Shengdong Zhao, and Morten Fjeld. 2014. Thaddeus: A Dual Device Interaction Space for Exploring Information Visualisation. Proceedings of the 8th Nordic Conference on Human-Computer Interaction NordiCHI '14, ACM Press, 41--50.
40. Koji Yatani, Koiti Tamura, Keiichi Hiroki, Masanori Sugimoto, and Hiromichi Hashizume. 2005. Toss-it: Intuitive Information Transfer Techniques for Mobile Devices. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, NY, NY, USA, 1881--1884.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)