Should I Interrupt or Not?: Understanding Interruptions in Head-Mounted Display Settings

ABSTRACT

Head-mounted displays (HMDs) are being used for VR and AR applications and increasingly permeate our everyday life. At the same time, a detailed understanding of interruptions in settings where people wearing an HMD (HMD user) and people not wearing an HMD (bystander) is missing. We investigate (a) whether bystanders are capable of identifying when HMD users switch tasks by observing their gestures, and hence exploit opportune moments for interruptions, and (b) which strategies bystanders employ. In a lab study (N=64) we found that bystanders are able to successfully identify both task switches (83%) and tasks (77%) within only a few seconds of the task switch. Furthermore, we identified interruption strategies of bystanders. From our results we derive implications meant to support designers and practitioners in building HMD applications that are used in a co-located collaborative setting.

References

1. 2016. (May 2016). https://developers.googleblog.com/2016/05/daydream-labs-exploring-and-sharing-vrs.html
2. Piotr D. Adamczyk and Brian P. Bailey. 2004. If Not Now, when?: The Effects of Interruption at Different Moments Within Task Execution. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 271--278.
3. Ilhan Aslan, Andreas Uhl, Alexander Meschtscherjakov, and Manfred Tscheligi. 2014. Mid-air authentication gestures: an exploration of authentication based on palm and finger motions. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, 311--318.
4. Daniel Avrahami, James Fogarty, and Scott E. Hudson. 2007. Biases in Human Estimation of Interruptibility: Effects and Implications for Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 50--60.
5. Abraham Bernstein, Peter Vorburger, and Patrice Egger. 2008. A scenario-based approach for direct interruptability prediction on wearable devices. International Journal of Pervasive Computing and Communications 3, 4 (2008), 426--438.
6. Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. 2001. The MagicBook: a transitional AR interface. Computers & Graphics 25, 5 (2001), 745--753.
7. Tobias Blum, Valerie Kleeberger, Christoph Bichlmeier, and Nassir Navab. 2012. mirracle: An augmented reality magic mirror system for anatomy education. In Virtual Reality Short Papers and Posters (VRW), 2012 IEEE. IEEE, 115--116.
8. Miss Bhavana Borkar, Miss Shiba Sheikh, and PD Kaware. 2016. 4d password mechanism. Imperial Journal of Interdisciplinary Research 2, 5 (2016).
9. Daniel BROSCHART and Peter ZEILE. ARchitecture--Augmented Reality in Architecture and Urban Planning. (????).
10. Erica A Cartmill, Özlem Ece Demir, and Susan Goldin-Meadow. 2012. 14 Studying Gesture. (2012).
11. Yun Suk Chang, Benjamin Nuernberger, Bo Luan, Tobias Höllerer, and John O'Donovan. 2017. Gesture-based augmented reality annotation. In Virtual Reality (VR), 2017 IEEE. IEEE, 469--470.
12. Jian Chen, Pardha S Pyla, and Doug A Bowman. 2004. Testbed evaluation of navigation and text display techniques in an information-rich virtual environment. In Virtual Reality, 2004. Proceedings. IEEE. IEEE, 181--289.
13. Laura Dabbish and Robert E. Kraut. 2004. Controlling Interruptions: Awareness Displays and Social Motivation for Coordination. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 182--191.
14. Tamara Denning, Zakariya Dehlawi, and Tadayoshi Kohno. 2014. In situ with bystanders of augmented reality glasses: Perspectives on recording and privacy-mediating technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2377--2386.
15. Isabelle Doll, Karsten Rexroth, and Petra Von Both. 2017. Augmented Reality in der Stadt: Neue Potenziale durch die Entwicklung einer Lichtmarker. Wireless Commun. 6 (2017), 65--75.
16. Joshua A Fisher, Amit Garg, Karan Pratap Singh, and Wesley Wang. 2017. Designing intentional impossible spaces in virtual reality narratives: A case study. In Virtual Reality (VR), 2017 IEEE. IEEE, 379--380.
17. James Fogarty and Scott E. Hudson. 2007. Toolkit Support for Developing and Deploying Sensor-based Statistical Models of Human Situations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 135--144.
18. James Fogarty, Scott E. Hudson, and Jennifer Lai. 2004. Examining the Robustness of Sensor-based Statistical Models of Human Interruptibility. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 207--214.
19. Markus Funk, Mareike Kritzler, and Florian Michahelles. 2017. HoloCollab: a shared virtual platform for physical assembly training using spatially-aware head-mounted displays. In Proceedings of the Seventh International Conference on the Internet of Things. ACM, 19.
20. Nirit Gavish, Teresa Gutiérrez, Sabine Webel, Jorge Rodr'iguez, Matteo Peveri, Uli Bockholt, and Franco Tecchia. 2015. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23, 6 (2015), 778--798.
21. Ceenu George, Manuel Demmler, and Heinrich Hussmann. 2018. Intelligent Interruptions for IVR: Investigating the Interplay between Presence, Workload and Attention. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW511.
22. Ceenu Goerge, Mohamed Khamis, Emanuel von Zezschwitz, Marinus Burger, Henri Schmidt, Florian Alt, and Heinrich Hussmann. 2017. Seamless and Secure VR: Adapting and Evaluating Established Authentication Systems for Virtual Reality. In Proceedings of the Network and Distributed System Security Symposium (NDSS 2017) (USEC '17). Internet Society.
23. Jean Ann Graham and Michael Argyle. 1975. A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology 10, 1 (1975), 57--67.
24. Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality Between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4021--4033.
25. Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, and Enrico Rukzio. 2018. FaceDisplay: Towards Asymmetric Multi-User Interaction for Nomadic Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 54, 13 pages.
26. Daniel A Guttentag. 2010. Virtual reality: Applications and implications for tourism. Tourism Management 31, 5 (2010), 637--651.
27. Mohammad S Hasan and Hongnian Yu. 2017. Innovative developments in HCI and future trends. International Journal of Automation and Computing 14, 1 (2017), 10--20.
28. Mary Jo Hatch. 1987. Physical barriers, task characteristics, and interaction activity in research and development firms. Administrative Science Quarterly (1987), 387--399.
29. Christian Heath and Paul Luff. 1993. Disembodied conduct: Interactional asymmetries in video-mediated communication. Technology in working order: Studies of work, interaction, and technology (1993), 35--54.
30. Chih-Yuan Ho, Mark I Nikolic, Molly J Waters, and Nadine B Sarter. 2004. Not now! Supporting interruption management by indicating the modality and urgency of pending tasks. Human Factors 46, 3 (2004), 399--409.
31. Joyce Ho and Stephen S Intille. 2005. Using context-aware computing to reduce the perceived burden of interruptions from mobile devices. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 909--918.
32. Paul Hockett and Tim Ingleby. 2016. Augmented reality with HoloLens: Experiential architectures embedded in the real world. arXiv preprint arXiv:1610.04281 (2016).
33. Eric Horvitz and Johnson Apacible. 2003. Learning and Reasoning About Interruption. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI '03). ACM, New York, NY, USA, 20--27.
34. Yichao Huang, Xiaorui Liu, Xin Zhang, and Lianwen Jin. 2016. A pointing gesture based egocentric interaction system: Dataset, approach and application. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 16--23.
35. James M. Hudson, Jim Christensen, Wendy A. Kellogg, and Thomas Erickson. 2002. "I'D Be Overwhelmed, but It's Just One More Thing to Do": Availability and Interruption in Research Management. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 97--104.
36. Scott Hudson, James Fogarty, Christopher Atkeson, Daniel Avrahami, Jodi Forlizzi, Sara Kiesler, Johnny Lee, and Jie Yang. 2003. Predicting Human Interruptibility with Sensors: A Wizard of Oz Feasibility Study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 257--264.
37. Shamsi T. Iqbal and Brian P. Bailey. 2005. Investigating the Effectiveness of Mental Workload As a Predictor of Opportune Moments for Interruption. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, New York, NY, USA, 1489--1492.
38. Shamsi T. Iqbal and Brian P. Bailey. 2006. Leveraging Characteristics of Task Structure to Predict the Cost of Interruption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 741--750.
39. J. Adam Jones, J. Edward Swan, II, Gurjot Singh, and Stephen R. Ellis. 2011. Peripheral Visual Information and Its Effect on Distance Judgments in Virtual and Augmented Environments. In Proceedings of the ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization (APGV '11). ACM, New York, NY, USA, 29--36.
40. Adam Kendon. 1986. Current issues in the study of gesture. The biological foundations of gestures: Motor and semiotic aspects 1 (1986), 23--47.
41. Adam Kendon. 1994. Introduction to the special issue: gesture and understanding in interaction. Research on Language and Social Interaction 27, 3 (1994), 171--173.
42. Andrew L Kun and Christian P Janssen. 2017. Calling While Driving : An Initial Experiment with HoloLens. Driving Assessment Conference University (2017), 200--206.
43. Gordon Kurtenbach. 1990. Gestures in human-computer communication. The Art of Human-Computer Interface Design, B. (1990).
44. Glyn Lawson, Davide Salanitri, and Brian Waterfield. 2016. Future directions for the development of virtual reality within an automotive manufacturer. Applied ergonomics 53 (2016), 323--330.
45. Morgan Le Chénéchal, Thierry Duval, Valérie Gouranton, Jérôme Royan, and Bruno Arnaldi. 2016. Vishnu: Virtual immersive support for helping users-an interaction paradigm for collaborative remote guiding in mixed reality. In 3DCVE 2016: International Workshop on Collaborative Virtual Environments. IEEE, 1--5.
46. Matthew Lombard, Frank Biocca, Jonathan Freeman, Wijnand IJsselsteijn, and Rachel J. Schaevitz. 2015. Immersed in Media: Telepresence Theory, Measurement & Technology. Springer Publishing Company, Incorporated.
47. Aidong Lu, Jian Huang, Shaoting Zhang, Chuang Wang, and Weichao Wang. 2016. Towards mobile immersive analysis: A study of applications. In Immersive Analytics (IA), 2016 Workshop on. IEEE, 25--30.
48. Christian Mai, Lukas Rambold, and Mohamed Khamis. 2017. TransparentHMD: Revealing the HMD User's Face to Bystanders. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia (MUM '17). ACM, New York, NY, USA, 6.
49. Tomasz Mazuryk and Michael Gervautz. 1996. Virtual reality-history, applications, technology and future. (1996).
50. Mark A McDaniel, Gilles O Einstein, Thomas Graham, and Erica Rall. 2004. Delaying execution of intentions: Overcoming the costs of interruptions. Applied Cognitive Psychology 18, 5 (2004), 533--547.
51. Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2143--2152.
52. Yoshiro Miyata and Donald A Norman. 1986. Psychological issues in support of multiple activities. User centered system design: New perspectives on human-computer interaction (1986), 265--284.
53. Calkin S Montero, Jason Alexander, Mark T Marshall, and Sriram Subramanian. 2010. Would you do that?: understanding social acceptance of gestural interfaces. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. ACM, 275--278.
54. Martin Muhlenbrock, Oliver Brdiczka, Dave Snowdon, and J-L Meunier. 2004. Learning to detect user activity and availability from a variety of sensor data. In Pervasive Computing and Communications, 2004. PerCom 2004. Proceedings of the Second IEEE Annual Conference on. IEEE, 13--22.
55. Darren Newtson. 1973. Attribution and the unit of perception of ongoing behavior. Journal of Personality and Social Psychology 28, 1 (1973), 28.
56. Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 249--256.
57. Kim O'Connell. 16.11.2016. 4 Tipps für erste Anwendungen von Virtual Reality in der Architektur. https://www.autodesk.de/redshift/virtual-reality-in-architecture/. (16.11.2016).
58. Noushin Radnia, Alireza Karduni, Isaac Cho, Zachary Wartell, and Eric Sauda. 2017. Augmented : Reality : : Architecture : Interface.
59. A Joy Rivera. 2014. A socio-technical systems approach to studying interruptions: Understanding the interrupter's perspective. Applied ergonomics 45, 3 (2014), 747--756.
60. AJ Rivera-Rodriguez and Ben-Tzion Karsh. 2010. Interruptions and distractions in healthcare: review and reappraisal. Quality and Safety in Health Care 19, 4 (2010), 304--312.
61. Rufat Rzayev, Paweł W. Wo'zniak, Tilman Dingler, and Niels Henze. 2018a. Reading on Smart Glasses: The Effect of Text Position, Presentation Type and Walking. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 45, 9 pages.
62. Rufat Rzayev, Paweł W. Wo'zniak, Tilman Dingler, and Niels Henze. 2018b. Reading on Smart Glasses: The Effect of Text Position, Presentation Type and Walking. ACM, New York, NY, USA. 45:1--45:9 pages.
63. Maria V Sanchez-Vives and Mel Slater. 2005. From presence to consciousness through virtual reality. Nat Rev Neurosci 6, 4 (2005), 332--339.
64. Paul Sawers. 2017, Lastchecked: 2018-09--10. Worldpay demos system for making payments inside virtual reality worlds. (May 2017, Lastchecked: 2018-09--10). https://venturebeat.com/2017/05/25/worldpay-demos-system-for-making-payments-inside-virtual-reality-worlds/.
65. Jeffrey H. Shuhaiber. 2004. Augmented reality in surgery. Archives of Surgery 139, 2 (2004), 170--174.
66. Mel Slater, Vasilis Linakis, Martin Usoh, Rob Kooper, and Gower Street. 1996. Immersion, presence, and performance in virtual environments: An experiment with tri-dimensional chess. In ACM virtual reality software and technology (VRST), Vol. 163. ACM Press New York, NY, 72.
67. Søren S. Sørensen. 2006. The development of Augmented Reality as a tool in Architectural and Urban design. Nordic Association for Architectural Research 19, 4 (2006), 8.
68. Takahiro Tanaka and Kinya Fujita. 2011. Study of User Interruptibility Estimation Based on Focused Application Switching. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (CSCW '11). ACM, New York, NY, USA, 721--724.
69. Burak S Tekin and Stuart Reeves. 2017. Ways of spectating: Unravelling spectator participation in Kinect play. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1558--1570.
70. Welderufael B. Tesfay, Peter Hofmann, Toru Nakamura, Shinsaku Kiyomoto, and Jetzabel Serna. 2018. PrivacyGuide: Towards an Implementation of the EU GDPR on Internet Privacy Policy Evaluation. In Proceedings of the Fourth ACM International Workshop on Security and Privacy Analytics (IWSPA '18). ACM, New York, NY, USA, 15--21.
71. Liam D. Turner, Stuart M. Allen, and Roger M. Whitaker. 2015. Interruptibility Prediction for Ubiquitous Systems: Conventions and New Directions from a Growing Field. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 801--812.
72. Kaisa V"a"an"anen and Klaus Böhm. 1993. Gesture driven interaction as a human factor in virtual environments-an approach with neural networks. Virtual reality systems (1993), 93--106.
73. Nina Valkanova, Robert Walter, Andrew Vande Moere, and Jörg Müller. 2014. MyPosition: Sparking Civic Discourse by a Public Interactive Poll Visualization. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, New York, NY, USA, 1323--1332.
74. Radu-Daniel Vatavu. 2012. User-defined gestures for free-hand TV control. In Proceedings of the 10th European conference on Interactive tv and video. ACM, 45--48.
75. Radu-Daniel Vatavu. 2015. Audience silhouettes: peripheral awareness of synchronous audience kinesics for social television. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video. ACM, 13--22.
76. Marcel Walch, Julian Frommel, Katja Rogers, Felix Schüssel, Philipp Hock, David Dobbelstein, and Michael Weber. 2017. Evaluating VR Driving Simulation from a Player Experience Perspective. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2982--2989.
77. Robert Walter, Gilles Bailly, Nina Valkanova, and Jörg Müller. 2014. Cuenesics: using mid-air gestures to select items on interactive public displays. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. ACM, 299--308.
78. Alan Wexelblat. 1997. Research challenges in gesture: Open issues and unsolved problems. In International Gesture Workshop. Springer, 1--11.
79. Bob G Witmer and Michael J Singer. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments 7, 3 (1998), 225--240.
80. Z. Yu, H. Liang, C. Fleming, and K. L. Man. 2016. An exploration of usable authentication mechanisms for virtual reality systems. In 2016 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS). 458--460.
81. Farbod Zorriassatine, Catherine Wykes, Robert Parkin, and Nabil Gindy. 2003. A survey of virtual prototyping techniques for mechanical product development. Proceedings of the institution of mechanical engineers, Part B: Journal of engineering manufacture 217, 4 (2003), 513--530.
82. Manuela Züger, Christopher Corley, André N. Meyer, Boyang Li, Thomas Fritz, David Shepherd, Vinay Augustine, Patrick Francis, Nicholas Kraft, and Will Snipes. 2017. Reducing Interruptions at Work: A Large-Scale Field Study of FlowLight. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 61--72.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User studies

    • Interaction paradigms

      • Mixed / augmented reality

      • Virtual reality