“Stop over there”: natural gesture and speech interaction for non-critical spontaneous intervention in autonomous driving

ABSTRACT

We propose a new multimodal input technique for Non-critical Spontaneous Situations (NCSSs) in autonomous driving scenarios such as selecting a parking lot or picking up a hitchhiker. Speech and deictic (pointing) gestures were combined to instruct the car about desired interventions which include spatial references to the current environment (e.g., ''stop over [pointing] there'' or ''take [pointing] this parking lot''). In this way, advantages from both modalities were exploited: Speech allows for selecting from many maneuvres and functions in the car (e.g., stop, park), whereas deictic gestures provide a natural and intuitive way of indicating spatial discourse referents used in these interventions (e.g., near this tree, that parking lot). The speech and pointing gesture input was compared to speech and touch-based input in a user study with 38 participants. The touch-based input was selected as a baseline due to its widespread use in in-car touch screens. The evaluation showed that speech and pointing gestures are perceived more natural, intuitive and less cognitively demanding compared to speech and touch and are thus recommended as NCSSs intervention technique for autonomous driving.

References

1. Leonardo Angelini, Andreas Sonderegger, Jürgen Baumgartner, Francesco Carrino, Stefano Carrino, Maurizio Caon, Omar Abou Khaled, Jürgen Sauer, Denis Lalanne, and Elena Mugellini. 2016. Comparing Gesture, Speech and Touch Interaction Modalities for In-Vehicle Infotainment Systems. Actes de la 28ieme conference francophone sur l’Interaction Homme-Machine on - IHM ’16: 188-196.
2. Richard Bolt. 1980. ”Put-that-there.” Proceedings of the 7th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’80: 262-270.
3. James Bucanek. 2009. Model-View-Controller Pattern. In Learn Objective-C for Java Developers. Apress, Berkeley, CA, 353-402.
4. Stephen Casner, Edwin Hutchins, and Don Norman. 2016. The Challenges of Partially Automated Driving. Communications of the ACM 59, 5: 70-77.
5. David Cohen, Akshay Chandrashekaran, Ian Lane, and Antoine Raux. 2014. The HRI-CMU Corpus of Situated In-Car Interactions. International Workshop Series on Spoken Dialogue Systems Technology: 201-212.
6. Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies - why and how. Knowledge-Based Systems 6, 4: 258-266.
7. Murat Dikmen and Catharine Burns. 2016. Autonomous Driving in the Real World: Experiences with Tesla Autopilot and Summon. In Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ’16), Ann Arbor, MI, USA., 225-228.
8. Mica Endsley and David Kaber. 1999. Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 42, 3: 462-492.
9. Martin Fischbach. 2015. Software Techniques for Multimodal Input Processing in Realtime Interactive Systems. In Proceedings of the International Conference on Multimodal Interaction - ICMI’15: 623-627.
10. Yannick Forster, Frederik Naujoks, and Alexandra Neukum. 2016. Your Turn or My Turn? Design of a Human-Machine Interface for Conditional Automation Your Turn or My Turn? Design of a Human-Machine Interface for Conditional Automation. In Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutomotiveUI’16, 253-260.
11. Tom Michael Gasser. 2012. Ergebnisse der Projektgruppe Automatisierung: Rechtsfolgen zunehmender Fahrzeugautomatisierung. Bergisch Gladbach.
12. Daniel Göhring, David Latotzky, Miao Wang, and Raul Rojas. 2013. Semi-autonomous Car Control Using Brain Computer Interfaces. Springer, Berlin, Heidelberg, 393-408.
13. Sandra Hart. 2006. Nasa-Task Load Index (NASA-TLX) 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9: 904-908.
14. Sandra Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology 52, C: 139-183.
15. Renate Hauslschmid, Benjamin Menrad, and Andreas Butz. 2015. Freehand vs. micro gestures in the car: Driving performance and user experience. 2015 IEEE Symposium on 3D User Interfaces (3DUI) 336: 159-160.
16. Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits, and atoms. In Proceedings of the 8th international conference on Intelligent user interfaces, 234-241.
17. Michaela Kauer, Benjamin Franz, Michael Schreiber, Ralph Bruder, and Sebastian Geyer. 2012. User acceptance of cooperative maneuverbased driving - A summary of three studies. Work 41, SUPPL.1: 4258-4264.
18. Martin Kienle, Daniel Damböck, Heiner Bubb, and Klaus Bengler. 2013. The ergonomic value of a bidirectional haptic interface when driving a highly automated vehicle. Cognition, Technology and Work 15, 4: 475-482.
19. Denis Lalanne, Laurence Nigay, Philippe Palanque, Peter Robinson, Jean Vanderdonckt, and Jean-Francois Ladry. 2009.
20. Fusion engines for multimodal input: a survey. International Conference on Multimodal Interfaces: 153-160.
21. Marc Erich Latoschik. 2001. A General Framework for Multi-Modal Interaction in Virtual Reality Systems: PrOSA. In The Future of VR and AR Interfaces-Multimodal, 21-25.
22. Marc Erich Latoschik. 2002. Designing transition networks for multimodal VR-interactions using a markup language. In Proceedings of the International Conference on Multimodal Interfaces - ICMI’02: 411-416.
23. Marc Erich Latoschik. 2005. A user interface framework for multimodal VR interactions. In Proceedings of the International Conference on Multimodal Interfaces - ICMI’05, 76-83.
24. Marc Erich Latoschik and Christian Fröhlich. 2007. Semantic reflection for intelligent virtual environments. In Proceedings - IEEE Virtual Reality, 305-306.
25. Marc Erich Latoschik and Ipke Wachsmuth. 1998. Exploiting distant pointing gestures for object selection in a virtual environment. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 185-196.
26. Sebastian Loehmann, Martin Knobel, Melanie Lamara, and Andreas Butz. 2013. Culturally independent gestures for incar interactions. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 538-545.
27. Jannette Maciej and Mark Vollrath. 2009. Comparison of manual vs. speech-based interaction with in-vehicle information systems. Accident Analysis & Prevention 41, 5: 924-930.
28. Microsoft Corporation. 2014. Visual Gesture Builder (VGB). Retrieved February 24, 2017 from https://msdn.microsoft. com/de-de/library/dn785304.aspx
29. Anja Naumann, Jörn Hurtienne, Johann Habakuk Israel, Carsten Mohs, Martin Christof Kindsmüller, Herbert A. Meyer, Steffi Husslein, and IUUI Research Group. 2007. Intuitive use of user interfaces: Defining a vague concept. Engineering Psychology and Cognitive Ergonomics: 128-136.
30. NHTSA. 2013. Preliminary statement of policy concerning automated vehicles. National Highway Traffic Safety Administration. National Highway Traffic Safety Administration: 1-14. ICMI’17, November 13–17, 2017, Glasgow, UK Tscharn, Latoschik, Löffler, and Hurtienne
31. Sharon Oviatt. 1999. Ten myths of multimodal interaction. Communications of the ACM 42, 11: 74-81.
32. Thies Pfeiffer and Marc Erich Latoschik. 2004. Resolving object references in multimodal dialogues for immersive virtual environments. In Proceedings - Virtual Reality Annual International Symposium, 35-42.
33. Bastian Pfleging, Stefan Schneegass, and Albrecht Schmidt. 2012. Multimodal interaction in the car - combining speech and gestures on the steering wheel. In Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 155-162.
34. Bernhard Preim and Raimund Dachselt. 2015. Interaktive Systeme: Band 2: User Interface Engineering, 3D-Interaktion. Springer Vieweg: Wiesbaden.
35. Jonas Radlmayr and Klaus Bengler. 2015. Literaturanalyse und Methodenauswahl zur Gestaltung von Systemen zum hochautomatisierten Fahren. FAT-Schriftenreihe 276: 1-57.
36. Jonas Radlmayr, Christian Gold, Lutz Lorenz, Mehdi Farid, and Klaus Bengler. 2014. How Traffic Situations and Nondriving-Related Tasks Affect the Takeover Quality in Highly Automated Driving. In Human Factors and Ergonomics Annual Meeting, 2063-2067.
37. Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998. The Office of the FutureâĂŕ: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays. SIGGRAPH ’98 Proceedings of the 25th annual conference on Computer graphics and interactive techniques: 1-10.
38. Leah M. Reeves, Jean-Claude Martin, Michael McTear, TV Raman, Kay M. Stanney, Hui Su, Qian Ying Wang, Jennifer Lai, James A. Larson, Sharon Oviatt, T. S. Balaji, Stephanie Buisine, Penny Collings, Phil Cohen, and Ben Kraal. 2004.
39. Guidelines for multimodal user interface design. Communications of the ACM 47, 1: 57-59.
40. Sonja Rümelin, Chadly Marouane, and Andreas Butz. 2013. Free-hand pointing for identification and interaction with distant objects. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ’13): 40-47.
41. JR Treat. 1977. Tri-Level Study of the Causes of Traffic Accidents: An overview of final results. In Proceedings of the American Association for Automotive Medicine Annual Conference, 391-403.
42. Matthew Turk. 2014. Multimodal interaction: A review. Pattern Recognition Letters 36, 189-195.
43. Volvo. 2013. Volvo Trucks European Accident Research and Safety Report 2013. Retrieved February 20, 2017 from https:// www.kenallenlaw.com/2013/02/new-volvo-truck-study-9out-of-10-truck-accidents-in-europe-caused-by-humanfactor-including-distracted-driving-while-ntsa-proposesnew-trucking-regs-to-fmsca-after-2011-nevada-crash/
44. Marcel Walch, Tobias Sieber, Philipp Hock, Martin Baumann, and Michael Weber. 2016. Towards Cooperative Driving: Involving the Driver in an Autonomous Vehicle’s Decision Making. In Proceedings of AutomotiveUI’16: 261-268.
45. Daniel Wigdor and Dennis Wixon. 2011. Brave NUI world: designing natural user interfaces for touch and gesture. Elsevier.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction techniques