skip to main content
10.1145/3196709.3196733acmconferencesArticle/Chapter ViewAbstractPublication PagesdisConference Proceedingsconference-collections
research-article

Interweaving Visual and Audio-Haptic Augmented Reality for Urban Exploration

Published: 08 June 2018 Publication History

Abstract

While ordinary touchscreen-based interfaces on urban explorer applications draw much of a user's attention onto the screen, visual and audio-haptic augmented reality interfaces have emerged as the two main streams for enabling direct focus on the surroundings. However, neither interface alone satisfies users in the highly dynamic urban environment. This research investigates how the two complementary augmentation can coexist on one system and how people adapt to the situation by selecting the more suitable interface. A prototype was deployed in a field experiment in which participants explored points of interest in an urban environment with both interfaces. The engagement with the surroundings was compared with a touchscreen-based application. Most participants spontaneously switched between the two interfaces, which manifests the value of the availability of both interfaces on one system. The results point at the situated advantages of either interface and reveal the users' preferences when both interfaces are available.

Supplementary Material

suppl.mov (disfp249.mp4)
Supplemental video

References

[1]
Anupriya Ankolekar, Thomas Sandholm, and Louis Yu. 2013. Play it by ear: a case for serendipitous discovery of places with musicons. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, Paris, France, 2959--2968.
[2]
Matthias Baldauf, Peter Frohlich, Kathrin Masuch, and Thomas Grechenig. 2011. Comparing Viewing and Filtering Techniques for Mobile Urban Exploration. Journal of Location Based Services 5, 1 (March 2011), 38--57.
[3]
Michel Beaudouin-Lafon. 2004. Designing Interaction, Not Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '04). ACM, New York, NY, USA, 15--22.
[4]
Mark Billinghurst and Bruce H. Thomas. 2011. Mobile Collaborative Augmented Reality. Springer New York, New York, NY, 1--19.
[5]
Anne Braun and Rod McCall. 2010. User Study for Mobile Mixed Reality Devices. In Joint Virtual Reality Conference of EGVE - EuroVR - VEC. The Eurographics Association.
[6]
Jaewoo Chung, Francesco Pagnini, and Ellen Langer. 2016. Mindful navigation for pedestrians: Improving engagement with augmented reality. Technology in Society 45 (2016), 29--33.
[7]
Andrew Dillon, John Richardson, and Cliff McKnight. 1990. The effects of display size and text splitting on reading lengthy text from screen. Behaviour & Information Technology 9, 3 (1990), 215--227.
[8]
Peter Fröhlich, Antti Oulasvirta, Matthias Baldauf, and Antti Nurminen. 2011. On the move, wirelessly connected to the world. Commun. ACM 54, 1 (jan 2011), 132--138.
[9]
Maribeth Gandy, Richard Catrambone, Blair MacIntyre, Chris Alvarez, Elsa Eiriksdottir, Matthew Hilimire, Brian Davidson, and Anne Collins McLaughlin. 2010. Experiences with an AR evaluation test bed: Presence, performance, and physiological measurement. In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on. IEEE, 127--136.
[10]
Frank A. Geldard. 1957. Adventures in tactile literacy. American Psychologist 12, 3 (1957), 115--124.
[11]
Shigeru Haga, Ayaka Sano, Yuri Sekine, Hideka Sato, Saki Yamaguchi, and Kosuke Masuda. 2015. Effects of using a Smart Phone on Pedestrians' Attention and Walking. Procedia Manufacturing 3 (2015), 2574--2580.
[12]
Faizan Haque, Mathieu Nancel, and Daniel Vogel. 2015. Myopoint: Pointing and Clicking Using Forearm Mounted Electromyography and Inertial Motion Sensors. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15. ACM Press, Seoul, Republic of Korea, 3653--3656.
[13]
Linda A Henkel. 2014. Point-and-shoot memories: the influence of taking photos on memory for a museum tour. Psychological science 25, 2 (feb 2014), 396--402.
[14]
Yi-Ta Hsieh, Antti Jylhä, Valeria Orso, Luciano Gamberini, and, Giulio Jacucci. 2016. Designing a Willing-to-Use-in-Public Hand Gestural Interaction Technique for Smart Glasses. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 4203--4215.
[15]
Ira E Hyman, S Matthew Boss, Breanne M Wise, Kira E McKenzie, and Jenna M Caggiano. 2010. Did you see the unicycling clown? Inattentional blindness while walking and talking on a cell phone. Applied Cognitive Psychology 24, 5 (oct 2010), 597--607.
[16]
Lei Jing, Zixue Cheng, Yinghui Zhou, Junbo Wang, and Tongjun Huang. 2013. Magic Ring: a self-contained gesture input device on finger. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia - MUM '13. ACM Press, New York, New York, USA, 1--4.
[17]
Lynette a Jones and Nadine B Sarter. 2008. Tactile displays: guidance for their design and application. Human factors 50, 1 (2008), 90--111.
[18]
Matt Jones, George Buchanan, and Harold Thimbleby. 2003. Improving web search on small screen devices. Interacting with Computers 15, 4 (2003), 479--495.
[19]
Antti Jylhä, Yi-Ta Hsieh, Valeria Orso, Salvatore Andolina, Luciano Gamberini, and Giulio Jacucci. 2015. A Wearable Multimodal Interface for Exploring Urban Points of Interest. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM Press, New York, New York, USA. 175--182
[20]
Amy K Karlson, Benjamin B Bederson, and Jose L Contreras-Vidal. 2008. Understanding one-handed use of mobile devices. In Handbook of research on user interface design and evaluation for mobile technology. IGI Global, 86--101.
[21]
Andreas Komninos, Peter Barrie, Vassilios Stefanis, and Athanasios Plessas. 2012. Urban exploration using audio scents. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services - MobileHCI '12. ACM, 349--358.
[22]
Sang-su Lee, Sohyun Kim, and Kun-pyo Lee. 2010a. How Users Manipulate Deformable Displays as Input Devices. Methodology (2010), 1647--1656.
[23]
W. Lee, Y. Park, V. Lepetit, and W. Woo. 2010b. Point-and-shoot for ubiquitous tagging on mobile phones. In 2010 IEEE International Symposium on Mixed and Augmented Reality. 57--64.
[24]
Charlotte Magnusson, Kirsten Rassmus-Gröhn, and Delphine Szymczak. 2010. Scanning angles for directional pointing. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services - MobileHCI '10. ACM Press, Lisbon, Portugal, 399--400.
[25]
Charlotte Magnusson, Kirsten Rassmus-Gröhn, and Delphine Szymczak. 2014. Exploring history: a mobile inclusive virtual tourist guide. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction Fun, Fast, Foundational - NordiCHI '14. ACM Press, New York, New York, USA, 69--78.
[26]
David McGookin, Stephen Brewster, and Pablo Priego. 2009. Audio bubbles: Employing non-speech audio to support tourist wayfinding. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5763 LNCS (2009), 41--50.
[27]
Jack Nasar, Peter Hecht, and Richard Wener. 2008. Mobile telephones, distracted attention, and pedestrian safety. Accident; analysis and prevention 40, 1, 69--75.
[28]
Rui Nóbrega, Diogo Cabral, Giulio Jacucci, and António Coelho. 2015. NARI: Natural Augmented Reality Interface - Interaction Challenges for AR Applications. In Proceedings of the 10th International Conference on Computer Graphics Theory and Applications (VISIGRAPP 2015). SCITEPRESS, 504--510.
[29]
Eamonn O'Neill. 2014. Haptic and audio displays for augmented reality tourism applications. In 2014 IEEE Haptics Symposium (HAPTICS). IEEE, Houston, TX, USA, 485--488.
[30]
Martin Pielot, Niels Henze, and Susanne Boll. 2009. Supporting Map-based Wayfinding with Tactile Cues. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09). ACM, New York, NY, USA, Article 23, 10 pages.
[31]
Martin Pielot, Wilko Heuten, Stephan Zerhusen, and Susanne Boll. 2012. Dude, where's my car?: in-situ evaluation of a tactile car finder. In Proceedings of the 7th Nordic Conference on Human-Computer Interaction Making Sense Through Design - NordiCHI '12. ACM Press, Copenhagen, Denmark, 166--169.
[32]
Dieter Schmalstieg and Tobias Höllerer. 2015. Augmented Reality: Principles and Practice. Addison Wesley Professional.
[33]
Julia Snell. 2011. Interrogating video data: Systematic quantitative analysis versus micro-ethnographic analysis. 14 (05 2011), 253--258.
[34]
Mayuree Srikulwong and Eamonn O'Neill. 2011. A comparative study of tactile representation techniques for landmarks on a wearable device. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '11. ACM, Vancouver, BC, Canada, 2029--2038.
[35]
Ying-Chao Tung, Chun-Yen Hsu, Han-Yu Wang, Silvia Chyou, Jhe-Wei Lin, Pei-Jung Wu, Andries Valstar, and Mike Y Chen. 2015. User-Defined Game Input for Smart Glasses in Public Space. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15. ACM Press, 3327--3336.
[36]
Matthew Turk. 2014. Review Article: Multimodal Interaction: A Review. Pattern Recogn. Lett. 36 (Jan. 2014), 189--195.
[37]
Tuomas Vaittinen and David McGookin. 2016. Phases of Urban Tourists' Exploratory Navigation: A Field Study. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16). ACM, New York, NY, USA, 1111--1122.
[38]
Florian van de Camp, Alexander Schick, and Rainer Stiefelhagen. 2013. How to click in mid-air. In International Conference on Distributed, Ambient, and Pervasive Interactions. Springer, 78--86.
[39]
Yolanda Vazquez-Alvarez, Ian Oakley, and Stephen A Brewster. 2012. Auditory display design for exploration in mobile audio-augmented reality. Personal and Ubiquitous Computing 16, 8 (sep 2012), 987--999.
[40]
J. Wither, S. DiVerdi, and T. Hollerer. 2007. Evaluating Display Types for AR Selection and Annotation. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on. 95--98.
[41]
Z Yovcheva, Dimitrios Buhalis, and C Gatzidis. 2012. Smartphone Augmented Reality Applications for Tourism. e-Review of Tourism Research (eRTR) 10, 2 (2012), 63--66.

Cited By

View all
  • (2024)MR-Sense: A Mixed Reality Environment Search Assistant for Blind and Visually Impaired People2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00029(166-175)Online publication date: 17-Jan-2024
  • (2022)“I was Holding a Magic Box”: Investigating the Effects of Private and Projected Displays in Outdoor Heritage WalksProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533468(1565-1580)Online publication date: 13-Jun-2022
  • (2022)Multi-mode soft haptic thimble for haptic augmented reality based application of texture overlayingDisplays10.1016/j.displa.2022.10227274(102272)Online publication date: Sep-2022
  • Show More Cited By

Index Terms

  1. Interweaving Visual and Audio-Haptic Augmented Reality for Urban Exploration

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference
    June 2018
    1418 pages
    ISBN:9781450351980
    DOI:10.1145/3196709
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 June 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. audio-haptic interface
    2. augmented reality
    3. multimodal interaction
    4. urban exploration

    Qualifiers

    • Research-article

    Funding Sources

    • European Union Seventh Framework Programme (FP7/2007-2013)
    • Fundação para a Ciência e a Tecnologia

    Conference

    DIS '18
    Sponsor:

    Acceptance Rates

    DIS '18 Paper Acceptance Rate 107 of 487 submissions, 22%;
    Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

    Upcoming Conference

    DIS '25
    Designing Interactive Systems Conference
    July 5 - 9, 2025
    Funchal , Portugal

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)35
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)MR-Sense: A Mixed Reality Environment Search Assistant for Blind and Visually Impaired People2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00029(166-175)Online publication date: 17-Jan-2024
    • (2022)“I was Holding a Magic Box”: Investigating the Effects of Private and Projected Displays in Outdoor Heritage WalksProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533468(1565-1580)Online publication date: 13-Jun-2022
    • (2022)Multi-mode soft haptic thimble for haptic augmented reality based application of texture overlayingDisplays10.1016/j.displa.2022.10227274(102272)Online publication date: Sep-2022
    • (2021)Multi-modal Spatial Object Localization in Virtual Reality for Deaf and Hard-of-Hearing People2021 IEEE Virtual Reality and 3D User Interfaces (VR)10.1109/VR50410.2021.00084(588-596)Online publication date: Mar-2021
    • (2019)Augmented object selection through smart glassesProceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia10.1145/3365610.3368416(1-5)Online publication date: 26-Nov-2019
    • (2019)Audio-augmented museum experiences with gaze trackingProceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia10.1145/3365610.3368415(1-5)Online publication date: 26-Nov-2019

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media