Crossmodal congruence: the look, feel and sound of touchscreen widgets

ABSTRACT

Our research considers the following question: how can visual, audio and tactile feedback be combined in a congruent manner for use with touchscreen graphical widgets? For example, if a touchscreen display presents different styles of visual buttons, what should each of those buttons feel and sound like? This paper presents the results of an experiment conducted to investigate methods of congruently combining visual and combined audio/tactile feedback by manipulating the different parameters of each modality. The results indicate trends with individual visual parameters such as shape, size and height being combined congruently with audio/tactile parameters such as texture, duration and different actuator technologies. We draw further on the experiment results using individual quality ratings to evaluate the perceived quality of our touchscreen buttons then reveal a correlation between perceived quality and crossmodal congruence. The results of this research will enable mobile touchscreen UI designers to create realistic, congruent buttons by selecting the most appropriate audio and tactile counterparts of visual button styles.

References

1. Hall, M., Hoggan, E. and Brewster, S. A. T-Bars: Towards Tactile User Interfaces for Touchscreen Mobiles. In Proc MobileHCI' 08, ACM Press (2008).
2. Lee, J. C., Dietz, P. H., Leigh, D., Yerazunis, W. S. and Hudson, S. E. Haptic Pen: A Tactile Feedback Stylus for Touch Screens In Proc 17th annual ACM symposium on User interface software and technology ACM Press (2004), 291--294.
3. Kaaresoja, T., Brown, L. M. and Linjama, J. Snap-Crackle-Pop: Tactile Feedback for Mobile Touch Screens. In Proc Eurohaptics '06, (2006), 565--566.
4. Poupyrev, I., Maruyama, S. and Rekimoto, J. Ambient Touch: Designing Tactile Interfaces for Handheld Devices. In Proc 15th annual ACM symposium on User interface software and technology, ACM Press (2002), 51--60.
5. Fukumoto, M. and Sugimura, T. Active Click: Tactile Feedback for Touch Panels. In Proc CHI' 01, ACM Press (2001), 121--122.
6. Hoggan, E., Brewster, S. A. and Johnston, J. Investigating the Effectiveness of Tactile Feedback for Mobile Touchscreens. In Proc CHI' 08, ACM Press (2008), 1573--1582.
7. Brewster, S. A. Overcoming the Lack of Screen Space on Mobile Computers. Personal and Ubiquitous Computing 6, 3 (2002), 188--205.
8. McGee, M. R., Gray, P. and Brewster, S. A. Mixed Feelings: Multimodal Perception of Virtual Roughness. In Proc EuroHaptics, Springer LNCS (2002),
9. Hoggan, E. and Brewster, S. A. Designing Audio and Tactile Crossmodal Icons for Mobile Devices. In Proc Proceedings of the 9th international conference on Multimodal interfaces, ACM (2007), 162--169.
10. Lewkowicz, D. J. The Development of Intersensory Temporal Perception: An Epigenetic Systems/Limitations View. Psychological Bulletin 126, (2000), 281--308.
11. Shimojo, S. and Shams, L. Sensory Modalities Are Not Separate Modalities: Plasticity and Interaction. Current Opinion in Neurobiology 11, 4 (2001), 505--509.
12. Huang, G., Metaxas, D. and Govindaraj, M. Feel The "Fabric": An Audio-Haptic Interface. In Proc Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association (2003), 52--61.
13. Driver, J. and Spence, C. Multisensory Perception: Beyond Modularity and Convergence. Current Biology 10, 20 (2000), 731--735.
14. McGurk, H. and MacDonald, J. W. Hearing Lips and Seeing Voices. Nature 64, 2 (1976), 746--748.
15. Bermant, R. I. and Welch, R. B. The Effect of Degree of Visual-Auditory Stimulus Separation and Eye Position Upon the Spatial Interaction of Vision and Audition. Perceptual and Motor Skill 43, (1976), 487--493.
16. Jousmaki, V. and Hari, R. Parchment-Skin Illusion: Sound-Biased Touch. Current Biology 8, 6 (1998), 190--191.
17. Laitinen, P. and Mäenpää, J. Enabling Mobile Haptic Design: Piezoelectric Actuator Technology Properties in Hand Held Devices. In Proc HAVE'06 (Haptic Audio Virtual Environments), (2006).
18. Pesqueux, L. and Rouaud, M., Vibration Level of Mobile Phones' Silent Alerts, PhD Thesis - Department of Acoustics, Aalborg University, 2005
19. Apple Macintosh Human Interface Guidelines. Addison-Wesley, 1992.
20. Zhou, L., Raju, B. I. and Srinivasan, M. A. Relevant Stimuli and Their Relationships to Primate Sa-I Mechanoreceptive Responses under Static Sinusoidan Indentation. In Proc WORLDCOMP'06 / The 2006 International Conference on Modeling, Simulation & Visualization Methods, CSREA Press, USA (2006), 67--73.
21. Lawrence, D. A. and Chapel, J. D. Performance Trade-Offs for Hand Controlled Design. In Proc 1994 IEEE Conference on Robotics and Automation, (1994), 3211--3216.
22. Bolanowski, S. J., Jr., Gescheider, G. A., Verrillo, R. T. and Checkosky, C. M. Four Channels Mediate the Mechanical Aspects of Touch. Journal of the Acoustic Society of America 84, 5 (1988), 1680--1694.
23. Fletcher, H. and A., M. W. Loudness: Its Definition, Measurement, and Calculation. Journal of Acoustical Society of America 5, (1933), 82--108.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Haptic devices

    • Interaction techniques

      • Auditory feedback

  • Interaction design

    • Systems and tools for interaction design