Auditory bias of visual attention for perceptually-guided selective rendering of animations

ABSTRACT

The developers and users of real-time graphics, such as games and virtual reality, are demanding ever more realistic computer generated images. Despite the availability of modern graphics hardware, such real-time high fidelity graphics is still not feasible on a single PC. Research on visual perception has shown that the perceived quality of rendered graphics depends not only on the fidelity of the generated imagery but also on the characteristics of visual attention and the limitations of the human visual system. The findings of this research have been used to define perceptually driven criteria for rendering with the aim to reduce rendering times. Furthermore, in reality there are strong crossmodal interactions between auditory and visual stimuli, with a number of studies showing that stimuli reaching the various senses are not, in general, processed independently. In this paper we investigate whether auditory stimuli, and more specifically sound effects with abrupt onsets, affect a viewer's perceived quality of rendered images while watching computer generated animations. In fact, we show how we can potentially accelerate the rendering of animations by directing the viewer's attention towards the source of a sound and selectively render at high quality only the sound emitting object. For this purpose a renderer was implemented which selectively renders the sound emitting objects and the surrounding pixels to high quality while the rest of the scene is rendered at a significantly lower quality. A psychophysical experiment with 120 participants was run which revealed a significant effect of sound effects on the perceived rendering quality. Our results show that audio stimuli, and in particular sound effects, can be exploited when rendering animations, to significantly reduce rendering time without any loss in the user's perception of delivered quality.

REFERENCES

• 1: Broadbent, D. 1958. Perception and Communication. London: Pergamon Press.
• 2: Bullot, N. J., 2004. Auditory and cross-modal attention for the cognitive access to objects. (Under review). http://nicolasbullot.org/Publi/PDF/Bullot_200x_Crossmodal-Attention_1.pdf Last accessed: 14/4/2005.
• 3: Carlson, N. R. 1993. Psychology, The Science of Behavior. Needham Heights, MA: Allyn and Bacon.
• 4: Cater, K., and Chalmers, A. 2003. Maintaining perceived quality for interactive tasks. In IS&T/SPIE Conference on Human Vision and Electronic Imaging VIII, SPIE Proceedings, vol. 5007-21.
• 5: K. Cater , A. Chalmers , G. Ward, Detail to attention: exploiting visual tasks for selective rendering, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 6: Driver, J., and Spence, C. 1998. Attention and the crossmodal construction of space. Trends in Cognitive Sciences 2, 7, 254--262.
• 7: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558 ]
• 8: David P. Luebke , Benjamin Hallen, Perceptually-Driven Simplification for Interactive Rendering, Proceedings of the 12th Eurographics Workshop on Rendering Techniques, p.223-234, June 25-27, 2001
• 9: Mack, A., and Rock, I. 1998a. Inattentional Blindness. Cambridge, MA: MIT Press.
• 10: The Effect of Music on the Perception of Display Rate and Duration of Animated Sequences: An Experimental Study, Proceedings of the Theory and Practice of Computer Graphics 2004 (TPCG'04), p.128-134, June 08-10, 2004 [doi>10.1109/TPCG.2004.34]
• 11: Georgia Mastoropoulou , Kurt Debattista , Alan Chalmers , Tom Troscianko, The influence of sound effects on the perceived smoothness of rendered animations, Proceedings of the 2nd symposium on Applied perception in graphics and visualization, August 26-28, 2005, A Coroña, Spain [doi>10.1145/1080402.1080404]
• 12: Karol Myszkowski , Takehiro Tawara , Hiroyuki Akamine , Hans-Peter Seidel, Perception-guided global illumination solution for animation rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.221-230, August 2001 [doi>10.1145/383259.383284]
• 13: Noisternig, M., Musil, T., Sontacchi, A., and Hldrich, R. 2003. A 3d real time rendering engine for binaural sound reproduction. In Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, vol. 3663.
• 14: O'Sullivan, C., Howlett, S., Morvan, Y., McDonnell, R., and O'Conor, K. 2004. Star report on perceptually adaptive graphics. In Eurographics 2004, Eurographics Association.
• 15: Sumanta N. Pattanaik , Jack Tumblin , Hector Yee , Donald P. Greenberg, Time-dependent visual adaptation for fast realistic image display, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.47-54, July 2000 [doi>10.1145/344779.344810]
• 16: Peterson, L. R., and Peterson, M. J. 1959. Short-term memory retention of individual items. Journal of Experimental Psychology 58, 193--198.
• 17: Pettersson, R., 1999. Attention an information design perspective! (International Institute for Information Design (IIID), Vienna, Austria).
• 18: Mahesh Ramasubramanian , Sumanta N. Pattanaik , Donald P. Greenberg, A perceptually based physical error metric for realistic image synthesis, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.73-82, July 1999 [doi>10.1145/311535.311543]
• 19: Rensink, R. 2000b. Seeing, sensing, and scrutinizing. Vision Research 40, 1469--1487.
• 20: Simons, D. J., and Chabris, C. 1999. Gorillas in our midst sustained inattentional blindness for dynamic events. Perception 28, 1059--1074.
• 21: Simons, D. 2000. Attentional capture and inattentional blindness. Trends in Cognitive Sciences 4, 147--155.
• 22: Soto-Faraco, S., Lyons, J., Gazzaniga, M. S., Spence, C., and Kingstone, A. 2002. The ventriloquist in motion: Illusory capture of dynamic information across sensory modalities. Cognitive Brain Research 14, 139--146.
• 23: Storms, R. L. 1998. Auditory-Visual Cross-Modal Perception Phenomena. PhD thesis, Naval Postgraduate School, Monterey, California.
• 24: Tellinghuisen, D. J., and Nowak, E. J. 2003. The inability to ignore auditory distractors as a function of visual task perceptual load. Perception & Psychophysics 65, 5, 817--828.
• 25: Gregory J. Ward, The RADIANCE lighting simulation and rendering system, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.459-472, July 1994 [doi>10.1145/192161.192286]
• 26: Benjamin Watson , Alinda Friedman , Aaron McGaffey, Measuring and predicting visual fidelity, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.213-220, August 2001 [doi>10.1145/383259.383283]
• 27: Hector Yee , Sumanita Pattanaik , Donald P. Greenberg, Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments, ACM Transactions on Graphics (TOG), v.20 n.1, p.39-65, Jan. 2001 [doi>10.1145/383745.383748]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Keywords:
3D graphics, auditory-visual interactions, cross-modal interactions, perceptually-adaptive techniques, rendering techniques, selective rendering