Parametric directional coding for precomputed sound propagation

Abstract

Convincing audio for games and virtual reality requires modeling directional propagation effects. The initial sound's arrival direction is particularly salient and derives from multiply-diffracted paths in complex scenes. When source and listener straddle occluders, the initial sound and multiply-scattered reverberation stream through gaps and portals, helping the listener navigate. Geometry near the source and/or listener reveals its presence through anisotropic reflections. We propose the first precomputed wave technique to capture such directional effects in general scenes comprising millions of polygons. These effects are formally represented with the 9D directional response function of 3D source and listener location, time, and direction at the listener, making memory use the major concern. We propose a novel parametric encoder that compresses this function within a budget of ~100MB for large scenes, while capturing many salient acoustic effects indoors and outdoors. The encoder is complemented with a lightweight signal processing algorithm whose filtering cost is largely insensitive to the number of sound sources, resulting in an immediately practical system.

References

1. Lakulish Antani, Anish Chandak, Micah Taylor, and Dinesh Manocha. 2012. Direct-to-indirect Acoustic Radiance Transfer. IEEE Transactions on Visualization and Computer Graphics 18, 2 (Feb. 2012), 261--269.
2. P. Bilinski, Ahrens J., Thomas M. R. P., Tashev I. J., and Platt J. C. 2014. HRTF magnitude synthesis via sparse representation of anthropometric features. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence. 4468--4472. http://ieeexplore.ieee.org/stamp/stamp-jsp?tp=&arnumber=6854447&isnumber=6853544
3. J. Blauert. 1997. An introduction to binaural technology. In Binaural and Spatial Hearing in Real and Virtual Environments, R. Gilkey and T. R. Anderson (Eds.). Lawrence Erlbaum, USA.
4. Jeroen Breebaart, Sascha Disch, Christof Faller, Jürgen Herre, Gerard Hotho, Kristofer Kjörling, Francois Myburg, Matthias Neusinger, Werner Oomen, Heiko Purnhagen, and Jonas Rödén. 2005. MPEG Spatial Audio Coding / MPEG Surround: Overview and Current Status. In Audio Engineering Society Convention 119. http://www.aes.org/e-lib/browse.cfm?elib=13333
5. Chunxiao Cao, Zhong Ren, Carl Schissler, Dinesh Manocha, and Kun Zhou. 2016. Interactive Sound Propagation with Bidirectional Path Tracing, to appear. ACM Transactions on Graphics (SIGGRAPH Asia 2016) (2016).
6. Jeffrey N. Chadwick, Steven S. An, and Doug L. James. 2009. Harmonic shells: a practical nonlinear sound model for near-rigid thin shells. In SIGGRAPH Asia '09: ACM SIGGRAPH Asia 2009 papers. ACM, New York, NY, USA, 1--10.
7. Anish Chandak, Christian Lauterbach, Micah Taylor, Zhimin Ren, and Dinesh Manocha. 2008. AD-Frustum: Adaptive Frustum Tracing for Interactive Sound Propagation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1707--1722.
8. Jean-Jacques Embrechts. 2016. Review on the applications of directional impulse responses in room acoustics. In Proceedings of CFA 2016. Société française d'acoustique (SFA). http://orbi.ulg.ac.be/handle/2268/193820
9. Kenji Fujii, Takuya Hotehama, Kosuke Kato, Ryota Shimokura, Yosuke Okamoto, Yukio Suzumura, and Yoichi Ando. 2004. Spatial Distribution of Acoustical Parameters in Concert Halls: Comparison of Different Scattered Reflections. 4 (01 2004).
10. Anders Gade. 2007. Acoustics in Halls for Speech and Music. In Springer Handbook of Acoustics (2007 ed.), Thomas Rossing (Ed.). Springer, Chapter 9. http://www.worldcat.org/isbn/0387304460
11. Michael A. Gerzon. 1973. Periphony: With-Height Sound Reproduction. J. Audio Eng. Soc 21, 1 (1973), 2--10. http://www.aes.org/e-lib/browse.cfm?elib=2012
12. Nail A. Gumerov and Ramani Duraiswami. 2005. Fast Multipole Methods for the Helmholtz Equation in Three Dimensions (Elsevier Series in Electromagnetism) (1 ed.). Elsevier Science, http://www.worldcat.org/isbn/0080443710
13. Brian Hamilton, Stefan Bilbao, Brian Hamilton, and Stefan Bilbao. 2017. FDTD Methods for 3-D Room Acoustics Simulation With High-Order Accuracy in Space and Time. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 25, 11 (Nov. 2017), 2112--2124.
14. Jürgen Herre, Johannes Hilpert, Achim Kuntz, and Jan Plogsties. 2015. MPEG-H Audio - The New Standard for Universal Spatial/3D Audio Coding. J. Audio Eng. Soc 62, 12 (2015), 821--830. http://www.aes.org/e-lib/browse.cfm?elib=17556
15. Doug L. James, Jernej Barbie, and Dinesh K. Pai. 2006. Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources. ACM Transactions on Graphics 25, 3 (July 2006), 987--995.
16. Heinrich Kuttruff. 2000. Room Acoustics (4 ed.). Taylor & Francis. http://www.worldcat.org/isbn/0419245804
17. Mikko V. Laitinen, Tapani Pihlajamäki, Cumhur Erkut, and Ville Pulkki. 2012. Parametric Time-frequency Representation of Spatial Sound in Virtual Worlds. ACM Trans. Appl. Percept. 9, 2 (June 2012).
18. Dingzeyu Li, Yun Fei, and Changxi Zheng. 2015. Interactive Acoustic Transfer Approximation for Modal Sound. ACM Trans. Graph. 35, 1 (Dec. 2015).
19. Ruth Y. Litovsky, Steven H. Colburn, William A. Yost, and Sandra J. Guzman. 1999. The precedence effect. The Journal of the Acoustical Society of America 106, 4 (1999), 1633--1654.
20. Ravish Mehra, Nikunj Raghuvanshi, Lakulish Antani, Anish Chandak, Sean Curtis, and Dinesh Manocha. 2013. Wave-based Sound Propagation in Large Open Scenes Using an Equivalent Source Formulation. ACM Trans. Graph. 32, 2 (April 2013).
21. Ravish Mehra, Nikunj Raghuvanshi, Lauri Savioja, Ming C. Lin, and Dinesh Manocha. 2012. An efficient GPU-based time domain solver for the acoustic wave equation. Applied Acoustics 73, 2 (Feb. 2012), 83--94.
22. Ravish Mehra, Atul Rungta, Abhinav Golas, Ming Lin, and Dinesh Manocha. 2015. WAVE: Interactive Wave-based Sound Propagation for Virtual Environments. IEEE transactions on visualization and computer graphics 21, 4 (April 2015), 434--442. http://view.ncbi.nlm.nih.gov/pubmed/26357093
23. Juha Merimaa and Ville Pulkki. 2005. Spatial Impulse Response Rendering I: Analysis and Synthesis. J. Audio Eng. Soc 53, 12 (2005), 1115--1127. http://www.aes.org/e-lib/browse.cfm?elib=13401
24. Brian Moore and Brian Glasberg. 1996. A Revision of Zwicker's Loudness Model. 82 (03 1996), 335--345.
25. D. Murphy, A. Kelloniemi, J. Mullen, and S. Shelley. 2007. Acoustic Modeling Using the Digital Waveguide Mesh. IEEE Signal Processing Magazine 24, 2 (March 2007), 55--66.
26. Juhani Paasonen, Aleksandr Karapetyan, Jan Plogsties, and Ville Pulkki. 2017. Proximity of Surfaces - Acoustic and Perceptual Effects. J. Audio Eng. Soc 65, 12 (2017), 997--1004. http://www.aes.org/e-lib/browse.cfm?elib=19365
27. Allan D. Pierce. 1989. Acoustics: An Introduction to Its Physical Principles and Applications. Acoustical Society of America, http://www.worldcat.org/isbn/0883186128
28. Boaz Rafaely 2015. Fundamentals of Spherical Array Processing (Springer Topics in Signal Processing) (2015 ed.). Springer, http://www.worldcat.org/isbn/9783662456644
29. Nikunj Raghuvanshi, Rahul Narain, and Ming C. Lin. 2009a. Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition. IEEE Transactions on Visualization and Computer Graphics 15, 5 (2009), 789--801.
30. Nikunj Raghuvanshi, Rahul Narain, and Ming C. Lin. 2009b. Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition. IEEE Transactions on Visualization and Computer Graphics 15, 5 (2009), 789--801.
31. Nikunj Raghuvanshi and John Snyder. 2014. Parametric Wave Field Coding for Precomputed Sound Propagation. ACM Trans. Graph. 33, 4 (July 2014).
32. Nikunj Raghuvanshi, John Snyder, Ravish Mehra, Ming C. Lin, and Naga K. Govindaraju. 2010. Precomputed Wave Simulation for Real-Time Sound Propagation of Dynamic Sources in Complex Scenes. ACM Transactions on Graphics 29, 3 (July 2010).
33. Jens H. Rindel and Claus L. Christensen. 2013. The use of colors, animations and auralizations in room acoustics. In Internoise 2013.
34. Lauri Savioja and U. Peter Svensson. 2015. Overview of geometrical room acoustic modeling techniques. The Journal of the Acoustical Society of America 138, 2 (01 Aug. 2015), 708--730.
35. Carl Schissler, Ravish Mehra, and Dinesh Manocha. 2014. High-order Diffraction and Diffuse Reflections for Interactive Sound Propagation in Large Environments. ACM Trans. Graph. 33, 4 (July 2014).
36. Dirk Schröder. 2011. Physically Based Real-Time Auralization of Interactive Virtual Environments. Logos Verlag, http://www.worldcat.org/isbn/3832530312
37. Jonathan Sheaffer, Maarten Van Walstijn, Boaz Rafaely, and Konrad Kowalczyk. 2015. Binaural Reproduction of Finite Difference Simulations Using Spherical Array Processing. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 23, 12 (Dec. 2015), 2125--2135.
38. S. Siltanen, T. Lokki, and L. Savioja. 2010a. Rays or Waves? Understanding the Strengths and Weaknesses of Computational Room Acoustics Modeling Techniques. In Proc. Int. Symposium on Room Acoustics. Melbourne, Australia.
39. Samuel Siltanen, Tapio Lokki, and Lauri Savioja. 2010b. Room acoustics modeling with acoustic radiance transfer. Proc. ISRA Melbourne (2010).
40. Julius O. III Smith. 2007. Introduction to Digital Filters with Audio Applications. (2007). https://ccrma.stanford.edu/~jos/filters/
41. Alex Southern, Damian T. Murphy, and Lauri Savioja. 2012. Spatial Encoding of Finite Difference Time Domain Acoustic Models for Auralization. Trans. Audio, Speech and Lang. Proc. 20, 9 (Nov. 2012), 2420--2432.
42. Micah T. Taylor, Anish Chandak, Lakulish Antani, and Dinesh Manocha. 2009. RESound: interactive sound rendering for dynamic virtual environments. In Proceedings of ACM conference on Multimedia. ACM, New York, NY, USA, 271--280.
43. Sakari Tervo, Jukka Pätynen, Antti Kuusinen, and Tapio Lokki. 2013. Spatial Decomposition Method for Room Impulse Responses. J. Audio Eng. Soc 61, 1/2 (2013), 17--28.
44. Nicolas Tsingos. 2009. Pre-computing geometry-based reverberation effects for games. In 35th AES Conference on Audio for Games.
45. Nicolas Tsingos, Carsten Dachsbacher, Sylvain Lefebvre, and Matteo Dellepiane. 2007. Instant Sound Scattering. In Rendering Techniques (Proceedings of the Eurographics Symposium on Rendering). http://www-sop.inria.fr/reves/Basilic/2007/TDLD07
46. Michael Vorländer. 2007. Auralization: Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic Virtual Reality (RWTHedition) (1 ed.). Springer, http://www.worldcat.org/isbn/3540488294
47. Hengchin Yeh, Ravish Mehra, Zhimin Ren, Lakulish Antani, Dinesh Manocha, and Ming Lin. 2013. Wave-ray Coupling for Interactive Sound Propagation in Large Complex Scenes. ACM Trans. Graph. 32, 6 (Nov. 2013).
48. Wen Zhang, Thushara D. Abhayapala, Rodney A. Kennedy, and Ramani Duraiswami. 2010. Insights into head-related transfer function: Spatial dimensionality and continuous representation. The Journal of the Acoustical Society of America 127, 4 (01 April 2010), 2347--2357.

Index Terms

• Applied computing

  • Arts and humanities

    • Sound and music computing

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality