A multi-stream adaptation framework for bandwidth management in 3D tele-immersion

ABSTRACT

Tele-immersive environments will improve the state of collaboration among distributed participants. However, along with the promise a new set of challenges have emerged including the real-time acquisition, streaming and rendering of 3D scenes to convey a realistic sense of immersive spaces. Unlike 2D video conferencing, a 3D tele-immersive environment employs multiple 3D cameras to cover a much wider field of view, thus generating a very large volume of data that need to be carefully coordinated, organized, and synchronized for Internet transmission, rendering and display. This is a challenging task and a dynamic bandwidth management must be in place. To achieve this goal, we propose a multi-stream adaptation framework for bandwidth management in 3D tele-immersion. The adaptation framework relies on the hierarchy of mechanisms and services that exploits the semantic link of multiple 3D video streams in the tele-immersive environment. We implement a prototype of the framework that integrates semantic stream selection, content adaptation, and 3D data compression services with user preference. The experimental results have demonstrated that the framework shows a good quality of the resulting composite 3D rendered video in case of sufficient bandwidth, while it adapts individual 3D video streams in a coordinated and user-friendly fashion, and yields graceful quality degradation in case of low bandwidth availability.

REFERENCES

• 1: Report on 3dav exploration. International Organisation for Standardisation, ISO/IEC JTC1/SC29/WG11 N5878, July 2003.
• 2: Survey of algorithms used for multi-view video coding. International Organisation for Standardisation, ISO/IEC JTC1/SC29/WG11 N6909, January 2005.
• 3: B. Bai and J. Harms. A multiview video transcoder. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 503--506, New York, NY, USA, 2005. ACM Press.
• 4: H. Baker, N. Bhatti, D. Tanguay, I. Sobel, D. Gelb, M. Goss, W. Culbertson, and T. Malzbender. Understanding performance in coliseum, an immersive videoconferencing system. ACM Transactions on Multimedia Computing, Communications, and Applications, 1, 2005.
• 5: K. Daniilidis, J. Mulligan, R. McKendall, D. Schmid, G. Kamberova, and R. Bajcsy. Real-time 3d-teleimmersion. In Confluence of Computer Vision and Computer Graphics, pages 253--265, 2000.
• 6: M. Gross, S. Würmlin, M. Naef, E. Lamboray, C. Spagno, A. Kunz, E. Koller-Meier, T. Svoboda, L. V. Gool, S. Lang, K. Strehlke, A. V. Moere, and O. Staadt. blue-c: a spatially immersive display and 3d video portal for telepresence. ACM Trans. Graph., 22(3):819--827, 2003.
• 7: M. Hosseini and N. D. Georganas. Design of a multi-sender 3d videoconferencing application over an end system multicast protocol. In MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia, pages 480--489, New York, NY, USA, 2003. ACM Press.
• 8: P. Kauff and O. Schreer. An immersive 3d video-conferencing system using shared virtual team user environments. In CVE '02: Proceedings of the 4th international conference on Collaborative virtual environments, pages 105--112, New York, NY, USA, 2002. ACM Press.
• 9: S.-U. Kum and K. Mayer-Patel. Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl., 1(2):128--150, 2005.
• 10: S.-U. Kum, K. Mayer-Patel, and H. Fuchs. Real-time compression for dynamic 3d environments. In MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia, pages 185--194, New York, NY, USA, 2003. ACM Press.
• 11: J.-G. Lou, H. Cai, and J. Li. A real-time interactive multi-view video system. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 161--170, New York, NY, USA, 2005. ACM Press.
• 12: W. Matusik and H. Pfister. 3d tv: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph., 23(3):814--824, 2004.
• 13: J. Mulligan and K. Daniilidis. Real time trinocular stereo for tele-immersion. In International Conference on Image Processing, pages III: 959--962, 2001.
• 14: S. Mürmlin, E. Lamboray, and M. Gross. 3d video fragmens: dynamic point samples for real-time free-viewpoint video. In Technical Report No. 397, Institute of Scientific Computing, ETH, Zurich, 2003.
• 15: D. E. Ott and K. Mayer-Patel. Coordinated multi-streaming for 3d tele-immersion. In MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia, pages 596--603, New York, NY, USA, 2004. ACM Press.
• 16: R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs. The office of the future: a unified approach to image-based modeling and spatially immersive displays. In SIGGRAPH '98: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, pages 179--188, New York, NY, USA, 1998. ACM Press.
• 17: H. Towles, W.-C. Chen, R. Yang, S.-U. Kum, and H. F. et al. 3d tele-collaboration over internet2. In International Workshop on Immersive Telepresence (ITP 2002), December 2002.
• 18: H. Towles, S.-U. Kum, T. Sparks, S. Sinha, S. Larsen, and N. Beddes. Transport and rendering challenges of multi-stream, 3d tele-immersion data. In NSF Lake Tahoe Workshop on Collaborative Virtual Reality and Visualization, Octomber 2003.
• 19: R. Yang, C. Kurashima, A. Nashel, H. Towles, A. Lastra, and H. Fuchs. Creating adaptive views for group video teleconferencing - an image-based approach. In International Workshop on Immersive Telepresence (ITP 2002), 2002.
• 20: Z. Yang, Y. Cui, Z. Anwar, R. Bocchino, N. Kiyanclar, K. Nahrstedt, R. H. Campbell, and W. Yurcik. Real-time 3d video compression for tele-immersive environments. In SPIE Multimedia Computing and Networking (MMCN 2006), San Jose, CA, January 2006.
• 21: Z. Yang, K. Nahrstedt, Y. Cui, B. Yu, J. Liang, S. hack Jung, and R. Bajscy. Teeve: The next generation architecture for tele-immersive environments. In IEEE International Symposium on Multimedia (ISM2005), Irvine, CA, USA, 2005.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Design, Performance

Keywords:
3D tele-immersion, adaptation, bandwidth management