A 3D tele-immersion streaming approach using skeleton-based prediction

ABSTRACT

3D collaborative Tele-Immersive environments allow reconstruction of real world 3D scenes in the virtual world across multiple physical locations. This kind of reconstruction results in a lot of 3D data being transmitted over the internet in real time. The current systems allow for transmission at low frame rates due to the large volume of data and network bandwidth restrictions. In this paper we propose a prediction based approach that generates future frames by animating the live model based on few skeleton points. By doing so the magnitude of data transmitted is reduced to few hundred bytes. The prediction errors are corrected when an entire frame is received. This approach allows minimal amounts (few bytes) of data to be transmitted per frame, thus allowing for high frame rates and still maintain an acceptable visual quality of reconstruction at the receiver side.

References

1. Z. Yang, B. Yu, W. Wu, and K. Nahrstedt, "Collaborative dancing in Tele-immersive environment," in in Proc. of ACM Multimedia (MM'06), 2006.
2. A. Redert, M.O. de Beeck, C. Fehn, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek, I. Sexton, and P. Surman, "Advanced three-dimensional television system technologies," in 3D Data Processing Visualization and Transmission, 2002, pp. 313--319.
3. Z. Yang, Y. Cui, Z. Anwar, R. Bocchino, N. Kiyanclar, K. Nahrstedt, R. H. Campbell, and W. Yurcik, "Realtime 3d video compression for tele-immersive environments," in in Proceedings of Multimedia Computing and Networking 2006, 2006.
4. S. Shi, W. J. Jeon, K. Nahrstedt, and R. H. Campbell, "M-teeve: Real-time 3d video interaction and broadcasting framework for mobile devices," in ICST International Conference on Immersive Telecommunications (IMMERSCOM), Berkeley, CA, USA, 2009.
5. S-U. Kum and K. Mayer-Patel, "Real-time multidepth stream compression," ACM Trans. Multimedia Comput. Commun. Appl., vol. 1, no. 2, pp. 128--150, May 2005.
6. J-M. Lien, G. Kurillo, and R. Bajcsy, "Multi-camera tele-immersion system with real-time model driven data compression: A new model-based compression method for massive dynamic point data," Vis. Comput., vol. 26, no. 1, pp. 3--15, Nov. 2009.
7. Z. Tang, G. Rong, X. Guo, and B. Prabhakaran, "Streaming 3D shape deformations in collaborative virtual environment," in Virtual Reality Conference (VR), 2010 IEEE, march 2010, pp. 183--186.
8. S. Yoshizawa, A.G. Belyaev, and H-P.Seidel, "Free-form skeleton-driven mesh deformations," in Proceedings of the eighth ACM symposium on Solid modeling and applications, New York, NY, USA, 2003, SM '03, pp. 247--253, ACM.
9. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images," in Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 2011, CVPR '11, pp. 1297--1304, IEEE Computer Society.
10. F. Aurenhammer, "Voronoi diagrams - a survey of a fundamental geometric data structure," ACM Comput. Surv., vol. 23, no. 3, pp. 345--405, Sept. 1991.
11. D. Salomon, Data Compression: The Complete Reference, Springer-Verlag, Berlin, Germany / Heidelberg, Germany / London, UK / etc., 2007.
12. W. Wu, A. Arefin, G. Kurillo, P. Agarwal, K. Nahrstedt, and R. Bajcsy, "Color-plus-depth level-of-detail in 3d tele-immersive video: a psychophysical approach," in Proceedings of the 19th ACM international conference on Multimedia, New York, NY, USA, 2011, MM '11, pp. 13--22, ACM.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Hierarchical representations

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

    • Shape modeling

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Mixed / augmented reality

      • Virtual reality