An Immersive Bidirectional System for Life-size 3D Communication

ABSTRACT

Telecommunication and video conferencing are an integral part of modern society with implications in many aspects of everyday life. However, compared to a meeting in person, the sense of presence is still limited in electronic communication. In this paper, we present a novel system for life-size 3D telecommunication. It is designed to create an immersive user experience by seamlessly embedding a remote conversation partner into the local environment. To achieve this, users are captured in 3D by hybrid (color+depth) sensors and displayed on a life-size transparent 3D display. We have built two instances of this system in Zurich and Singapore. They form a complete and fully functional prototype enabling bidirectional communication in real-time over a long distance. We further demonstrate alternative hardware setups, which make our system flexible and adaptable to different usage scenarios.

References

1. Web real-time communication (WebRTC), 2011.
2. T. Balogh and P. T. Kovács. Real-time 3D light field transmission. In SPIE Photonics Europe, pages 772406--772406. Int. Soc. for Optics and Photonics, 2010.
3. S. Beck, A. Kunert, A. Kulik, and B. Froehlich. Immersive group-to-group telepresence. IEEE TVCG, 19(4):616--625, 2013.
4. Y. Chen, M. M. Hannuksela, T. Suzuki, and S. Hattori. Overview of the MVC + D 3D video coding standard. J. Visual Communication and Image Representation, 25(4):679--688, 2014.
5. H. Fuchs, G. Bishop, K. Arthur, L. McMillan, R. Bajcsy, S. Lee, H. Farid, and T. Kanade. Virtual space teleconferencing using a sea of cameras. In Proc. First Int. Conf. on Medical Robotics and Computer Assisted Surgery, volume 26, pages 161--167, 1994.
6. M. Gross, S. Würmlin, M. Näf, E. Lamboray, C. P. Spagno, A. M. Kunz, E. Koller-Meier, T. Svoboda, L. Van Gool, S. Lang, K. Strehlke, A. V. Moere, and O. G. Staadt. blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans. Graphics (Proc. SIGGRAPH), 22(3):819--827, 2003.
7. S.-R. Han, T. Yamasaki, and K. Aizawa. Time-varying mesh compression using an extended block matching algorithm. IEEE Trans. Circuits Syst. Video Techn., 17(11):1506--1518, 2007.
8. A. Jones, M. Lang, G. Fyffe, X. Yu, J. Busch, I. McDowall, M. Bolas, and P. Debevec. Achieving eye contact in a one-to-many 3D video teleconferencing system. In ACM Trans. Graphics (Proc. SIGGRAPH), volume 28, page 64, 2009.
9. T. Kanade, P. Rander, and P. Narayanan. Virtualized reality: Constructing virtual worlds from real scenes. IEEE Multimedia, 4(1):34--47, 1997.
10. A. Kulik, A. Kunert, S. Beck, R. Reichel, R. Blach, A. Zink, and B. Froehlich. C1x6: a stereoscopic six-user display for co-located collaboration in shared virtual environments. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 30(6):188, 2011.
11. C. Kuster, J.-C. Bazin, A. C. Öztireli, T. Deng, T. Martin, T. Popa, and M. Gross. Spatio-temporal geometry fusion for multiple hybrid cameras using moving least squares surfaces. CGF (Eurographics), 33(2):1--10, 2014.
12. M. Lang, O. Wang, T. Aydin, A. Smolic, and M. Gross. Practical temporal consistency for image-based graphics applications. ACM Trans. Graphics (Proc. SIGGRAPH), 31(4):34, 2012.
13. B. Lee and J. Hong. Transparent 3d display for augmented reality. In Photonics Asia, pages 855602--855602. Int. Soc. for Optics and Photonics, 2012.
14. A. Maimone and H. Fuchs. Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In IEEE/ACM ISMAR, pages 137--146, 2011.
15. A. Maimone, X. Yang, N. Dierk, A. State, M. Dou, and H. Fuchs. General-purpose telepresence with head-worn optical see-through displays and projector-based lighting. In IEEE VR, pages 23--26, 2013.
16. W. Matusik and H. Pfister. 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. In ACM Trans. Graphics (Proc. SIGGRAPH), volume 23, pages 814--824, 2004.
17. D. Nguyen and J. Canny. Multiview: improving trust in group video conferencing through spatial faithfulness. In Proceedings of the SIGCHI conf. on Human factors in computing systems, pages 1465--1474. ACM, 2007.
18. V. Nguyen, J. Lu, S. Zhao, D. Jones, and M. Do. Teleimmersive audio-visual communication using commodity hardware {applications corner}. IEEE Signal Processing Magazine, 31(6):118--136, 2014.
19. T. Oskiper, M. Sizintsev, V. Branzoi, S. Samarasekera, and R. Kumar. Augmented reality binoculars. In IEEE/ACM ISMAR, pages 219--228, 2013.
20. T. Peterka, R. Kooima, D. Sandin, A. Johnson, J. Leigh, and T. DeFanti. Advances in the dynallax solid-state dynamic parallax barrier autostereoscopic visualization display system. IEEE TVCG, 14(3):487--499, 2008.
21. N. Ranieri and M. Gross. Vision-based calibration of parallax barrier displays. In IS&T/SPIE Electronic Imaging, pages 90111D--90111D. Int. Society for Optics and Photonics, 2014.
22. N. Ranieri, H. Seifert, and M. Gross. Transparent stereoscopic display and application. In IS&T/SPIE Electronic Imaging, pages 90110P--90110P. Int. Society for Optics and Photonics, 2014.
23. C. Richardt, C. Stoll, N. A. Dodgson, H.-P. Seidel, and C. Theobalt. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. CGF (Eurographics), 31(2):247--256, 2012.
24. O. Schreer, I. Feldmann, N. Atzpadin, P. Eisert, P. Kauff, and H. Belt. 3dpresence -a system concept for multi-user and multi-party immersive 3d videoconferencing. In 5th Europ. Conf. on Visual Media Production (CVMP), pages 1--8, 2008.
25. J. Steinmeyer. Hiding the Elephant: How Magicians Invented the Impossible and Learned to Disappear. Carroll & Graf Publishers, 2003.
26. T. Sun, S. Wu, and B. Cheng. 54.4: Novel transparent emissive display on optic-clear phosphor screen. In SID Symposium Digest of Technical Papers, volume 44, pages 755--758. Wiley Online Library, 2013.
27. Y. Taguchi, T. Koike, K. Takahashi, and T. Naemura. TransCAIP: a live 3D TV system using a camera array and an integral photography display with interactive control of viewing parameters. IEEE TVCG, 15(5):841--852, 2009.
28. E. Tola, C. Zhang, Q. Cai, and Z. Zhang. Virtual view generation with a hybrid camera array. CVLAB-Report-2009-001 (EPFL), 2009.
29. H. Towles, W. Chen, R. Yang, S. Kum, H. Fuchs, N. Kelshikar, J. Mulligan, K. Daniilidis, L. Holden, B. Zeleznik, A. Sadagic, and J. Lanier. 3D tele-collaboration over Internet2. In Int. Workshop on Immersive Telepresence, 2002.
30. R. Vasudevan, G. Kurillo, E. Lobaton, T. Bernardin, O. Kreylos, R. Bajcsy, and K. Nahrstedt. High-quality visualization for geographically distributed 3-D teleimmersive applications. IEEE Trans. on Multimedia, 13(3):573--584, 2011.
31. A. Vetro, T. Wiegand, and G. J. Sullivan. Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard. Proc. of the IEEE, 99(4):626--642, 2011.
32. T. Wiegand, G. J. Sullivan, G. Bjntegaard, and A. Luthra. Overview of the H.264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Techn., 13(7):560--576, 2003.
33. J. Yang, C. Kim, and S. Lee. Semi-regular representation and progressive compression of 3-d dynamic mesh sequences. IEEE TIP, 15(9):2531--2544, 2006.
34. Z. Zhang. Flexible camera calibration by viewing a plane from unknown orientations. In ICCV, volume 1, pages 666--673, 1999.
35. M. Zwicker, H.-P. Pfister, J. Van Baar, and M. Gross. Surface splatting. In Proceedings of the 28th annual conf. on Computer graphics and interactive techniques, pages 371--378. ACM, 2001.

• Information systems

  • World Wide Web

    • Web applications

      • Internet communications tools