MirageTable: freehand interaction on a projected augmented reality tabletop

ABSTRACT

Instrumented with a single depth camera, a stereoscopic projector, and a curved screen, MirageTable is an interactive system designed to merge real and virtual worlds into a single spatially registered experience on top of a table. Our depth camera tracks the user's eyes and performs a real-time capture of both the shape and the appearance of any object placed in front of the camera (including user's body and hands). This real-time capture enables perspective stereoscopic 3D visualizations to a single user that account for deformations caused by physical objects on the table. In addition, the user can interact with virtual objects through physically-realistic freehand actions without any gloves, trackers, or instruments. We illustrate these unique capabilities through three application examples: virtual 3D model creation, interactive gaming with real and virtual objects, and a 3D teleconferencing experience that not only presents a 3D view of a remote person, but also a seamless 3D shared task space. We also evaluated the user's perception of projected 3D objects in our system, which confirmed that the users can correctly perceive such objects even when they are projected over different background colors and geometries (e.g., gaps, drops).

References

1. Agrawala, M., Beers, A. C., McDowall, I., Fröhlich, B., Bolas, M., and Hanrahan. P. 1997. The two-user Responsive Workbench: support for collaboration through individual views of a shared space. In Proc. of ACM SIGGRAPH '97. 327--332.
2. Anderson, D., Frankel, J., Marks, J., Agarwala, A., Beardsley, P., Hodgins, J., Leigh, D., Ryall, K., Sullivan, E., and Yedidia, J. S. 2000. Tangible interaction + graphical interpretation: a new approach to 3D modeling. In Proc. of ACM SIGGRAPH '00. 393--402.
3. Bimber, O. and Raskar, R. 2005. Spatial Augmented Reality: Merging Real and Virtual Worlds. A. K. Peters, Ltd., Natick, MA, USA.
4. Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnacao, L. M. 2002. The virtual showcase. IEEE Comput. Graph. Appl. 21, 6 (November 2001). 48--55.
5. Bimber, O., Encarnacao, L. M., and Branco, P. 2001. The Extended Virtual Table: An Optical Extension for Table-Like Projection Systems. Presence: Teleoper. Virtual Environ. 10, 6, 613--631.
6. Cruz-Neira, C., Sandin, D. J., and DeFanti, T. A. 1993. Surround-screen projection-based virtual reality: The design and implementation of the CAVE. In Proc. of ACM SIGGRAPH '93. 135--142.
7. Grossman, T. and Balakrishnan, R. 2006. An evaluation of depth perception on volumetric displays. In Proc. of ACM Advanced Visual Interfaces (AVI '06). 193--200.
8. Hilliges, O., Izadi, S., Wilson, A., Hodges, S., Garcia-Mendoza, A., and Butz, A. 2009. Interactions in the Air: Adding Further Depth to Interactive Tabletops. In Proc. of ACM UIST '09. 139--148.
9. Hilliges, O., Kim, D., Izadi, S., Weiss, M. and Wilson, A. D. 2012. Holodesk: Direct 3D Interactions with a Situated See-Through Display. In Proc. of ACM SIGCHI '12.
10. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., and Fitzgibbon, A. 2011. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. In Proc. of ACM UIST '11. 559--568.
11. Jota, R., and Benko, H. 2011. Constructing Virtual 3D Models with Physical Building Blocks. In CHI 2011 Extended Abstracts. 2173--2178.
12. McCann, J. J. 1992. Rules for colour constancy. Ophthal. Physiol. Opt., Vol. 12. 175--177.
13. Pinhanez, C. S. 2001. The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. In Proc. of UBICOMP '01. 315--331.
14. Piper, B., Ratti, C., Ishii, H. 2002. Illuminating clay: a 3-D tangible interface for landscape analysis. In Proc. of ACM SIGCHI '02. 355--362.
15. Raskar, R., Brown, M. S., Yang, R. Chen, W.-C., Welch, G., Towles, H., Seales, B., and Fuchs. H. 1999. Multi-projector displays using camera-based registration. In Proc. of Visualization (VIS '99). 161--168.
16. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L. and Fuchs, H. 1998. The office of the future: a unified approach to image-based modeling and spatially immersive displays. In Proc. of ACM SIGGRAPH '98. 179--188.
17. Raskar, R., Welch, G., Low, K.-L., and Bandyopadhyay, D. 2001. Shader Lamps: Animating Real Objects With Image-Based Illumination. In Proc. of the Eurographics Workshop on Rendering Techniques. 89--102.
18. Rekimoto, J. and Saitoh, M. 1999. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments. In Proc. of ACM SIGCHI '99. 378--385.
19. Segal, M., Korobkin, C., van Widenfelt, R., Foran, J., and Haeberli, P. 1992. Fast shadows and lighting effects using texture mapping. In Proc. of ACM SIGGRAPH '92. 249--252.
20. Starner, T., Leibe, B., Minner, D., Westyn, T., Hurst, A., and Weeks, J. 2003. The Perceptive Workbench: Computer-visionbased gesture tracking, object tracking, and 3D reconstruction for augmented desks. In Journal of Machine Vision and Applications, vol. 14. 59--71.
21. Sutherland, I. E. 1965. The Ultimate Display. In Proc. of the IFIP Congress. 506--508.
22. Tognazzini, B. 1994. The "Starfire" video prototype project: a case history. In Proc. of ACM SIGCHI '94. 99--105.
23. Underkoffler, J., Ullmer, B., and Ishii, H. 1999. Emancipated pixels: Real-world graphics in the luminous room. In Proc. of ACM SIGGRAPH '99. 385--392.
24. Wanger, L. 1992. The effect of shadow quality on the perception of spatial relationships in computer generated imagery. In Proc. of ACM Interactive 3D Graphics '92. 39--42.
25. Ware, C., Arthur, K. and Booth, K. 1993. Fish tank virtual reality. In Proc. of ACM SIGCHI '93. 37--42.
26. Weiss, M., Voelker, S., Sutter, C., and Borchers, J. 2010. BendDesk: dragging across the curve. In Proc. of ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). 1--10.
27. Wilson, A. 2007. Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. In Proc. of IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '07). 201--204.
28. Wilson, A. and Robbins, D. C. PlayTogether: Playing Games across Multiple Interactive Tabletops, IUI 2007 Workshop on Tangible Play: Research and Design for Tangible and Tabletop Games.
29. Wilson A. and Benko H. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proc. of ACM UIST '10. 273--282.
30. Wilson, A. D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., and Kirk, D. 2008. Bringing physics to the surface. In Proc. of ACM UIST '08. 67--76.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)