Remixed Reality: Manipulating Space and Time in Augmented Reality

ABSTRACT

We present Remixed Reality, a novel form of mixed reality. In contrast to classical mixed reality approaches where users see a direct view or video feed of their environment, with Remixed Reality they see a live 3D reconstruction, gathered from multiple external depth cameras. This approach enables changing the environment as easily as geometry can be changed in virtual reality, while allowing users to view and interact with the actual physical world as they would in augmented reality. We characterize a taxonomy of manipulations that are possible with Remixed Reality: spatial changes such as erasing objects; appearance changes such as changing textures; temporal changes such as pausing time; and viewpoint changes that allow users to see the world from different points without changing their physical location. We contribute a method that uses an underlying voxel grid holding information like visibility and transformations, which is applied to live geometry in real time.

References

1. Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 19681979.
2. Hrvoje Benko, Andrew D. Wilson, and Federico Zannier. 2014. Dyadic Projected Spatial Augmented Reality. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 645--655.
3. Paul J. Besl and Neil D. McKay. 1992. A Method for Registration of 3-D Shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2 (February 1992), 239--256.
4. Mark Billinghurst, Adrian Clark, and Gun Lee. 2015. A Survey of Augmented Reality. Found. Trends Hum.Comput. Interact. 8, 2--3, 73--272.
5. Oliver Bimber and Ramesh Raskar. 2005. Spatial Augmented Reality: Merging Real and Virtual Worlds. A. K. Peters, Ltd., Natick, MA, USA.
6. Gerd Bruder, Frank Steinicke, Klaus H. Hinrichs. 2009. Arch-Explore: A natural user interface for immersive architectural walkthroughs. In Proceedings of the IEEE Symposium on 3D User Interfaces (3DUI'09). 75--82.
7. Kai Bürger, Jens Krüger and Rüdiger Westermann. Direct Volume Editing. 2008. In IEEE Transactions on Visualization and Computer Graphics, 14 / 6, 13881395.
8. Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, and Shahram Izadi. 2016. Fusion4D: real-time performance capture of challenging scenes. In ACM Trans. Graph. 35/4.
9. Steven Feiner, Blair MacIntyre, Marcus Haupt, and Eliot Solomon. 1993. Windows on the world: 2D windows for 3D augmented reality. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '93). ACM, New York, NY, USA, 145--155.
10. Sebastian Freitag, Dominik Rausch, Torsten Kuhlen. 2014 Reorientation in virtual environments using interactive portals. In Proceedings of the IEEE Symposium on 3D User Interfaces (3DUI'14). 119--122.
11. Jan Herling and Wolfgang Broll. 2012. PixMix: A realtime approach to high-quality Diminished Reality. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR '12). IEEE Computer Society, Washington, DC, USA, 141--150.
12. Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects As Tangible Proxies in Augmented Reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1957--1967.
13. Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. (2016) VolumeDeform: Real-Time Volumetric Nonrigid Reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV '16). 362--379.
14. Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 559--568.
15. Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom: Peripheral Projected Illusions for Interactive Experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 869-- 878.
16. Brett R. Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew D. Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 637--644.
17. André Kunert, Alexander Kulik, Stephan Beck, and Bernd Froehlich. 2014. Photoportals: shared references in space and time. In Proceedings of the ACM conference on Computer supported cooperative work & social computing (CSCW '14). ACM, New York, NY, USA, 1388--1399.
18. Joshua Lifton, and Joseph A. Paradiso. 2010. Dual Reality: Merging the Real and Virtual. In Facets of Virtual Environments (FaVE '09). Springer, Berlin, Heidelberg.
19. Yung-Ta Lin, Yi-Chi Liao, Shan-Yuan Teng, Yi-Ju Chung, Liwei Chan, and Bing-Yu Chen. 2017. OutsideIn: Visualizing Out-of-Sight Regions-of-Interest in a 360 Video Using Spatial Picture-in-Picture Previews. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA.
20. David Lindlbauer, Jörg Müller, and Marc Alexa. 2016. Changing the Appearance of Physical Interfaces Through Controlled Transparency. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA.
21. David Lindlbauer, Jörg Müller, and Marc Alexa. 2017. Changing the Appearance of Real-World Objects By Modifying Their Surroundings. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 39543965.
22. Sang-won Leigh and Pattie Maes. 2015. AfterMath: Visualizing Consequences of Actions through Augmented Reality. In Proceedings of the Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). ACM, New York, NY, USA, 941--946.
23. Steve Mann. Mediated reality. Technical Report MITML Percom TR-260, University of Toronto, 1994.
24. Steve Mann and James Fung. 2001. Videoorbits on eye tap devices for deliberately diminished reality or altering the visual perception of rigid planar patches of a real world scene. In Proceedings of the International Symposium on Mixed Reality (ISMR'01). 48--55.
25. Steve Mann and James Fung. 2002. EyeTap devices for augmented, deliberately diminished, or otherwise altered visual perception of rigid planar patches of realworld scenes. In Presence: Teleoperators and Virtual Environments 11, 2 158--175.
26. Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays. In Proceedings of the Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2143--2152.
27. Paul Milgram and Fumio Kishino. A taxonomy of mixed reality visual displays. In IIEICE Transactions on Information Systems, Vol E77-D, No.12 December 1994.
28. Takashi Miyaki and Jun Rekimoto. 2016. LiDARMAN: reprogramming reality with egocentric laser depth scanning. In ACM SIGGRAPH 2016 Emerging Technologies (SIGGRAPH '16). ACM, New York, NY, USA, Article 15.
29. Mathias Möhring, Christian Lessig, and Oliver Bimber, Video see-through AR on consumer cell-phones. 2004 In Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality, (ISMAR '04). 252--253.
30. Simon Olberding, Michael Wessely, and Jürgen Steimle. 2014. PrintScreen: Fabricating Highly Customizable Thin-film Touch-displays. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 281-- 290.
31. Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 741--754.
32. Shigeru Owada, Frank Nielsen, and Takeo Igarashi. 2005. Volume catcher. In Proceedings of the Symposium on interactive 3D graphics and games (I3D '05). ACM, New York, NY, USA, 111--116.
33. Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The go-go interaction technique: non-linear mapping for direct manipulation in VR. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '96). ACM, New York, NY, USA, 79--80.
34. Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998. The Office of the Future: A Unified Approach to Image-based Modeling and Spatially Immersive Displays. In Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). ACM, New York, NY, USA, 179--188.
35. Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. 2001. Shader lamps: animating real objects with image-based illumination. In Proceedings of the Eurographics conference on Rendering (EGWR'01). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 89--101.
36. Majken K. Rasmussen, Esben W. Pedersen, Marianne G. Petersen, and Kasper Hornbæk. 2012. Shape-changing Interfaces: A Review of the Design Space and Open Research Questions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 735--744.
37. Sharif Razzaque, Zachariah Kohn, and Mary C. Whitton. 2001. Redirected Walking. In Proceedings of Eurographics (EG '01), 289--294.
38. Jun Rekimoto and Katashi Nagao. 1995. The world through the computer: computer augmented interaction with real world environments. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '95). ACM, New York, NY, USA, 29--36.
39. Joan Sol Roo, Martin Hachet. Towards a Hybrid Space Combining Spatial Augmented Reality and Virtual Reality. In Proceedings of the 2017 IEEE Symposium on 3D User Interfaces (3DUI '17), Los Angeles, CA, USA, 195--198.
40. Joan Sol Roo, Martin Hachet. One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA.
41. Claus Scheiblauer, Michael Wimmer, Out-of-core selection and editing of huge point clouds. 2011. Computers & Graphics, Volume 35, Issue 2, Pages 342--351, ISSN 0097--8493.
42. Ian Stavness, Billy Lam, and Sidney Fels. 2010. pCubee: A Perspective-corrected Handheld Cubic Display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1381--1390.
43. Vikas Singh, Deborah Silver and Nicu D. Cornea. 2003. Real-time volume manipulation. In Proceedings of the Eurographics/IEEE TVCG Workshop on Volume graphics (VG '03). ACM, New York, NY, USA, 45--51.
44. Adriana De Souza E Silva. 2009. Hybrid Reality and Location-Based Gaming: Redefining Mobility and Game Spaces in Urban Environments. Simul. Gaming 40, 3 (June 2009), 404--424.
45. Misha Sra, Sergio Garrido-Jurado, Chris Schmandt, and Pattie Maes. 2016. Procedurally generated virtual reality from 3D reconstructed physical space. In Proceedings of the ACM Conference on Virtual Reality Software and Technology (VRST '16). ACM, New York, NY, USA, 191--200.
46. Richard Stoakley, Matthew J. Conway, and Randy Pausch. 1995. Virtual reality on a WIM: interactive worlds in miniature. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM, New York, NY, USA, 265--272.
47. John Underkoffler and Hiroshi Ishii. 1998. Illuminating Light: An Optical Design Tool with a Luminous-tangible Interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98). ACM, New York, NY, USA, 542--549.
48. John Underkoffler and Hiroshi Ishii. 1999. Urp: A Luminous-tangible Workbench for Urban Planning and Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 386--393.
49. Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, and Philip Torr. 2015. SemanticPaint: Interactive 3D Labeling and Learning at Your Fingertips. ACM Trans. Graph. 34, 5, Article 154.
50. John Viega, Matthew J. Conway, George Williams, and Randy Pausch. 1996. 3D magic lenses. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '96). ACM, New York, NY, USA, 51--58.
51. Andrew D. Wilson. 2017. Fast lossless depth image compression. In Proceedings of ACM Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA.
52. Ya-Ting Yue, Yong-Liang Yang, Gang Ren, and Wenping Wang. 2017. SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA.

Index Terms

• Human-centered computing