Article

Consistent interactive augmentation of live camera images with correct near-field illumination

Authors:

Thorsten Grosch,

Stefan MuellerAuthors Info & Claims

VRST '07: Proceedings of the 2007 ACM symposium on Virtual reality software and technology

Pages 125 - 132

https://doi.org/10.1145/1315184.1315207

Published: 05 November 2007 Publication History

Abstract

Inserting virtual objects in real camera images with correct lighting is an active area of research. Current methods use a high dynamic range camera with a fish-eye lens to capture the incoming illumination. The main problem with this approach is the limitation to distant illumination. Therefore, the focus of our work is a real-time description of both near - and far-field illumination for interactive movement of virtual objects in the camera image of a real room. The daylight, which is coming in through the windows, produces a spatially varying distribution of indirect light in the room; therefore a near-field description of incoming light is necessary. Our approach is to measure the daylight from outside and to simulate the resulting indirect light in the room. To accomplish this, we develop a special dynamic form of the irradiance volume for real-time updates of indirect light in the room and combine this with importance sampling and shadow maps for light from outside. This separation allows object movements with interactive frame rates (10--17 fps). To verify the correctness of our approach, we compare images of synthetic objects with real objects.

References

[1]

Agusanto, K., Li, L., Chuangui, Z., and Sing, N. W. 2003. Photorealistic rendering for augmented reality using environment illumination. In ISMAR, 208--216.

Digital Library

[2]

Annen, T., Kautz, J., Durand, F., and Seidel, H.-P. 2004. Spherical Harmonic Gradients for Mid-Range Illumination. In Proceedings of the Eurographics Symposium on Rendering, 331--336.

Digital Library

[3]

Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. SIGGRAPH '98, 189--198.

Digital Library

[4]

Dmitriev, K., Annen, T., Krawczyk, G., Myszkowski, K., and Seidel, H.-P. 2004. A cave system for interactive modeling of global illumination in car interior. In ACM Symposium on Virtual Reality Software and Technology (VRST 2004), ACM, Hong Kong, R. Lau and G. Baciu, Eds., 137--145.

Digital Library

[5]

Drettakis, G., Robert, L., and Bougnoux, S. 1997. Interactive common illumination for computer augmented reality. In Eighth Eurographics Workshop on Rendering, 45--56.

Digital Library

[6]

Fournier, A., Gunavan, A., and Romanzin, C. 1993. Common illumination between real and computer generated scenes. In Proc. of Graphics Interface 1993, 254--262.

[7]

Gibson, S., and Howard, T. 2000. Interactive reconstruction of virtual environments from photographs, with application to scene-of-crime analysis. In Proc. of ACM Symposium on Virtual Reality Software and Technology 2000, Seoul, Korea, October 2000, 41--48.

Digital Library

[8]

Gibson, S., and Murta, A. 2000. Interactive rendering with real-world illumination. In Eleventh Eurographics Workshop on Rendering, 365--376.

Digital Library

[9]

Gibson, S., Cook, J., Howard, T., and Hubbold, R. 2003. Rapid shadow generation in real-world lighting environments. In Eurographics Symposium on Rendering, 219--229.

Digital Library

[10]

Goesele, M., Granier, X., Heidrich, W., and Seidel, H.- P. 2003. Accurate light source acquisition and rendering. In SIGGRAPH '03: ACM SIGGRAPH 2003 Papers, ACM Press, New York, NY, USA, 621--630.

Digital Library

[11]

Greger, G., Shirley, P., Hubbard, P. M., and Greenberg, D. P. 1998. The irradiance volume. IEEE Computer Graphics and Applications 18, 2, 32--43.

Digital Library

[12]

Grosch, T., Mueller, S., and Kresse, W. 2003. Goniometric light reconstruction for augmented reality image synthesis. In Graphiktag im Rahmen der GI Jahrestagung, Frankfurt am Main,.

[13]

Grosch, T. 2005. Differential photon mapping: Consistent augmentation of photographs with correction of all light paths. In Eurographics 2005, Trinity College, Dublin, Ireland.

[14]

Grosch, T. 2005. PanoAR: Interactive augmentation of omnidirectional images with consistent lighting. In Mirage 2005, Computer Vision / Computer Graphics Collaboration Techniques and Applications, 25--34.

[15]

Haller, M., Drab, S., and Hartmann, W. 2003. A real-time shadow approach for an augmented reality application using shadow volumes. In VRST '03: Proceedings of the ACM symposium on Virtual reality software and technology, ACM Press, New York, NY, USA, 56--65.

Digital Library

[16]

Hartley, R., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision. Cambridge University Press.

Digital Library

[17]

Hasan, M., Pellacini, F., and Bala, K. 2006. Direct-to-Indirect Transfer for Cinematic Relighting. ACM Trans. Graph. 25, 3, 1089--1097.

Digital Library

[18]

Havran, V., Smyk, M., Krawczyk, G., Myszkowski, K., and Seidel, H.-P. 2005. Importance Sampling for Video Environment Maps. In Eurographics Symposium on Rendering 2005, ACM SIGGRAPH, Konstanz, Germany, K. Bala and P. Dutré, Eds., 31--42.

Digital Library

[19]

Hoefflinger, B. 2007. High-Dynamic-Range (HDR) Vision: Microelectronics, Image Processing, Computer Graphics (Springer Series in Advanced Microelectronics). Springer-Verlag New York, Inc., Secaucus, NJ, USA.

Digital Library

[20]

Kanbara, M., and Yokoya, N. 2002. Geometric and photometric registration for real-time augmented reality. In In IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR) (September 2002), 279.

Digital Library

[21]

Kanbara, M., and Yokoya, N. 2004. Real-time estimation of light source environment for photorealistic augmented reality. In ICPR (2), 911--914.

Digital Library

[22]

Kato, H., and Billinghurst, M. 1999. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In Proceedings of the 2nd International Workshop on Augmented Reality (IWAR 99).

Digital Library

[23]

Kontkanen, J., and Laine, S. 2006. Sampling Precomputed volumetric lighting. Journal of Graphics Tools 11, 3.

[24]

Kontkanen, J., Turquin, E., Holzschuch, N., and Sillion, F. X. 2006. Wavelet Radiance Transport for Interactive Indirect Lighting. In 17th Eurographics Symposium on Rendering, 161--172.

Digital Library

[25]

Korn, M., Stange, M., Von Arb, A., Blum, L., Kreil, M., Kunze, K., Anhenn, J., Wallrath, T., and Grosch, T. 2006. Interactive augmentation of live images using a HDR stereo camera. In Dritter Workshop Virtuelle und Erweiterte Realitt der GI-Fachgruppe VR/AR, vol. 3, 107--118.

[26]

Krawczyk, G., Goesele, M., and Seidel, H.-P. 2005. Photometric calibration of high dynamic range cameras. Research Report MPI-I-2005-4-005, Max-Planck-Institut für Informatik, Stuhlsatzenhausweg 85, 66123 Saarbrücken, Germany, April.

[27]

Kristensen, A. W., Akenine-Mller, T., and Jensen, H. W. 2005. Precomputed Local Radiance Transfer for Real-Time Lighting Design. ACM Trans. Graph. 24, 3, 1208--1215.

Digital Library

[28]

Loscos, C., Frasson, M., Drettakis, G., Walter, B., Granier, X., and Poulin, P. 1999. Interactive virtual relighting and remodeling of real scenes. In Tenth Eurographics Workshop on Rendering, 329--340.

Digital Library

[29]

Mantiuk, R., Pattanaik, S., and Myszkowski, K. 2002. Cube-map data structure for interactive global illumination computation in dynamic diffuse environments. In ICCVG.

[30]

Nakamae, E., Harada, K., Ishizaki, T., and Nishita, T. 1986. A montage method: The overlaying of the computer generated images onto a background photograph. In Computer Graphics (Proceedings of SIGGRAPH 86), 207--214.

Digital Library

[31]

Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM Trans. Graph. 22, 3, 376--381.

Digital Library

[32]

Nijasure, M., Pattanaik, S. N., and Goel, V. 2005. Real-time global illumination on GPUs. journal of graphics tools 10, 2, 55--71.

[33]

Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994. Efficient Re-rendering of Naturally Illuminated Environments. In Fifth Eurographics Workshop on Rendering, Springer-Verlag, Darmstadt, Germany, 359--373.

[34]

Perez, R., Seals, R., and Michalsky, J. 1993. All-weather model for sky luminance distribution-preliminary configuration and validation. In Solar Energy 50, 3, 235--245.

[35]

Pharr, M., and Humphreys, G. 2004. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann, August.

Digital Library

[36]

Ramamoorthi, R., and Hanrahan, P. 2001. An efficient representation for irradiance environment maps. In SIGGRAPH, 497--500.

Digital Library

[37]

Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers, Dec.

Digital Library

[38]

Ritschel, T., and Grosch, T. 2006. On-line estimation of diffuse materials. In Dritter Workshop Virtuelle und Erweiterte Realitt der GI-Fachgruppe VR/AR, vol. 3, 95--106.

[39]

Sato, I., Sato, Y., and Ikeuchi, K. 1999. Acquiring a radiance distribution to superimpose virtual objects onto a real scene. IEEE Trans. Vis. Comput. Graph. 5, 1, 1--12.

Digital Library

[40]

Sillion, F. X., and Puech, C. 1994. Radiosity and Global Illumination. Morgan Kaufmann Publishers, Inc., San Francisco, California.

Digital Library

[41]

Sloan, P.-P. J., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Trans. Graph. 21, 3, 527--536.

Digital Library

[42]

Williams, L. 1978. Casting curved shadows on curved surfaces. In SIGGRAPH '78: Proceedings of the 5th annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY, USA, 270--274.

Digital Library

Cited By

Xu YJiang YWang SLi KGeng G(2023)Delta Path Tracing for Real-Time Global Illumination in Mixed Reality2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00020(44-52)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00020
Iketani SImura M(2022)Photorealistic Augmented Reality Image Generation with Generative Adversarial Network Focusing on Structural Edge3次元構造エッジを重視したGANによる写実的AR画像の生成Transactions of the Institute of Systems, Control and Information Engineers10.5687/iscie.35.13335:6(133-144)Online publication date: 15-Jun-2022
https://doi.org/10.5687/iscie.35.133
Zhao YMa CHuang HGuo T(2022)LITARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35502916:3(1-29)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550291
Show More Cited By

Index Terms

Consistent interactive augmentation of live camera images with correct near-field illumination
1. Computing methodologies
  1. Computer graphics

Recommendations

Ray tracing-based interactive diffuse indirect illumination

Despite great efforts in recent years to accelerate global illumination computation, the real-time ray tracing of fully dynamic scenes to support photorealistic indirect illumination effects has yet to be achieved in computer graphics. In this paper, we ...
Splatting indirect illumination
I3D '06: Proceedings of the 2006 symposium on Interactive 3D graphics and games

In this paper we present a novel method for plausible real-time rendering of indirect illumination effects for diffuse and non-diffuse surfaces. The scene geometry causing indirect illumination is captured by an extended shadow map, as proposed in ...
Interactive Display of Isosurfaces with Global Illumination

In many applications, volumetric data sets are examined by displaying isosurfaces, surfaces where the data, or some function of the data, takes on a given value. Interactive applications typically use local lighting models to render such surfaces. This ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '07: Proceedings of the 2007 ACM symposium on Virtual reality software and technology

November 2007

259 pages

ISBN:9781595938633

DOI:10.1145/1315184

Conference Chairs:
Aditi Majumder
UC Irvine
,
Larry Hodges
UNC - Charlotte
,
Daniel Cohen-Or
Tel Aviv University
,
Editor:
Stephen N. Spencer
University of Washington

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

VRST07

Sponsor:

VRST07: The ACM Symposium on Virtual Reality Software and Technology

November 5 - 7, 2007

California, Newport Beach

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
396
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu YJiang YWang SLi KGeng G(2023)Delta Path Tracing for Real-Time Global Illumination in Mixed Reality2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00020(44-52)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00020
Iketani SImura M(2022)Photorealistic Augmented Reality Image Generation with Generative Adversarial Network Focusing on Structural Edge3次元構造エッジを重視したGANによる写実的AR画像の生成Transactions of the Institute of Systems, Control and Information Engineers10.5687/iscie.35.13335:6(133-144)Online publication date: 15-Jun-2022
https://doi.org/10.5687/iscie.35.133
Zhao YMa CHuang HGuo T(2022)LITARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35502916:3(1-29)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550291
Dai DShi XWang LLi X(2022)Interactive Mixed Reality Rendering on Holographic Pyramid2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR51125.2022.00068(483-492)Online publication date: Mar-2022
https://doi.org/10.1109/VR51125.2022.00068
Gomes CMeyers BDenil JThule CLausdahl KVangheluwe HDe Meulenaere P(2020)Semantic adaptation for FMI co-simulation with hierarchical simulatorsSimulation10.1177/003754971875977595:3(241-269)Online publication date: 18-Jun-2020
https://dl.acm.org/doi/10.1177/0037549718759775
Chatterjee DRao S(2020)Computational SustainabilityACM Computing Surveys10.1145/340979753:5(1-29)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3409797
Álvarez-García JCvetković BLuštrek M(2020)A Survey on Energy Expenditure Estimation Using Wearable DevicesACM Computing Surveys10.1145/340448253:5(1-35)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3404482
Ismail LMaterwala H(2020)Computing Server Power Modeling in a Data CenterACM Computing Surveys10.1145/339060553:3(1-34)Online publication date: 12-Jun-2020
https://dl.acm.org/doi/10.1145/3390605
Prenkaj BVelardi PStilo GDistante DFaralli S(2020)A Survey of Machine Learning Approaches for Student Dropout Prediction in Online CoursesACM Computing Surveys10.1145/338879253:3(1-34)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3388792
Beaumont OCanon LEyraud-Dubois LLucarelli GMarchal LMommessin CSimon BTrystram D(2020)Scheduling on Two Types of ResourcesACM Computing Surveys10.1145/338711053:3(1-36)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3387110
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten