Deep image-based relighting from optimal sparse samples

Abstract

We present an image-based relighting method that can synthesize scene appearance under novel, distant illumination from the visible hemisphere, from only five images captured under pre-defined directional lights. Our method uses a deep convolutional neural network to regress the relit image from these five images; this relighting network is trained on a large synthetic dataset comprised of procedurally generated shapes with real-world reflectances. We show that by combining a custom-designed sampling network with the relighting network, we can jointly learn both the optimal input light directions and the relighting function. We present an extensive evaluation of our network, including an empirical analysis of reconstruction quality, optimal lighting configurations for different scenarios, and alternative network architectures. We demonstrate, on both synthetic and real scenes, that our method is able to reproduce complex, high-frequency lighting effects like specularities and cast shadows, and outperforms other image-based relighting methods that require an order of magnitude more images.

References

1. Aayush Bansal, Bryan Russell, and Abhinav Gupta. 2016. Marr Revisited: 2D-3D Model Alignment via Surface Normal Prediction. CVPR (2016).
2. Ronen Basri and David W. Jacobs. 2003. Lambertian Reflectance and Linear Subspaces. IEEE Trans. Pattern Anal Mach. Intell. 25, 2 (Feb. 2003), 218--233.
3. Peter N. Belhumeur and David J. Kriegman. 1998. What Is the Set of Images of an Object Under All Possible Illumination Conditions? International Journal of Computer Vision 28, 3 (01 Jul 1998), 245--260.
4. Brett Burley. 2012. Physically-based shading at Disney. In ACM SIGGRAPH 2012 Courses.
5. Ayan Chakrabarti. 2016. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems. 3081--3089.
6. Manmohan Chandraker. 2016. The information available to a moving observer on shape with unknown, isotropic BRDFs. IEEE transactions on pattern analysis and machine intelligence 38, 7 (2016), 1283--1297.
7. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 145--156.
8. David Eigen and Rob Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. ICCV (2015).
9. Per Einarsson, Charles-Felix Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond, Tim Hawkins, Mark T Bolas, Sebastian Sylwan, and Paul E Debevec. 2006. Relighting Human Locomotion with Flowed Reflectance Fields. Rendering techniques 2006 (2006), 17th.
10. John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views From the World's Imagery. In CVPR.
11. Martin Fuchs, Volker Blanz, Hendrik Lensch, and Hans-Peter Seidel. 2007. Adaptive sampling of reflectance fields. ACM Transactions on Graphics (TOG) 26, 2 (2007), 10.
12. Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to Predict Indoor Illumination from a Single Image. ACM Transactions on Graphics (SIGGRAPH Asia) 9, 4 (2017).
13. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Tinne Tuytelaars, and Luc Van Gool. 2017. What Is Around The Camera?. In ICCV.
14. Dan B Goldman, Brian Curless, Aaron Hertzmann, and Steven M Seitz. 2010. Shape and spatially-varying brdfs from photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 6 (2010), 1060--1071.
15. Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In CVPR.
16. Z. Hui and A. C. Sankaranarayanan. 2017. Shape and Spatially-Varying Reflectance Estimation from Virtual Exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 10 (Oct 2017), 2060--2073.
17. Wenzel Jakob. 2010. Mitsuba renderer. (2010). http://www.mitsuba-renderer.org.
18. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 193.
19. Eric P Lafortune and Yves D Willems. 1993. Bi-directional path tracing. (1993).
20. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Transactions on Graphics (TOG) 36, 4 (2017), 45.
21. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. In ICCV. 2261--2269.
22. Dhruv Mahajan, Ira Kemelmacher Shlizerman, Ravi Ramamoorthi, and Peter Belhumeur. 2007. A Theory of Locally Low Dimensional Light Transport. ACM Trans. Graph. 26, 3, Article 62 (July 2007).
23. Tom Malzbender, Dan Gelb, and Hans Wolters. 2001. Polynomial Texture Maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). 519--528.
24. Wojciech Matusik, Matthew Loper, and Hanspeter Pfister. 2004. Progressively-Refined Reflectance Functions from Natural Illumination. In Eurographics Workshop on Rendering, Alexander Keller and Henrik Wann Jensen (Eds.).
25. Shree K. Nayar, Peter N. Belhumeur, and Terry E. Boult. 2004. Lighting Sensitive Display. ACM Trans. Graph. 23, 4 (Oct. 2004), 963--979.
26. Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2003. All-frequency shadows using non-linear wavelet lighting approximation. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 376--381.
27. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On optimal, minimal BRDF sampling for reflectance acquisition. ACM Transactions on Graphics (TOG) 34, 6(2015), 186.
28. Geoffrey Oxholm and Ko Nishino. 2016. Shape and reflectance estimation in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 38, 2 (2016), 376--389.
29. Pieter Peers and Philip Dutré. 2005. Inferring reflectance functions from wavelet noise. In Proceedings of the Sixteenth Eurographics conference on Rendering Techniques. Eurographics Association, 173--182.
30. Pieter Peers, Dhruv K Mahajan, Bruce Lamond, Abhijeet Ghosh, Wojciech Matusik, Ravi Ramamoorthi, and Paul Debevec. 2009. Compressive light transport sensing. ACM Transactions on Graphics (TOG) 28, 1 (2009), 3.
31. Ravi Ramamoorthi and Pat Hanrahan. 2001. On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. J. Opt. Soc. Am. A 18, 10 (Oct 2001), 2448--2459.
32. Dikpal Reddy, Ravi Ramamoorthi, and Brian Curless. 2012. Frequency-space Decomposition and Acquisition of Light Transport Under Spatially Varying Illumination. In Proceedings of the 12th European Conference on Computer Vision - Volume Part VI (ECCV'12). Springer-Verlag, Berlin, Heidelberg, 596--610.
33. Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep reflectance maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4508--4516.
34. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Transactions on Graphics (TOG) 34, 4 (2015), 111.
35. O. Ronneberger, P.Fischer, and T. Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI) (LNCS), Vol. 9351. Springer, 234--241.
36. Christopher Schwartz, Michael Weinmann, Roland Ruiters, and Reinhard Klein. 2011. Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage. In VAST. Eurographics Association, 25--32.
37. Amnon Shashua. 1997. On Photometric Issues in 3D Visual Recognition from a Single 2D Image. International Journal of Computer Vision 21, 1 (01 Jan 1997), 99--122.
38. Peter-Pike Sloan, Jesse Hall, John Hart, and John Snyder. 2003. Clustered Principal Components for Precomputed Radiance Transfer. ACM Trans. Graph. 22, 3 (July 2003), 382--391.
39. Kalyan Sunkavalli, Todd Zickler, and Hanspeter Pfister. 2010. Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows. In ECCV. 251--264.
40. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 195--206.
41. Jiaping Wang, Yue Dong, Xin Tong, Zhouchen Lin, and Baining Guo. 2009. Kernel Nyström method for light transport. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 29.
42. Robert J. Woodham. 1980. Photometric Method For Determining Surface Orientation From Multiple Images. Optical Engineering 19 (1980), 19 - 19 - 6.
43. Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal BRDF sampling for two-shot near-field reflectance acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 188.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering