Performance relighting and reflectance transformation with time-multiplexed illumination

ABSTRACT

We present a technique for capturing an actor's live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-multiplexed basis lighting conditions, and to record these conditions with a high-speed video camera so that many conditions are recorded in the span of the desired output frame interval. We investigate several lighting bases for representing the sphere of incident illumination using a set of discrete LED light sources, and we estimate and compensate for subject motion using optical flow and image warping based on a set of tracking frames inserted into the lighting basis. To composite the illuminated performance into a new background, we include a time-multiplexed matte within the basis. We also show that the acquired data enables time-varying surface normals, albedo, and ambient occlusion to be estimated, which can be used to transform the actor's reflectance to produce both subtle and stylistic effects.

REFERENCES

• 1: Black, M. J., and Anandan, P. 1993. A framework for the robust estimation of optical flow. In Fourth International Conf. on Computer Vision, 231--236.
• 2: Gabriel J. Brostow , Irfan Essa, Image-based motion blur for stop motion animation, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.561-566, August 2001 [doi>10.1145/383259.383325]
• 3: Calahan, S. 2000. Advanced Renderman: Creating CGI for Motion Pictures. Morgan Kaufman Publishers, San Francisco, ch. Storytelling through lighting, a computer perspective, 337382.
• 4: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 5: Paul Debevec , Andreas Wenger , Chris Tchou , Andrew Gardner , Jamie Waese , Tim Hawkins, A lighting reproduction approach to live-action compositing, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 6: Fielding, R. 1985. The Technique of Special Effects Cinematography, 4th ed. Hastings House, New York.
• 7: Athinodoros S. Georghiades , Peter N. Belhumeur , David J. Kriegman, Illumination-Based Image Synthesis: Creating Novel Images of Human Faces Under Differing Pose and Lighting, Proceedings of the IEEE Workshop on Multi-View Modeling & Analysis of Visual Scenes, p.47, June 21-23, 1999
• 8: Athinodoros S. Georghiades, Recovering 3-D shape and reflectance from a small number of photographs, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 9: Reid Gershbein , Pat Hanrahan, A fast relighting engine for interactive cinematic lighting design, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.353-358, July 2000 [doi>10.1145/344779.344938]
• 10: Ned Greene, Environment mapping and other applications of world projections, IEEE Computer Graphics and Applications, v.6 n.11, p.21-29, November 1986 [doi>10.1109/MCG.1986.276658]
• 11: Brian Guenter , Cindy Grimm , Daniel Wood , Henrique Malvar , Fredric Pighin, Making faces, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.55-66, July 1998 [doi>10.1145/280814.280822]
• 12: Haeberli, P. 1992. Synthetic lighting for photography. Available at http://www.sgi.com/grafica/synth/index.html, January.
• 13: G. D. Hager , P. N. Belhumeur, Real-time tracking of image regions with changes in geometry and illumination, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.403, June 18-20, 1996
• 14: Marshall Hall, Jr., Combinatorial theory (2nd ed.), John Wiley & Sons, Inc., New York, NY, 1998
• 15: Harwit, M., and Sloane, N. J. A. 1979. Hadamard transform optics. Academic Press, New York.
• 16: Hawkins, T., Wenger, A., Tchou, C., and Debevec, A. G. F. G. P. 2004. Animatable facial reflectance fields. In Eurographics Symposium on Rendering: 15th Eurographics Workshop on Rendering.
• 17: Hewlett-Packard Components Group. 1998. Noise sources in cmos image sensors. Tech. rep., Hewlett-Packard.
• 18: Sing Bing Kang , Matthew Uyttendaele , Simon Winder , Richard Szeliski, High dynamic range video, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 19: Landis, H., 2002. Production-ready global illumination. Course Notes for SIGGRAPH 2002 Course 16, RenderMan in Production.
• 20: Tom Malzbender , Dan Gelb , Hans Wolters, Polynomial texture maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.519-528, August 2001 [doi>10.1145/383259.383320]
• 21: J. Marks , B. Andalman , P. A. Beardsley , W. Freeman , S. Gibson , J. Hodgins , T. Kang , B. Mirtich , H. Pfister , W. Ruml , K. Ryall , J. Seims , S. Shieber, Design galleries: a general approach to setting parameters for computer graphics and animation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.389-400, August 1997 [doi>10.1145/258734.258887]
• 22: Stephen R. Marschner , Brian K. Guenter , Sashi Raghupathy, Modeling and Rendering for Realistic Facial Animation, Proceedings of the Eurographics Workshop on Rendering Techniques 2000, p.231-242, June 26-28, 2000
• 23: Masselus, V., Peers, P., Dutre, P., and Willems, Y. D. 2004. Smooth reconstruction and compact representation of reflectance functions for image-based relighting. In 15th Eurographics Symposium on Rendering, no. Norrkoping, Sweden.
• 24: Miller, G. S., and Hoffman, C. R. 1984. Illumination and reflection maps: Simulated objects in simulated and real environments. In SIGGRAPH 84 Course Notes for Advanced Computer Graphics Animation.
• 25: Ren Ng , Ravi Ramamoorthi , Pat Hanrahan, All-frequency shadows using non-linear wavelet lighting approximation, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 26: Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994, Efficient re-rendering of naturally illuminated environments. In Fifth Eurographics Workshop on Rendering, 359--373.
• 27: Ko Nishino , Shree K. Nayar, Eyes for relighting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 28: Lena Petrović , Brian Fujito , Lance Williams , Adam Finkelstein, Shadows for cel animation, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.511-516, July 2000 [doi>10.1145/344779.345073]
• 29: Thomas Porter , Tom Duff, Compositing digital images, ACM SIGGRAPH Computer Graphics, v.18 n.3, p.253-259, July 1984
• 30: Ramesh Raskar , Kar-Han Tan , Rogerio Feris , Jingyi Yu , Matthew Turk, Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 31: Szymon Rusinkiewicz , Olaf Hall-Holt , Marc Levoy, Real-time 3D model acquisition, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 32: Yoav Y. Schechner , Shree K. Nayar , Peter N. Belhumeur, A Theory of Multiplexed Illumination, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.808, October 13-16, 2003
• 33: Peter-Pike Sloan , Jan Kautz , John Snyder, Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 34: Sloan, N. J. A., 1999. A library of hadamard matrices. http://www.research.att.com/~njas/hadamard/.
• 35: Alvy Ray Smith , James F. Blinn, Blue screen matting, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.259-268, August 1996 [doi>10.1145/237170.237263]
• 36: Trumbull, D. 2000. Personal communication. January.
• 37: Wen, Z., Liu, Z., and Huang, T. S. 2003. Face relighting with radiance environment maps. In 2003 Conference on Computer Vision and Pattern Recognition (CVPR 2003), 158--165.
• 38: Williams, L. 1991. Shading in two dimensions. In Graphics Interface '91, 143--151.
• 39: Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1, 139--144.
• 40: Xiaolin Wu, An efficient antialiasing technique, Proceedings of the 18th annual conference on Computer graphics and interactive techniques, p.143-152, July 1991
• 41: Li Zhang , Noah Snavely , Brian Curless , Steven M. Seitz, Spacetime faces: high resolution capture for modeling and animation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 42: Douglas E. Zongker , Dawn M. Werner , Brian Curless , David H. Salesin, Environment matting and compositing, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.205-214, July 1999 [doi>10.1145/311535.311558]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking

Keywords:
compositing, environmental illumination, image-based rendering, reflectance models, relighting