Article

Eyes for relighting

Authors:

Shree K. NayarAuthors Info & Claims

SIGGRAPH '04: ACM SIGGRAPH 2004 Papers

Pages 704 - 711

https://doi.org/10.1145/1186562.1015783

Published: 01 August 2004 Publication History

Abstract

The combination of the cornea of an eye and a camera viewing the eye form a catadioptric (mirror + lens) imaging system with a very wide field of view. We present a detailed analysis of the characteristics of this corneal imaging system. Anatomical studies have shown that the shape of a normal cornea (without major defects) can be approximated with an ellipsoid of fixed eccentricity and size. Using this shape model, we can determine the geometric parameters of the corneal imaging system from the image. Then, an environment map of the scene with a large field of view can be computed from the image. The environment map represents the illumination of the scene with respect to the eye. This use of an eye as a natural light probe is advantageous in many relighting scenarios. For instance, it enables us to insert virtual objects into an image such that they appear consistent with the illumination of the scene. The eye is a particularly useful probe when relighting faces. It allows us to reconstruct the geometry of a face by simply waving a light source in front of the face. Finally, in the case of an already captured image, eyes could be the only direct means for obtaining illumination information. We show how illumination computed from eyes can be used to replace a face in an image with another one. We believe that the eye not only serves as a useful tool for relighting but also makes relighting possible in situations where current approaches are hard to use.

Supplemental Material

MOV File

Download
11.84 KB

References

[1]

AGARWAL, S., RAMAMOORTHI, R., BELONGIE, S., AND JENSEN, H. 2003. Structured Importance Sampling of Environment Maps. In Proc. of ACM SIGGRAPH 2003, 605--612.

Digital Library

[2]

BAKER, S., AND NAYAR, S. 1999. A Theory of Single-Viewpoint Catadioptric Image Formation. IJCV 35, 2 (Nov.), 1--22.

Digital Library

[3]

BAKER, T. 1943. Ray tracing through non-spherical surfaces. Proc. of The Royal Society of London 55, 361--364.

[4]

BARSKY, B., BARGTEIL, A., GARCIA, D., AND KLEIN, S. 2002. Introducing Vision-Realistic Rendering. In Proc. of EGWR 2002, 1--7.

[5]

BARSKY, B. 2003. Geometry for Analysis of Corneal Shape. In Computational Geometry: Lectures at Morningside Center of Mathematics. Amer Mathematical Society, 33--56.

[6]

BLANZ, V., AND VETTER, T. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proc. of ACM SIGGRAPH 99.

Digital Library

[7]

BOIVIN, S., AND GAGALOWICZ, A. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. of ACM SIGGRAPH 2001, 197--116.

Digital Library

[8]

BOLT, R. 1982. Eyes at the Interface. In Proc. of ACM CHI 82, 360--362.

Digital Library

[9]

BURKHARD, D., AND SHEALY, D. 1973. Flux Density for Ray Propoagation in Gemoetrical Optics. JOSA 63, 3 (Mar.), 299--304.

[10]

CORNBLEET, S. 1984. Microwave and Optical Ray Geometry. John Wiley and Sons.

[11]

DAUGMAN, J. 1993. High Confidence Visual Recognition of Persons by a Test of Statistical Independence. IEEE TPAMI 15, 11 (Nov.), 1148--1161.

Digital Library

[12]

DEBEVEC, P., AND MALIK, J. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. of ACM SIGGRAPH 97, 369--378.

Digital Library

[13]

DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., AND SAROKIN, W. 2000. Acquiring the Reflectance Field of a Human Face. In Proc. of ACM SIGGRAPH 2000, 145--156.

Digital Library

[14]

DEBEVEC, P. 1998. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography. In Proc. of ACM SIGGRAPH 98, 189--198.

Digital Library

[15]

FLOM, L., AND SAFIR, A., 1987. Iris Recognition System. US patent 4,641,349.

[16]

FRANKOT, R., AND CHELLAPPA, R. 1988. A Method for Enforcing Integrability in Shape from Shading Algorithms. IEEE TPAMI 10, 4, 439--451.

Digital Library

[17]

GEORGHIADES, A. 2003. Recovering 3-D Shape and Reflectance From a Small Number of Photographs. In Proc. of EGSR 2003, 230--240.

Digital Library

[18]

HALSTEAD, M., BARSKY, B., KLEIN, S., AND MANDELL, R. 1996. Reconstructing Curved Surfaces From Specular Reflection Patterns Using Spline Surface Fitting of Normals. In Proc. of ACM SIGGRAPH 96, 335--342.

Digital Library

[19]

HUTCHINSON, T., WHITE, K., REICHERT, K., AND FREY, L. 1989. Human-computer Interaction using Eye-gaze Input. IEEE TSMC 19 (Nov./Dec.), 1527--1533.

[20]

JACOB, R. 1990. What You Look At is What You Get: Eye Movement-Based Interaction Techniques. In Proc. of ACM CHI 90, 11--18.

Digital Library

[21]

KAUFMAN, P., AND ALM, A., Eds. 2003. Adler's Physiology of the Eye: Clinical Application, 10th ed. Mosby.

[22]

LEFOHN, A., CARUSO, R., REINHARD, E., BUDGE, B., AND SHIRLEY, P. 2003. An Ocularist's Approach to Human Iris Synthesis. IEEE CGA 23, 6 (Nov./Dec.), 70--75.

Digital Library

[23]

LENSCH, H., KAUTZ, J., GOESELE, M., HEIDRICH, W., AND SEIDEL, H.-P. 2001. Image-Based Reconstruction of Spatially Varying Materials. In Proc. of EGWR 2001, 104--115.

Digital Library

[24]

LIN, Z., WONG, T.-T., AND SHUM, H.-Y. 2001. Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction. In Proc. of IEEE CVPR 2001, vol. 1, 561--567.

[25]

MAGDA, S., KRIEGMAN, D., ZICKLER, T., AND BELHUMEUR, P. 2001. Beyond Lambert: Reconstructing Surfaces with Arbitrary BRDFs. In Proc. of IEEE ICCV 01, vol. II, 391--398.

[26]

MARSCHNER, S., AND GREENBERG, D. 1997. Inverse Lighting for Photography. In Proc. of IS&T/SID CIC, 262--265.

[27]

MATUSIK, W., PFISTER, H., NGAN, A., BEARDSLEY, P., ZIEGLER, R., AND MCMILLAN, L. 2002. Image-based 3D Photography using Opacity Hulls. In Proc. of ACM SIGGRAPH 2002, 427--437.

Digital Library

[28]

MITSUNAGA, T., AND NAYAR, S. 1999. Radiometric Self Calibration. In Proc. of IEEE CVPR 99, vol. 1, 1374--1380.

[29]

MOSTAFAWY, S., KERMANI, O., AND LUBATSCHOWSKI, H. 1997. Virtual Eye: Retinal Image Visualization of the Human Eye. IEEE CGA 17, 1 (Jan./Feb.), 8--12.

Digital Library

[30]

NAYAR, S., AND MITSUNAGA, T. 2000. High Dynamic Range Imaging: Spatially Varying Pixel Exposures. In Proc. of IEEE CVPR 00, 1472--1479.

[31]

NISHINO, K., AND NAYAR, S. 2004. Corneal imaging system: Environment from eyes. Tech. rep., Dept. of Computer Science, Columbia University. In preparation.

[32]

RAMAMOORTHI, R., AND HANRAHAN, P. 2001. A Signal-Processing Framework for Inverse Rendering. In Proc. of ACM SIGGRAPH 2001.

Digital Library

[33]

SAGAR, M., BULLIVANT, D., MALLINSON, G., AND HUNTER, P. 1994. A Virtual Environment and Model of the Eye for Surgical Simulation. In Proc. of ACM SIGGRAPH 94, 205--212.

Digital Library

[34]

SATO, Y., WHEELER, M., AND IKEUCHI, K. 1997. Object shape and reflectance modeling from observation. In Proc. of ACM SIGGRAPH 97, 379--387.

Digital Library

[35]

SATO, I., SATO, Y., AND IKEUCHI, K. 2003. Illumination from Shadows. IEEE TPAMI 25, 3, 290--300.

Digital Library

[36]

STEIN, C. 1995. Accurate Internal Camera Calibration using Rotation, with Analysis of Sources of Errors. In Proc. of ICCV 95, 230--236.

Digital Library

[37]

SZELISKI, R., AND SHUM, H.-Y. 1997. Creating Full View Panoramic Image Mosaics and Environment Maps. In Proc. of ACM SIGGRAPH 97, 251--258.

Digital Library

[38]

TELLER, S., ANTONE, M., BODNER, Z., BOSSE, M., COORG, S., JETHWA, M., AND MASTER, N. 2003. Calibrated, Registered Images of an Extended Urban Area. IJCV 53, 1, 93--107.

Digital Library

[39]

TORRANCE, K., AND SPARROW, E. 1967. Theory for off-specular reflection from roughened surfaces. JOSA, 57, 1105--1114.

[40]

TSUMURA, N., DANG, M., MAKINO, T., AND MIYAKE, Y. 2003. Estimating the Directions to Light Sources Using Images of Eye for Reconstructing 3D Human Face. In Proc. of IS&T/SID's CIC, 77--81.

[41]

VON HELMHOLTZ, H. 1909. Physiologic Optics, third ed., vol. 1 and 2. Voss, Hamburg, Germany.

[42]

WOODHAM, R. 1978. Photometric Stereo: A Reflectance Map Technique for Determining Surface Orientation from a Single View. In Proc. of SPIE, vol. 155, 136--143.

[43]

YAGI, Y., AND YACHIDA, M. 1991. Real-time Generation of Environmental Map and Obstacle Avoidance using Omnidirectional Image Sensor with Conic Mirror. In Proc. of IEEE CVPR 91, 160--165.

[44]

YU, Y., DEBEVEC, P., MALIK, J., AND HAWKINS, T. 1999. Inverse Globall Illumination: Recovering Reflectance Models of Real Scenes From Photographs. In Proc. of ACM SIGGRAPH 99, 215--224.

Digital Library

Cited By

Manuri FSanna APetrucci C(2020)PDIF: Pupil Detection After Isolation and FittingIEEE Access10.1109/ACCESS.2020.29730058(30826-30837)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2973005
Prakash SBahremand ANguyen LLiKamWa RSong JKim MLane NBalan R(2019)GLEAMProceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services10.1145/3307334.3326098(142-154)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1145/3307334.3326098
Lander CSpeicher MKerber FKrüger AMandryk RHancock MPerry MCox A(2018)Towards Fixation Extraction in Corneal Imaging Based Eye Tracking DataExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3188597(1-6)Online publication date: 20-Apr-2018
https://dl.acm.org/doi/10.1145/3170427.3188597
Show More Cited By

Index Terms

Eyes for relighting
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
      2. Image and video acquisition
        Epipolar geometry
  2. Computer graphics
    1. Image manipulation
      1. Texturing

Recommendations

Eyes for relighting

The combination of the cornea of an eye and a camera viewing the eye form a catadioptric (mirror + lens) imaging system with a very wide field of view. We present a detailed analysis of the characteristics of this corneal imaging system. Anatomical ...
Interactive relighting with dynamic BRDFs

We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance and ...
Interactive relighting with dynamic BRDFs
SIGGRAPH '07: ACM SIGGRAPH 2007 papers

We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '04: ACM SIGGRAPH 2004 Papers

August 2004

684 pages

ISBN:9781450378239

DOI:10.1145/1186562

Editor:
Joe Marks
Mitsubishi Electric Research Laboratories (MERL)

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGGRAPH04

Sponsor:

SIGGRAPH

SIGGRAPH04: Special Interest Group on Computer Graphics and Interactive Techniques

August 8 - 12, 2004

California, Los Angeles

Acceptance Rates

SIGGRAPH '04 Paper Acceptance Rate 83 of 478 submissions, 17%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,789
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Manuri FSanna APetrucci C(2020)PDIF: Pupil Detection After Isolation and FittingIEEE Access10.1109/ACCESS.2020.29730058(30826-30837)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2973005
Prakash SBahremand ANguyen LLiKamWa RSong JKim MLane NBalan R(2019)GLEAMProceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services10.1145/3307334.3326098(142-154)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1145/3307334.3326098
Lander CSpeicher MKerber FKrüger AMandryk RHancock MPerry MCox A(2018)Towards Fixation Extraction in Corneal Imaging Based Eye Tracking DataExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3188597(1-6)Online publication date: 20-Apr-2018
https://dl.acm.org/doi/10.1145/3170427.3188597
Lander Clöchtefeld MKrüger A(2018)hEYEbridProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31611661:4(1-29)Online publication date: 8-Jan-2018
https://dl.acm.org/doi/10.1145/3161166
Lander CGehring SLöchtefeld MBulling AKrüger AHenze NWoźniak PVäänänen KWilliamson JSchneegass S(2017)EyemirrorProceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia10.1145/3152832.3152839(279-291)Online publication date: 26-Nov-2017
https://dl.acm.org/doi/10.1145/3152832.3152839
Schneider DGrubert JSubramanian SSteimle JDachselt RMartinez Plasencia DGrossman T(2017)Towards Around-Device Interaction using Corneal ImagingProceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces10.1145/3132272.3134127(287-293)Online publication date: 17-Oct-2017
https://dl.acm.org/doi/10.1145/3132272.3134127
Ren HQiu HHe FLeng K(2016)A Survey on Image-Based Approaches of Synthesizing Objects2016 International Conference on Virtual Reality and Visualization (ICVRV)10.1109/ICVRV.2016.50(264-269)Online publication date: Sep-2016
https://doi.org/10.1109/ICVRV.2016.50
Plopski AItoh YNitschke CKiyokawa KKlinker GTakemura H(2015)Corneal-Imaging Calibration for Optical See-Through Head-Mounted DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2015.239185721:4(481-490)Online publication date: 18-Apr-2015
https://dl.acm.org/doi/10.1109/TVCG.2015.2391857
Zhang ZIsola PAdelson E(2015)SparkleVision: Seeing the world through random specular microfacets2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW.2015.7301369(1-9)Online publication date: Jun-2015
https://doi.org/10.1109/CVPRW.2015.7301369
KOC EBALCISOY S(2014)Light Source Estimation in Mobile Augmented Reality Scenes by Using Human Face GeometryIEICE Transactions on Information and Systems10.1587/transinf.E97.D.1974E97.D:8(1974-1982)Online publication date: 2014
https://doi.org/10.1587/transinf.E97.D.1974
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten