Removing photography artifacts using gradient projection and flash-exposure sampling

ABSTRACT

Flash images are known to suffer from several problems: saturation of nearby objects, poor illumination of distant objects, reflections of objects strongly lit by the flash and strong highlights due to the reflection of flash itself by glossy surfaces. We propose to use a flash and no-flash (ambient) image pair to produce better flash images. We present a novel gradient projection scheme based on a gradient coherence model that allows removal of reflections and highlights from flash images. We also present a brightness-ratio based algorithm that allows us to compensate for the falloff in the flash image brightness due to depth. In several practical scenarios, the quality of flash/no-flash images may be limited in terms of dynamic range. In such cases, we advocate using several images taken under different flash intensities and exposures. We analyze the flash intensity-exposure space and propose a method for adaptively sampling this space so as to minimize the number of captured images for any given scene. We present several experimental results that demonstrate the ability of our algorithms to produce improved flash images.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Canon. http://www.cps.canon-europe.com/kb/.
• 3: Chen, H., Belhumeur, P., and Jacobs, D. 2000. In search of illumination invariants. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1, 254--261.
• 4: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 5: Elmar Eisemann , Frédo Durand, Flash photography enhancement via intrinsic relighting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 6: Farid, H., and Adelson, E. H. 1999. Separating reflections and lighting using independent components analysis. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1, 1262--1267.
• 7: Raanan Fattal , Dani Lischinski , Michael Werman, Gradient domain high dynamic range compression, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 8: Graham D. Finlayson , Steven D. Hordley , Mark S. Drew, Removing Shadows from Images, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.823-836, May 28-31, 2002
• 9: Finlayson, G. D., Drew, M. S., and Lu, C. 2004. Intrinsic images by entropy minimization. In Proc. of European Conf. on Computer Vision, vol. 3, 582--595.
• 10: Grossberg, M. D., and Nayar, S. K. 2003. High dynamic range from multiple images: which exposures to combine? In Proc. of ICCV Workshop on Color and Photometric Methods in Computer Vision.
• 11: Bernd Jahne, Spatio-Temporal Image Processing: Theory and Scientific Applications, Springer-Verlag New York, Inc., Secaucus, NJ, 1993
• 12: Kodak. 2001. CCD image sensor noise sources. Application note MPT/PS-0233.
• 13: Levin, A., Zomet, A., and Weiss, Y. 2004. Separating reflections from a single image using local features. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1, 306--313.
• 14: Lichtenauer, J., Reinders, M., and Hendriks, E. 2004. Influence of the observation likelihood function on particle filtering performance in tracking applications. In Sixth IEEE Int'l. Conf. on Automatic Face and Gesture Recognition, 767--772.
• 15: Mann, S., and Picard, R. W. 1995. Being undigital with digital cameras: extending dynamic range by combining differently exposed pictures. In Proc. of IS&T 48th annual conference, 422--428.
• 16: Nayar, S. K., and Mitsunaga, T. 2000. High dynamic range imaging: spatially varying pixel exposures. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1, 472--479.
• 17: Shree K. Nayar , Xi-Sheng Fang , Terrance Boult, Separation of Reflection Components Using Color and Polarization, International Journal of Computer Vision, v.21 n.3, p.163-186, Feb. 1997 [doi>10.1023/A:1007937815113 ]
• 18: Patrick Pérez , Michel Gangnet , Andrew Blake, Poisson image editing, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 19: Georg Petschnigg , Richard Szeliski , Maneesh Agrawala , Michael Cohen , Hugues Hoppe , Kentaro Toyama, Digital photography with flash and no-flash image pairs, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 20: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, New York, NY, 1992
• 21: Ramesh Raskar , Kar-Han Tan , Rogerio Feris , Jingyi Yu , Matthew Turk, Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 22: Ramesh Raskar , Adrian Ilie , Jingyi Yu, Image fusion for context enhancement and video surrealism, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987671]
• 23: Yoav Y. Schechner , Nahum Kiryati , Ronen Basri, Separation of Transparent Layers using Focus, International Journal of Computer Vision, v.39 n.1, p.25-39, Aug. 2000 [doi>10.1023/A:1008166017466 ]
• 24: Jian Sun , Jiaya Jia , Chi-Keung Tang , Heung-Yeung Shum, Poisson matting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 25: Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1, 246--243.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Projections

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement
          Subjects: Geometric correction
      I.4.8 Scene Analysis
          Subjects: Photometry

General Terms:
Algorithms, Experimentation, Performance

Keywords:
flash, flash-exposure sampling, gradient projection, high dynamic range (HDR) imaging, reflection removal