Video enhancement using per-pixel virtual exposures

ABSTRACT

We enhance underexposed, low dynamic range videos by adaptively and independently varying the exposure at each photoreceptor in a post-process. This virtual exposure is a dynamic function of both the spatial neighborhood and temporal history at each pixel. Temporal integration enables us to expand the image's dynamic range while simultaneously reducing noise. Our non-linear exposure variation and denoising filters smoothly transition from temporal to spatial for moving scene elements. Our virtual exposure framework also supports temporally coherent per frame tone mapping. Our system outputs restored video sequences with significantly reduced noise, increased exposure time of dark pixels, intact motion, and improved details.

REFERENCES

• 1: Acosta-Serafini, P. M., Masaki, I., and Sodini, C. G. 2004, Predictive Multiple Sampling Algorithm with Overlapping Integration Intervals for Linear Wide Dynamic Range Integrating Image Sensors. IEEE Transactions on Intelligent Transportation Systems, 5, 1, 33--41.
• 2: Danny Barash, A Fundamental Relationship between Bilateral Filtering, Adaptive Smoothing, and the Nonlinear Diffusion Equation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.6, p.844-847, June 2002 [doi>10.1109/TPAMI.2002.1008390 ]
• 3: Eric P. Bennett , Leonard McMillan, Proscenium: a framework for spatio-temporal video editing, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957049]
• 4: Bidermann, W., El Gamal, A., Ewedemi, S., Reyneri, J., Tian, H., Wile, D., and Yang, D., 2003. A .18μm High Dynamic Range NTSC/PAL Imaging System-on-Chip with Embedded DRAM Frame Buffer, In Proceedings of the IEEE International Solid-State Circuits Conference, 212--213.
• 5: Rein van den Boomgaard , Joost van de Weijer, On the Equivalence of Local-Mode Finding, Robust Estimation and Mean-Shift Analysis as Used in Early Vision Tasks, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, p.30927, August 11-15, 2002
• 6: Prasun Choudhury , Jack Tumblin, The trilateral filter for high contrast images and meshes, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 7: Cohen, M., Colburn, A., and Drucker, S. 2003. Image Stacks. Microsoft Research Technical Report, MSR-TR-2003-40.
• 8: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 9: Drago, F., Myszkowski, K., Annen, T., and Chiba, N. 2003. Adaptive Logarithmic Mapping for Displaying High Contrast Scenes. In Proceedings of EUROGRAPHICS 2003, 22, 3, 419--426.
• 10: Dubois, E. and Sabri, S., 1984. Noise Reduction in Image Sequences Using Motion-Compensated Temporal Filtering, IEEE Transactions on Communications, 32, 7, 826--831.
• 11: Frédo Durand , Julie Dorsey, Fast bilateral filtering for the display of high-dynamic-range images, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 12: Elmar Eisemann , Frédo Durand, Flash photography enhancement via intrinsic relighting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 13: Raanan Fattal , Dani Lischinski , Michael Werman, Gradient domain high dynamic range compression, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 14: Francis, J. J. and Jager, G. D. 2003. The Bilateral Median Filter. In Proceedings of the 14th Symposium of the Pattern Recognition Association of South Africa.
• 15: Jobson, D. J., Rahman, Z.-U., and Woodell, G. A. 1997. A Multiscale Retinex for Bridging the Gap Between Color Images and the Human Observation of Scenes. IEEE Transactions on Image Processing, 6, 7, 965--976.
• 16: Jostschulte, K., Amer, A., Schu, M., and Scroder, H., 1998. Perception Adaptive Temporal TV-Noise Reduction Using Contour Preserving Prefilter Techniques, IEEE Transactions on Consumer Electronics, 44, 3 (August), 1091--1096.
• 17: Sing Bing Kang , Matthew Uyttendaele , Simon Winder , Richard Szeliski, High dynamic range video, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 18: Lee, S. H. and Kang, M. G. 1998. Spatio-Temporal Video Filtering Algorithm based on 3-D Anisotropic Diffusion Equation. In Proceedings of the International Conference on Image Processing, 98, 2, 447--450.
• 19: Liu, X., and El Gamal, A., 2003. Synthesis of High Dynamic Range Motion Blur Free Image From Multiple Captures. IEEE Transactions on Circuits and Systems, Fundamental Theory and Applications, 50, 4, 530--539.
• 20: Shree K. Nayar , Vlad Branzoi, Adaptive Dynamic Range Imaging: Optical Control of Pixel Exposures Over Space and Time, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1168, October 13-16, 2003
• 21: Nayar, S. and Branzoi, V. 2004. Programmable Imaging Using a Digital Micromirror Array. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 436--443.
• 22: Sumanta N. Pattanaik , Jack Tumblin , Hector Yee , Donald P. Greenberg, Time-dependent visual adaptation for fast realistic image display, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.47-54, July 2000 [doi>10.1145/344779.344810]
• 23: P. Perona , J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.7, p.629-639, July 1990 [doi>10.1109/34.56205 ]
• 24: Georg Petschnigg , Richard Szeliski , Maneesh Agrawala , Michael Cohen , Hugues Hoppe , Kentaro Toyama, Digital photography with flash and no-flash image pairs, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 25: Ramesh Raskar , Adrian Ilie , Jingyi Yu, Image fusion for context enhancement and video surrealism, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987671]
• 26: Reibel, Y., Jung, M., Bouhifd. M., Cunin, B., and Draman, C. 2003. CCD or CMOS Camera Noise Characteristics. In Proceedings of the European Physical Journal of Applied Physics, 75--80.
• 27: Peter Sand , Seth Teller, Video matching, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 28: Stockham, T. G. 1972. Image Processing in the Context of a Visual Model, In Proceedings of the IEEE, 60, 828--842.
• 29: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 30: Jack Tumblin , Holly Rushmeier, Tone Reproduction for Realistic Images, IEEE Computer Graphics and Applications, v.13 n.6, p.42-48, November 1993 [doi>10.1109/38.252554 ]
• 31: Jack Tumblin , Greg Turk, LCIS: a boundary hierarchy for detail-preserving contrast reduction, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.83-90, July 1999 [doi>10.1145/311535.311544]
• 32: Ward, G. 1991. Real Pixels. Graphics Gems II. Academic Press. 80--83.
• 33: Hector Yee , Sumanita Pattanaik , Donald P. Greenberg, Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments, ACM Transactions on Graphics (TOG), v.20 n.1, p.39-65, Jan. 2001 [doi>10.1145/383745.383748]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Keywords:
bilateral filter, digital video, exposure, low dynamic range, noise reduction, restoration, tone mapping