Deep high dynamic range imaging of dynamic scenes

Abstract

Producing a high dynamic range (HDR) image from a set of images with different exposures is a challenging process for dynamic scenes. A category of existing techniques first register the input images to a reference image and then merge the aligned images into an HDR image. However, the artifacts of the registration usually appear as ghosting and tearing in the final HDR images. In this paper, we propose a learning-based approach to address this problem for dynamic scenes. We use a convolutional neural network (CNN) as our learning model and present and compare three different system architectures to model the HDR merge process. Furthermore, we create a large dataset of input LDR images and their corresponding ground truth HDR images to train our system. We demonstrate the performance of our system by producing high-quality HDR images from a set of three LDR images. Experimental results show that our method consistently produces better results than several state-of-the-art approaches on challenging scenes.

References

1. A. Badki, N. Khademi Kalantari, and P. Sen. 2015. Robust Radiometric Calibration for Dynamic Scenes in the Wild. In IEEE ICCP. 1--10.
2. Luca Bogoni. 2000. Extending Dynamic Range of Monochrome and Color Images through Fusion. In IEEE ICPR. 3007--3016.
3. Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu. 2013. Large Displacement Optical Flow from Nearest Neighbor Fields. In IEEE CVPR. 2443--2450.
4. Z. Cheng, Q. Yang, and B. Sheng. 2015. Deep Colorization. In IEEE ICCV. 415--423.
5. Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH. 369--378.
6. A. Dosovitskiy, P. Fischery, E. Ilg, P. HÃd'usser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox. 2015. FlowNet: Learning Optical Flow with Convolutional Networks. In ICCV. 2758--2766.
7. John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views from the WorldâĂŹs Imagery. In IEEE CVPR. 5515--5524.
8. Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel. 2014. Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. SPIE 9023 (2014), 90230X--90230X-10.
9. Brian Funt and Lilong Shi. 2010. The Rehabilitation of MaxRGB. Color and Imaging Conference 2010, 1 (2010), 256--259.
10. O. Gallo, N. Gelfand, W. Chen, M. Tico, and K. Pulli. 2009. Artifact-free High Dynamic Range Imaging. In IEEE ICCP. 1--7.
11. O. Gallo, A. Troccoli, J. Hu, K. Pulli, and J. Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In IEEE CVPRW. 48--55.
12. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, Vol. 9. 249--256.
13. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Networks. ArXiv e-prints (June 2014). arXiv:stat.ML/1406.2661
14. M. Granados, B. Ajdin, M. Wand, C. Theobalt, H. P. Seidel, and H. P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In IEEE CVPR. 215--222.
15. Miguel Granados, Kwang In Kim, James Tompkin, and Christian Theobalt. 2013. Automatic Noise Modeling for Ghost-free HDR Reconstruction. ACM TOG 32, 6, Article 201 (2013), 10 pages.
16. T. Grosch. 2006. Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement. In Vision, Modeling and Visualization. 277--284.
17. Michael D. Grossberg and Shree K. Nayar. 2003. Determining the Camera Response from Images: What Is Knowable? IEEE PAMI 25, 11 (Nov. 2003), 1455--1467.
18. Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM TOG 30, 4, Article 70 (2011), 10 pages.
19. Saghi Hajisharif, Joel Kronander, and Jonas Unger. 2015. Adaptive dualISO HDR reconstruction. EURASIP Journal on Image and Video Processing 2015, 1 (2015), 41.
20. S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In CVPR. 553--560.
21. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-light Imaging on Mobile Cameras. ACM TOG 35, 6, Article 192 (Nov. 2016), 12 pages.
22. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A Flexible Camera Image Processing Framework. ACM TOG 33, 6, Article 231 (Nov. 2014), 13 pages.
23. Yong Seok Heo, Kyoung Mu Lee, Sang Uk Lee, Youngsu Moon, and Joonhyuk Cha. 2010. Ghost-free high dynamic range imaging. In ACCV, Vol. 4. 486--500.
24. Jun Hu, Orazio Gallo, and Kari Pulli. 2012. Exposure Stacks of Live Scenes with Hand-Held Cameras. Springer Berlin Heidelberg, Berlin, Heidelberg, 499--512.
25. J. Hu, O. Gallo, K. Pulli, and X. Sun. 2013. HDR Deghosting: How to Deal with Saturation?. In IEEE CVPR. 1163--1170.
26. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let There Be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM TOG 35, 4, Article 110 (July 2016), 11 pages.
27. Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2016. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. CoRR abs/1612.01925 (2016).
28. K. Jacobs, C. Loscos, and G. Ward. 2008. Automatic High-Dynamic Range Image Generation for Dynamic Scenes. IEEE Computer Graphics and Applications 28, 2 (Mar.-Apr. 2008), 84--93.
29. T. Jinno and M. Okuda. 2008. Motion blur free HDR image acquisition using multiple exposures. In IEEE ICIP. 1304--1307.
30. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM TOG 35, 6, Article 193 (Nov. 2016), 10 pages.
31. Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003. High dynamic range video. ACM TOG 22, 3 (2003), 319--325.
32. Kanita Karaduzovic-Hadziabdic, Jasminka Hasic Telalovic, and Rafal Mantiuk. 2016. Subjective and Objective Evaluation of Multi-exposure High Dynamic Range Image Deghosting Methods. In EG, T. Bashford-Rogers and L. P. Santos (Eds.). The Eurographics Association.
33. E.A. Khan, A.O. Akyüz, and E. Reinhard. 2006. Ghost Removal in High Dynamic Range Images. In IEEE ICIP. 2005--2008.
34. C. Lee, Y. Li, and V. Monga. 2014. Ghost-Free High Dynamic Range Imaging via Rank Minimization. IEEE SPL 21, 9 (Sept 2014), 1045--1049.
35. Z. Li, J. Zheng, Z. Zhu, and S. Wu. 2014. Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects. IEEE Transactions on Image Processing 23, 10 (Oct 2014), 4372--4382.
36. C. Liu. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral thesis. Massachusetts Institute of Technology.
37. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In IJCAI. 674--679.
38. Steve Mann and Rosalind W. Picard. 1995. On Being 'undigital' With Digital Cameras: Extending Dynamic Range By Combining Differently Exposed Pictures. In Proc. of Society for Imaging Science and Technology. 442--448.
39. Rafat Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM TOG 30, 4, Article 40 (July 2011), 14 pages.
40. M. McGuire, W. Matusik, H. Pfister, B. Chen, J.F. Hughes, and S.K. Nayar. 2007. Optical Splitting Trees for High-Precision Monocular Imaging. IEEE Computer Graphics and Applications 27, 2 (march-april 2007), 32--42.
41. T. H. Oh, J. Y. Lee, Y. W. Tai, and I. S. Kweon. 2015. Robust High Dynamic Range Imaging by Rank Minimization. IEEE PAMI 37, 6 (2015), 1219--1232.
42. F. Pece and J. Kautz. 2010. Bitmap Movement Detection: HDR for Dynamic Scenes. In CVMP. 1--8.
43. Photomatix. 2017. Commercially-available HDR processing software. (2017). http://www.hdrsoft.com/.
44. Shanmuganathan Raman and Subhasis Chaudhuri. 2011. Reconstruction of high contrast images for dynamic scenes. The Visual Computer 27, 12 (Dec. 2011), 1099--1114.
45. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning Representations by Back-propagating Errors. Nature 323 (1986), 533--536.
46. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM TOG 31, 6, Article 203 (Nov. 2012), 11 pages.
47. Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional Sparse Coding for High Dynamic Range Imaging. CGF 35, 2 (2016), 153--163.
48. Michael D. Tocci, Chris Kiser, Nora Tocci, and Pradeep Sen. 2011. A versatile HDR video production system. ACM TOG 30, 4, Article 41 (July 2011), 10 pages.
49. A. Tomaszewska and R. Mantiuk. 2007. Image Registration for Multi-exposure High Dynamic Range Image Acquisition. In WSCG.
50. Okan Tarhan Tursun, Ahmet Oğuz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. CGF 35, 2 (2016), 139--152.
51. Okan Tarhan Tursun, Ahmet Oħuz AkyAijz, Aykut Erdem, and Erkut Erdem. 2015. The State of the Art in HDR Deghosting: A Survey and Evaluation. CGF 34, 2 (2015).
52. Andrea Vedaldi and Karel Lenc. 2015. MatConvNet: Convolutional neural networks for Matlab. In ACMMM. 689--692.
53. Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal of Graphics, GPU, and Game Tools 8, 2 (2003), 17--30.
54. L. Zhang, A. Deshpande, and X. Chen. 2010. Denoising vs. deblurring: HDR imaging techniques using moving cameras. In CVPR. 522--529.
55. Wei Zhang and Wai-Kuen Cham. 2012. Gradient-Directed Multiexposure Composition. IEEE TIP 21, 4 (April 2012), 2318--2323.
56. H. Zhao, B. Shi, C. Fernandez-Cull, S. K. Yeung, and R. Raskar. 2015. Unbounded High Dynamic Range Photography Using a Modulo Camera. In ICCP 2015. 1--10.
57. J. Zheng, Z. Li, Z. Zhu, S. Wu, and S. Rahardja. 2013. Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects. IEEE TIP 22, 12 (Dec 2013), 5190--5201.
58. Henning Zimmer, AndrÃl's Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. CGF 30, 2 (April 2011), 405--414.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Computational photography