Mixed illumination analysis in single image for interactive color grading

ABSTRACT

Colorists often use keying or rotoscoping tools to access and edit particular colors or parts of the scene. Although necessary, this is a time-consuming and potentially imprecise process, as it is not possible to fully separate the influence of light sources in the scene from the colors of objects and actors within it. To simplify this process, we present a new solution for automatically estimating the color and influence of multiple illuminants, based on image variation analysis. Using this information, we present a new color grading tool for simply and interactively editing the colors of detected illuminants, which fits naturally in color grading workflows. We demonstrate the use of our solution in several scenes, evaluating the quality of our results by means of a psychophysical study.

References

1. Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Susstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Machine Intell. 34, 11 (2012), 2274--2282.
2. Yağiz Aksoy, Tunç Ozan Aydin, Aljoša Smolić, and Marc Pollefeys. 2017. Unmixing-based soft color segmentation for image manipulation. ACM Transactions on Graphics (TOG) 36, 2 (2017), 19.
3. Shida Beigpour, Christian Riess, Joost Van De Weijer, and Elli Angelopoulou. 2014. Multi-illuminant estimation with conditional random fields. IEEE Trans. Im. Proc. 23, 1 (2014), 83--96.
4. Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic Images in the Wild. ACM Trans. on Graphics (Proc. Siggraph) 33, 4 (2014).
5. Michael Bleier, Christian Riess, Shida Beigpour, Eva Eibenberger, Elli Angelopoulou, Tobias Tröger, and André Kaup. 2011. Color constancy and non-uniform illumination: Can existing algorithms work?. In ICCV Workshops. IEEE, 774--781.
6. Nicolas Bonneel, Kalyan Sunkavalli, Sylvain Paris, and Hanspeter Pfister. 2013. Example-Based Video Color Grading. ACM Trans. on Graphics (Proc. Siggraph) 32, 4 (2013).
7. Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015. Blind Video Temporal Consistency. ACM Transactions on Graphics (SIGGRAPH Asia 2015) 34, 6 (2015).
8. Adrien Bousseau, Sylvain Paris, and Frédo Durand. 2009. User-assisted intrinsic images. ACM Trans. on Graphics 28, 5 (2009), 130.
9. Ivaylo Boyadzhiev, Kavita Bala, Sylvain Paris, and Frédo Durand. 2012. User-guided white balance for mixed lighting conditions. ACM Trans. on Graphics 31, 6 (2012), 200.
10. B. Bratt. 2011. Rotoscoping: Techniques and Tools for the Aspiring Artist. Elsevier/Focal Press. https://books.google.fr/books?id=8PsAcEhBu_0C
11. Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Transactions on Graphics (TOG) 34, 4 (2015), 139.
12. Dongliang Cheng, Abdelrahman Abdelhamed, Brian Price, Scott Cohen, and Michael S Brown. 2016. Two Illuminant Estimation and User Correction Preference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 469--477.
13. Dorin Comaniciu, Peter Meer, and Senior Member. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Machine Intell. 24 (2002), 603--619.
14. Paul Debevec, Chris Tchou, Andrew Gardner, Tim Hawkins, Charis Poullis, Jessi Stumpfel, Andrew Jones, Nathaniel Yun, Per Einarsson, Therese Lundgren, Marcos Fajardo, and Philippe Martinez. 2004. Estimating Surface reflectance properties of a Complex Scene under captured natural Illumination. Technical Report ICT TR 06 2004. http://ict.usc.edu/pubs/ICT-TR-06.2004.pdf
15. Liuyun Duan and Florent Lafarge. 2015. Image partitioning into convex polygons. In Proc. CVPR.
16. Sylvain Duchêne, Clement Riant, Gaurav Chaurasia, Jorge Lopez-Moreno, Pierre-Yves Laffont, Stefan Popov, Adrien Bousseau, and George Drettakis. 2015. Multi-View Intrinsic Images of Outdoors Scenes with an Application to Relighting. ACM Trans. on Graphics (2015), 16. https://hal.inria.fr/hal-01164841
17. Marc Ebner. 2007. Color Constancy. John Wiley & Sons, West Sussex, England.
18. Marc Ebner. 2009. Color constancy based on local space average color. Mach. Vis. Appl. 20, 5 (2009), 283--301.
19. H Sheikh Faridul, T Pouli, C Chamaret, J Stauder, E Reinhard, D Kuzovkin, and A Tremeau. 2015. Colour Mapping: A Review of Recent Methods, Extensions and Applications. In Computer Graphics Forum. Wiley Online Library.
20. Elena Garces, Adolfo Munoz, Jorge Lopez-Moreno, and Diego Gutierrez. 2012. Intrinsic Images by Clustering. Computer Graphics Forum (Proc. EGSR) 31, 4 (2012).
21. Eduardo S. L. Gastal and Manuel M. Oliveira. 2011. Domain Transform for Edge-Aware Image and Video Processing. ACM Trans. on Graphics (Proc. Siggraph) 30, 4, Article 69 (2011), 69:1--69:12 pages.
22. Theo Gevers, Arjan Gijsenij, Joost Van de Weijer, and Jan-Mark Geusebroek. 2012. Color in computer vision: fundamentals and applications. John Wiley & Sons.
23. Arjan Gijsenij, Rui Lu, and Theo Gevers. 2012. Color constancy for multiple light sources. IEEE Trans. Im. Proc. 21, 2 (2012), 697--707.
24. Eugene Hsu, Tom Mertens, Sylvain Paris, Shai Avidan, and Frédo Durand. 2008. Light mixture estimation for spatially varying white balance. In ACM Trans. on Graphics, vol. 27. ACM, 70.
25. Tapas Kanungo, David M. Mount, Nathan S. Netanyahu, Christine D. Piatko, Ruth Silverman, and Angela Y. Wu. 2004. A local search approximation algorithm for k-means clustering. Comput. Geom. 28, 2--3 (2004), 89--112.
26. Pierre-Yves Laffont, Adrien Bousseau, Sylvain Paris, Frédo Durand, and George Drettakis. 2012. Coherent Intrinsic Images from Photo Collections. ACM Trans. on Graphics (Proc. Siggraph Asia) 31, 6 (2012).
27. Wenbin Li, Fabio Viola, Jonathan Starck, Gabriel J Brostow, and Neill DF Campbell. 2016. Roto++: Accelerating professional rotoscoping using shape manifolds. ACM Transactions on Graphics (TOG) 35, 4 (2016), 62.
28. Tania Pouli and Erik Reinhard. 2011. Progressive color transfer for images of arbitrary dynamic range. Computers & Graphics 35, 1 (2011), 67--80.
29. Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications 21, 5 (2001), 34--41.
30. Erik Reinhard and Tania Pouli. 2011. Colour spaces for colour transfer. In Computational Color Imaging. Springer, 1--15.
31. Joan Sol Roo and Christian Richardt. 2014. Temporally Coherent Video De-Anaglyph. In SIGGRAPH Talks.
32. Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Machine Intell. 22, 8 (2000), 888--905.
33. Jianchao Tan, Jyh-Ming Lien, and Yotam Gingold. 2016. Decomposing Images into Layers via RGB-Space Geometry. ACM Transactions on Graphics (TOG) 36, 1 (2016), 7.
34. Fuzhang Wu, Weiming Dong, Yan Kong, Xing Mei, Jean-Claude Paul, and Xiaopeng Zhang. 2013. Content-Based Colour Transfer. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 190--203.
35. Su Xue, Aseem Agarwala, Julie Dorsey, and Holly Rushmeier. 2013. Learning and applying color styles from feature films. In Computer Graphics Forum, vol. 32. Wiley Online Library, 255--264.
36. Qing Zhang, Chunxia Xiao, Hanqiu Sun, and Feng Tang. 2017. Palette-Based Image Recoloring Using Color Decomposition Optimization. IEEE Transactions on Image Processing 26, 4 (2017), 1952--1964.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Computational photography

  • Computer graphics

    • Image manipulation