Intrinsic images in the wild

Abstract

Intrinsic image decomposition separates an image into a reflectance layer and a shading layer. Automatic intrinsic image decomposition remains a significant challenge, particularly for real-world scenes. Advances on this longstanding problem have been spurred by public datasets of ground truth data, such as the MIT Intrinsic Images dataset. However, the difficulty of acquiring ground truth data has meant that such datasets cover a small range of materials and objects. In contrast, real-world scenes contain a rich range of shapes and materials, lit by complex illumination.

In this paper we introduce Intrinsic Images in the Wild, a large-scale, public dataset for evaluating intrinsic image decompositions of indoor scenes. We create this benchmark through millions of crowdsourced annotations of relative comparisons of material properties at pairs of points in each scene. Crowdsourcing enables a scalable approach to acquiring a large database, and uses the ability of humans to judge material comparisons, despite variations in illumination. Given our database, we develop a dense CRF-based intrinsic image algorithm for images in the wild that outperforms a range of state-of-the-art intrinsic image algorithms. Intrinsic image decomposition remains a challenging problem; we release our code and database publicly to support future research on this problem, available online at http://intrinsic.cs.cornell.edu/.

References

1. Adams, A., Baek, J., and Davis, A. 2010. Fast high-dimensional filtering using the permutohedral lattice. Computer Graphics Forum (Eurographics) 29, 2.
2. Barron, J. T., and Malik, J. 2012. Color constancy, intrinsic images, and shape estimation. In Proc. European Conference on Computer Vision.
3. Barron, J. T., and Malik, J. 2012. Shape, albedo, and illumination from a single image of an unknown object. In Proc. Computer Vision and Pattern Recognition.
4. Barron, J. T., and Malik, J. 2013. Intrinsic scene properties from a single RGB-D image. In Proc. Computer Vision and Pattern Recognition.
5. Barron, J. T., and Malik, J. 2013. Shape, illumination, and reflectance from shading. Tech. rep., UC Berkeley.
6. Bell, S., Upchurch, P., Snavely, N., and Bala, K. 2013. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. on Graphics (SIGGRAPH) 32, 4.
7. Bostock, M., 2013. D3. http://d3js.org/. {Online; accessed 24-Mar-2014}.
8. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. on Graphics (SIGGRAPH Asia) 28, 5.
9. Boyadzhiev, I., Paris, S., and Bala, K. 2013. User-assisted image compositing for photographic lighting. ACM Trans. on Graphics (SIGGRAPH) 32, 4.
10. Branson, S., Wah, C., Babenko, B., Schroff, F., Welinder, P., Perona, P., and Belongie, S. 2010. Visual recognition with humans in the loop. In Proc. European Conference on Computer Vision.
11. Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. on Graphics (SIGGRAPH).
12. Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In Proc. International Conference on Computer Vision.
13. Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., and Singh, M. 2009. How well do line drawings depict shape? ACM Trans. on Graphics (SIGGRAPH) 28, 3.
14. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. In J. Roy. Statistical Society.
15. Felzenszwalb, P., McAllester, D., and Ramanan, D. 2008. A discriminatively trained, multiscale, deformable part model. In Proc. Computer Vision and Pattern Recognition.
16. Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. Computer Graphics Forum (Eurographics Symposium on Rendering) 31, 4.
17. Gehler, P., Rother, C., Kiefel, M., Zhang, L., and Scholkopf, B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In Neural Information Processing Systems.
18. Gingold, Y., Shamir, A., and Cohen-Or, D. 2012. Micro perceptual human computation. ACM Trans. on Graphics 31, 5.
19. Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In Proc. International Conference on Computer Vision.
20. Haber, T., Fuchs, C., Bekaer, P., Seidel, H.-P., Goesele, M., and Lensch, H. P. 2009. Relighting objects from image collections. In Proc. Computer Vision and Pattern Recognition.
21. Hauagge, D., Wehrwein, S., Bala, K., and Snavely, N. 2013. Photometric ambient occlusion. Proc. Computer Vision and Pattern Recognition.
22. Krähenbühl, P., and Koltun, V. 2011. Efficient inference in fully connected CRFs with gaussian edge potentials. In Neural Information Processing Systems.
23. Krähenbühl, P., and Koltun, V. 2013. Parameter learning and convergent inference for dense random fields. In Proc. International Conference on Machine Learning.
24. Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., and Drettakis, G. 2012. Coherent intrinsic images from photo collections. ACM Trans. on Graphics (SIGGRAPH Asia) 31, 6.
25. Laffont, P., Bousseau, A., and Drettakis, G. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. on Visualization and Computer Graphics 19, 2.
26. Land, E. H., and McCann, J. J. 1971. Lightness and retinex theory. J. Opt. Soc. Am. 61, 1.
27. Liao, Z., Rock, J., Wang, Y., and Forsyth, D. 2013. Non-parametric filtering for geometric detail extraction and material representation. In Proc. Computer Vision and Pattern Recognition.
28. Liu, X., Jiang, L., Wong, T.-T., and Fu, C.-W. 2012. Statistical invariance for texture synthesis. IEEE Trans. on Visualization and Computer Graphics 18, 11.
29. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. Conference on Comp. Graphics and Interactive Techniques (SIGGRAPH).
30. Omer, I., and Werman, M. 2004. Color lines: image specific color representation. In Proc. Computer Vision and Pattern Recognition.
31. Ostrovsky, Y., Cavanagh, P., and Sinha, P. 2005. Perceiving illumination inconsistencies in scenes. Perception 34, 11.
32. Rubinstein, M., Gutierrez, D., Sorkine, O., and Shamir, A. 2010. A comparative study of image retargeting. ACM Trans. on Graphics (SIGGRAPH Asia) 29, 6.
33. Shen, L., and Yeo, C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In Proc. Computer Vision and Pattern Recognition.
34. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In Proc. Computer Vision and Pattern Recognition.
35. Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In Proc. Computer Vision and Pattern Recognition.
36. Tappen, M., Freeman, W., and Adelson, E. 2005. Recovering intrinsic images from a single image. IEEE Trans. on Pattern Analysis and Machine Intelligence.
37. Tappen, M. F., Adelson, E. H., and Freeman, W. T. 2006. Estimating intrinsic component images using non-linear regression. In Proc. Computer Vision and Pattern Recognition.
38. van Wijk, J. J., and Nuij, W. A. 2003. Smooth and efficient zooming and panning. In Proc. IEEE Conference on Information Visualization (INFOVIS).
39. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In Proc. International Conference on Computer Vision.
40. Welinder, P., Branson, S., Belongie, S., and Perona, P. 2010. The multidimensional wisdom of crowds. In Neural Information Processing Systems.
41. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. on Pattern Analysis and Machine Intelligence 34, 7.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding