research-article

Sky is not the limit: semantic-aware sky replacement

Authors:

Kalyan Sunkavalli,

Ming-Hsuan YangAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 4

Article No.: 149, Pages 1 - 11

https://doi.org/10.1145/2897824.2925942

Published: 11 July 2016 Publication History

Abstract

Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.

Supplementary Material

ZIP File (a149-tsai-supp.zip)

Supplemental files.

Download
128.09 MB

MP4 File (a149.mp4)

Download
292.14 MB

References

[1]

Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. (proc. SIGGRAPH) 27, 3.

Digital Library

[2]

Boykov, Y., and Kolmogorov, V. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI, 1124--1137.

Digital Library

[3]

Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. 2015. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR.

[4]

Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., and Pfister, H. 2009. Image restoration using online photo collections. In ICCV.

[5]

Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (proc. SIGGRAPH) 31, 4.

Digital Library

[6]

Galasso, F., Nagaraja, N., Cardenas, T., Brox, T., and Schiele, B. 2013. A unified video segmentation benchmark: Annotation, metrics and analysis. In ICCV.

Digital Library

[7]

HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graph. (proc. SIGGRAPH) 30, 4.

Digital Library

[8]

Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3.

Digital Library

[9]

He, K., Sun, J., and Tang, X. 2013. Guided image filtering. PAMI 35, 6, 1397--1409.

Digital Library

[10]

Hoiem, D., Efros, A. A., and Hebert, M. 2007. Recovering surface layout from an image. IJCV 75, 1.

Digital Library

[11]

Johnson, M. K., Dale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. Vis. Comp. Graph. 17, 9.

Digital Library

[12]

Kaufman, L., Lischinski, D., and Werman, M. 2012. Content-aware automatic photo enhancement. Comp. Graph. Forum 31, 8.

Digital Library

[13]

Kumar, M. P., Torr, P., and Zisserman, A. 2005. Obj cut. In CVPR.

Digital Library

[14]

Laffont, P.-Y., Ren, Z., Tao, X., Qian, C., and Hays, J. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. (proc. SIGGRAPH) 33, 4.

Digital Library

[15]

Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In ICCV.

[16]

Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3.

Digital Library

[17]

Lalonde, J.-F., Narasimhan, S. G., and Efros, A. A. 2010. What do the sun and the sky tell us about the camera? IJCV 88, 1.

Digital Library

[18]

Lalonde, J.-F., Efros, A. A., and Narasimhan, S. G. 2011. Estimating the natural illumination conditions from a single outdoor image. IJCV 98, 2, 123--145.

Digital Library

[19]

Lazebnik, S., Schmid, C., and Ponce, J. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR.

Digital Library

[20]

Lee, J.-Y., Sunkavalli, K., Lin, Z., Shen, X., and Kweon, I. S. 2016. Automatic content-aware color and tone stylization. In CVPR.

[21]

Liu, Y., and Yu, Y. 2012. Interactive image segmentation based on level sets of probabilities. IEEE Transactions on Visualization and Computer Graphics 18, 2, 202--213.

Digital Library

[22]

Liu, C., Yuen, J., and Torralba, A. 2011. Nonparametric scene parsing via label transfer. PAMI 33, 12, 2368--2382.

Digital Library

[23]

Liu, Y., Cohen, M., Uyttendaele, M., and Rusinkiewicz, S. 2014. Autostyle: Automatic style transfer from image collections to users image. Comp. Graph. Forum 33, 4.

[24]

Long, J., Shelhamer, E., and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In CVPR.

[25]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. (proc. SIGGRAPH) 22, 3.

Digital Library

[26]

Pitié, F., and Kokaram, A. 2007. The linear mongekantorovitch linear colour mapping for example-based colour transfer. In CVMP.

[27]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comp. Graph. Appl. 21, 5, 34--41.

Digital Library

[28]

Rother, C., Kolmogorov, V., and Blake, A. 2004. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. (proc. SIGGRAPH) 23, 3.

Digital Library

[29]

Shahrian, E., Rajan, D., Price, B., and Cohen, S. 2013. Improving image matting using comprehensive sampling sets. In CVPR.

Digital Library

[30]

Shih, Y., Paris, S., Durand, F., and Freeman, W. T. 2013. Data-driven hallucination of different times of day from a single outdoorphoto. ACM Trans. Graph. (proc. SIGGRAPH Asia) 32, 6.

Digital Library

[31]

Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. ACM Trans. Graph. (proc. SIGGRAPH) 29, 4.

Digital Library

[32]

Tai, Y.-W., Jia, J., and Tang, C.-K. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In CVPR.

Digital Library

[33]

Tao, L., Yuan, L., and Sun, J. 2009. Skyfinder: Attribute-based sky image search. ACM Trans. Graph. (proc. SIGGRAPH) 28, 3.

Digital Library

[34]

Tao, M. W., Johnson, M. K., and Paris, S. 2013. Error-tolerant image compositing. IJCV 103, 2, 178--189.

Digital Library

[35]

Tighe, J., and Lazebnik, S. 2013. Superparsing: Scalable non-parametric image parsing with superpixels. IJCV 101, 2, 329--349.

Digital Library

[36]

Torralba, A., Oliva, A., Castelhano, M., and Hen-derso, J. M. 2006. Contextual guidance of attention in natural scenes: The role of global features on object search. Psychological Review 113, 10, 766--786.

[37]

Tsai, Y.-H., Hamsici, O., and Yang, M.-H. 2015. Adaptive region pooling for object detection. In CVPR.

[38]

Wang, X., Yang, M., Zhu, S., and Lin, Y. 2013. Regionlets for generic object detection. In ICCV.

Digital Library

[39]

Wu, F., Dong, W., Knog, Y., Mei, X., Paul, J.-C., and Zhang, X. 2013. Content-based coulour transfer. Comp. Graph. Forum 32, 1.

[40]

Xue, S., Agarwala, A., Dorsey, J., and Rushmeier, H. 2012. Understanding and improving the realism of image composites. ACM Trans. Graph. (proc. SIGGRAPH) 31, 4.

Digital Library

[41]

Yan, Z., Zhang, H., Wang, B., Paris, S., and Yu, Y. 2016. Automatic photo adjustment using deep neural networks. ACM Trans. Graph. 35, 2.

Digital Library

[42]

Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., and Torr, P. 2015. Conditional random fields as recurrent neural networks. In ICCV.

Digital Library

[43]

Zhu, J.-Y., Krähenbühl, P., Shechtman, E., and Efros, A. A. 2015. Learning a discriminative model for the perception of realism in composite images. In ICCV.

Digital Library

Cited By

Wu MSun YJiang S(2024)Adaptive Color Transfer From Images to Terrain VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329512230:8(5538-5552)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3295122
Wang XLv QChen GZhang JWei ZDong JFu HZhu ZLiu JJin X(2024)MobileSky: Real-Time Sky Replacement for Mobile ARIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325784030:7(4304-4320)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3257840
Weng SShi B(2024)Conditional Image RepaintingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333127546:4(2285-2298)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3331275
Show More Cited By

Index Terms

Sky is not the limit: semantic-aware sky replacement
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

A physically-based night sky model
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

This paper presents a physically-based model of the night sky for realistic image synthesis. We model both the direct appearance of the night sky and the illumination coming from the Moon, the stars, the zodiacal light, and the atmosphere. To accurately ...
Total relighting: learning to relight portraits for background replacement

We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images ...
The Sky’s the Limit: Relightable Outdoor Scenes via a Sky-Pixel Constrained Illumination Prior and Outside-In Visibility
Computer Vision – ECCV 2024
Abstract
Inverse rendering of outdoor scenes from unconstrained image collections is a challenging task, particularly illumination/albedo ambiguities and occlusion of the illumination environment (shadowing) caused by geometry. However, there are many cues ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 4

July 2016

1396 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2897824

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Published in TOG Volume 35, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
979
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)6

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu MSun YJiang S(2024)Adaptive Color Transfer From Images to Terrain VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329512230:8(5538-5552)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3295122
Wang XLv QChen GZhang JWei ZDong JFu HZhu ZLiu JJin X(2024)MobileSky: Real-Time Sky Replacement for Mobile ARIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325784030:7(4304-4320)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3257840
Weng SShi B(2024)Conditional Image RepaintingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333127546:4(2285-2298)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3331275
Duan LWu MLou HYin JLi X(2024)MRCAN: Multi-scale Region Correlation-driven Adaptive Normalization for Image Harmonization2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC54092.2024.10831083(5206-5211)Online publication date: 6-Oct-2024
https://doi.org/10.1109/SMC54092.2024.10831083
Fu XWang BShu YYang WTian LDing Y(2024)Video Sky Replacement Based on Dual-Branch Segmentation Network2024 4th International Conference on Consumer Electronics and Computer Engineering (ICCECE)10.1109/ICCECE61317.2024.10504221(243-247)Online publication date: 12-Jan-2024
https://doi.org/10.1109/ICCECE61317.2024.10504221
Bao ZFu GChen ZXiao C(2024)Illuminator: Image-based illumination editing for indoor scene harmonizationComputational Visual Media10.1007/s41095-023-0397-610:6(1137-1155)Online publication date: 5-Jul-2024
https://doi.org/10.1007/s41095-023-0397-6
Lin TLin HLi FHe DWu WWang MLi XLiu YWilliams BChen YNeville J(2023)AdaCMProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v37i2.25248(1613-1621)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1609/aaai.v37i2.25248
Zhang WZhang RWang GLi WLiu XYang YHu D(2023)Physics Guided Remote Sensing Image Synthesis Network for Ship DetectionIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2023.324810661(1-14)Online publication date: 2023
https://doi.org/10.1109/TGRS.2023.3248106
Zhu NBouronopoulos ALeduc TServières MRenaudin V(2023)Evaluation of the Human Body Mask Effects on GNSS Wearable Devices for Outdoor Pedestrian Navigation Using Fisheye Sky Views2023 IEEE/ION Position, Location and Navigation Symposium (PLANS)10.1109/PLANS53410.2023.10140056(841-850)Online publication date: 24-Apr-2023
https://doi.org/10.1109/PLANS53410.2023.10140056
Gerhardt CWeidner FBroll W(2023)SkyCloud: Neural Network-Based Sky and Cloud Segmentation from Natural Images2023 8th International Conference on Image, Vision and Computing (ICIVC)10.1109/ICIVC58118.2023.10270450(343-351)Online publication date: 27-Jul-2023
https://doi.org/10.1109/ICIVC58118.2023.10270450
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents