research-article

Multi-scale image harmonization

Authors:

Kalyan Sunkavalli,

Micah K. Johnson,

Wojciech Matusik,

Hanspeter PfisterAuthors Info & Claims

SIGGRAPH '10: ACM SIGGRAPH 2010 papers

Article No.: 125, Pages 1 - 10

https://doi.org/10.1145/1833349.1778862

Published: 26 July 2010 Publication History

Abstract

Traditional image compositing techniques, such as alpha matting and gradient domain compositing, are used to create composites that have plausible boundaries. But when applied to images taken from different sources or shot under different conditions, these techniques can produce unrealistic results. In this work, we present a framework that explicitly matches the visual appearance of images through a process we call image harmonization, before blending them. At the heart of this framework is a multi-scale technique that allows us to transfer the appearance of one image to another. We show that by carefully manipulating the scales of a pyramid decomposition of an image, we can match contrast, texture, noise, and blur, while avoiding image artifacts. The output composite can then be reconstructed from the modified pyramid coefficients while enforcing both alpha-based and seamless boundary constraints. We show how the proposed framework can be used to produce realistic composites with minimal user interaction in a number of different scenarios.

Supplementary Material

Supplemental material. (125.zip)

The auxillary material consists of a PDF document with larger versions of the results in the paper.

Download
36.05 MB

References

[1]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics 23, 3, 294--302.

Digital Library

[2]

Agarwala, A. 2007. Efficient gradient-domain compositing using quadtrees. ACM Transactions on Graphics 26, 3, 94.

Digital Library

[3]

Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. ACM Transactions on Graphics 25, 3, 637--645.

Digital Library

[4]

Burt, P. J., and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications COM-31, 4, 532--540.

[5]

Burt, P. J., and Adelson, E. H. 1983. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics 2, 4, 217--236.

Digital Library

[6]

Chen, J., Tang, C.-K., and Wang, J. 2009. Noise brush: interactive high quality image-noise separation. ACM Transactions on Graphics 28, 5, 1--10.

Digital Library

[7]

Cohen-Or, D., Sorkine, O., Gal, R., Leyvand, T., and Xu, Y.-Q. 2006. Color harmonization. ACM Transactions on Graphics 25, 3, 624--630.

Digital Library

[8]

Davis, T. A. 2006. Direct Methods for Sparse Linear Systems (Fundamentals of Algorithms 2). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA.

Digital Library

[9]

Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics 27, 3, 1--10.

Digital Library

[10]

Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Transactions on Graphics 28, 3, 1--9.

Digital Library

[11]

Fattal, R., Agrawala, M., and Rusinkiewicz, S. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Transactions on Graphics 26, 3, 51.

Digital Library

[12]

Georgiev, T. 2004. Photoshop healing brush: a tool for seamless cloning. In Workshop on Applications of Computer Vision (ECCV 2004), 1--8.

[13]

Gonzalez, R. C., and Woods, R. E. 2001. Digital Image Processing. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[14]

Heeger, D. J., and Bergen, J. R. 1995. Pyramid-based texture analysis/synthesis. SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 229--238.

Digital Library

[15]

Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Transactions on Graphics 25, 3, 631--637.

Digital Library

[16]

Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In IEEE International Conference on Computer Vision.

[17]

Levin, A., Zomet, A., Peleg, S., and Weiss, Y. 2004. Seamless image stitching in the gradient domain. In European Conference on Computer Vision.

[18]

Li, Y., Sharan, L., and Adelson, E. H. 2005. Compressing and companding high dynamic range images with subband architectures. ACM Transactions on Graphics 24, 3, 836--844.

Digital Library

[19]

McCann, J., and Pollard, N. S. 2008. Real-time gradient-domain painting. ACM Transactions on Graphics 27, 3, 1--7.

Digital Library

[20]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics 22, 3, 313--318.

Digital Library

[21]

Pitie, F., Kokaram, A. C., and Dahyot, R. 2005. N-dimensional probablility density function transfer and its application to colour transfer. In IEEE International Conference on Computer Vision.

Digital Library

[22]

Porter, T., and Duff, T. 1984. Compositing digital images. In SIGGRAPH '84: Proceedings of the 11th annual conference on Computer graphics and interactive techniques, 253--259.

Digital Library

[23]

Portilla, J., and Simoncelli, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1, 49--70.

Digital Library

[24]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5, 34--41.

Digital Library

[25]

Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics 23, 3, 315--321.

Digital Library

[26]

Szeliski, R. 2006. Locally adapted hierarchical basis preconditioning. ACM Transactions on Graphics 25, 3, 1135--1143.

Digital Library

[27]

Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In IEEE International Conference on Computer Vision.

Digital Library

[28]

Wang, J., Agrawala, M., and Cohen, M. F. 2007. Soft scissors: an interactive tool for realtime high quality matting. ACM Transactions on Graphics 26, 3, 9.

Digital Library

Cited By

Zhang SHuang HZhu Y(2023)Multi-Scale Context Aggregation Network for Inharmonious Region Localization2023 IEEE 15th International Conference on Advanced Infocomm Technology (ICAIT)10.1109/ICAIT59485.2023.10367321(412-416)Online publication date: 13-Oct-2023
https://doi.org/10.1109/ICAIT59485.2023.10367321
Sofiiuk KPopenova PKonushin A(2021)Foreground-aware Semantic Representations for Image Harmonization2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00166(1619-1628)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00166
Zhang HZhang JPerazzi FLin ZPatel V(2021)Deep Image Compositing2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00041(365-374)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00041
Show More Cited By

Index Terms

Multi-scale image harmonization
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Multi-scale image harmonization

Traditional image compositing techniques, such as alpha matting and gradient domain compositing, are used to create composites that have plausible boundaries. But when applied to images taken from different sources or shot under different conditions, ...
Drag-and-drop pasting

In this paper, we present a user-friendly system for seamless image composition, which we call drag-and-drop pasting. We observe that for Poisson image editing [Perez et al. 2003] to work well, the user must carefully draw a boundary on the source image ...
Nonlinear multiresolution image blending

We study contrast enhancement for multiresolution image blending. In image compositing, image stitching, and image fusion, a blending operator combines coefficients of a pixel array, an image pyramid, a wavelet decomposition, or a gradient domain ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '10: ACM SIGGRAPH 2010 papers

July 2010

984 pages

ISBN:9781450302104

DOI:10.1145/1833349

Conference Chair:
Tony DeRose,
Editor:
Hugues Hoppe
ACM Transactions on Graphics

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGGRAPH '10

Sponsor:

SIGGRAPH

SIGGRAPH '10: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 26 - 30, 2010

California, Los Angeles

Acceptance Rates

SIGGRAPH '10 Paper Acceptance Rate 103 of 390 submissions, 26%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
1,958
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang SHuang HZhu Y(2023)Multi-Scale Context Aggregation Network for Inharmonious Region Localization2023 IEEE 15th International Conference on Advanced Infocomm Technology (ICAIT)10.1109/ICAIT59485.2023.10367321(412-416)Online publication date: 13-Oct-2023
https://doi.org/10.1109/ICAIT59485.2023.10367321
Sofiiuk KPopenova PKonushin A(2021)Foreground-aware Semantic Representations for Image Harmonization2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00166(1619-1628)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00166
Zhang HZhang JPerazzi FLin ZPatel V(2021)Deep Image Compositing2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00041(365-374)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00041
Inoue NIto DHold‐Geoffroy YMai LPrice BYamasaki T(2020)RGB2AO: Ambient Occlusion Generation from RGB ImagesComputer Graphics Forum10.1111/cgf.1394339:2(451-462)Online publication date: 13-Jul-2020
https://doi.org/10.1111/cgf.13943
He MLiao JChen DYuan LSander P(2019)Progressive Color Transfer With Dense Semantic CorrespondencesACM Transactions on Graphics10.1145/329248238:2(1-18)Online publication date: 5-Apr-2019
https://dl.acm.org/doi/10.1145/3292482
Zhao YPrice BCohen SGurari D(2019)Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image2019 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV.2019.00166(1514-1523)Online publication date: Jan-2019
https://doi.org/10.1109/WACV.2019.00166
Vashisht MBhatia M(2019)Role of Mathematics in Image Processing2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon)10.1109/COMITCon.2019.8862438(538-543)Online publication date: Feb-2019
https://doi.org/10.1109/COMITCon.2019.8862438
Luan FParis SShechtman EBala K(2018)Deep Painterly HarmonizationComputer Graphics Forum10.1111/cgf.1347837:4(95-106)Online publication date: 20-Jul-2018
https://doi.org/10.1111/cgf.13478
Wang WShen WGuo SZhu RChen BSun Y(2018)Image Artistic Style Migration Based on Convolutional Neural Network2018 5th International Conference on Systems and Informatics (ICSAI)10.1109/ICSAI.2018.8599512(967-972)Online publication date: Nov-2018
https://doi.org/10.1109/ICSAI.2018.8599512
Zhao HShen XLin ZSunkavalli KPrice BJia J(2018)Compositing-Aware Image SearchComputer Vision – ECCV 201810.1007/978-3-030-01219-9_31(517-532)Online publication date: 7-Oct-2018
https://doi.org/10.1007/978-3-030-01219-9_31
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten