research-article

Painterly animation using video semantics and feature correspondence

Authors:

Song-Chun ZhuAuthors Info & Claims

NPAR '10: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering

Pages 73 - 80

https://doi.org/10.1145/1809939.1809948

Published: 07 June 2010 Publication History

Abstract

We present an interactive system that stylizes an input video into a painterly animation. The system consists of two phases. The first is an Video Parsing phase that extracts and labels semantic objects with different material properties (skin, hair, cloth, and so on) in the video, and then establishes robust correspondence between frames for discriminative image features inside each object. The second Painterly Rendering phase performs the stylization based on the video semantics and feature correspondence. Compared to the previous work, the proposed method advances painterly animation in three aspects: Firstly, we render artistic painterly styles using a rich set of example-based brush strokes. These strokes, placed in multiple layers and passes, are automatically selected according to the video semantics. Secondly, we warp brush strokes according to global object deformations, so that the strokes appear to be tightly attached to the object surfaces. Thirdly, we propose a series of novel teniques to reduce the scintillation effects. Results applying our system to several video clips show that it produces expressive oil painting animations.

References

[1]

Agarwala, A. 2002. SnakeToonz: A Semi-Automatic Approach to Creating Cel Animation from Video. Int'l Symposium on Non-Photorealistic Animation and Rendering. 139--146.

Digital Library

[2]

Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based Tracking for Rotoscoping and Animation. ACM Trans. Graphics (Proc. SIGGRAPH). 23, 3, 584--591.

Digital Library

[3]

Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video SnapCut: Robust Video Object Cutout Using Localized Classifiers. ACM Trans. Graphics (Proc. SIGGRAPH).

Digital Library

[4]

Bookstein, L. 1989. Principal Warps: Thin-plate Splines and the Decomposition of Deformations. IEEE Trans. on Pattern Anal. and Mach. Intell. 11, 6, 567--585.

Digital Library

[5]

Bousseau, A., Neyret, F., Thollot, J., and Salesin D. 2007. Video Watercolorization using Bidirectional Texture Advection. ACM Trans. Graphics (Proc. SIGGRAPH). 26, 3, 104:1--104:7.

Digital Library

[6]

Boykov, Y., and Jolly, P. 2001. Fast Approximate Energy Minimization via Graph Cuts. IEEE Trans. on Pattern Anal. and Mach. Intell. 23, 11, 1222--1238.

Digital Library

[7]

Breslav, S., Szerszen, K., Markosian, L., Barla, P., and Thollot, J. 2007. Dynamic 2D Patterns for Shading 3D Scenes. ACM Trans. Graphics (Proc. SIGGRAPH).

Digital Library

[8]

Chen, H., and Zhu, S. C. 2006. A Generative Sketch Model for Human Hair Analysis and Synthesis. IEEE Trans. on Pattern Anal. and Mach. Intell. 28, 7, 1025--1040.

Digital Library

[9]

Collomosse, P. and Hall, M. 2002. Painterly rendering using image salience. Eurographics. 122--128.

Digital Library

[10]

Collomosse, P., Rowntree, D., and Hall, M. 2005. Stroke Surfaces: Temporally Coherent Artistic Animations from Video. IEEE Trans. on VCG. 11, 5, 540--549.

Digital Library

[11]

Hays, J., and Essa, I. 2004. Image and Video Based Painterly Animation. Int'l Symposium on Non-Photorealistic Animation and Rendering, 113--120.

Digital Library

[12]

Hertzmann, A. 1998. Painterly Rendering with Curved Brush Strokes of Multiple Sizes. Proc. SIGGRAPH. 453--460.

Digital Library

[13]

Hertzmann, A., and Perlin, K. 2000. Painterly Rendering for Video and Interaction. Int'l Symposium on Non-Photorealistic Animation and Rendering, 7--12.

Digital Library

[14]

Jonker, R., and Volgenant, A. 1987. A Shortest Augmenting Path Algorithm for Dense and Sparse Linear Assignment Problems. Computing. 38, 325--340.

Digital Library

[15]

Kalnins, R. Davidson, P. Markosian, L. and Finkelstein, A. 2003. Coherent Stylized Silhouettes. ACM Trans. Graphics (Proc. SIGGRAPH). 22, 3, 856--861.

Digital Library

[16]

Klein, W., Sloan, J., Finkelstein, A., and Cohen, F. 2002. Stylized Video Cubes. ACM SIGGRAPH Symposium on Computer Animation.

Digital Library

[17]

Kolliopoulos, A., Wang, J. M., and Hertzmann, A. 2006. Segmentation-Based 3D Artistic Rendering. Proc. Eurographics Symposium on Rendering. 361--370.

Digital Library

[18]

Litwinowicz, P. 1997. Processing Image and Video for An Impressionist Effect. Proc. of SIGGRAPH. 407--414.

Digital Library

[19]

Lowe, D. 2004. Distinctive Image Features from Scale-Invariant Keypoints. Int'l Journal of Computer Vision. 60, 2, 91--110.

Digital Library

[20]

Matas, J., Chum, O., Urban, M. and Pajdla, T. 2002. Robust Wide Baseline Stereo from Maximally Stable Extremal Regions. Proc. British Machine Vision Conference. 384--393.

[21]

Meier, B. 1996. Painterly Rendering For Animation. ACM Trans. Graphics (Proc. SIGGRAPH). 477--484.

Digital Library

[22]

Nocedal, J., and Wright, S. J. 1999. Numerical Optimization. Springer.

[23]

Roweis, S. and Saul, L. 2000. Nonlinear dimensionality reduction by locally linear embedding. Science. 290, 5500, 2323--2326.

[24]

Santella A. and Decarlo D. 2002. Abstracted painterly renderings using eye-tracking data. Int'l Symposium on Non-Photorealistic Animation and Rendering. 769--776.

Digital Library

[25]

Shotton, J., Winn, J., Rother, C., and Criminisi, A. 2009. TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context, International Journal of Computer Vision. 81, 1, 2--23.

Digital Library

[26]

Wang, J., Xu Y., Shum, H. Y., and Cohen, F. 2004. Video Tooning. ACM Trans. Graphics (Proc. SIGGRAPH). 23, 3, 574--583.

Digital Library

[27]

Wang, J., Bhat, P., Colburn, R., Agarwala M. and Cohen, M. F. 2005. Interactive Video Cutout. ACM Trans. Graphics (Proc. SIGGRAPH). 24, 3, 585--594.

Digital Library

[28]

Winnemoller, H., Olsen, S. C., and Gooch, B. 2006. Real-Time Video Abstraction. ACM Trans. Graphics (Proc. SIGGRAPH). 25, 3, 1221--1226.

Digital Library

[29]

Xu, S., Xu, Y., Kang, S. B., Salesin, D. H., Pan, Y., and Shum, H.-Y. 2006. Animating Chinese Painting Through Stroke-based Decomposition. ACM Trans. Graphics. 25, 2, 239--267.

Digital Library

[30]

Zeng, K., Zhao, M. T., and Zhu, S. C. 2009. From Image Parsing to Painterly Rendering. ACM Trans. on Graphics. 29. 1, 1--11.

Digital Library

Cited By

Ilinkin ISong DKim Y(2023)Stroke-Based Rendering and Planning for Robotic Performance of Artistic Drawing2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10341808(8062-8068)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10341808
Tong ZWang XYuan SChen XWang JFang XMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive SamplingProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547759(1035-1046)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547759
Yang Min (2019)A Multi-Column Deep Framework for Recognizing Artistic MediaElectronics10.3390/electronics81112778:11(1277)Online publication date: 2-Nov-2019
https://doi.org/10.3390/electronics8111277
Show More Cited By

Index Terms

Painterly animation using video semantics and feature correspondence
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Tracking
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Authoring and animating painterly characters

Artists explore the visual style of animated characters through 2D concept art, since it affords them a nearly unlimited degree of creative freedom. Realizing the desired visual style, however, within the 3D character animation pipeline is often ...
Real-time painterly rendering for MR applications
GRAPHITE '04: Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia

In this paper we describe a real-time system for AR/MR rendering applications in a painterly style. Impressionistic images are created using a large number of brush strokes, which are organized as 3d particles to achieve frame-to-frame coherence. ...
Real-Time Painterly Rendering of Terrains
ICVGIP '08: Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing

We present a non-photo realistic, real-time painterly rendering technique for terrains. The painterly appearance and the impression of terrains is created by effectively rendering several brush strokes. The strokes have fixed locations on the surfaces ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

NPAR '10: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering

June 2010

183 pages

ISBN:9781450301251

DOI:10.1145/1809939

General Chairs:
Morgan McGuire
Williams College
,
John Collomosse
University of Surrey

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Adobe
CNRS: Centre National De La Rechercue Scientifique
INRIA: Institut Natl de Recherche en Info et en Automatique

In-Cooperation

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
715
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ilinkin ISong DKim Y(2023)Stroke-Based Rendering and Planning for Robotic Performance of Artistic Drawing2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10341808(8062-8068)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10341808
Tong ZWang XYuan SChen XWang JFang XMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive SamplingProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547759(1035-1046)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547759
Yang Min (2019)A Multi-Column Deep Framework for Recognizing Artistic MediaElectronics10.3390/electronics81112778:11(1277)Online publication date: 2-Nov-2019
https://doi.org/10.3390/electronics8111277
Huang ZZhou SHeng W(2019)Learning to Paint With Model-Based Deep Reinforcement Learning2019 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV.2019.00880(8708-8717)Online publication date: Oct-2019
https://doi.org/10.1109/ICCV.2019.00880
Yang HMin K(2019)Classification of basic artistic media based on a deep convolutional approachThe Visual Computer10.1007/s00371-019-01641-6Online publication date: 28-Feb-2019
https://doi.org/10.1007/s00371-019-01641-6
Liu JYang WSun XZeng W(2018)Photo Stylistic Brush: Robust Style Transfer via Superpixel-Based Bipartite GraphIEEE Transactions on Multimedia10.1109/TMM.2017.278076120:7(1724-1737)Online publication date: Jul-2018
https://doi.org/10.1109/TMM.2017.2780761
Selim AElgharib MDoyle L(2016)Painting style transfer for head portraits using convolutional neural networksACM Transactions on Graphics10.1145/2897824.292596835:4(1-18)Online publication date: 11-Jul-2016
https://dl.acm.org/doi/10.1145/2897824.2925968
Tang YZhang YShi XFan J(2015)Multi-style video stylization based on texture advection基于纹理传输的多风格视频艺术化处理Science China Information Sciences10.1007/s11432-014-5255-958:11(1-13)Online publication date: 8-Sep-2015
https://doi.org/10.1007/s11432-014-5255-9
Seo SLee H(2015)Pixel based stroke generation for painterly effect using maximum homogeneity neighbor filterMultimedia Tools and Applications10.1007/s11042-013-1835-874:10(3317-3328)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1007/s11042-013-1835-8
Villegas JForbes AHua KRui YSteinmetz RHanjalic ANatsev AZhu W(2014)Analysis/synthesis approaches for creatively processing video signalsProceedings of the 22nd ACM international conference on Multimedia10.1145/2647868.2654944(37-46)Online publication date: 3-Nov-2014
https://dl.acm.org/doi/10.1145/2647868.2654944
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten