Video watercolorization using bidirectional texture advection

Abstract

In this paper, we present a method for creating watercolor-like animation, starting from video as input. The method involves two main steps: applying textures that simulate a watercolor appearance; and creating a simplified, abstracted version of the video to which the texturing operations are applied. Both of these steps are subject to highly visible temporal artifacts, so the primary technical contributions of the paper are extensions of previous methods for texturing and abstraction to provide temporal coherence when applied to video sequences. To maintain coherence for textures, we employ texture advection along lines of optical flow. We furthermore extend previous approaches by incorporating advection in both forward and reverse directions through the video, which allows for minimal texture distortion, particularly in areas of disocclusion that are otherwise highly problematic. To maintain coherence for abstraction, we employ mathematical morphology extended to the temporal domain, using filters whose temporal extents are locally controlled by the degree of distortions in the optical flow. Together, these techniques provide the first practical and robust approach for producing watercolor animations from video, which we demonstrate with a number of examples.

References

1. Alp, B., Haavisto, P., Jarske, T., Oistamo, K., and Neuvo, Y. A. 1990. Median-based algorithms for image sequence processing. In SPIE Vol. 1360, Visual Communications and Image Processing, 122--134.
2. Bousseau, A., Kaplan, M., Thollot, J., and Sillion, F. X. 2006. Interactive watercolor rendering with temporal coherence and abstraction. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 141 -- 149.
3. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics (Proc. SIGGRAPH 2005) 21, 3 (July), 243--248.
4. Collomosse, J. P., Rowntree, D., and Hall, P. M. 2005. Stroke surfaces: Temporally coherent artistic animations from video. IEEE Transactions on Visualization and Computer Graphics 11, 5 (Sept.), 540--549.
5. Cunzi, M., Thollot, J., Paris, S., Debunne, G., Gascuel, J.-D., and Durand, F. 2003. Dynamic canvas for immersive non-photorealistic walkthroughs. In Graphics Interface, 121--130.
6. Curtis, C. J., Anderson, S. E., Seims, J. E., Fleischer, K. W., and Salesin, D. H. 1997. Computer-generated water-color. In SIGGRAPH 97, 421--430.
7. Fang, H. 2006. Rototexture: Automated tools for texturing raw video. IEEE Transactions on Visualization and Computer Graphics 12, 6, 1580--1589.
8. Haralick, R. M., Sternberg, S. R., and Zhuang, X. 1987. Image analysis using mathematical morphology. IEEE Trans. Pattern Anal. Mach. Intell. 9, 4, 532--550.
9. Hays, J., and Essa, I. 2004. Image and video based painterly animation. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 113--120.
10. Hertzmann, A., and Perlin, K. 2000. Painterly rendering for video and interaction. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 7--12.
11. Horn, B. K. P. 1986. Robot Vision. MIT Press. ISBN 0-262-08159-8.
12. Jobard, B., Erlebacher, G., and Hussaini, M. Y. 2001. Lagrangian-eulerian advection for unsteady flow visualization. In VIS '01: Conference on Visualization '01, 53--60.
13. Johan, H., Hashimota, R., and Nishita, T. 2005. Creating watercolor style images taking into account painting techniques. Journal of the Society for Art and Science 3, 4, 207--215.
14. Klein, A. W., Sloan, P.-P. J., Finkelstein, A., and Cohen, M. F. 2002. Stylized video cubes. In ACM-SIGGRAPH/EG Symposium on Computer Animation (SCA), 15--22.
15. Kokaram, A. C. 1998. Motion Picture Restoration: Digital Algorithms for Artefact Suppression in Degraded Motion Picture Film and Video. Springer-Verlag. ISBN 3-540-76040-7.
16. Laveau, N., and Bernard, C. 2005. Structuring elements following the optical flow. In Mathematical Morphology: 40 Years On, Proceedings of the 7th International Symposium on Mathematical Morphology, Springer-Verlag, Ed., 43--52.
17. Lei, E., and Chang, C.-F. 2004. Real-time rendering of water-color effects for virtual environments. In IEEE 2004 Pacific-Rim Conference on Multimedia, 474--481.
18. Litwinowicz, P. C. 1997. Processing images and video for an impressionist effect. In SIGGRAPH 97, 407--414.
19. Luft, T., and Deussen, O. 2006. Real-time watercolor illustrations of plants using a blurred depth test. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 11--20.
20. Lum, E. B., and Ma, K.-L. 2001. Non-photorealistic rendering using watercolor inspired textures and illumination. In Pacific Graphics, 322--331.
21. Max, N., and Becker, B. 1995. Flow visualization using moving textures. In Proceedings of the ICASW/LaRC Symposium on Visualizing Time-Varying Data, 77--87.
22. Neyret, F. 2003. Advected textures. In ACM-SIGGRAPH/EG Symposium on Computer Animation (SCA), 147 -- 153.
23. Ozkan, M. K., Sezan, M. I., and Tekalp, A. M. 1993. Adaptive motion-compensated filtering of noisy image sequences. IEEE transactions on circuits and systems for video technology 3, 4, 277--290.
24. Sand, P., and Teller, S. 2006. Particle video: Long-range motion estimation using point trajectories. In CVPR, 2195 -- 2202.
25. Serra, J., and Vincent, L. 1992. An overview of morphological filtering. Circuits Syst. Signal Process. 11, 1, 47--108.
26. Sims, K. 1992. Choreographed image flow. The Journal of Visualization and Computer Animation 3, 1, 31--43.
27. Small, D. 1991. Modeling watercolor by simulating diffusion, pigment, and paper fibers. In SPIE, vol. 1460, 140--146.
28. Stam, J. 1999. Stable fluids. In SIGGRAPH 99, 121--128.
29. Van Laerhoven, T., Liesenborgs, J., and Van Reeth, F. 2004. Real-time watercolor painting on a distributed paper model. In Computer Graphics International, 640--643.
30. Wang, J. Y. A., and Adelson, E. H. 1994. Representing Moving Images with Layers. The IEEE Transaction on Image Processing Special Issue: Image Sequence Compression 3, 5, 625--638.
31. Wang, J., Xu, Y., Shum, H.-Y., and Cohen, M. 2004. Video tooning. ACM Transactions on Graphics (proc. of SIGGRAPH 2004) 23, 3, 574 -- 583.
32. WEISS, B. 2006. Fast median and bilateral filtering. ACM Transactions on Graphics (proc. of SIGGRAPH 2006) 25, 3, 519--526.
33. Winnemoller, H., Olsen, S. C., and Gooch, B. 2006. Realtime video abstraction. ACM Transactions on Graphics (proc. of SIGGRAPH 2006) 25, 3, 1221 -- 1226.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Appearance and texture representations

      • Computer vision tasks

        • Video summarization

  • Computer graphics

    • Animation

    • Image manipulation

      • Texturing