Transfusive image manipulation

Abstract

We present a method for consistent automatic transfer of edits applied to one image to many other images of the same object or scene. By introducing novel, content-adaptive weight functions we enhance the non-rigid alignment framework of Lucas-Kanade to robustly handle changes of view point, illumination and non-rigid deformations of the subjects. Our weight functions are content-aware and possess high-order smoothness, enabling to define high-quality image warping with a low number of parameters using spatially-varying weighted combinations of affine deformations. Optimizing the warp parameters leads to subpixel-accurate alignment while maintaining computation efficiency. Our method allows users to perform precise, localized edits such as simultaneous painting on multiple images in real-time, relieving them from tedious and repetitive manual reapplication to each individual image.

References

1. Agarwala, A., Hertzmann, A., Salesin, D., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. Graph. 23, 3, 584--591.
2. An, X., and Pellacini, F. 2008. AppProp: all-pairs appearance-space edit propagation. ACM Trans. Graph. 27, 3.
3. Andersen, E. D., and Andersen, K. D. 2000. The mosek interior point optimizer for linear programming: an implementation of the homogeneous algorithm. In High Performance Optimization. Kluwer Academic Publishers, 197--232.
4. Baker, S., and Matthews, I. 2004. Lucas-Kanade 20 years on: A unifying framework. Int. J. Comput. Vision 56, 3, 221--255.
5. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2011. A database and evaluation methodology for optical flow. Int. J. Comput. Vision 92, 1, 1--31.
6. Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The generalized patchmatch correspondence algorithm. In Proc. ECCV.
7. Berthouzoz, F., Li, W., Dontcheva, M., and Agrawala, M. 2011. A framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations. ACM Trans. Graph. 30, 5, 120.
8. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Curless, B., Cohen, M., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Proc. EGSR, 327--338.
9. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P. N., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. ACM Trans. Graph. 27, 3.
10. Brand, M., and Pletscher, P. 2008. A conditional random field for automatic photo editing. In Proc. CVPR.
11. Cohen-Or, D., Sorkine, O., Gal, R., Leyvand, T., and Xu, Y.-Q. 2006. Color harmonization. ACM Trans. Graph. 25, 3.
12. Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27, 3.
13. Farbman, Z., Fattal, R., and Lischinski, D. 2010. Diffusion maps for edge-aware image editing. ACM Trans. Graph. 29, 6.
14. Fattal, R., Carroll, R., and Agrawala, M. 2009. Edge-based image coarsening. ACM Trans. Graph. 29, 1, 1--11.
15. Fattal, R. 2009. Edge-avoiding wavelets and their applications. ACM Trans. Graph. 28, 3.
16. Finch, M., Snyder, J., and Hoppe, H. 2011. Freeform vector graphics with controlled thin-plate splines. ACM Trans. Graph. 30, 6, 166:1--166:10.
17. HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graph. 30, 4.
18. Hasinoff, S. W., Jóźwiak, M., Durand, F., and Freeman, W. T. 2010. Search-and-replace editing for personal photo collections. In Proc. ICCP.
19. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. Graph. 26, 3.
20. Jacobson, A., and Sorkine, O. 2012. A cotangent laplacian for images as surfaces. Tech. Rep. 757, ETH Zurich, April.
21. Jacobson, A., Baran, I., Popovic, J., and Sorkine, O. 2011. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph. 30, 4.
22. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. 23, 3, 689--694.
23. Li, Y., Adelson, E. H., and Agarwala, A. 2008. Scribble-boost: Adding classification to edge-aware interpolation of local image and video adjustments. Comput. Graph. Forum 27, 4.
24. Li, Y., Ju, T., and Hu, S.-M. 2010. Instant propagation of sparse edits on images and videos. Comput. Graph. Forum 29, 7.
25. Lischinski, D., Farbman, Z., Uyttendaele, M., and Szeliski, R. 2006. Interactive local adjustment of tonal values. ACM Trans. Graph. 25, 3, 646--653.
26. Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. 2008. Sift flow: Dense correspondence across different scenes. In Proc. ECCV: Part III, 28--42.
27. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110.
28. Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc. International Joint Conference on Artificial Intelligence.
29. Meyer, M., Desbrun, M., Schröder, P., and Barr, A. H. 2003. Discrete differential-geometry operators for triangulated 2-manifolds. In Visualization and Mathematics III. 35--57.
30. Photoshop. 2012. Version 12.0.4. Adobe Systems, Inc.
31. Pinkall, U., and Polthier, K. 1993. Computing discrete minimal surfaces and their conjugates. Experiment. Math. 2, 1.
32. Rav-Acha, A., Kohli, P., Rother, C., and Fitzgibbon, A. W. 2008. Unwrap mosaics: a new representation for video editing. ACM Trans. Graph. 27, 3.
33. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. and Applications 21, 5, 34--41.
34. Sand, P., and Teller, S. J. 2004. Video matching. ACM Trans. Graph. 23, 3, 592--599.
35. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision 47, 1--3, 7--42.
36. Zimmer, H., Bruhn, A., and Weickert, J. 2011. Optic flow in harmony. Int. J. Comput. Vision 93, 3, 368--388.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Computer graphics

    • Image manipulation

    • Rendering