research-article

Practical temporal consistency for image-based graphics applications

Authors:

Aljoscha Smolic,

Markus GrossAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 31, Issue 4

Article No.: 34, Pages 1 - 8

https://doi.org/10.1145/2185520.2185530

Published: 01 July 2012 Publication History

Abstract

We present an efficient and simple method for introducing temporal consistency to a large class of optimization driven image-based computer graphics problems. Our method extends recent work in edge-aware filtering, approximating costly global regularization with a fast iterative joint filtering operation. Using this representation, we can achieve tremendous efficiency gains both in terms of memory requirements and running time. This enables us to process entire shots at once, taking advantage of supporting information that exists across far away frames, something that is difficult with existing approaches due to the computational burden of video data. Our method is able to filter along motion paths using an iterative approach that simultaneously uses and estimates per-pixel optical flow vectors. We demonstrate its utility by creating temporally consistent results for a number of applications including optical flow, disparity estimation, colorization, scribble propagation, sparse data up-sampling, and visual saliency computation.

Supplementary Material

MP4 File (tp115_12.mp4)

Download
19.65 MB

References

[1]

Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1, 1--31.

Digital Library

[2]

Bhat, P., Zitnick, C. L., Cohen, M. F., and Curless, B. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graph. 29, 2.

Digital Library

[3]

Chen, J., Paris, S., and Durand, F. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3, 103.

Digital Library

[4]

Criminisi, A., Sharp, T., Rother, C., and Pérez, P. 2010. Geodesic image and video editing. ACM Trans. Graph. 29, 5, 134.

Digital Library

[5]

Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In CVPR, 1141--1148.

[6]

Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21, 3, 257--266.

Digital Library

[7]

Gastal, E. S. L., and Oliveira, M. M. 2011. Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30, 4, 69.

Digital Library

[8]

Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. In CVPR, IEEE Computer Society.

[9]

He, K., Sun, J., and Tang, X. 2010. Guided image filtering. In ECCV (1), Springer, vol. 6311 of Lecture Notes in Computer Science, 1--14.

Digital Library

[10]

Höffken, M., Oberhoff, D., and Kolesnik, M. 2011. Temporal prediction and spatial regularization in differential optical flow. In ACIVS, Springer, vol. 6915 of Lecture Notes in Computer Science, 576--585.

Digital Library

[11]

Horn, B. K. P., and Schunck, B. G. 1981. Determining optical flow. Artif. Intell. 17, 1--3, 185--203.

Digital Library

[12]

Hosni, A., Rhemann, C., Bleyer, M., and Gelautz, M. 2011. Temporally consistent disparity and optical flow via efficient spatio-temporal filtering. In PSIVT (1), Springer, vol. 7087 of Lecture Notes in Computer Science, 165--177.

Digital Library

[13]

Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26, 3, 96.

Digital Library

[14]

Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. H. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5.

Digital Library

[15]

Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. 23, 3, 689--694.

Digital Library

[16]

Levin, A., Lischinski, D., and Weiss, Y. 2006. A closed form solution to natural image matting. In CVPR (1), IEEE Computer Society, 61--68.

Digital Library

[17]

Nehab, D., Maximo, A., Lima, R. S., and Hoppe, H. 2011. Gpu-efficient recursive filtering and summed-area tables. ACM Trans. Graph. 30, 6, 176.

Digital Library

[18]

Paris, S., Kornprobst, P., and Tumblin, J. 2009. Bilateral Filtering. Now Publishers Inc., Hanover, MA, USA.

Digital Library

[19]

Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 7, 629--639.

Digital Library

[20]

Rhemann, C., Hosni, A., Bleyer, M., Rother, C., and Gelautz, M. 2011. Fast cost-volume filtering for visual correspondence and beyond. In CVPR, IEEE, 3017--3024.

Digital Library

[21]

Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 1--3, 7--42.

Digital Library

[22]

Sun, D., Roth, S., and Black, M. J. 2010. Secrets of optical flow estimation and their principles. In CVPR, 2432--2439.

[23]

Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In ICCV, 839--846.

Digital Library

[24]

Volz, S., Bruhn, A., Valgaerts, L., and Zimmer, H. 2011. Modeling temporal coherence for optical flow. In Proc. 13th International Conference on Computer Vision (ICCV), IEEE Computer Society Press, Barcelona.

Digital Library

[25]

Wang, O., Lang, M., Frei, M., Hornung, A., Smolic, A., and Gross, M. H. 2011. Stereobrush: Interactive 2d to 3d conversion using discontinuous warps. In SBM, Eurographics Association, 47--54.

Digital Library

[26]

Wildeboer, M. O., Yendo, T., Tehrani, M. P., and Tanimoto, M. 2010. A semi-automatic multi-view depth estimation method. Proceedings of SPIE, Visual Communications and Image Processing 7744.

[27]

Xiao, J., Cheng, H., Sawhney, H. S., Rao, C., and Isnardi, M. A. 2006. Bilateral filtering-based optical flow estimation with occlusion detection. In ECCV (1), Springer, vol. 3951 of Lecture Notes in Computer Science, 211--224.

Digital Library

[28]

Yang, Q., Yang, R., Davis, J., and Nistér, D. 2007. Spatial-depth super resolution for range images. In CVPR, IEEE Computer Society.

[29]

Zimmer, H., Bruhn, A., and Weickert, J. 2011. Optic flow in harmony. International Journal of Computer Vision 93, 3, 368--388.

Digital Library

Cited By

Fulari AMulleti SRajwade A(2024)Unsupervised Model-based Learning for Simultaneous Video Deflickering and Deblotching2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00407(4105-4113)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00407
Wu STan HTian ZChen YQi XJia J(2024)SaCo Loss: Sample-Wise Affinity Consistency for Vision-Language Pre-Training2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02583(27348-27359)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02583
Chen XFang LYe LZhang Q(2024)Deep Video Harmonization by Improving Spatial-temporal ConsistencyMachine Intelligence Research10.1007/s11633-023-1447-321:1(46-54)Online publication date: 15-Jan-2024
https://doi.org/10.1007/s11633-023-1447-3
Show More Cited By

Recommendations

Blind video temporal consistency

Extending image processing techniques to videos is a non-trivial task; applying processing independently to each video frame often leads to temporal inconsistencies, and explicitly encoding temporal consistency requires algorithmic changes. We describe a ...
Occlusion-aware Video Temporal Consistency
MM '17: Proceedings of the 25th ACM international conference on Multimedia

Image color editing techniques such as color transfer, HDR tone mapping, dehazing, and white balance have been widely used and investigated in recent decades. However, naively employing them to videos frame-by-frame often leads to flickering or color ...
Foveated 3D graphics

We exploit the falloff of acuity in the visual periphery to accelerate graphics computation by a factor of 5-6 on a desktop HD display (1920x1080). Our method tracks the user's gaze point and renders three image layers around it at progressively higher ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 31, Issue 4

July 2012

935 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2185520

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2012

Published in TOG Volume 31, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

83
Total Citations
View Citations
1,530
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fulari AMulleti SRajwade A(2024)Unsupervised Model-based Learning for Simultaneous Video Deflickering and Deblotching2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00407(4105-4113)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00407
Wu STan HTian ZChen YQi XJia J(2024)SaCo Loss: Sample-Wise Affinity Consistency for Vision-Language Pre-Training2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02583(27348-27359)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02583
Chen XFang LYe LZhang Q(2024)Deep Video Harmonization by Improving Spatial-temporal ConsistencyMachine Intelligence Research10.1007/s11633-023-1447-321:1(46-54)Online publication date: 15-Jan-2024
https://doi.org/10.1007/s11633-023-1447-3
Shekhar SReimann MHilscher MSemmo ADöllner JTrapp M(2023)Interactive Control over Temporal Consistency while Stylizing Video StreamsComputer Graphics Forum10.1111/cgf.1489142:4Online publication date: 26-Jul-2023
https://doi.org/10.1111/cgf.14891
Lei CRen XZhang ZChen Q(2023)Blind Video Deflickering by Neural Filtering with a Flawed Atlas2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01006(10439-10448)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01006
Suri Z(2023)Pose Constraints for Consistent Self-supervised Monocular Depth and Ego-MotionImage Analysis10.1007/978-3-031-31438-4_23(340-353)Online publication date: 18-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-31438-4_23
Sheng BLi PAli RChen C(2022)Improving Video Temporal Consistency via Broad Learning SystemIEEE Transactions on Cybernetics10.1109/TCYB.2021.307931152:7(6662-6675)Online publication date: Jul-2022
https://doi.org/10.1109/TCYB.2021.3079311
Wei YJia ZYang JKasabov N(2021)High-Brightness Image Enhancement AlgorithmApplied Sciences10.3390/app11231149711:23(11497)Online publication date: 4-Dec-2021
https://doi.org/10.3390/app112311497
Zhang YWang CCui MRen PXie XHua XBao HHuang QXu WShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Attention-guided Temporally Coherent Video Object MattingProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475623(5128-5137)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475623
Abbasi AToosi RAkhaee M(2021)Fast and Temporal Consistent Video Style Transfer2021 5th International Conference on Pattern Recognition and Image Analysis (IPRIA)10.1109/IPRIA53572.2021.9483531(1-6)Online publication date: 28-Apr-2021
https://doi.org/10.1109/IPRIA53572.2021.9483531
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents