Perceptual Temporal Incoherence Aware Stereo Video Retargeting

ABSTRACT

Stereo video retargeting aims to avoid shape and depth distortions while maintaining temporal coherence of shape and depth while resizing a stereo video to a desired size. Existing methods resort to extending stereo image retargeting schemes to stereo video retargeting by imposing temporal constraints to consistently resize all corresponding regions so as to maintain temporal coherence. However, such a direct extension often incurs conflicts among the requirements for preserving shape information and depth information and maintaining their temporal coherence, thereby failing to meet one or more of these requirements. We find that properly relaxing temporal constraints for non-paired regions at frame boundaries can effectively mitigate conflicts among depth, shape, and temporal constraints without severely degrading temporal coherence perceptually. Based on this new finding, we derive effective temporal constraints to improve the viewing experience of a 3D scene for stereo video retargeting. Accordingly, we propose an efficient grid-based implementation for our method. Experimental results show that our method achieves superior visual quality over existing methods.

References

1. S. Anstis. 1998. Picturing peripheral acuity. Perception 27, 7 (1998), 817--825.
2. T. D. Basha, Y. Moses, and S. Avidan. 2013. Stereo Seam Carving a Geometrically Consistent Approach. IEEE Trans. Pattern Anal. Mach. Intell. 35 (2013), 2513--2525.
3. B.Li, L.-Y. Duan, C.-W. Lin, and W. Gao. 2014. Region-based depth-preserving stereoscopic image retargeting. In Proc. IEEE Int. Conf. Image Process. 2903--2907.
4. C.-H. Chang, C.-K. Liang, and Y.-Y. Chuang. 2011. Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans. Multimedia 13 (2011), 589--601.
5. A. Duchowski. 2007. Eye tracking methodology: Theory and practice. Vol. 373. Springer Science & Business Media.
6. K. He, H. Chang, and J. Sun. 2013. Content-aware rotation. In Proc. IEEE Int. Conf. Comput. Vis. 553--560.
7. C.-C. Hsu, C.-W. Lin, Y. Fang, and W. Lin. 2014. Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss. IEEE J. Sel. Topics Signal Process. (2014), 377--389.
8. S. Kopf, B. Guthier, C. Hipp, J. Kiess, and W. Effelsberg. 2014. Warping-based retargeting for stereoscopic video. In IEEE Int. Conf. Image Process. 2898--2902.
9. P. Krähenbühl, M. Lang, A. Hornung, and M. Gross. 2009. A system for retargeting of streaming video. ACM Trans. Graph. (2009), 126:1--126:10.
10. M. Lang, A. Hornung, O. Wang, S. Poulakos, A. Smolic, and M. Gross. 2010. Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graph. 29, 4 (2010), 75:1--75:10.
11. K.-Y. Lee, C.-D. Chung, and Y.-Y. Chuang. 2012. Scene warping: Layer-based stereoscopic image resizing. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recongnit. 49--56.
12. B. Li, Y. Chen, J. Wang, L.-Y. Duan, and W. Gao. 2011. Fast retargeting with adaptive grid optimization. In Proc. IEEE Int. Conf. Mutimedia Expo. 1--4.
13. B. Li, L.-Y. Duan, C.-W. Lin, T. Huang, and W. Gao. 2015. Depth-preserving warping for stereo image retargeting. IEEE Trans. Image Process. 24, 9 (2015), 2811--2826.
14. B. Li, L.-Y. Duan, J.Wang, J. Chen, R. Ji, andW. Gao. 2011. Grid-based retargeting with transformation consistency smoothing. In Proc. Multimedia Modeling.
15. B. Li, L.-Y. Duan, J. Wang, R. Ji, C.-W. Lin, and W. Gao. 2014. Spatiotemporal Grid Flow for Video Retargeting. IEEE Trans. Image Process. 23, 4 (2014), 1615--1628.
16. H.-S. Lin, S.-H. Guan, C.-T. Lee, and M. Ouhyoung. 2011. Stereoscopic 3D Experience Optimization Using Cropping and Warping. In SIGGRAPH Asia (SA '11). 40:1--40:2.
17. S.-S. Lin, C.-H. Lin, S.-H. Chang, and T.-Y. Lee. 2014. Object-coherence warping for stereoscopic image retargeting. IEEE Trans. Circuits Syst. Video Technol. 24, 5 (2014), 759--768.
18. S.-S. Lin, C.-H. Lin, Y.-H. Kuo, and T.-Y. Lee. 2016. Consistent volumetric warping using floating boundaries for stereoscopic video retargeting. IEEE Trans. Circuits Syst. Video Technol. 26, 5 (May 2016), 801--813.
19. Y. Niu, W.-C. Feng, and F. Liu. 2012. Enabling Warping on Stereoscopic Images. ACM Trans. Graph. (2012).
20. J. Nocedal and S. Wright. 2006. Numerical Optimization (2nd ed.). Springer.
21. M. Rubinstein, A. Shamir, and S. Avidan. 2008. Improved seam carving for video retargeting. ACM Trans. Graph. (2008).
22. A. Shamir and O. Sorkine. 2009. Visual media retargeting. In ACM SIGGRAPH ASIA 2009 Courses. 11:1--11:13.
23. K. Utsugi, T. Shibahara, T. Koike, K. Takahashi, and T. Naemura. 2010. Seam carving for stereo images. In Proc. 3DTV-Conference. 1--4.
24. Y.-S. Wang, H. Fu, O. Sorkine, T.-Y. Lee, and H.-P. Seidel. 2009. Motion-Aware Temporal Coherence for Video Resizing. ACM Trans. Graph. (2009).
25. Y.-S. Wang, J.-H. Hsiao, O. Sorkine, and T.-Y. Lee. 2011. Scalable and Coherent Video Resizing with Per-Frame Optimization. ACM Trans. Graph. (2011).
26. Y.-S. Wang, H.-C. Lin, O. Sorkine, and T.-Y. Lee. 2010. Motion-based Video Retargeting with Optimized Crop-and-Warp. ACM Trans. Graph. (2010).
27. L.Wolf, M. Guttmann, and D. Cohen-Or. 2007. Non-homogeneous Content-driven Video-retargeting.. In Proc. IEEE Conf. Comput. Vis. 1--6.
28. T.-C. Yen, C.-M. Tsai, and C.-W. Lin. 2011. Maintaining Temporal Coherence in Video Retargeting Using Mosaic-Guided Scaling. IEEE Trans. Image Process. (2011), 2339--2351.
29. J.W. Yoo, S. Yea, and I. K. Park. 2013. Content-Driven Retargeting of Stereoscopic Images. IEEE Signal Process. Lett. (2013), 519--522.
30. Y.-F. Zhang, S.-M. Hu, and R. R. Martin. 2008. Shrinkability maps for contentaware video resizing. Comput. Graph. Forum (2008), 1797--1804.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

  • Computer graphics

    • Image manipulation