Video tapestries with continuous temporal zoom

Abstract

We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly into the image to reveal additional temporal details. We call these artifacts tapestries because their continuous nature is akin to medieval tapestries and other narrative depictions predating the advent of motion pictures. We propose a set of criteria for such a summarization, and a series of optimizations motivated by these criteria. These can be performed as an entirely offline computation to produce high quality renderings, or by adjusting some optimization parameters the later stages can be solved in real time, enabling an interactive interface for video navigation. Our video tapestries combine the best aspects of two common visualizations, providing the visual clarity of DVD chapter menus with the information density and multiple scales of a video editing timeline representation. In addition, they provide continuous transitions between zoom levels. In a user study, participants preferred both the aesthetics and efficiency of tapestries over other interfaces for visual browsing.

References

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graphics 23, 3, 294--302.
2. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: pose selection and illustration. In ACM Intl. Conference on Computer Graphics and Interactive Techniques, 667--676.
3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graphics 28, 3.
4. Berkhin, P. 2002. Grouping Multidimensional Data: A survey of clustering data mining techniques. Springer.
5. Bernstein, S. 1994. Film Production, Second Edition. Focal Press.
6. Boreczky, J., Girgensohn, A., Golovchinsky, G., and Uchihashi, S. 2000. An interactive comic book presentation for exploring video. In Proceedings of SIGCHI, ACM, 185--192.
7. Bourdev, L., and Brandt, J. 2005. Robust object detection via soft cascade. In IEEE CVPR 2005, vol. 2.
8. Chiu, P., Girgensohn, A., and Liu, Q. 2004. Stained-glass visualization for highly condensed video summaries. In IEEE ICME 2004.
9. Christel, M., Hauptmann, A., Wactlar, H., and Ng, T. 2002. Collages as dynamic summaries for news video. In ACM Multimedia, 561--569.
10. Cockburn, A., Karlson, A., and Bederson, B. B. 2008. A review of overview+detail, zooming, and focus+context interfaces. ACM Comput. Surv. 41, 1, 1--31.
11. Correa, C. D., and Ma, K.-L. 2010. Dynamic video narratives. ACM Trans. Graphics 29, 3.
12. Davis, M. 1995. Media streams: representing video for retrieval and repurposing. PhD thesis, Wesleyan University.
13. Dementhon, D., Kobla, V., and Doermann, D. 1998. Video summarization by curve simplifiation. In ACM Multimedia, 211--218.
14. Hauser, T. 2008. The Art of Wall-E. Chronicle Books LLC.
15. Kang, H., Matsushita, Y., Tang, X., Chen, X., Hefei, P., and Beijing, P. 2006. Space-time video montage. In CVPR06, 1331--1338.
16. Kim, K., Essa, I., and Abowd, G. D. 2006. Interactive mosaic generation for video navigation. In ACM Multimedia, 655--658.
17. Kraaij, W., Smeaton, A., Over, P., and Arlandis, J. 2004. Trecvid 2004-an overview. In TRECVID video retrieval online proceedings.
18. Kwatra, V., Schdl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graphics 22, 3, 277--286.
19. Ma, Y., and Zhang, H. 2002. A model of motion attention for video skimming. In Proc. Image Processing, Int'l Conf., vol. 1, I-129--I-132.
20. Mei, T., Yang, B., Yang, S., and Hua, X. 2009. Video collage: presenting a video sequence using a single image. The Visual Computer 25, 1, 39--51.
21. Murch, W. 1995. In the Blink of an Eye: A Perspective on Film Editing. Silman-James Press, Los Angeles.
22. Rother, C., Kumar, S., Kolmogorov, V., and Blake, A. 2005. Digital tapestry. In IEEE CVPR, I: 589--596.
23. Rother, C., Bordeaux, L., Hamadi, Y., and Blake, A. 2006. Autocollage. ACM Trans. Graphics 25, 3, 847--852.
24. Shipman, F., Girgensohn, A., and Wilcox, L. 2003. Generation of interactive multi-level video summaries. In ACM Multimedia, 392--401.
25. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR 2008.
26. Sivic, J., Kaneva, B., Torralba, A., Avidan, S., and Freeman, W. 2008. Creating and exploring a large photorealistic virtual space. In IEEE CVPR Workshops, 2008., 1--8.
27. Smith, M., and Kanade, T. 1995. Video skimming for quick browsing based on audio and image characterization. Technical Report CMU-CS-95-186, School of Computer Science, Carnegie Mellon University.
28. Smith, M., and Kanade, T. 1997. Video skimming and characterization through the combination of image and language understanding techniques. In 1997 IEEE CVPR, 775--781.
29. Taniguchi, Y., Akutsu, A., and Tonomura, Y. 1997. PanoramaExcerpts: Extracting and packing panoramas for video browsing. In ACM Multimedia, 427--436.
30. Truong, B. T., and Venkatesh, S. 2007. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl. 3, 1, 3.
31. Uchihashi, S., Foote, J., Girgensohn, A., and Boreczky, J. 1999. Video manga: generating semantically meaningful video summaries. In ACM Multimedia, ACM, 383--392.
32. Wang, T., Mei, T., Hua, X.-S., Liu, X., and Zhou, H.-Q. 2007. Video collage: A novel presentation of video sequence. In ICME, IEEE, 1479--1482.
33. Yang, B., Mei, T., Sun, L.-F., Yang, S.-Q., and Hua, X.-S. 2008. Free-shaped video collage. Multi-Media Modeling (MMM), 175--185.

Index Terms

• Computing methodologies

  • Computer graphics