research-article

Unwrap mosaics: a new representation for video editing

Authors:

Pushmeet Kohli,

Carsten Rother,

Andrew FitzgibbonAuthors Info & Claims

SIGGRAPH '08: ACM SIGGRAPH 2008 papers

Article No.: 17, Pages 1 - 11

https://doi.org/10.1145/1399504.1360616

Published: 01 August 2008 Publication History

Abstract

We introduce a new representation for video which facilitates a number of common editing tasks. The representation has some of the power of a full reconstruction of 3D surface models from video, but is designed to be easy to recover from a priori unseen and uncalibrated footage. By modelling the image-formation process as a 2D-to-2D transformation from an object's texture map to the image, modulated by an object-space occlusion mask, we can recover a representation which we term the "unwrap mosaic". Many editing operations can be performed on the unwrap mosaic, and then re-composited into the original sequence, for example resizing objects, repainting textures, copying/cutting/pasting objects, and attaching effects layers to deforming objects.

Supplementary Material

MOV File (a17-rav_acha.mov)

Download
43.64 MB

References

[1]

2d3 Ltd., 2008. Boujou 4: The virtual interchangeable with the real. http://www.2d3.com.

[2]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S. M., Colburn, A., Curless, B., Salesin, D., and Cohen, M. F. 2004. Interactive digital photomontage. ACM Trans. Graph. (Proc. of SIGGRAPH) 23, 3, 294--302.

Digital Library

[3]

Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M., and Szeliski, R. 2007. A database and evaluation methodology for optical flow. In Proc. ICCV.

[4]

Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering.

Digital Library

[5]

Black, M. J., and Anandan, P. 1993. A framework for the robust estimation of optical flow. In Proc. ICCV, 231--236.

[6]

Blake, A., and Zisserman, A. 1987. Visual Reconstruction. MIT Press.

Digital Library

[7]

Boykov, Y., and Jolly, M.-P. 2001. Interactive graph cuts for optimal boundary and region segmentation of objects in n-D images. In Proc. ICCV, 105--112.

[8]

Brand, M. 2001. Morphable 3D models from video. In Proc. CVPR, vol. 2, 456--463.

[9]

Bregler, C., Hertzmann, A., and Biermann, H. 2000. Recovering non-rigid 3D shape from image streams. In Proc. CVPR, 690--696.

[10]

Brown, M., and Lowe, D. G. 2007. Automatic panoramic image stitching using invariant features. Intl. J. Comput. Vision 74, 1, 59--73.

Digital Library

[11]

Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV, 25--36.

[12]

Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. Intl. J. of Computer Vision 61, 3, 211--231.

Digital Library

[13]

Costeira, J. P., and Kanade, T. 1998. A multibody factorization method for independently moving objects. Intl. J. of Computer Vision 29, 3, 159--179.

Digital Library

[14]

Cox, M., and Cox, M. A. A. 2001. Multidimensional Scaling. Chapman and Hall.

[15]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs. In Proc. ACM Siggraph.

Digital Library

[16]

Fleet, D., Jepson, A., and Black, M. 2002. A layered motion representation with occlusion and compact spatial support. In Proc. ECCV, 692--706.

Digital Library

[17]

Frey, B. J., Jojic, N., and Kannan, A. 2003. Learning appearance and transparency manifolds of occluded objects in layers. In Proc. CVPR.

Digital Library

[18]

Gay-Bellile, V., Bartoli, A., and Sayd, P. 2007. Direct estimation of non-rigid registrations with image-based self-occlusion reasoning. In Proc. ICCV.

[19]

Gu, X., Gortler, S. J., and Hoppe, H. 2002. Geometry images. ACM Trans. Graph. (Proc. of SIGGRAPH), 355--361.

Digital Library

[20]

Irani, M., Anandan, P., and Hsu, S. 1995. Mosaic based representations of video sequences and their applications. In Proc. ICCV.

Digital Library

[21]

Lempitsky, V., and Ivanov, D. 2007. Seamless mosaicing of image-based texture maps. In Proc. CVPR, 1--6.

[22]

Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Trans. Graph. (Proc. of SIGGRAPH) 24, 3, 595--600.

Digital Library

[23]

Rav-Acha, A., Kohli, P., Rother, C., and Fitzgibbon, A. 2008. Unwrap mosaics. Tech. rep., Microsoft Research. http://research.microsoft.com/unwrap.

[24]

Sand, P., and Teller, S. J. 2006. Particle video: Longrange motion estimation using point trajectories. In Proc. CVPR, 2195--2202.

Digital Library

[25]

Seetzen, H., Heidrich, W., Stuerzlinger, W., Ward, G., Whitehead, L., Trentacoste, M., Ghosh, A., and Vorozcovs, A. 2004. High dynamic range display systems. ACM Trans. Graph. (Proc. of SIGGRAPH) 23, 3, 760--768.

Digital Library

[26]

Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. CVPR, vol. 1, 519--526.

Digital Library

[27]

Seymour, M. 2006. Art of optical flow. fxguide.com: Feature Stories (Dec.).

[28]

Shade, J. W., Gortler, S. J., He, L.-W., and Szeliski, R. 1998. Layered depth images. In Proc. ACM Siggraph, 231--242.

Digital Library

[29]

Shi, J., and Malik, J. 1997. Normalized cuts and image segmentation. In Proc. CVPR, 731--743.

Digital Library

[30]

Thormählen, T., and Broszio, H., 2008. Voodoo Camera Tracker: A tool for the integration of virtual and real scenes. http://www.digilab.uni-hannover.de/docs/manual.html.

[31]

Toklu, C., Erdem, A. T., and Tekalp, A. M. 2000. Two-dimensional mesh-based mosaic representation for manipulation of video objects with occlusion. IEEE Trans. Image Proc. 9, 9, 1617--1630.

Digital Library

[32]

Torresani, L., Hertzmann, A., and Bregler, C. 2008. Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors. IEEE Trans. PAMI, (to appear).

Digital Library

[33]

Turk, G., and Levoy, M. 1994. Zippered polygon meshes from range images. In Proc. ACM Siggraph, 311--318.

Digital Library

[34]

van den Hengel, A., Dick, A., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. VideoTrace: Rapid interactive scene modelling from video. ACM Trans. Graph. (Proc. of SIGGRAPH).

Digital Library

[35]

Wang, J. Y. A., and Adelson, E. H. 1994. Representing moving images with layers. IEEE Trans. Image Proc. 3, 5, 625--638.

Digital Library

[36]

Woodford, O. J., Reid, I. D., and Fitzgibbon, A. W. 2007. Efficient new-view synthesis using pairwise dictionary priors. In Proc. CVPR.

[37]

Zhou, K., Wang, X., Tong, Y., Desbrun, M., Guo, B., and Shum, H.-Y. 2005. Texture-Montage: Seamless texturing of surfaces from multiple images. ACM Trans. Graph. (Proc. of SIGGRAPH), 1148--1155.

Digital Library

[38]

Zigelman, G., Kimmel, R., and Kiryati, N. 2002. Texture mapping using surface flattening via multi-dimensional scaling. IEEE Trans. on Visualization and Computer Graphics 8, 2, 198--207.

Digital Library

Cited By

Chugunov IJoshi AMurthy KBleibel FHeide F(2024)Neural Light Spheres for Implicit Image Stitching and View SynthesisSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687660(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687660
Lee JKim GCho S(2024)RNA: Video Editing with ROI-Based Neural AtlasComputer Vision – ACCV 202410.1007/978-981-96-0960-4_17(278-293)Online publication date: 8-Dec-2024
https://doi.org/10.1007/978-981-96-0960-4_17
Suzuki TKikuchi KYamaguchi K(2024)Fast Sprite Decomposition from Animated GraphicsComputer Vision – ECCV 202410.1007/978-3-031-72855-6_12(200-215)Online publication date: 9-Nov-2024
https://doi.org/10.1007/978-3-031-72855-6_12
Show More Cited By

Recommendations

Layered neural atlases for consistent video editing

We present a method that decomposes, and "unwraps", an input video into a set of layered 2D atlases, each providing a unified representation of the appearance of an object (or background) over the video. For each pixel in the video, our method estimates ...
Interactive intrinsic video editing

Separating a photograph into its reflectance and illumination intrinsic images is a fundamentally ambiguous problem, and state-of-the-art algorithms combine sophisticated reflectance and illumination priors with user annotations to create plausible ...
Virtual video editing in interactive multimedia applications

Drawing examples from four interrelated sets of multimedia tools and applications under development at MIT, the authors examine the role of digitized video in the areas of entertainment, learning, research, and communication.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '08: ACM SIGGRAPH 2008 papers

August 2008

887 pages

ISBN:9781450301121

DOI:10.1145/1399504

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGGRAPH '08

Sponsor:

SIGGRAPH

SIGGRAPH '08: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 11 - 15, 2008

California, Los Angeles

Acceptance Rates

SIGGRAPH '08 Paper Acceptance Rate 90 of 518 submissions, 17%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
1,767
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)3

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chugunov IJoshi AMurthy KBleibel FHeide F(2024)Neural Light Spheres for Implicit Image Stitching and View SynthesisSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687660(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687660
Lee JKim GCho S(2024)RNA: Video Editing with ROI-Based Neural AtlasComputer Vision – ACCV 202410.1007/978-981-96-0960-4_17(278-293)Online publication date: 8-Dec-2024
https://doi.org/10.1007/978-981-96-0960-4_17
Suzuki TKikuchi KYamaguchi K(2024)Fast Sprite Decomposition from Animated GraphicsComputer Vision – ECCV 202410.1007/978-3-031-72855-6_12(200-215)Online publication date: 9-Nov-2024
https://doi.org/10.1007/978-3-031-72855-6_12
Yu EBlackburn-Matzen KNguyen CWang OHabib Kazi RBousseau A(2023)VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware CanvasesACM Transactions on Graphics10.1145/359241342:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592413
Esser PChiu JAtighehchian PGranskog JGermanidis A(2023)Structure and Content-Guided Video Synthesis with Diffusion Models2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00675(7312-7322)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00675
Lei CRen XZhang ZChen Q(2023)Blind Video Deflickering by Neural Filtering with a Flawed Atlas2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01006(10439-10448)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01006
Zhang LWong TLiu Y(2022)Sprite-from-SpriteACM Transactions on Graphics10.1145/3550454.355543941:6(1-12)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555439
Ye VLi ZTucker RKanazawa ASnavely N(2022)Deformable Sprites for Unsupervised Video Decomposition2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00268(2647-2656)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.00268
Bar-Tal OOfri-Amar DFridman RKasten YDekel T(2022)Text2LIVE: Text-Driven Layered Image and Video EditingComputer Vision – ECCV 202210.1007/978-3-031-19784-0_41(707-723)Online publication date: 31-Oct-2022
https://doi.org/10.1007/978-3-031-19784-0_41
Xu YAlBahar BHuang J(2022)Temporally Consistent Semantic Video EditingComputer Vision – ECCV 202210.1007/978-3-031-19784-0_21(357-374)Online publication date: 31-Oct-2022
https://doi.org/10.1007/978-3-031-19784-0_21
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten