Article

Face transfer with multilinear models

Authors:

Hanspeter Pfister,

Jovan PopovićAuthors Info & Claims

SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

Pages 426 - 433

https://doi.org/10.1145/1186822.1073209

Published: 01 July 2005 Publication History

Abstract

Face Transfer is a method for mapping videorecorded performances of one individual to facial animations of another. It extracts visemes (speech-related mouth articulations), expressions, and three-dimensional (3D) pose from monocular video or film footage. These parameters are then used to generate and drive a detailed 3D textured face mesh for a target identity, which can be seamlessly rendered back into target footage. The underlying face model automatically adjusts for how the target performs facial expressions and visemes. The performance data can be easily edited to change the visemes, expressions, pose, or even the identity of the target---the attributes are separably controllable. This supports a wide variety of video rewrite and puppetry applications.Face Transfer is based on a multilinear model of 3D face meshes that separably parameterizes the space of geometric variations due to different attributes (e.g., identity, expression, and viseme). Separability means that each of these attributes can be independently varied. A multilinear model can be estimated from a Cartesian product of examples (identities × expressions × visemes) with techniques from statistical analysis, but only after careful preprocessing of the geometric data set to secure one-to-one correspondence, to minimize cross-coupling artifacts, and to fill in any missing examples. Face Transfer offers new solutions to these problems and links the estimated model with a face-tracking algorithm to extract pose, expression, and viseme parameters.

Supplementary Material

MP4 File (pps004.mp4)

Download
30.96 MB

References

[1]

Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Transactions on Graphics 22, 3 (July), 587--594.

Digital Library

[2]

Bascle, B., and Blake, A. 1998. Separability of pose and expression in facial tracking and animation. In International Conference on Computer Vision (ICCV), 323--328.

Digital Library

[3]

Birchfield, S., 1996. KLT: An implementation of the kanade-lucas-tomasi feature tracker. http://www.ces.clemson.edu/~stb/.

[4]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of SIGGRAPH 99, Computer Graphics Proceedings, Annual Conference Series, 187--194.

Digital Library

[5]

Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Computer Graphics Forum 22, 3 (Sept.), 641--650.

[6]

Brand, M. E., and Bhotika, R. 2001. Flexible flow for 3D nonrigid tracking and shape recovery. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, 315--322.

[7]

Brand, M. E. 2002. Incremental singular value decomposition of uncertain data with missing values. In European Conference on Computer Vision (ECCV), vol. 2350, 707--720.

Digital Library

[8]

Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of SIGGRAPH 97, Computer Graphics Proceedings, Annual Conference Series, 353--360.

Digital Library

[9]

Bregler, C., Hertzmann, A., and Biermann, H. 2000. Recovering non-rigid 3D shape from image streams. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, 690--696.

[10]

Cao, Y., Faloutsos, P., and Pighin, F. 2003. Unsupervised learning for speech motion editing. In Eurographics/SIGGRAPH Symposium on Computer Animation (SCA), 225--231.

Digital Library

[11]

Chai, J.-X., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Eurographics/SIGGRAPH Symposium on Computer Animation (SCA), 193--206.

Digital Library

[12]

Chuang, E. S., Deshpande, H., and Bregler, C. 2002. Facial expression space learning. In Pacific Conference on Computer Graphics and Applications (PG), 68--76.

Digital Library

[13]

De Lathauwer, L. 1997. Signal Processing based on Multilinear Algebra. PhD thesis, Katholieke Universiteit Leuven, Belgium.

[14]

DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 231--238.

Digital Library

[15]

DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. International Journal of Computer Vision 38, 2, 99--127.

Digital Library

[16]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: Using input from video. In Computer Animation '96, 68--79.

Digital Library

[17]

Ezzat, T., and Poggio, T. 2000. Visual speech synthesis by morphing visemes. International Journal of Computer Vision 38, 1, 45--57.

Digital Library

[18]

Freeman, W. T., and Tenenbaum, J. B. 1997. Learning bilinear models for two factor problems in vision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 554--560.

Digital Library

[19]

Georghiades, A., Belhumeur, P., and Kriegman, D. 2001. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 23, 6, 643--660.

Digital Library

[20]

Gotsman, C., Gu, X., and Sheffer, A. 2003. Fundamentals of spherical parameterization for 3D meshes. ACM Transactions on Graphics 22, 3 (July), 358--363.

Digital Library

[21]

Jones, T. R., Durand, F., and Desbrun, M. 2003. Non-iterative, feature-preserving mesh smoothing. ACM Transactions on Graphics 22, 3 (July), 943--949.

Digital Library

[22]

Koch, R. M., Gross, M. H., Carls, F. R., Von Büren, D. F., Fankhauser, G., and Parish, Y. 1996. Simulating facial surgery using finite element methods. In Proceedings of SIGGRAPH 96, Computer Graphics Proceedings, Annual Conference Series, 421--428.

Digital Library

[23]

Kraevoy, V., and Sheffer, A. 2004. Cross-parameterization and compatible remeshing of 3D models. ACM Transactions on Graphics 23, 3 (Aug.), 861--869.

Digital Library

[24]

Kroonenberg, P. M., and De Leeuw, J. 1980. Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika 45, 69--97.

[25]

Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. In Proceedings of SIGGRAPH 95, Computer Graphics Proceedings, Annual Conference Series, 55--62.

Digital Library

[26]

Li, H., Roivainen, P., and Forchheimer, R. 1993. 3-D motion estimation in model-based facial image coding. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 15, 6, 545-555.

Digital Library

[27]

Noh, J.-Y., and Neumann, U. 2001. Expression cloning. In Proceedings of SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 277--288.

Digital Library

[28]

Parke, F. I. 1974. A parametric model for human faces. PhD thesis, University of Utah, Salt Lake City, Utah.

Digital Library

[29]

Parke, F. I. 1982. Parameterized models for facial animation. IEEE Computer Graphics & Applications 2 (Nov.), 61--68.

Digital Library

[30]

Pentland, A., and Sclaroff, S. 1991. Closed-form solutions for physically based shape modeling and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 13, 7, 715--729.

Digital Library

[31]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics 22, 3 (July), 313--318.

Digital Library

[32]

Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 75--84.

Digital Library

[33]

Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In International Conference on Computer Vision (ICCV), 143--150.

[34]

Praun, E., and Hoppe, H. 2003. Spherical parameterization and remeshing. ACM Transactions on Graphics 22, 3 (July), 340--349.

Digital Library

[35]

Robertson, B. 2004. Locomotion. Computer Graphics World (Dec.).

[36]

Roweis, S. 1997. EM algorithms for PCA and SPCA. In Advances in neural information processing systems 10 (NIPS), 626--632.

Digital Library

[37]

Sirovich, L., and Kirby, M. 1987. Low dimensional procedure for the characterization of human faces. Journal of the Optical Society of America A 4, 519--524.

[38]

Sumner, R. W., and Popović, J. 2004. Deformation transfer for triangle meshes. ACM Transactions on Graphics 23, 3 (Aug.), 399--405.

Digital Library

[39]

Tipping, M. E., and Bishop, C. M. 1999. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B 61, 3, 611--622.

[40]

Torresani, L., Yang, D., Alexander, E., and Bregler, C. 2001. Tracking and modeling non-rigid objects with rank constraints. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, 493--450.

[41]

Tucker, L. R. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31, 3 (Sept.), 279--311.

[42]

Vasilescu, M. A. O., and Terzopoulos, D. 2002. Multilinear analysis of image ensembles: Tensorfaces. In European Conference on Computer Vision (ECCV), 447--460.

Digital Library

[43]

Vasilescu, M. A. O., and Terzopoulos, D. 2004. Tensortextures: multilinear image-based rendering. ACM Transactions on Graphics 23, 3 (Aug.), 336-342.

Digital Library

[44]

Viola, P., and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, 511--518.

[45]

Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. Computer Graphics Forum 23, 3 (Sept.), 677--686.

[46]

Waters, K. 1987. A muscle model for animating three-dimensional facial expression. In Computer Graphics (Proceedings of SIGGRAPH 87), vol. 21, 17--24.

Digital Library

[47]

Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proceedings of SIGGRAPH 90), vol. 24, 235--242.

Digital Library

[48]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Space-time faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548-558.

Digital Library

Cited By

Bednarik JWood EChoutas VBolkart TWang DWu CBeeler T(2024)Learning to Stabilize FacesComputer Graphics Forum10.1111/cgf.1503843:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15038
Tretschk EKairanda NB R MDabral RKortylewski AEgger BHabermann MFua PTheobalt CGolyanik V(2023)State of the Art in Dense Monocular Non‐Rigid 3D ReconstructionComputer Graphics Forum10.1111/cgf.1477442:2(485-520)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14774
Wu RYu YZhan FZhang JLiao SLu S(2023)POCE: Pose-Controllable Expression EditingIEEE Transactions on Image Processing10.1109/TIP.2023.332935832(6210-6222)Online publication date: 2023
https://doi.org/10.1109/TIP.2023.3329358
Show More Cited By

Index Terms

Face transfer with multilinear models
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
  2. Computer graphics
    1. Animation

Recommendations

Face transfer with multilinear models

Face Transfer is a method for mapping videorecorded performances of one individual to facial animations of another. It extracts visemes (speech-related mouth articulations), expressions, and three-dimensional (3D) pose from monocular video or film ...
Face transfer with multilinear models
SIGGRAPH '06: ACM SIGGRAPH 2006 Courses

Face Transfer is a method for mapping videorecorded performances of one individual to facial animations of another. It extracts visemes (speech-related mouth articulations), expressions, and three-dimensional (3D) pose from monocular video or film ...
A Live Face Swapper
MM '16: Proceedings of the 24th ACM international conference on Multimedia

In this technical demonstration, we propose a face swapping framework, which is able to interactively change the appearance of a face in the wild to a different person/creature's face in real time on a mobile device. To realize this objective, we develop ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

July 2005

826 pages

ISBN:9781450378253

DOI:10.1145/1186822

Editor:
Markus Gross
Eidgenössische Technische Hochschule Zürich

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGGRAPH05

Sponsor:

SIGGRAPH

SIGGRAPH05: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 31 - August 4, 2005

California, Los Angeles

Acceptance Rates

SIGGRAPH '05 Paper Acceptance Rate 98 of 461 submissions, 21%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

90
Total Citations
View Citations
3,493
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bednarik JWood EChoutas VBolkart TWang DWu CBeeler T(2024)Learning to Stabilize FacesComputer Graphics Forum10.1111/cgf.1503843:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15038
Tretschk EKairanda NB R MDabral RKortylewski AEgger BHabermann MFua PTheobalt CGolyanik V(2023)State of the Art in Dense Monocular Non‐Rigid 3D ReconstructionComputer Graphics Forum10.1111/cgf.1477442:2(485-520)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14774
Wu RYu YZhan FZhang JLiao SLu S(2023)POCE: Pose-Controllable Expression EditingIEEE Transactions on Image Processing10.1109/TIP.2023.332935832(6210-6222)Online publication date: 2023
https://doi.org/10.1109/TIP.2023.3329358
Mallya AWang TLiu MKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Implicit warping for animation with image setsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3601900(22438-22450)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3601900
Cao CSimon TKim JSchwartz GZollhoefer MSaito SLombardi SWei SBelko DYu SSheikh YSaragih J(2022)Authentic volumetric avatars from a phone scanACM Transactions on Graphics10.1145/3528223.353014341:4(1-19)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530143
Mao JZhang YLiu CTao ZYi JWang Y(2022)Dense 3D Face Reconstruction from a Single RGB Image2022 IEEE 25th International Conference on Computational Science and Engineering (CSE)10.1109/CSE57773.2022.00013(24-31)Online publication date: Dec-2022
https://doi.org/10.1109/CSE57773.2022.00013
Hu CXie X(2022)One-Shot Decoupled Face Reenactment with Vision TransformerPattern Recognition and Artificial Intelligence10.1007/978-3-031-09282-4_21(246-257)Online publication date: 29-May-2022
https://doi.org/10.1007/978-3-031-09282-4_21
Pic MMahfoudi GTrabelsi ADugelay J(2022)Face Manipulation Detection in Remote Operational SystemsHandbook of Digital Face Manipulation and Detection10.1007/978-3-030-87664-7_19(413-436)Online publication date: 31-Jan-2022
https://doi.org/10.1007/978-3-030-87664-7_19
Zhao YYang LPei EOveneke MAlioscha‐Perez MLi LJiang DSahli H(2021)Action Unit Driven Facial Expression Synthesis from a Single Image with Patch Attentive GANComputer Graphics Forum10.1111/cgf.1420240:6(47-61)Online publication date: 17-Mar-2021
https://doi.org/10.1111/cgf.14202
Grasshof SAckermann HBrandt SOstermann J(2021)Multilinear Modelling of Faces and ExpressionsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2020.298649643:10(3540-3554)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TPAMI.2020.2986496
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten