skip to main content
10.5555/1272690.1272698acmconferencesArticle/Chapter ViewAbstractPublication PagesscaConference Proceedingsconference-collections
Article

Dynamic, expressive speech animation from a single mesh

Published: 03 August 2007 Publication History

Abstract

In this work we present a method for human face animation which allows us to generate animations for a novel person given just a single mesh of their face. These animations can be of arbitrary text and may include emotional expressions. We build a multilinear model from data which encapsulates the variation in dynamic face motions over changes in identity, expression, and over different texts. We then describe a synthesis method consisting of a phoneme planning and a blending stage which uses this model as a base and attempts to preserve both face shape and dynamics given a novel text and an emotion at each point in time.

References

[1]
{BBVP03} Blanz V., Basso C., Vetter T., Poggio T.: Reanimating faces in images and video. In EUROGRAPHICS 2003 (EUROGRAPHICS-03): the European Association for Computer Graphics, 24th Annual Conference (Granada, Spain, 2003), Brunet P., Fellner D. W., (Eds.), vol. 22 of Computer Graphics Forum, The Eurographics Association, Blackwell, pp. 641--650.
[2]
{BCS97} Bregler C., Covell M., Slaney M.: Video rewrite: driving visual speech with audio. In SIGGRAPH '97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1997), ACM Press/Addison-Wesley Publishing Co., pp. 353--360.
[3]
{Bra99} Brand M.: Voice puppetry. In SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1999), ACM Press/Addison-Wesley Publishing Co., pp. 21--28.
[4]
{BV99} Blanz V., Vetter T.: A morphable model for the synthesis of 3d faces. In SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1999), ACM Press/Addison-Wesley Publishing Co., pp. 187--194.
[5]
{Cam} Cambridge University Engineering Department: Hidden markov model toolkit. http://htk.eng.cam.ac.uk/.
[6]
{CB05} Chuang E., Bregler C.: Mood swings: expressive speech animation. ACM Trans. Graph. 24, 2 (2005), 331--347.
[7]
{CE05} Chang Y.-J., Ezzat T.: Transferable videorealistic speech animation. In SCA '05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation (New York, NY, USA, 2005), ACM Press, pp. 143--151.
[8]
{CTFP05} Cao Y., Tien W. C., Faloutsos P., Pighin F.: Expressive speech-driven facial animation. ACM Trans. Graph. 24, 4 (2005), 1283--1302.
[9]
{DLN05} Deng Z., Lewis J., Neumann U.: Synthesizing speech animation by learning compact speech co-articulation models. In Computer Graphics International (2005), pp. 19--25.
[10]
{EF78} Ekman P., Friesen W.: The facial action coding system: A technique for the measurement of facial movement, 1978.
[11]
{EGP02} Ezzat T., Geiger G., Poggio T.: Trainable videorealistic speech animation. In SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactive techniques (New York, NY, USA, 2002), ACM Press, pp. 388--398.
[12]
{JP98} Jones M. J., Poggio T.: Multidimensional morphable models: A framework for representing and matching object classes. Int. J. Comput. Vision 29, 2 (1998), 107--131.
[13]
{KB06} Kim T.-Y., Bulut M.: Expressive facial animation synthesis by learning speech coarticulation and expression spaces. IEEE Transactions on Visualization and Computer Graphics 12, 6 (2006), 1523--1534. Member-Zhigang Deng and Member-Ulrich Neumann and Member-J. P. Lewis and Senior Member-Shrikanth Narayanan.
[14]
{KP05} King M.-S. A., Parent M.-R. E.: Creating speech-synchronized animation. IEEE Transactions on Visualization and Computer Graphics 11, 3 (2005), 341--352.
[15]
{Lat97} Lathauwer L. D.: Signal Processing based on Multilinear Algebra. PhD thesis, Faculteit der Toegepaste Wetenschappen. Katholieke Universiteit Leuven, 1997.
[16]
{LTW95} Lee Y., Terzopoulos D., Walters K.: Realistic modeling for facial animation. In SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1995), ACM Press, pp. 55--62.
[17]
{LZPW03} Levin A., Zomet A., Peleg S., Weiss Y.: Seamless image stitching in the gradient domain, 2003.
[18]
{MKPG05} Mueller P., Kalberer G. A., Proesmans M., Gool L. V.: Realistic speech animation based on observed 3d face dynamics. IEE Proc. Vision, Image and Signal Processing 152 (August 2005), 491--500.
[19]
{MZD05} Matusik W., Zwicker M., Durand F.: Texture design using a simplicial complex of morphable textures. In SIGGRAPH '05: ACM SIGGRAPH 2005 Papers (New York, NY, USA, 2005), ACM Press, pp. 787--794.
[20]
{Par82} Parke F. I.: Parameterized models for facial animation. j-IEEE-CGA 2, 9 (nov 1982), 61--64, 66--68.
[21]
{PL06} Pighin F., Lewis J. P.: Facial motion retargeting. In SIGGRAPH '06: ACM SIGGRAPH 2006 Courses (New York, NY, USA, 2006), ACM Press, p. 2.
[22]
{SSRMF06} Sifakis E., Selle A., Robinson-Mosher A., Fedkiw R.: Simulating speech with a physics-based facial muscle model. In SCA '06: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation (Aire-la-Ville, Switzerland, Switzerland, 2006), Eurographics Association, pp. 261--270.
[23]
{VBPP05} Vlasic D., Brand M., Pfister H., Popović J. P.: Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (2005), 426--433.
[24]
{VT02} Vasilescu M. A. O., Terzopoulos D.: Multilinear analysis of image ensembles: Tensorfaces. In ECCV '02: Proceedings of the 7th European Conference on Computer Vision-Part I (London, UK, 2002), Springer-Verlag, pp. 447--460.
[25]
{VT04} Vasilescu M. A. O., Terzopoulos D.: Ten-sortextures: multilinear image-based rendering. In SIGGRAPH '04: ACM SIGGRAPH 2004 Papers (New York, NY, USA, 2004), ACM Press, pp. 336--342.
[26]
{Wat87} Waters K.: A muscle model for animation three-dimensional facial expression. In SIGGRAPH '87: Proceedings of the 14th annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1987), ACM Press, pp. 17--24.
[27]
{Wil90} Williams L.: Performance-driven facial animation. In SIGGRAPH '90: Proceedings of the 17th annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1990), ACM Press, pp. 235--242.
[28]
{WWS*05} Wang H., Wu Q., Shi L., Yu Y., Ahuja N.: Out-of-core tensor approximation of multidimensional matrices of visual data. ACM Trans. Graph. 24, 3 (2005), 527--535.
[29]
{xCXH03} xiang Chai J., Xiao J., Hodgins J.: Vision-based control of 3d facial animation. In SCA '03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation (Aire-la-Ville, Switzerland, Switzerland, 2003), Eurographics Association, pp. 193--206.
[30]
{ZSCS04} Zhang L., Snavely N., Curless B., Seitz S. M.: Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3 (2004), 548--558.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SCA '07: Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation
August 2007
287 pages
ISBN:9781595936240

Sponsors

Publisher

Eurographics Association

Goslar, Germany

Publication History

Published: 03 August 2007

Check for updates

Qualifiers

  • Article

Conference

SCA07
Sponsor:

Acceptance Rates

SCA '07 Paper Acceptance Rate 28 of 81 submissions, 35%;
Overall Acceptance Rate 183 of 487 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2016)High-fidelity facial and speech animation for VR HMDsACM Transactions on Graphics10.1145/2980179.298025235:6(1-14)Online publication date: 5-Dec-2016
  • (2015)Video-audio driven real-time facial animationACM Transactions on Graphics10.1145/2816795.281812234:6(1-10)Online publication date: 2-Nov-2015
  • (2012)Dynamic units of visual speechProceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation10.5555/2422356.2422395(275-284)Online publication date: 29-Jul-2012
  • (2012)Dynamic units of visual speechProceedings of the 11th ACM SIGGRAPH / Eurographics conference on Computer Animation10.5555/2421731.2421770(275-284)Online publication date: 29-Jul-2012
  • (2012)Lip-synced character speech animation with dominated animeme modelsSIGGRAPH Asia 2012 Technical Briefs10.1145/2407746.2407772(1-4)Online publication date: 28-Nov-2012
  • (2008)Perceptually guided expressive facial animationProceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation10.5555/1632592.1632603(67-76)Online publication date: 7-Jul-2008

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media