research-article

Perceptually guided expressive facial animation

Authors:

Xiaohan MaAuthors Info & Claims

SCA '08: Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation

Pages 67 - 76

Published: 07 July 2008 Publication History

Abstract

Most of current facial animation approaches largely focus on the accuracy or efficiency of their algorithms, or how to optimally utilize pre-collected facial motion data. However, human perception, the ultimate measuring stick of the visual fidelity of synthetic facial animations, was not effectively exploited in these approaches. In this paper, we present a novel perceptually guided computational framework for expressive facial animation, by bridging objective facial motion patterns with subjective perceptual outcomes. First, we construct a facial perceptual metric (FacePEM) using a hybrid of region-based facial motion analysis and statistical learning techniques. The constructed FacePEM model can automatically measure the emotional expressiveness of a facial motion sequence. We showed how the constructed FacePEM model can be effectively incorporated into various facial animation algorithms. For the sake of clear demonstrations, we choose data-driven expressive speech animation generation and expressive facial motion editing as two concrete application examples. Through a comparative user study, we showed that comparing with the traditional facial animation algorithms, the introduced perceptually guided expressive facial animation algorithms can significantly increase the emotional expressiveness and perceptual believability of synthesized facial animations.

References

[1]

{Ado02} Adolphs R.: Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews 1, 1 (2002), 21--62.

[2]

{BCS97} Bregler C., Covell M., Slaney M.: Video rewrite: Driving visual speech with audio. Proc. of ACM SIGGRAPH'97 31 (1997), 353--360.

Digital Library

[3]

{Bra99} Brand M.: Voice puppetry. In Proc. of ACM SIGGRAPH'99 (1999), pp. 21--28.

Digital Library

[4]

{BV99} Blanz V., Vetter T.: A morphable model for the synthesis of 3D faces. In Proc. of ACM SIGGRAPH'99 (1999), pp. 187--194.

Digital Library

[5]

{CBK*03} Cunningham D. W., Breidt M., Kleiner M., Wallraven C., Bülthoff H. H.: How believable are real faces? towards a perceptual basis for conversational animation. In Proc. of IEEE CASA'03 (2003).

Digital Library

[6]

{CDB02} Chuang E., Deshpande H., Bregler C.: Facial expression space learning. In Proc. of PG'02 (2002).

Digital Library

[7]

{CFKP04} Cao Y., Faloutsos P., Kohler E., Pighin F.: Real-time speech motion synthesis from recorded motions. In Proc. of SCA'04 (2004), pp. 345--353.

Digital Library

[8]

{CFP03} Cao Y., Faloutsos P., Pighin F.: Unsupervised learning for speech motion editing. In Proc. of SCA'03 (2003).

Digital Library

[9]

{CH07} Chai J., Hodgins J. K.: Constraint-based motion optimization using a statistical dynamic model. ACM Trans. Graph. 26, 3 (2007).

Digital Library

[10]

{CKBW04} Cunningham D. W., Kleiner M., Bülthoff H. H., Wallraven C.: The components of conversational facial expressions. In Proc. of APGV '04 (2004), pp. 143--150.

Digital Library

[11]

{CL01} Chang C.-C., Lin C.-J.: LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

[12]

{CPM*05} Cosker D., Paddock S., Marshall D., Rosin P. L., Rushton S.: Toward perceptually realistic talking heads: Models, methods, and mcgurk. ACM Trans. on Appl. Percep. 2, 3 (2005), 270--285.

Digital Library

[13]

{DBLN06} Deng Z., Bailenson J., Lewis J. P., Neumann U.: Perceiving visual emotions with speech. In Proc. of the 6th Int'l Conf. on Intelligent Virtual Agents (IVA) 2006 (August 2006), pp. 107--120.

Digital Library

[14]

{DLN05} Deng Z., Lewis J. P., Neumann U.: Automated eye motion synthesis using texture synthesis. IEEE CG&A (March/April 2005), 24--30.

Digital Library

[15]

{DN06} Deng Z., Neumann U.: eFASE: Expressive facial animation synthesis and editing with phoneme-isomap controls. In Proc. of SCA'06 (2006), pp. 251--259.

Digital Library

[16]

{DN07} Deng Z., Noh J. Y.: Computer facial animation: A survey. In Data-Driven 3D Facial Animation, Z. Deng and U. Neumann (Eds.) (2007), Springer-Verlag Press, pp. 1--28.

[17]

{DN08} Deng Z., Neumann U.: Expressive speech animation synthesis with phoneme-level control. Computer Graphics Forum 27, 6 (2008), in press.

[18]

{EF78} Ekman P., Friesen W. V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, 1978.

[19]

{EGP02} Ezzat T., Geiger G., Poggio T.: Trainable video-realistic speech animation. In ACM Trans. on Graph. (2002), vol. 21, pp. 388--398.

Digital Library

[20]

{FS01} Frank M., Stennett J.: The forced-choice paradigm and the perception of facial expressions of emotion. Journal of Personality and Social Psychology 80 (2001), 75--85.

[21]

{GEP03} Geiger G., Ezzat T., Poggio T.: Perceptual evaluation of video-realistic speech. MIT-AI-Memo 2003--003 (Feb. 2003).

[22]

{HHN88} Horn B. K. P., Hilden H. M., Negahdaripour S.: Closed-form solution of absolute orientation using orthonormal matrices. J. Opt. Soc. Am. A 5, 7 (1988), 1127.

[23]

{HOT98} Hodgins J. K., O'Brien J. F., Tumblin J.: Perception of human motion with different geometric models. IEEE Trans. Vis. Comp. Graph. 4, 4 (1998), 307--316.

Digital Library

[24]

{JL08} Ju E., Lee J.: Expressive facial gestures from motion capture data. Computer Graphics Forum 27, 2 (2008).

[25]

{JTDP03} Joshi P., Tien W., Desbrun M., Pighin F.: Learning controls for blend shape based realistic facial animation. In Proc. of SCA'03 (2003), pp. 35--42.

Digital Library

[26]

{KGT00} Kshirsagar S., Garchery S., Thalmann N. M.: Feature point based mesh deformation applied to mpeg-4 facial animation. In Proc. Deform'2000 (November 2000), pp. 23--34.

Digital Library

[27]

{KT03} Kshirsagar S., Thalmann N. M.: Visyllable based speech animation. Computer Graphics Forum 22, 3 (2003).

[28]

{LD07} Li Q., Deng Z.: Facial motion capture editing by automated orthogonal blendshape construction and weight propagation. IEEE Computer Graphics and Applications (accepted for publication 2007).

[29]

{LMDN05} Lewis J. P., Mooser J., Deng Z., Neumann U.: Reducing blendshape interference by selected motion attenuation. In Proc. of I3DG'05 (2005), pp. 25--29.

Digital Library

[30]

{LTW95} Lee Y., Terzopoulos D., Waters K.: Realistic modeling for facial animation. In Proc. of ACM SIGGRAPH'95 (1995), pp. 55--62.

Digital Library

[31]

{MDCO06} McDonnell R., Dobbyn S., Collins S., O'Sullivan C.: Perceptual evaluation of lod clothing for virtual humans. In Proc. of SCA'06 (2006), pp. 117--126.

Digital Library

[32]

{MNO07} McDonnell R., Newell F., O'Sullivan C.: Smooth movers: perceptually guided human motion simulation. In Proc. of SCA'07 (2007), pp. 259--269.

Digital Library

[33]

{NN01} Noh J.-Y., Neumann U.: Expression cloning. In Proc. of ACM SIGGRAPH'01 (2001), pp. 277--288.

Digital Library

[34]

{OD01} O'Sullivan C., Dingliana J.: Collisions and perception. ACM Trans. on Graph. 20, 3 (2001), 151--168.

Digital Library

[35]

{ODGK03} O'Sullivan C., Dingliana J., Giang T., Kaiser M. K.: Evaluating the visual fidelity of physically based animations. In Proc. of ACM SIGGRAPH'03 (2003), pp. 527--536.

Digital Library

[36]

{OHM*04} O'Sullivan C., Howlett S., Morvan Y., McDonnell R., O'Conor K.: Perceptually Adaptive Graphics. In STAR - Proc. of Eurographics 2004 (2004), pp. 141--164.

[37]

{PHL*98} Pighin F., Hecker J., Lischinski D., Szeliski R., Salesin D. H.: Synthesizing realistic facial expressions from photographs. Proc. of ACM SIGGRAPH'98 (1998), 75--84.

Digital Library

[38]

{POM99} Pandzic I., Ostermann J., Millen D.: Users evaluations: synthetic talking faces for interactive services. The Visual Computer 15 (1999), 330--340.

Digital Library

[39]

{PR00} Pantic M., Rothkrantz L.: Automatic analysis of facial expressions: The state of the art. IEEE Trans. on PAMI 22, 12 (2000), 1424--1445.

Digital Library

[40]

{PRM00} Pavlovic V., Rehg J. M., MacCormick J.: Learning switching linear models of human motion. In NIPS (2000), pp. 981--987.

[41]

{RP03} Reitsma P. S. A., Pollard N. S.: Perceptual metrics for character animation: sensitivity to errors in ballistic motion. In ACM Trans. on Graph. (2003), vol. 22, pp. 537--542.

Digital Library

[42]

{SF98} Singh K., Fiume E.: Wires: A geometric deformation technique. In Proc. of ACM SIGGRAPH'98 (1998), pp. 405--414.

Digital Library

[43]

{SNF05} Sifakis E., Neverov I., Fedkiw R.: Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Trans. on Graph. 24, 3 (2005), 417--425.

Digital Library

[44]

{SP04} Sumner R. W., Popović J.: Deformation transfer for triangle meshes. ACM Trans. Graph. 23, 3 (2004), 399--405.

Digital Library

[45]

{SWCCG06} Schwaninger A., Wallraven C., Cunningham D. W., Chiller-Glaus S. D.: Processing of facial identity and expression a psychophysical, physiological and computational perspective. Progress in Brain Research 156 (2006).

[46]

{TKC01} Tian Y., Kanade T., Cohn J.: Recognizing action units for facial expression analysis. IEEE Trans. on PAMI 23, 2 (February 2001), 97--115.

Digital Library

[47]

{VBPP05} Vlasic D., Brand M., Pfister H., Popović J.: Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (2005), 426--433.

Digital Library

[48]

{VP06} Valstar M., Pantic M.: Fully automatic facial action unit detection and temporal analysis. In CVPRW '06: Proc. of the 2006 Conf. on Computer Vision and Pattern Recognition Workshop (2006), IEEE Computer Society.

Digital Library

[49]

{WB04} Wang J., Bodenheimer B.: Computing the duration of motion transitions: an empirical approach. In Proc. of SCA'04 (2004), pp. 335--344.

Digital Library

[50]

{WBCB05} Wallraven C., Breidt M., Cunningham D., Bulthoff H. H.: Psychophysical evaluation of animated facial expressions. In Proc. of APGV'05 (August 2005), pp. 17--24.

Digital Library

[51]

{WBCB08} Wallraven C., Breidt M., Cunningham D. W., Bulthoff H. H.: Evaluating the perceptual realism of animated facial expressions. ACM Trans. on Appl. Percep. (January 2008), 1--20.

Digital Library

[52]

{WFM01} Watson B., Friedman A., McGaffey A.: Measuring and predicting visual fidelity. In Proc. of ACM SIGGRAPH '01 (2001), pp. 213--220.

Digital Library

[53]

{Wil90} Williams L.: Performance-driven facial animation. In Proc. of ACM SIGGRAPH'90 (1990), pp. 235--242.

Digital Library

[54]

{WSZP07} Wampler K., Sasaki D., Zhang L., Popović Z.: Dynamic, expressive speech animation from a single mesh. In Proc. of SCA'07 (2007), pp. 53--62.

Digital Library

[55]

{ZLGS03} Zhang Q., Liu Z., Guo B., Shum H.: Geometry-driven photorealistic facial expression synthesis. In Proc. of SCA'03 (2003), pp. 177--186.

Digital Library

[56]

{ZSCS04} Zhang L., Snavely N., Curless B., Seitz S. M.: Spacetime faces: High-resolution capture for modeling and animation. ACM Trans. on Graph. 23, 3 (2004), 548--558.

Digital Library

Cited By

Carrigan EZibrek KDahyot RMcDonnell R(2020)Investigating perceptually based models to predict importance of facial blendshapesProceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games10.1145/3424636.3426904(1-6)Online publication date: 16-Oct-2020
https://dl.acm.org/doi/10.1145/3424636.3426904
Fan SWang RNg TTan CHerberg JKoenig B(2014)Human Perception of Visual Realism for Photo and Computer-Generated Face ImagesACM Transactions on Applied Perception10.1145/262003011:2(1-21)Online publication date: 30-Jul-2014
https://dl.acm.org/doi/10.1145/2620030
Zeng MLiang LLiu XBao H(2012)Video-driven state-aware facial animationComputer Animation and Virtual Worlds10.1002/cav.145523:3-4(167-178)Online publication date: 1-May-2012
https://dl.acm.org/doi/10.1002/cav.1455
Show More Cited By

Index Terms

Perceptually guided expressive facial animation
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging
  2. Computer graphics
    1. Animation
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces

Synthesizing expressive facial animation is a very challenging topic within the graphics community. In this paper, we present an expressive facial animation synthesis system enabled by automated learning from facial motion capture data. Accurate 3D ...
Mood swings: expressive speech animation

Motion capture-based facial animation has recently gained popularity in many applications, such as movies, video games, and human-computer interface designs. With the use of sophisticated facial motions from a human performer, animated characters are ...
Artist friendly facial animation retargeting

This paper presents a novel facial animation retargeting system that is carefully designed to support the animator's workflow. Observation and analysis of the animators' often preferred process of key-frame animation with blendshape models informed our ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SCA '08: Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation

July 2008

230 pages

ISBN:9783905674101

Conference Chairs:
Eugene Fiume,
Jerry Tessendorf,
Program Chairs:
Markus Gross,
Doug James

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Eurographics Association

Goslar, Germany

Publication History

Published: 07 July 2008

Check for updates

Qualifiers

Research-article

Conference

SCA08

Sponsor:

SIGGRAPH
EUROGRAPHICS

SCA08: The ACM SIGGRAPH / Eurographics Symposium on Computer Animation

July 7 - 9, 2008

Dublin, Ireland

Acceptance Rates

SCA '08 Paper Acceptance Rate 24 of 60 submissions, 40%;

Overall Acceptance Rate 183 of 487 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
401
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Carrigan EZibrek KDahyot RMcDonnell R(2020)Investigating perceptually based models to predict importance of facial blendshapesProceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games10.1145/3424636.3426904(1-6)Online publication date: 16-Oct-2020
https://dl.acm.org/doi/10.1145/3424636.3426904
Fan SWang RNg TTan CHerberg JKoenig B(2014)Human Perception of Visual Realism for Photo and Computer-Generated Face ImagesACM Transactions on Applied Perception10.1145/262003011:2(1-21)Online publication date: 30-Jul-2014
https://dl.acm.org/doi/10.1145/2620030
Zeng MLiang LLiu XBao H(2012)Video-driven state-aware facial animationComputer Animation and Virtual Worlds10.1002/cav.145523:3-4(167-178)Online publication date: 1-May-2012
https://dl.acm.org/doi/10.1002/cav.1455
Hill ABonner MMacIntyre B(2011)ClearspaceProceedings of the 2011 international conference on Virtual and mixed reality: systems and applications - Volume Part II10.5555/2028716.2028757(333-342)Online publication date: 9-Jul-2011
https://dl.acm.org/doi/10.5555/2028716.2028757
Ma XLe BDeng ZBodenheimer BTamstorf RGrinspun EHodgins J(2009)Style learning and transferring for facial animation editingProceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation10.1145/1599470.1599486(123-132)Online publication date: 1-Aug-2009
https://dl.acm.org/doi/10.1145/1599470.1599486

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten