skip to main content
10.1145/1322192.1322198acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Faces of pain: automated measurement of spontaneousallfacial expressions of genuine and posed pain

Published: 12 November 2007 Publication History

Abstract

We present initial results from the application of an automated facial expression recognition system to spontaneous facial expressions of pain. In this study, 26 participants were videotaped under three experimental conditions: baseline, posed pain, and real pain. In the real pain condition, subjects experienced cold pressor pain by submerging their arm in ice water. Our goal was to automatically determine which experimental condition was shown in a 60 second clip from a previously unseen subject. We chose a machine learning approach, previously used successfully to categorize basic emotional facial expressions in posed datasets as well as to detect individual facial actions of the Facial Action Coding System (FACS) (Littlewort et al, 2006; Bartlett et al., 2006). For this study, we trained 20 Action Unit (AU) classifiers on over 5000 images selected from a combination of posed and spontaneous facial expressions. The output of the system was a real valued number indicating the distance to the separating hyperplane for each classifier. Applying this system to the pain video data produced a 20 channel output stream, consisting of one real value for each learned AU, for each frame of the video. This data was passed to a second layer of classifiers to predict the difference between baseline and pained faces, and the difference between expressions of real pain and fake pain. Naíve human subjects tested on the same videos were at chance for differentiating faked from real pain, obtaining only 52% accuracy. The automated system was successfully able to differentiate faked from real pain. In an analysis of 26 subjects, the system obtained 72% correct for subject independent discrimination of real versus fake pain on a 2-alternative forced choice. Moreover, the most discriminative facial action in the automated system output was AU 4 (brow lower), which all was consistent with findings using human expert FACS codes.

References

[1]
Bartlett M. S., Littlewort G. C., Frank M. G., Lainscsek C., Fasel I., and Movellan J. R., "Automatic recognition of facial actions in spontaneous expressions.," Journal of Multimedia., 1(6) p. 22--35.
[2]
Cohn, J. F. & Schmidt, K. L. (2004). The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing, Vol. 2, No. 2, pp. 121--132.
[3]
Craig K. D, Hyde S., Patrick C. J. (1991). Genuine, supressed, and faked facial behaviour during exacerbation of chronic low back pain. Pain 46:161--172.
[4]
Craig K. D, Patrick C. J. (1985). Facial expression during induced pain. J Pers Soc Psychol. 48(4):1080--91.
[5]
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P. & Sejnowski, T. J. (1999). Classifying facial actions. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 21, No. 10, pp. 974--989.
[6]
Ekman P. and Friesen, W. Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press, Palo Alto, CA, 1978.
[7]
Ekman, P. (2001). Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton, New York, USA.
[8]
Ekman, P. & Rosenberg, E. L., (Eds.), (2005). What the face reveals: Basic and applied studies of spontaneous expression using the FACS, Oxford University Press, Oxford, UK.
[9]
Fasel I., Fortenberry B., Movellan J. R. "A generative framework for real-time object detection and classification.," Computer Vision and Image Understanding 98, 2005.
[10]
Fishbain D. A, Cutler R., Rosomoff H. L, Rosomoff R. S. (1999). Chronic pain disability exaggeration/malingering and submaximal effort research. Clin J Pain. 15(4):244--74.
[11]
Fishbain D. A, Cutler R., Rosomoff H. L, Rosomoff R. S. (2006). Accuracy of deception judgments. Pers Soc Psychol Rev. 10(3):214--34.
[12]
Frank M. G., Ekman P., Friesen W. V. (1993). Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol. 64(1):83--93.
[13]
Grossman, S., Shielder, V., Swedeen, K., Mucenski, J. (1991). Correlation of patient and caregiver ratings of cancer pain. Journal of Pain and Symptom Management 6(2), p. 53--57.
[14]
Hadjistavropoulos H. D., Craig K. D., Hadjistavropoulos T., Poole GD. (1996). Subjective judgments of deception in pain expression: accuracy and errors. Pain. 65(2-3):251--8.
[15]
Hill M. L., Craig K. D. (2002) Detecting deception in pain expressions: the structure of genuine and deceptive facial displays. Pain. 98(1-2):135--44.
[16]
Kanade, T., Cohn, J. F. and Tian, Y., "Comprehensive database for facial expression analysis," in Proceedings of the fourth IEEE International conference on automatic face and gesture recognition (FG'00), Grenoble, France, 2000, pp. 46--53.
[17]
Larochette A. C., Chambers C. T., Craig K. D. (2006). Genuine, suppressed and faked facial expressions of pain in children. Pain. 2006 Dec 15;126(1-3):64--71.
[18]
Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J. & Movellan, J. (2006). Dynamics of facial expression extracted automatically from video. J. Image & Vision Computing, Vol. 24, No. 6, pp. 615--625.
[19]
Morecraft R. J, Louie J. L., Herrick J. L., Stilwell-Morecraft KS. (2001). Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. Brain 124(Pt 1):176--208.
[20]
Pantic, M., Pentland, A., Nijholt, A. & Huang, T. (2006). Human Computing and machine understanding of human behaviour: A Survey, Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 239--248.
[21]
Pantic, M. F. Valstar, R. Rademaker and L. Maat, "Web-based Database for Facial Expression Analysis", Proc. IEEE Int'l Conf. Multmedia and Expo (ICME'05), Amsterdam, The Netherlands, July 2005.
[22]
Prkachin K. M. (1992). The consistency of facial expressions of pain: a comparison across modalities. Pain. 51(3):297--306.
[23]
Prkachin K. M., Schultz I., Berkowitz J., Hughes E., Hunt D. Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav Res Ther. 40(5):595--607.
[24]
Rinn W. E. The neuropsyhology of facial expression: a review of the neurological and psychological mechanisms for producing facial expression. Psychol Bull 95:52--77.
[25]
Schmand B., Lindeboom J., Schagen S., Heijt R., Koene T., Hamburger H. L. Cognitive complaints in patients after whiplash injury: the impact of malingering. J Neurol Neurosurg Psychiatry. 64(3):339--43.
[26]
Schmidt K. L., Cohn J. F., Tian Y. (2003). Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 65(1):49--66.
[27]
Schneiderman, H. and Kanade, T. (1998). Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 45--51.
[28]
Viola, P. & Jones, M. (2004). Robust real-time face detection. J. Computer Vision, Vol. 57, No. 2, pp. 137--154.
[29]
Vural, E., Ercil, A., Littlewort, G.C., Bartlett, M.S., and Movellan, J.R. (2007).allMachine learning systems for detecting driver drowsiness. Proceedings of the Biennial Conference on Digital Signal Processing for in-Vehicle and Mobile Systems.

Cited By

View all
  • (2025)Individuality Prediction of Pain Facial Expressions Based on Permanent WrinklesGenetic and Evolutionary Computing10.1007/978-981-96-1535-3_3(21-27)Online publication date: 14-Feb-2025
  • (2024)An Iterative Approach to Build a Semantic Dataset for Facial Expression of PersonalityProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696333(1-11)Online publication date: 21-Nov-2024
  • (2023)Denver pain authenticity stimulus set (D-PASS)Behavior Research Methods10.3758/s13428-023-02283-256:4(2992-3008)Online publication date: 22-Nov-2023
  • Show More Cited By

Index Terms

  1. Faces of pain: automated measurement of spontaneousallfacial expressions of genuine and posed pain

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces
      November 2007
      402 pages
      ISBN:9781595938176
      DOI:10.1145/1322192
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 12 November 2007

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. FACS
      2. computer vision
      3. deception
      4. facial action coding system
      5. facial expression recognition
      6. machine learning
      7. pain
      8. spontaneous behavior

      Qualifiers

      • Research-article

      Conference

      ICMI07
      Sponsor:
      ICMI07: International Conference on Multimodal Interface
      November 12 - 15, 2007
      Aichi, Nagoya, Japan

      Acceptance Rates

      Overall Acceptance Rate 453 of 1,080 submissions, 42%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)29
      • Downloads (Last 6 weeks)6
      Reflects downloads up to 19 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Individuality Prediction of Pain Facial Expressions Based on Permanent WrinklesGenetic and Evolutionary Computing10.1007/978-981-96-1535-3_3(21-27)Online publication date: 14-Feb-2025
      • (2024)An Iterative Approach to Build a Semantic Dataset for Facial Expression of PersonalityProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696333(1-11)Online publication date: 21-Nov-2024
      • (2023)Denver pain authenticity stimulus set (D-PASS)Behavior Research Methods10.3758/s13428-023-02283-256:4(2992-3008)Online publication date: 22-Nov-2023
      • (2023)Development of a binary classifier model from extended facial codes toward video-based pain recognition in cancer patientsScandinavian Journal of Pain10.1515/sjpain-2023-001123:4(638-645)Online publication date: 5-Sep-2023
      • (2023)Analyzing facial action units in children to differentiate genuine and fake pain during inferior alveolar nerve block: a cross-sectional studyScientific Reports10.1038/s41598-023-42982-613:1Online publication date: 20-Sep-2023
      • (2023)BAUFER: A Baseline-Enabled Facial Expression Recognition Pipeline Trained with Limited AnnotationsArtificial Intelligence for Personalized Medicine10.1007/978-3-031-36938-4_17(221-236)Online publication date: 2-Sep-2023
      • (2022)Pain E-motion Faces Database (PEMF): Pain-related micro-clips for emotion researchBehavior Research Methods10.3758/s13428-022-01992-455:7(3831-3844)Online publication date: 17-Oct-2022
      • (2022)Assessing Pain Research: A Narrative Review of Emerging Pain Methods, Their Technosocial Implications, and Opportunities for Multidisciplinary ApproachesFrontiers in Pain Research10.3389/fpain.2022.8962763Online publication date: 2-Jun-2022
      • (2022)Game Outlier Behavior Detection System Based on Dynamic Time Warp AlgorithmComputer Modeling in Engineering & Sciences10.32604/cmes.2022.018413131:1(219-237)Online publication date: 2022
      • (2022)A Multimodal Non-Intrusive Stress Monitoring From the Pleasure-Arousal Emotional DimensionsIEEE Transactions on Affective Computing10.1109/TAFFC.2020.298845513:2(1044-1056)Online publication date: 1-Apr-2022
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media