Affective Grounding in Human-Robot Interaction

ABSTRACT

Participating in interaction requires not only coordination on content and process, as previously proposed, but also on affect. The term affective grounding is introduced to refer to the coordination of affect in interaction with the purpose of building shared understanding about what behavior can be exhibited, and how behavior is interpreted emotionally and responded to. Affective Ground is achieved when interactants have reached shared understanding about how behavior should be interpreted emotionally. The paper contributes a review and critique of current perspectives on emotion in HRI. Further it outlines how research on emotion in HRI can benefit from taking an affective grounding perspective and outlines implications for the design of robots capable of participating in the coordination on affect in interaction.

References

1. Ahn, H. S., Lee, D.-W., Choi, D., Lee, D.-Y., Hur, M., and Lee, H. (2012). Appropriate emotions for facial expressions of 33-dofs android head ever-4 h33. In RO-MAN, 2012 IEEE, pages 1115--1120. IEEE.
2. Andersson, L. M., & Pearson, C. M. (1999). Tit for tat? The spiraling effect of incivility in the workplace. Academy of Management Review, 24(3): 452--471.
3. Barrett, L. F. (2006). Are emotions natural kinds?. Perspectives on psychological science, 1(1), 28--58.
4. Bennett, C. C. and Šabanović, S. (2014). Deriving minimal features for human-like facial expressions in robotic faces. International Journal of Social Robotics, 6(3):367--381.
5. Bethel,C.L., & Murphy, R. R. (2007). Non-facial/non-verbal methods of affective expression as applied to robot- assisted victim assessment. Proceedings of HRI 2007 (287--294). NY: ACM Press
6. Boehner, K., DePaula, R., Dourish, P., & Sengers, P. (2007). How emotion is made and measured. International Journal of Human-Computer Studies, 65(4), 275--291.
7. Bowman, M., Debray, S. K., and Peterson, L. L. 1993. Reasoning about naming systems. ACM Trans. Program. Lang. Syst. 15, 5 (Nov. 1993), 795--825.
8. Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1), 119--155.
9. Breazeal, C.L. (2004). Designing sociable robots. Cambridge: The MIT Press.
10. Breazeal, C. and B. Scassellati. (1999). How to build robots that make friends and influence people. Proceedings of IROS '99.
11. Breazeal, C., DePalma, N., Orkin, J., Chernova, S., & Jung, M. (2013). Crowdsourcing human-robot interaction: New methods and system evaluation in a public environment. Journal of Human-Robot Interaction, 2(1), 82--111.
12. Breazeal, C., Kidd, C. D., Thomaz, A. L., Hoffman, G., & Berlin, M. (2005). Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. Proceedings of IROS 2005 (708--713).
13. Bretan, M., Hoffman, G., and Weinberg, G. (2015). Emotionally expressive dynamic physical behaviors in robots. International Journal of Human-Computer Studies, 78:1--16.
14. Brown, L. D., Hua, H., and Gao, C. 2003. A widget framework for augmented interaction in SCAPE. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 1--10.
15. Chu, V., Bullard, K., & Thomaz, A. L. (2014, September). Multimodal real-time contingency detection for HRI. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 3327--3332). IEEE.
16. Clark, H. H. & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, R. M. Levine, & S. D. Teasley (Eds.). Perspectives on socially shared cognition (pp. 127--149). Washington, DC: APA Press.
17. Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22, 1--39.
18. Clark, M. S., Pataki, S. P., & Carver, V. H. 1996. Some thoughts and findings on self-presentation of emotions in relationships. In G. J. O. Fletcher & J. Fitness (Eds.), Knowledge structures in close relationships: A social psychological approach: 247--274). Mahwah, NJ: Erlbaum.
19. Coan, J. A., & Gottman, J. M. (2007). The specific affect coding system (SPAFF). In J. A. Coan & J. J. B. Allen (Eds.) Handbook of Emotion Elicitation and Assessment. (106--123). New York, NY: Oxford University Press.
20. Costa, S., Soares, F., and Santos, C. (2013). Facial expressions and gestures to convey emotions with a humanoid robot. In Social Robotics, pages 542--551. Springer.
21. Couper-Kuhlen, E. (2009). On combining clauses and actions in interaction. Virittäjä, 113(3).
22. Curhan, J. R., & Pentland, A. (2007). Thin slices of negotiation: predicting outcomes from conversational dynamics within the first 5 minutes. Journal of Applied Psychology, 92(3), 802.Å
23. Damm, O., Dreier, K., Hegel, F., Jaecks, P., Stenneken, P., Wrede, B., & Hielscher-Fastabend, M. (2011, January). Communicating emotions in robotics: Towards a model of emotional alignment. In Proceedings of the workshop "Expectations in intuitive interaction" on the 6th HRI International conference on Human-Robot Interaction.
24. Ding, W. and Marchionini, G. 1997. A Study on Video Browsing Strategies. Technical Report. University of Maryland at College Park.
25. Driver, J. L., & Gottman, J. M. 2004. Daily marital interactions and positive affect during marital conflict among newlywed couples. Family Processes, 43(3): 301--314.
26. Ekman, P. (1984). Expression and the nature of emotion. In P. Ekman & K. Scherer (Eds.), Approaches to emotion (pp. 319--343). Hillsdale. NJ: Erlbaum.
27. Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., ... & Scherer, K. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of personality and social psychology, 53(4), 712.
28. Ekman. P. (1993). Facial expression and emotion. American Psychologist, 48, 384--392.
29. Ekman, P., & Friesen, W. V. (1977). Facial action coding system.
30. Fernández-Dols, J.-M., Sanchez, F. , Carrera, P., Ruiz-Belda, M.-A. (1997) Are spontaneous expressions and emotions linked? An experimental test of coherence, Journal of Nonverbal Behavior, vol. 31, pp. 163--177.
31. Fincannon, T., Barnes, L. E., Murphy, R. R., & Riddle, D. L. (2004). Evidence of the need for social intelligence in rescue robots. Proceedings of IROS 2004. (pp. 1089--1095). IEEE/RSJ 2.
32. Fischer, A. H., & Manstead, A. S. 2008. Social functions of emotion. In Lewis, M., Haviland-Jones, J. M., and Barrett, L. F. (Eds.), Handbook of emotions, (3rd ed.): 456--468. New York, NY: The Guilford Press
33. Forman, G. 2003. An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res. 3 (Mar. 2003), 1289--1305.
34. Fröhlich, B. and Plate, J. 2000. The cubic mouse: a new device for three-dimensional input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. ACM, New York, NY, 526--531.
35. Giese-Davis, J., Spiegel, D., (2003) Emotional expression and cancer progression. Handbook of affective sciences. (pp. 1053--1082). New York: Oxford University Press.
36. Gockley, R., Simmons, R., and Forlizzi, J. (2006). Modeling affect in socially interactive robots. In Robot and Human Interactive Communication, 2006. ROMAN 2006. The 15th IEEE Inter- national Symposium on, pages 558--563. IEEE.
37. Goffman, E. (1961). Fun in games. Encounters, 15--81.
38. Goodwin, M. H., & Goodwin, C. (2000). Emotion within situated activity. Communication: An arena of development, 33--53.
39. Gottman, J. M., & Levenson, R. W. (1992). Marital processes predictive of later dissolution: behavior, physiology, and health. Journal of personality and social psychology, 63(2), 221.
40. Gross, J. J. (1998). The emerging field of emotion regulation: An integrative review. Review of General Psychology 2, 271--299.
41. Gross, J. J. (2013). Emotion regulation: taking stock and moving forward. Emotion, 13(3), 359.
42. Hiolle, A., Lewis, M., & Cañamero, L. (2014). Arousal regulation and affective adaptation to human responsiveness by a robot that explores and learns a novel environment. Frontiers in neurorobotics, 8.
43. Hochschild, A. R. (2003). The managed heart: Commercialization of human feeling. Univ of California Press.
44. Horney, K. (1945). Our inner conflicts. New York: Norton.
45. Hoque, M. E., McDuff, D. J., & Picard, R. W. (2012). Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Transactions on Affective Computing, 3(3), 323--334.
46. Jefferson, G.,;Sacks, H., Schegloff, E. (1987). Notes on laughter in the pursuit of intimacy. Button, G. (Ed); Lee, J. R. E. (Ed). . Talk and social organisation., (pp. 152--205). Clevedon, England: Multilingual Matters, 335 pp.
47. Jung, M., Chong, J., & Leifer, L. (2012). Group hedonic balance and pair programming performance: a ective interaction dynamics as indicators of performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 829--838). ACM.
48. Jung, M. F. (2016). Coupling Interactions and Performance: Predicting Team Performance from Thin Slices of Conflict. ACM Transactions on Computer-Human Interaction (TOCHI), 23(3), 18.
49. Jung, M. F., Lee, J. J., DePalma, N., Adalgeirsson, S. O., Hinds, P. J., & Breazeal, C. (2013, February). Engaging robots: easing complex human-robot teamwork using backchanneling. In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 1555--1566). ACM.
50. Jung, M. F., Martelaro, N., & Hinds, P. J. (2015). Using robots to moderate team conflict: the case of repairing violations. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (pp. 229--236). ACM.
51. Kaindl, H., Popp, R., Raneburger, D., Ertl, D., Falb, J., Szep, A., & Bogdan, C. (2011). Robot-supported cooperative work: A shared-shopping scenario. In System Sciences (HICSS), 2011 44th Hawaii International Conference on (pp. 1--10). IEEE.
52. Karg, M., Schwimmbeck, M., Kuhnlenz, K., and Buss, M. (2010). Towards mapping emotive gait patterns from human to robot. In RO-MAN, 2010 IEEE, pages 258--263. IEEE.
53. Kedzierski, J., Muszynški, R., Zoll, C., Oleksy, A., and Frontkiewicz, M. (2013). Emys--emotive head of a social robot. International Journal of Social Robotics, 5(2):237--249.
54. Keltner, D., & Haidt, J. 1999. Social Functions of Emotions at Four Levels of Analysis. Cognition & Emotion, 13(5): 505--521.
55. Kirstein, F., Fischer, K., Erkent, Ö., & Piater, J. (2015). Human smile distinguishes between collaborative and solitary tasks in human-robot interaction. In Proceedings of HRI 2015. (pp. 145--146). ACM.
56. Kishi, T., Kojima, T., Endo, N., Destephe, M., Otani, T., Jamone, L., Kryczka, P., Trovato, G., Hashimoto, K., Cosentino, S., et al. (2013). Impression survey of the emotion expression hu- manoid robot with mental model based dynamic emotions. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 1663--1668. IEEE.
57. Knight, H. and Simmons, R. (2014). Expressive motion with x, y and theta: Laban effort features for mobile robots. In Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on, pages 267--273. IEEE.
58. Kraut, R. E., & Johnston, R. E. (1979). Social and emotional messages of smiling: An ethological approach. Journal of personality and social psychology, 37(9), 1539.
59. Leahu, L., & Sengers, P. (2014, June). Freaky: performing hybrid human-machine emotion. In Proceedings of the 2014 conference on Designing interactive systems (pp. 607--616). ACM.
60. Levenson, R. W., Carstensen, L. L., & Gottman, J. M. 1994. Influence of age and gender on affect, physiology, and their interrelations: A study of long-term marriages. Journal of Personality and Social Psychology, 67(1): 56--68.
61. Levenson, R. W., & Gottman, J. M. 1985. Physiological and affective predictors of change in relationship satisfaction. Journal of Personality and Social Psychology, 49(1): 85--94.
62. Lewis, M., & Canamero, L. (2014, August). An affective autonomous robot toddler to support the development of self-efficacy in diabetic children. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication.
63. Lütkebohle, I., Hegel, F., Schulz, S., Hackel, M., Wrede, B., Wachsmuth, S., and Sagerer, G. (2010). The bielefeld anthropomorphic robot head "flobi". In Robotics and Automation, 2010. ICRA 2010. Proceedings IEEE International Conference on, volume 3(7), pages 3384--3391.
64. Masuda, M. and Kato, S. (2010). Motion rendering system for emotion expression of human form robots based on laban movement analysis. In RO-MAN, 2010 IEEE, pages 324--329. IEEE.
65. Morris, M. W., & Keltner, D. 2000. How emotions work: The social functions of emotional expression in negotiations. Research in Organizational Behavior, 22: 1--50.
66. Murphy, R. R., Lisetti, C. L., Tardif, R., Irish, L., & Gage, A. (2002). Emotion-based control of cooperating heterogeneous mobile robots. IEEE Transactions on Robotics and Automation 18(5), 744--757.
67. Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H., & Hagita, N. (2009). Footing in human-robot conversations: how robots might shape participant roles using gaze cues. Proceedings of HRI 2009 (61--68). NY: ACM Press.
68. Pasupathi, M., Carstensen, L. L., Levenson, R. W., & Gottman, J. M. (1999). Responsive listening in long-married couples: A psycholinguistic perspective. Journal of Nonverbal Behavior, 23(2), 173--193.
69. Perakyla, A., & Ruusuvuori, J. (2012). Facial Expression and Interactional Regulation of Emotion. Emotion in interaction, ed. A Perakyla, M Sorjonen, 64--91.
70. Reeck, C., Ames, D. R., & Ochsner, K. N. (2016). The social regulation of emotion: An integrative, cross-disciplinary model. Trends in cognitive sciences, 20(1), 47--63.
71. Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 375--382). IEEE.
72. Rimé, B. (2007). Interpersonal emotion regulation. Handbook of emotion regulation, 1, 466--468.
73. Rose, R., Scheutz, M, and Schermerhorn, P. (2008). Empirical investigations into the believability of robot affect. Proceedings of the AAAI Spring Symposium.
74. Rosenberg, E. L., Ekman, P. (1994) Coherence between expressive and experiential systems in emotion," Cognition and Emotion, vol. 8, pp. 201--229.
75. Ruusuvuori, J. (2013). Emotion, affect and conversation. The handbook of conversation analysis, 330--349.
76. Russell, J. A. (1978). Evidence of convergent validity on the dimensions of affect. Journal of personality and social psychology, 36(10), 1152.
77. Russell, J. A., & Barrett, L. F. (1999). Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. Journal of personality and social psychology, 76(5), 805.
78. Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. language, 696--735.
79. Sannella, M. J. 1994. Constraint Satisfaction and Debugging for Interactive User Interfaces. Doctoral Thesis. UMI Order Number: UMI Order No. GAX95-09398., University of Washington.
80. Schegloff, E. (2007). Sequence organization in interaction: Volume 1: A primer in conversation analysis (Vol. 1). Cambridge, UK: Cambridge University Press.
81. Scheutz, M., Schermerhorn, P., & Kramer, J. (2006). The utility of affect expression in natural language interactions in joint human-robot tasks. In Proceedings of HRI 2006 (226--233). NY: ACM Press.
82. Selting, M. (1994). Emphatic speech style mdash; with special focus on the prosodic signalling of heightened emotive involvement in conversation. Journal of pragmatics, 22(3), 375--408.
83. Sharma, M., Hildebrandt, D., Newman, G., Young, J. E., and Eskicioglu, R. (2013). Communicating affect via flight path exploring use of the laban effort system for designing affective locomotion paths. In Human-Robot Interaction (HRI), 2013 8th ACM/IEEE International Conference on, pages 293--300. IEEE.
84. Shibata, H., Kano, M., Kato, S., and Itoh, H. (2006). A system for converting robot 'emotion' into facial expressions. In Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, pages 3660--3665. IEEE.
85. Spector, A. Z. 1989. Achieving application requirements. In Distributed Systems, S. Mullender, Ed. ACM Press Frontier Series. ACM, New York, NY, 19--33.
86. Tavel, P. 2007. Modeling and Simulation Design. AK Peters Ltd., Natick, MA.
87. Thomaz, A., Hoffman, G., Cakmak, M., (In Press) Computational Human-Robot Interaction. Foundations and Trends in Human Computer Interaction.
88. Tielman, M., Neerincx, M., Meyer, J. J., & Looije, R. (2014, March). Adaptive emotional expression in robot-child interaction. In Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction (pp. 407--414). ACM.
89. Van Kleef, G. A. (2009). How emotions regulate social life the emotions as social information (EASI) model. Current directions in psychological science, 18(3): 184--188.
90. Van Kleef, G. A., De Dreu, C. K., & Manstead, A. S. (2010). An interpersonal approach to emotion in social decision making: The emotions as social information model. In Mark P. Zanna, editor: Advances in experimental social psychology, 42, 45--96.
91. Watson, D., & Tellegen, A. 1985. Toward a consensual structure of mood. Psychological Bulletin, 98(2): 219--235.
92. Wilkinson, S., & Kitzinger, C. (2006). Surprise as an interactional achievement: Reaction tokens in conversation. Social psychology quarterly, 69(2), 150--182.
93. Xu, J.,Broekens, J., Hindriks, K., and Neerincx, M. A. (2015). Mood contagion of robot body language in human robot interaction. Autonomous Agents and Multi-Agent Systems 29, 6. 1216--1248.
94. Yu, Y. T. and Lau, M. F. 2006. A comparison of MC/DC, MUMCUT and several other coverage criteria for logical decisions. J. Syst. Softw. 79, 5 (May. 2006), 577--590.
95. Zaki, J., & Williams, W. C. (2013). Interpersonal emotion regulation. Emotion, 13(5), 803.
96. Zhang, J. and Sharkey, A. J. (2011). Listening to sad music while seeing a happy robot face. In Social Robotics, pages 173--182. Springer.
97. Ziemke, T., & Lowe, R. (2009). On the role of emotion in embodied cognitive architectures: From organisms to robots. Cognitive computation, 1(1), 104--117.

Index Terms

• Computer systems organization

  • Embedded and cyber-physical systems

    • Robotics

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI theory, concepts and models