skip to main content
10.1145/1027933.1027942acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
Article

Multimodal transformed social interaction

Published: 13 October 2004 Publication History

Abstract

Understanding human-human interaction is fundamental to the long-term pursuit of powerful and natural multimodal interfaces. Nonverbal communication, including body posture, gesture, facial expression, and eye gaze, is an important aspect of human-human interaction. We introduce a paradigm for studying multimodal and nonverbal communication in collaborative virtual environments (CVEs) called Transformed Social Interaction (TSI), in which a user's visual representation is rendered in a way that strategically filters selected communication behaviors in order to change the nature of a social interaction. To achieve this, TSI must employ technology to detect, recognize, and manipulate behaviors of interest, such as facial expressions, gestures, and eye gaze. In [13] we presented a TSI experiment called <i>non-zero-sum gaze</i> (NZSG) to determine the effect of manipulated eye gaze on persuasion in a small group setting. Eye gaze was manipulated so that each participant in a three-person CVE received eye gaze from a presenter that was normal, less than normal, or greater than normal. We review this experiment and discuss the implications of TSI for multimodal interfaces.

References

[1]
Argyle, M. Bodily Communication (2nd edition). London, England, UK: Methuen, 1988.
[2]
Depaulo, B. M., and Friedman, H. S. Nonverbal communication. In D. T. Gilbert, S. T. Fiske, and G. Lindzey (eds.), The Handbook of Social Psychology (4th edition), pp. 3--40, Vol. 2, Boston: McGraw-Hill, 1998.
[3]
Quek, F., McNeill, D., Bryll, B., Duncan, S., Ma, X., Kirbas, C., McCullough, K-E, and Ansari, R. Multimodal Human Discourse: Gesture and Speech &lt;http://vislab.cs.wright.edu/Publications/2001/Queetal01.html&gt;, ACM Transactions on Computer-Human Interaction, Vol. 9, No. 3, September 2002, pp. 171--193.
[4]
Oviatt, S. L. Multimodal interface research: A science without borders. Proceedings of the International Conference of Spoken Language Processing (ICSLP'2000), ed. by B. Yuan, T. Huang & X. Tang, Beijing: Chinese Friendship Publishers, 2000, vol. 3, 1--6.
[5]
Zhang, X. and Furnas, G. Social interactions in multiscale CVEs. Proceedings of the ACM Conference on Collaborative Virtual Environments, 2002.
[6]
Normand, V. et al. The COVEN project: exploring applicative, technical and usage dimensions of collaborative virtual environment. Presence: Teleoperators and Virtual Environments, 8(2), 1999.
[7]
Bailenson, J., Beall, A., Blascovich, J., Loomis, J. and Turk, M. Transformed social interaction: decoupling representation from behavior and form in collaborative virtual environments. PRESENCE: Teleoperators and Virtual Environments, MIT press, Vol. 13, No. 4, August 2004.
[8]
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., and Bailenson, J. N. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychological Inquiry, 13, 146--149, 2001.
[9]
Beall, A.C., Bailenson, J.N., Loomis, J., Blascovich, J., and Rex, C. Non-Zero-Sum Mutual Gaze in Collaborative Virtual Environments. Proceedings of HCI International, Crete, 2003.
[10]
Cappella, J. N. and Pelachaud, C. Rules for responsive robots: Using human interactions to build virtual interactions. In Vangelisti, A.L., Reis, H.T., and Fitzpatrick, M. (Eds.), Stability and Change in Relationships, pp. 325--354, Cambridge, MA: Cambridge University Press, 2002.
[11]
Morton, G. Effect of eye contact and distance on the verbal reinforcement of attitude. Journal of Social Psychology, 111, pp. 73--78, 1980.
[12]
Sherwood, J. V. Facilitative effects of gaze upon learning. Perceptual and Motor Skills, 64, pp. 1275--1278, 1987.
[13]
Bailenson, J., Beall, A., Blascovich, J., Loomis, J., and Turk, M. Non-zero-sum gaze and persuasion in immersive virtual environments. Manuscript under review.
[14]
Gigorovici, G. Persuasive effects of presence in immersive virtual environments. In G. Riva, F. Davide (Eds), Being There: concepts effects and measurement of user presence in synthetic environments, Amsterdam, The Netherlands: IOS Press, online, 2003.
[15]
Guadagno, R. E. and Cialdini, R. B. Online persuasion: An examination of gender differences in computer-mediated interpersonal influence. Group Dynamics: Theory Research and Practice. Special Issue on Internet Research, 6, pp. 38--51, 2002.
[16]
Lanier, J. Virtually there. Scientific American, April, 2001.

Cited By

View all
  • (2022)Decentralized, not Dehumanized in the Metaverse: Bringing Utility to NFTs through Multimodal InteractionProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3558176(662-667)Online publication date: 7-Nov-2022
  • (2013)Avatar interfaces for biobehavioral feedbackProceedings of the Second international conference on Design, User Experience, and Usability: health, learning, playing, cultural, and cross-cultural user experience - Volume Part II10.1007/978-3-642-39241-2_47(424-434)Online publication date: 21-Jul-2013
  • (2013)Factors of Influence in Co-located Multimodal InteractionsIntelligent Interactive Technologies and Multimedia10.1007/978-3-642-37463-0_21(231-245)Online publication date: 2013
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces
October 2004
368 pages
ISBN:1581139950
DOI:10.1145/1027933
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2004

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. computer-mediated communication
  2. multimodal processing
  3. transformed social interaction

Qualifiers

  • Article

Conference

ICMI04
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)16
  • Downloads (Last 6 weeks)1
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Decentralized, not Dehumanized in the Metaverse: Bringing Utility to NFTs through Multimodal InteractionProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3558176(662-667)Online publication date: 7-Nov-2022
  • (2013)Avatar interfaces for biobehavioral feedbackProceedings of the Second international conference on Design, User Experience, and Usability: health, learning, playing, cultural, and cross-cultural user experience - Volume Part II10.1007/978-3-642-39241-2_47(424-434)Online publication date: 21-Jul-2013
  • (2013)Factors of Influence in Co-located Multimodal InteractionsIntelligent Interactive Technologies and Multimedia10.1007/978-3-642-37463-0_21(231-245)Online publication date: 2013
  • (2006)Evaluating Perception of Interaction Initiation in Virtual Environments using Humanoid AgentsProceedings of the 2006 conference on ECAI 2006: 17th European Conference on Artificial Intelligence August 29 -- September 1, 2006, Riva del Garda, Italy10.5555/1567016.1567032(46-50)Online publication date: 22-May-2006
  • (2005)A model of attention and interest using Gaze behaviorLecture Notes in Computer Science10.1007/11550617_20(229-240)Online publication date: 12-Sep-2005

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media