skip to main content
10.1145/1056808.1056886acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Article

Children's and adults' multimodal interaction with 2D conversational agents

Authors Info & Claims
Published:02 April 2005Publication History

ABSTRACT

Few systems combine both Embodied Conversational Agents (ECAs) and multimodal input. This research aims at modeling the behavior of adults and children during their multimodal interaction with ECAs. A Wizard-of-Oz setup was used and users were video-recorded while interacting with 2D ECAs in a game scenario with speech and pen as input modes. We found that frequent social cues and natural Human-Human syntax condition the verbal interaction of both groups with ECAs. Multimodality accounted for 21% of inputs: it was used for integrating conversational and social aspects (by speech) into task-oriented actions (by pen). We closely examined temporal and semantic integration of modalities: most of the time, speech and gesture overlapped and produced complementary or redundant messages; children also tended to produce concurrent multimodal inputs, as a way of doing several things at the same time. Design implications of our results for multimodal bidirectional ECAs and game systems are discussed.

References

  1. Cassell, J., Sullivan, J., Prevost, S., Churchill, E. Embodied Conversational Agents, MIT Press, 2000.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Hauptmann, A.G. Speech and gestures for graphic image manipulation. In Proc. CHI'89 (1989), pp. 241--245. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Kipp, M. Anvil - A generic annotation tool for multimodal dialogue. In Proc. Eurospeech'01 (2001), pp. 1367--1370.Google ScholarGoogle Scholar
  4. Oviatt, S.L. Multimodal interfaces for dynamic interactive maps. In Proc. CHI '96 (1996), pp. 95--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Oviatt, S. L. Ten myths of multimodal interaction. Communications of the ACM 42, (1999), pp. 74--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Oviatt, S., Coulston, R., Tomko, S., Xiao, B., Lunsford, R., Wesson, M., Carmichael, L. Toward a theory of organized multimodal integration patterns during Human-Computer Interaction. In Proc. ICMI'03, ACM Press (2003), pp. 44--51. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Oviatt, S. L., DeAngeli, A. & Kuhn, K. Integration and synchronization of input modes during multimodal human-computer interaction. In Proc. CHI '97 (1997), pp. 415--422. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Reeves, B., Nass, C. The Media Equation. Cambridge University Press, New York, 1996.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Xiao, B., Girand, C., Oviatt, S.L. Multimodal integration patterns in children. In Proc. ICSLP'02, Casual Prod. Ltd (2002), pp. 629--632.Google ScholarGoogle Scholar

Index Terms

  1. Children's and adults' multimodal interaction with 2D conversational agents

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in
        • Published in

          cover image ACM Conferences
          CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems
          April 2005
          1358 pages
          ISBN:1595930027
          DOI:10.1145/1056808

          Copyright © 2005 ACM

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 2 April 2005

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • Article

          Acceptance Rates

          Overall Acceptance Rate6,164of23,696submissions,26%

          Upcoming Conference

          CHI '24
          CHI Conference on Human Factors in Computing Systems
          May 11 - 16, 2024
          Honolulu , HI , USA

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader