Children's and adults' multimodal interaction with 2D conversational agents

ABSTRACT

Few systems combine both Embodied Conversational Agents (ECAs) and multimodal input. This research aims at modeling the behavior of adults and children during their multimodal interaction with ECAs. A Wizard-of-Oz setup was used and users were video-recorded while interacting with 2D ECAs in a game scenario with speech and pen as input modes. We found that frequent social cues and natural Human-Human syntax condition the verbal interaction of both groups with ECAs. Multimodality accounted for 21% of inputs: it was used for integrating conversational and social aspects (by speech) into task-oriented actions (by pen). We closely examined temporal and semantic integration of modalities: most of the time, speech and gesture overlapped and produced complementary or redundant messages; children also tended to produce concurrent multimodal inputs, as a way of doing several things at the same time. Design implications of our results for multimodal bidirectional ECAs and game systems are discussed.

REFERENCES

• 1: Cassell, J., Sullivan, J., Prevost, S., Churchill, E. Embodied Conversational Agents, MIT Press, 2000.
• 2: A. G. Hauptmann, Speech and gestures for graphic image manipulation, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.241-245, March 1989
• 3: Kipp, M. Anvil - A generic annotation tool for multimodal dialogue. In Proc. Eurospeech'01 (2001), pp. 1367--1370.
• 4: Sharon Oviatt, Multimodal interfaces for dynamic interactive maps, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.95-102, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238438]
• 5: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 6: Sharon Oviatt , Rachel Coulston , Stefanie Tomko , Benfang Xiao , Rebecca Lunsford , Matt Wesson , Lesley Carmichael, Toward a theory of organized multimodal integration patterns during human-computer interaction, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958443]
• 7: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 8: Reeves, B., Nass, C. The Media Equation. Cambridge University Press, New York, 1996.
• 9: Xiao, B., Girand, C., Oviatt, S.L. Multimodal integration patterns in children. In Proc. ICSLP'02, Casual Prod. Ltd (2002), pp. 629--632.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); User-centered design

Keywords:
adults vs. children, behavioral corpus, embodied conversational agents, multimodal interface