ABSTRACT
Few systems combine both Embodied Conversational Agents (ECAs) and multimodal input. This research aims at modeling the behavior of adults and children during their multimodal interaction with ECAs. A Wizard-of-Oz setup was used and users were video-recorded while interacting with 2D ECAs in a game scenario with speech and pen as input modes. We found that frequent social cues and natural Human-Human syntax condition the verbal interaction of both groups with ECAs. Multimodality accounted for 21% of inputs: it was used for integrating conversational and social aspects (by speech) into task-oriented actions (by pen). We closely examined temporal and semantic integration of modalities: most of the time, speech and gesture overlapped and produced complementary or redundant messages; children also tended to produce concurrent multimodal inputs, as a way of doing several things at the same time. Design implications of our results for multimodal bidirectional ECAs and game systems are discussed.
- Cassell, J., Sullivan, J., Prevost, S., Churchill, E. Embodied Conversational Agents, MIT Press, 2000.Google ScholarDigital Library
- Hauptmann, A.G. Speech and gestures for graphic image manipulation. In Proc. CHI'89 (1989), pp. 241--245. Google ScholarDigital Library
- Kipp, M. Anvil - A generic annotation tool for multimodal dialogue. In Proc. Eurospeech'01 (2001), pp. 1367--1370.Google Scholar
- Oviatt, S.L. Multimodal interfaces for dynamic interactive maps. In Proc. CHI '96 (1996), pp. 95--102. Google ScholarDigital Library
- Oviatt, S. L. Ten myths of multimodal interaction. Communications of the ACM 42, (1999), pp. 74--81. Google ScholarDigital Library
- Oviatt, S., Coulston, R., Tomko, S., Xiao, B., Lunsford, R., Wesson, M., Carmichael, L. Toward a theory of organized multimodal integration patterns during Human-Computer Interaction. In Proc. ICMI'03, ACM Press (2003), pp. 44--51. Google ScholarDigital Library
- Oviatt, S. L., DeAngeli, A. & Kuhn, K. Integration and synchronization of input modes during multimodal human-computer interaction. In Proc. CHI '97 (1997), pp. 415--422. Google ScholarDigital Library
- Reeves, B., Nass, C. The Media Equation. Cambridge University Press, New York, 1996.Google ScholarDigital Library
- Xiao, B., Girand, C., Oviatt, S.L. Multimodal integration patterns in children. In Proc. ICSLP'02, Casual Prod. Ltd (2002), pp. 629--632.Google Scholar
Index Terms
Children's and adults' multimodal interaction with 2D conversational agents
Recommendations
Conversational Interaction with Multiple Agents Initiated via Proxemics and Gaze
HAI '18: Proceedings of the 6th International Conference on Human-Agent InteractionThis paper explores the use of proxemics and gaze in multimodal multiparty interaction in virtual reality (VR). While many applications with sophisticated implementations have been developed, we sought to create, using more lightweight tools, a ...
Embodied conversational agents: computing and rendering realistic gaze patterns
PCM'06: Proceedings of the 7th Pacific Rim conference on Advances in Multimedia Information ProcessingWe describe here our efforts for modeling multimodal signals exchanged by interlocutors when interacting face-to-face. This data is then used to control embodied conversational agents able to engage into a realistic face-to-face interaction with human ...
Multimodal and mobile conversational Health and Fitness Companions
Multimodal conversational spoken dialogues using physical and virtual agents provide a potential interface to motivate and support users in the domain of health and fitness. This paper describes how such multimodal conversational Companions can be ...
Comments