Individual differences in multimodal integration patterns: what are they and why do they exist?

ABSTRACT

Techniques for information fusion are at the heart of multimodal system design. To develop new user-adaptive approaches for multimodal fusion, the present research investigated the stability and underlying cause of major individual differences that have been documented between users in their multimodal integration pattern. Longitudinal data were collected from 25 adults as they interacted with a map system over six weeks. Analyses of 1,100 multimodal constructions revealed that everyone had a dominant integration pattern, either simultaneous or sequential, which was 95-96% consistent and remained stable over time. In addition, coherent behavioral and linguistic differences were identified between these two groups. Whereas performance speed was comparable, sequential integrators made only half as many errors and excelled during new or complex tasks. Sequential integrators also had more precise articulation (e.g., fewer disfluencies), although their speech rate was no slower. Finally, sequential integrators more often adopted terse and direct command-style language, with a smaller and less varied vocabulary, which appeared focused on achieving error-free communication. These distinct interaction patterns are interpreted as deriving from fundamental differences in reflective-impulsive cognitive style. Implications of these findings are discussed for the design of adaptive multimodal systems with substantially improved performance characteristics.

REFERENCES

• 1: Centre for Lexical Information, M.P.I.f.P., The CELEX lexical database. Linguistic Data Consortium.
• 2: Drake, D.M., Perceptual correlates of impulsive and reflective behavior. Dev. Psych., 1970. 2(2), 202--214.
• 3: Kagan, J. and Kogan, N., Individual variation in cognitive processes, Carmichael's manual of child psych., P. Mussen, Ed. 1970, Wiley, New York. 1273--1365.
• 4: Kagan, J., Rosman, B.L., Day, D., Albert, J., and Phillips, W., Information processing in the child: significance of analytic and reflective attitudes. Psychological Monographs, 1964. 78(1), 1--37.
• 5: Maccoby, E., Social Development: Psychological growth and the parent-child relationship. 1980, New York: Harcourt Brace Jovanovich.
• 6: Messer, S.B., Reflection-impulsivity: A review. Psychological Bulletin, 1976. 83(6), 1026--1052.
• 7: Mollon, J.D. and Perkins, A.J., Errors of judgement at Greenwich in 1796. Nature, 1996. 380, 101--102.
• 8: Oviatt, S., Coulston, R., and Lunsford, R. Just do what I tell you: The limited impact of instructions on multimodal integration patterns, in submission.
• 9: Oviatt, S.L., Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 1995. 9, 19--35.
• 10: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 11: Sharon Oviatt , Rachel Coulston , Stefanie Tomko , Benfang Xiao , Rebecca Lunsford , Matt Wesson , Lesley Carmichael, Toward a theory of organized multimodal integration patterns during human-computer interaction, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958443]
• 12: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 13: Stone, J.V., Hunkin, N.M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M., and Porter, N.R., When is now? Perception of simultaneity. Proc. Royal Society: Biological Sciences, 2001. 268, 31--38.
• 14: Xiao, B., Girand, C., and Oviatt, S.L. Multimodal integration patterns in children. Proc. ICSLP'02. Causal Productions, Ltd. 629--632.
• 15: Benfang Xiao , Rebecca Lunsford , Rachel Coulston , Matt Wesson , Sharon Oviatt, Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958480]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Voice I/O; Input devices and strategies (e.g., mouse, touchscreen); Prototyping; Evaluation/methodology; Natural language

Keywords:
commands, conversations, disfluencies, errors, impulsive-reflective cognitive style, individual differences, multimodal integration patterns, simultaneous or sequential input