A pattern mining method for interpretation of interaction

ABSTRACT

This paper proposes a novel mining method for multimodal interactions to extract important patterns of group activities. These extracted patterns can be used as machine-readable event indices in developing an interaction corpus based on a huge collection of human interaction data captured by various sensors. The event indices can be used, for example, to summarize a set of events and to search for particular events because they contain various pieces of context information. The proposed method extracts simultaneously occurring patterns of primitive events in interaction, such as gaze and speech, that in combination occur more consistently than randomly. The proposed method provides a statistically plausible definition of interaction events that is not possible through intuitive top-down definitions. We demonstrate the effectiveness of our method for the data captured in an experimental setup of a poster-exhibition scene. Several interesting patterns are extracted by the method, and we examined their interpretations.

REFERENCES

• 1: R. A. Brooks, The Intelligent Room project, Proceedings of the 2nd International Conference on Cognitive Technology (CT '97), p.271, August 25-28, 1997
• 2: Barry Brumitt , Brian Meyers , John Krumm , Amanda Kern , Steven A. Shafer, EasyLiving: Technologies for Intelligent Environments, Proceedings of the 2nd international symposium on Handheld and Ubiquitous Computing, p.12-29, September 25-27, 2000, Bristol, UK
• 3: Gemma Casas-Garriga. Discovering unbounded episodes in sequential data. In PKDD, pages 83--94, 2003.
• 4: Heikki Mannila , Hannu Toivonen , A. Inkeri Verkamo, Discovery of Frequent Episodes in Event Sequences, Data Mining and Knowledge Discovery, v.1 n.3, p.259-289, 1997 [doi>10.1023/A:1009748302351 ]
• 5: Cory D. Kidd , Robert Orr , Gregory D. Abowd , Christopher G. Atkeson , Irfan A. Essa , Blair MacIntyre , Elizabeth D. Mynatt , Thad Starner , Wendy Newstetter, The Aware Home: A Living Laboratory for Ubiquitous Computing Research, Proceedings of the Second International Workshop on Cooperative Buildings, Integrating Information, Organization, and Architecture, p.191-198, October 01-02, 1999
• 6: Hisashi Miyamori , Shu-ichi Iisaku, Video Annotation for Content-based Retrieval using Human Behavior Analysis and Domain Knowledge, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.320, March 26-30, 2000
• 7: Michihiro Kuramochi , George Karypis, Frequent Subgraph Discovery, Proceedings of the 2001 IEEE International Conference on Data Mining, p.313-320, November 29-December 02, 2001
• 8: Alex Pentland. Smart rooms. Scientific American, 274(4):68--76, 1996.
• 9: Yasuyuki Sumi, Sadanori Ito, Tetsuya Matsuguchi, Sidney Fels, and Kenji Mase. Collaborative capturing and interpretation of interactions. In Pervasive 2004 Workshop on Memory and Sharing of Experiences, pages 1--7, 2004.
• 10: Masashi Takahashi , Sadanori Ito , Yasuyuki Sumi , Megumu Tsuchikawa , Kiyoshi Kogure , Kenji Mase , Toyoaki Nishida, A layered interpretation of human interactions captured by ubiquitous sensors, Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences, October 15-15, 2004, New York, New York, USA [doi>10.1145/1026653.1026657]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
activity patterns, behavior mining, interaction corpus, multimodal interaction patterns