From conversational tooltips to grounded discourse: head poseTracking in interactive dialog systems

ABSTRACT

Head pose and gesture offer several key conversational grounding cues and are used extensively in face-to-face interaction among people. While the machine interpretation of these cues has previously been limited to output modalities, recent advances in face-pose tracking allow for systems which are robust and accurate enough to sense natural grounding gestures. We present the design of a module that detects these cues and show examples of its integration in three different conversational agents with varying degrees of discourse model complexity. Using a scripted discourse model and off-the-shelf animation and speech-recognition components, we demonstrate the use of this module in a novel "conversational tooltip" task, where additional information is spontaneously provided by an animated character when users attendto various physical objects or characters in the environment. We further describe the integration of our module in two systems where animated and robotic characters interact with users based on rich discourse and semantic models.

REFERENCES

• 1: AT&T. Natural Voices. http://www.naturalvoices.att.com.
• 2: S. Basu , I. Essa , A. Pentland, Motion Regularization for Model-Based Head Tracking, Proceedings of the International Conference on Pattern Recognition (ICPR '96) Volume III-Volume 7276, p.611, August 25-29, 1996
• 3: M. J. Black , Y. Yacoob, Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion, Proceedings of the Fifth International Conference on Computer Vision, p.374, June 20-23, 1995
• 4: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 5: J. Cassell , T. Bickmore , M. Billinghurst , L. Campbell , K. Chang , H. Vilhjálmsson , H. Yan, Embodiment in conversational interfaces: Rea, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.520-527, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303150]
• 6: Timothy Bickmore , Justine Cassell, Relational agents: a model and implementation of building user trust, Proceedings of the SIGCHI conference on Human factors in computing systems, p.396-403, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365304]
• 7: Timothy F. Cootes , Gareth J. Edwards , Christopher J. Taylor, Active Appearance Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.681-685, June 2001 [doi>10.1109/34.927467 ]
• 8: V. Design. MEGA-D Megapixel Digital Stereo Head. http://www.ai.sri.com/ konolige/svs/, 2000.
• 9: Gregory D. Hager , Peter N. Belhumeur, Efficient Region Tracking With Parametric Models of Geometry and Illumination, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.10, p.1025-1039, October 1998 [doi>10.1109/34.722606 ]
• 10: Haptek. Haptek Player. http://www.haptek.com.
• 11: Ashish Kapoor , Rosalind W. Picard, A real-time head nod and shake detector, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971509]
• 12: Rick Kjeldsen, Head Gestures for Computer Control, Proceedings of the IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems (RATFG-RTS'01), p.61, July 13-August 13, 2001
• 13: Marco La Cascia , Stan Sclaroff , Vassilis Athitsos, Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.4, p.322-336, April 2000 [doi>10.1109/34.845375 ]
• 14: Christopher Lee , Neal Lesh , Candace L. Sidner , Louis-Philippe Morency , Ashish Kapoor , Trevor Darrell, Nodding in conversations with a robot, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.985935]
• 15: L.-P. Morency and T. Darrell. Stereo tracking using ICP and normal flow. In Proceedings International Conference on Pattern Recognition, 2002.
• 16: Louis-Philippe Morency , Ali Rahimi , Neal Checka , Trevor Darrell, Fast Stereo-Based Head Tracking for Interactive Environments, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, p.390, May 20-21, 2002
• 17: L.-P. Morency, A. Rahimi, and T. Darrell. Adaptive view-based appearance model. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, 2003.
• 18: K. Murphy. Bayes Net Toolbox for Matlab. http://www.ai.mit.edu/ murphyk/Software/BNT/bnt.html.
• 19: Yukiko I. Nakano , Gabe Reinstein , Tom Stocky , Justine Cassell, Towards a model of face-to-face grounding, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.553-561, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075166]
• 20: Nuance. Nuance. http://www.nuance.com.
• 21: J. Pierrehumbert. The phonology and phonetic of English intonation. Massachusetts Institute of Technology, 1980.
• 22: Charles Rich , Candace L. Sidner , Neal Lesh, Collagen: applying collaborative discourse theory to human-computer interaction, AI Magazine, v.22 n.4, p.15-25, Winter 2001
• 23: A. Schodl, A. Haro, and I. Essa. Head tracking using a textured polygonal model. In PUI98, 1998.
• 24: Candace L. Sidner , Cory D. Kidd , Christopher Lee , Neal Lesh, Where to look: a study of human-robot engagement, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964458]
• 25: C. Sidner, C. Lee, and N. Lesh. Engagement when looking: Behaviors for robots when collaborating with people. In Diabruck: Proceedings of the 7th workshop on the Semantic and Pragmatics of Dialogue, pages 123--130, University of Saarland, 2003. I. Kruiff-Korbayova and C. Kosny (eds.).
• 26: Michael Siracusa , Louis-Philippe Morency , Kevin Wilson , John Fisher , Trevor Darrell, A multi-modal approach for determining speaker location and focus, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958449]
• 27: Tracking Focus of Attention in Meetings, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.273, October 14-16, 2002
• 28: Yoshinao Takemae , Kazuhiro Otsuka , Naoki Mukawa, Impact of video editing based on participants' gaze in multiparty conversation, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986057]
• 29: P. Viola and M. Jones. Robust real-time face detection. In ICCV, page II: 747, 2001.
• 30: Laurenz Wiskott , Jean-Marc Fellous , Norbert Krüger , Christopher von der Malsburg, Face Recognition by Elastic Bunch Graph Matching, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.775-779, July 1997 [doi>10.1109/34.598235 ]
• 31: Christopher Richard Wren , Ali Azarbayejani , Trevor Darrell , Alex Paul Pentland, Pfinder: Real-Time Tracking of the Human Body, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.780-785, July 1997 [doi>10.1109/34.598236 ]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Motion

General Terms:
Algorithms

Keywords:
conversational tooltips, grounding, head gesture recognition, head pose tracking, human-computer interaction, interactive dialog system