Modeling the impact of shared visual information on collaborative reference

ABSTRACT

A number of recent studies have demonstrated that groups benefit considerably from access to shared visual information. This is due, in part, to the communicative efficiencies provided by the shared visual context. However, a large gap exists between our current theoretical understanding and our existing models. We address this gap by developing a computational model that integrates linguistic cues with visual cues in a way that effectively models reference during tightly-coupled, task-oriented interactions. The results demonstrate that an integrated model significantly outperforms existing language-only and visual-only models. The findings can be used to inform and augment the development of conversational agents, applications that dynamically track discourse and collaborative interactions, and dialogue managers for natural language interfaces.

REFERENCES

• 1: James Allen , George Ferguson , Mary Swift , Amanda Stent , Scott Stoness , Lucian Galescu , Nathan Chambers , Ellen Campana , Gregory Aist, Two diverse systems built using generic components for spoken dialogue: (recent progress on TRIPS), Proceedings of the ACL 2005 on Interactive poster and demonstration sessions, p.85-88, June 25-30, 2005, Ann Arbor, Michigan [doi>10.3115/1225753.1225775]
• 2: Bangalore, S., and Johnston, M. (2004). Balancing data-driven and rule-based approaches in the context of a multimodal conversational system. In Proceedings of HLT-NAACL '04, 33--40.
• 3: Brennan, S.E. (1995). Centering attention in discourse. Language & Cognitive Processes, 10 (2), 137--167.
• 4: Brennan, S.E. (2005). How conversation is shaped by visual and spoken evidence. In Trueswell, J., and Tanenhaus, M. (Eds.) Approaches to studying world situated language use: Bridging the language-as-product and language-as-action traditions, pp. 95--130. MIT Press, Cambridge, MA.
• 5: Susan E. Brennan , Marilyn W. Friedman , Carl J. Pollard, A centering approach to pronouns, Proceedings of the 25th annual meeting on Association for Computational Linguistics, p.155-162, July 06-09, 1987, Stanford, California [doi>10.3115/981175.981197]
• 6: Byron, D.K., Mampilly, T., Sharma, V., and Xu, T. (2005). Utilizing visual attention for cross-modal coreference interpretation. In Proceedings of CONTEXT '05, 83--96.
• 7: Byron, D.K., and Stoia, L. (2005). An analysis of proximity markers in collaborative dialog. Appeared at 41st annual meeting of the Chicago Linguistic Society.
• 8: Cassell, J. (2004). Towards a Model of Technology and Literacy Development: Story Listening Systems. Journal of Applied Developmental Psychology, 25 (1), 75--105.
• 9: Justine Cassell , Matthew Stone , Hao Yan, Coordination and context-dependence in the generation of embodied conversation, Proceedings of the first international conference on Natural language generation, June 12-16, 2000, Mitzpe Ramon, Israel [doi>10.3115/1118253.1118277]
• 10: Joyce Y. Chai , Zahar Prasov , Joseph Blaim , Rong Jin, Linguistic theories in efficient multimodal reference resolution: an empirical investigation, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040850]
• 11: Chomsky, N. (1982). Some concepts and consequences of the theory of government and binding. MIT Press, Cambridge, MA.
• 12: Clark, H.H., and Brennan, S.E. (1991). Grounding in communication. In Resnick, L., Levine, J., and Teasley, S. (Eds.) Perspectives on socially shared cognition, pp. 127--149. APA, Washington DC.
• 13: Clark, H.H., and Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22, 1--39.
• 14: Daelemans, W., Buchholz, S., and Veenstra, J. (1999). Memory-based shallow parsing. In Proceedings of CoNLL Workshop, 53--60.
• 15: Daelemans, W., Zavrel, J., van der Sloot, K., and van den Bosch, A. (2001). TiMBL: Tilburg memory based learner, version 4.0, reference guide. Technical Report ILK Technical Report 00-01, Tilburg University.
• 16: David DeVault , Natalia Kariaeva , Anubha Kothari , Iris Oved , Matthew Stone, An information-state approach to collaborative reference, Proceedings of the ACL 2005 on Interactive poster and demonstration sessions, p.1-4, June 25-30, 2005, Ann Arbor, Michigan [doi>10.3115/1225753.1225754]
• 17: Pinar Dönmez , Carolyn Rosé , Karsten Stegmann , Armin Weinberger , Frank Fischer, Supporting CSCL with automatic corpus analysis technology, Proceedings of th 2005 conference on Computer support for collaborative learning: learning 2005: the next 10 years!, p.125-134, May 30-June 04, 2005, Taipei, Taiwan
• 18: Eisenstein, J., and Christoudias, C.M. (2004). A salience-based approach to gesture-speech alignment. In Proceedings of HLT-NAACL-04, 25--32.
• 19: Fussell, S.R., Setlock, L.D., Yang, J., Ou, J., Mauer, E.M., and Kramer, A. (2004). Gestures over video streams to support remote collaboration on physical tasks. Human-Computer Interaction, 19, 273--309.
• 20: Darren R. Gergle , Robert E. Kraut, The value of shared visual information for task-oriented collaboration, Carnegie Mellon University, Pittsburgh, PA, 2006
• 21: Darren Gergle , Robert E. Kraut , Susan R. Fussell, Action as language in a shared visual space, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031687]
• 22: Darren Gergle , Robert E. Kraut , Susan R. Fussell, The impact of delayed visual feedback on collaborative performance, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124968]
• 23: Barbara J. Grosz , Aravind K. Joshi , Scott Weinstein, Providing a unified account of definite noun phrases in discourse, Proceedings of the 21st annual meeting on Association for Computational Linguistics, June 15-17, 1983, Cambridge, Massachusetts [doi>10.3115/981311.981320]
• 24: Barbara J. Grosz , Scott Weinstein , Aravind K. Joshi, Centering: a framework for modeling the local coherence of discourse, Computational Linguistics, v.21 n.2, p.203-225, June 1995
• 25: Barbara J. Grosz , Candace L. Sidner, Attention, intentions, and the structure of discourse, Computational Linguistics, v.12 n.3, p.175-204, July-September 1986
• 26: Gundel, J.K., Hedberg, N., and Zacharski, R. (1993). Cognitive status and the form of referring expressions in discourse. Language, 69 (2), 274--307.
• 27: Gahgene Gweon , Carolyn Rose , Regan Carey , Zachary Zaiss, Providing support for adaptive scripting in an on-line collaborative learning environment, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124810]
• 28: Peter A. Heeman , James F. Allen, Dialogue Transcription Tools, University of Rochester, Rochester, NY, 1995
• 29: Hobbs, J.R. (1978). Resolving pronoun references. Lingua, 44, 311--338.
• 30: Hudson, S.B., Tanenhaus, M.K., and Dell, G.S. (1986). The effect of the discourse center on the local coherence of a discourse. In Proceedings of the Cognitive Science Society -- 86, 96--101. Lawrence Erlbaum Associates.
• 31: Carla Huls , Wim Claassen , Edwin Bos, Automatic referent resolution of deictic and anaphoric expressions, Computational Linguistics, v.21 n.1, p.59-79, March 1995
• 32: Andrew Kehler, Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.685-690, July 30-August 03, 2000
• 33: Kraut, R.E., Fussell, S.R., and Siegel, J. (2003). Visual information as a conversational resource in collaborative physical tasks. Human-Computer Interaction, 18(1), 13--49.
• 34: Robert E. Kraut , Darren Gergle , Susan R. Fussell, The use of visual information in shared visual spaces: informing the development of virtual co-presence, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587084]
• 35: Kumar, R., Rosé, C.P., Aleven, V., Iglesias, A., and Robinson, A. (2006). Evaluating the effectiveness of tutorial dialogue instruction in an exploratory learning context. In Proceedings of the Intelligent Tutoring Systems Conference.
• 36: Mitkov, R. (2000). Towards a more consistent and comprehensive evaluation of anaphora resolution algorithms and systems. In Proceedings of DAARC 2000, 96--107.
• 37: Müller, C. (2006). Automatic detection of non-referential it in spoken multi-party dialog. In Proceedings of EACL 2006, 49--56.
• 38: Bonnie A. Nardi , Heinrich Schwarz , Allan Kuchinsky , Robert Leichner , Steve Whittaker , Robert Sclabassi, Turning away from talking heads: the use of video-as-data in neurosurgery, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.327-334, May 1993, Amsterdam, The Netherlands
• 39: Jiazhi Ou , Lui Min Oh , Susan R. Fussell , Tal Blum , Jie Yang, Analyzing and predicting focus of attention in remote collaborative tasks, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088485]
• 40: Oviatt, S., Levow, G.-A., Moreton, E., and MacEachern, M. (1998). Modeling global and focal hyperarticulation during human-computer error resolution. Journal of the Acoustical Society of America, 104 (5), 3080--3091.
• 41: Sharon Oviatt , Margaret MacEachern , Gina-Anne Levow, Predicting hyperarticulate speech during human-computer error resolution, Speech Communication, v.24 n.2, p.87-110, May 1, 1998 [doi>10.1016/S0167-6393(98)00005-3 ]
• 42: Oviatt, S.L. (1997). Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 12, 93--129.
• 43: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 44: Abhishek Ranjan , Jeremy P. Birnholtz , Ravin Balakrishnan, An exploratory analysis of partner action and camera control in a video-mediated collaborative task, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada [doi>10.1145/1180875.1180936]
• 45: Scholl, B.J. (2001). Objects and attention: the state of the art. Cognition, 80, 1--46.
• 46: Stacey D. Scott , Sheelagh Carpendale, Guest Editors' Introduction: Interacting with Digital Tabletops, IEEE Computer Graphics and Applications, v.26 n.5, p.24-27, September 2006 [doi>10.1109/MCG.2006.107]
• 47: Michael Strube, Never look back: an alternative to centering, Proceedings of the 17th international conference on Computational linguistics, p.1251-1257, August 10-14, 1998, Montreal, Quebec, Canada
• 48: Michael Strube , Udo Hahn, Functional centering: grounding referential coherence in information structure, Computational Linguistics, v.25 n.3, p.309-344, Septmeber 1999
• 49: Joel R. Tetreault, A corpus-based evaluation of centering and pronoun resolution, Computational Linguistics, v.27 n.4, December 2001
• 50: Joel Tetreault , James Allen, Empirical evaluations of pronoun resolution, University of Rochester, Rochester, NY, 2005
• 51: Marilyn A. Walker, Evaluating discourse processing algorithms, Proceedings of the 27th annual meeting on Association for Computational Linguistics, p.251-261, June 26-29, 1989, Vancouver, British Columbia, Canada [doi>10.3115/981623.981654]
• 52: Whittaker, S. (2003). Things to talk about when talking about things. Human-Computer Interaction, 18 (1-2), 149--170.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Computer-supported cooperative work

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Collaborative computing

General Terms:
Human Factors

Keywords:
communication, discourse, language use, modeling, multimodal interaction, shared visual information