Article

Taking advantage of the situation: non-linguistic context for natural language interfaces to interactive virtual environments

Authors:

Michael Fleischman,

Eduard HovyAuthors Info & Claims

IUI '06: Proceedings of the 11th international conference on Intelligent user interfaces

Pages 47 - 54

https://doi.org/10.1145/1111449.1111467

Published: 29 January 2006 Publication History

Abstract

We introduce a framework for learning situated Natural Language Interfaces (NLIs) to interactive virtual environments. The framework exploits the non-linguistic context, or situation, explicitly modeled in such interactive applications. This situation model is integrated with a model of word meaning in a principled manner using a noisy channel approach to language understanding. Preliminary experimentation in an independently designed interactive application, i.e. the Mission Rehearsal Exercise (MRE), shows that this situated NLI outperforms a state of the art NLI on both whole frame accuracy and F-Score metrics. Further, use of the situation model in the situated NLI is shown to increase robustness to the noise introduced by the use of automatic speech recognition.

References

[1]

Bhagat, R., Leuski, A., and Hovy, E. Statistical Shallow Semantic Parsing despite Little Training Data. In prep.]]

[2]

Fillmore, C. Frame Semantics and the Nature of Language. Annals of the New York Academy of Science Conference on the Origin and Development of Language and Speech, Volume 280 (pp. 20--32), 1976.]]

[3]

Fleischman, M., Kwon, N., and Hovy, E. 2003. Maximum Entropy Models for FrameNet Classification. EMNLP, Sapporo, Japan, 2003.]]

Digital Library

[4]

Fleischman, M. B. and Roy, D. Why Verbs are Harder to Learn than Nouns: Initial Insights from a Computational Model of Intention Recognition in Situated Word Learning. 27th Annual Meeting of the Cognitive Science Society, Stresa, Italy. July 2005.]]

[5]

Fleischman, M. B. and Roy, D. Intentional Context in Situated Language Learning. Ninth Conference on Computational Natural Language Learning, Ann Arbor, MI. June 2005.]]

Digital Library

[6]

Ge, R. and Mooney, R.J. A Statistical Semantic Parser that Integrates Syntax and Semantics Proceedings of the Ninth Conference on Computational Natural Language Learning, Ann Arbor, MI, pp. 9--16, June 2005.]]

Digital Library

[7]

Gildea, D. and Jurafsky, D. Automatic Labeling of Semantic Roles. Computational Linguistics, 28(3) 245--288 14, 2002.]]

Digital Library

[8]

Gorniak, P. and Roy, D. Probabilistic Grounding of Situated Speech using Plan Recognition and Reference Resolution. ICMI, 2005.]]

Digital Library

[9]

Moore, R. Improving IBM Word Alignment Model 1. in Proc. of 42nd ACL, Barcelona, Spain., 2004.]]

Digital Library

[10]

Pynadath, D. Probabilistic Grammars for Plan Recognition. Ph.D. Thesis, University of Michigan, 1999.]]

Digital Library

[11]

Reiter, E. and Roy. D. (in press). Connecting Language to the World. Special issue of Artificial Intelligence.]]

[12]

Roy, D. "Grounding Words in Perception and Action: Insights from Computational Models". Trends in Cognitive Science, 2005.]]

[13]

Traum, D., Rickel, J., Gratch, J.and Marsella, S. "Negotiation over Tasks in Hybrid Human-Agent Teams for Simulation-Based Training," in 2nd International Conference on Autonomous Agents and Multiagent Systems, Melbourne, Australia, July 2003.]]

Digital Library

[14]

Swartout, W., Gratch, J., Hill, R.W. Jr., Hovy, E., Lindheim, R., Marsella, S., Rickel, J., Traum, D.R. Simulation in Hollywood: Integrating Graphics, Sound, Story and Character for Immersive Simulation. Multimodal Intelligent Information Presentation (Oliviero Stock and Massimo Zancanaro Eds), 2005.]]

[15]

Zettlemoyer, L. and Collins, M. Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars. In proceedings of UAI 2005.]]

[16]

Brown, P. F. Della Pietra, V. J. Della Pietra S. A. & Mercer., R. L. The Mathematics of Statistical Machine Translation: Parameter Estimation, Computational Linguistics 19(2), 1993.]]

Digital Library

[17]

Miller, S., R. Bobrow, R. Ingria, and R. Schwartz. 1994. Hidden Understanding Models of Natural Language, Proceedings of the Association of Computational Linguistics (ACL) conference, pp. 25--32.]]

Digital Library

[18]

Horvitz, E., Breese, J., Heckerman, D., Hovel, D, and Rommelse., K. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users. In Proc. of the 14th Conference on UAI, 1998.]]

[19]

DE Appelt, ME Pollack. Weighted abduction for plan ascription In Proceedings of User Modelling and User-Adapted Interaction, 1992.]]

[20]

Carberry, S. Plan Recognition in Natural Language Dialogue. MIT Press, Cambridge MA. 1990.]]

Digital Library

Cited By

Seon CLee HKim HSeo J(2014)Improving domain action classification in goal-oriented dialogues using a mutual retraining methodPattern Recognition Letters10.1016/j.patrec.2014.03.02145(154-160)Online publication date: Aug-2014
https://doi.org/10.1016/j.patrec.2014.03.021
Orkin JRoy D(2011)Semi-Automated Dialogue Act Classification for Situated Social Agents in GamesAgents for Games and Simulations II10.1007/978-3-642-18181-8_11(148-162)Online publication date: 2011
https://doi.org/10.1007/978-3-642-18181-8_11
Orkin JSmith TReckman HRoy DJhala ARiedl MRoberts D(2010)Semi-automatic task recognition for interactive narratives with EAT & RUNProceedings of the Intelligent Narrative Technologies III Workshop10.1145/1822309.1822312(1-8)Online publication date: 18-Jun-2010
https://dl.acm.org/doi/10.1145/1822309.1822312
Show More Cited By

Index Terms

Taking advantage of the situation: non-linguistic context for natural language interfaces to interactive virtual environments
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Knowledge Extraction from Task Narratives
iWOAR '17: Proceedings of the 4th International Workshop on Sensor-based Activity Recognition and Interaction

One of the major difficulties in activity recognition stems from the lack of a model of the world where activities and events are to be recognised. When the domain is fixed and repetitive we can manually include this information using some kind of ...
Taking advantage of contextualized interactions while users watch TV

While watching TV, viewers use the remote control to turn the TV set on and off, change channel and volume, to adjust the image and audio settings, etc. Worldwide, research institutes collect information about audience measurement, which can also be ...
Ontology-based situation awareness

The notions of ''situation'' and ''situation awareness'' have been formulated by many authors in various contexts. In this paper, we present a formalization of situations that is compatible with the interpretation of situation awareness in terms of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '06: Proceedings of the 11th international conference on Intelligent user interfaces

January 2006

392 pages

ISBN:1595932879

DOI:10.1145/1111449

General Chairs:
Ernest Edmonds
University of Technology Sydney, Australia
,
Doug Riecken
IBM Watson Research, USA
,
Program Chairs:
Cécile L. Paris
CSIRO ICT Centre, Australia
,
Candace L. Sidner
MERL, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 January 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

IUI06

Sponsor:

IUI06: 11th International Conference on Intelligent User Interfaces

January 29 - February 1, 2006

Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
534
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Seon CLee HKim HSeo J(2014)Improving domain action classification in goal-oriented dialogues using a mutual retraining methodPattern Recognition Letters10.1016/j.patrec.2014.03.02145(154-160)Online publication date: Aug-2014
https://doi.org/10.1016/j.patrec.2014.03.021
Orkin JRoy D(2011)Semi-Automated Dialogue Act Classification for Situated Social Agents in GamesAgents for Games and Simulations II10.1007/978-3-642-18181-8_11(148-162)Online publication date: 2011
https://doi.org/10.1007/978-3-642-18181-8_11
Orkin JSmith TReckman HRoy DJhala ARiedl MRoberts D(2010)Semi-automatic task recognition for interactive narratives with EAT & RUNProceedings of the Intelligent Narrative Technologies III Workshop10.1145/1822309.1822312(1-8)Online publication date: 18-Jun-2010
https://dl.acm.org/doi/10.1145/1822309.1822312
Leuski ALavrenko VYu PTsotras VFox ELiu B(2006)Tracking dragon-hunters with language modelsProceedings of the 15th ACM international conference on Information and knowledge management10.1145/1183614.1183714(698-707)Online publication date: 6-Nov-2006
https://dl.acm.org/doi/10.1145/1183614.1183714

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten