Taking advantage of the situation: non-linguistic context for natural language interfaces to interactive virtual environments

ABSTRACT

We introduce a framework for learning situated Natural Language Interfaces (NLIs) to interactive virtual environments. The framework exploits the non-linguistic context, or situation, explicitly modeled in such interactive applications. This situation model is integrated with a model of word meaning in a principled manner using a noisy channel approach to language understanding. Preliminary experimentation in an independently designed interactive application, i.e. the Mission Rehearsal Exercise (MRE), shows that this situated NLI outperforms a state of the art NLI on both whole frame accuracy and F-Score metrics. Further, use of the situation model in the situated NLI is shown to increase robustness to the noise introduced by the use of automatic speech recognition.

REFERENCES

• 1: Bhagat, R., Leuski, A., and Hovy, E. Statistical Shallow Semantic Parsing despite Little Training Data. In prep.
• 2: Fillmore, C. Frame Semantics and the Nature of Language. Annals of the New York Academy of Science Conference on the Origin and Development of Language and Speech, Volume 280 (pp. 20--32), 1976.
• 3: Michael Fleischman , Namhee Kwon , Eduard Hovy, Maximum entropy models for FrameNet classification, Proceedings of the 2003 conference on Empirical methods in natural language processing, p.49-56, July 11, 2003 [doi>10.3115/1119355.1119362]
• 4: Fleischman, M. B. and Roy, D. Why Verbs are Harder to Learn than Nouns: Initial Insights from a Computational Model of Intention Recognition in Situated Word Learning. 27th Annual Meeting of the Cognitive Science Society, Stresa, Italy. July 2005.
• 5: Fleischman, M. B. and Roy, D. Intentional Context in Situated Language Learning. Ninth Conference on Computational Natural Language Learning, Ann Arbor, MI. June 2005.
• 6: Ge, R. and Mooney, R.J. A Statistical Semantic Parser that Integrates Syntax and Semantics Proceedings of the Ninth Conference on Computational Natural Language Learning, Ann Arbor, MI, pp. 9--16, June 2005.
• 7: Daniel Gildea , Daniel Jurafsky, Automatic labeling of semantic roles, Computational Linguistics, v.28 n.3, p.245-288, September 2002 [doi>10.1162/089120102760275983]
• 8: Peter Gorniak , Deb Roy, Probabilistic grounding of situated speech using plan recognition and reference resolution, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088489]
• 9: Moore, R. Improving IBM Word Alignment Model 1. in Proc. of 42nd ACL, Barcelona, Spain., 2004.
• 10: David Varghese Pynadath , Michael P. Wellman, Probabilistic grammars for plan recognition, 1999
• 11: Deb Roy , Ehud Reiter, Connecting language to the world, Artificial Intelligence, v.167 n.1-2, p.1-12, September 2005 [doi>10.1016/j.artint.2005.06.002]
• 12: Roy, D. "Grounding Words in Perception and Action: Insights from Computational Models". Trends in Cognitive Science, 2005.
• 13: David Traum , Jeff Rickel , Jonathan Gratch , Stacy Marsella, Negotiation over tasks in hybrid human-agent teams for simulation-based training, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860646]
• 14: Swartout, W., Gratch, J., Hill, R.W. Jr., Hovy, E., Lindheim, R., Marsella, S., Rickel, J., Traum, D.R. Simulation in Hollywood: Integrating Graphics, Sound, Story and Character for Immersive Simulation. Multimodal Intelligent Information Presentation (Oliviero Stock and Massimo Zancanaro Eds), 2005.
• 15: Zettlemoyer, L. and Collins, M.. Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars. In proceedings of UAI 2005.
• 16: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 17: Scott Miller , Robert Bobrow , Robert Ingria , Richard Schwartz, Hidden understanding models of natural language, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.25-32, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981736]
• 18: Horvitz, E., Breese, J., Heckerman, D., Hovel, D, and Rommelse., K. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users. In Proc. of the 14th Conference on UAI, 1998.
• 19: DE Appelt, ME Pollack. Weighted abduction for plan ascription In Proceedings of User Modelling and User-Adapted Interaction, 1992.
• 20: Sandra Carberry, Plan Recognition in Natural Language Dialogue, MIT Press, Cambridge, MA, 1990

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

General Terms:
Algorithms, Design, Performance

Keywords:
interactive virtual environments, mission rehearsal exercise, natural language interfaces/understanding, non-linguistic context, plan recognition, situated NLI, situation models