Object schemas for responsive robotic language use

ABSTRACT

The use of natural language should be added to a robot system without sacrificing responsiveness to the environment. In this paper, we present a robot that manipulates objects on a tabletop in response to verbal interaction. Reactivity is maintained by using concurrent interaction processes, such as visual trackers and collision detection processes. The interaction processes and their associated data are organized into object schemas, each representing a physical object in the environment, based on the target of each process. The object schemas then serve as discrete structures of coordination between reactivity, planning, and language use, permitting rapid integration of information from multiple sources.

REFERENCES

• 1: C. Bauckhage, Evaluating Integrated Speech- and Image Understanding, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.9, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166961]
• 2: P. Beeson, M. MacMahon, J. Modayil, A. Murarka, B. Kuipers, and B. Stankiewicz. Integrating multiple representations of spatial knowledge for mapping, navigation, and communication. In Proceedings of the AAAI 2007 Spring Symposium on Control Mechanisms for Spatial Knowledge Processing in Cognitive / Intelligent Systems, 2007.
• 3: S. Belongie , J. Malik , J. Puzicha, Shape Matching and Object Recognition Using Shape Contexts, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.4, p.509-522, April 2002 [doi>10.1109/34.993558]
• 4: M. Berlin, J. Gray, A. L. Thomaz, and C. Breazeal. Perspective taking: An organizing principle for learning in human-robot interaction. In Proceedings of AAAI 2006, 2006.
• 5: C. Breazeal, M. Berlin, A. Brooks, J. Gray, and A. L. Thomaz. Using perspective taking to learn from ambiguous demonstrations. Journal of Robotics and Autonomous Systems Special Issue on Robot Programming by Demonstration, 54(5), 2006.
• 6: R. A. Brooks. A robust layered control system for a mobile robot. IEEE Journal of Robotics and Automation, 2:14--23, 1986.
• 7: Rodney A. Brooks, Intelligence without representation, Artificial Intelligence, v.47 n.1-3, p.139-159, Jan. 1991 [doi>10.1016/0004-3702(91)90053-M]
• 8: J. Bruce, T. Balch, and M. Veloso. Fast and inexpensive color image segmentation for interactive robots. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2000.
• 9: Joanna Joy Bryson , Lynn Andrea Stein, Intelligence by design: principles of modularity and coordination for engineering complex adaptive agents, Massachusetts Institute of Technology, 2001
• 10: J. J. Bryson and L. A. Stein. Modularity and design in reactive intelligence. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1115--1120, August 2001.
• 11: R. Burke, D. Isla, M. Downie, Y. Ivanov, and B. Blumberg. Creature smarts: The art and architecture of a virtual brain. In Proceedings of the Game Developers Conference, pages 147--166, 2001.
• 12: A. Cangelosi. The grounding and sharing of symbols. Pragmatics and Cognition, 14:275--285, 2006.
• 13: Dorin Comaniciu , Peter Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.603-619, May 2002 [doi>10.1109/34.1000236]
• 14: Gary L. Drescher, Made-up minds: a constructivist approach to artificial intelligence, MIT Press, Cambridge, MA, 1991
• 15: Paul Michael Fitzpatrick , Rodney A. Brooks, From first contact to close encounters: a developmentally deep perceptual system for a humanoid robot, Massachusetts Institute of Technology, 2003
• 16: P. Fitzpatrick and G. Metta. Grounding vision through experimental manipulation. Philosophical Transactions of the Royal Society: Mathematical, Physical, and Engineering Sciences, 361(1811):2165--2185, 2003.
• 17: E. Gat. Integrating planning and reaction in a heterogeneous asynchronous architecture for controlling mobile robots. In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI), 1992.
• 18: Erann Gat, Three-layer architectures, Artificial intelligence and mobile robots: case studies of successful robot systems, MIT Press, Cambridge, MA, 1998
• 19: J. J. Gibson. The Ecological Approach to Visual Perception. Erlbaum, 1979.
• 20: P. Gorniak and D. Roy. Situated language understanding as filtering perceived affordances. Cognitive Science, 31(2):197--231, 2007.
• 21: K. Hsiao and D. Roy. A habit system for an interactive robot. In AAAI Fall Symposium: From Reactive to Anticipatory Cognitive Embodied Systems, 2005.
• 22: D. Isla, R. Burke, M. Downie, and B. Blumberg. A layered brain architecture for synthetic creatures. In Proceedings of IJCAI, 2001.
• 23: Luís Seabra Lopes , Jonathan H. Connell, Guest Editors' Introduction: Semisentient Robots-- Routes to Integrated Intelligence, IEEE Intelligent Systems, v.16 n.5, p.10-14, September 2001 [doi>10.1109/5254.956076]
• 24: J. Modayil and B. Kuipers. Where do actions come from? Autonomous robot learning of objects and actions. In Proceedings of the AAAI 2007 Spring Symposium on Control Mechanisms for Spatial Knowledge Processing in Cognitive / Intelligent Systems, 2007.
• 25: J. Piaget. The Construction of Reality in the Child. Basic Books, 1955.
• 26: T. Regier and L. Carlson. Grounding spatial language in perception: An empirical and computational investigation. Journal of Experimental Psychology, 130(2):273--298, 2001.
• 27: D. Roy. Grounding words in perception and action: computational insights. Trends in Cognitive Science, 9(8):389--396, 2005.
• 28: Deb Roy, Semiotic schemas: a framework for grounding language in action and perception, Artificial Intelligence, v.167 n.1-2, p.170-205, September 2005 [doi>10.1016/j.artint.2005.04.007]
• 29: D. Roy, K. Hsiao, and N. Mavridis. Mental imagery for a conversational robot. IEEE Transactions on Systems, Man, and Cybernetics, 34:1374--1383, 2004.
• 30: J. M. Siskind. Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. Journal of Artificial Intelligence Research, 15:31--90, August 2001.
• 31: M. Skubic, D. Perzanowski, S. Blisard, A. Schultz, W. Adams, M. Bugajska, and D. Brock. Spatial language for human-robot dialogs. IEEE Transactions on SMC Part C, Special Issue on Human-Robot Interaction, 34(2):154--167, May 2004.
• 32: L. Steels and T. Belpaeme. Coordinating perceptually grounded categories through language: A case study for colour. Behavioral and Brain Sciences, 28:469--529, 2005.
• 33: A. Stoytchev. Behavior-grounded representation of tool affordances. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2005.
• 34: P. Vogt and F. Divina. Social symbol grounding and language evolution. Interaction Studies, 8(1):31--52, 2007.
• 35: Terry Winograd, Understanding Natural Language, Academic Press, Inc., Orlando, FL, 1972

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.9 Robotics

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding

General Terms:
Algorithms, Design

Keywords:
affordances, behavior-based, language grounding, object schema, robot