Attention driven visual processing for an interactive dialog robot

ABSTRACT

In this paper we propose an attention-based vision system for the JAST interactive dialog robot. The robotic vision system incorporates three submodules: object recognition, gesture recognition and self recognition.

The performance boost of our biologically inspired vision system is based on two assumptions: first, generally attention is attracted by regions of high intensity or hue gardients as well as scene dynamics (bottom-up attention attraction), and second, attentioninal focus can be directed by higher level modules, whether volitional or not, in an inhibitory or reinforcing way (top-down attention control). The system proposed in this paper is able to utilize these assumptions and organize its computational efforts accordingly.

Integrated into an efficient data management architecture, the vision system is capable of continuously publishing results to the cognitive layer of the robot and thus enables operations in realtime. Furthermore, the modular system structure and the asynchronous communication paradigm allows for efficient integration of additional modules, be it visual or any other sensory input data.

The main contribution of this work is the application of neuroscience findings and biologically plausible theories of attention based visual processing to a real-world robotic setup. Here, our experimental results show tremendous speed-ups using either the bottom-up attention attractors or the principle of top-down attention control as input data filters for further visual analysis, reaching the peak in a combination of the two.

REFERENCES

• 1: P. Reinagel and A. M. Zador, "Natural scene statistics at the center of gaze," Network: Computation in Neural Systems, vol. 10, no. 4, pp. 341--350, 1999.
• 2: Timor Kadir , Michael Brady, Saliency, Scale and Image Description, International Journal of Computer Vision, v.45 n.2, p.83-105, November 2001 [doi>10.1023/A:1012460413855]
• 3: S. Gilles, "Robust description and matching of images," Ph.D. dissertation, University of Oxford, 1998.
• 4: W. Kienzle, F. A. Wichmann, B. Schölkopf, and M. O. Franz, "A nonparametric approach to bottom-up visual saliency," in Proc. NIPS '06, 2006.
• 5: J. K. O'Regan and A. Noë, "A sensorimotor account of vision and visual consciousness," Behavioural and Brain Sciences, vol. 24, pp. 939--1031, 2001.
• 6: M. Posner and Y. Cohen, "Components of visual orienting," Attention and Performance, vol. 10, pp. 531--556, 1984.
• 7: L. Itti and C. Koch, "Computational modelling of visual attention," Nature Reviews Neuroscience, vol. 2, pp. 194--203, March 2001.
• 8: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 9: Dirk Walther , Laurent Itti , Maximilian Riesenhuber , Tomaso Poggio , Christof Koch, Attentional Selection for Object Recognition – A Gentle Way, Proceedings of the Second International Workshop on Biologically Motivated Computer Vision, p.472-479, November 22-24, 2002
• 10: Mary Ellen Foster , Tomas By , Markus Rickert , Alois Knoll, Human-Robot dialogue for joint construction tasks, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181009]
• 11: M. Rickert, M. E. Foster, M. Giuliani, T. By, G. Panin, and A. Knoll, "Integrating language, vision, and action for human robot dialog systems," in Proc. ICMI, 2007.
• 12: M. Giuliani and A. Knoll, "Integrating multimodal cues using grammar based models," in Proc. ICMI, 2007.
• 13: David E. Culler , Anoop Gupta , Jaswinder Pal Singh, Parallel Computer Architecture: A Hardware/Software Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 14: Håkan Sundell , Philippas Tsigas, Fast and Lock-Free Concurrent Priority Queues for Multi-Thread Systems, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, p.84.2, April 22-26, 2003
• 15: T. Müller, P. Ziaie, and A. Knoll, "A wait-free realtime system for optimal distribution of vision tasks on multicore architectures," in Proc. ICINCO '08, 2008.
• 16: Maurice Herlihy, Wait-free synchronization, ACM Transactions on Programming Languages and Systems (TOPLAS), v.13 n.1, p.124-149, Jan. 1991 [doi>10.1145/114005.102808]
• 17: P. Ziaie, T. Müller, M. E. Foster, and A. Knoll, "A naïve bayes classifier with distance weighting for hand-gesture recognition," in Proc. Intl. CSI Computer Conference '08, 2008.
• 18: Yiqun Hu , Deepu Rajan , Liang-Tien Chia, Detection of visual attention regions in images using robust subspace analysis, Journal of Visual Communication and Image Representation, v.19 n.3, p.199-216, April, 2008 [doi>10.1016/j.jvcir.2007.11.001]
• 19: Greg Pass , Ramin Zabih, Comparing images using joint histograms, Multimedia Systems, v.7 n.3, p.234-240, May 1999 [doi>10.1007/s005300050125]
• 20: Thomas Müller , Claus Lenz , Simon Barner , Alois Knoll, Accelerating Integral Histograms Using an Adaptive Approach, Proceedings of the 3rd international conference on Image and Signal Processing, July 01-03, 2008, Cherbourg-Octeville, France [doi>10.1007/978-3-540-69905-7_24]
• 21: W. Li, V. Piëch, and C. D. Gilbert, "Perceptual learning and top-down influences in primary visual cortex," Nature Neuroscience, vol. 7, pp. 651--657, May 2004.
• 22: M. I. Posner, R. D. Rafal, L. S. Choate, and J. Vaughan, "Inhibition of return: Neural basis function," Cognitive Neuropsychology, vol. 2, no. 3, pp. 211--228, August 1985.
• 23: J. R. Stroop, "Studies of interference in serial verbal reactions," Journal of Experimental Pyschology, vol. 18, pp. 643--662, 1935.
• 24: J. D. Cohen, G. Aston-Jones, and M. S. Gilzenrat, "A systems-level perspective on attention and cognitive control: Guided activation, adaptive gating, conflict monitoring, and exploitation versus exploration," in Cognitive Neuroscience of Attention. Guilford Publications, 2004, pp. 71--90.

INDEX TERMS

Keywords:
active perception and vision, autonomous robots, bioinspired robot vision, computer vision, gesture, object and self recognition, realtime visual processing