A 3D interactive kiosk system

ABSTRACT

One of the long-term goals in human-computer interaction is to utilize more intuitive and natural methods such as speech and hand gesture that a user would employ for communication. In this paper, we present a multi-modal 3D interaction mechanism, in which user can interact with a 3D model of a tourist location displayed on the kiosk screen from a one meter distance by means of gestures and voice commands without wearing any special device in a public place, with a complex and non-static background environment. The system can be used in many applications such entertainment, touring, education, museum displays, and advertising.

REFERENCES

• 1: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, Proceedings of the 7th annual conference on Computer graphics and interactive techniques, p.262-270, July 14-18, 1980, Seattle, Washington, United States
• 2: Gary R. Bradski, Real Time Face and Object Tracking as a Component of a Perceptual User Interface, Proceedings of the 4th IEEE Workshop on Applications of Computer Vision (WACV'98), p.214, October 19-21, 1998
• 3: Marcio C. Cabral , Carlos H. Morimoto , Marcelo K. Zuffo, On the usability of gesture interfaces in virtual reality environments, Proceedings of the 2005 Latin American conference on Human-computer interaction, p.100-108, October 23-26, 2005, Cuernavaca, Mexico [doi>10.1145/1111360.1111370]
• 4: GOMONKEY. 2006. http://www.gomonkey.at/. IBM Corporation 1999 ViaVoice Technology. http://www-306.ibm.com/software/voice/viavoice/.
• 5: Lim, B. P., Li, H., Chen, Y., 2004. Language identification through large vocabulary continuous speech recognition, In proceedings of International Symposium on Chinese Spoken Language Processing 2004, 49--52.
• 6: Kawato, S., and Ohya, J. 2000. Automatic skin-color distribution extraction for face detection and tracking. In Proceedings of 5th International Conference on Signal Processing 2000. WCCC-ICSP., 2, 1415--1418.
• 7: A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.349, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167020]
• 8: David M. Krum , Olugbenga Omoteso , William Ribarsky , Thad Starner , Larry F. Hodges, Speech and gesture multimodal control of a whole Earth 3D visualization environment, Proceedings of the symposium on Data Visualisation 2002, May 27-29, 2002, Barcelona, Spain
• 9: Marc Erich Latoschik, A user interface framework for multimodal VR interactions, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088479]
• 10: Manders, C., Aimone, C. and Mann, S. 2004. Camera response function recovery from different illuminations of identical subject matter, In Proceedings of 2004 International Conference on Image Processing, 2004. ICIP '04., 5, 24--27, 2965--2968
• 11: Manders, C., and Mann, S. 2005. Digital Camera Sensor Noise Estimation from Different Illuminations of Identical Subject Matter," In Proceedings of the Fifth International Conference on Information, Communications and Signal Processing 2005, 1292--1296.
• 12: Moeslund, T. B., Storring, M., and Granum, E. 2000 Vision-based user interface for interacting with a virtual environment. In Proc. of Danish Conference on Pattern Recognition and Image Analysis, DANKOMB 2000.
• 13: Konstantinos Moustakas , Dimitrios Tzovaras , Michael G. Strintzis , Sebastien Carbini , Olivier Bernier , Jean Emmanuel Viallet , Stephan Raidt , Matei Mancas , Mariella Dimiccoli , Enver Yagci , Serdar Balci , Eloisa Ibanez Leon, Masterpiece: Physical Interaction and 3D Content-Based Search in VR Applications, IEEE MultiMedia, v.13 n.3, p.92-100, July 2006 [doi>10.1109/MMUL.2006.65]
• 14: Kai Nickel , Rainer Stiefelhagen, Pointing gesture recognition based on 3D-tracking of face, hands and head orientation, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958460]
• 15: Matheen Siddiqui , Gérard Medioni, Robust real-time upper body limb detection and tracking, Proceedings of the 4th ACM international workshop on Video surveillance and sensor networks, October 27-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178782.1178792]
• 16: Maribeth Gandy , Thad Starner , Jake Auxier , Daniel Ashbrook, The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring, Proceedings of the 4th IEEE International Symposium on Wearable Computers, p.87, October 18-21, 2000
• 17: Damien Touraine , Patrick Bourdot , Yacine Bellik , Laurence Bolot, A framework to manage multimodal fusion of events for advanced interactions within virtual environments, Proceedings of the workshop on Virtual environments 2002, May 30-31, 2002, Barcelona, Spain
• 18: P. Kakumanu , S. Makrogiannis , N. Bourbakis, A survey of skin-color modeling and detection methods, Pattern Recognition, v.40 n.3, p.1106-1122, March, 2007 [doi>10.1016/j.patcog.2006.06.010]
• 19: Wilson, A., and Cutrell, E. 2005. FlowMouse: A computer vision-based pointing and gesture input device. In Proc. of 2005 Intl. Conf. on Human-computer Interaction, INTERACT '05, Sep. 2005, 565--578.
• 20: Wilson, A., and Oliver, N. 2003. GWindows: Towards robust perception-based UI. In Proc. of Computer Vision and Pattern Recognition, CVPR 2003.
• 21: Yao, Y., and Zhu, M. L. 2004. Hand tracking in time-varying illumination. In Proc. of 2004 Intl. Conf. on Machine Learning and Cybernetics, 4071--4075.
• 22: Zhu, Q., Cheng, K. T., Wu, C. T., and Wu Y. L. 2004. Adaptive learning of an accurate skin-color model," In Proceedings of the. Sixth IEEE International Conference on Automatic Face and Gesture Recognition 2004., 37--42.
• 23: Xiaojin Zhu , Jie Yang , Alex Waibel, Segmenting Hands of Arbitrary Color, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.446, March 26-30, 2000

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Keywords:
gesture tracking, multimodal human computer interaction, voice recognition