On the usability of gesture interfaces in virtual reality environments

ABSTRACT

This paper discusses several usability issues related to the use of gestures as an input mode in multimodal interfaces. The use of gestures has been suggested before as a natural solution for applications that require hands-free and notouch interaction with computers, such as in virtual reality (VR) environments. We introduce a simple but robust 2D computer vision based gesture recognition system that was successfully used for interaction in VR environments such as CAVEs and Powerwalls. This interface was tested under 3 different scenarios, as a regular pointing device in a GUI interface, as a navigation tool, and as a visualization tool. Our experiments show that the time to completion of simple pointing tasks is considerably slower when compared to a mouse and that its use during even short periods of time causes fatigue. Despite, these drawbacks, the use of gestures as an alternative mode in multimodal interfaces offers several advantages, such as quick access to computing resources that might be embedded in the environment, using a natural and intuitive way, and that scales nicely to group and collaborative applications, where gestures can be used sporadically.

REFERENCES

• 1: M. C. Cabral. Interface baseada em gestos para ambientes de realidade virtual usando visão computacional. Master's thesis, Departamento de Ciência da Computação, Universidade de São Paulo, 2005.
• 2: J. Cassell. A framework for gesture generation and interpretation. In R. Cipolla and A. Pentland, editors, Computer Vision in Human-Machine Interaction, New York: Cambridge University Press, 1998.
• 3: G. Cheung, S. Baker, and T. Kanade. Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. CVPR, 2003.
• 4: Carolina Cruz-Neira , Daniel J. Sandin , Thomas A. DeFanti, Surround-screen projection-based virtual reality: the design and implementation of the CAVE, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.135-142, September 1993 [doi>10.1145/166117.166134]
• 5: P. M. Fitts. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology: Human Perception and Performance, 47, 1954.
• 6: W. T. Freeman and C. D. Weissman. Television control by hand gestures. In Proc. of the Int. Workshop on Automatic Face and Gesture Recognition, Zurich, Switzerland, June 1995.
• 7: Pengyu Hong , Thomas S. Huang , Matthew Turk, Gesture Modeling and Recognition Using Finite State Machines, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.410, March 26-30, 2000
• 8: C. Hummels , P. J. Stappers, Meaningful Gestures for Human Computer Interaction: Beyond Hand Postures, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.591, April 14-16, 1998
• 9: Point Grey Research Inc. Dragon fly firewire camera specifications - http://www.ptgrey.com/products/dragonfly/specifications.html.
• 10: Simeon Keates , Peter Robinson, The use of gestures in multimodal input, Proceedings of the third international ACM conference on Assistive technologies, p.35-42, April 15-17, 1998, Marina del Rey, California, United States [doi>10.1145/274497.274505]
• 11: R. Kehl and L. V. Gool. Real-time pointing gesture recognition for an immersive environment. IEEE Intl. Conference on Automatic Face and Gesture Recognition, 2004.
• 12: Frederik C. M. Kjeldsen, Visual interpretation of hand gestures as a practical interface modality, Columbia University, New York, NY, 1997
• 13: D. Lemmerman, A. Forsberg, L. Monroe, K. Chartrand, B. Greene, D. Modl, and J. Olivares. Towards human-centered user interfaces in an immersive context. Proc. IEEE VR 2004 - Beyond Wand and Glove Based Interaction, 2004.
• 14: C. Leubner, C. Brockmann, and H. Muller. Computer-vision-based human-computer interaction with a back projection wall using arm gestures. Euromicro Conference, 2001.
• 15: I. Scott MacKenzie , Abigail Sellen , William A. S. Buxton, A comparison of input devices in element pointing and dragging tasks, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.161-166, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108868]
• 16: Ivana Mikić , Mohan Trivedi , Edward Hunter , Pamela Cosman, Human Body Model Acquisition and Tracking Using Voxel Data, International Journal of Computer Vision, v.53 n.3, p.199-223, July-August 2003 [doi>10.1023/A:1023012723347]
• 17: Thomas B. Moeslund , Moritz Störring , Erik Granum, A Natural Interface to a Virtual Environment through Computer Vision-Estimated Pointing Gestures, Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction, p.59-63, April 18-20, 2001
• 18: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 19: M. Nielsen, T. Moeslund, M. Störring, and E. Granum. A procedure for developing intuitive and ergonomic gesture interfaces for human computer interaction. In Proc. of the 5th Internation Gesture Workshop, GW 2003, Genova, Italy, April 2003.
• 20: T. Starner and A. Pentland. Visual recognition of american sigh language using hidden markov models. IEEE Int. Symp. on Computer Vision, 1995.
• 21: N. Streitz, P. Tandler, C. Mller-Tomfelde, and S. Konomi. Roomware: Towards the next generation of human-computer interaction based on an integrated design of real and virtual worlds. In J. Carroll, editor, Human-Computer Interaction in the New Millenium. Addison-Wesley, 2001.
• 22: Christopher Richard Wren , Ali Azarbayejani , Trevor Darrell , Alex Paul Pentland, Pfinder: Real-Time Tracking of the Human Body, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.780-785, July 1997 [doi>10.1109/34.598236]
• 23: J. Zuffo and M. K. Zuffo. Caverna digital - sistema de multiprojeção estereoscopico baseado em aglomerados de pcs para aplicações imersivas em realidade virtual. In 4th SBC Simposio de Realidade Virtual. Sociedade Brasileira de Computação, Outubro 2001.

INDEX TERMS

Keywords:
computer vision, usability of gesture interfaces, virtual reality