Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper

ABSTRACT

This paper presents a vision-based interface system, VISUAL PANEL, which employs an arbitrary quadrangle-shaped panel (e.g., an ordinary piece of paper) and a tip pointer (e.g., fingertip) as an intuitive, wireless and mobile input device. The system can accurately and reliably track the panel and the tip pointer. The panel tracking continuously determines the projective mapping between the panel at the current position and the display, which in turn maps the tip position to the corresponding position on the display. By detecting the clicking and dragging actions, the system can fulfill many tasks such as controlling a remote large display, and simulating a physical keyboard. Users can naturally use their fingers or other tip pointers to issue commands and type texts. Furthermore, by tracking the 3D position and orientation of the visual panel, the system can also provide 3D information, serving as a virtual joystick, to control 3D virtual objects.

REFERENCES

• 1: S. Ahmad. A usable real-time 3D hand tracker. In Proc. IEEE Asilomar Conf., 1994.
• 2: G. Berry. Small-wall: A multimodal human computer intelligent interaction test bed with applications. Master's thesis, Dept. of ECE, University of Illinois at Urbana-Champaign, 1998.
• 3: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 4: J. Crowley, F. Berard, and J. Coutaz. Finger tracking as an input device for augmented reality. In Proc. Int'l Workshop Automatic Face Gesture Recognition, pages 195--200, 1995.
• 5: Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, Cambridge, MA, 1993
• 6: D. Hall, C. Le Gal, J. Martin, O. Chomat, T. Kapuscinski, and J. Crowley. Magicboard: A contribution to an intelligent office environment. In Proc. Int'l Symposium on Intelligent Robotic Systems, University of Coimbra, Portugal, 1999.
• 7: K. Imagawa , S. Lu , S. Igi, Color-Based Hands Tracking System for Sign Language Recognition, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.462, April 14-16, 1998
• 8: Ramesh Jain , Rangachar Kasturi , Brian G. Schunck, Machine vision, McGraw-Hill, Inc., New York, NY, 1995
• 9: K. -H. Jo , Y. Kuno , Y. Shirai, Manipulative Hand Gesture Recognition Using Task Knowledge for Human Computer Interaction, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.468, April 14-16, 1998
• 10: Nebojsa Jojic , Thomas Huang , Barry Brumitt , Brian Meyers , Steve Harris, Detection and Estimation of Pointing Gestures in Dense Disparity Maps, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.468, March 26-30, 2000
• 11: Shanon X. Ju , Michael J. Black , Scott Minneman , Don Kimber, Analysis of Gesture and Action in Technical Talks for Video Indexing, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.595, June 17-19, 1997
• 12: Jennifer Mankoff , Gregory D. Abowd, Cirrin: a word-level unistroke keyboard for pen input, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.213-214, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288611]
• 13: Thomas P. Moran , Eric Saund , William Van Melle , Anuj U. Gujar , Kenneth P. Fishkin , Beverly L. Harrison, Design and technology for Collaborage: collaborative collages of information on physical walls, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.197-206, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322602]
• 14: Ken Perlin, Quikwriting: continuous stylus-based text entry, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.215-216, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288613]
• 15: Francis K. H. Quek, Unencumbered Gestural Interaction, IEEE MultiMedia, v.3 n.4, p.36-47, December 1996 [doi>10.1109/93.556459]
• 16: J. M. Rehg , T. Kanade, Model-based tracking of self-occluding articulated objects, Proceedings of the Fifth International Conference on Computer Vision, p.612, June 20-23, 1995
• 17: J. G. Semple and L. Roth. Introduction to Algebraic Geometry. Oxford: Clarendon Press, 1949. Reprinted 1987.
• 18: Maribeth Gandy , Thad Starner , Jake Auxier , Daniel Ashbrook, The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring, Proceedings of the 4th IEEE International Symposium on Wearable Computers, p.87, October 18-21, 2000
• 19: Thad Starner , Joshua Weaver , Alex Pentland, A Wearable Computer Based American Sign Language Recognizer, Proceedings of the 1st IEEE International Symposium on Wearable Computers, p.130, October 13-14, 1997
• 20: G. Strang. Introduction To Applied Mathematics. Wellesley-Cambridge Press, 1986.
• 21: N. Takao, J. Shi, S. Baker, I. Matthews, and B. Nabble. Tele-Graffiti: A pen and paper-based remote sketching system. In Proc. 8th Int'l Conf. Computer Vision, volume II, page 750, 2001.
• 22: J. Triesch , C. von der Malsburg, Robust classification of hand postures against complex backgrounds, Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG '96), p.170, October 14-16, 1996
• 23: Christian Vogler , Dimitris Metaxas, ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis, Proceedings of the Sixth International Conference on Computer Vision, p.363, January 04-07, 1998
• 24: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 25: C. Wren, A. Azarbayejani, T. Darrel, and A. Pentland. Pfinder: Real-time tracking of the human body. In Photonics East, SPIE Proceedings, volume 2615, 1995.
• 26: Y. Wu and T. S. Huang. Human hand modeling, analysis and animation in the context of HCI. In Proc. Int'l Conf. Image Processing, 1999.
• 27: M. Zeller , J. C. Phillips , A. Dalke , W. Humphrey , K. Schulten , T. S. Huang , V. I. Pavlovic , Y. Zhao , Z. Lo , S. Chu , R. Sharma, A Visual Computing Environment for Very Large Scale Biomolecular Modeling, Proceedings of the IEEE International Conference on Application-Specific Systems, Architectures and Processors, p.3, July 14-16, 1997
• 28: Zhengyou Zhang, A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.11, p.1330-1334, November 2000 [doi>10.1109/34.888718]

INDEX TERMS

Keywords:
new control device, new input device, plane projectivity, virtual joystick, virtual keyboard, virtual mouse, vision-based user interface, visual panel