A camera-based interface for interaction with mobile handheld computers

ABSTRACT

Recent advances in mobile computing allow the users to deal with 3D interactive graphics on handheld computers. Although the computing resources and screen resolutions grow steadily, user interfaces for handheld computers do not change significantly. Consequently, we designed a new 3-DOF interface adapted to the characteristics of handheld computers. This interface tracks the movement of a target that the user holds behind the screen by analyzing the video stream of the handheld computer camera. The position of the target is directly inferred from the color-codes that are printed on it using an efficient algorithm. The users can easily interact in real-time in a mobile setting. The visualization of the data is good as the target does not occlude the screen and the interaction techniques are not dependent on the orientation of the handheld computer. We used the interface in several test applications for the visualization of large images such as maps, the manipulation of 3D models, and the navigation in 3D scenes. This new interface favors the development of 2D and 3D interactive applications on handheld computers.

REFERENCES

• 1: Ravin Balakrishnan , Ken Hinckley, The role of kinesthetic reference frames in two-handed input performance, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.171-178, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322599]
• 2: Benjamin B. Bederson , James D. Hollan, Pad++: a zooming graphical interface for exploring alternate interface physics, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.17-26, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192435]
• 3: W. Buxton , B. Myers, A study in two-handed input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.321-326, April 13-17, 1986, Boston, Massachusetts, United States
• 4: Dusan, S., Gadbois, G., and Flanagan, J. 2003. Multi-modal interaction on pda's integrating speech and pen inputs. In Proceedings of EUROSPEECH, 2225--2228.
• 5: Encarnao, L., Bimber, O., Schmalstieg, D., and Chandler, S. 1999. A translucent sketchpad for the virtual table exploring motion-based gesture recognition. In Proceedings of Eurographics '99, 179--190.
• 6: Daniel Fällmana , Andreas Lund , Mikael Wiberg, ScrollPad: Tangible Scrolling with Mobile Devices, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 9, p.90286.3, January 05-08, 2004
• 7: George W. Fitzmaurice , Shumin Zhai , Mark H. Chignell, Virtual reality for palmtop computers, ACM Transactions on Information Systems (TOIS), v.11 n.3, p.197-218, July 1993 [doi>10.1145/159161.159160]
• 8: George W. Fitzmaurice, Situated information spaces and spatially aware palmtop computers, Communications of the ACM, v.36 n.7, p.39-49, July 1993 [doi>10.1145/159544.159566]
• 9: Gong, J., and Tarasewich, P. 2004. Guidelines for handheld device interface design. In Proceedings of DSI 2004 (To appear).
• 10: Ken Hinckley , Mike Sinclair , Erik Hanson , Richard Szeliski , Matt Conway, The VideoMouse: a camera-based multi-degree-of-freedom input device, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.103-112, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322591]
• 11: Ken Hinckley , Jeff Pierce , Mike Sinclair , Eric Horvitz, Sensing techniques for mobile interaction, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.91-100, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354417]
• 12: Takeo Igarashi , Ken Hinckley, Speed-dependent automatic zooming for browsing large documents, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.139-148, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354435]
• 13: Kjedskov, J., and Skov, M. B. 2002. Interaction design for handheld computers. In Proceedings of the 5th Asian Pacific Conference on Human-Computer Interaction, APCHI 2002, Science Press.
• 14: David M. Krum , Rob Melby , William Ribarsky , Larry Hodges, Isometric pointer interfaces for wearable 3D visualization, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765983]
• 15: Mark Raymond Mine, Exploiting proprioception in virtual-environment interaction, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1998
• 16: Pierce, J., Mahaney, H., and Abowd, G. 2003. Opportunistic annexing for handheld devices: Opportunities and challenges. Tech. rep., Georgia Institute of Technology GIT-GVU-03-31.
• 17: Pouderoux, J., Gonzato, J.-C., Guitton, P., and Granier, X., 2004. A software for reconstructing 3d-terrains from scanned maps. ACM SIGGRAPH 2004 Sketches and Applications, http://iparla.labri.fr/softwares/automnt/.
• 18: Rantakokko, T., and Plomp, J. 2003. An adaptive map-based interface for situated services. In Proceedings of Smart Object Conference.
• 19: Jun Rekimoto , Katashi Nagao, The world through the computer: computer augmented interaction with real world environments, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.29-36, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215639]
• 20: Jun Rekimoto, Tilting operations for small screen interfaces, Proceedings of the 9th annual ACM symposium on User interface software and technology, p.167-168, November 06-08, 1996, Seattle, Washington, United States [doi>10.1145/237091.237115]
• 21: Rohs, M. 2004. Real-world interaction with camera-phones. In 2nd International Symposium on Ubiquitous Computing Systems (UCS 2004).
• 22: Ben Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997
• 23: Daniel Wagner , Dieter Schmalstieg, First Steps Towards Handheld Augmented Reality, Proceedings of the 7th IEEE International Symposium on Wearable Computers, p.127, October 21-23, 2003
• 24: Eric Woods , Paul Mason , Mark Billinghurst, MagicMouse: an inexpensive 6-degree-of-freedom mouse, Proceedings of the 1st international conference on Computer graphics and interactive techniques in Australasia and South East Asia, February 11-14, 2003, Melbourne, Australia [doi>10.1145/604471.604539]
• 25: Ka-Ping Yee, Peephole displays: pen interaction on spatially aware handheld computers, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642613]

INDEX TERMS

Primary Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices
          Subjects: Channels and controllers

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Keywords:
handheld computer, interaction, user interface, video tracking, visualization