Mobile visual interaction: enhancing local communication and collaboration with visual interactions

ABSTRACT

User interaction with applications and data is traditionally based on Menu driven, mouse or pen based user interfaces. As increasing number of mobile devices are equipped with cameras, mobile devices can be a key tool for local and remote visual interaction and communications. In this paper we discuss our Mobile Visual Interaction system that enables pointing with mobile camera devices on larger displays, and key findings in using mobile cameras for human computer interaction. The mobile pointing software can identify and interpret camera-detectable data elements inserted on data surfaces such as computer displays or projected screens. The user can interact with this system by pointing the mobile camera device to the surface and performing gestures. In this article we share the key challenges and key results found during developing the system.

REFERENCES

• 1: Rafael Ballagas , Michael Rohs , Jennifer G. Sheridan, Sweep and point and shoot: phonecam-based interactions for large public displays, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056876]
• 2: Chande, S. 2006. mForms: An Application Framework for Mobilizing Business Applications, January 2006, COMSWARE 2006, New Delhi
• 3: Gandalf Computer Vision Library URL: http://gandalf-library.sourceforge.net/
• 4: Seokhee Jeon , Jane Hwang , Gerard J. Kim , Mark Billinghurst, Interaction techniques in large display environments using hand-held devices, Proceedings of the ACM symposium on Virtual reality software and technology, November 01-03, 2006, Limassol, Cyprus [doi>10.1145/1180495.1180516]
• 5: Scott R. Klemmer , Mark W. Newman , Ryan Farrell , Mark Bilezikjian , James A. Landay, The designers' outpost: a tangible interface for collaborative web site, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502350]
• 6: Lee, B. and Klemmer, S. R., 2005. Notebooks that Share and Walls that Remember: Electronic Capture of Design Education Artifacts. In Conference Supplement to UIST 2005. URL: http://hci.stanford.edu/research/ideas/
• 7: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 8: Madhavapeddy, A., Scott, D., Sharp, R. and Upton, E., 2004. Using Camera-Phones to Enhance Human-Computer Interaction. In The adjunct proceedings of the Sixth International Conference on Ubiquitous Computing. URL: http://anil.recoil.org/papers/2004-ubicomp-camera.pdf
• 9: Kento Miyaoku , Suguru Higashino , Yoshinobu Tonomura, C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029657]
• 10: Elizabeth D. Mynatt , Takeo Igarashi , W. Keith Edwards , Anthony LaMarca, Flatland: new dimensions in office whiteboards, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.346-353, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303108]
• 11: Jun Rekimoto, Pick-and-drop: a direct manipulation technique for multiple computer environments, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.31-39, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263505]
• 12: SmartTech's SmartBoard touch-screen. URL: http://www.smarttech.com/smartboard/ {15-Sep-2006}

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Keywords:
imaging, large displays, mobile camera, user interfaces, visual interaction