Sweep and point and shoot: phonecam-based interactions for large public displays

ABSTRACT

This paper focuses on enabling interactions with large public displays using the most ubiquitous personal computing device, the mobile phone. Two new interaction techniques are introduced that use the embedded camera on mobile phones as an enabling technology. The "Point & Shoot" technique allows users to select objects using visual codes to set up an absolute coordinate system on the display surface instead of tagging individual objects on the screen. The "Sweep" technique enables users to use the phone like an optical mouse with multiple degrees of freedom and allows interaction without having to point the camera at the display. Prototypes of these interactions have been implemented and evaluated using modern mobile phone technologies. This proof of concept provides a performance baseline and gives valuable insights to guide future research and development. These techniques are intended to inspire and enable new classes of large public display applications.

REFERENCES

• 1: Stuart K. Card , Jock D. Mackinlay , George G. Robertson, A morphological analysis of the design space of input devices, ACM Transactions on Information Systems (TOIS), v.9 n.2, p.99-122, April 1991 [doi>10.1145/123078.128726]
• 2: ISO. Ergonomic requirements for office work with visual display terminals (VDTs) - Requirements for non-keyboard input devices. ISO 9241-9/ (2000).
• 3: Madhavapeddy, A., Scott, D., Sharp, R., and Upton, E. Using camera-phones to enhance human-computer interaction. Adjuct Proc. of Ubicomp. Springer-Verlag (2004).
• 4: Kento Miyaoku , Suguru Higashino , Yoshinobu Tonomura, C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029657]
• 5: Brad A. Myers , Herb Stiel , Robert Gargiulo, Collaboration using multiple PDAs connected to a PC, Proceedings of the 1998 ACM conference on Computer supported cooperative work, p.285-294, November 14-18, 1998, Seattle, Washington, United States [doi>10.1145/289444.289503]
• 6: Rohs, M. Real-world interaction with camera-phones. Proc. UCS. IPSJ Press (2004).

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Haptic I/O; Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Experimentation, Human Factors

Keywords:
camera phones, large displays, mobile phones, visual codes