Shoot & copy: phonecam-based information transfer from public displays onto mobile phones

ABSTRACT

Large public displays have become pervasive in our everyday lives, but up to now, they are mostly information screens without any interaction possibilities. Users tend to forget what they saw relatively fast after leaving such a display. In this paper, we present a new interaction technique for transferring information from a public display onto a personal mobile phone with its built-in camera. Instead of having to rely on their memory, users simply take a picture of the information of interest. Instead of just storing the image, our system then retrieves the actual data represented on the screen, such as a stock quote, news text, or piece of music. The Shoot & Copy technique does not require visual codes that interfere with shown content or reduce screen real estate. Our prototype allows users to capture an arbitrary region of a standard desktop screen, containing icons, which represent pieces of data. The captured image is then analyzed and a reference to the corresponding data is sent back to the mobile phone. Once the user has time to view the information in more detail, our system allows retrieving the actual data from this reference. We present our prototype and the methods it uses for image processing, as well as an evaluation of our interaction technique illustrating its potential use and applications.

REFERENCES

• 1: AR Toolkit. http://www.hitl.washington.edu/artoolkit/
• 2: Rafael Ballagas , Michael Rohs , Jennifer G. Sheridan, Sweep and point and shoot: phonecam-based interactions for large public displays, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056876]
• 3: Ballagas, R., Rohs, M., Sheridan, J., Borchers, J., BYOD: Bring your own Device, In Proceedings of UbiComp 2004 Workshop on Ubiquitous Display Environments, Nottingham, UK, 2004
• 4: BeeTagg. http://www.beetagg.com/
• 5: Costanza, E., Leinss, M., Telling a Story on a Tag: The Importance of Markers's Visual Design for Real World Applications, MIRW 2006, workshop in conjunction with MobileHCI 2006, Espoo, Finland, ACM, 2006, 6--9
• 6: Gregor Broll , Sven Siorpaes , Enrico Rukzio , Massimo Paolucci , John Hamard , Matthias Wagner , Albrecht Schmidt, Supporting Mobile Service Usage through Physical Mobile Interaction, Proceedings of the Fifth IEEE International Conference on Pervasive Computing and Communications, p.262-271, March 19-23, 2007 [doi>10.1109/PERCOM.2007.35]
• 7: Greenberg, S., Boyle, M., Laberge, J., PDAs and Shared Public Displays: Making Personal Information Public, and Public Information Personal, In Personal and Ubiquitous Computing 3(1/2), London, UK, Springer, 1999, 54--64
• 8: International Organization for Standardization: Information Technology - International Symbology Specification - Data Matrix. ISO/IEC 16022, 2000
• 9: International Organization for Standardization: Information Technology - Automatic Identification and Data Capture Techniques - Bar Code Symbology - QR Code. ISO/IEC 18004, 2000
• 10: Naohiko Kohtake , Jun Rekimoto , Yuichiro Anzai, InfoPoint: A Device that Provides a Uniform User Interface to Allow Appliances to Work Together over a Network, Personal and Ubiquitous Computing, v.5 n.4, p.264-274, December 2001 [doi>10.1007/s007790170005]
• 11: Hirokazu Kato , Mark Billinghurst, Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System, Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, p.85, October 20-21, 1999
• 12: Kento Miyaoku , Suguru Higashino , Yoshinobu Tonomura, C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029657]
• 13: Jun Rekimoto, Pick-and-drop: a direct manipulation technique for multiple computer environments, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.31-39, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263505]
• 14: Rohs, M., Gfeller, B., Using Camera-Equipped Mobile Phones for Interacting with Real-World Objects, In Advances in Pervasive Computing, Austrian Computer Society (OCG), Vienna, Austria, 2004, 265--271
• 15: Rohs, M., Real-World Interaction with Camera Phones, In Proceedings of UCS 2004, Tokyo, Japan, Revised Selected Papers, LNCS 3598, 2005, 74--89
• 16: Enrico Rukzio , Albrecht Schmidt , Heinrich Hussmann, Physical Posters as Gateways to Context-Aware Services for Mobile Devices, Proceedings of the Sixth IEEE Workshop on Mobile Computing Systems and Applications, p.10-19, December 02-03, 2004 [doi>10.1109/MCSA.2004.20]
• 17: Semacode. http://semacode.org/
• 18: Semapedia. http://www.semapedia.org/
• 19: ShotCode. http://www.shotcode.com/

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Computer vision

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
image processing, large public displays, mobile camera phones