Camera phone based motion sensing: interaction techniques, applications and performance study

ABSTRACT

This paper presents TinyMotion, a pure software approach for detecting a mobile phone user's hand movement in real time by analyzing image sequences captured by the built-in camera. We present the design and implementation of TinyMotion and several interactive applications based on TinyMotion. Through both an informal evaluation and a formal 17-participant user study, we found that 1. TinyMotion can detect camera movement reliably under most background and illumination conditions. 2. Target acquisition tasks based on TinyMotion follow Fitts' law and Fitts law parameters can be used for TinyMotion based pointing performance measurement. 3. The users can use Vision TiltText, a TinyMotion enabled input method, to enter sentences faster than MultiTap with a few minutes of practicing. 4. Using camera phone as a handwriting capture device and performing large vocabulary, multilingual real time handwriting recognition on the cell phone are feasible. 5. TinyMotion based gaming is enjoyable and immediately available for the current generation camera phones. We also report user experiences and problems with TinyMotion based interaction as resources for future design and development of mobile interfaces.

REFERENCES

• 1: Rafael Ballagas , Michael Rohs , Jennifer G. Sheridan, Sweep and point and shoot: phonecam-based interactions for large public displays, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056876]
• 2: Drab, S., Artner, N., Motion Detection as Interaction Technique for Games & Applications on Mobile Devices, In Ext. Abstracts of PERVASIVE '05: (PERMID), Munich, Germany.
• 3: EyeMobile, http://www.eyemobile.com
• 4: Fitts, P. M. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381--391, 1954.
• 5: George W. Fitzmaurice , Shumin Zhai , Mark H. Chignell, Virtual reality for palmtop computers, ACM Transactions on Information Systems (TOIS), v.11 n.3, p.197-218, July 1993 [doi>10.1145/159161.159160]
• 6: David A. Forsyth , Jean Ponce, Computer Vision: A Modern Approach, Prentice Hall Professional Technical Reference, 2002
• 7: William T. Freeman , David B. Anderson , Paul A. Beardsley , Chris N. Dodge , Michal Roth , Craig D. Weissman , William S. Yerazunis , Hiroshi Kage , Kazuo Kyuma , Yasunari Miyake , Ken-ichi Tanaka, Computer Vision for Interactive Computer Graphics, IEEE Computer Graphics and Applications, v.18 n.3, p.42-53, May 1998 [doi>10.1109/38.674971]
• 8: Borivoje Furht , Borko Furht , Joshua Greenblatt, Motion Estimation Algorithms for Video Compression, Kluwer Academic Publishers, Norwell, MA, 1996
• 9: Hannuksela, J., Sangi, P., and Heikkila J., A Vision-Based Approach for Controlling User Interfaces of Mobile Devices, In Proc. of IEEE Workshop on Vision for Human-Computer Interaction (V4HCI), 2005
• 10: Thomas Riisgaard Hansen , Eva Eriksson , Andreas Lykke-Olesen, Mixed interaction space: designing for camera based interaction with mobile devices, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057060]
• 11: Beverly L. Harrison , Kenneth P. Fishkin , Anuj Gujar , Carlos Mochon , Roy Want, Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.17-24, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274647]
• 12: Ken Hinckley , Jeff Pierce , Mike Sinclair , Eric Horvitz, Sensing techniques for mobile interaction, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.91-100, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354417]
• 13: Berthold K.P. Horn , Brian G. Schunck, Determining Optical Flow, Massachusetts Institute of Technology, Cambridge, MA, 1980
• 14: Peter M. Kuhn , Kuhn Peter M., Algorithms, Complexity Analysis and VLSI Architectures for MPEG-4 Motion Estimation, Kluwer Academic Publishers, Norwell, MA, 1999
• 15: Gordon Kurtenbach , William Buxton, User learning and performance with marking menus, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.258-264, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191759]
• 16: Moehring, M., Lessig C., and Bimber O. Optical tracking and video see-through AR on consumer cell phones. In Proc. of Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR, pp. 193--204. 2004
• 17: Thomas B. Moeslund , Erik Granum, A survey of computer vision-based human motion capture, Computer Vision and Image Understanding, v.81 n.3, p.231-268, March 2001 [doi>10.1006/cviu.2000.0897]
• 18: Jun Rekimoto, Tilting operations for small screen interfaces, Proceedings of the 9th annual ACM symposium on User interface software and technology, p.167-168, November 06-08, 1996, Seattle, Washington, United States [doi>10.1145/237091.237115]
• 19: Jun Rekimoto , Yuji Ayatsuka, CyberCode: designing augmented reality environments with visual tags, Proceedings of DARE 2000 on Designing augmented reality environments, p.1-10, April 2000, Elsinore, Denmark [doi>10.1145/354666.354667]
• 20: Rohs, M., Zweifel, P., A Conceptual Framework for Camera Phone-based Interaction Techniques, In Proc. of PERVASIVE 2005, Munich, Germany, May 2005
• 21: The Development of Camera Phone Module Industry, 2005-2006, http://www.okokok.com.cn/Abroad/Abroad_show.asp?ArticleID=1034
• 22: Wang, L., Hu, W., and Tan, T., Recent developments in human motion analysis, Pattern. Recognition, 36 pp. 585--601, 2003
• 23: Kurt Partridge , Saurav Chatterjee , Vibha Sazawal , Gaetano Borriello , Roy Want, TiltType: accelerometer-supported text entry for very small devices, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572013]
• 24: Daniel Wigdor , Ravin Balakrishnan, TiltText: using tilt for text input to mobile phones, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.81-90, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964705]
• 25: Ka-Ping Yee, Peephole displays: pen interaction on spatially aware handheld computers, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642613]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen); Theory and methods

General Terms:
Design, Human Factors, Performance

Keywords:
Fitts' law, camera phones, computer vision, gesture recognition, handwriting recognition, human performance, input techniques and devices, mobile devices, mobile phones, motion estimation