Eyepatch: prototyping camera-based interaction through examples

ABSTRACT

Cameras are a useful source of input for many interactive applications, but computer vision programming is difficult and requires specialized knowledge that is out of reach for many HCI practitioners. In an effort to learn what makes a useful computer vision design tool, we created Eyepatch, a tool for designing camera-based interactions, and evaluated the Eyepatch prototype through deployment to students in an HCI course. This paper describes the lessons we learned about making computer vision more accessible, while retaining enough power and flexibility to be useful in a wide variety of interaction scenarios.

REFERENCES

• 1: Amavasai, B. P. Principles of Vision Systems. IEEE Systems, Man, and Cybernetics: Principles and Applications Workshop, 2002.
• 2: Brian Bagnall, Core LEGO MINDSTORMS Programming, Prentice Hall PTR, Upper Saddle River, NJ, 2002
• 3: Michael J. Black , Allan D. Jepson, A Probabilistic Framework for Matching Temporal Trajectories: CONDENSATION-Based Recognition of Gestures and Expressions, Proceedings of the 5th European Conference on Computer Vision-Volume I, p.909-924, June 02-06, 1998
• 4: Bradski, G. The OpenCV Library. Dr. Dobb's Journal November 2000, Computer Security, 2000.
• 5: Bradski, G. R. Computer vision face tracking for use in a perceptual user interface. Intel Technology Journal 2(2). pp. 12--21, 1998.
• 6: Gary R. Bradski , James W. Davis, Motion segmentation and pose recognition with motion history gradients, Machine Vision and Applications, v.13 n.3, p.174-184, July 2002 [doi>10.1007/s001380100064]
• 7: Diaz-Marino, R. and S. Greenberg. CAMBIENCE: A Video-Driven Sonic Ecology for Media Spaces. Video Proceedings of ACM CSCW'06 Conference on Computer Supported Cooperative Work, 2006.
• 8: Jerry Fails , Dan Olsen, A design tool for camera-based interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642690]
• 9: Jerry Alan Fails , Dan Olsen Jr., Light widgets: interacting in every-day spaces, Proceedings of the 7th international conference on Intelligent user interfaces, January 13-16, 2002, San Francisco, California, USA [doi>10.1145/502716.502729]
• 10: William T. Freeman , David B. Anderson , Paul A. Beardsley , Chris N. Dodge , Michal Roth , Craig D. Weissman , William S. Yerazunis , Hiroshi Kage , Kazuo Kyuma , Yasunari Miyake , Ken-ichi Tanaka, Computer Vision for Interactive Computer Graphics, IEEE Computer Graphics and Applications, v.18 n.3, p.42-53, May 1998 [doi>10.1109/38.674971]
• 11: Gregory D. Hager , Kentaro Toyama, X Vision: a portable substrate for real-time vision applications, Computer Vision and Image Understanding, v.69 n.1, p.23-37, Jan. 1998 [doi>10.1006/cviu.1997.0586]
• 12: Hancher, M. D., M. J. Broxton, and L. J. Edwards. A User's Guide to the NASA Vision Workbench. Intelligent Systems Division, NASA Ames Research Center, 2006.
• 13: Björn Hartmann , Leith Abdulla , Manas Mittal , Scott R. Klemmer, Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240646]
• 14: Björn Hartmann , Scott R. Klemmer , Michael Bernstein , Leith Abdulla , Brandon Burr , Avi Robinson-Mosher , Jennifer Gee, Reflective physical prototyping through integrated design, test, and analysis, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166300]
• 15: Igarashi, T., Student projects in the "Media Informatics" course at the University of Tokyo, 2005. http://www-ui.is.s.u-tokyo.ac.jp/~kwsk/media2005/projects.html
• 16: Iivarinen, J., M. Peura, J. Sarela, and A. Visa. Comparison of combined shape descriptors for irregular objects. Proceedings of the 8th British Machine Vision Conference 2. pp. 430--39, 1997.
• 17: Michael Isard , Andrew Blake, Contour Tracking by Stochastic Propagation of Conditional Density, Proceedings of the 4th European Conference on Computer Vision-Volume I, p.343-356, April 15-18, 1996
• 18: Hirokazu Kato , Mark Billinghurst, Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System, Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, p.85, October 20-21, 1999
• 19: Scott R. Klemmer , Jack Li , James Lin , James A. Landay, Papier-Mache: toolkit support for tangible input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.399-406, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985743]
• 20: Scott R. Klemmer , Mark W. Newman , Ryan Farrell , Mark Bilezikjian , James A. Landay, The designers' outpost: a tangible interface for collaborative web site, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502350]
• 21: Kolsch, M., M. Turk, and T. Hollerer. Vision-based interfaces for mobility. Mobile and Ubiquitous Systems: Networking and Services, 2004. MOBIQUITOUS 2004. The First Annual International Conference on. pp. 86--94, 2004.
• 22: Myron W. Krueger , Thomas Gionfriddo , Katrin Hinrichsen, VIDEOPLACE—an artificial reality, ACM SIGCHI Bulletin, v.16 n.4, p.35-40, April 1985
• 23: Larssen, A. T., L. Loke, T. Robertson, and J. Edwards. Understanding Movement as Input for Interaction-A Study of Two EyeToy" Games. Proceedings of OZCHI 2004, 2004.
• 24: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 25: Maynes-Aminzade, D., Website for the course "Designing Applications that See" at Stanford University, 2007. http://cs377s.stanford.edu/
• 26: Techniques for Interactive Audience Participation, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.15, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166962]
• 27: Brad Myers , Scott E. Hudson , Randy Pausch, Past, present, and future of user interface software tools, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.1, p.3-28, March 2000 [doi>10.1145/344949.344959]
• 28: Casey Reas , Benjamin Fry, Processing: a learning environment for creating interactive Web graphics, ACM SIGGRAPH 2003 Web Graphics, July 27-31, 2003, San Diego, California [doi>10.1145/965333.965356]
• 29: Jeffrey S. Shell , Roel Vertegaal , Alexander W. Skaburskis, EyePliances: attention-seeking devices that respond to visual attention, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765981]
• 30: Maribeth Gandy , Thad Starner , Jake Auxier , Daniel Ashbrook, The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring, Proceedings of the 4th IEEE International Symposium on Wearable Computers, p.87, October 18-21, 2000
• 31: John Underkoffler , Hiroshi Ishii, Illuminating light: an optical design tool with a luminous-tangible interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.542-549, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274717]
• 32: Viola, P. and M. Jones. Rapid object detection using a boosted cascade of simple features. Proc. CVPR 1. pp. 511--18, 2001.
• 33: Daniel Wagner , Dieter Schmalstieg, Handheld Augmented Reality Displays, Proceedings of the IEEE conference on Virtual Reality, p.321, March 25-29, 2006 [doi>10.1109/VR.2006.67]
• 34: Tracy Westeyn , Helene Brashear , Amin Atrash , Thad Starner, Georgia tech gesture toolkit: supporting experiments in gesture recognition, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958452]
• 35: Andrew D. Wilson, PlayAnywhere: a compact interactive tabletop projection-vision system, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095047]
• 36: Andrew D. Wilson, TouchLight: an imaging touch screen and display for gesture-based interaction, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027946]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Prototyping; Theory and methods

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking; Object recognition; Color
      I.4.9 Applications

General Terms:
Algorithms, Design, Human Factors

Keywords:
classification, computer vision, image processing, interaction, machine learning