SmartSkin: an infrastructure for freehand manipulation on interactive surfaces

ABSTRACT

This paper introduces a new sensor architecture for making interactive surfaces that are sensitive to human hand and finger gestures. This sensor recognizes multiple hand positions and shapes and calculates the distance between the hand and the surface by using capacitive sensing and a mesh-shaped antenna. In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems. This paper describes the sensor architecture, as well as two working prototype systems: a table-size system and a tablet-size system. It also describes several interaction techniques that would be difficult to perform without using this architecture

REFERENCES

• 1: Eric A. Bier , Maureen C. Stone , Ken Pier , William Buxton , Tony D. DeRose, Toolglass and magic lenses: the see-through interface, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.73-80, September 1993 [doi>10.1145/166117.166126]
• 2: George W. Fitzmaurice , Hiroshi Ishii , William A. S. Buxton, Bricks: laying the foundations for graspable user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.442-449, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223964]
• 3: Masaaki Fukumoto , Toshiaki Sugimura, Active click: tactile feedback for touch panels, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634141]
• 4: Y. Guiard. Asymmetric divisoin of labor in human skilled bimanual action: the kinematic chain as a model. Journal of Motor Behavior, pages 485--517, 1987.
• 5: Ken Hinckley , Randy Pausch , John C. Goble , Neal F. Kassell, Passive real-world interface props for neurosurgical visualization, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.452-458, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191821]
• 6: Ken Hinckley , Mike Sinclair, Touch-sensing input devices, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.223-230, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303045]
• 7: IEEE. Proceedings of the fourth ieee international conference on automatic face and gesture recognition, 2000.
• 8: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 9: Hideki Koike , Yoichi Sato , Yoshinori Kobayashi , Hiroaki Tobita , Motoki Kobayashi, Interactive textbook and interactive Venn diagram: natural and intuitive interfaces on augmented desk system, Proceedings of the SIGCHI conference on Human factors in computing systems, p.121-128, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332415]
• 10: SK Lee , William Buxton , K. C. Smith, A multi-touch three dimensional touch-sensitive tablet, Proceedings of the SIGCHI conference on Human factors in computing systems, p.21-25, April 1985, San Francisco, California, United States
• 11: M. Mathews and W. Schloss. The radiodrum as a synthesis controller. In Proceedings international computer music conference, 1989.
• 12: Nobuyuki Matsushita , Yuji Ayatsuka , Jun Rekimoto, Dual touch: a two-handed interface for pen-based PDAs, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.211-212, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354774]
• 13: Nobuyuki Matsushita , Jun Rekimoto, HoloWall: designing a finger, hand, body, and object sensitive wall, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.209-210, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263549]
• 14: Shunichi Numazaki , Akira Morshita , Naoko Umeki , Minoru Ishikawa , Miwako Doi, A kinetic and 3D image input device, CHI 98 conference summary on Human factors in computing systems, p.237-238, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286723]
• 15: Polhemus, Inc., Colchester, Vermont. 3SPACE ISOTRAK User's Manual, 1987.
• 16: Jun Rekimoto , Masanori Saitoh, Augmented surfaces: a spatially continuous work space for hybrid computing environments, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.378-385, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303113]
• 17: Jun Rekimoto , Eduardo Sciammarella, ToolStone: effective use of the physical manipulation vocabularies of input devices, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.109-117, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354421]
• 18: Jun Rekimoto , Brygg Ullmer , Haruo Oba, DataTiles: a modular platform for mixed physical and graphical interactions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.269-276, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365115]
• 19: Meredith Ringel , Henry Berg , Yuhui Jin , Terry Winograd, Barehands: implement-free interaction with a wall-mounted display, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634284]
• 20: Norbert A. Streitz , Jörg Geißler , Torsten Holmer , Shin'ichi Konomi , Christian Müller-Tomfelde , Wolfgang Reischl , Petra Rexroth , Peter Seitz , Ralf Steinmetz, i-LAND: an interactive landscape for creativity and innovation, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.120-127, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303010]
• 21: Brygg Ullmer , Hiroshi Ishii, The metaDESK: models and prototypes for tangible user interfaces, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.223-232, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263551]
• 22: John Underkoffler , Hiroshi Ishii, Illuminating light: an optical design tool with a luminous-tangible interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.542-549, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274717]
• 23: Pierre Wellner, The DigitalDesk calculator: tangible manipulation on a desk top display, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.27-33, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120785]
• 24: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 25: T. G. Zimmerman, Personal area networks: near-field intrabody communication, IBM Systems Journal, v.35 n.3-4, p.609-617, 1996
• 26: Thomas G. Zimmerman , Joshua R. Smith , Joseph A. Paradiso , David Allport , Neil Gershenfeld, Applying electric field sensing to human-computer interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.280-287, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223940]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments
          Subjects: Interactive environments

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.9 Robotics
          Subjects: Sensors
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques
  I.5 PATTERN RECOGNITION
      I.5.5 Implementation
          Subjects: Interactive systems

General Terms:
Design, Human Factors, Performance

Keywords:
augmented tables, gesture recognition, interactive surfaces, touch-sensitive interfaces, two-handed interfaces