Token+constraint systems for tangible interaction with digital information

ABSTRACT

We identify and present a major interaction approach for tangible user interfaces based upon systems of tokens and constraints. In these interfaces, tokens are discrete physical objects which represent digital information. Constraints are confining regions that are mapped to digital operations. These are frequently embodied as structures that mechanically channel how tokens can be manipulated, often limiting their movement to a single degree of freedom. Placing and manipulating tokens within systems of constraints can be used to invoke and control a variety of computational interpretations.We discuss the properties of the token+constraint approach; consider strengths that distinguish them from other interface approaches; and illustrate the concept with eleven past and recent supporting systems. We present some of the conceptual background supporting these interfaces, and consider them in terms of Bellotti et al.'s [2002] five questions for sensing-based interaction. We believe this discussion supports token+constraint systems as a powerful and promising approach for sensing-based interaction.

REFERENCES

• 1: Christopher Ahlberg , Ben Shneiderman, Visual information seeking: tight coupling of dynamic query filters with starfield displays, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.313-317, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191775]
• 2: Aish, R. and Noakes, P. 1984. Architecture without numbers---CAAD based on a 3D modelling system. In Computer-Aided Design 16, 6 (Nov.) 321--328.
• 3: Anagnostou, G., Dewey, D., and Patera, A. 1989. Geometry-defining processors for engineering design and analysis. In The Visual Computer. Chapter 5, 304--315.
• 4: David Anderson , James L. Frankel , Joe Marks , Aseem Agarwala , Paul Beardsley , Jessica Hodgins , Darren Leigh , Kathy Ryall , Eddie Sullivan , Jonathan S. Yedidia, Tangible interaction + graphical interpretation: a new approach to 3D modeling, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.393-402, July 2000 [doi>10.1145/344779.344960]
• 5: Rafael Ballagas , Meredith Ringel , Maureen Stone , Jan Borchers, iStuff: a physical user interface toolkit for ubiquitous computing environments, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642705]
• 6: Bell, R. 1979. Board and Table Games from Many Civilizations. Dover Publications, New York, NY.
• 7: Victoria Bellotti , Maribeth Back , W. Keith Edwards , Rebecca E. Grinter , Austin Henderson , Cristina Lopes, Making sense of sensing systems: five questions for designers and researchers, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503450]
• 8: Eduardo H. Calvillo-Gámez , Nancy Leland , Orit Shaer , Robert J. K. Jacob, The TAC paradigm: unified conceptual framework to represent Tangible User Interfaces, Proceedings of the Latin American conference on Human-computer interaction, August 17-20, 2003, Rio de Janeiro, Brazil
• 9: Jonathan Cohen , Meg Withgott , Philippe Piernot, Logjam: a tangible multi-person interface for video logging, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.128-135, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303013]
• 10: Mark R. Cutkosky , Robert D. Howe, Human grasp choice and robotic grasp analysis, Dextrous robot hands, Springer-Verlag New York, Inc., New York, NY, 1990
• 11: Durham, J. 2002a. Abrasives, trust, and how the Abacus got its name. http://bart.cba.nau. edu/∼durham-j/newsite/id153.htm {Feb}.
• 12: Durham, J. 2002b. Personal communications. Feb. 10, 2002.
• 13: Fernandes, L. 2001. The abacus: The art of calculating with beads. http://www.ee.ryerson. ca:8080/∼elf/abacus/ {Feb. 2002}.
• 14: George W. Fitzmaurice , Hiroshi Ishii , William A. S. Buxton, Bricks: laying the foundations for graspable user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.442-449, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223964]
• 15: Morten Fjeld , Martin Bichsel , Matthias Rauterberg, BUILD-IT: An Intuitive Design Tool Based on Direct Object Manipulation, Proceedings of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction, p.297-308, September 17-19, 1997
• 16: Frazer, J. 1995. An Evolutionary Architecture. Architectural Association. London, UK.
• 17: Frazer J. H., Frazer J. M., and Frazer P. A. 1989. Intelligent physical three-dimensional modelling systems. Computer Graphics' 80 Conference, Conference Proceedings, Online Publications. 359--370.
• 18: Hans W. Gellersen , Albercht Schmidt , Michael Beigl, Multi-sensor context-awareness in mobile devices and smart artifacts, Mobile Networks and Applications, v.7 n.5, p.341-351, October 2002 [doi>10.1023/A:1016587515822]
• 19: Gibson, J. 1979. The Ecological Approach to Visual Perception. Erlbaum Associates, New York, NY.
• 20: Guiard, Y. 1987. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. J. Motor Behav. 19, 4, 486--517.
• 21: Ken Hinckley , Randy Pausch , Dennis Proffitt , Neal F. Kassell, Two-handed virtual manipulation, ACM Transactions on Computer-Human Interaction (TOCHI), v.5 n.3, p.260-302, Sept. 1998 [doi>10.1145/292834.292849]
• 22: Lars Erik Holmquist , Johan Redström , Peter Ljungstrand, Token-Based Acces to Digital Information, Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing, p.234-245, September 27-29, 1999, Karlsruhe, Germany
• 23: Hornecker, E. 2002. Understanding the benefits of graspable interfaces for cooperative use. In Proceedings of Cooperative Systems Design 2002. 71--87.
• 24: Georges Ifrah , E. F. Harding, The Universal History of Computing: From the Abacus to the Quantum Computer, John Wiley & Sons, Inc., New York, NY, 2001
• 25: Hiroshi Ishii , Eran Ben-Joseph , John Underkoffler , Luke Yeung , Dan Chak , Zahra Kanji , Ben Piper, Augmented Urban Planning Workbench: Overlaying Drawings, Physical Models and Digital Simulation, Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR'02), p.203, September 30-October 01, 2002
• 26: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 27: Robert J. K. Jacob , Hiroshi Ishii , Gian Pangaro , James Patten, A tangible interface for organizing information using a grid, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503437]
• 28: William P. Jones , Susan T. Dumais, The spatial metaphor for user interfaces: experimental tests of reference by location versus name, ACM Transactions on Information Systems (TOIS), v.4 n.1, p.42-63, Jan. 1986 [doi>10.1145/5401.5405]
• 29: David Kirsh, The intelligent use of space, Artificial Intelligence, v.73 n.1-2, p.31-68, Feb. 1995 [doi>10.1016/0004-3702(94)00017-U]
• 30: Klemmer, S. 2003. Papier-Mâhé: Toolkit support for tangible interaction. In Proceedings of User Interface Software and Technology 1995.
• 31: Larkin, J. and Simon, H. 1987. Why a diagram is (sometimes) worth ten thousand words. Cognit. Sci. 11, 65--99.
• 32: Lütjens, J. 2002. Abacus online museum. http://www.joernluetjens.de/sammlungen/abakus/abakus-en.htm {(Feb.) 2002}.
• 33: Karon E. MacLean , Scott S. Snibbe , Golan Levin, Tagged handles: merging discrete and continuous manual control, Proceedings of the SIGCHI conference on Human factors in computing systems, p.225-232, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332435]
• 34: Masters, J. 2002. The royal game of Ur and Tau. http://www.tradgames.org.uk/games/Royal-Game-Ur.htm {(Aug.) 2002}.
• 35: Ali Mazalek , Tristan Jehan, Interacting with music in a social setting, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633441]
• 36: Mcnerney, T. 2000. Tangible programming bricks: An approach to making programming accessible to everyone. MS Thesis, MIT Media Laboratory.
• 37: Les Nelson , Satoshi Ichimura , Elin Rønby Pedersen , Lia Adams, Palette: a paper interface for giving presentations, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.354-361, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303109]
• 38: Neurosmith. 1999. MusicBlocks product. http://www.neurosmith.com/.
• 39: Donald A. Norman, Affordance, conventions, and design, interactions, v.6 n.3, p.38-43, May/June 1999 [doi>10.1145/301153.301168]
• 40: Norman, D. 1993. Things that Make Us Smart. Addison-Wesley, Reading, MA.
• 41: Oxford English Dictionary (OED). 1989. OED Online Oxford University Press.
• 42: Gian Pangaro , Dan Maynes-Aminzade , Hiroshi Ishii, The actuated workbench: computer-controlled actuation in tabletop tangible interfaces, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572011]
• 43: James Patten , Ben Recht , Hiroshi Ishii, Audiopad: a tag-based interface for musical performance, Proceedings of the 2002 conference on New interfaces for musical expression, May 24-26, 2002, Dublin, Ireland
• 44: James Patten , Hiroshi Ishii , Jim Hines , Gian Pangaro, Sensetable: a wireless object tracking platform for tangible user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.253-260, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365112]
• 45: Perlman, R. 1976. Using computer technology to provide a creative learning environment for preschool children. MIT Logo Memo #24.
• 46: Marian Petre, Why looking isn't always seeing: readership skills and graphical programming, Communications of the ACM, v.38 n.6, p.33-44, June 1995 [doi>10.1145/203241.203251]
• 47: Polynor, R. 1995. The hand that rocks the cradle. I.D. (May/June), 60--65.
• 48: Jenny Preece , Yvonne Rogers , Helen Sharp, Interaction Design, John Wiley & Sons, Inc., New York, NY, 2002
• 49: Redström, J. 2001. Designing everyday computational things. Ph.D. thesis, Göteberg University.
• 50: Jun Rekimoto , Brygg Ullmer , Haruo Oba, DataTiles: a modular platform for mixed physical and graphical interactions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.269-276, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365115]
• 51: Gudula Retz-Schmidt, Various views on spatial prepositions, AI Magazine, v.9 n.2, p.95-105, July/August 1988
• 52: Mike Scaife , Yvonne Rogers, External cognition: how do graphical representations work?, International Journal of Human-Computer Studies, v.45 n.2, p.185-213, Aug. 1996 [doi>10.1006/ijhc.1996.0048]
• 53: Kai Schäfer , Volker Brauer , Willi Bruns, A new approach to human-computer interaction—synchronous modelling in real and virtual spaces, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.335-344, August 18-20, 1997, Amsterdam, The Netherlands [doi>10.1145/263552.263638]
• 54: Schieβl, S. 2002. Digital cubes. http://www.aec.at/festival2002/texte/schieβl_e.asp.
• 55: Schmandt-Besserat, D. 1997. How Writing Came About. University of Texas Press, Austin, TX.
• 56: Shneiderman, B. 1983. Direct manipulation: A step beyond programming languages. IEEE Comput. 16, 8, 57--69.
• 57: Andrew Singer , Debby Hindus , Lisa Stifelman , Sean White, Tangible progress: less is more in Somewire audio spaces, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.104-111, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303007]
• 58: David Canfield Smith, Pygmalion: a creative programming environment., 1975
• 59: Suzuki, H. and Kato, H. 1993. AlgoBlock: A tangible programming language, a tool for collaborative learning. In Proceedings of 4th European Logo Conference. 297--303.
• 60: Ten Hagen, P. 1981. Interaction and syntax. Int. J. Man-Mach. Stud. 15.
• 61: Tomoe Soroban Co., Ltd. 2002. Soroban museum: Roman Soroban. http://www.soroban. com/museum/∼5s_eng.html {Feb. 2002}.
• 62: Ullmer, B., Ishii, H., and Jacob, R. 2003. Tangible query interfaces: Physically constrained tokens for manipulating database queries. To appear in Proceedings of International Conference on Computer-Human Interaction 2003.
• 63: Brygg Anders Ullmer , Hiroshi Ishii, Tangible interfaces for manipulating aggregates of digital information, 2002
• 64: Ullmer, B., and Ishii, H. 2001. Emerging Frameworks for Tangible User Interfaces. In HCI in the New Millenium, John M. Carroll, Ed. 579--601.
• 65: Brygg Ullmer , Hiroshi Ishii, The metaDESK: models and prototypes for tangible user interfaces, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.223-232, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263551]
• 66: Brygg Ullmer , Hiroshi Ishii , Dylan Glas, mediaBlocks: physical containers, transports, and controls for online media, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.379-386, July 1998 [doi>10.1145/280814.280940]
• 67: John Underkoffler , Brygg Ullmer , Hiroshi Ishii, Emancipated pixels: real-world graphics in the luminous room, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.385-392, July 1999 [doi>10.1145/311535.311593]
• 68: Want, R. and Russell, D. 2000. Ubiquitous electronic tagging. In IEEE Distrib. Syst. Online 1, 2 (Sept.).
• 69: Paul Yarin , Hiroshi Ishii, TouchCounters: designing interactive electronic labels for physical containers, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.362-369, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303110]
• 70: Zhang, J. 1997. The nature of external representations in problem solving. Cogn. Sci. 21, 2, 179--217.
• 71: Zhang, J. and Norman, D. 1994. Representations in distributed cognitive tasks. Cogn. Sci. 18, 87--122.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Design, Theory

Keywords:
Tangible interfaces, token+constraint interfaces