Reality-based interaction: a framework for post-WIMP interfaces

ABSTRACT

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

REFERENCES

• 1: Gregory D. Abowd , Alan J. Dix, Integrating status and event phenomena in formal specifications of interactive systems, Proceedings of the 2nd ACM SIGSOFT symposium on Foundations of software engineering, p.44-52, December 06-09, 1994, New Orleans, Louisiana, United States
• 2: Anand Agarawala , Ravin Balakrishnan, Keepin' it real: pushing the desktop metaphor with physics, piles and the pen, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124965]
• 3: Ängeslevä, J., Oakley, I., Hughes, S. and O'Modhrain, S., Body Mnemonics Portable device interaction design concept. in UIST, (2003).
• 4: Trent Apted , Judy Kay , Aaron Quigley, Tabletop sharing of digital photographs for the elderly, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124887]
• 5: Seok-Hyung Bae , Takahiro Kobayash , Ryugo Kijima , Won-Sup Kim, Tangible NURBS-curve manipulation techniques using graspable handles on a large display, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029646]
• 6: Michel Beaudouin-Lafon, Instrumental interaction: an interaction model for designing post-WIMP user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.446-453, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332473]
• 7: Ashweeni Beeharee , Anthony Steed, Exploiting real world knowledge in ubiquitous applications, Personal and Ubiquitous Computing, v.11 n.6, p.429-437, August 2007 [doi>10.1007/s00779-006-0091-6]
• 8: Victoria Bellotti , Maribeth Back , W. Keith Edwards , Rebecca E. Grinter , Austin Henderson , Cristina Lopes, Making sense of sensing systems: five questions for designers and researchers, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503450]
• 9: Steve Benford , Holger Schnädelbach , Boriana Koleva , Rob Anastasi , Chris Greenhalgh , Tom Rodden , Jonathan Green , Ahmed Ghali , Tony Pridmore , Bill Gaver , Andy Boucher , Brendan Walker , Sarah Pennington , Albrecht Schmidt , Hans Gellersen , Anthony Steed, Expected, sensed, and desired: A framework for designing sensing-based interaction, ACM Transactions on Computer-Human Interaction (TOCHI), v.12 n.1, p.3-30, March 2005 [doi>10.1145/1057237.1057239]
• 10: Doug A. Bowman , Ernst Kruijff , Joseph J. LaViola , Ivan Poupyrev, 3D User Interfaces: Theory and Practice, Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, 2004
• 11: Jacob Buur , Mads Vedel Jensen , Tom Djajadiningrat, Hands-only scenarios and video action walls: novel methods for tangible user interaction design, Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques, August 01-04, 2004, Cambridge, MA, USA [doi>10.1145/1013115.1013141]
• 12: Buxton, B. http://www.billbuxton.com/multitouchOverview.html, 2007.
• 13: Georgios Christou, Towards a new method for the evaluation of reality based interaction, CHI '07 extended abstracts on Human factors in computing systems, April 28-May 03, 2007, San Jose, CA, USA [doi>10.1145/1240866.1240974]
• 14: Céline Coutrix , Laurence Nigay, Mixed reality: a model of mixed interaction, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133274]
• 15: Connor Dickie , Roel Vertegaal , Changuk Sohn , Daniel Cheng, eyeLook: using attention to facilitate mobile media consumption, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095050]
• 16: Where the action is: the foundations of embodied interaction, MIT Press, Cambridge, MA, 2001
• 17: Dubois, E. and Gray, P. A Design-Oriented Information-Flow Refinement of the ASUR Interaction Model. Proceedings of EHCI 2007.
• 18: Kenneth P. Fishkin, A taxonomy for and analysis of tangible interfaces, Personal and Ubiquitous Computing, v.8 n.5, p.347-358, September 2004 [doi>10.1007/s00779-004-0297-4]
• 19: Kenneth P. Fishkin , Thomas P. Moran , Beverly L. Harrison, Embodied User Interfaces: Towards Invisible User Interfaces, Proceedings of the IFIP TC2/TC13 WG2.7/WG13.4 Seventh Working Conference on Engineering for Human-Computer Interaction, p.1-18, September 14-18, 1998
• 20: Clifton Forlines , Chia Shen, DTLens: multi-user tabletop spatial data exploration, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095055]
• 21: Hayes, P.J. The second naive physics manifesto Cognitive Science Technical Report URCS-10, University of Rochester, 1983.
• 22: Eva Hornecker , Jacob Buur, Getting a grip on tangible interaction: a framework on physical space and social interaction, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124838]
• 23: Jörn Hurtienne , Johann Habakuk Israel, Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction, Proceedings of the 1st international conference on Tangible and embedded interaction, February 15-17, 2007, Baton Rouge, Louisiana [doi>10.1145/1226969.1226996]
• 24: Hutchins, E.L., Hollan, J.D. and Norman, D.A. Direct Manipulation Interfaces. Draper, D.A.N.a.S.W. ed. User Centered System Design: New Perspectives on Human-computer Interaction, Lawrence Erlbaum, Hillsdale, N.J., 1986, 87--124.
• 25: Randolph L. Jackson , Eileen Fagan, Collaboration and learning within immersive virtual reality, Proceedings of the third international conference on Collaborative virtual environments, p.83-92, September 2000, San Francisco, California, United States [doi>10.1145/351006.351018]
• 26: Robert J. K. Jacob , Audrey Girouard , Leanne M. Hirshfield , Michael S. Horn , Orit Shaer , Erin Treacy Solovey , Jamie Zigelbaum, Reality-based interaction: unifying the new generation of interaction styles, CHI '07 extended abstracts on Human factors in computing systems, April 28-May 03, 2007, San Jose, CA, USA [doi>10.1145/1240866.1241025]
• 27: Robert Jacob , Audrey Girouard , Leanne M. Hirshfield , Michael Horn , Orit Shaer , Erin Treacy Solovey , Jamie Zigelbaum, CHI2006: what is the next generation of human-computer interaction?, interactions, v.14 n.3, May + June 2007 [doi>10.1145/1242421.1242459]
• 28: Kalanithi, J. Connectibles: Tangible Social Networking, MS Thesis, MIT Media Lab, Cambridge, 2007, 132pp.
• 29: Scott R. Klemmer , Björn Hartmann , Leila Takayama, How bodies matter: five themes for interaction design, Proceedings of the 6th conference on Designing Interactive systems, June 26-28, 2006, University Park, PA, USA [doi>10.1145/1142405.1142429]
• 30: MacLean, A., Young, R.M., Bellotti, V.M.E. and Moran, T.P. Questions, Options, and Criteria: Elements of Design Space Analysis. Human-Computer Interaction, 6. 201--250.
• 31: John Maeda, The Laws of Simplicity (Simplicity: Design, Technology, Business, Life), The MIT Press, 2006
• 32: Merrill, D. and Maes, P., Augmenting Looking, Pointing and Reaching Gestures to Enhance the Searching and Browsing of Physical Objects. in Proc. Pervasive'07, (2007).
• 33: Milgram, P. and Kishino, F., Augmented reality: A class of displays on the reality-virtuality continuum. in SPIE, (1994), 282--292.
• 34: Betty J. Mohler , William B. Thompson , Sarah H. Creem-Regehr , Peter Willemsen , Herbert L. Pick, Jr. , John J. Rieser, Calibration of locomotion resulting from visual motion in a treadmill-based virtual environment, ACM Transactions on Applied Perception (TAP), v.4 n.1, p.4-es, January 2007 [doi>10.1145/1227134.1227138]
• 35: Jakob Nielsen, Noncommand user interfaces, Communications of the ACM, v.36 n.4, p.83-99, April 1993 [doi>10.1145/255950.153582]
• 36: Ivan Poupyrev , Henry Newton-Dunn , Olivier Bau, D20: interaction with multifaceted display devices, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125683]
• 37: Rohrer, T. Metaphors We Compute By: Bringing Magic into Interface Design, Center for the Cognitive Science of Metaphor, Philosophy Department, University of Oregon, 1995.
• 38: Second Life. http://secondlife.com/.
• 39: B. Shneiderman, Direct Manipulation: A Step Beyond Programming Languages, Computer, v.16 n.8, p.57-69, August 1983 [doi>10.1109/MC.1983.1654471]
• 40: Ben Shneiderman, Leonardo's Laptop: Human Needs and the New Computing Technologies, MIT Press, Cambridge, MA, 2002
• 41: Ben Shneiderman, Why Not Make Interfaces Better than 3D Reality?, IEEE Computer Graphics and Applications, v.23 n.6, p.12-15, November 2003 [doi>10.1109/MCG.2003.1242376]
• 42: Mel Slater , Martin Usoh , Anthony Steed, Taking steps: the influence of a walking technique on presence in virtual reality, ACM Transactions on Computer-Human Interaction (TOCHI), v.2 n.3, p.201-219, Sept. 1995 [doi>10.1145/210079.210084]
• 43: John D. Smith , Roel Vertegaal , Changuk Sohn, ViewPointer: lightweight calibration-free eye tracking for ubiquitous handsfree deixis, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095043]
• 44: Streitz, N.A., Tandler, P., Muller-Tomfelde, C. and Konomi, S. Roomware: Toward the Next Generation of Human-Computer Interaction Based on an Integrated Design of Real and Virtual Worlds. Carroll, J.M. ed. Human-Computer Interaction in the New Millenium, Addison-Wesley/ACM Press, Reading, Mass., 2001.
• 45: Tanriverdi, V. A Virtual Reality Interface Design (VRID) Model and Methodology, Tufts University, 2001.
• 46: Vildan Tanriverdi , Robert J. K. Jacob, Interacting with eye movements in virtual environments, Proceedings of the SIGCHI conference on Human factors in computing systems, p.265-272, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332443]
• 47: Ullmer, B. and Ishii, H. Emerging Frameworks for Tangible User Interfaces. Carroll, J.M. ed. Human-Computer Interaction in the New Millenium, Addison-Wesley/ACM Press, Reading, Mass., 2001.
• 48: Brygg Ullmer , Hiroshi Ishii , Robert J. K. Jacob, Token+constraint systems for tangible interaction with digital information, ACM Transactions on Computer-Human Interaction (TOCHI), v.12 n.1, p.81-118, March 2005 [doi>10.1145/1057237.1057242]
• 49: John Underkoffler , Hiroshi Ishii, Urp: a luminous-tangible workbench for urban planning and design, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.386-393, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303114]
• 50: Martin Usoh , Kevin Arthur , Mary C. Whitton , Rui Bastos , Anthony Steed , Mel Slater , Frederick P. Brooks, Jr., Walking > walking-in-place > flying, in virtual environments, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.359-364, July 1999 [doi>10.1145/311535.311589]
• 51: Andries van Dam, Post-WIMP user interfaces, Communications of the ACM, v.40 n.2, p.63-67, Feb. 1997 [doi>10.1145/253671.253708]
• 52: Norman G. Vinson, Design guidelines for landmarks to support navigation in virtual environments, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.278-285, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303062]
• 53: Daniel Vogel , Ravin Balakrishnan, Distant freehand pointing and clicking on very large, high resolution displays, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095041]
• 54: Warren, J. A Wii Workout: When Videogames Hurt Wall Street Journal Online, 2006.
• 55: Weiser, M. The Computer for the Twenty-first Century Scientific American, 1991, 94--104.
• 56: Jamie Zigelbaum , Michael S. Horn , Orit Shaer , Robert J. K. Jacob, The tangible video editor: collaborative video editing with active tokens, Proceedings of the 1st international conference on Tangible and embedded interaction, February 15-17, 2007, Baton Rouge, Louisiana [doi>10.1145/1226969.1226978]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

General Terms:
Design

Keywords:
context-aware, interaction styles, multimodal, next-generation, post-wimp interfaces, reality-based interaction, tangible interfaces, ubiquitous computing, virtual reality