Mixed reality: a model of mixed interaction

ABSTRACT

Mixed reality systems seek to smoothly link the physical and data processing (digital) environments. Although mixed reality systems are becoming more prevalent, we still do not have a clear understanding of this interaction paradigm. Addressing this problem, this article introduces a new interaction model called Mixed Interaction model. It adopts a unified point of view on mixed reality systems by considering the interaction modalities and forms of multimodality that are involved for defining mixed environments. This article presents the model and its foundations. We then study its unifying and descriptive power by comparing it with existing classification schemes. We finally focus on the generative and evaluative power of the Mixed Interaction model by applying it to design and compare alternative interaction techniques in the context of RAZZLE, a mobile mixed reality game for which the goal of the mobile player is to collect digital jigsaw pieces localized in space.

REFERENCES

• 1: Michel Beaudouin-Lafon, Designing interaction, not interfaces, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989865]
• 2: Bernsen, - Taxonomy of HCI Systems: State of the Art. ESPRIT BR GRACE, delivrable 2.1, 1993.
• 3: Jullien Bouchet , Laurence Nigay , Thierry Ganille, ICARE software components for rapidly developing multimodal interfaces, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027975]
• 4: Emmanuel Dubois , Laurence Nigay , Jocelyne Troccaz, Consistency in Augmented Reality Systems, Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, p.111-122, May 11-13, 2001
• 5: Kenneth P. Fishkin, A taxonomy for and analysis of tangible interfaces, Personal and Ubiquitous Computing, v.8 n.5, p.347-358, September 2004 [doi>10.1007/s00779-004-0297-4]
• 6: George W. Fitzmaurice , Hiroshi Ishii , William A. S. Buxton, Bricks: laying the foundations for graspable user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.442-449, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223964]
• 7: Lars Erik Holmquist , Johan Redström , Peter Ljungstrand, Token-Based Acces to Digital Information, Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing, p.234-245, September 27-29, 1999, Karlsruhe, Germany
• 8: Milgram, Kishino, A Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information Systems, Vol. E77-D, No. 12.
• 9: Nigay, Coutaz, The CARE Properties and Their Impact on Software Design. Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, John Lee, AAAI Press, 1997.
• 10: Norman, Cognitive Engineering. Book chapter of User Centered System Design, New Perspectives on Human-Computer Interaction, 1986, 31--61.
• 11: H. Rex Hartson , Antonio C. Siochi , D. Hix, The UAN: a user-oriented representation for direct manipulation interface designs, ACM Transactions on Information Systems (TOIS), v.8 n.3, p.181-203, July 1990 [doi>10.1145/98188.98191]
• 12: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 13: Hiroshi Ishii , Craig Wisneski , Scott Brave , Andrew Dahley , Matt Gorbet , Brygg Ullmer , Paul Yarin, ambientROOM: integrating ambient media with architectural space, CHI 98 conference summary on Human factors in computing systems, p.173-174, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286652]
• 14: Jun Rekimoto , Katashi Nagao, The world through the computer: computer augmented interaction with real world environments, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.29-36, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215639]
• 15: Renevier, Nigay, Bouchet, Pasqualetti, Generic Interaction Techniques for Mobile Collaborative Mixed Systems. CADUI'2004, 307--320.
• 16: Orit Shaer , Nancy Leland , Eduardo H. Calvillo-Gamez , Robert J. K. Jacob, The TAC paradigm: specifying tangible user interfaces, Personal and Ubiquitous Computing, v.8 n.5, p.359-369, September 2004 [doi>10.1007/s00779-004-0298-3]
• 17: Zsolt Szalavári , Erik Eckstein , Michael Gervautz, Collaborative gaming in augmented reality, Proceedings of the ACM symposium on Virtual reality software and technology, p.195-204, November 02-05, 1998, Taipei, Taiwan [doi>10.1145/293701.293740]
• 18: Brygg Ullmer , Hiroshi Ishii, mediaBlocks: tangible interfaces for online media, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632739]
• 19: Ullmer, Ishii, Emerging Frameworks for Tangible User Interfaces, Human-Computer Interaction in the New Millenium,, John M. Carroll, ed.; August 2001, 579--601.
• 20: Ullmer, Ishii, Jacob, Tangible Query Interfaces: Physically Constrained Tokens for Manipulating Database Queries. INTERACT'03.
• 21: Brygg Ullmer , Hiroshi Ishii , Robert J. K. Jacob, Token+constraint systems for tangible interaction with digital information, ACM Transactions on Computer-Human Interaction (TOCHI), v.12 n.1, p.81-118, March 2005 [doi>10.1145/1057237.1057242]
• 22: Vernier, Nigay, A Framework for the Combination and Characterization of Output Modalities. DSVIS'00, 32--48.
• 23: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); User-centered design; Graphical user interfaces (GUI)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Design, Theory

Keywords:
augmented reality-virtuality, instrumental model, interaction modality, interaction model, mixed reality, multimodality