Designing context-aware multimodal virtual environments

ABSTRACT

Despite of decades of research, creating intuitive and easy to learn interfaces for 3D virtual environments (VE) is still not obvious, requiring VE specialists to define, implement and evaluate solutions in an iterative way, often using low-level programming code. Moreover, quite frequently the interaction with the virtual environment may also vary dependent on the context in which it is applied, such as the available hardware setup, user experience, or the pose of the user (e.g. sitting or standing). Lacking other tools, the context-awareness of an application is usually implemented in an ad-hoc manner, using low-level programming, as well. This may result in code that is difficult and expensive to maintain. One possible approach to facilitate the process of creating these highly interactive user interfaces is by adopting a model-based user interface design. This lifts the creation of a user interface to a higher level allowing the designer to reason more in terms of high-level concepts, rather than writing programming code. In this paper, we adopt a model-based user interface design (MBUID) process for the creation of VEs, and explain how a context system using an Event-Condition-Action paradigm is added. We illustrate our approach by means of a case study.

REFERENCES

• 1: Matthias Baldauf , Schahram Dustdar , Florian Rosenberg, A survey on context-aware systems, International Journal of Ad Hoc and Ubiquitous Computing, v.2 n.4, p.263-277, June 2007 [doi>10.1504/IJAHUC.2007.014070]
• 2: W. Beer, V. Christian, A. Ferscha, and L. Mehrmann. Modeling Context-aware Behavior by Interpreted ECA Rules. volume 2790, pages 1064--1073. Springer.
• 3: Doug A. Bowman , Ernst Kruijff , Joseph J. LaViola , Ivan Poupyrev, 3D User Interfaces: Theory and Practice, Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, 2004
• 4: D. Carr. Interaction object graphs: An executable graphical notation for specifying user interfaces. In Formal Methods for Computer-Human Interaction, pages 141--156. Springer-Verlag, 1997.
• 5: T. Clerckx. Model-Based Development of Context-Aware Interactive Applications in Ambient Intelligence Environments. PhD thesis, transnationale Universiteit Limburg, June 2007.
• 6: K. Coninx, E. Cuppens, J. De Boeck, and C. Raymaekers. Integrating support for usability evaluation into high level interaction descriptions with NiMMiT. In Proceedings of 13th International Workshop on Design, Specification and Verification of Interactive Systems (DSVIS'06), volume 4385, Dublin, Ireland, July 26-28 2006.
• 7: J. Coutaz and G. Rey. Foundations for a Theory of Contextors. In C. Kolski and J. Vanderdonckt, editors, Computer-Aided Design of User Interfaces III, volume 3, pages 13--33. Kluwer Academic, 2002. Invited talk.
• 8: J. De Boeck, C. Raymaekers, and K. Coninx. A tool supporting model based user interface design in 3d virtual environments. In Proceedings of the International Conference on Computer Graphics Theory and Applications (GRAPP08), Funchal, Portugal, January 22-25 2008.
• 9: J. De Boeck, D. Vanacken, C. Raymaekers, and K. Coninx. High-level modeling of multimodal interaction techniques using nimmit. Journal of Virtual Reality and Broadcasting, 4(2), September 2007. urn:nbn:de:0009-6-11615.
• 10: Anind Kumar Dey , Gregory D. Abowd, Providing architectural support for building context-aware applications, Georgia Institute of Technology, Atlanta, GA, 2000
• 11: Pierre Dragicevic , Jean-Daniel Fekete, Support for input adaptability in the ICON toolkit, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027969]
• 12: Bruno Dumas , Denis Lalanne , Dominique Guinard , Reto Koenig , Rolf Ingold, Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces, Proceedings of the 2nd international conference on Tangible and embedded interaction, February 18-20, 2008, Bonn, Germany [doi>10.1145/1347390.1347403]
• 13: R. Etter, P. Costa, and T. Broens. A Rule-Based Approach Towards Context-Aware User Notification Services. pages 281--284, 2006.
• 14: Vitor Fernandes , Tiago Guerreiro , Bruno Araújo , Joaquim Jorge , João Pereira, Extensible middleware framework for multimodal interfaces in distributed environments, Proceedings of the 9th international conference on Multimodal interfaces, November 12-15, 2007, Nagoya, Aichi, Japan [doi>10.1145/1322192.1322230]
• 15: Pablo Figueroa , Mark Green , H. James Hoover, InTml: a description language for VR applications, Proceedings of the seventh international conference on 3D Web technology, p.53-58, February 24-28, 2002, Tempe, Arizona, USA [doi>10.1145/504502.504511]
• 16: Frans Flippo , Allen Krebs , Ivan Marsic, A framework for rapid development of multimodal interfaces, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958455]
• 17: S. Irawati, S. Ahn, J. Kim, and H. Ko. VARU Framework: Enabling Rapid Prototyping of VR, AR and Ubiquitous Applications. In Virtual Reality Conference, 2008. VR'08. IEEE, pages 201--208, 2008.
• 18: R. Kernchen, P. Boda, K. Moessner, B. Mrohs, M. Boussard, and G. Giuliani. Multimodal user interfaces for context-aware mobile applications. In 16th Annual IEEE International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), pages 2268--2273, 2005.
• 19: J. Lee, G. Rhee, H. Kim, K. Lee, Y. Suh, and K. Kim. Convergence of Context-Awareness and Augmented Reality for Ubiquitous Services and Immersive Interactions. In Computational Science and Its Applications - ICCSA 2006, pages 466--474. Springer, 2006.
• 20: Chao Li , Katharine Willis, Modeling context aware interaction for wayfinding using mobile devices, Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, September 12-15, 2006, Helsinki, Finland [doi>10.1145/1152215.1152235]
• 21: Blair MacIntyre , Steven Feiner, Language-level support for exploratory programming of distributed virtual environments, Proceedings of the 9th annual ACM symposium on User interface software and technology, p.83-94, November 06-08, 1996, Seattle, Washington, United States [doi>10.1145/237091.237104]
• 22: Mario Gutierrez , Daniel Thalmann , Frederic Vexo, Semantic Virtual Environments with Adaptive Multimodal Interfaces, Proceedings of the 11th International Multimedia Modelling Conference, p.277-283, January 12-14, 2005 [doi>10.1109/MMMC.2005.65]
• 23: D. Navarre, P. Palanque, R. Bastide, A. Schyn, M. Winckler, L. Nedel, and C. Freitas. A formal description of multimodal interaction techniques for immersive virtual reality applications. In Proceedings of Tenth IFIP TC13 International Conference on Human-Computer Interaction, Rome, IT, September 12-16 2005.
• 24: J. M. S. O., J. Serrat, K. Yang, and E. S. C. Modelling context information for managing pervasive network services. In Proc. of the International Conference on Modelling and Simulation (ICMS '05), pages 35--39, 2005.
• 25: D. Preuveneers, J. Van den Bergh, D. Wagelaar, A. Georges, P. Rigole, T. Clerckx, Y. Berbers, K. Coninx, V. Jonckers, and K. D. Bosschere. Towards an Extensible Context Ontology for Ambient Intelligence. In P. Markopoulos, B. Eggen, E. Aarts, and J. L. Crowley, editors, Second European Symposium on Ambient Intelligence, volume 3295 of LNCS, pages 148--59, Eindhoven, The Netherlands, Nov 8 - 11 2004. Springer.
• 26: C. Raymaekers, K. Coninx, J. D. Boeck, E. Cuppens, and E. Flerackers. High-vel interaction modelling to facilitate the development of virtual environments. 2004 May 12-14. Proceedings of Virtual Reality International Conference, Laval, FR.
• 27: Cyril Rousseau , Yacine Bellik , Frédéric Vernier, Multimodal output specification / simulation platform, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088480]
• 28: Marcos Serrano , Laurence Nigay , Jean-Yves L. Lawson , Andrew Ramsay , Roderick Murray-Smith , Sebastian Denef, The openinterface framework: a tool for multimodal interaction., CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358881]
• 29: Timothy Sohn , Anind Dey, iCAP: an informal tool for interactive prototyping of context-aware applications, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.766102]
• 30: L. Vanacken, E. Cuppens, T. Clerckx, and K. Coninx. Extending a dialog model with contextual knowledge. In M. Winckler, H. Johnson, and P. A. Palanque, editors, TAMODIA, volume 4849 of Lecture Notes in Computer Science, pages 28--41. Springer, 2007.
• 31: James S. Willans , Michael D. Harrison, A toolset supported approach for designing and testing virtual environment interaction techniques, International Journal of Human-Computer Studies, v.55 n.2, p.145-165, August 2001 [doi>10.1006/ijhc.2001.0474]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Design, Human Factors

Keywords:
context-awareness, model-based user interface design, multimodal interaction techniques