A conceptual framework for developing adaptive multimodal applications

ABSTRACT

This article presents FAME, a model-based Framework for Adaptive Multimodal Environments. FAME proposes an architecture for adaptive multimodal applications, a new way to represent adaptation rules - the behavioral matrix - and a set of guidelines to assist the design process of adaptive multimodal applications. To demonstrate FAME's validity, the development process of an adaptive Digital Talking Book player is summarized.

REFERENCES

• 1: Eric Blechschmitt , Christoph Strödecke, An architecture to provide adaptive, synchronized and multimodal human computer interaction, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641066]
• 2: Jullien Bouchet , Laurence Nigay , Thierry Ganille, ICARE software components for rapidly developing multimodal interfaces, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027975]
• 3: Gaelle Calvary , Joëlle Coutaz , David Thevenin , Quentin Limbourg , Nathalie Souchon , Laurent Bouillon , Murielle Florins , Jean Vanderdonckt, Plasticity of User Interfaces: A Revised Reference Framework, Proceedings of the First International Workshop on Task Models and Diagrams for User Interface Design, p.127-134, July 18-19, 2002
• 4: Paul De Bra , Geert-Jan Houben , Hongjing Wu, AHAM: a Dexter-based reference model for adaptive hypermedia, Proceedings of the tenth ACM Conference on Hypertext and hypermedia : returning to our diverse roots: returning to our diverse roots, p.147-156, February 21-25, 1999, Darmstadt, Germany [doi>10.1145/294469.294508]
• 5: Pierre Dragicevic , Jean-Daniel Fekete, The Input Configurator toolkit: towards high input adaptability in interactive applications, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989904]
• 6: Christian Elting , Stefan Rapp , Gregor Möhler , Michael Strube, Architecture and implementation of multimodal plug and play, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958453]
• 7: Frans Flippo , Allen Krebs , Ivan Marsic, A framework for rapid development of multimodal interfaces, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958455]
• 8: A. Garg, V. PavloviĆ, and J. Rehg. Boosted learning in dynamic bayesian networks for multimodal speaker detection. Proceedings of the IEEE, 91(9):1355--1369, 2003.
• 9: David Gotz , Ketan Mayer-Patel, A general framework for multidimensional adaptation, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027671]
• 10: M. Harders and G. Székely. Enhancing human-computer interaction in medical segmentation. Proceedings of the IEEE, 91(9):1430--1442, 2003.
• 11: Charles Jacobs , Wilmot Li , Evan Schrier , David Bargeron , David Salesin, Adaptive grid-based document layout, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 12: Alfred Kobsa, Generic User Modeling Systems, User Modeling and User-Adapted Interaction, v.11 n.1-2, p.49-63, 2001 [doi>10.1023/A:1011187500863]
• 13: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 14: S. Oviatt. User-centered modeling and evaluation of multimodal interfaces. Proceedings of the IEEE, 91(9):1457--1468, 2003.
• 15: Sharon Oviatt , Rachel Coulston , Rebecca Lunsford, When do we interact multimodally?: cognitive load and multimodal communication patterns, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027957]
• 16: S. Oviatt, T. Darrell, and M. Flickner. Multimodal interfaces that flex, adapt, and persist. Commun. ACM, 47(1), 2004.
• 17: S. L. Oviatt, P. R. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro. Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. Human Computer Interaction, 15(4):263--322, 2000.
• 18: Fabio Paterno, Model-Based Design and Evaluation of Interactive Applications, Springer-Verlag, London, 1999
• 19: R. Sharma, M. Yeasin, N. Krahnstoever, I. Rauschert, G. Cai, I. Brewer, A. M. Maceachren, and K. Sengupta. Speech-gesture driven multimodal interfaces for crisis management. Proceedings of the IEEE, 91(9):1327--1354, 2003.
• 20: C. Stephanidis and A. Savidis. Universal access in the information society: Methods, tools and interaction technologies. Universal Access in the Information Society, 1(1):40--55, 2001.
• 21: C. Wickens and J. Hollands. Engineering Psychology and Human Performance. Prentice Hall, 1999.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

General Terms:
Design, Human Factors

Keywords:
IUI design, adaptive multimodal interfaces, behavioral matrix, digital talking books