Gaze-X: adaptive affective multimodal interface for single-user office scenarios

ABSTRACT

This paper describes an intelligent system that we developed to support affective multimodal human-computer interaction (AMM-HCI) where the user's actions and emotions are modeled and then used to adapt the HCI and support the user in his or her activity. The proposed system, which we named Gaze-X, is based on sensing and interpretation of the human part of the computer's context, known as W5+ (who, where, what, when, why, how). It integrates a number of natural human communicative modalities including speech, eye gaze direction, face and facial expression, and a number of standard HCI modalities like keystrokes, mouse movements, and active software identification, which, in turn, are fed into processes that provide decision making and adapt the HCI to support the user in his or her activity according to his or her preferences. To attain a system that can be educated, that can improve its knowledge and decision making through experience, we use case-based reasoning as the inference engine of Gaze-X. The utilized case base is a dynamic, incrementally self-organizing event-content-addressable memory that allows fact retrieval and evaluation of encountered events based upon the user preferences and the generalizations formed from prior input. To support concepts of concurrency, modularity/scalability, persistency, and mobility, Gaze-X has been built as an agent-based system where different agents are responsible for different parts of the processing. A usability study conducted in an office scenario with a number of users indicates that Gaze-X is perceived as effective, easy to use, useful, and affectively qualitative.

REFERENCES

• 1: Brigitte Bartsch-Spörl , Mario Lenz , André Hübner, Case-Based Reasoning: Survey and Future Directions, Proceedings of the 5th Biannual German Conference on Knowledge-Based Systems: Knowledge-Based Systems - Survey and Future Directions, p.67-89, March 03-05, 1999
• 2: Nadia Bianchi-Berthouze , Christine L. Lisetti, Modeling Multimodal Expression of User's Affective Subjective Experience, User Modeling and User-Adapted Interaction, v.12 n.1, p.49-84, February 2002 [doi>10.1023/A:1013365332180]
• 3: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 4: Browne, D., Norman, M. and Riches, D. Why Build Adaptive Interfaces? In Adaptive User Interfaces. Browne, D., Totterdell, P. and Norman, M., Eds. Academic Press, London, UK, 1990, 15--57.
• 5: Conati, C. Probabilistic assessment of user's emotions in educational games. Applied Artificial Intelligence, 16, 7-8 (Aug. 2002), 555--575.
• 6: Timothy F. Cootes , Gareth J. Edwards , Christopher J. Taylor, Active Appearance Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.681-685, June 2001 [doi>10.1109/34.927467]
• 7: Li Deng , Xuedong Huang, Challenges in adopting speech recognition, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962108]
• 8: Dey, A. K., Abowd, G. D. and Salber, D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. J. Human-Computer Interaction, 16, 2-4 (Dec. 2001), 97--166.
• 9: Duric, Z., Gray, W. D., Heishman, R., Li, F., Rosenfeld, A., Schoelles, M. J., Schunn, C. and Wechsler, H. Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction. Proceedings of the IEEE, 90, 7 (July 2002), 1272--1289.
• 10: Rana El Kaliouby , Peter Robinson, Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 10, p.154, June 27-July 02, 2004
• 11: Gu, H. and Ji, Q. An automated face reader for fatigue detection. In Proc. Int'l Conf. Face & Gesture Recognition, 111--116, 2004.
• 12: Gunes, H. and Piccardi, M. Affect Recognition from Face and Body: Early Fusion vs. Late Fusion, In Proc. Int'l Conf. Systems, Man and Cybernetics, 3437--3443, 2005.
• 13: Donna L. Hoffman , Thomas P. Novak , Alladi Venkatesh, Has the Internet become indispensable?, Communications of the ACM, v.47 n.7, p.37-42, July 2004 [doi>10.1145/1005817.1005818]
• 14: Eva Hudlicka, To feel or not to feel: the role of affect in human-computer interaction, International Journal of Human-Computer Studies, v.59 n.1-2, p.1-32, July 2003 [doi>10.1016/S1071-5819(03)00047-8]
• 15: Eva Hudlicka , Michael D. Mcneese, Assessment of User Affective and Belief States for Interface Adaptation: Application to an Air Force Pilot Task, User Modeling and User-Adapted Interaction, v.12 n.1, p.1-47, February 2002 [doi>10.1023/A:1013337427135]
• 16: Jaimes, A. and Sebe, N. Multimodal human computer interaction: A survey. In Proc. Int'l Workshop on HCI in conjunction with Int'l Conf. Computer Vision, 2005.
• 17: Ashish Kapoor , Rosalind W. Picard, Multimodal affect recognition in learning environments, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101300]
• 18: Keltner, D. and Ekman, P. Facial expression of emotion. In Handbook of Emotions, Lewis, M., and Haviland-Jones, J. M. Eds. The Guilford Press, New York, 2000, pp. 236--249.
• 19: van Kuilenburg, H., Wiering, M. and den Uyl, M. A model-based method for automatic facial expression recognition. Lecture Notes in Artificial Intelligence, 3720, 194--205, 2005.
• 20: Christine L. Lisetti , Fatma Nasoz, MAUI: a multimodal affective user interface, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641038]
• 21: Lisetti, C. L. and Schiano, D. J. Automatic facial expression interpretation: Where human-computer interaction, AI and cognitive science intersect. Pragmatics and Cognition, 8, 1 (Jan. 2000), 185--235.
• 22: Marsic, I., Medl, A. and Flanagan, J. Natural communication with information systems. Proceedings of the IEEE, 88, 8 (Aug. 2000), 1354--1366.
• 23: Jakob Nielsen, Multimedia and hypertext: the Internet and beyond, Academic Press Professional, Inc., San Diego, CA, 1995
• 24: Harriet J. Nock , Giridharan Iyengar , Chalapathy Neti, Multimodal processing by finding common cause, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962105]
• 25: Oviatt, S. User-centred modelling and evaluation of multimodal interfaces. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1457--1468.
• 26: Pantic, M., Grootjans, R. J. and Zwitserloot, R. Teaching Ad-hoc Networks using a Simple Agent Framework. In Proc. Int'l Conf. Information Technology Based Higher Education and Training, pp. 6--11, 2005.
• 27: Maja Pantic , Alex Pentland , Anton Nijholt , Thomas Huang, Human computing and machine understanding of human behavior: a survey, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181044]
• 28: Pantic, M. and Rothkrantz, L. J. M. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1370--1390.
• 29: Pantic, M. and Rothkrantz, L. J. M. Case-based reasoning for user-profiled recognition of emotions from face images. In Proc. Int'l Conf. Multimedia and Expo, 2004, 391--394.
• 30: Maja Pantic , Nicu Sebe , Jeffrey F. Cohn , Thomas Huang, Affective multimodal human-computer interaction, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101299]
• 31: Alex Pentland, Perceptual user interfaces: perceptual intelligence, Communications of the ACM, v.43 n.3, p.35-44, March 2000 [doi>10.1145/330534.330536]
• 32: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 33: Porta, M. Vision-based user interfaces: methods and applications. Int'l J. Human-Computer Studies, 57, 1 (July 2002), 27--73.
• 34: Prendinger, H. and Ishizuka, M. The empathic companion: A character-based interface that addresses users' affective states. Applied Artificial Intelligence, 19, 3-4 (Mar. 2005), 267--285.
• 35: Leah M. Reeves , Jennifer Lai , James A. Larson , Sharon Oviatt , T. S. Balaji , Stéphanie Buisine , Penny Collings , Phil Cohen , Ben Kraal , Jean-Claude Martin , Michael McTear , TV Raman , Kay M. Stanney , Hui Su , Qian Ying Wang, Guidelines for multimodal user interface design, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962106]
• 36: Zsófia Ruttkay , Catherine Pelachaud, From brows to trust: evaluating embodied conversational agents, Kluwer Academic Publishers, Norwell, MA, 2004
• 37: Schank, R. C. Memory based expert systems. AFOSR.TR. 84-0814, Comp. Science Dept., Yale University, 1984.
• 38: Ben Shneiderman, Universal usability, Communications of the ACM, v.43 n.5, p.84-91, May 2000 [doi>10.1145/332833.332843]
• 39: Sharma, R., Yeasin, M., Krahnstoever, N., Rauschert, I., Cai, G., Maceachren, A. M. and Sengupta, K. Speech-gesture driven multimodal interfaces for crisis management. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1327--1354.
• 40: David Tennenhouse, Proactive computing, Communications of the ACM, v.43 n.5, p.43-50, May 2000 [doi>10.1145/332833.332837]
• 41: Matthew Turk, Computer vision in the interface, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962107]
• 42: Vo, M. T. and Waibel, A. Multimodal human-computer interaction. In Proc. Int'l Symposium on Spoken Dialogue, 1993.
• 43: Alex Waibel , Minh Tue Vo , Paul Duchnowski , Stefan Manke, Multimodal interfaces, Artificial Intelligence Review, v.10 n.3-4, p.299-319, Aug. 1996 [doi>10.1007/BF00127684]
• 44: Ping Zhang , Na Li, The importance of affective quality, Communications of the ACM, v.48 n.9, p.105-108, September 2005 [doi>10.1145/1081992.1081997]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications

General Terms:
Design, Experimentation, Human Factors, Performance

Keywords:
affective computing, facial expressions, multimodal interfaces