Conversing with the user based on eye-gaze patterns

ABSTRACT

Motivated by and grounded in observations of eye-gaze patterns in human-human dialogue, this study explores using eye-gaze patterns in managing human-computer dialogue. We developed an interactive system, iTourist, for city trip planning, which encapsulated knowledge of eye-gaze patterns gained from studies of human-human collaboration systems. User study results show that it was possible to sense users' interest based on eye-gaze patterns and manage computer information output accordingly. Study participants could successfully plan their trip with iTourist and positively rated their experience of using it. We demonstrate that eye-gaze could play an important role in managing future multimodal human-computer dialogues.

REFERENCES

• 1: Allopenna, P.D., Magnuson, J.S. and Tanenhaus, M.K. Tacking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38, (1998). 419--439.
• 2: Argyle, M. and Graham, J. The central Europe experiment - Looking at persons and looking at things. Journal of Environmental Psychology and Nonverbal Behaviour, 1, (1977). 6--16.
• 3: Richard A. Bolt, Eyes at the interface, Proceedings of the 1982 conference on Human factors in computing systems, p.360-362, March 15-17, 1982, Gaithersburg, Maryland, United States [doi>10.1145/800049.801811]
• 4: Christopher S. Campbell , Paul P. Maglio, A robust algorithm for reading detection, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971503]
• 5: Cooper, R.M. The control of eye fixation by the meaning of spoken language - a new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, (1974). 84--107.
• 6: Greg Edwards, A tool for creating eye-aware applications that adapt to changes in user behaviors, Proceedings of the third international ACM conference on Assistive technologies, p.67-74, April 15-17, 1998, Marina del Rey, California, United States [doi>10.1145/274497.274511]
• 7: Goldberg, J.H. and Schryver, J.C. Eye-gaze determination of user intent at the computer interface. In Findlay, J.M., Walker, R. and Kentridge, R.W. (ed). Eye Movement Research -- Mechanisms, Processes and Applications, Elsevier Science, New York (1995).
• 8: Griffin, Z.M. and Bock, K. What the eye says about speaking. Psychological Science, 11, 4 (2000). 274--279.
• 9: Aulikki Hyrskykari , Päivi Majaranta , Antti Aaltonen , Kari-Jouko Räihä, Design issues of iDICT: a gaze-assisted translation aid, Proceedings of the 2000 symposium on Eye tracking research & applications, p.9-14, November 06-08, 2000, Palm Beach Gardens, Florida, United States [doi>10.1145/355017.355019]
• 10: Hyrskykari, A., Majaranta, P. and Räihä, K.-J., Proactive response to eye movements. Proc. INTERACT - IFIP Conference on Human-Computer Interaction (2003), 129--136.
• 11: Robert J. K. Jacob, What you look at is what you get: eye movement-based interaction techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.11-18, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97246]
• 12: Manpreet Kaur , Marilyn Tremaine , Ning Huang , Joseph Wilder , Zoran Gacovski , Frans Flippo , Chandra Sekhar Mantravadi, Where is "it"? Event Synchronization in Gaze-Speech Input Systems, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958463]
• 13: Kendon Some function of gaze direction in social interaction. Acta Psychologica, 32, (1967). 1--25.
• 14: Michael Li , Ted Selker, Eye Pattern Analysis in Intelligent Virtual Agents, Proceedings of the Third International Workshop on Intelligent Virtual Agents, p.23-35, September 10-11, 2001
• 15: Paul P. Maglio , Rob Barrett , Christopher S. Campbell , Ted Selker, SUITOR: an attentive information system, Proceedings of the 5th international conference on Intelligent user interfaces, p.169-176, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325821]
• 16: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 17: Qvarfordt, P. Eyes on multimodal interaction (Ph.D. Thesis) Department of Computer and Information Science, Linkoping University, Linkoping Studies in Science and Technology Dissertation No. 893 (2004).
• 18: Dario D. Salvucci, Inferring intent in eye-based interfaces: tracing eye movements with process models, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.254-261, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303055]
• 19: John L. Sibert , Mehmet Gokturk , Robert A. Lavine, The reading assistant: eye gaze triggered auditory prompting for reading remediation, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.101-107, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354418]
• 20: India Starker , Richard A. Bolt, A gaze-responsive self-disclosing display, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.3-10, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97245]
• 21: Katsumi Tanaka, A robust selection system using real-time multi-modal user-agent interactions, Proceedings of the 4th international conference on Intelligent user interfaces, p.105-108, January 05-08, 1999, Los Angeles, California, United States [doi>10.1145/291080.291099]
• 22: Tanenhaus, M.K., Magnuson, J.S., Dahan, D. and Chambers, C. Eye movements and lexical access in spoken-language comprehension: Linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistics Research, 29, 6 (2000). 557--580.
• 23: Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K.M. and Sedivy, J.C. Integration of visual and linguistic information in spoken language comprehension. Science, 268, 5217 (1995). 1635--1634.
• 24: Velichkovsky, B.M. Communicating attention-gaze position transfer in cooperative problem solving. Pragmatics and Cognition, 3, 2 (1995). 99--224.
• 25: Roel Vertegaal, The GAZE groupware system: mediating joint attention in multiparty communication and collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.294-301, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303065]
• 26: Roel Vertegaal , Robert Slagter , Gerrit van der Veer , Anton Nijholt, Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes, Proceedings of the SIGCHI conference on Human factors in computing systems, p.301-308, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365119]
• 27: Colin Ware , Harutune H. Mikaelian, An evaluation of an eye tracker as a device for computer input2, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.183-188, April 05-09, 1987, Toronto, Ontario, Canada
• 28: Shumin Zhai, What's in the eyes for attentive input, Communications of the ACM, v.46 n.3, March 2003 [doi>10.1145/636772.636795]
• 29: Shumin Zhai , Carlos Morimoto , Steven Ihde, Manual and gaze input cascaded (MAGIC) pointing, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.246-253, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303053]
• 30: Qiaohui Zhang , Atsumi Imamiya , Kentaro Go , Xiaoyang Mao, Overriding errors in a speech and gaze multimodal architecture, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964527]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Keywords:
dialogue systems, eye tracking, interest detection, multimodal interaction