Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes

ABSTRACT

In multi-agent, multi-user environments, users as well as agents should have a means of establishing who is talking to whom. In this paper, we present an experiment aimed at evaluating whether gaze directional cues of users could be used for this purpose. Using an eye tracker, we measured subject gaze at the faces of conversational partners during four-person conversations. Results indicate that when someone is listening or speaking to individuals, there is indeed a high probability that the person looked at is the person listened (p=88%) or spoken to (p=77%). We conclude that gaze is an excellent predictor of conversational attention in multiparty conversations. As such, it may form a reliable source of input for conversational systems that need to establish whom the user is speaking or listening to. We implemented our findings in FRED, a multi-agent conversational system that uses eye input to gauge which agent the user is listening or speaking to.

REFERENCES

• 1: Argyle,M.The Psychology of Interpersonal Behaviour .London:Penguin Books,1967.
• 2: Argyle,M.Social Interaction .London:Tavistock Publications,1969.
• 3: Argyle,M.and Cook,M.Ga e and Mutual Gaze . London:Cambridge University Press,1976.
• 4: Argyle,M.and Dean,J.Eye-contact,Distance and Affiliation .Sociometry 28,1965,pp.289-304.
• 5: Argyle,M.and Graham,J.The Central Europe Experiment -Looking at Persons and Looking at Things .Journal of Environmental Psychology and Nonverbal Behaviour 1,1977,pp.6-16.
• 6: Argyle,M.,Lalljee,M.,and Cook,M.The Effects of Visibility on Interaction in a Dyad .Human Relations 21,1968,pp.3-17.
• 7: Brady,P.T.A Statistical Analysis of On-off Patterns in 16 Conversations .The Bell System Technical Journal (Jan.)1968,pp.73-91.
• 8: J. Cassell , T. Bickmore , M. Billinghurst , L. Campbell , K. Chang , H. Vilhjálmsson , H. Yan, Embodiment in conversational interfaces: Rea, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.520-527, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303150]
• 9: Jaffe J.and Feldstein S.Rhythms of Dialogue .New York:Academic Press,1970.
• 10: Kendon,A.Some Functions of Gaze Direction in Social Interaction .Acta Psychologica 32,1967,pp.1- 25.
• 11: LC Technologies,Inc.The Eyegaze Communication System .Fairfax,VA:http://www.eyegaze.com ,1997.
• 12: Mobbs,N.A.Eye-contact in Relation to Social Introversion-Extraversion .British Journal of Social Clinical Psychology 7(305-306),1968.
• 13: Nijholt,A.and Hulstijn,J.Multimodal Interactions with Agents in Virtual Worlds.In Future Directions for Intelligent Information Systems and Information Science N.Kasabov (ed.),Physica-Verlag:Studies in Fuzziness and Soft Computing 45,2000,pp.148-173.
• 14: Abigail J. Sellen, Speech patterns in video-mediated conversations, Proceedings of the SIGCHI conference on Human factors in computing systems, p.49-59, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142756]
• 15: Boris M. Velichkovsky , John Paulin Hansen, New technological windows into mind: there is more in eyes and brains for human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.496-503, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238619]
• 16: Roel Vertegaal, The GAZE groupware system: mediating joint attention in multiparty communication and collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.294-301, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303065]
• 17: Vertegaal,R.Look Who's Talking to Whom .PhD Thesis.Enschede,The Netherlands:Cognitive Ergonomics Department,Twente University,1998.
• 18: Roel Vertegaal , Robert Slagter , Gerrit van der Veer , Anton Nijholt, Why conversational agents should catch the eye, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633442]
• 19: Vertegaal,R.Van der Veer,G.C.and Vons,H. Effects of Gaze on Multiparty Mediated Communication.In Proceedings of Graphics Interface 2000.Montreal,Canada:Morgan Kaufmann Publishers,2000,pp.95-102.
• 20: Waters,K.and Frisbee,J.A Coordinated Muscle Model for Speech Animation.In Proceedings of Graphics Interface '95.Canada,1995,pp.163-170.
• 21: Weisbrod,R.M.Looking Behavior in a Discussion Group.Unpublished paper,Department of Psychology, Cornell University,1965.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Multiagent systems

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
  H.4 INFORMATION SYSTEMS APPLICATIONS
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

General Terms:
Design, Human Factors, Languages, Management, Measurement, Performance, Theory

Keywords:
attention-based interfaces, attentive agents, conversational attention, gaze, multiparty communication, tracking