Shared reality: spatial intelligence in intuitive user interfaces

ABSTRACT

In this paper, we describe an interface that demonstrates spatial intelligence. This interface, an embodied conversational kiosk, builds on research in embodied conversational agents (ECAs) and on information displays in mixed reality and kiosk format. ECAs leverage people's abilities to coordinate information displayed in multiple modalities, particularly information conveyed in speech and gesture. Mixed reality depends on users' interactions with everyday objects that are enhanced with computational overlays. We describe an implementation, MACK (Media lab Autonomous Conversational Kiosk), an ECA who can answer questions about and give directions to the MIT Media Lab's various research groups, projects and people. MACK uses a combination of speech, gesture, and indications on a normal paper map that users place on a table between themselves and MACK. Research issues involve users' differential attention to hand gestures, speech and the map, and how reference using these modalities can be fused in input and generation.

REFERENCES

• 1: J. Cassell , T. Bickmore , M. Billinghurst , L. Campbell , K. Chang , H. Vilhjálmsson , H. Yan, Embodiment in conversational interfaces: Rea, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.520-527, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303150]
• 2: Cassell, J., T. Bickmore, L. Campbell, H. Vilhjalmsson, and H. Yan. "More Than Just a Pretty Face: Conversational Protocols and the Affordances of Embodiment." Knowledge- Based systems 14 (2001): 55-64.
• 3: Justine Cassell , Hannes Högni Vilhjálmsson , Timothy Bickmore, BEAT: the Behavior Expression Animation Toolkit, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.477-486, August 2001 [doi>10.1145/383259.383315]
• 4: Chang, J. "Action Scheduling in Humanoid Conversational Agents." M.S. Thesis in Electrical Engineering and Computer Science. Cambridge, MA: MIT (1998).
• 5: Glass, J. and E. Weinstein. "SpeechBuilder: Facilitating Spoken Dialogue System Development." Proc. EuroSpeech '01. Aarlborg, Denmark (September 2001).
• 6: Johnson, W., J. Rickel, and J. Lester. "Animated pedagogical agents: Face-to-face interaction in interactive learning environments." International Journal of Artificial Intelligence in Education 11 (2000): 47-78.
• 7: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 8: Rickel, J., N. Lesh, C. Rich, C.L. Sidner, and A. Gertner. "Building a Bridge between Intelligent Tutoring and Collaborative Dialogue Systems." Proc. Tenth International Conference on AI in Education (2001): 592-594. IOS Press.
• 9: Steiger, P and B. Ansel Suter. "MINELLI - Experiences with an Interactive Information Kiosk for Casual Users." Proc. UBILAB '94, Zurich (1994): 124-133.
• 10: Tversky, B. "Spatial schemas in depictions." In Spatial Schemas and Abstract Thought. Ed. M. Gattis. Cambridge, MA: MIT Press, 1999.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.5 Implementation
          Subjects: Interactive systems

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis

General Terms:
Design, Human Factors, Management, Performance, Theory

Keywords:
embodied conversational agents, interaction techniques, multimodal systems, public information kiosks