Plan-based interfaces: keeping track of user tasks and acting to cooperate

ABSTRACT

The ability to reason about the activity of a user is crucial to the implementation of any Intelligent User Interface. If it is able to recognize what a user is doing, a computer can act to cooperate. Most computer systems limit themselves to command-response interactions-their trivial understandings of their users cannot support a more complicated interaction. However, by looking at the tasks that their users are performing and reasoning about sequences of actions, a computer system can provide a more interesting level of interaction that is more efficient and does not demand as much of its users. Furthermore, the understanding of the user's activity provides a context within which to better understand future actions and to tune the sensing systems to look and listen for the actions that the user is most likely to take next. Finally, in many domains, such computer systems can recognize user tasks and act to cooperate without requiring a deep, goal-oriented understanding. In this paper, we look at the process-based interface used in the Intelligent Classroom, focusing on how a human lecturer can control it by simply going about her presentation. Also, we look at how the general ideas have been adapted to Jabberwocky, a speech-based interface to Microsoft PowerPoint that automatically switches slides, and how they are being applied to extend the functionality of Watson, an autonomous web research tool that uses the document a user is viewing as a search context.

REFERENCES

• 1: Bobick, A.; Intille, S.; Davis, J.; Baird, F.; Pinhanez, C.; Campbell, L.; Ivanov, Y.; Schutte, A.; and Wilson, A. 1998. Design decisions for interactive environments: Evaluating the KidsRoom. In Spring Symposium on Intelligent Environments.
• 2: Shannon Bradshaw , Andrei Scheinkman , Kristian Hammond, Guiding people to information: providing an interface to a digital library using reference as a basis for indexing, Proceedings of the 5th international conference on Intelligent user interfaces, p.37-43, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325774]
• 3: Jay Budzik , Kristian J. Hammond, User interactions with everyday applications as context for just-in-time information access, Proceedings of the 5th international conference on Intelligent user interfaces, p.44-51, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325776]
• 4: Michael H. Coen, Design principles for intelligent environments, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.547-554, July 1998, Madison, Wisconsin, United States
• 5: Firby, R. J.; Kahn, R. E.; Prokopowicz, P. N.; and Swain, M. J. 1995. An architecture for vision and action. In Fourteenth International Joint Conference on Artificial Intelligence.
• 6: Joshua Flachsbart , David Franklin , Kristian Hammond, Improving human computer interaction in a classroom environment using computer vision, Proceedings of the 5th international conference on Intelligent user interfaces, p.86-93, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325790]
• 7: David Franklin, Cooperating with people: the intelligent classroom, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.555-560, July 1998, Madison, Wisconsin, United States
• 8: David Franklin , Shannon Bradshaw , Kristian Hammond, Jabberwocky: you don't have to be a rocket scientist to change slides for a hydrogen combustion lecture, Proceedings of the 5th international conference on Intelligent user interfaces, p.98-105, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325792]
• 9: Horvitz, E.; Breese, J.; Heckerman, D.; Hovel, D.; and Rommelse, K. 1998. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users. In Fourth Annual Conference on Uncertainty in Artificial Intelligence.
• 10: Eric Horvitz, Principles of mixed-initiative user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.159-166, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303030]
• 11: Lieberman, H. 1995. Letizia: An Agent That Assists Web Browsing. In Fourteenth International Joint Conference on Artificial Intelligence.
• 12: Charles Rich , Candace L. Sidner , Neal Lesh, Collagen: applying collaborative discourse theory to human-computer interaction, AI Magazine, v.22 n.4, p.15-25, Winter 2001

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Collaborative computing

General Terms:
Design, Management, Performance, Theory

Keywords:
perceptual user interfaces, plan-based intelligent user interfaces, representation of activity, task recognition