Some virtues and limitations of action inferring interfaces

ABSTRACT

An action inferring facility for a multimodal interface called Edward is described. Based on the actions the user performs, Edward anticipates future actions and offers to perform them automatically. The system uses inductive inference to anticipate actions. It generalizes over arguments and results, and detects patterns on the basis of a small sequence of user actions, e.g. “copy a lisp file; change extension of original file into .org; put the copy in the backup folder”. Multimodality (particularly the combination of natural language and simulated pointing gestures) and the reuse of patterns are important new features. Some possibilities and problems of action inferring interfaces in general are addressed. Action inferring interfaces are particularly useful for professional users of general-purpose applications. Such users are unable to program repetitive patterns because either the applications do not provide the facilities or the users lack the capabilities.

REFERENCES

• 1: Edwin Bos, A multimodal syntax-directed graph editor, Structure-based editors and environments, Academic Press, Inc., Orlando, FL, 1996
• 2: Bos, E., Claassen, W., & Huls, C. (forthcoming). Edward: a multimodal interface. SPIN/MMC research report. Nijmegen: NICI.
• 3: Wim Claassen, Generating Referring Expressions in a Multimodal Environment, Proceedings of the 6th International Workshop on Natural Language Generation: Aspects of Automated Natural Language Generation, p.247-262, April 05-07, 1992
• 4: Claassen, W., Bos, E., & Huls, C. (1990). The Pooh Way in human-computer interaction: towards multimodal interfaces. SPIN~MC Research report #5. Nijmegen: NICI.
• 5: Claassen, W., Bos, E., & Huls, C. (submitted). Automatic referent resolution in deictic and anaphoric expressions. Submitted to Computational Linguistics.
• 6: Claassen, W. & Huls, C. (1991). DoNaLD: A Dutch Natural Language Dialogue system. SPIN/MMC research report #11. Nijmegen: NICI.
• 7: Cristina Conati , Jon Slack, Accessing information through graphics, Proceedings of the 10th European conference on Artificial intelligence, p.759-763, August 1992, Vienna, Austria
• 8: Allen Cypher, EAGER: programming repetitive tasks by example, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.33-39, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108850]
• 9: Cypher, A. (1992). Personal communication.
• 10: Forsyth, R. (Ed.) (1989). Machine learning. London: Chapman and Hall.
• 11: Frohlich, D.M. (1991) The design space of interfaces. In: Kjelldahl, L. (Ed.) Multimedia: principles, systems and applications. Berlin: Springer-Verlag.
• 12: Daniel Conrad Halbert, Programming by example, 1984
• 13: Lewis, C. & Norman, D. (1986). Designing for error. In: Norman, D.A. & Draper, S.W. (Eds.) User centered system design: new perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
• 14: David L. Maulsby , Ian H. Witten , Kenneth A. Kittlitz, Metamouse: specifying graphical procedures by example, Proceedings of the 16th annual conference on Computer graphics and interactive techniques, p.127-136, July 1989
• 15: Brad A. Myers, Creating user interfaces by demonstration, Academic Press Professional, Inc., San Diego, CA, 1988
• 16: Myers, B.A. (1990). Demonstrational interfaces: a step beyond direct manipulation. In: Diaper, D. & Hammond, N. (Eds.) People and computers VI. Cambridge: Cambridge University Press.
• 17: Brad A. Myers , Allen Cypher , David Maulsby , David C. Smith , Ben Shneiderman, Demonstrational interfaces: Coming soon?, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.393-396, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108966]
• 18: Norman, D.A. (1986). Cognitive engineering. In: Norman, D.A. & Draper, S.W. (Eds.) User centered system design: new perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
• 19: Richard M. Stallman, EMACS: The Extensible, Customizable, Self-Documenting Display Editor, Massachusetts Institute of Technology, Cambridge, MA, 1979

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Induction

General Terms:
Algorithms, Human Factors, Languages

Keywords:
demonstrational interfaces, multimodal interfaces, programming by example