Multimodal event parsing for intelligent user interfaces

ABSTRACT

Many intelligent interfaces must recognize patterns of user activity that cross a variety of different input channels. These multimodal interfaces offer significant challenges to both the designer and the software engineer. The designer needs a method of expressing interaction patterns that has the power to capture real use cases and a clear semantics. The software engineer needs a processing model that can identify the described interaction patterns efficiently while maintaining meaningful intermediate state to aid in debugging and system maintenanceIn this paper, we describe an input model, a general recognition model, and a series of important classes of recognition parsers with useful computational characteristics; that is, we can say with some certainty how efficient the recognizers will be, and the kind of patterns the recognizers will accept. Examples illustrate the ability of these recognizers to integrate information from multiple channels across varying time intervals.

REFERENCES

• 1: James F. Allen, Maintaining knowledge about temporal intervals, Communications of the ACM, v.26 n.11, p.832-843, Nov. 1983 [doi>10.1145/182.358434]
• 2: Eclipse.org, Eclipse Integrated Development Environment. 2002, http://www.eclipse.org.
• 3: Firby, R.J., et al. An architecture for vision and action. Proceedings of International Joint Conference on Artificial Intelligence. 1995.
• 4: William Alan Fitzgerald, Building embedded conceptual parsers, Northwestern University, Evanston, IL, 1994
• 5: W. Fitzgerald , R. J. Firby, The Dynamic Predictive Memory Architecture: Integrating Language with Task Execution, Proceedings of the IEEE International Joint Symposia on Intelligence and Systems, p.202, March 21-23, 1998
• 6: Joshua Flachsbart , David Franklin , Kristian Hammond, Improving human computer interaction in a classroom environment using computer vision, Proceedings of the 5th international conference on Intelligent user interfaces, p.86-93, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325790]
• 7: Ian D. Horswill, Specialization of perceptual processes, Massachusetts Institute of Technology, Cambridge, MA, 1993
• 8: Michael Johnston, Unification-based multimodal parsing, Proceedings of the 36th annual meeting on Association for Computational Linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 9: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 10: Martin, C.E., Case-based parsing and Micro-DMAP, in Inside Case-Based Reasoning, C.K. Riesbeck and R.C. Schank, Editors. 1989, Lawrence Erlbaum Associates: Hillsdale, NJ.
• 11: Martin, D.L., Cheyer, A.J., and Moran, D.B., The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence, 1999. 13: 91--128.
• 12: Douglas B. Moran , Adam J. Cheyer , Luc E. Julia , David L. Martin , Sangkyu Park, Multimodal user interfaces in the Open Agent Architecture, Proceedings of the 2nd international conference on Intelligent user interfaces, p.61-68, January 06-09, 1997, Orlando, Florida, United States [doi>10.1145/238218.238290]
• 13: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 14: Debra Schreckenghost , Carroll Thronesbery , Peter Bonasso , David Kortenkamp , Cheryl Martin, Intelligent Control of Life Support for Space Missions, IEEE Intelligent Systems, v.17 n.5, p.24-31, September 2002 [doi>10.1109/MIS.2002.1039829]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.5 Implementation
          Subjects: Special architectures

General Terms:
Human Factors

Keywords:
CERA, event recognition, multi-modal parsing