Multimodal error correction for continuous handwriting recognition in pen-based user interfaces

ABSTRACT

In this paper, we describe a multimodal error correction mechanism. It allows the user to correct errors in continuous handwriting recognition naturally by simultaneously using pen gesture and speech. A multimodal fusion algorithm is designed to enhance recognition accuracies of handwriting and speech through cross-modal influence. We have performed preliminary evaluation experiments and the results show that this multimodal mechanism can efficiently correct the errors in continuous handwriting recognition.

REFERENCES

• 1: Coen M. H. Coen. Multimodal integration a biological view. In Proceedings of 17th International Joint Conferenceon Artificial Intelligence, pages 1417--1424. Seattle, Washington, USA, August 4-10, 2001.
• 2: Clive Frankish , Dylan Jones , Kevin Hapeshi, Decline in accuracy of automatic speech recognition as a function of time on task: fatigue or voice drift?, International Journal of Man-Machine Studies, v.36 n.6, p.797-816, June 1992 [doi>10.1016/0020-7373(92)90073-T]
• 3: C. Halverson, D.B. Horn, C. Karat and J. Karat. The beauty of errors: patterns of error correction in desktop speech systems. In Proceedings of INTERACT '99, pages 133--140, IOS Press. Edinburgh, September 1999.
• 4: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 5: J. Mankoff and G. Abowd. Error correction techniques for handwriting, speech, and other ambiguous or error prone systems. GVU Technical Report Number: GIT-GVU-99-18, 1999.
• 6: J. Mankoff, G. Abowd and S. Hudson. OOPS: a toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces. Computers and Graphics, 24: 819--834, 2000.
• 7: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 8: Sharon Oviatt, Taming recognition errors with a multimodal interface, Communications of the ACM, v.43 n.9, p.45-51, Sept. 2000 [doi>10.1145/348941.348979]
• 9: Janet C. Read , Stuart MacFarlane , Chris Casey, Oops! silly me! errors in a handwriting recognition-based text entry interface for children, Proceedings of the second Nordic conference on Human-computer interaction, October 19-23, 2002, Aarhus, Denmark [doi>10.1145/572020.572026]
• 10: Bernhard Suhm , Brad Myers , Alex Waibel, Multimodal error correction for speech user interfaces, ACM Transactions on Computer-Human Interaction (TOCHI), v.8 n.1, p.60-98, March 2001 [doi>10.1145/371127.371166]
• 11: J. Sturm and L. Boves. Effective error recovery strategies for multimodal form-filling applications. Speech Communication, 45:289--303, 2005.
• 12: Mary Zajicek , Jill Hewitt, An investigation into the use of error recovery dialogues in a user interface management system for speech recognition, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.755-760, August 27-31, 1990

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Text processing

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Signal processing
      I.5.5 Implementation
          Subjects: Interactive systems

General Terms:
Algorithms, Performance

Keywords:
handwriting recognition, multimodal fusion, pen-based user interface, speech recognition