A voice and ink XML multimodal architecture for mobile e-commerce systems

ABSTRACT

This paper presents a multimodal interface architecture that combines standardized voice and ink formats to facilitate the creation of robust and efficient multimodal mobile e-Commerce systems, particularly for noisy mobile environments. The platform provides a Web interactive system for generic multimodal application development. By providing mutual disambiguation of input signals and superior error handling this architecture should broaden the spectrum of users to the general population, including permanently and temporarily disabled users. Integration of VoiceXML and InkXML provides a standard data format to facilitate Web based development and content delivery. We present a prototype platform and sample dialogues.

REFERENCES

• 1: Edgar, B. The VoiceXML Handbook, CMP Books, 2001.
• 2: Fujisaki, T, et al. Hybrid on-line handwriting recognition and optical character recognition system, U.S. Patent 6,011,865, 2000.
• 3: James A Larson , Sharon Oviatt , David L. Ferro, Designing the user interface for pen and speech multimedia applications, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632826]
• 4: Oviatt, S.L. Pen/Voice: complementary multimodal communication, in Proc. of Speech Technology '92, New York, 1992.
• 5: Oviatt, S.L. and Van Gent, R. Error resolution during multimodal human-computer interaction, in Proc. Int. Conf. on Spoken Language Processing, pp. 204--207, U. Delaware Press, 1996.
• 6: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 7: Oviatt, S.L. Multimodal interactive maps: designing for human performance, Human-Computer Interaction (special issue on Multimodal Interfaces), vol. 12, pp. 93--129, 1997.
• 8: Oviatt, S.L., Cohen, P.R., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J. and Ferro, D. Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions, Human Computer Interaction, vol. 15, no. 4, pp. 263--322, 2000.
• 9: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 10: Thomas G. Holzman, Computer-human interface solutions for emergency medical care, interactions, v.6 n.3, p.13-24, May/June 1999 [doi>10.1145/301153.301160]
• 11: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 12: Jennifer Lai , John Vergo, MedSpeak: report creation with continuous speech recognition, Proceedings of the SIGCHI conference on Human factors in computing systems, p.431-438, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258829]
• 13: David R. McGee , Philip R. Cohen , Sharon Oviatt, Confirmation in multimodal systems, Proceedings of the 36th annual meeting on Association for Computational Linguistics, p.823-829, August 10-14, 1998, Montreal, Quebec, Canada
• 14: Kay, M. Functional grammar, Proc. Fifth Annual Meeting of the Berkeley Linguistics Society, 142--158, 1979.
• 15: InkXML Documents, "http://www.easystreet.com/~lartech/InkXML/

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen); Interaction styles (e.g., commands, menus, forms, direct manipulation); User-centered design

General Terms:
Design

Keywords:
InkXML, VoiceXML, handwriting recognition, multimodal applications, mutual disambiguation, speech recognition