Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

We discuss the design of speech-based interaction systems from the perspective of affordance. It is our claim that many of the problems associated with speech interaction stems from a lack of a deeper understanding of the communicative nature of speech in combination with graphical interfaces. Today's graphical interfaces are almost universally based on the Direct Manipulation (DM) interaction paradigm, whereas speech interaction is in nature conversational. However, most multi modal systems accepting spoken input are designed with the DM paradigm as the underlying interaction model, often forcing the spoken input to correspond to simple commands, which are replicated in the GUI anyway. We will discuss how speech-based interfaces can be designed, taking into account that the DM paradigm is not supported by speech. We demonstrate this through the design of a concrete application for a portable device (PDA), and show how an underlying architecture can be built to support both spoken input and DM interfaces.

REFERENCES

• 1: VoiceXML Forum Specification: "Mobile X+V 1.2", published 2005: http://www.voicexml.org/specs/multimodal/x+v/mobile/12/
• 2: ETSI Standard ES 202 212. "Distributed speech recognition; extended advanced front-end feature extraction algorithm; compression algorithm, back-end speech reconstruction algorithm", November 2003.
• 3: L. Almeidal, et al: "The MUST guide to Paris: Implementation and expert evaluation of a multimodal tourist guide to Paris" In proceedings of ITRW Workshop on Multi-Modal Dialogue in Mobile Environments Germany 2002
• 4: J. Sturm, et al: "Comparing the Usability of a User Driven and a Mixed Initiative Multimodal Dialogue System for Train Timetable Information" In proc. of Eurospeech 03, Geneva, Switzerland 2003
• 5: Helen Sharp , Yvonne Rogers , Jenny Preece, Interaction Design: Beyond Human Computer Interaction, John Wiley & Sons, 2007
• 6: Donald Norman, "The Psychology of Everyday Things", 1988, ISBN 0-465-06710-7
• 7: William W. Gaver, Technology affordances, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.79-84, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108856]
• 8: Hartson, H. Rex, "Cognitive, Physical, and Perceptual Affordances in Interaction Design", Behaviour & Information Technology, September-October 2003, Vol. 22, No. 5, 1999
• 9: Jaakko Hakulinen, Markku Turunen, & Esa-Pekka Salonen. "Software Tutors for Dialogue Systems". In Proceedings of Text, Speech and Dialogue (TSD 2005), LNAI 3658, Springer, pages 412--419.
• 10: Xu, H., Tan, Z.-H., Dalsgaard, P., Mattethat, R. and Lindberg, B.: A configurable distributed speech recognition system, Biennial on DSP for in-Vehicle and Mobile Systems, Sesimbra, Portugal, Sep. 2005.
• 11: The CMU Sphinx Group Open Source Speech Recognition Engines. http://cmusphinx.sourceforge.net/html/cmusphinx.php
• 12: Brøndsted, T., Larsen, H. L., Larsen, L. B., Lindberg, B., Ortiz-Arroyo, D., Tan, Z.-H., Xu, H., "Mobile Information Access with Spoken Query Answering" in COST278 Final Workshop on "Applied Spoken Language Interaction in Distributed Environments" ASIDE, Aalborg, Denmark, Nov 2005
• 13: K. L. Jensen, L. B. Larsen. S. Larsen, M. H. Rasmussen "Implementation of the CNTK Car Rental Service" Technical Report (CNTK), Aalborg University, November 2006 ISSN: 0908-1224

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Design, Experimentation, Human Factors, Theory

Keywords:
affordance, distributed speech recognition, interface design on mobile devices, multi modal user interaction, speech interaction