Multimodal user input patterns in a non-visual context

ABSTRACT

How will users choose between speech and hand inputs to perform tasks when they are given equivalent choices between both modalities in a non-visual interface? This exploratory study investigates this question. The study was conducted using AudioBrowser, a non-visual information access for the visually impaired. Findings include: (1) Users chose between input modalities based on the type of operations undertaken. Navigation operations primarily used hand input on the touchpad, while non-navigation instructions primarily used speech input. (2) Surprisingly, multimodal error correction was not prevalent. Repeating a failed operation until it succeeded and trying other methods in the same input modality were dominant error-correction strategies. (3) The modality learned first was not necessarily the primary modality used later, but a training order effect existed. These empirical results provide implications for designing non-visual multimodal input dialogues.

REFERENCES

• 1: McNeill, D. (Ed.) Language and Gesture. Cambridge: Cambridge University Press, MA, 2000.
• 2: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 3: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 4: Sears, A., Feng, J., and Oseitutu, K., Hands-Free, Speech-Based Navigation during Dictation: Difficulties, Consequences, and Solutions, Human-Computer Interaction, 18 (2003), 229--257.
• 5: Bernhard Suhm , Brad Myers , Alex Waibel, Multimodal error correction for speech user interfaces, ACM Transactions on Computer-Human Interaction (TOCHI), v.8 n.1, p.60-98, March 2001 [doi>10.1145/371127.371166]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Design, Human Factors

Keywords:
accessibility, dialogue design, interfaces for the visually impaired, non-visual multimodal interaction