Longitudinal study of people learning to use continuous voice-based cursor control

ABSTRACT

We conducted a 2.5 week longitudinal study with five motor impaired (MI) and four non-impaired (NMI) participants, in which they learned to use the Vocal Joystick, a voice-based user interface control system. We found that the participants were able to learn the mapping between the vowel sounds and directions used by the Vocal Joystick, and showed marked improvement in their target acquisition performance. At the end of the ten session period, the NMI group reached the same level of performance as the previously measured "expert" Vocal Joystick performance, and the MI group was able to reach 70% of that level. Two of the MI participants were also able to approach the performance of their preferred device, a touchpad. We report on a number of issues that can inform the development of further enhancements in the realm of voice-driven computer control.

REFERENCES

• 1: Johnny Accot , Shumin Zhai, Beyond Fitts' law: models for trajectory-based HCI tasks, Proceedings of the SIGCHI conference on Human factors in computing systems, p.295-302, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258760]
• 2: Bates, R. Enhancing the performance of eye and head mice: a validated assessment method and an investigation into the performance of eye and head based assistive technology pointing devices. PhD Thesis, De Montfort University, May 2006.
• 3: Bates, R. and Istance, H.O. Why are eye mice unpopular? A detailed comparison of head and eye controlled assistive technology pointing devices. In Universal Access in the Information Society 2, 3 (2003), 280--290.
• 4: Jeff A. Bilmes , Xiao Li , Jonathan Malkin , Kelley Kilanski , Richard Wright , Katrin Kirchhoff , Amarnag Subramanya , Susumu Harada , James A. Landay , Patricia Dowden , Howard Chizeck, The vocal joystick: a voice-based human-computer interface for individuals with motor impairments, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.995-1002, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220700]
• 5: Capilouto, G.J., Higginbotham, D.J., McClenaghan, B., Williams, H.G., and Dickerson, J. Performance investigation of a head-operated device and expanded membrane cursor keys in a target acquisition task. In Technology and Disability 17, 3 (2005), 173--183.
• 6: Coyne, K.P. (2005) Conducting simple usability studies with users with disabilities. In Proc. HCI Int'l 2005, Lawrence Erlbaum Assoc. (2005). On proceedings CD.
• 7: Liwei Dai , Rich Goldman , Andrew Sears , Jeremy Lozier, Speech-based cursor control: a study of grid-based solutions, ACM SIGACCESS Accessibility and Computing
• 8: de Mauro, C., Gori, M., Maggini, M., and Martinelli, E. Easy access to graphical interfaces by voice mouse. Technical report, Università di Siena (2001). Available from the author at: [e-mail].
• 9: Susumu Harada , James A. Landay , Jonathan Malkin , Xiao Li , Jeff A. Bilmes, The vocal joystick:: evaluation of voice-based cursor control techniques, Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, October 23-25, 2006, Portland, Oregon, USA [doi>10.1145/1168987.1169021]
• 10: Susumu Harada , Jacob O. Wobbrock , James A. Landay, Voicedraw: a hands-free voice-driven drawing application for people with motor impairments, Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, October 15-17, 2007, Tempe, Arizona, USA [doi>10.1145/1296843.1296850]
• 11: Susumu Harada , T. Scott Saponas , James A. Landay, VoicePen: augmenting pen input with simultaneous non-linguisitic vocalization, Proceedings of the 9th international conference on Multimodal interfaces, November 12-15, 2007, Nagoya, Aichi, Japan [doi>10.1145/1322192.1322225]
• 12: Brandi House , Jonathan Malkin , Jeff Bilmes, The VoiceBot: a voice controlled robot arm, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA [doi>10.1145/1518701.1518731]
• 13: Takeo Igarashi , John F. Hughes, Voice as sound: using non-verbal voice input for interactive control, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502372]
• 14: Malkin, J., Li, X., and Bilmes, J.A. Energy and loudness for speed control in the Vocal Joystick. In IEEE Automatic Speech Recognition and Understanding Workshop, (2005), 409--414.
• 15: Yoshiyuki Mihara , Etsuya Shibayama , Shin Takahashi, The migratory cursor: accurate speech-based cursor movement by moving multiple ghost cursors using non-verbal vocalizations, Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility, October 09-12, 2005, Baltimore, MD, USA [doi>10.1145/1090785.1090801]
• 16: Mahmud, M., Sporka, A.J., Kurniawan, S.H., and Slavík, P. A comparative longitudinal study of non-verbal mouse pointer. In Proc. INTERACT 2007, Springer Berlin (2007), 489--502.
• 17: Riemer-Reiss, M.L. and Wacker, R.R. Factors associated with assistive technology discontinuance among individuals with disabilities. In Journal of Rehabilitation 66, 3 (2000), 44--50.
• 18: Andrew Sears , Mark Young, Physical disabilities and computing technologies: an analysis of impairments, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, L. Erlbaum Associates Inc., Hillsdale, NJ, 2002
• 19: J. Sporka , H. Kurniawan , Pavel Slavík, Acoustic control of mouse pointer, Universal Access in the Information Society, v.4 n.3, p.237-245, February 2006 [doi>10.1007/s10209-005-0010-z]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O

Additional Classification:
  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.2 Social Issues
          Subjects: Assistive technologies for persons with disabilities

General Terms:
Performance

Keywords:
longitudinal study, motor impairment, pointer control, speech recognition, voice-based interface