VoicePen: augmenting pen input with simultaneous non-linguisitic vocalization

ABSTRACT

This paper explores using non-linguistic vocalization as an additional modality to augment digital pen input on a tablet computer. We investigated this through a set of novel interaction techniques and a feasibility study. Typically, digital pen users control one or two parameters using stylus position and sometimes pen pressure. However, in many scenarios the user can benefit from the ability to continuously vary additional parameters. Non-linguistic vocalizations, such as vowel sounds, variation of pitch, or control of loudness have the potential to provide fluid continuous input concurrently with pen interaction. We present a set of interaction techniques that leverage the combination of voice and pen input when performing both creative drawing and object manipulation tasks. Our feasibility evaluation suggests that with little training people can use non-linguistic vocalization to productively augment digital pen interaction.

REFERENCES

• 1: Ravin Balakrishnan , Ken Hinckley, The role of kinesthetic reference frames in two-handed input performance, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.171-178, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322599]
• 2: Eric A. Bier , Maureen C. Stone , Ken Pier , William Buxton , Tony D. DeRose, Toolglass and magic lenses: the see-through interface, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.73-80, September 1993 [doi>10.1145/166117.166126]
• 3: Jeff A. Bilmes , Xiao Li , Jonathan Malkin , Kelley Kilanski , Richard Wright , Katrin Kirchhoff , Amarnag Subramanya , Susumu Harada , James A. Landay , Patricia Dowden , Howard Chizeck, The vocal joystick: a voice-based human-computer interface for individuals with motor impairments, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.995-1002, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220700]
• 4: Colin G. Butler , Robert St. Amant, HabilisDraw DT: a bimanual tool-based direct manipulation drawing environment, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986049]
• 5: W. Buxton , B. Myers, A study in two-handed input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.321-326, April 13-17, 1986, Boston, Massachusetts, United States
• 6: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 7: Richard C. Davis , T. Scott Saponas , Michael Shilman , James A. Landay, SketchWizard: Wizard of Oz prototyping of pen-based user interfaces, Proceedings of the 20th annual ACM symposium on User interface software and technology, October 07-10, 2007, Newport, Rhode Island, USA [doi>10.1145/1294211.1294233]
• 8: George W. Fitzmaurice , Hiroshi Ishii , William A. S. Buxton, Bricks: laying the foundations for graspable user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.442-449, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223964]
• 9: Mark D. Gross , Ellen Yi-Luen Do, Ambiguous intentions: a paper-like interface for creative design, Proceedings of the 9th annual ACM symposium on User interface software and technology, p.183-192, November 06-08, 1996, Seattle, Washington, United States [doi>10.1145/237091.237119]
• 10: Guiard, Y. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of Motor Behavior, 19 (1987), 486--517.
• 11: Susumu Harada , James A. Landay , Jonathan Malkin , Xiao Li , Jeff A. Bilmes, The vocal joystick:: evaluation of voice-based cursor control techniques, Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, October 23-25, 2006, Portland, Oregon, USA [doi>10.1145/1168987.1169021]
• 12: Ken Hinckley , Randy Pausch , Dennis Proffitt , James Patten , Neal Kassell, Cooperative bimanual action, Proceedings of the SIGCHI conference on Human factors in computing systems, p.27-34, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258571]
• 13: Ken Hinckley , Mary Czerwinski , Mike Sinclair, Interaction and modeling techniques for desktop two-handed input, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.49-58, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288572]
• 14: Takeo Igarashi , Ken Hinckley, Speed-dependent automatic zooming for browsing large documents, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.139-148, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354435]
• 15: Takeo Igarashi , John F. Hughes, Voice as sound: using non-verbal voice input for interactive control, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502372]
• 16: Susanne Jul , George W. Furnas, Critical zones in desert fog: aids to multiscale navigation, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.97-106, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288578]
• 17: L. Julia , C. Faure, Pattern recognition and beautification for a pen based interface, Proceedings of the Third International Conference on Document Analysis and Recognition (Volume 1), p.58, August 14-15, 1995
• 18: Paul Kabbash , William Buxton , Abigail Sellen, Two-handed input in a compound task, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.417-423, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191808]
• 19: Gordon Kurtenbach , George Fitzmaurice , Thomas Baudel , Bill Buxton, The design of a GUI paradigm based on tablets, two-hands, and transparency, Proceedings of the SIGCHI conference on Human factors in computing systems, p.35-42, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258574]
• 20: Andrea Leganchuk , Shumin Zhai , William Buxton, Manual and cognitive benefits of two-handed input: an experimental study, ACM Transactions on Computer-Human Interaction (TOCHI), v.5 n.4, p.326-359, Dec. 1998 [doi>10.1145/300520.300522]
• 21: Malkin, J., Li, X., and Bilmes, J. Energy and loudness for speed control in the Vocal Joystick. In IEEE Automatic Speech Recognition and Understanding Workshop (November 2005).
• 22: Yoshiyuki Mihara , Etsuya Shibayama , Shin Takahashi, The migratory cursor: accurate speech-based cursor movement by moving multiple ghost cursors using non-verbal vocalizations, Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility, October 09-12, 2005, Baltimore, MD, USA [doi>10.1145/1090785.1090801]
• 23: Newman, M. W., Lin, J., Hong, J. I., and Landay, J. A. Denim: An informal web site design tool inspired by observations of practice. Human-Computer Interaction 18, 3 (2003), 259--324.
• 24: Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J. A., Larson, J., and Ferro, D. Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions. Human-Computer Interaction 15 (2000), 263--322.
• 25: Russell Owen , Gordon Kurtenbach , George Fitzmaurice , Thomas Baudel , Bill Buxton, When it gets more difficult, use both hands: exploring bimanual curve manipulation, Proceedings of Graphics Interface 2005, May 09-11, 2005, Victoria, British Columbia
• 26: Gonzalo Ramos , Ravin Balakrishnan, Fluid interaction techniques for the control and annotation of digital video, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.105-114, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964708]
• 27: Gonzalo A. Ramos , Ravin Balakrishnan, Pressure marks, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240834]
• 28: Gonzalo Ramos , Matthew Boulos , Ravin Balakrishnan, Pressure widgets, Proceedings of the SIGCHI conference on Human factors in computing systems, p.487-494, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985754]
• 29: Gonzalo Ramos , Ravin Balakrishnan, Zliding: fluid zooming and sliding for high precision parameter manipulation, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095059]
• 30: Roope Raisamo, An alternative way of drawing, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.175-182, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303035]
• 31: Sporka, A. J., Kurniawan, S. H., and Slavík, P. Whistling user interface (U3I). In User Interfaces for All (2004), pp. 472--478.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O

General Terms:
Design, Human Factors

Keywords:
multimodal input, pen-based interface, voice-based interface