Blindsight: eyes-free access to mobile phones

ABSTRACT

Many mobile phones integrate services such as personal calendars. Given the social nature of the stored data, however, users often need to access such information as part of a phone conversation. In typical non-headset use, this re-quires users to interrupt their conversations to look at the screen.

We investigate a counter-intuitive solution: to avoid the need for interruption we replace the visual interface with one based on auditory feedback. Surprisingly, this can be done without interfering with the phone conversation. We present blindSight, a prototype application that replaces the traditionally visual in-call menu of a mobile phone. Users interact using the phone keypad, without looking at the screen. BlindSight responds with auditory feedback. This feedback is heard only by the user, not by the person on the other end of the line.

We present the results of two user studies of our prototype. The first study verifies that useful keypress accuracy can be obtained for the phone-at-ear position. The second study compares the blindSight system against a visual baseline condition and finds a preference for blindSight.

REFERENCES

• 1: microsoft.com/windowsmobile/voicecommand/default.mspx
• 2: www.tellme.com.
• 3: A. F. Aucella , S. F. Ehrlich, Voice messaging enhancing the user interface design based on field performance, Proceedings of the SIGCHI conference on Human factors in computing systems, p.156-161, April 13-17, 1986, Boston, Massachusetts, United States
• 4: Patrick Baudisch, Don't click, paint! Using toggle maps to manipulate sets of toggle switches, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.65-66, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288574]
• 5: Blanchard, H.E. and Lewis, S.H. (1999). Voice messaging user interface. In D. Gardner-Bonneau (Ed.), Human Factors and Voice Interactive Systems, Kluwer, pp. 257--284.
• 6: Stephen A. Brewster, Using nonspeech sounds to provide navigation cues, ACM Transactions on Computer-Human Interaction (TOCHI), v.5 n.3, p.224-259, Sept. 1998 [doi>10.1145/292834.292839]
• 7: Paul H. Dietz , William S. Yerazunis, Real-time audio buffering for telephone applications, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502383]
• 8: Masaaki Fukumoto , Yoshinobu Tonomura, Whisper: a wristwatch style wearable handset, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.112-119, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303009]
• 9: William W. Gayer, The SonicFinder: An Interface that Uses Auditory Icons (Abstract Only), ACM SIGCHI Bulletin, v.21 n.1, p.124, July 1989 [doi>10.1145/67880.1046601]
• 10: David Goldberg , Cate Richardson, Touch-typing with a stylus, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.80-87, May 1993, Amsterdam, The Netherlands
• 11: Iván E. González , Jacob O. Wobbrock , Duen Horng Chau , Andrew Faulring , Brad A. Myers, Eyes on the road, hands on the wheel: thumb-based interaction techniques for input on steering wheels, Proceedings of Graphics Interface 2007, May 28-30, 2007, Montreal, Canada [doi>10.1145/1268517.1268535]
• 12: Hansen, J.H.L., Plucienkowski, J., Gallant, S., Pellom, B.L., Ward, B. CU-Move: Robust Speech Processing for In-Vehicle Speech Systems. Proc. ICSLP'00, pp. 524--527.
• 13: Hiraoka, S., Miyamoto, I., Tomimatsu, K. (2003) Behind Touch, a Text Input Method for Mobile Phones by The Back and Tactile Sense Interface. Information Processing Society of Japan, Interaction'03. p. 131--138.
• 14: Hornstein, T. 1994. Telephone voice interfaces on the cheap. UBILAB Rep, Union Bank of Switzerland, Zurich.
• 15: Scott E. Hudson , Ian Smith, Electronic mail previews using non-speech audio, Conference companion on Human factors in computing systems: common ground, p.237-238, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257299]
• 16: Mizuko Ito , Daisuke Okabe , Misa Matsuda, Personal, Portable, Pedestrian: Mobile Phones in Japanese Life, The MIT Press, 2005
• 17: Joseph Luk , Jerome Pasquero , Shannon Little , Karon MacLean , Vincent Levesque , Vincent Hayward, A role for haptics in mobile interaction: initial design using a handheld tactile display prototype, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124800]
• 18: Kent Lyons , Christopher Skeels , Thad Starner , Cornelis M. Snoeck , Benjamin A. Wong , Daniel Ashbrook, Augmenting conversations using dual-purpose speech, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029674]
• 19: Kent Lyons , Thad Starner , Daniel Plaisted , James Fusia , Amanda Lyons , Aaron Drew , E. W. Looney, Twiddler typing: one-handed chording text entry for mobile phones, Proceedings of the SIGCHI conference on Human factors in computing systems, p.671-678, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985777]
• 20: Scott MacKenzie, Mobile text entry using three keys, Proceedings of the second Nordic conference on Human-computer interaction, October 19-23, 2002, Aarhus, Denmark [doi>10.1145/572020.572025]
• 21: Marics, M., and Engelbeck, G. (1997). Designing voice menu applications for telephones, in Handbook of Human-Computer Interaction, Helander, M., Landauer, T., and Prabhu, P. Edi-tors. Elsevier, pp. 1085--1102.
• 22: Saverio Perugini , Taylor J. Anderson , William F. Moroney, A study of out-of-turn interaction in menu-based, IVR, voicemail systems, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240770]
• 23: Paul Resnick , Robert A. Virzi, Skip and scan: cleaning up telephone interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.419-426, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142881]
• 24: Rosenberger, J., and Gasco, M. (1983). Comparing location estimators: Trimmed means, medians, and trimean. in Under-standing robust and exploratory data analysis, Hoagan, D., Mosteller, F., and Tukey, J., Editors. John Wiley: New York.
• 25: Thad E. Starner , Cornelis M. Snoeck , Benjamin A. Wong , R. Martin McGuire, Use of mobile appointment scheduling devices, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986100]
• 26: Masanori Sugimoto , Keiichi Hiroki, HybridTouch: an intuitive manipulation technique for PDAs using their front and rear surfaces, Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, September 12-15, 2006, Helsinki, Finland [doi>10.1145/1152215.1152243]
• 27: Teslar, L. The Smalltalk Environment, BYTE, 6,'81, 90--147.
• 28: Simon Tucker , Steve Whittaker, Time is of the essence: an evaluation of temporal compression algorithms, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124822]
• 29: Daniel Wigdor , Clifton Forlines , Patrick Baudisch , John Barnwell , Chia Shen, Lucid touch: a see-through mobile device, Proceedings of the 20th annual ACM symposium on User interface software and technology, October 07-10, 2007, Newport, Rhode Island, USA [doi>10.1145/1294211.1294259]
• 30: Jacob O. Wobbrock , Duen Horng Chau , Brad A. Myers, An alternative to push, press, and tap-tap-tap: gesturing on an isometric joystick for mobile phone text entry, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240728]
• 31: Jacob O. Wobbrock , Brad A. Myers , John A. Kembel, EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.61-70, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964703]
• 32: Min Yin , Shumin Zhai, The benefits of augmenting telephone voice menu navigation with visual browsing and search, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124821]
• 33: Shengdong Zhao , Pierre Dragicevic , Mark Chignell , Ravin Balakrishnan , Patrick Baudisch, Earpod: eyes-free menu selection using touch input and reactive audio feedback, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240836]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices

General Terms:
Human Factors

Keywords:
devices, eyes-free, mobile phone, non-speech audio, user interfaces