VoiceLabel: using speech to label mobile sensor data

ABSTRACT

Many mobile machine learning applications require collecting and labeling data, and a traditional GUI on a mobile device may not be an appropriate or viable method for this task. This paper presents an alternative approach to mobile labeling of sensor data called VoiceLabel. VoiceLabel consists of two components: (1) a speech-based data collection tool for mobile devices, and (2) a desktop tool for offline segmentation of recorded data and recognition of spoken labels. The desktop tool automatically analyzes the audio stream to find and recognize spoken labels, and then presents a multimodal interface for reviewing and correcting data labels using a combination of the audio stream, the system's analysis of that audio, and the corresponding mobile sensor data. A study with ten participants showed that VoiceLabel is a viable method for labeling mobile sensor data. VoiceLabel also illustrates several key features that inform the design of other data labeling tools.

REFERENCES

• 1: Bao, L. and Intille, S. S. 2004. Activity recognition from user-annotated acceleration data. In Proc. of Pervasive 2004.
• 2: S. Basu , B. Clarkson , A. Pentland, Smart headphones: enhancing auditory awareness through robust speech detection and source localization, Proceedings of the Acoustics, Speech, and Signal Processing, 2001. on IEEE International Conference, p.3361-3364, May 07-11, 2001 [doi>10.1109/ICASSP.2001.940379]
• 3: Jeff A. Bilmes , Xiao Li , Jonathan Malkin , Kelley Kilanski , Richard Wright , Katrin Kirchhoff , Amarnag Subramanya , Susumu Harada , James A. Landay , Patricia Dowden , Howard Chizeck, The vocal joystick: a voice-based human-computer interface for individuals with motor impairments, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.995-1002, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220700]
• 4: Chen, M, Sohn, T., Chmelev, D., Haehnel, D., Hightower, J., Hughes, J., and LaMarca, A. 2006. Practical metropolitan-scale positioning for GSM phones. In Proc. of Ubicomp 2006.
• 5: Chen, S., Gaur, A., Muthukrishnan, S., and Rosenbluth, D. 2004. Wireless in loco sensor data collection and applications. In Proc. of WWW 2004 Workshop on Emerging Applications for Wireless and Mobile Access.
• 6: Sunny Consolvo , David W. McDonald , Tammy Toscos , Mike Y. Chen , Jon Froehlich , Beverly Harrison , Predrag Klasnja , Anthony LaMarca , Louis LeGrand , Ryan Libby , Ian Smith , James A. Landay, Activity sensing in the wild: a field trial of ubifit garden, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357335]
• 7: V. Fontaine , H. Bourlard, Speaker-Dependent Speech Recognition Based on Phone-Like Units Models - Application to Voice Dialing, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2, p.1527, April 21-24, 1997
• 8: Goto, M., Itou, K., and Kobayashi, T. 2005. Speech interface exploiting intentionally--controlled nonverbal speech information. In Proc. of UIST 2005.
• 9: Jason Hill , Robert Szewczyk , Alec Woo , Seth Hollar , David Culler , Kristofer Pister, System architecture directions for networked sensors, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.93-104, November 2000, Cambridge, Massachusetts, United States
• 10: Lamming, M., and Bohm, D. 2003. SPECs: Another approach to human context and activity sensing research, using tiny peer-to-peer wireless computers. In Proc. of Ubicomp 2003.
• 11: Lester, J., Choudhury, T., Kern, N., Borriello, G., and Hannaford, B. 2005. A hybrid discriminative-generative approach for modeling human activities. In Proc. of IJCAI 2005.
• 12: Lester, J., Choudhury, T., and Borriello, G. 2006. A practical approach to recognizing physical activities. In Proc. of Pervasive 2006.
• 13: Logan, B., Healey, J., Philipose, M., Tapia, E.M., and Intille, S. 2007. A long-term evaluation of sensing modalities for activity recognition. In Proc. of Ubicomp 2007.
• 14: Nissenbaum, H. 1998. Protecting privacy in an information age: The problem of privacy in public. In Law and Philosophy 17 (5-6).
• 15: Michael Shilman , Desney S. Tan , Patrice Simard, CueTIP: a mixed-initiative interface for correcting handwriting errors, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166304]
• 16: Saponas, T., Lester, J., Froehlich, J, Fogarty, J., Landay, J. 2008. iLearn on the iPhone: Real-Time Human Activity Classification on Commodity Mobile Phones. University of Washington CSE Tech Report UW-CSE-08-04-02.
• 17: Tapia, E.M., Intille, S.S., and Larson, K. 2004. Activity recognition in the home setting using simple and ubiquitous sensors. In Proc. of Pervasive 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O

General Terms:
Design, Human Factors

Keywords:
data collection, machine learning, mobile devices, sensors, speech recognition