Toward open-microphone engagement for multiparty interactions

ABSTRACT

There currently is considerable interest in developing new open-microphone engagement techniques for speech and multimodal interfaces that perform robustly in complex mobile and multiparty field environments. State-of-the-art audio-visual open-microphone engagement systems aim to eliminate the need for explicit user engagement by processing more implicit cues that a user is addressing the system, which results in lower cognitive load for the user. This is an especially important consideration for mobile and educational interfaces due to the higher load required by explicit system engagement. In the present research, longitudinal data were collected with six triads of high-school students who engaged in peer tutoring on math problems with the aid of a simulated computer assistant. Results revealed that amplitude was 3.25dB higher when users addressed a computer rather than human peer when no lexical marker of intended interlocutor was present, and 2.4dB higher for all data. These basic results were replicated for both matched and adjacent utterances to computer versus human partners. With respect to dialogue style, speakers did not direct a higher ratio of commands to the computer, although such dialogue differences have been assumed in prior work. Results of this research reveal that amplitude is a powerful cue marking a speaker's intended addressee, which should be leveraged to design more effective microphone engagement during computer-assisted multiparty interactions.

REFERENCES

• 1: Alexander M. Arthur , Rebecca Lunsford , Matt Wesson , Sharon Oviatt, Prototyping novel collaborative multimodal systems: simulation, data collection and analysis tools for the next decade, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181039]
• 2: Boersma, P. & Weenink, D., Praat: Doing phonetics by computer (version 4.2). 2005. (URL:www.praat.org).
• 3: Coulston, R., Oviatt, S. L., & Darves, C. Amplitude convergence in children's conversational speech with animated personas. In Proceedings of the International Conference on Spoken Language Processing (ICSLP'2002), 2002, Casual Prod. Ltd., Denver, CO: 2689--2692.
• 4: Escera, C., Corral, M.-J., & Yago, E., An electrophysiological and behavioral investigation of involuntary attention towards auditory frequency, duration and intensity changes. Cognitive Brain Research, 14, 3: 325--332.
• 5: Julia Hirschberg , Barbara Grosz, Intonational features of local and global discourse structure, Proceedings of the workshop on Speech and Natural Language, February 23-26, 1992, Harriman, New York [doi>10.3115/1075527.1075632]
• 6: Julia Hirschberg , Christine H. Nakatani, A prosodic analysis of discourse segments in direction-giving monologues, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.286-293, June 24-27, 1996, Santa Cruz, California [doi>10.3115/981863.981901]
• 7: Michael Katzenmaier , Rainer Stiefelhagen , Tanja Schultz, Identifying the addressee in human-human-robot interactions based on head pose and speech, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027959]
• 8: Levow, G.-A. Prosodic cues to discourse segment boundaries in human-computer dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue 2004: 93--96.
• 9: Rebecca Lunsford , Sharon Oviatt , Rachel Coulston, Audio-visual cues distinguishing self- from system-directed speech in younger and older adults, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088494]
• 10: Messer, D. J., The identification of names in maternal speech to infants. Journal of Psycholinguistic Research, 10, 1 (January 1981): 69--77.
• 11: Neti, C., Iyengar, G., Potamianos, G., Senior, A., & Maison, B. Perceptual interfaces for information interaction: Joint processing of audio and visual information for human-computer interaction. In Proceedings of the International Conference on Spoken Language Processing, 2000 (Beijing). 3, Chinese Friendship Publishers: 11--14.
• 12: Sharon Oviatt , Courtney Darves , Rachel Coulston, Toward adaptive conversational interfaces: Modeling speech convergence with animated personas, ACM Transactions on Computer-Human Interaction (TOCHI), v.11 n.3, p.300-328, September 2004 [doi>10.1145/1017494.1017498]
• 13: Sharon Oviatt , Margaret MacEachern , Gina-Anne Levow, Predicting hyperarticulate speech during human-computer error resolution, Speech Communication, v.24 n.2, p.87-110, May 1, 1998 [doi>10.1016/S0167-6393(98)00005-3]
• 14: Paek, T., Horvitz, E., & Ringger, E. Continuous listening for unconstrained spoken dialog. In Proceedings of the International Conference on Spoken Language Processing, 2000 (Beijing, China). Chinese Freindship Publishers: 138--141.
• 15: Schroger, E., A neural mechanism for involuntary attention shifts to changes in auditory stimulation. Journal of Cognitive Neuroscience, 8, 6 (November 1996): 527--539.
• 16: Koen van Turnhout , Jacques Terken , Ilse Bakx , Berry Eggen, Identifying the intended addressee in mixed human-human and human-computer interaction from non-verbal features, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088495]
• 17: Welkowitz, J., Feldstein, S., Finklestein, M., & Aylesworth, L., Changes in vocal intensity as a function of interspeaker influence. Perceptual and Motor Skills, 35: 715--718.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Voice I/O; Natural language; Input devices and strategies (e.g., mouse, touchscreen); Evaluation/methodology; Theory and methods; Prototyping

General Terms:
Design, Human Factors, Performance, Reliability

Keywords:
collaborative peer tutoring, computer-supported collaborative work, dialogue style, intended addressee, multimodal interaction, open-microphone engagement, spoken amplitude, user communication modeling