Effects of real-time transcription on non-native speaker's comprehension in computer-mediated communications

ABSTRACT

We performed an empirical study to understand the relative contributions of real-time transcription to a non-native speaker's comprehension in audio/video meetings. 48 participants were assigned to 2 presentation modes (audio, audio+video) and 3 transcription modes (no transcript, real-time transcripts in the streaming mode, transcripts with all past records) in a 3x2 factorial experimental design. The results suggest that comprehension can be significantly improved for both audio and audio+video conditions when real-time transcription is provided. Also, the participants reported positive subjective responses to the presence of real-time transcription in terms of usefulness, preference, and willingness to use such a feature if provided. No cognitive load issues were reported by the participants in the ability to synthesize across modalities. Implications for system development and design, as well as future work utilizing automation speech recognition to provide the transcripts are discussed.

REFERENCES

• 1: Tyler, M.D. The Effect of Background Knowledge on First and Second Language Comprehension Difficulty. In Proc. of ICSLP 1998 (International Conference on Spoken Language Processing.)
• 2: Nakamura S., Markov K., Nakaiwa H., et al. The ATR Multilingual Speech-to-Speech Translation System. IEEE Transactions on Audio, Speech, and Language Processing 10, 2 (2006), 365--376.
• 3: Kazunori Imoto , Munehiko Sasajima , Taishii Shimomori , Noriko Yamanaka , Makoto Yajima , Yasuyuki Masai, A multi modal supporting tool for multi lingual communication by inducing partner's reply, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111527]
• 4: Chen S., Kingsbury B., Mangu L., et al. Advances in Speech Transcription at IBM under the DAPAR EARS Program. IEEE Transactions on Audio, Speech, and Language Processing 14, 5 (2006), 1596--1608.
• 5: Cui X., Gu L., Xiang B., et al. Developing High Performance ASR in the IBM Multilingual Speech-to-Speech Translation System. In Proc. ICASSP 2008 (International Conference on Acoustics, Speech, and Signal Processing), IEEE Press (2008), 5121--5124.
• 6: Markham P.L., Peter L. A., McCarthy T.J. The Effects of Native Language vs. Target Language Captions on Foreign Language Students' DVD Video Comprehension. Foreign Language Annals 34, 5 (2001), 439--445.
• 7: Jin Y., Psychological Measurement. East China Normal University Press, China, 2005.
• 8: Elizabeth S. Veinott , Judith Olson , Gary M. Olson , Xiaolan Fu, Video helps remote work: speakers who need to negotiate common ground benefit from seeing each other, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.302-309, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303067]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Computer-supported cooperative work

General Terms:
Experimentation

Keywords:
cmc, experiment, multimodal, non-native speakers, real-time transcription