SCANMail: a voicemail interface that makes speech browsable, readable and searchable

ABSTRACT

Increasing amounts of public, corporate, and private speech data are now available on-line. These are limited in their usefulness, however, by the lack of tools to permit their browsing and search. The goal of our research is to provide tools to overcome the inherent difficulties of speech access, by supporting visual scanning, search, and information extraction. We describe a novel principle for the design of UIs to speech data: What You See Is Almost What You Hear (WYSIAWYH). In WYSIAWYH, automatic speech recognition (ASR) generates a transcript of the speech data. The transcript is then used as a visual analogue to that underlying data. A graphical user interface allows users to visually scan, read, annotate and search these transcripts. Users can also use the transcript to access and play specific regions of the underlying message. We first summarize previous studies of voicemail usage that motivated the WYSIAWYH principle, and describe a voicemail UI, SCANMail, that embodies WYSIAWYH. We report on a laboratory experiment and a two-month field trial evaluation. SCANMail outperformed a state of the art voicemail system on core voicemail tasks. This was attributable to SCANMail's support for visual scanning, search and information extraction. While the ASR transcripts contain errors, they nevertheless improve the efficiency of voicemail processing. Transcripts either provide enough information for users to extract key points or to navigate to important regions of the underlying speech, which they can then play directly

REFERENCES

• 1: Barry Michael Arons , Christopher M. Schmandt, Interactively skimming recorded speech, 1994
• 2: Askwall, S. Computer supported reading vs reading text on paper: a comparison of two reading situations, International Journal of Man Machine Studies, 22, 425--439, 1985.
• 3: John Boreczky , Andreas Girgensohn , Gene Golovchinsky , Shingo Uchihashi, An interactive comic book presentation for exploring video, Proceedings of the SIGCHI conference on Human factors in computing systems, p.185-192, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332428]
• 4: Barbara L. Chalfonte , Robert S. Fish , Robert E. Kraut, Expressive richness: a comparison of speech and text as media for revision, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.21-26, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108848]
• 5: Leo Degen , Richard Mander , Gitta Salomon, Working with audio: integrating personal tape recorders and desktop computers, Proceedings of the SIGCHI conference on Human factors in computing systems, p.413-418, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142877]
• 6: Alexander G. Hauptmann , Michael J. Witbrock, Informedia: news-on-demand multimedia information acquisition and retrieval, Intelligent multimedia information retrieval, MIT Press, Cambridge, MA, 1997
• 7: Debby Hindus , Chris Schmandt , Chris Horner, Capturing, structuring, and representing ubiquitous audio, ACM Transactions on Information Systems (TOIS), v.11 n.4, p.376-400, Oct. 1993 [doi>10.1145/159764.159761]
• 8: Hirschberg, J., Bacchiani, M., Hindle, D., Isenhour, P., Rosenberg, A., Stark, L., Stead, L., Zamchick, G., and Whittaker, S. SCANMail: Browsing and Searching Speech Data by Content, Proceedings of Eurospeech 2001, Aalborg, 2001.
• 9: Hirschberg, J. and Nakatani, C. Acoustic indicators of topic segmentation. In ICSLP98, 1998.
• 10: G. J. F. Jones , J. T. Foote , K. Spärck Jones , S. J. Young, Retrieving spoken documents by combining multiple index sources, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.30-38, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243208]
• 11: Rick Kazman , Reem Al-Halimi , William Hunt , Marilyn Mantei, Four Paradigms for Indexing Video Conferences, IEEE MultiMedia, v.3 n.1, p.63-73, January 1996 [doi>10.1109/93.486705]
• 12: Thomas P. Moran , Leysia Palen , Steve Harrison , Patrick Chiu , Don Kimber , Scott Minneman , William van Melle , Polle Zellweger, “I'll get that off the audio”: a case study of salvaging multimedia meeting records, Proceedings of the SIGCHI conference on Human factors in computing systems, p.202-209, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258704]
• 13: Ronald E. Rice , Douglas E. Shook, Voice messaging, coordination, and communication, Intellectual teamwork: social and technological foundations of cooperative work, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1990
• 14: Rice, R.E., & Tyler, J. (1995). Individual and organizational influences on voicemail use and evaluation. Behaviour and Information Technology, 14(6), 329--341.
• 15: Salton, G. The SMART Retrieval System, Prentice-Hall, Englewood Cliffs, NJ, 1971.
• 16: Stark, L., Whittaker, S., and Hirschberg, J. ASR satisficing: the effects of ASR accuracy on speech retrieval. In Proceedings of International Conference on Spoken Language Processing, 2000.
• 17: Lisa Stifelman , Barry Arons , Chris Schmandt, The audio notebook: paper and pen interaction with structured speech, Proceedings of the SIGCHI conference on Human factors in computing systems, p.182-189, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365096]
• 18: Steve Whittaker , Richard Davis , Julia Hirschberg , Urs Muller, Jotmail: a voicemail interface that enables you to see what was said, Proceedings of the SIGCHI conference on Human factors in computing systems, p.89-96, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332411]
• 19: Steve Whittaker , Julia Hirschberg , John Choi , Don Hindle , Fernando Pereira , Amit Singhal, SCAN: designing and evaluating user interfaces to support retrieval from speech archives, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.26-33, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312639]
• 20: Steve Whittaker , Julia Hirschberg , Christine H. Nakatani, All talk and all action: strategies for managing voicemail messages, CHI 98 conference summary on Human factors in computing systems, p.249-250, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286732]
• 21: Steve Whittaker , Julia Hirschberg , Christine H. Nakatani, Play it again: a study of the factors underlying speech browsing behavior, CHI 98 conference summary on Human factors in computing systems, p.247-248, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286731]
• 22: Steve Whittaker , Patrick Hyland , Myrtle Wiley, FILOCHAT: handwritten notes provide access to recorded conversations, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.271-277, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191763]
• 23: Steve Whittaker , Candace Sidner, Email overload: exploring personal information management of email, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.276-283, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238530]
• 24: Wilcox, L. Chen, F., Kimber, D. and Balasubramanian, V. Segmentation of Speech Using Speaker Identification. Proc. ICASSP, 1994.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing

General Terms:
Design, Human Factors, Performance, Theory

Keywords:
"speech as data", asynchronous communication, empirical evaluation, speech access, voicemail, what you see is almost what you hear