Capturing, structuring, and representing ubiquitous audio

ABSTRACT

Although talking is an integral part of collaboration, there has been little computer support for acquiring and accessing the contents of conversations. Our approach has focused on ubiquitous audio, or the unobtrusive capture of speech interactions in everyday work environments. Speech recognition technology cannot yet transcribe fluent conversational speech, so the words themselves are not available for organizing the captured interactions. Instead, the structure of an interaction is derived from acoustical information inherent in the stored speech and augmented by user interaction during or after capture. This article describes applications for capturing and structuring audio from office discussions and telephone calls, and mechanisms for later retrieval of these stored interactions. An important aspect of retrieval is choosing an appropriate visual representation, and this article describes the evolution of a family of representations across a range of applications. Finally, this work is placed within the broader context of desktop audio, mobile audio applications, and social implications.

REFERENCES

• 1: ADES, S., AND SWINEHART, D.C. 1986. Voice annotation and editing in a workstation enwronment. In Proceedings of the 1986 Conference. The American Voice I/O Society, San Jose, Calif., 13 28.
• 2: Barry Arons, SpeechSkimmer: interactively skimming recorded speech, Proceedings of the 6th annual ACM symposium on User interface software and technology, p.187-196, December 1993, Atlanta, Georgia, United States [doi>10.1145/168642.168661]
• 3: ARONS, B. 1992a. Techniques, perception, and applications of time-compressed speech. In Proceedings of the 1992 Conference. The American Voice I/O Society, San Jose, Calif., 169-177.
• 4: Barry Arons, Tools for building asynchronous servers to support speech and audio applications, Proceedings of the 5th annual ACM symposium on User interface software and technology, p.71-78, November 15-18, 1992, Monteray, California, United States [doi>10.1145/142621.142630]
• 5: Barry Arons, Hyperspeech: navigating in speech-only hypermedia, Proceedings of the third annual ACM conference on Hypertext, p.133-146, December 15-18, 1991, San Antonio, Texas, United States [doi>10.1145/122974.122989]
• 6: BEATTIE, G. W., AND BARNARD, P. J 1979. The temporal structure of natural telephone conversations (directory enquiry calls) Lmguistics 17, 213 229.
• 7: BELLOTTI, V., AND SELLEN, A. 1993. Design for privacy in ubiquitous computing environments. In Proceedings of European Conference oil Computer Szepported Cooperative Work. Available as Rank Xerox EuroPARC Tech Rep EPC-93-103
• 8: Sara A. Bly , Steve R. Harrison , Susan Irwin, Media spaces: bringing people together in a video, audio, and computing environment, Communications of the ACM, v.36 n.1, p.28-46, Jan. 1993 [doi>10.1145/151233.151235]
• 9: Barbara L. Chalfonte , Robert S. Fish , Robert E. Kraut, Expressive richness: a comparison of speech and text as media for revision, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.21-26, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108848]
• 10: CHEN, F. R., AND WITHGOTT, M M. 1992 The use of emphasis to automatically summarize a spoken discourse. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing IEEE, New York, 1-229-232.
• 11: Leo Degen , Richard Mander , Gitta Salomon, Working with audio: integrating personal tape recorders and desktop computers, Proceedings of the SIGCHI conference on Human factors in computing systems, p.413-418, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142877]
• 12: Alan R. Dennis , Joey F. George , Len M. Jessup , Jay F. Nunamaker, Jr. , Douglas R. Vogel, “Information technology to support electronic meetings", Management Information Systems Quarterly, v.12 n.4, p.591-624, December 1988 [doi>10.2307/249135]
• 13: DOURISH, P. 1993. Culture and control in a media space In Proceedzngs of the European Conference on Computer Supported Cooperatme Work. Available as Rank Xerox EuroPARC Tech Rep. EPC-93-101
• 14: Charles Dunlop , Rob Kling, Computerization and controversy: value conflicts and social choices, Academic Press Professional, Inc., San Diego, CA, 1991
• 15: Carmen Egido, Teleconferencing as a technology to support cooperative work: its possibilities and limitations, Intellectual teamwork: social and technological foundations of cooperative work, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1990
• 16: Robert S. Fish , Robert E. Kraut , Mary D. P. Leland, Quilt: a collaborative tool for cooperative writing, Proceedings of the ACM SIGOIS and IEEECS TC-OA 1988 conference on Office information systems, p.30-37, March 23-25, 1988, Palo Alto, California, United States
• 17: Robert S. Fish , Robert E. Kraut , Robert W. Root , Ronald E. Rice, Video as a technology for informal communication, Communications of the ACM, v.36 n.1, p.48-61, Jan. 1993 [doi>10.1145/151233.151237]
• 18: William Gaver , Thomas Moran , Allan MacLean , Lennart Lövstrand , Paul Dourish , Kathleen Carter , William Buxton, Realizing a video environment: EuroPARC's RAVE system, Proceedings of the SIGCHI conference on Human factors in computing systems, p.27-35, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142754]
• 19: HINDUS, D. 1992. Semi-structured capture and display of telephone conversations. Master's thesis, Massachusetts Institute of Technolog~y, Cambridge, Mass.
• 20: HORNER, C. 1993. NewsTime: A graphical user interface to audio news. Master's thesis, Massachusetts Institute of Technology, Cambridge, Mass.
• 21: H. Ishii, TeamWorkStation: towards a seamless shared workspace, Proceedings of the 1990 ACM conference on Computer-supported cooperative work, p.13-26, October 07-10, 1990, Los Angeles, California, United States [doi>10.1145/99332.99337]
• 22: Ellen A. Isaacs , John C. Tang, What video can and can't do for collaboration: a case study, Proceedings of the first ACM international conference on Multimedia, p.199-206, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.166289]
• 23: LAMMING, M., AND NEWMAN, W. 1992. Activity-based information retrieval: Technology in support of human memory. Tech. Rep. 92-002, Rank Xerox EuroPARC.
• 24: W. E. Mackay , T. W. Malone , K. Crowston , R. Rao , D. Rosenblitt , S. K. Card, How do experienced information lens users use rules?, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.211-216, March 1989
• 25: Thomas W. Malone , Kenneth R. Grant , Kum-Yew Lai , Ramana Rao , David Rosenblitt, Semistructured messages are surprisingly useful for computer-supported coordination, ACM Transactions on Information Systems (TOIS), v.5 n.2, p.115-131, April 1987 [doi>10.1145/27636.27637]
• 26: Marilyn Mantei, Capturing the capture concepts: a case study in the design of computer-supported meeting environments, Proceedings of the 1988 ACM conference on Computer-supported cooperative work, p.257-270, September 26-28, 1988, Portland, Oregon, United States [doi>10.1145/62266.62287]
• 27: Marilyn M. Mantei , Ronald M. Baecker , Abigail J. Sellen , William A. S. Buxton , Thomas Milligan , Barry Wellman, Experiences in the use of a media space, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.203-208, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108888]
• 28: Michael Mills , Jonathan Cohen , Yin Yin Wong, A magnifier tool for video data, Proceedings of the SIGCHI conference on Human factors in computing systems, p.93-98, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142764]
• 29: M. J. Muller , J. E. Daniel, Toward a definition of voice documents, Proceedings of the ACM SIGOIS and IEEE CS TC-OA conference on Office information systems, p.174-183, April 25-27, 1990, Cambridge, Massachusetts, United States
• 30: Brad A. Myers, The importance of percent-done progress indicators for computer-human interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.11-17, April 1985, San Francisco, California, United States
• 31: OSCHMAN, a. B., AND CHAPANIS, h. 1974. The effects of ten communication modes on the behavior of teams during co-operative problem solving. Int. J. Man/Machine Syst. 6, 579 619.
• 32: Stephen Reder , Robert G. Schwab, The temporal structure of cooperative activity, Proceedings of the 1990 ACM conference on Computer-supported cooperative work, p.303-316, October 07-10, 1990, Los Angeles, California, United States [doi>10.1145/99332.99363]
• 33: Paul Resnick, HyperVoice: a phone-based CSCW platform, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.218-225, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143482]
• 34: Paul Resnick , Robert A. Virzi, Skip and scan: cleaning up telephone interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.419-426, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142881]
• 35: ROTHFEDER, J. 1992. Privacy for Sale. Simon and Schuster, New York.
• 36: RUq~rER. D.R. 1987. Communicating by Telephone. Pergamon Press, New York.
• 37: Chris Schmandt, Phoneshell: the telephone as computer terminal, Proceedings of the first ACM international conference on Multimedia, p.373-382, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.168430]
• 38: SCHMANgT, C. 1990. Caltalk: A multi-media calendar. In Proceedings of the 1990 Conference. The American Voice I/O Society, San Jose, Calif., 71-75.
• 39: SCUMANDT, C. 1981. The Intelligent Ear: A graphical interfaceto digital audio. In Proceedings of the IEEE Conference on Cybernctlc~' altd Hocle(v. IEEE, New York, 393 397.
• 40: SCHMANDT, C., AND ARONS, B. 1985. Phone Slave: A graphical telecommunications interface. Proc. Soc. Inf. D~splay 26, 1, 79 82.
• 41: SOCLOF, M., AND ZUE, V. 1990. Collection and analysis of spontaneous and read corpora for spoken language system development. In Proceedmgs of ICSLP. 1105-1108.
• 42: Lee Sproull , Sara Kiesler, Connections: new ways of working in the networked organization, MIT Press, Cambridge, MA, 1991
• 43: TIFELMAN, L.J. 1992. VoiceNotes: An application for a voice-controlled hand-held computer. Master's thesm, Massachusetts Institute of Technology, Cambridge, Mass
• 44: STIFELMAN, L. J. 1991. Not just another voice mail system. In Proceedings of the 1991 Conference. American Voice I/O Society, San Jose, Calif., 21-26.
• 45: Lisa J. Stifelman , Barry Arons , Chris Schmandt , Eric A. Hulteen, VoiceNotes: a speech interface for a hand-held voice notetaker, Proceedings of the SIGCHI conference on Human factors in computing systems, p.179-186, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169150]
• 46: Roy Want , Andy Hopper , Veronica Falcão , Jonathan Gibbons, The active badge location system, ACM Transactions on Information Systems (TOIS), v.10 n.1, p.91-102, Jan. 1992 [doi>10.1145/128756.128759]
• 47: WATABE, K., SAKATA, S., MAENO, K., FUKUOKA, H., AND OHMORI, T. 1991. Distributed desktop conferenclng system with multluser multimedia interface. IEEE J. Sel. Areas Commun. 9, 4, 531 539.
• 48: WEISER, M. 1991. The computer for the 21st century. Sc~. Am. 265, 3 (Sept.), 66 75.
• 49: WILCOX, L., AND BUSH, M. 1991. HMM-based wordspotting for vmce editing and indexing. In Proceedings of Eurospeech 91. 25 28.
• 50: ZELLWECER, P., TERRY, D., ANO SWlNE~ART, D. 1988. An overview of the Etherphone system and its applications. In Proceedings of the 2nd IEEE Conference on Computer Workstatmns. IEEE, New York, 160-168.
• 51: ZuE, V.W. 1991. From signals to symbols to meaning. On machine understanding of spoken language. In Proceedings of the 12th International Congress of Phonetic Sciences.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
  H.4 INFORMATION SYSTEMS APPLICATIONS
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Audio input/output
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)
      H.5.3 Group and Organization Interfaces
          Subjects: Synchronous interaction; Asynchronous interaction

General Terms:
Design, Human Factors

Keywords:
audio interactions, collaborative work, multimedia workstation software, semi-structured data, software telephony, stored speech, ubiquitous computing