One-sided measures for evaluating ranked retrieval effectiveness with spontaneous conversational speech

ABSTRACT

Early speech retrieval experiments focused on news broadcasts, for which adequate Automatic Speech Recognition (ASR) accuracy could be obtained. Like newspapers, news broadcasts are a manually selected and arranged set of stories. Evaluation designs reflected that, using known story boundaries as a basis for evaluation. Substantial advances in ASR accuracy now make it possible to build search systems for some types of spontaneous conversational speech, but present evaluation designs continue to rely on known topic boundaries that are no longer well matched to the nature of the materials. We propose a new class of measures for speech retrieval based on manual annotation of points at which a user with specific topical interests would wish replay to begin.

REFERENCES

• 1: Barry Arons, SpeechSkimmer: a system for interactively skimming recorded speech, ACM Transactions on Computer-Human Interaction (TOCHI), v.4 n.1, p.3-38, March 1997 [doi>10.1145/244754.244758]
• 2: Garofolo, J. et al., The TREC Spoken Document Retrieval Track: A Success Story, in TREC-8, 2000.
• 3: Samuel Gustman , Dagobert Soergel , Douglas Oard , William Byrne , Michael Picheny , Bhuvana Ramabhadran , Douglas Greenberg, Supporting access to large digital oral history archives, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544224]
• 4: Jaana Kekäläinen , Kalervo Järvelin, Using graded relevance assessments in IR evaluation, Journal of the American Society for Information Science and Technology, v.53 n.13, p.1120-1129, November 2002 [doi>10.1002/asi.10137]
• 5: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.m Miscellaneous

General Terms:
Design, Experimentation

Keywords:
evaluation measures, simulation, speech retrieval