Building an information retrieval test collection for spontaneous conversational speech

ABSTRACT

Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.

REFERENCES

• 1: EU-US working group on spoken-word audio collections, 2003. http://www.dcs.shef.ac.uk/spandh/projects/swag/.
• 2: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 3: William Byrne et al. Automated recognition of spontaneous speech for access to multilingual oral history archives. IEEE Transactions on Speech and Audio Processing, 12(4), 2004.
• 4: Jean Carletta, Assessing agreement on classification tasks: the kappa statistic, Computational Linguistics, v.22 n.2, June 1996
• 5: S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. In Computer Speech and Language, 1999.
• 6: Christopher Cieri , Stephanie Strassel , David Graff , Nii Martey , Kara Rennert , Mark Liberman, Corpora for topic detection and tracking, Topic detection and tracking: event-based information organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 7: C. Cleverdon. The Cranfield tests on index language devices. ASLIB Proceedings, 19(6):173--194, 1967.
• 8: Gordon V. Cormack , Christopher R. Palmer , Charles L. A. Clarke, Efficient construction of large test collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.282-289, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291009]
• 9: Martin Franz et al. Ad hoc and multilingual information retrieval at IBM. In TREC-7, 1998.
• 10: J. S. Garofolo et al. The TREC spoken document retrieval track: A success story. In TREC-8, 1999.
• 11: J. Godfrey et al. SWITCHBOARD: telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 517--520, 1992.
• 12: Samuel Gustman , Dagobert Soergel , Douglas Oard , William Byrne , Michael Picheny , Bhuvana Ramabhadran , Douglas Greenberg, Supporting access to large digital oral history archives, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544224]
• 13: Xiaoli Huang and Dagobert Soergel. Relevance judges' understanding of topical relevance types: An explication of an enriched concept of topical relevance. In Annual Meeting of the American Society for Information Science and Technology, 2004. to appear.
• 14: R. V. Katter. The influence of scale form on relevance judgment. Information Storage and Retrieval, 4(1):1--11, 1968.
• 15: B. Ramabhadran et al. Towards automatic transcription of large spoken archives - English ASR for the MALACH project. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2003.
• 16: S. E. Robertson et al. Okapi at TREC-3. In TREC-3, pages 109--126, 1994.
• 17: Rong Tang , William M. Shaw, Jr. , Jack L. Vevea, Towards the identification of the optimal number of relevance categories, Journal of the American Society for Information Science, v.50 n.3, p.254-264, March 1999 [doi>10.1002/(SICI)1097-4571(1999)50:3<254::AID-ASI8>3.3.CO;2-P]
• 18: L. Schamber. Variations in relevance and information behavior. In Annual Review of Information Science and Technology, volume 29, pages 3--48. 2000.
• 19: K. Sparck-Jones and C. J. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59--72, 1976.
• 20: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]
• 21: Steve Whittaker , Julia Hirschberg , Christine H. Nakatani, Play it again: a study of the factors underlying speech browsing behavior, CHI 98 conference summary on Human factors in computing systems, p.247-248, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286731]
• 22: Yiming Yang , Xin Liu, A re-examination of text categorization methods, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.42-49, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312647]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Measurement

Keywords:
assessment, automatic speech recognition, oral history, search-guided relevance