Multilingual information access: the contribution of evaluation

ABSTRACT

Since evaluation of cross-language information retrieval systems began at TREC in 1997 and NTCIR in 1998 and, in particular, with the launch of the Cross-Language Evaluation Forum (CLEF) in 2000, considerable progress has been made in this particular sector of IR. Advances can be considered in two stages. The first stage regarded in particular the development of text retrieval systems from simple so-called "bilingual" systems in which a query in one language is used to search a document collection in another to truly "multilingual" retrieval systems where a query in one language can find relevant results from a collection of documents in multiple languages. In the second stage, the focus was no longer just on multilingual document retrieval but was diversified to include different kinds of text retrieval across languages (e.g multilingual question answering) and retrieval on different kinds of media (e.g. collections containing images or speech). However, although the results from the research perspective have been interesting, there has been little real take-up by the applications communities. In the paper we describe the results achieved by CLEF over the years and propose a third stage for multilingual system evaluation which gives far more attention to questions regarding usability and user satisfaction but also provides ways for the results achieved to be transferred to the operational context.

REFERENCES

• 1: Martin Braschler , Carol Peters, Cross-Language Evaluation Forum: Objectives, Results, Achievements, Information Retrieval, v.7 n.1-2, p.7-31, January-April 2004 [doi>10.1023/B:INRT.0000009438.69013.fa]
• 2: Cyril Cleverdon, The Cranfield tests on index language devices, Readings in information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 3: Clough et al. Overview of the ImageCLEF 2006 Photographic Retrieval and Object Annotation Tasks. Working Notes. http://clef.isti.cnr.it/2006/working_notes/CLEF2006WN-Contents1.htm
• 4: Gey, F. C. et al. GeoCLEF 2006: the CLEF 2006 Cross-Language Geographic Information Retrieval Track Overview. CLEF 2006 Working Notes. http://clef.isti.cnr.it/2006/working_notes/CLEF2006WN-Contents1.htm
• 5: Gonzalo, J, Karlgren, J., Clough, P. iCLEF2006 Overview: Searching the Flikr WWWW photo-sharing repository. CLEF 2006 Working Notes. op cit.
• 6: Magnini, B. et al. Overview of the CLEF 2006 Multilingual Question Answer ing Track. CLEF 2006 Working Notes. op cit.
• 7: Müller, H. et al. Overview of the ImageCLEFmed 2006 Medical Retrieval and Annotation Tasks. CLEF 2006 Working Notes. op cit.
• 8: Oard et al. Overview of the CLEF-2006 Cross-Language Speech Retrieval Track. CLEF 2006 Working Notes. op cit.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

General Terms:
Experimentation, Languages, Measurement, Performance

Keywords:
cross-language information retrieval, multilingual information access, system evaluation, test collections