Resource selection for domain-specific cross-lingual IR

ABSTRACT

An under-explored question in cross-language information retrieval (CLIR) is to what degree the performance of CLIR methods depends on the availability of high-quality translation resources for particular domains. To address this issue, we evaluate several competitive CLIR methods - with different training corpora - on test documents in the medical domain. Our results show severe performance degradation when using a general-purpose training corpus or a commercial machine translation system (SYSTRAN), versus a domain-specific training corpus. A related unexplored question is whether we can improve CLIR performance by systematically analyzing training resources and optimally matching them to target collections. We start exploring this problem by suggesting a simple criterion for automatically matching training resources to target corpora. By using cosine similarity between training and target corpora as resource weights we obtained an average of 5.6% improvement over using all resources with no weights. The same metric yields 99.4% of the performance obtained when an oracle chooses the optimal resource every time.

REFERENCES

• 1: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 2: Carbonell J. G, Yang, Y., Frederking, R. E., Brown, R., Geng, Y., Lee, D. Translingual Information Retrieval: A Comparative Evaluation. In Proceedings of the IJCAI (1) 1997: 708--715.
• 3: Aitao Chen , Hailing Jiang , Fredric Gey, Combining multiple sources for short query translation in Chinese-English cross-language information retrieval, Proceedings of the fifth international workshop on on Information retrieval with Asian languages, p.17-23, September 30-October 01, 2000, Hong Kong, China [doi>10.1145/355214.355217]
• 4: Darwish, K. and Oard, D. CLIR Experiments at Maryland for TREC-2002: Evidence Combination for Arabic-English Retrieval. In TREC 2002 Proceedings.
• 5: Franz, M., McCarley, J. S, and Roukos, S. Ad hoc and multilingual information retrieval at IBM. In The Seventh Text REtrieval Conference, pages 157--168, November 1998. NIST Special Publication 500--242.
• 6: Franz, M. and McCarley, J.S. Arabic Information Retrieval at IBM. In TREC 2002 proceedings.
• 7: Fraser, A., Xu, J., Weischedel, R. 2002. TREC 2002 Cross-lingual Retrieval at BBN. In TREC 2002 proceedings.
• 8: Gey, F. and Jiang H. 1999. English-German cross-language retrieval for the GIRT collection -- Exploiting a multilingual thesaurus. In TREC-8 proceedings.
• 9: Kando, N. Overview of the Third NTCIR Workshop. Working notes of the Third NTCIR Workshop Meeting. Part I:Overview. Tokyo. Japan. October 2002. p.1--16.
• 10: Khudanpur, S., Kim, W., 2002. Using cross-language cues for story-specific language modeling. In Proceedings of the International Conference on Spoken Language Processing, p. 513--516.
• 11: Khudanpur , S. Kim, W., 1999. A maximum entropy language model to integrate n-grams and topic dependencies for conversational speech recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 553--556.
• 12: Michael Kluck , Fredric C. Gey, The Domain-Specific Task of CLEF - Specific Evaluation Strategies in Cross-Language Information Retrieval, Revised Papers from the Workshop of Cross-Language Evaluation Forum on Cross-Language Information Retrieval and Evaluation, p.48-56, September 21-22, 2000
• 13: Koehn, P. Europarl: A Multilingual Corpus for Evaluation of Machine Translation. Draft, Unpublished.
• 14: Nie, J. Y., Simard, M. and Foster, G.. Using parallel web pages for multi-lingual IR. In C. Peters(Ed.), Proceedings of the CLEF 2000 evaluation forum.
• 15: Oard, D. W. and F. Gey, The TREC-2002 Arabic/English CLIR Track. In TREC 2002 proceedings.
• 16: Oard, D. When You Come to a Fork in the Road, Take It: Multiple Futures for CLIR Research. Cross-Language Information Retrieval: A Research Roadmap. Workshop at SIGIR-2002, Tampere Finland August 15, 2002.
• 17: Franz Josef Och , Hermann Ney, Improved statistical alignment models, Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, p.440-447, October 03-06, 2000, Hong Kong [doi>10.3115/1075218.1075274]
• 18: Ogilvie, P. and Callan, J. Experiments using the Lemur toolkit. In Proceedings of the Tenth Text Retrieval Conference (TREC-10). (2001).
• 19: Peters, C. Results of the CLEF 2003 Cross-Language System Evaluation Campaign. Working Notes for the CLEF 2003 Workshop, 21-22 August, Trondheim, Norway.
• 20: Philip Resnik, Mining the Web for bilingual text, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.527-534, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034757]
• 21: Rogati, M and Yang, Y. Multilingual Information Retrieval using Open, Transparent Resources in CLEF 2003 . In C. Peters (Ed.), Results of the CLEF2003 cross-language evaluation forum.
• 22: Jacques Savoy, A stemming procedure and stopword list for general French corpora, Journal of the American Society for Information Science, v.50 n.10, p.944-952, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:10<944::AID-ASI9>3.3.CO;2-H]
• 23: Seymore, K., Rosenfeld, R. 1997. Using story topics for language model adaptation. In Proceedings of the European Conference on Speech Communication and Technology.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Dictionaries

General Terms:
Algorithms, Experimentation, Measurement, Performance

Keywords:
cross-language information retrieval, domain-specific translation