Simultaneous multilingual search for translingual information retrieval

ABSTRACT

We consider the problem of translingual information retrieval, where monolingual searchers issue queries in a different language than the document language(s) and the results must be returned in the language they know, the query language. We present a framework for translingual IR that integrates document translation and query translation into the retrieval model. The corpus is represented as an aligned, jointly indexed "pseudo-parallel" corpus, where each document contains the text of the document along with its translation into the query language. The queries are formulated as multilingual structured queries, where each query term and its translations into the document language(s) are treated as synonym sets. This model leverages simultaneous search in multiple languages against jointly indexed documents to improve the accuracy of results over search using document translation or query translation alone. For query translation, we compared a statistical machine translation (SMT) approach to a dictionary-based approach. We found that using a Wikipedia-derived dictionary for named entities combined with an SMT-based dictionary worked better than SMT alone. Simultaneous multilingual search also has other important features suited to translingual search, since it can provide an indication of poor document translation when a match with the source document is found. We show how close integration of CLIR and SMT allows us to improve result translation in addition to IR results.

REFERENCES

• 1: Chen, A. and F. Gey: Combining Query Translation and Document Translation in Cross-Language Retrieval. CLEF 2003: 108--121.
• 2: Kareem Darwish , Douglas W. Oard, Probabilistic structured query methods, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860497]
• 3: Ferrández, S., Toral, A., Ferrández, O., Ferrández, A., Muñoz, R. Applying Wikipedia's Multilingual Knowledge to Cross-Lingual Question Answering. In Proceedings of the 12th International Conference on applications of Natural Language to Information Systems. Paris (France). pp. 352--363. June 2007.
• 4: Habash, N. and Ghoneim, Personal communication, 2007.
• 5: Kalervo Järvelin , Jaana Kekäläinen, IR evaluation methods for retrieving highly relevant documents, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.41-48, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345545]
• 6: Ji, H. and R. Grishman. Collaborative Entity Extraction and Translation. Proc. International Conference on Recent Advances in Natural Language Processing 2007. Borovets, Bulgaria. Sept 2007.
• 7: Diane Kelly , Jimmy Lin, Overview of the TREC 2006 ciQA task, ACM SIGIR Forum, v.41 n.1, p.107-116, June 2007 [doi>10.1145/1273221.1273231]
• 8: Kraaij, W., Variations on Language Modeling on Information Retrieval, Ph.D. thesis, University of Twente, 2004.
• 9: Leah S. Larkey , Lisa Ballesteros , Margaret E. Connell, Improving stemming for Arabic information retrieval: light stemming and co-occurrence analysis, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564425]
• 10: Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
• 11: J. Scott McCarley, Should we translate the documents or the queries in cross-language information retrieval?, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.208-214, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034716]
• 12: Donald Metzler , W. Bruce Croft, Combining the language model and inference network approaches to retrieval, Information Processing and Management: an International Journal, v.40 n.5, p.735-750, September 2004 [doi>10.1016/j.ipm.2004.05.001]
• 13: Miller, George A., Richard Beckwith, Christiane Fellbaum, Derek Gross and Katherine J. Miller. "Introduction to WordNet: an on-line lexical database." In: International Journal of Lexicography 3 (4), 1990, pp. 235 -- 244.
• 14: Nie, J-Y. and Jin, F. A Multilingual Approach to Multilingual Information Retrieval. Proceedings of the Cross-Language Evaluation Forum, 2003.
• 15: Douglas W. Oard, A Comparative Study of Query and Document Translation for Cross-Language Information Retrieval, Proceedings of the Third Conference of the Association for Machine Translation in the Americas on Machine Translation and the Information Soup, p.472-483, October 28-31, 1998
• 16: Franz Josef Och , Hermann Ney, The Alignment Template Approach to Statistical Machine Translation, Computational Linguistics, v.30 n.4, p.417-449, December 2004 [doi>10.1162/0891201042544884]
• 17: Kishore Papineni , Salim Roukos , Todd Ward , Wei-Jing Zhu, BLEU: a method for automatic evaluation of machine translation, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073135]
• 18: Ari Pirkola, The effects of query structure and dictionary setups in dictionary-based cross-language information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.55-63, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290957]
• 19: Jianqiang Wang , Douglas W. Oard, Combining bidirectional translation and synonymy for cross-language information retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148208]
• 20: Wikipedia. "Wikipedia: Multilingual Statistics.", January, 2008.http://en.wikipedia.org/wiki/Wikipedia:Multilingual_statistics
• 21: Xu, J. and R. Weischedel, "TREC-9 Cross-lingual Retrieval at BBN," In Proceedings of The Ninth Text Retrieval Conference, National Institutes of Standards and Technology, 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Design, Experimentation

Keywords:
cross-lingual IR, document translation, query translation