MT-based Japanese-Enlish cross-language IR experiments using the TREC test collections

ABSTRACT

This paper evaluates the effectiveness of MT-based Japanese-English CLIR using a subcollection of the TREC test collections and two bilingual researchers to separately translate the TREC requests into Japanese. Our main findings are as follows: (1)With the aid of pseudo-relevance feedback, MT-based J-E CLIR can be as effective as “best-case” monolingual IR. In particular, although poor MT quality often leads to poor initial CLIR performance, pseudo-relevance feedback is useful for alleviating the harm; (2)The manual request translation process that is inherent in conventional CLIR performance evaluation can be a more dominant factor than query length and overall MT quality.

REFERENCES

• 1: Gregory Grefenstette , G. Grefenstette, Cross-Language Information Retrieval, Kluwer Academic Publishers, Norwell, MA, 1998
• 2: J. Scott McCarley, Should we translate the documents or the queries in cross-language information retrieval?, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.208-214, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034716]
• 3: Oard, D. W.: Alternative Approaches for Cross-Language Text Retrieval, AAAI Spring Symposium on Cross-Language Text and Speech Retrieval Electronic Working Notes, 1997. Available from http://www.ece.umd.edu/ medlab/filter/sss/papers.
• 4: Gey, F. C., Jiang, H., Chen, A. and Larson, R. R.: Manual Queries and Machine Translation in Cross-language Retrieval and Interactive Retrieval with Cheshire II at TREC-7, TREC-7 Proceedings, 1999, pp. 527-540.
• 5: Oard, D. W. and Hackett, P.: Document Translation for Cross-Language Text Retrieval at the University of Maryland, TREC-6 Proceedings, 1998, pp. 687--696.
• 6: Sakai, T. et al.: A Study on English-to-Japanese / Japanese-to-English Cross-Language Information Retrieval using Machine Translation (in Japanese), IPSJ Transactions, Vol. 40, No. 11, 1999, pp. 4075--4086.
• 7: David Eichmann , Miguel E. Ruiz , Padmini Srinivasan, Cross-language information retrieval with the UMLS metathesaurus, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.72-80, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290959]
• 8: David A. Hull , Gregory Grefenstette, Querying across languages: a dictionary-based approach to multilingual information retrieval, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.49-57, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243212]
• 9: Ari Pirkola, The effects of query structure and dictionary setups in dictionary-based cross-language information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.55-63, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290957]
• 10: Lisa Ballesteros , W. Bruce Croft, Resolving ambiguity for cross-language retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.64-71, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290958]
• 11: Jian-Yun Nie , Michel Simard , Pierre Isabelle , Richard Durand, Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the Web, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.74-81, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312656]
• 12: Jones, G. et al.: Exploring the use of Machine Translation Resources for English-Japanese Cross-Language Information Retrieval, Post-Conference Workshop on Machine Translation for Cross Language Information Retrieval at AAMT Machine Translation Summit VIII, 1999, pp. 15-22.
• 13: Tetsuya Sakai , Tsuyoshi Kitani , Yasushi Ogawa , Tetsuya Ishikawa , Haruo Kimoto , Ikuo Keshi , Jun Toyoura , Toshikazu Fukushima , Kunio Matsui , Yoshihiro Ueda , Takenobu Tokunaga , Hiroshi Tsuruoka , Hidekazu Nakawatase , Teru Agata , Noriko Kando, BMIR-J2: a test collection for evaluation of Japanese information retrieval systems, ACM SIGIR Forum, v.33 n.1, p.13-17, Fall 1999 [doi>10.1145/331403.331406]
• 14: Tetsuya Sakai , Masahiro Kajiura , Kazuo Sumita, A first step towards flexible local feedback for ad hoc retrieval, Proceedings of the fifth international workshop on on Information retrieval with Asian languages, p.95-102, September 30-October 01, 2000, Hong Kong, China [doi>10.1145/355214.355228]
• 15: NTCIR: http://www.rd.nacsis.ac.jp/ "ntcadm/index-en.html
• 16: Karen Sparck Jones, Further reflections on TREC, Information Processing and Management: an International Journal, v.36 n.1, p.37-85, Jan. 2000 [doi>10.1016/S0306-4573(99)00044-8]
• 17: Kwok, K. L.: English-Chinese Cross-Language Retrieval based on a Translation Package, Post-Conference Workshop on Machine Translation for Cross Language InJormation Retrieval at AAMT Machine Translation Summit VIII, 1999.
• 18: Braschler, M., Peters, C. and Schauble, P.: Cross-Language Information Retrieval(CLIR) Track Overview, TREC.8 Proceedings, 2000.
• 19: Qu, Y. et al.: CLARIT TREC-8 CLIR Experiments, TREC-8 Proceedings, 2000.
• 20: Harmon, D.: What We Have Learned, and Not Learned, from TREC, BCS IRSG '2000 Proceedings, 2000, pp. 2-20.
• 21: Nakayama, K. and Kumano, A.: Collection of Dictionary Data through Internet Translation Service, AAMT Machine Translation Summit VIII Proceedings, 1999, pp. 586-592.
• 22: Sakai, T. et al.: Cross-Language Information Retrieval for NTCIR at Toshiba, NTCIR Workshop 1 Proceedings, 1999, pp. 137-144. http://www.rd.nacsis.ac.jp/-ntcadm/workshop/ OnlinePro ceedings/017-IR-Sakai.pdf
• 23: Robertson, S. E. and Walker, S.: Okapi/Keenbow at TREC-8, TREC-8 Proceedings, 2000.
• 24: Robertson, S. E. and Sparck Jones, K: Simple, Proven Approaches to Text Retrieval, Computer Laboratory, University of Cambridge, 1994.
• 25: Ian H. Witten , Alistair Moffat , Timothy C. Bell, Managing gigabytes (2nd ed.): compressing and indexing documents and images, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999

INDEX TERMS

Keywords:
cross-language information retrieval, machine translation