Towards effective strategies for monolingual and bilingual information retrieval: Lessons learned from NTCIR-4

ABSTRACT

At the NTCIR-4 workshop, Justsystem Corporation (JSC) and Clairvoyance Corporation (CC) collaborated in the cross-language retrieval task (CLIR). Our goal was to evaluate the performance and robustness of our recently developed commercial-grade CLIR systems for English and Asian languages. The main contribution of this article is the investigation of different strategies, their interactions in both monolingual and bilingual retrieval tasks, and their respective contributions to operational retrieval systems in the context of NTCIR-4. We report results of Japanese and English monolingual retrieval and results of Japanese-to-English bilingual retrieval. In monolingual retrieval analysis, we examine two special properties of the NTCIR experimental design (two levels of relevance and identical queries in multiple languages) and explore how they interact with strategies of our retrieval system, including pseudo-relevance feedback, multi-word term down-weighting, and term weight merging strategies. Our analysis shows that the choice of language (English or Japanese) does not have a significant impact on retrieval performance. Query expansion is slightly more effective with relaxed judgments than with rigid judgments. For better retrieval performance, weights of multi-word terms should be lowered. In the bilingual retrieval analysis, we aim to identify robust strategies that are effective when used alone and when used in combination with other strategies. We examine cross-lingual specific strategies such as translation disambiguation and translation structuring, as well as general strategies such as pseudo-relevance feedback and multi-word term down-weighting. For shorter title topics, pseudo-relevance feedback is a major performance enhancer, but translation structuring affects retrieval performance negatively when used alone or in combination with other strategies. All experimented strategies improve retrieval performance for the longer description topics, with pseudo-relevance feedback and translation structuring as the major contributors.

REFERENCES

• 1: Allan, J., Connell, M. E., Croft, W. B., Feng, F., Fisher, D., and Li, X. 2000. Inquery and TREC-9. In Proceedings of the 9th Text REtrieval Conference (TREC 2000). National Institute of Standards and Technology (NIST), Gaithersburg, MD.
• 2: Lisa Ballesteros , W. Bruce Croft, Dictionary Methods for Cross-Lingual Information Retrieval, Proceedings of the 7th International Conference on Database and Expert Systems Applications, p.791-801, September 09-13, 1996
• 3: Lisa Ballesteros , W. Bruce Croft, Resolving ambiguity for cross-language retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.64-71, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290958]
• 4: Aitao Chen , Jianzhang He , Liangjie Xu , Fredric C. Gey , Jason Meggs, Chinese text retrieval without using a dictionary, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.42-49, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 5: Mark W. Davis , William C. Ogden, QUILT: implementing a large-scale cross-language text retrieval system, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.92-98, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 6: David A. Evans , Robert G. Lefferts, CLARIT-TREC experiments, Information Processing and Management: an International Journal, v.31 n.3, p.385-395, May-June 1995
• 7: David A. Evans , Chengxiang Zhai, Noun-phrase analysis in unrestricted text for information retrieval, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.17-24, June 24-27, 1996, Santa Cruz, California [doi>10.3115/981863.981866]
• 8: Fujita, S. 1999. Notes on phrasal indexing: JSCB evaluation experiments at NTCIR AD HOC. In Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition. National Center for Science Information Systems (NACSIS), Tokyo, Japan.
• 9: Grefenstette, G. 1998. The problem of cross-language information retrieval. In Cross-Language Information Retrieval. G. Grefenstette (ed). Kluwer Academic, Boston, MA, 1--9.
• 10: David A. Hull , Gregory Grefenstette, Querying across languages: a dictionary-based approach to multilingual information retrieval, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.49-57, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243212]
• 11: Kando, N., Kuriyama, K., Nozue, T., Eguchi, K., Kato, H., and Hidaka, S. 1999. Overview of IR tasks at the first NTCIR workshop. In Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition. National Center for Science Information Systems (NACSIS), Tokyo, Japan, 11--44.
• 12: Kishida, K., Chen, K., Lee, S., Kuriyama, K., Kando, N., Chen, H., Myaeng, S. H., and Eguchi, K. 2004. Overview of CLIR task at the fourth NTCIR workshop. In NTCIR-4 Workshop Meeting: Working Notes of the Fourth NTCIR Workshop Meeting. National Institute of Informatics, Tokyo, Japan, 1--60.
• 13: Kishida, K. and Kando, N. 2004. Two-stages refinement of query translation for pivot language approach to cross-lingual information retrieval. In Comparative Evaluation of Multilingual Information Access Systems, 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003 (Trondheim, Norway, Aug. 21-22, 2003). Revised selected papers. P. C. Gonzalo et al. (eds). Lecture Notes in Computer Science 3237, Springer, New York, 253--262.
• 14: K. L. Kwok, Comparing representations in Chinese information retrieval, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.34-41, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 15: K. L. Kwok, Employing multiple representations for Chinese information retrieval, Journal of the American Society for Information Science, v.50 n.8, p.709-723, June 1999 [doi>10.1002/(SICI)1097-4571(1999)50:8<709::AID-ASI8>3.0.CO;2-V]
• 16: Hideo Fujii , W. Bruce Croft, A comparison of indexing techniques for Japanese text retrieval, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.237-246, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160728]
• 17: Lindman, H. R. 1974. Analysis of Variances in Complex Experimental Designs. Freeman, New York.
• 18: Littman, M. L., Dumais, S. T., and Landauer, T. K. 1998. Automatic cross-language information retrieval using latent semantic indexing. In Cross-Language Information Retrieval. G. Grefenstette (ed). Kluwer Academic, Boston, MA, 51--62.
• 19: Christopher D. Manning , Hinrich Schütze, Foundations of statistical natural language processing, MIT Press, Cambridge, MA, 1999
• 20: J. Scott McCarley, Should we translate the documents or the queries in cross-language information retrieval?, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.208-214, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034716]
• 21: Nakagawa, T. and Kitamura, M. 2004. NTCIR-4 CLIR experiments at Oki. In NTCIR-4 Workshop Meeting: Working Notes of the Fourth NTCIR Workshop Meeting. National Institute of Informatics, Tokyo, 96--99.
• 22: Oyama, K., Ishida, E., and Kando, N. (Eds) 2003. NTCIR Workshop 3: Proceedings of the Third NTCIR Workshop on Research in Information Retrieval, Automatic Text Summarization and Question Answering, National Institute of Informatics, Tokyo, Japan.
• 23: Douglas W. Oard, A Comparative Study of Query and Document Translation for Cross-Language Information Retrieval, Proceedings of the Third Conference of the Association for Machine Translation in the Americas on Machine Translation and the Information Soup, p.472-483, October 28-31, 1998
• 24: Oard, D. W. and Wang, J. 2001. NTCIR-2 experiments at Maryland: Comparing structured queries and balanced translation. In Proceedings of the 2nd NTCIR Workshop on Research in Chinese & Japanese Text Retrieval and Text Summarization. National Institute of Informatics, Tokyo, Japan.
• 25: Ari Pirkola, The effects of query structure and dictionary setups in dictionary-based cross-language information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.55-63, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290957]
• 26: Ari Pirkola , Deniz Puolamäki , Kalervo Järvelin, Applying query structuring in cross-language retrieval, Information Processing and Management: an International Journal, v.39 n.3, p.391-402, May 2003 [doi>10.1016/S0306-4573(02)00091-2]
• 27: Qu, Y., Grefenstette, G., and Evans, D. A. 2003. Resolving translation ambiguity using monolingual corpora. In Advances in Cross-Language Information Retrieval: Third Workshop of the Cross-Language Evaluation Forum (CLEF 2002, Rome, Italy, Sept. 19-20, 2002), C. Peters et al. (eds). Lecture Notes in Computer Science 2785, Springer, New York, 223--241.
• 28: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 29: Savoy, J. 2004. Report on CLIR task for the NTCIR-4 evaluation campaign. In NTCIR-4 Workshop Meeting: Working Notes of the Fourth NTCIR Workshop Meeting. National Institute of Informatics, Tokyo, Japan, 178--192.
• 30: Ruth Sperer , Douglas W. Oard, Structured translation for cross-language information retrieval, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345562]
• 31: Sakai, T., Koyama, M., Kumano, A., and Manabe, T. 2004. Toshiba BIRDJE at NTCIR-4 CLIR: Monolingual/bilingual IR and flexible feedback. In NTCIR-4 Workshop Meeting: Working Notes of the Fourth NTCIR Workshop Meeting. National Institute of Informatics, Tokyo, Japan, 65--72.
• 32: Páraic Sheridan , Jean Paul Ballerini, Experiments in multilingual information retrieval using the SPIDER system, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.58-65, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243213]
• 33: Tong, X., Zhai, C., Milic-Frayling, N., and Evans, D. A. 1996. Experiments on Chinese text indexing---CLARIT TREC-5 Chinese track report. In Proceedings of the Fifth Text REtrieval Conference (TREC-5, Gaithersburg, MD). National Institute of Standards and Technology (NIST), Special Publication 500-238.
• 34: Masao Utiyama , Hitoshi Isahara, Reliable measures for aligning Japanese-English news articles and sentences, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.72-79, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075106]
• 35: Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.355-370, September 03-04, 2001
• 36: Tatsuo Yamashita , Yuji Matsumoto, Language independent morphological analysis, Proceedings of the sixth conference on Applied natural language processing, p.232-238, April 29-May 04, 2000, Seattle, Washington [doi>10.3115/974147.974179]
• 37: Yang, L., Ji, D., and Tang, L. 2004. Chinese information retrieval based on terms and ontology. In NTCIR-4 Workshop Meeting: Working Notes of the Fourth NTCIR Workshop Meeting, National Institute of Informatics, Tokyo, Japan, 136--142.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis

Keywords:
Monolingual information retrieval, NTCIR, comparison, cross-language information retrieval