Quantifying the utility of parallel corpora

ABSTRACT

Our English-Chinese cross-language IR system is trained from parallel corpora; we investigate its performance as a function of training corpus size for three different training corpora. We find that the performance of the system as trained on the three parallel corpora can be related by a simple measure, namely the out-of-vocabulary rate of query words.

REFERENCES

• 1: D.Harman and E.Voorhees, ed. The 9th Text Retrieval Conference (TREC-9) and references therein.
• 2: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 3: J. Scott McCarley, Should we translate the documents or the queries in cross-language information retrieval?, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.208-214, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034716]
• 4: S.E. Robertson, S. Walker, S. Jones, M.M. Hancock-Beaulieu, M. Gatford, "Okapi at TREC-3" in Proceedings of the Third Text REtrieval Conference (TREC-3) ed. by D.K. Harman. NIST Special Publication 500-225, 1995.
• 5: http://morph.ldc.upenn.edu/Projects/Chinese/

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models; Search process

General Terms:
Design, Experimentation, Performance