A comparison of indexing techniques for Japanese text retrieval

ABSTRACT

A series of Japanese full-text retrieval experiments were conducted using an inference network document retrieval model. The retrieval performance of two major indexing methods, character-based and word-based, were evaluated. Using structured queries, the character-based indexing performed retrieval as well as, or slightly better, than the word-based system. This result has practical significance since the character-based indexing speed is considerably faster than the traditional word-based indexing. All the queries in this experiment were automatically formulated from natural language input.

REFERENCES

• CCH92: Callan J. P., Croft W. B., Harding S. M., "The INQUERY Retrieval System", 3rd International Conference on Database and Exp~rt Sy~tem~ Applications, 1992.
• CTL91: W. Bruce Croft , Howard R. Turtle , David D. Lewis, The use of phrases and structured queries in information retrieval, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.32-45, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122864]
• Fag87: Fagan J., "Experiments in Automatic Phrase Indexing for Document Retrieval, A Comparison of Syntactic and Non-Syntactic Methods. Ph.D., Cornell University, 1987.
• Fra92: William B. Frakes , Ricardo Baeza-Yates, Information retrieval: data structures and algorithms, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• GHO+84: Goto T., Hosono K., Oe T., et al., "The Relationship between Electrical Engineering and Chinese Characters Appearing in Artides Related to that Field", JICST 21st Annual Meeting on information Science and Technology, pp.209-216, 1984.
• HKF89: Hatakeyama A., Kawaguchi H., Fujiawa H., "Processing of Synonym and Variant for Free Term Search", IPSJ 39th Zenkoku- Taikai, p.1077, 1989.
• HGH+89: Hosono K., Gotoh T., Harada T., et al., "Analysing Characteristics of Kanji and Katakana Scripts of Technical Terms in Science and Technology Fields", 2nd international Conference on Japanese Information in Science, Technology and Commerce, pp.472-484, 1989.
• INT+86: Ishiwata T., Nishimura H., Tanaka H., et al.,"Nihongo Joho-Shori {Japanese Information Processing}", edited by N. Takahashi, Kindai Kagaku-sha, Tokyo, 1986.
• ISN92: Iwabuchi T., Suda A., Nakata Y., "Automatic Indexing Method of Japanese Language on Full-Text Documents", iPSJ 44th Zenkoku-Taikai, 4, pp.95-96, 1992.
• Kam89: Kamio T., "Automated indexing for Making of a Newspaper Artide Database", JICST Joho Kanri, 32(4), pp.283-293, 1989.
• KHK88: Kaneda K., Hanada J., Kato K., "Analysis of Retrieval Term and Evaluation on and Registration of New Keywords for the inhouse Technical Document Management System", JICST 25th Annual Meetimg on Information Science and Technology, pp.139- 154, 1988.
• Kim87: Kimoto H., "Automatic Keyword Extraction by a Natural Language Processing", 1st Jinkou-Chinou Gakkai, pp.389-392, 1987.
• KMK80: Hiroshi Kinukawa , Hiroshi Matsuoka , Mutsuko Kimura, Japanese sentence analysis for automatic indexing, Proceedings of the 8th conference on Computational linguistics, September 30-October 04, 1980, Tokyo, Japan [doi>10.3115/990174.990268]
• MT92: Masuoka T., Takubo Y., "Kiso Nihongo Bunpou {Basics of Japanese Grammar}", Kuroshio, Tokyo, 1992.
• MKM+91: Matsumoto Y., Kurohasi S., Myoki Y., et al., "Riyousha Teigi Kanou-na Nihongo Keitaiso Kaiseki System, JUMAN Shiyou Setsumeisho {User's Guide for the JUMAN System, A User-Extensible Morphological Analyzer for Japanese}", Nagao Laboratory, Kyoto University, 1991.
• MUH+91: Morohashi M., Umeda S., Hosono K., et al., "Statistical Analysis of Japanese Text by Kanji Usage Frequendes and its Applications", 1991 ASIS Workshop on Natural Language Information Processing, 1991.
• Nak87: Nakamura T., "Automatic Adjustment for Okurigana Variation in Japanese Segmentation", IPSJ 34thZenkoku-Taikai, pp.1297- 1298, 1987.
• Neg89: Negishi M., "Problems in Construction and Service of Full Text Databases, Discussed with a Case at National Center for Science Information System", IPSJ Jouhou-gaku Kiso 14-1, pp.1-8, 1989.
• Neg92: Negishi M., "Technology Trend in Full-text Database", IPSj Joho-shori, 33(4), pp.413- 420, 1992.
• OKT92: Ogawa R., Kikuchi Y., Takahasi K., "Recent Developments in Full-Text Database Technologies", IPSJ Joho-shori, 33(4), pp.404-412, 1992.
• Pea89: Judea Pearl, Probabilistic reasoning in intelligent systems: networks of plausible inference, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1988
• Sal86: Gerard Salton, Another look at automatic text-retrieval systems, Communications of the ACM, v.29 n.7, p.648-656, July 1986 [doi>10.1145/6138.6149]
• SM83: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• San86: Sano H., "Use of English in Japanese Technical Information", 49th ASIS Annual Meeting, pp.300-305, 1986.
• Sas89: Sasaki T., "Similar Expression Retrieval based on Meaning of Kanji Character", IPSJ 39th Zenkoku-Taikai, pp.742-743, 1989.
• Sas88: Sasaki Y., "An Automatic Query Construction Experiment", Proc. of the 25th Annual Meeting on Information Science and Technology, pp.43-48, 1988.
• Som85: Someya K., "Toward Building Databases in Japanese - View of Keywords', JICST 22nd Annual Meeting on Information Science and Technology, pp.37-46, 1985.
• TAK+81: Takano F., Araki K., Kaneko A., et al., "Japanese Keywords Automatic Selection System(JAKAS) -Development of Implementation System", JICST 18th Annual Meetimg on Information Science and Technology, pp.35-44, 1981.
• Tan89: Tanaka H., "Shizen-gengo Kaiseki no Kiso {Foundation of Natural Language Analysis}", Sangyo-Tosho, Tokyo, 1989.
• TOT83: Tanaka K., Osada K., Tsuchiya S., "Analysis of Katakana Strings in Scientific/ Technological Document Abstracts", IPSJ 26th Zenkoku-Taikai, pp.1205-1206, 1983.
• Tom89: Tomatsuri Y., "Discussion on a Program for Selecting Keywords Automatically out of Japannese Text - in the Case of Geographical Articles", IPSJ 1989 Information Science Symposium, pp.135-144, 1989.
• Tsu91: Tsunoda T., "Sekai no Gengo to Nihongo {World Languages and Japanese}", Kuroshio, Tokyo, 1991.
• Tur91: Howard Robert Turtle, Inference networks for document retrieval, University of Massachusetts, Amherst, MA, 1991
• TC91: Howard Turtle , W. Bruce Croft, Evaluation of an inference network-based retrieval model, ACM Transactions on Information Systems (TOIS), v.9 n.3, p.187-222, July 1991 [doi>10.1145/125187.125188]
• Van79: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• Yam92: Yamada K., private letter.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  G. Mathematics of Computing
  G.2 DISCRETE MATHEMATICS
      G.2.2 Graph Theory
          Subjects: Network problems
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo)

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

General Terms:
Experimentation, Languages, Performance