Two learning schemes in information retrieval

ABSTRACT

Two methods are given to improve weighting schemes by using relevance information of a set of queries. The first method is to estimate parameter values of two independence models in information retrieval — the binary independence model and the non-binary independence model. The parameters estimated here are used to calculate optimal weights for terms in a different set of queries. Performance of this estimation is compared to the inverse document frequency method, the cosine measure, and the statistical similarity measure. The second method is to learn optimal weights of the non-binary independence model adaptively by a learning formula. Experiments are performed on three different document collections CISI, MEDLARS, and CRN4NUL for both methods, and results are reported. Both methods show improvements compared to the existing weighting schemes. Experimental results show that the second method gives slightly better performance than the first one, and has simpler implementation.

REFERENCES

• BoSw: Bookstein, A. and Swanson, D. "A Decision Theoretic Foundation.for Indexing", J. of the American Society for Information Science, January 1975, pp45-50.
• Brei: Breiman, L.. et al. "Classification and regression trees", WADSWORTH INTERNATIONAL GROUP, 1984, pl 1.
• CrHa: Croft, W.B. and Harper, D.J. "Using Probabilistic Models of Document Retrieval without Relevance Information", J. of Documentation, Vol. 35, 1979 pp285-295.
• Crof: Croft, W.B. "Experiments wifl# Representation in a Document Retrieval System", Information Technology:Research and Development, Vol. 2, January 1983, ppl-21.
• RoSp: Robertson, S.E. and Sparck Jones, K. "Relevance Weighing of Search Term", J. of the American Society for Information Science, May-June 1976, pp129-146.
• Salt: G. Salton, The SMART Retrieval System—Experiments in Automatic Document Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1971
• Salt2: Gerard Salton, Recent trends in automatic information retrieval, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.1-10, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253171]
• SaMc: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• Spar: Sparck Jones, K. "A Statistical Interpretation of Term Specificity and its Application in Retrieval", J. of Documentation, Vol. 28, No. I, March 1972, ppl 1-20.
• WoYa: S. K. M. Wong , Y. Y. Yao, A statistical similarity measure, Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, June 03-05, 1987, New Orleans, Louisiana, United States [doi>10.1145/42005.42006]
• WoZW: S. K. M. Wong , Wojciech Ziarko , Patrick C. N. Wong, Generalized vector spaces model in information retrieval, Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval, p.18-25, June 05-07, 1985, Montreal, Quebec, Canada [doi>10.1145/253495.253506]
• Yu: Yu, C.T. "A Framework for Effective Retrieval", Technical Report, the University of lltinois at Chicago, 1987.
• YuLe: C. T. Yu , T. C. Lee, Non-binary independence model, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.265-268, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253224]
• YuSa: C. T. Yu , G. Salton, Precision Weighting—An Effective Automatic Indexing Method, Journal of the ACM (JACM), v.23 n.1, p.76-88, Jan. 1976 [doi>10.1145/321921.321930]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Algorithms, Design, Performance, Theory