Probabilistic document indexing from relevance feedback data

ABSTRACT

Based on the binary independence indexing model, we apply three new concepts for probabilistic document indexing from relevance feedback data: Abstraction from specific terms and documents, which overcomes the restriction of limited relevance information for parameter estimation. Flexibility of the representation, which allows the integration of new text analysis and knowledge-based methods in our approach as well as the consideration of more complex document structures or different types of terms (e.g. single words and noun phrases). Probabilistic learning or classification methods for the estimation of the indexing weights making better use of the available relevance information. We give experimental results for five test collections which show improvements over other indexing methods.

REFERENCES

• 1: Beinke-Geiser, U.; Lustig, G.; Putze-Meier, G. (1986). Indexieren mit dem System DAISY. In: Lustig, G. (ed.) : Au#ornatische Indezierung zwischen Forschung und Anwendung, pages 73-97. Olms, Hildesheim.
• 2: P. Biebricher , N. Fuhr , G. Lustig , M. Schwantner , G. Knorz, The automatic indexing system AIR/PHYS - from research to applications, Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval, p.333-342, May 1988, Grenoble, France [doi>10.1145/62437.62470]
• 3: Chow, C. K.; Liu, C. N. (1968). Approximating Discrete Probability Distributions with Dependence Trees. 1EEE Transactions on Information Theor#t 14(3), pages 462--467.
• 4: Croft, W. B. (1081). Document Representation in Probabilistic Models of Information Retrieval. Journal of the American Socie# for lnforraa#ion Science 3P, pages 451-457.
• 5: Croft, W. B. (1083). Experiments with Representation ,in a Document Retrieval System. Information Technology: Research and Development #, pages i-22.
• 6: Croft, W. 13. (1986). Boolean Queries and Term Dependencies in Probabilistic Retrieval Models. Journal of the American Society for Information Science 37(#), pages 71-77.
• 7: J. Fagan, Automatic phrase indexing for document retrieval, Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval, p.91-101, June 03-05, 1987, New Orleans, Louisiana, United States [doi>10.1145/42005.42016]
• 8: Joel L. Fagan, The effectiveness of a nonsyntatic approach to automatic phrase indexing for document retrieval, Journal of the American Society for Information Science, v.40 n.2, p.115-132, March 1989 [doi>10.1002/(SICI)1097-4571(198903)40:2<115::AID-ASI6>3.0.CO;2-B]
• 9: Faiflt, S. (1990). Developmen~ of Indezing Functions Based on Probabilistic Decision TPees (in German). Diploma thesis, TH Darmstadt, FB Informatik, Datenverwaltungssysteme II.
• 10: D. H. Freeman, Jr., Applied categorical data analysis, Marcel Dekker, Inc., New York, NY, 1987
• 11: Fuhr, N. (1988). Probabilistisches lndexing nnd Retrieval. Dissertation, TH Darmstadt, Faehbereich Informatik.
• 12: Norbert Fuhr, Models for retrieval with probabilistic indexing, Information Processing and Management: an International Journal, v.25 n.1, p.55-72, 1989 [doi>10.1016/0306-4573(89)90091-5]
• 13: N. Fuhr, Optimum polynomial retrieval functions, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.69-76, June 25-28, 1989, Cambridge, Massachusetts, United States
• 14: Norbert Fuhr, Optimum polynomial retrieval functions based on the probability ranking principle, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.183-204, July 1989 [doi>10.1145/65943.65944]
• 15: M. Gordon, Probabilistic and genetic algorithms in document retrieval, Communications of the ACM, v.31 n.10, p.1208-1218, Oct. 1988 [doi>10.1145/63039.63044]
• 16: Knorz, G. (1983). Automatisches lndezieren als Erkennen abstrakter Objekte. Niemeyer, Ttibingen.
• 17: K. L. Kwok , William Kuan, Experiments with document components for indexing and retrieval, Information Processing and Management: an International Journal, v.24 n.4, p.405-417, May 1988 [doi>10.1016/0306-4573(88)90044-1]
• 18: K. L. Kwok, An interpretation of index term weighting schemes based on document components, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.275-283, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253226]
• 19: K. L. Kwok, A neural network for probabilistic information retrieval, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-30, June 25-28, 1989, Cambridge, Massachusetts, United States
• 20: M. E. Maron , J. L. Kuhns, On Relevance, Probabilistic Indexing and Information Retrieval, Journal of the ACM (JACM), v.7 n.3, p.216-244, July 1960 [doi>10.1145/321033.321035]
• 21: M. E. Maron, Probabilistic approaches to the document retrieval problem, Proceedings of the 5th annual ACM conference on Research and development in information retrieval, p.98-107, May 18-20, 1982, West Berlin, Germany
• 22: Pfeifer, U. (1990). Development of Log-Linear and Linear-Iterative Indexing Functions (in German). Diploma thesis, TH Darmstadt, FB Informatik, Datenverwaltungssysteme If.
• 23: Quinlan, J. R. (1986). The Effect of Noise on Concept Learning. In: Michalski, R. S.; Carbonell, $. G.; Mitchell, T. M. (ed.) : Machine Learning: An Artificial Intelligence Approach , Vol. 11, pages 149-166. Morgan Kaufmann, Los Altos, California.
• 24: van Rijsbergen, C. J. (1977). A Theoretical Basis for the Use of Cx>-Occurrcnce Data in Information Retrieval. journal of Documentation 33, pages 106-119.
• 25: S. E. Robertson , C. J. van Rijsbergen , M. F. Porter, Probabilistic models of indexing and searching, Proceedings of the 3rd annual ACM conference on Research and development in information retrieval, p.35-56, June 23-27, 1980, Cambridge, England
• 26: l#obertson, S. E.; Maron, M. F..; Cooper, W. S. (1982). Probability of Relevance: A Unification of Two Competing Models for Document Retrieval. Information Technology: Research and Developrnen# 1, pages 1-21.
• 27: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 28: Salton, G.; Yang, C. S.; Yu, C. T. (1975). A Theory of Term Irnportwaee in Automatic Text Analysis. Journal of the American Society for Information Science 36, pages 33--44.
• 29: Alan F. Smeaton, Incorporating syntactic information into a document retrieval strategy: an investigation, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.103-113, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253193]
• 30: Tietze, A. (1989). Approximation of Discrete Probability Distributions by Dependence Trees and #heir Application as lndezin# Functions (in German). Diploma thesis, TH Darmstadt, FB tnformatik, Datenverwaltungssysteme iI.
• 31: Andrew K. C. Wong , David K. Y. Chiu, Synthesizing statistical knowledge from incomplete mixed-mode data, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.9 n.6, p.796-805, November 1, 1987 [doi>10.1109/TPAMI.1987.4767986]
• 32: S. K. M. Wong , Y. Y. Yao, A probability distribution model for information retrieval, Information Processing and Management: an International Journal, v.25 n.1, p.39-53, 1989 [doi>10.1016/0306-4573(89)90090-3]
• 33: C. T. Yu , H. Mizuno, Two learning schemes in information retrieval, Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval, p.201-218, May 1988, Grenoble, France [doi>10.1145/62437.62451]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Query processing
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Design, Experimentation, Theory