N-Poisson document modelling

ABSTRACT

This paper is a report of a study investigating the validity of the Multiple Poisson (nP) model of word distribution in document collections. An nP distribution is a mixture of n Poisson distributions with different means. We describe a practical algorithm for determining if a certain word is distributed acording to an nP distribution and computing the distribution parameters. The algorithm was applied to every word in four different document collections. It was found that over 70% of frequently occurring words and terms indeed behave according to the nP distributions. The results indicate that the proportion of nP words depends on the collection size, document length and the frequency of the individual words. Most of the nP words recognised are distributed according to the mixture of relatively few single Poisson distributions (two, three or four). There is an indication that the number of single Poisson components in the mixture of relatively few single Poisson distributions (two, three or four). There is an indication that the number of single Poisson components in the mixture depends on the collection frequency of words.

REFERENCES

• 1: Agha, M., & Ibrahim, M. T. (1984). Maximum Likelihood Estimation of Mixtures of Distributions. Applied Statistics, Journal of the Royal Statistzcal Soczety, 33,327-332.
• 2: B~rtschi, M. (1985). An Overview of Information Retrieval Subjects. Compuler, 18(5),67-84.
• 3: Bookstein, A., & Swanson, D. (1974). Probabilistic Models for Automatic Indexing. Journal of the American Society for {nformatzon Science, 25(5),312-318.
• 4: Breiman, L. (1973). Statistics: Wtth a V~ew Towards Applications. Houghton-Mifflin Company.
• 5: Damerau, F. J. (1965). An Experiment in Automatic Indexing. Amemcan Documentation, 16(4).
• 6: David K. Gifford, Polychannel systems for mass digital communications, Communications of the ACM, v.33 n.2, p.141-151, Feb. 1990 [doi>10.1145/75577.75580]
• 7: Harter, S. P. (1975). A Probabilistic Approach to Automatic Keyword Indexing: Part 1. Journal of the American Society for Information Science, 26(4),197-206.
• 8: Hasselblad, V. (1969). }~'stimation of Finite Mixtures o{" Distributions from the Exponential Family. American Statistzcal Association journal, 64,1459-1471.
• 9: Karp, R. M. (1990). An IJltroduction to Randomized AI.- gorithms. Technical Report TR-90-024, Computer Science Division, University of California, Berkeley.
• 10: Luhn, H. P. (1958). The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development, 2,156-165.
• 11: Margulis, E. L. (1991). N-Poisson Document Modelling Revisited. Technical Report 166, ETH-Ziirich, Departement Informatik.
• 12: Porter, M. (1980). An A_lgorithm for Suffix Stripping. Program, 24(3),130-137.
• 13: S. E. Robertson , C. J. van Rijsbergen , M. F. Porter, Probabilistic models of indexing and searching, Proceedings of the 3rd annual ACM conference on Research and development in information retrieval, p.35-56, June 23-27, 1980, Cambridge, England
• 14: Padmini Srinivason, On generalizing the Two-Poisson model, Journal of the American Society for Information Science, v.41 n.1, p.61-66, Jan. 1990 [doi>10.1002/(SICI)1097-4571(199001)41:1<61::AID-ASI5>3.0.CO;2-Q]
• 15: Stone, D. C., & Rubinoff, M. (1968). Statistical Generation of a Technical Vocabulary. American Documentation, 19(4).

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.1 Document and Text Editing
          Subjects: Languages**

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo)

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing; Indexing methods
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation, Languages