Information retrieval using a singular value decomposition model of latent semantic structure

ABSTRACT

In a new method for automatic indexing and retrieval, implicit higher-order structure in the association of terms with documents is modeled to improve estimates of term-document association, and therefore the detection of relevant documents on the basis of terms found in queries. Singular-value decomposition is used to decompose a large term by document matrix into 50 to 150 orthogonal factors from which the original matrix can be approximated by linear combination; both documents and terms are represented as vectors in a 50- to 150- dimensional space. Queries are represented as pseudo-documents vectors formed from weighted combinations of terms, and documents are ordered by their similarity to the query. Initial tests find this automatic method very promising.

REFERENCES

• ATHERTON65: Atherton, P. and Borko, H. A test of factor-analytically derived automated classification methods. AIP rept AIP-DRP 65-1, Jan. 1965.
• BAKER62: Frank B. Baker, Information Retrieval Based upon Latent Class Analysis, Journal of the ACM (JACM), v.9 n.4, p.512-521, Oct. 1962 [doi>10.1145/321138.321148]
• BATES86: Bates, M.J. Subject access in online catalogs: A design model. JASIS, 1986, 37 (6), 357-376.
• BORKO63: Harold Borko , Myrna Bernick, Automatic Document Classification, Journal of the ACM (JACM), v.10 n.2, p.151-162, April 1963 [doi>10.1145/321160.321165]
• CARROLL70: Carroll, J.D. and Chang, J.J. Analysis of individual differences in multidimensional scaling via an N-way generalization of "Eckart-Young" decomposition. Psychometrika, 1970, 35, 283-319.
• CARROLL80: Carroll, J.D. and Arabie, P. Multidimensional scaling. In M.R. Rosenzweig & L.W. Porter (Eds.). Annual Review of Psychology, 1980, 31,607-649.
• COOMBS64: Coombs, C.H. A Theory of Data. New York: Wiley, 1964.
• DESARBO85: Desarbo, W.S., and Carroll, J.D. Three-way metric unfolding via alternating weighted least squares. Psychometrika, 1985, 50(3), 275-300.
• DUMAIS89: Dumais, S.T., Deerwester, S., Furnas, G.W., Landauer T.K., and Harshman, R., Indexing by Latent Structure Analysis. Journal of the American Society for Information Science. 1989, in press.
• FIDEL85: Fidel, R. Individual variability in online searching behavior. In C.A. Parkhurst (Ed.). ASIS'85: Proceedings of the ASIS 48th Annual Meeting, Vol. 22, October 20-24, 1985, Las Vegas, 69-72.
• FORSYTHE77: George E. Forsythe , Michael A. Malcolm , Cleve B. Moler, Computer Methods for Mathematical Computations, Prentice Hall Professional Technical Reference, 1977
• FURNAS80: Furnas, G.W. Objects and their features: The metric representation of two-class data. Ph.D. Dissertation. Stanford University, 1980.
• FURNAS83: Furnas, G.W., Landauer, T.K., Dumais, S.T., and Gomez, L.M. Statistical semantics: Analysis of the potential performance of key-word information systems. Bell System Technical Journal, 1983, 62(6), 1753-1806.
• HARSHMAN70: Harshman, R.A. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modaJ factor analysis. UCLA Work: Papers Phonetics, 1970, 16, 86pp.
• HARSHMAN84: Harshman, R.A. and {.undy, M.E. Data preprocessing and the extended PARA.FAC model. In H.G. Law, C.W. Snyder, Jr., .I.A. Hattie, and R.P. McDonald (Eds.). Resea#'ch Methods for Multimode Data Anatysia, Praeger, 1984b.
• HARSHMAN84: Harshman, R.A. and Lundy, M.E. The PARAFAC model for three-way factor analysis and multi-dimensional scaling. In H.G. Law, C.W. Snyder, Jr., J.A. Hattie, and R.P. McDonald (Eds.). Research Methods for Multimode Data Analysis, Praeger, 1984a.
• HEISER81: Heiser, W.J. Unfolding Analysis of Proximity Data. Leiden, The Netherlands: Reprodienst Psychologic RUL, 1981.
• JARDIN71: Jardin, N. and van Rijsbergen, C.J. The use of hierarchic clustering in infomlation retrieval. Information Storage and Retrieval, 1971, 7, 217-240.
• KOLL79: Matthew B. Koll, WEIRD: an approach to concept-based information retrieval, ACM SIGIR Forum, v.13 n.4, Spring 1979 [doi>10.1145/1095366.1095368]
• KRUSKAL78: Kruskal, J.B. Factor analysis and principal components: Bilinear methods. In H. Ka#askai, J.M. Tanur (Eds.). lnternatiG, nal Encyclopedia of Statistics, New York: Free Press, 1978.
• LILEY54: Liley, O. Evaluation of the subject catalog. American Documentation, 1954, 5(2), 41-60,
• OSSORIO66: Ossorio, P.G. Classification space: A multivariate procedure for automatic document indexing and retrieval. Multivariate Behavioral Research, October 1966, 479-524.
• SALTON68: Gerard. Salton, Automatic Information Organization and Retrieval., McGraw Hill Text, 1968
• SALTON83: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• SPARCK71: Sparck Jones, K. Automatic Keyword Classification for Information Retrieval, Buttersworth, London, 1971.
• SPARCK72: Sparck Jones, K. A stati.stical interpretation of term specificity and its applications in retrieval. Journal of Documentation, March 1972, 28(1), 11-21.
• STREETER87: Streeter, L. A, and l_x#hb#um, K. E. An expert/expert-locating system based on aultomatic representation of semantic suucture. Proceedings of the Fourth Conference on Artificial Intelligence Applications. March 14-18, 1987, San Deigo, CA., pp. 345-350.
• TARR74: Tart, D. and Borko, H. Factors influencing inter-indexer consistency. In Proceedings of the ASIS 37th Annual Meeting, Vol. 11, 1974, 50-55.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Design, Performance