An analysis of latent semantic term self-correlation

ABSTRACT

Latent semantic analysis (LSA) is a generalized vector space method that uses dimension reduction to generate term correlations for use during the information retrieval process. We hypothesized that even though the dimension reduction establishes correlations between terms, the dimension reduction is causing a degradation in the correlation of a term to itself (self-correlation). In this article, we have proven that there is a direct relationship to the size of the LSA dimension reduction and the LSA self-correlation. We have also shown that by altering the LSA term self-correlations we gain a substantial increase in precision, while also reducing the computation required during the information retrieval process.

REFERENCES

• 1: Chris Buckley, Implementation of the SMART Information Retrieval System, Cornell University, Ithaca, NY, 1985
• 2: Chris Buckley , Gerard Salton , James Allan, The effect of adding relevance information in a relevance feedback environment, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.292-300, July 03-06, 1994, Dublin, Ireland
• 3: Chris H. Q. Ding, A probabilistic model for Latent Semantic Indexing: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.6, p.597-608, April 2005 [doi>10.1002/asi.v56:6]
• 4: Dumais, S. T. 1991. Improving the retrieval of information from external sources. Behav. Resear. Meth., Instrum. Comput. 23, 2, 229--236.
• 5: Dumais, S. T. 1994. Latent semantic indexing (lsi): Trec-3 report. In Proceedings of the 3rd Text REtrieval Conference (TREC-3).
• 6: Eckart, C. and Young, G. 1936. The approximation of one matrix by another of lower rank. Psychometrika 1, 3, 211--218.
• 7: Miles Efron, Eigenvalue-based model selection during latent semantic indexing: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.9, p.969-988, July 2005 [doi>10.1002/asi.v56:9]
• 8: Harman, D. 1994. Overview of the 3rd text retrieval conference (TREC-3). D. Harman Ed. National Institute of Standards and Technology Special Publication 500-226. 1--20.
• 9: David Hull, Improving text retrieval for the routing problem using latent semantic indexing, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.282-291, July 03-06, 1994, Dublin, Ireland
• 10: Parry Husbands , Horst Simon , Chris Ding, Term norm distribution and its effects on latent semantic indexing, Information Processing and Management: an International Journal, v.41 n.4, p.777-787, July 2005 [doi>10.1016/j.ipm.2004.03.006]
• 11: Fan Jiang , Michael L. Littman, Approximate Dimension Equalization in Vector-based Information Retrieval, Proceedings of the Seventeenth International Conference on Machine Learning, p.423-430, June 29-July 02, 2000
• 12: K. Sparck Jones , S. Walker , S. E. Robertson, A probabilistic model of information retrieval: development and comparative experiments Part 2, Information Processing and Management: an International Journal, v.36 n.6, p.809-840, Nov.06.2000 [doi>10.1016/S0306-4573(00)00016-9]
• 13: Kontostathis, A. and Pottenger, W. 2002. Detecting patterns in the lsi term-term matrix. Tech. rep. LU-CSE-02-010, Department of Computer Science and Engineering, Lehigh University.
• 14: April Kontostathis , William M. Pottenger, A framework for understanding latent semantic indexing (LSI) performance, Information Processing and Management: an International Journal, v.42 n.1, p.56-73, January 2006 [doi>10.1016/j.ipm.2004.11.007]
• 15: Laurence A. F. Park , Kotagiri Ramamohanarao, Hybrid Pre-Query Term Expansion using Latent Semantic Analysis, Proceedings of the Fourth IEEE International Conference on Data Mining, p.178-185, November 01-04, 2004
• 16: S. K.M. Wong , W. Ziarko , V. V. Raghavan , P. C.N. Wong, On modeling of information retrieval concepts in vector spaces, ACM Transactions on Database Systems (TODS), v.12 n.2, p.299-321, June 1987 [doi>10.1145/22952.22957]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Experimentation, Theory

Keywords:
Latent semantic analysis, term correlation