Using latent semantic analysis to improve access to textual information

ABSTRACT

This paper describes a new approach for dealing with the vocabulary problem in human-computer interaction. Most approaches to retrieving textual materials depend on a lexical match between words in users' requests and those in or assigned to database objects. Because of the tremendous diversity in the words people use to describe the same object, lexical matching methods are necessarily incomplete and imprecise [5]. The latent semantic indexing approach tries to overcome these problems by automatically organizing text objects into a semantic structure more appropriate for matching user requests. This is done by taking advantage of implicit higher-order structure in the association of terms with text objects. The particular technique used is singular-value decomposition, in which a large term by text-object matrix is decomposed into a set of about 50 to 150 orthogonal factors from which the original matrix can be approximated by linear combination. Terms and objects are represented by 50 to 150 dimensional vectors and matched against user queries in this “semantic” space. Initial tests find this completely automatic method widely applicable and a promising way to improve users' access to many kinds of textual materials, or to objects and services for which textual descriptions are available.

REFERENCES

• 1: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 2: Harold Borko , Myrna Bernick, Automatic Document Classification, Journal of the ACM (JACM), v.10 n.2, p.151-162, April 1963 [doi>10.1145/321160.321165]
• 3: Jeff Conklin, Hypertext: an introduction and survey, Computer, v.20 n.9, p.17-41, Sept. 1987 [doi>10.1109/MC.1987.1663693]
• 4: Deerwester, S., Dumais, S.T., Fumas, G.W., Landauer, T.K., and Harshman, R.A. Indexing by latent semantic analysis. Journal of the American Society for Information Science, in press.
• 5: Fumas, G.W., Landauer, T.K., Gomez, UM., and Dumais, S.T. Statistical semanUcs: Analysis of the potential performance of key-word information systems. Bell System Technical Journal, 1983, 62(6), 1753-1806.
• 6: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 7: George E. Forsythe , Michael A. Malcolm , Cleve B. Moler, Computer Methods for Mathematical Computations, Prentice Hall Professional Technical Reference, 1977
• 8: Michael D. Good , John A. Whiteside , Dennis R. Wixon , Sandra J. Jones, Building a user-derived interface, Communications of the ACM, v.27 n.10, p.1032-1043, Oct 1984 [doi>10.1145/358274.358284]
• 9: Matthew B. Koll, WEIRD: An approach to concept-based information retrieval, Proceedings of the 1st annual international ACM SIGIR conference on Information storage and retrieval, p.1, May 10-12, 1978
• 10: Thomas W Malone , Kenneth R Grant , Franklyn A Turbak , Stephen A Brobst , Michael D Cohen, Intelligent information-sharing systems, Communications of the ACM, v.30 n.5, p.390-402, May 1987 [doi>10.1145/22899.22903]
• 11: P. Orwick , J. T. Jaynes , T. R. Barstow , L. S. Bohn, DOMAIN/DELPHI: retrieving documents online, Proceedings of the SIGCHI conference on Human factors in computing systems, p.114-121, April 13-17, 1986, Boston, Massachusetts, United States
• 12: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 13: Sparck Jones, K. Automatic keyword classification for information retrieval. Buttersworth, 1971.
• 14: Streeter, L.A. and Lochbaum, K.E. An expert exert-locating system based on automatic representation of semantic structure. In Proceedings of lEEE Conference on AI Applications. San Diego, CA, March 1988.
• 15: Ellen M. Voorhees, The cluster hypothesis revisited, Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval, p.188-196, June 05-07, 1985, Montreal, Quebec, Canada [doi>10.1145/253495.253524]
• 16: Weyer, S. The design of a dynamic book for information search. International Journal of Man Machine Studies, 1982,17, 87-107.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.1 Document and Text Editing
          Subjects: Languages**

  J. Computer Applications
  J.5 ARTS AND HUMANITIES
      Subjects: Linguistics

General Terms:
Human Factors, Languages