An approach to natural language for document retrieval

ABSTRACT

Document retrieval systems have been restricted, by the nature of the task, to techniques that can be used with large numbers of documents and broad domains. The most effective techniques that have been developed are based on the statistics of word occurrences in text. In this paper, we describe an approach to using natural language processing (NLP) techniques for what is essentially a natural language problem - the comparison of a request text with the text of document titles and abstracts. The proposed NLP techniques are used to develop a request model based on “conceptual case frames” and to compare this model with the texts of candidate documents. The request model is also used to provide information to statistical search techniques that identify the candidate documents. As part of a preliminary evaluation of this approach, case frame representations of a set of requests from the CACM collection were constructed. Statistical searches carried out using dependency and relative importance information derived from the request models indicate that performance benefits can be obtained.

REFERENCES

• ALSH85: Alshawi, H.; Boguraev, B.; Briscoe, T. "Towards a Dictionary Support Environment for Real Time Parsing". Technical Report, Computer Laboratory, University of Cambridge, 1985.
• BECK75: Becket, J. D. "The Phrasal Lexicon". Bolt, Beranek, and Newman Inc. Report No. 3081, May 1975.
• BIRN81: Birnbaum, L.; Selfridge, M. "Conceptual Analysis of Natural Language." In Inside Computer Understanding : Five Programs Plus Miniatures. Edited by R. Schank and C. Riesbeck, 318-353. Hillsdale : Lawrence Erlbaum, 1981.
• BRUC75: Bruce, B. "Case Systems for Natural Language." Artificial Intelligence, 6: 327-360; 1975.
• CROF81: Croft, W. B. "Document Representation in Probabilistic Models of Information Retrieval". Journal of the American Society of Information Science, 32: 451- 457; 1981.
• CROF84: Croft, W.B. "A Comparison of the Cosine Correlation and the Modified Probabilistic Model". Information Technology, 2: 113-114; 1984.
• CROF86a: Croft, W. B. "Boolean Queries and Term Dependencies in Probabilistic Retrieval Models". Journal of the American society for Information Science, 37: 71-77; 1986.
• CROF86b: W. B. Croft, User-specified domain knowledge for document retrieval, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.201-206, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253211]
• CROF86c: W. B. Croft , R. H. Thompson, I3R: a new approach to the design of document retrieval systems, Journal of the American Society for Information Science, v.38 n.6, p.389-404, Nov. 1987 [doi>10.1002/(SICI)1097-4571(198711)38:6<389::AID-ASI1>3.0.CO;2-4]
• CULL86: Richard E. Cullingford, Natural language processing: a knowledge-engineering approach, Rowman & Littlefield, Totowa, NJ, 1986
• DEJO79: De Jong, G.F. "Skimming Stories in Real Time: An Experiment in Integrated Understanding." Research Report 158, Yale University Department of Computer Science, New Haven, Connecticut, 1979.
• DILL83: Dillon, M.; Gray, A.S. "FASIT: A fully automatic syntactically based indexing system." Journal of the American Society for Information Science. 34:99-108; 1983.
• RIJS79: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• SALT83b: Gerard Salton , Edward A. Fox , Harry Wu, Extended Boolean information retrieval, Communications of the ACM, v.26 n.11, p.1022-1036, Nov. 1983 [doi>10.1145/182.358466]
• SCHA75: Roger C. Schank, Conceptual Information Processing, Elsevier Science Inc., New York, NY, 1975
• SMEA86: Alan F. Smeaton, Incorporating syntactic information into a document retrieval strategy: an investigation, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.103-113, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253193]
• SPAR74: Sparck Jones, K. "Automatic Indexing". Journal of Documentation, 80: 393-432; 1974.
• SPAR84: Sparck Jones, K.; Tait, J. I. "Automatic Search Term Variant Generation". Journal of Documentation, 40: 50-66; 1984.
• TAIT82: Tait, J.I. "Automatic Summarizing of English Texts." Technical Report 47, University of Cambridge Computer Laboratory, Cambridge, England, 1982.
• THUR86: G. Thurmair, A common architecture for different text processing techniques in an information retrieval environment, Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval, p.138-143, September 1986, Palazzo dei Congressi, Pisa, Italy [doi>10.1145/253168.253198]
• WOOD70: W. A. Woods, Transition network grammars for natural language analysis, Communications of the ACM, v.13 n.10, p.591-606, Oct. 1970 [doi>10.1145/355598.362773]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing
      H.3.3 Information Search and Retrieval
          Subjects: Search process

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Languages, Performance