Subtopic structuring for full-length document access

ABSTRACT

We argue that the advent of large volumes of full-length text, as opposed to short texts like abstracts and newswire, should be accompanied by corresponding new approaches to information access. Toward this end, we discuss the merits of imposing structure on full-length text documents; that is, a partition of the text into coherent multi-paragraph units that represent the pattern of subtopics that comprise the text. Using this structure, we can make a distinction between the main topics, which occur throughout the length of the text, and the subtopics, which are of only limited extent. We discuss why recognition of subtopic structure is important and how, to some degree of accuracy, it can be found. We describe a new way of specifying queries on full-length documents and then describe an experiment in which making use of the recognition of local structure achieves better results on a typical information retrieval task than does a standard IR measure.

REFERENCES

• 1: W. B. Croft , R. Krovetz , H. Turtle, Interactive retrieval of complex documents, Information Processing and Management: an International Journal, v.26 n.5, p.593-613, 1990 [doi>10.1016/0306-4573(90)90104-A]
• 2: U. Hahn, Topic parsing: accounting for text macro structures in full-text analysis, Information Processing and Management: an International Journal, v.26 n.1, p.135-170, 1990 [doi>10.1016/0306-4573(90)90013-R]
• 3: HEARST, MARTI A. 1993a. Cases as structured indexes for full-length documents. In Procee&ngs of the 1993 AAAI Spring Symposzum on Case-based Reasonzng and Information Retrieval, Stanford,CA.
• 4: Marti A. Hearst, TextTiling: A Quantitative Approach to Discourse, University of California at Berkeley, Berkeley, CA, 1993
• 5: Elizabeth DuRoss Liddy, The discourse-level structure of empirical abstracts: an exploratory study, Information Processing and Management: an International Journal, v.27 n.1, p.55-81, Jan. 1991 [doi>10.1016/0306-4573(91)90031-G]
• 6: RABINEtt, LAWRENCE R. ~L RONALD W. SCHAFER. 1978. Digital processing of speech signals. New Jersey: Prentice-Hall, Inc.
• 7: RO, JUNG SOON. 1988a. An evaluation of the applicability of ranking algorithms to improve the effectiveness of full-text retrieval, i. on the effectiveness of full-text retrieval. Journal of lhe Amemcan Soczety for Information b"czence, 39(2):73-78.
• 8: Ro, JUNG SOON. 1988b. An evaluation of the applicability of ranking algorithms to improve the effectiveness of full-text retrieval, i. on the effectiveness of ranking algorithms on full-text retrieval. Journal of the Amcrzcan Society for Informatzon Science, 39(3):147-160.
• 9: Gerald Salton, Automatic text processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988
• 10: Gerard Salton , Chris Buckley, Automatic text structuring and retrieval-experiments in automatic encyclopedia searching, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-30, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122863]
• 11: SALTON, GERARD & CHRIS BUCKLEY. 1991b. Global text matching for information retrieval. Sczence, 253:1012-1015.
• 12: Craig Stanfill , David L. Waltz, Statistical methods, artificial intelligence, and information retrieval, Text-based intelligent systems: current research and practice in information extraction and retrieval, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1992
• 13: Carol Tenopir , Jung Soon Ro, Full text databases, Greenwood Publishing Group Inc., Westport, CT, 1990
• 14: Howard Turtle , W. Bruce Croft, Evaluation of an inference network-based retrieval model, ACM Transactions on Information Systems (TOIS), v.9 n.3, p.187-222, July 1991 [doi>10.1145/125187.125188]
• 15: David Yarowsky, Word-sense disambiguation using statistical models of Roget's categories trained on large corpora, Proceedings of the 14th conference on Computational linguistics, August 23-28, 1992, Nantes, France [doi>10.3115/992133.992140]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.2 Document Preparation
          Subjects: Format and notation

General Terms:
Algorithms, Documentation, Experimentation, Performance