Extracting sentence segments for text summarization: a machine learning approach

ABSTRACT

With the proliferation of the Internet and the huge amount of data it transfers, text summarization is becoming more important. We present an approach to the design of an automatic text summarizer that generates a summary by extracting sentence segments. First, sentences are broken into segments by special cue markers. Each segment is represented by a set of predefined features (e.g. location of the segment, average term frequencies of the words occurring in the segment, number of title words in the segment, and the like). Then a supervised learning algorithm is used to train the summarizer to extract important sentence segments, based on the feature vector. Results of experiments on U.S. patents indicate that the performance of the proposed approach compares very favorably with other approaches (including Microsoft Word summarizer) in terms of precision, recall, and classification accuracy.

REFERENCES

• 1: H. P. Edmundson, New Methods in Automatic Extracting, Journal of the ACM (JACM), v.16 n.2, p.264-285, April 1969 [doi>10.1145/321510.321519]
• 2: Vasant G. Honavar , Leonard Uhr, Generative learning structures and processes for generalized connectionist networks, Information Sciences: an International Journal, v.70 n.1-2, p.75-108, May 1993 [doi>10.1016/0020-0255(93)90049-R]
• 3: Julian Kupiec , Jan Pedersen , Francine Chen, A trainable document summarizer, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.68-73, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215333]
• 4: H. Luhn. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159-165, 1958.
• 5: W. Mann and S. Thompson. Rhetorical structure theory: Toward a functional theory of text. Text, 8(3):243-281, 1988.
• 6: Daniel C. Marcu , Graeme Hirst, The rhetorical parsing, summarization, and generation of natural language texts, 1998
• 7: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 8: T. Nguyen and V. Srinivasan. Accessing a relational database over the internet using macro language files, 1998. http://www.uspto.gov/.
• 9: J. R. Quinlan, Induction of Decision Trees, Machine Learning, v.1 n.1, p.81-106 [doi>10.1023/A:1022643204877]
• 10: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 11: S. Teufel and M. Moens. Sentence extraction and rhetorical classification for flexible abstracts. In D. Radev and E. Hovy, editors, Intelligent Text Summarization, AAAI Spring Symposium, pages 16-25. AAAI Press, Menlo Park, CA, 1998.
• 12: J. Yang, R. Parekh, and V. Honavar. DistAl: An inter-pattern distance-based constructive learning algorithm. Intelligent Data Analysis, 3:55- 73, 1999.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Abstracting methods

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis

General Terms:
Algorithms, Experimentation, Performance, Reliability, Theory

Keywords:
machine learning, sentence segment extraction, text summarization