Efficient text summarization using lexical chains

ABSTRACT

The rapid growth of the Internet has resulted in enormous amounts of information that has become more difficult to access efficiently. Internet users require tools to help manage this vast quantity of information. The primary goal of this research is to create an efficient and effective tool that is able to summarize large documents quickly. This research presents a linear time algorithm for calculating lexical chains which is a method of capturing the “aboutness” of a document. This method is compared to previous, less efficient methods of lexical chain extraction. We also provide alternative methods for extracting and scoring lexical chains. We show that our method provides similar results to previous research, but is substantially more efficient. This efficiency is necessary in Internet search applications where many large documents may need to be summarized at once, and where the response time to the end user is extremely important.

REFERENCES

• 1: Barzilay, Regina and Michael Elhadad. Using kxical Chains for Text Summarization. in Proceedings of the Intelligent Scalable Text Summarization Workshop (ISTS'97), ACL Madrid, 1997.
• 2: Halliday, Michael and Ruqaiya Hasan. Cohesion in English. Longman, London, 1976.
• 3: Marti A. Hearst, Multi-paragraph segmentation of expository text, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.9-16, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981734]
• 4: Jones, Karen Sparck. What might be in summary? Information Retrieval, 1993.
• 5: Luhn, H.P. The automatic creation of literature abstracts, In H.P. Luhn: Pioneer of Information Science. Schultz, editor. Spartan, 1968.
• 6: Jane Morris , Graeme Hirst, Lexical cohesion computed by thesaural relations as an indicator of the structure of text, Computational Linguistics, v.17 n.1, p.21-48, March 1991
• 7: Stairmond, Mark A. A Computational Analysis of Lexical Cohesion with Applications in Information Retrieval. Ph.D. thesis, Center for Computational Linguistics, UMIST, Manchester, 1999.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Algorithms, Design, Experimentation, Theory

Keywords:
NLP, algorithm, cohesion, lexical chains, linguistics, summarization