Text summarization via hidden Markov models

ABSTRACT

A sentence extract summary of a document is a subset of the document's sentences that contains the main ideas in the document. We present an approach to generating such summaries, a hidden Markov model that judges the likelihood that each sentence should be contained in the summary. We compare the results of this method with summaries generated by humans, showing that we obtain significantly higher agreement than do earlier methods.

REFERENCES

• 1: C. Aone, M. Okurowski, J. Gorlinsky, and B. Larsen. A scalable summarization system using robust nlp. Proceeding of the ACL'97/EACL'97 Workshop on Intelligent Scalable Text Summarization, pages 66-73, 1997.
• 2: L. E. Baum, T. Petrie, G. Soules, and N. Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains. Ann. Math. Stat., 41:164-171, 1970.
• 3: J. M. Conroy and D. P. O'Leary. Text summarization via hidden markov models and pivoted qr matrix decomposition. Technical report, University of Maryland, College Park, Maryland, March 2001.
• 4: Udo Hahn , Inderjeet Mani, The Challenges of Automatic Summarization, Computer, v.33 n.11, p.29-36, November 2000 [doi>10.1109/2.881692]
• 5: Hongyan Jing , Kathleen R. McKeown, The decomposition of human-written summary sentences, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.129-136, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312666]
• 6: Julian Kupiec , Jan Pedersen , Francine Chen, A trainable document summarizer, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.68-73, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215333]
• 7: H. P. Luhn. The automatic creation of literture abstracts. IBM Journal of Research Development, 2:159-165, 1958.
• 8: L. R. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. of the IEEE, 77:257-285, 1989.
• 9: TREC Conference Series. Text REtrieval Conference (TREC) text research collection. Technical Report http://trec.nist.gov/, National Institute of Standards and Technology, Gaithersburg, Maryland, 1994, 1996, 1997.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Markov processes

Keywords:
automatic summarization, document summarization, extract summaries, hidden Markov models, text summarization