|
 ABSTRACT
Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
REFERENCES
Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.
| |
1
|
|
| |
2
|
DUC. 2002. The Document Understanding Conference. http://duc.nist.gov.
|
| |
3
|
Fukusima, T. and Okumura, M. 2001. Text Summarization Challenge: Text Summarization Evaluation at NTCIR Workshop2. In Proceedings of the Second NTCIR Workshop on Research in Chinese & Japanese Text Retrieval and Text Summarization, NII, Tokyo, Japan, 2001.
|
| |
4
|
Lin, C.-Y. 2001. Summary Evaluation Environment. http://www.isi.edu/~cyl/SEE.
|
| |
5
|
|
| |
6
|
McKeown, K., R. Barzilay, D. Evans, V. Hatzivassiloglou, J. L. Klavans, A. Nenkova, C. Sable, B. Schiffman, S. Sigelman. Tracking and Summarizing News on a Daily Basis with Columbia's Newsblaster. In Proceedings of Human Language Technology Conference 2002 (HLT 2002). San Diego, CA, 2002.
|
| |
7
|
Mani, I., D. House, G. Klein, L. Hirschman, L. Obrst, T. Firmin, M. Chrzanowski, and B. Sundheim. 1998. The TIPSTER SUMMAC Text Summarization Evaluation: Final Report. MITRE Corp. Tech. Report.
|
| |
8
|
NIST. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-Occurrence Statistics.
|
| |
9
|
Over, P. 2003. Personal Communication.
|
| |
10
|
Papineni, K., S. Roukos, T. Ward, W.-J. Zhu. 2001. BLEU: a Method for Automatic Evaluation of Machine Translation. IBM Research Report RC22176 (W0109-022).
|
| |
11
|
Porter, M. F. 1980. An Algorithm for Suffix Stripping. Program, 14, pp. 130--137.
|
| |
12
|
Dragomir R. Radev , Sasha Blair-Goldensohn , Zhu Zhang , Revathi Sundara Raghavan, NewsInEssence: a system for domain-independent, real-time news clustering and multi-document summarization, Proceedings of the first international conference on Human language technology research, p.1-4, March 18-21, 2001, San Diego
[doi> 10.3115/1072133.1072208]
|
| |
13
|
|
| |
14
|
Rath, G. J., Resnick, A., and Savage, T. R. 1961. The Formation of Abstracts by the Selection of Sentences. American Documentation, 12(2), pp. 139--143. Reprinted in Mani, I., and Maybury, M., eds, Advances in Automatic Text Summarization, MIT Press, pp. 287--292.
|
| |
15
|
WAS. 2000. Workshop on Automatic Summarization, post-conference workshop of ANLP-NAACL-2000, Seattle, WA, 2000.
|
| |
16
|
WAS. 2001. Workshop on Automatic Summarization, pre-conference workshop of NAACL-2001, Pittsburgh, PA, 2001.
|
| |
17
|
WAS. 2002. Workshop on Automatic Summarization, post-conference workshop of ACL-2002, Philadelphia, PA, 2002.
|
CITED BY 70
|
|
Wenjie Li , Mingli Wu , Qin Lu , Wei Xu , Chunfa Yuan, Extractive summarization using inter- and intra- event relevance, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL, p.369-376, July 17-18, 2006, Sydney, Australia
|
|
|
|
|
|
|
|
Hal Daumé, III, Book Review, Inderjeet Mani: Automatic Summarization, John Benjamins Publishing Co., Amsterdam, The Netherlands, 2001, xi + 286 pp., Machine Translation, v.18 n.4, p.343-347, December 2004
|
|
|
William P. Doran , Nicola Stokes , Eamonn Newman , John Dunnion , Joe Carthy, A hybrid statistical/linguistic model for generating news story gists, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom
|
|
|
|
|
|
|
|
|
Gabriel Murray , Steve Renals , Jean Carletta , Johanna Moore, Incorporating speaker and discourse features into speech summarization, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.367-374, June 04-09, 2006, New York, New York
|
|
|
Enrique Amigó , Julio Gonzalo , Anselmo Peñas , Felisa Verdejo, QARLA: a framework for the evaluation of text summarization systems, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, p.280-289, June 25-30, 2005, Ann Arbor, Michigan
|
|
|
|
|
|
|
|
|
Enrique Amigó , Julio Gonzalo , Víctor Peinado , Anselmo Peñas , Felisa Verdejo, An empirical study of information synthesis tasks, Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, p.207-es, July 21-26, 2004, Barcelona, Spain
|
|
|
|
|
|
|
|
|
|
|
|
Fuchun Peng , Ralph Weischedel , Ana Licuanan , Jinxi Xu, Combining deep linguistics analysis and surface pattern learning: a hybrid approach to Chinese definitional question answering, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.307-314, October 06-08, 2005, Vancouver, British Columbia, Canada
|
|
|
|
|
|
Tsutomu Hirao , Manabu Okumura , Hideki Isozaki, Kernel-based approach for automatic evaluation of natural language generation technologies: application to automatic summarization, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.145-152, October 06-08, 2005, Vancouver, British Columbia, Canada
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Jian-Tao Sun , Dou Shen , Hua-Jun Zeng , Qiang Yang , Yuchang Lu , Zheng Chen, Web-page summarization using clickthrough data, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
|
|
|
|
|
|
|
|
|
Liang Zhou , Chin-Yew Lin , Dragos Stefan Munteanu , Eduard Hovy, ParaEval: using paraphrases to evaluate summaries automatically, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.447-454, June 04-09, 2006, New York, New York
|
|
|
Kathleen McKeown , Rebecca J. Passonneau , David K. Elson , Ani Nenkova , Julia Hirschberg, Do summaries help?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
|
|
|
Tianyong Hao , Zhi Lu , Shitong Wang , Tiansong Zou , Shenhua GU , Liu Wenyin, Categorizing and ranking search engine's results by semantic similarity, Proceedings of the 2nd international conference on Ubiquitous information management and communication, January 31-February 01, 2008, Suwon, Korea
|
|
|
|
|
|
Xu Ling , Jing Jiang , Xin He , Qiaozhu Mei , Chengxiang Zhai , Bruce Schatz, Generating gene summaries from biomedical literature: A study of semi-structured summarization, Information Processing and Management: an International Journal, v.43 n.6, p.1777-1791, November, 2007
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Chin-Yew Lin , Guihong Cao , Jianfeng Gao , Jian-Yun Nie, An information-theoretic approach to automatic evaluation of summaries, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.463-470, June 04-09, 2006, New York, New York
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Jaehui Park , Tomohiro Fukuhara , Ikki Ohmukai , Hideaki Takeda , Sang-goo Lee, Web content summarization using social bookmarks: a new approach for social summarization, Proceeding of the 10th ACM workshop on Web information and data management, October 30-30, 2008, Napa Valley, California, USA
|
|
|
|
|
|
Liangda Li , Ke Zhou , Gui-Rong Xue , Hongyuan Zha , Yong Yu, Enhancing diversity, coverage and balance for summarization through structure learning, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Anton Leuski , Chin-Yew Lin , Liang Zhou , Ulrich Germann , Franz Josef Och , Eduard Hovy, Cross-lingual C*ST*RD: English access to Hindi information, ACM Transactions on Asian Language Information Processing (TALIP), v.2 n.3, p.245-269, September 2003
|
|
|
|
|
|
Lawrence H. Reeve , Hyoil Han , Saya V. Nagori , Jonathan C. Yang , Tamara A. Schwimmer , Ari D. Brooks, Concept frequency distribution in biomedical text summarization, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Adobe Acrobat
QuickTime
Windows Media Player
Real Player
Pdf