Can we learn a template-independent wrapper for news article extraction from a single training site?

ABSTRACT

Automatic news extraction from news pages is important in many Web applications such as news aggregation. However, the existing news extraction methods based on template-level wrapper induction have three serious limitations. First, the existing methods cannot correctly extract pages belonging to an unseen template. Second, it is costly to maintain up-to-date wrappers for a large amount of news websites, because any change of a template may invalidate the corresponding wrapper. Last, the existing methods can merely extract unformatted plain texts, and thus are not user friendly. In this paper, we tackle the problem of template-independent Web news extraction in a user-friendly way. We formalize Web news extraction as a machine learning problem and learn a template-independent wrapper using a very small number of labeled news pages from a single site. Novel features dedicated to news titles and bodies are developed. Correlations between news titles and news bodies are exploited. Our template-independent wrapper can extract news pages from different sites regardless of templates. Moreover, our approach can extract not only texts, but also images and animates within the news bodies and the extracted news articles are in the same visual style as in the original pages. In our experiments, a wrapper learned from 40 pages from a single news site achieved an accuracy of 98.1% on 3,973 news pages from 12 news sites.

REFERENCES

• 1: Arvind Arasu , Hector Garcia-Molina, Extracting structured data from Web pages, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872799]
• 2: Shui-Lung Chuang , Jane Yung-jen Hsu, Tree-Structured Template Generation for Web Pages, Proceedings of the 2004 IEEE/WIC/ACM International Conference on Web Intelligence, p.327-333, September 20-24, 2004 [doi>10.1109/WI.2004.143]
• 3: Valter Crescenzi , Giansalvatore Mecca , Paolo Merialdo, RoadRunner: Towards Automatic Data Extraction from Large Web Sites, Proceedings of the 27th International Conference on Very Large Data Bases, p.109-118, September 11-14, 2001
• 4: Sergio Flesca , Giuseppe Manco , Elio Masciari , Eugenio Rende , Andrea Tagarelli, Web wrapper induction: a brief survey, AI Communications, v.17 n.2, p.57-61, April 2004
• 5: Suhit Gupta , Gail Kaiser , David Neistadt , Peter Grimm, DOM-based content extraction of HTML documents, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775182]
• 6: Chun-Nan Hsu , Ming-Tzung Dung, Generating finite-state transducers for semi-structured data extraction from the Web, Information Systems, v.23 n.9, p.521-538, Dec. 1998
• 7: Yunhua Hu , Guomao Xin , Ruihua Song , Guoping Hu , Shuming Shi , Yunbo Cao , Hang Li, Title extraction from bodies of HTML documents and its application to web page retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076079]
• 8: Utku Irmak , Torsten Suel, Interactive wrapper generation with minimal user effort, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135859]
• 9: Nitin Jindal , Bing Liu, Review spam detection, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242759]
• 10: K. A. M. C. Kamba, T. Bharat. An interactive, personalized, newspaper on the www. In WWW'95, 1995.
• 11: Nicholas Kushmerick, Wrapper induction: efficiency and expressiveness, Artificial Intelligence, v.118 n.1-2, p.15-68, April 2000 [doi>10.1016/S0004-3702(99)00100-9]
• 12: Alberto H. F. Laender , Berthier A. Ribeiro-Neto , Altigran S. da Silva , Juliana S. Teixeira, A brief survey of web data extraction tools, ACM SIGMOD Record, v.31 n.2, June 2002 [doi>10.1145/565117.565137]
• 13: B. Liu. Web content mining (tutorial). In WWW'05, 2005.
• 14: Ion Muslea , Steve Minton , Craig Knoblock, A hierarchical approach to wrapper induction, Proceedings of the third annual conference on Autonomous Agents, p.190-197, April 1999, Seattle, Washington, United States [doi>10.1145/301136.301191]
• 15: J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 1999.
• 16: D. C. Reis , P. B. Golgher , A. S. Silva , A. F. Laender, Automatic web news extraction using tree edit distance, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988740]
• 17: S. Sarawagi. Automation in information extraction and data integration (tutorial). In VLDB'02, 2002.
• 18: S. M. Selkow. The tree-to-tree editing problem. Information Processing Letters, 6(6):184--186, 1977.
• 19: Heng Tao Shen , Beng Chin Ooi , Kian-Lee Tan, Giving meanings to WWW images, Proceedings of the eighth ACM international conference on Multimedia, p.39-47, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.376098]
• 20: Ruihua Song , Haifeng Liu , Ji-Rong Wen , Wei-Ying Ma, Learning block importance models for web pages, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988700]
• 21: V. Vapnik. Principles of risk minimization for learning theory. In NIPS'91, pages 831--838, 1991.
• 22: Yanhong Zhai , Bing Liu, Web data extraction based on partial tree alignment, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060761]
• 23: Hongkun Zhao , Weiyi Meng , Zonghuan Wu , Vijay Raghavan , Clement Yu, Fully automatic wrapper generation for search engines, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060760]
• 24: Hongkun Zhao , Weiyi Meng , Clement Yu, Mining templates from search result records of search engines, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281286]
• 25: S. Zheng, R. Song, and J. Wen. Template-independent news extraction based on visual consistency. In AAAI'07, volume 22, pages 1507--1513, 2007.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

General Terms:
Algorithms, Experimentation

Keywords:
classification, data extraction, web mining