Template extraction from candidate template set generation: a structure and content approach

ABSTRACT

This paper introduces a new approach of webpage template extraction. Unlike traditional methods which concern only content information, this paper considers both structure and content similarity. It uses natural table structure as content units instead of text blocks or pagelets. This paper novelly and formally defines the templates and other concepts. It introduces a new concept, candidate template, which is an intermediate level of abstract table structure. A candidate template only covers the most informative tables, and abstracts a large page set with similar structures. This paper proposes a novel approach of template extraction by solving three sub problems surrounding candidate template set. The involving of candidate template set solves the accuracy and efficiency problems of traditional approaches. This paper also introduces a new model for structural similarity, and for table informativeness based on six heuristics.

REFERENCES

• 1: S. Lawrence and C. L. Giles. Accessibility of information on the Web. Nature, 400(6740):107--109, 1999.
• 2: Kevin Chen-Chuan Chang , Bin He , Chengkai Li , Mitesh Patel , Zhen Zhang, Structured databases on the web: observations and implications, ACM SIGMOD Record, v.33 n.3, September 2004 [doi>10.1145/1031570.1031584]
• 3: Wang Zhulong, Yu Hao, and Nishino Fumihito. 2004. Automatic Special type Website Detection Based on Webpage Type Classification. International Workshop on Web Engineering in conjunction with ACM Hypertext 2004, Aug. 2004
• 4: Ziv Bar-Yossef , Sridhar Rajagopalan, Template detection via data mining and its applications, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511522]
• 5: J. Tiedemann. Automatic Construction of Weighted String Similarity Measures. Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, 1999.
• 6: Aixin Sun , Ee-Peng Lim , Wee-Keong Ng, Web classification using support vector machine, Proceedings of the 4th international workshop on Web information and data management, November 08-08, 2002, McLean, Virginia, USA [doi>10.1145/584931.584952]
• 7: Yiming Yang , Seán Slattery , Rayid Ghani, A Study of Approaches to Hypertext Categorization, Journal of Intelligent Information Systems, v.18 n.2-3, p.219-241, March-May 2002 [doi>10.1023/A:1013685612819]
• 8: Yiming Yang, An Evaluation of Statistical Approaches to Text Categorization, Information Retrieval, v.1 n.1-2, p.69-90, 1999 [doi>10.1023/A:1009982220290]
• 9: L. Kaufman and P. J. Rousseeuw, Finding Groups in Data: An Introduction to Cluster Analysis, Wiley & Sons, New York, 1990.
• 10: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 11: Roni Rosenfeld, Two decades of Statistical Language Modeling: Where Do We Go From Here?, Proceedings of the IEEE 88(8), August 2000
• 12: Tengjiao Wang , Shiwei Tang , Dongqing Yang , Jun Gao , Yuqing Wu , Jian Pei, COMMIX: towards effective web information extraction, integration and query answering, Proceedings of the 2002 ACM SIGMOD international conference on Management of data, June 03-06, 2002, Madison, Wisconsin [doi>10.1145/564691.564774]
• 13: Prasanna Ganesan , Hector Garcia-Molina , Jennifer Widom, Exploiting hierarchical domain structure to compute similarity, ACM Transactions on Information Systems (TOIS), v.21 n.1, p.64-93, January 2003 [doi>10.1145/635484.635487]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications

General Terms:
Algorithms, Measurement

Keywords:
candidate template set, structure similarity, table informativeness, template extraction