Discovering informative content blocks from Web documents

ABSTRACT

In this paper, we propose a new approach to discover informative contents from a set of tabular documents (or Web pages) of a Web site. Our system, InfoDiscoverer, first partitions a page into several content blocks according to HTML tag <TABLE> in a Web page. Based on the occurrence of the features (terms) in the set of pages, it calculates entropy value of each feature. According to the entropy value of each feature in a content block, the entropy value of the block is defined. By analyzing the information measure, we propose a method to dynamically select the entropy-threshold that partitions blocks into either informative or redundant. Informative content blocks are distinguished parts of the page, whereas redundant content blocks are common parts. Based on the answer set generated from 13 manually tagged news Web sites with a total of 26,518 Web pages, experiments show that both recall and precision rates are greater than 0.956. That is, using the approach, informative blocks (news articles) of these sites can be automatically separated from semantically redundant contents such as advertisements, banners, navigation panels, news categories, etc. By adopting InfoDiscoverer as the preprocessor of information retrieval and extraction applications, the retrieval and extracting precision will be increased, and the indexing size and extracting complexity will also be reduced.

REFERENCES

• 1: Bear, J., Israel D., Petit, J., and Martin, D., "Using Information Extraction to Improve Document Retrieval," the Sixth Text Retrieval Conference (TREC 6), 1997, pp. 367--378.
• 2: Richard E. Blahut, Principles and practice of information theory, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1987
• 3: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Proceedings of the seventh international conference on World Wide Web 7, p.107-117, April 1998, Brisbane, Australia
• 4: Brin, S. and Page, L., Google Search Engine, http://www.google.com/.
• 5: Cardie, C., "Empirical Methods in Information Extraction," AI Magazine, 18(4):5--79, 1997.
• 6: Soumen Chakrabarti, Integrating the document object model with hyperlinks for enhanced topic distillation and information extraction, Proceedings of the 10th international conference on World Wide Web, p.211-220, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372054]
• 7: Boris Chidlovskii , Jon Ragetli , Maarten de Rijke, Wrapper Generation via Grammar Induction, Proceedings of the 11th European Conference on Machine Learning, p.96-108, May 31-June 02, 2000
• 8: Lee-Feng Chien, PAT-tree-based keyword extraction for Chinese information retrieval, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.50-58, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 9: Jim Cowie , Wendy Lehnert, Information extraction, Communications of the ACM, v.39 n.1, p.80-91, Jan. 1996 [doi>10.1145/234173.234209]
• 10: William B. Frakes , Ricardo Baeza-Yates, Information retrieval: data structures and algorithms, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 11: Dayne Brian Freitag , Tom Mitchell, Machine learning for information extraction in informal domains, 1999
• 12: Chun-Nan Hsu , Ming-Tzung Dung, Generating finite-state transducers for semi-structured data extraction from the Web, Information Systems, v.23 n.9, p.521-538, Dec. 1998
• 13: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM), v.46 n.5, p.604-632, Sept. 1999 [doi>10.1145/324133.324140]
• 14: Raymond Kosala , Hendrik Blockeel, Web mining research: a survey, ACM SIGKDD Explorations Newsletter, v.2 n.1, p.1-15, June, 2000 [doi>10.1145/360402.360406]
• 15: Nicholas Kushmerick , Daniel S. Weld, Wrapper induction for information extraction, 1997
• 16: Porter, M., "The Porter Stemming Algorithm," http://www.tartarus.org/~martin/PorterStemmer/.
• 17: Gerard Salton, Automatic text processing: the transformation, analysis, and retrieval of information by computer, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 18: Shannon, C., "A Mathematical Theory of Communication," Bell System Technical Journal, Vol. 27, pp. 379--423 and 623--656, July and October, 1948.
• 19: Dennis Shasha , Tsong-Li Wang, New techniques for best-match retrieval, ACM Transactions on Information Systems (TOIS), v.8 n.2, p.140-158, Apr. 1990 [doi>10.1145/96105.96111]
• 20: W3C DOM, "Document Object Model (DOM)," http://www.w3.org/DOM/.
• 21: W3C HTML, "HyperText Markup Language," http://www.w3.org/MarkUp/.
• 22: W3C XML, "Extensible Markup Language," http://www.w3.org/XML/.
• 23: Ke Wang , Huiqing Liu, Discovering Structural Association of Semistructured Data, IEEE Transactions on Knowledge and Data Engineering, v.12 n.3, p.353-371, May 2000 [doi>10.1109/69.846290]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.3 Clustering
          Subjects: Similarity measures

General Terms:
Algorithms, Experimentation, Performance

Keywords:
entropy, information extraction, information retrieval, informative content discovery