Extracting context to improve accuracy for HTML content extraction

ABSTRACT

Previous work on content extraction utilized various heuristics such as link to text ratio, prominence of tables, and identification of advertising. Many of these heuristics were associated with "settings", whereby some heuristics could be turned on or off and others parameterized by minimum or maximum threshold values. A given collection of settings - such as removing table cells with high linked to non-linked text ratios and removing all apparent advertising -- might work very well for a news website, but leave little or no content left for the reader of a shopping site or a web portal We present a new technique, based on incrementally clustering websites using search engine snippets, to associate a newly requested website with a particular "genre", and then employ settings previously determined to be appropriate for that genre, with dramatically improved content extraction results overall.

REFERENCES

• 1: Wolfgang Reichl, Bob Carpenter, Jennifer Chu-Carroll, Wu Chou, "Language Modeling for Content Extraction in Human-Computer Dialogues", In International Conference on Spoken Language Processing (ICSLP) 1998
• 2: Ion Muslea , Steve Minton , Craig Knoblock, A hierarchical approach to wrapper induction, Proceedings of the third annual conference on Autonomous Agents, p.190-197, April 1999, Seattle, Washington, United States [doi>10.1145/301136.301191]
• 3: Min-Yen Kan, Judith Klavans, Kathleen McKeown, "Linear Segmentation and Segment Relevance", In Proc. of 6th Int. Workshop of Very Large Corpora (WVLC-6), 1998
• 4: Suhit Gupta , Gail Kaiser , David Neistadt , Peter Grimm, DOM-based content extraction of HTML documents, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775182]
• 5: Suhit Gupta , Gail E. Kaiser , Peter Grimm , Michael F. Chiang , Justin Starren, Automating Content Extraction of HTML Documents, World Wide Web, v.8 n.2, p.179-224, June 2005 [doi>10.1007/s11280-004-4873-3]
• 6: Oren Zamir , Oren Etzioni, Web document clustering: a feasibility demonstration, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.46-54, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290956]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.4 Electronic Publishing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

General Terms:
Algorithms, Human Factors, Standardization

Keywords:
DOM trees, HTML, accessibility, content extraction, context, reformatting, speech rendering