DOM-based content extraction of HTML documents

ABSTRACT

Web pages often contain clutter (such as pop-up ads, unnecessary images and extraneous links) around the body of an article that distracts a user from actual content. Extraction of "useful and relevant" content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to removing clutter or making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike "Content Reformatting", which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses "Content Extraction". We have developed a framework that employs easily extensible set of techniques that incorporate advantages of previous work on content extraction. Our key insight is to work with the DOM trees, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages.

REFERENCES

• 1: Aidan Finn, Nicholas Kushmerick and Barry Smyth. "Fact or fiction: Content classification for digital libraries". In Joint DELOS-NSF Workshop on Personalisation and Recommender Systems in Digital Libraries (Dublin), 2001.
• 2: A. F. R. Rahman, H. Alam and R. Hartono. "Content Extraction from HTML Documents". In 1st Int. Workshop on Web Document Analysis (WDA2001), 2001.
• 3: Orkut Buyukkokten , Hector Garcia-Molina , Andreas Paepcke, Accordion summarization for end-game browsing on PDAs and cellular phones, Proceedings of the SIGCHI conference on Human factors in computing systems, p.213-220, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365102]
• 4: Orkut Buyukkokten , Hector Garcia-Molina , Andreas Paepcke, Seeing the whole in parts: text summarization for web browsing on handheld devices, Proceedings of the 10th international conference on World Wide Web, p.652-662, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372178]
• 5: Eija Kaasinen , Matti Aaltonen , Juha Kolari , Suvi Melakoski , Timo Laakko, Two approaches to bringing Internet services to WAP devices, Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking, p.231-246, June 2000, Amsterdam, The Netherlands
• 6: Stuart Hanzlik "Gorilla Design Studios Presents: The Hosts File". Gorilla Design Studios. August 31, 2002. http://accs-net.com/hosts/.
• 7: Marc H. Brown , Robert A. Shillner, A new paradigm for browsing the web, Conference companion on Human factors in computing systems, p.320-321, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223694]
• 8: K.R. McKeown, R. Barzilay, D. Evans, V. Hatzivassiloglou, M.Y. Kan, B. Schiffman and S. Teufel. "Columbia Multi-document Summarization: Approach and Evaluation", In Document Understanding Conf., 2001.
• 9: Nina Wacholder , Dvid K. Evans , Judith L. Klavans, Automatic identification and organization of index terms for interactive browsing, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.126-134, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379468]
• 10: Orkut Buyukkokten , Hector Garcia-Molina , Andreas Paepcke, Seeing the whole in parts: text summarization for web browsing on handheld devices, Proceedings of the 10th international conference on World Wide Web, p.652-662, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372178]
• 11: Manuela Kunze and Dietmar Rosner. "An XML-based Approach for the Presentation and Exploitation of Extracted Information". In 19th International Conference on Computational Linguistics, (Coling) 2002.
• 12: A. F. R. Rahman, H. Alam and R. Hartono. "Understanding the Flow of Content in Summarizing HTML Documents". In Int. Workshop on Document Layout Interpretation and its Applications, DLIA01, Sep., 2001.
• 13: Wolfgang Reichl, Bob Carpenter, Jennifer Chu-Carroll and Wu Chou. "Language Modeling for Content Extraction in Human-Computer Dialogues". In ICSLP, what year?. What in the world is ICSLP??
• 14: Ion Muslea , Steve Minton , Craig Knoblock, A hierarchical approach to wrapper induction, Proceedings of the third annual conference on Autonomous Agents, p.190-197, April 1999, Seattle, Washington, United States [doi>10.1145/301136.301191]
• 15: Min-Yen Kan, Judith L. Klavans and Kathleen R. McKeown. "Linear Segmentation and Segment Relevance". In Proc. of 6th Int. Workshop of Very Large Corpora (WVLC-6), 1998.
• 16: http://www.opera.com
• 17: http://www.bitstream.com/wireless
• 18: http://sourceforge.net/projects/wpar
• 19: http://www.webwiper.com
• 20: http://www.junkbusters.com
• 21: http://www.openxml.org
• 22: Private communication, Min-Yen Kan, Columbia NLP group, 2002.

INDEX TERMS

Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

General Terms:
Algorithms, Human Factors, Standardization

Keywords:
DOM trees, HTML documents, accessibility, content extraction, reformatting, speech rendering