A flexible learning system for wrapping tables and lists in HTML documents

ABSTRACT

A program that makes an existing website look like a database is called a wrapper. Wrapper learning is the problem of learning website wrappers from examples. We present a wrapper-learning system called WL² that can exploit several different representations of a document. Examples of such different representations include DOM-level and token-level representations, as well as two-dimensional geometric views of the rendered page (for tabular data) and representations of the visual appearance of text asm it will be rendered. Additionally, the learning system is modular, and can be easily adapted to new domains and tasks. The learning system described is part of an "industrial-strength" wrapper management system that is in active use at WhizBang Labs. Controlled experiments show that the learner has broader coverage and a faster learning rate than earlier wrapper-learning systems.

REFERENCES

• 1: Hilde Adé , Luc De Raedt , Maurice Bruynooghe, Declarative Bias for Specific-to-General ILP Systems, Machine Learning, v.20 n.1-2, p.119-154, July/Aug. 1995 [doi>10.1007/BF00993477]
• 2: A. Blum. Empirical support for WINNOW and weighted majority algorithms: results on a calendar scheduling domain. In Machine Learning: Proceedings of the Twelfth International Conference, Lake Tahoe, California, 1995. Morgan Kaufmann.
• 3: Boris Chidlovskii , Jon Ragetli , Maarten de Rijke, Wrapper Generation via Grammar Induction, Proceedings of the 11th European Conference on Machine Learning, p.96-108, May 31-June 02, 2000
• 4: XML path language (XPath) version 1.0. Available from http://www.w3.org/TR/1999/REC-xpath-19991116, 1999.
• 5: William W. Cohen, Grammatically biased learning: learning logic programs using an explicit antecedent description language, Artificial Intelligence, v.68 n.2, p.303-366, Aug. 1994 [doi>10.1016/0004-3702(94)90070-1]
• 6: William W. Cohen, Recognizing structure in Web pages using similarity queries, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.59-66, July 18-22, 1999, Orlando, Florida, United States
• 7: William W. Cohen , Wei Fan, Learning page-independent heuristics for extracting data from Web pages, Proceeding of the eighth international conference on World Wide Web, p.1641-1652, May 1999, Toronto, Canada
• 8: De Raedt, Advances in Inductive Logic Programming, IOS Press, Amsterdam, The Netherlands, 1996
• 9: Dayne Freitag, Multistrategy Learning for Information Extraction, Proceedings of the Fifteenth International Conference on Machine Learning, p.161-169, July 24-27, 1998
• 10: Dayne Freitag , Nicholas Kushmerick, Boosted Wrapper Induction, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.577-583, July 30-August 03, 2000
• 11: C.-N. Hsu. Initial results on wrapping semistructured web pages with finite-state transducers and contextual rules. In Papers from the 1998 Workshop on AI and Information Integration, Madison, WI, 1998. AAAI Press.
• 12: HTML 4.01 specification. http://www.w3.org/TR/html4/, 1999.
• 13: M. Hurst. The Interpretation of Tables in Texts. PhD thesis, University of Edinburgh, School of Cognitive Science, Informatics, University of Edinburgh, 2000.
• 14: L. S. Jensen and W. W. Cohen. A structured wrapper induction system for extracting information from semi-structured documents. In Proceedings of the IJCAI-2001 Workshop on Adaptive Text Extraction and Mining, Seattle, WA, 2001.
• 15: Nicholas Kushmerick, Regression testing for wrapper maintenance, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.74-79, July 18-22, 1999, Orlando, Florida, United States
• 16: Nicholas Kushmerick, Wrapper induction: efficiency and expressiveness, Artificial Intelligence, v.118 n.1-2, p.15-68, April 2000 [doi>10.1016/S0004-3702(99)00100-9]
• 17: David D. Lewis, Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval, Proceedings of the 10th European Conference on Machine Learning, p.4-15, April 21-23, 1998
• 18: Nick Littlestone, Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm, Machine Learning, v.2 n.4, p.285-318, April 1988 [doi>10.1023/A:1022869011914]
• 19: A. McCallum and K. Nigam. A comparison of event models for naive bayes text classification. In AAAI-98 Workshop on Learning for Text Categorization, 1998.
• 20: S. Muggleton and L. De Raedt. Inductive logic programming: Theory and methods. Journal of Logic Programming, 19/20(7):629--679, 1994.
• 21: Ion Muslea , Steve Minton , Craig Knoblock, A hierarchical approach to wrapper induction, Proceedings of the third annual conference on Autonomous Agents, p.190-197, April 1999, Seattle, Washington, United States [doi>10.1145/301136.301191]
• 22: I. Muslea, S. Minton, and C. Knoblock. Wrapper induction for semistructured information sources. Journal of Autonomous Agents and Multi-Agent Systems, 16(12), 1999.
• 23: K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. In Proceedings of Machine Learning for Information Filtering Workshop, IJCAI '99, Stockholm, Sweden, 1999.
• 24: Pat Langley, Editorial: Advice to Machine Learning Authors, Machine Learning, v.5 n.3, p.233-237, Aug. 1990 [doi>10.1023/A:1022647305786]
• 25: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 26: J. Ross Quinlan , R. Mike Cameron-Jones, FOIL: A Midterm Report, Proceedings of the European Conference on Machine Learning, p.3-20, April 05-07, 1993
• 27: Arnaud Sahuguet , Fabien Azavant, Building Light-Weight Wrappers for Legacy Web Data-Sources Using W4F, Proceedings of the 25th International Conference on Very Large Data Bases, p.738-741, September 07-10, 1999
• 28: Clean up your web pages with HTML TIDY. http://www.w3.org/People/Raggett/tidy/, 1999.
• 29: Xinxin Wang , Derick Wood, Tabular abstraction, editing, and formatting, 1996
• 30: John M. Zelle , Raymond J. Mooney, Inducing deterministic Prolog parsers from treebanks: a machine learning approach, Proceedings of the twelfth national conference on Artificial intelligence (vol. 1), p.748-753, October 1994, Seattle, Washington, United States

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Textual databases
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Algorithms, Experimentation

Keywords:
canopy, learning, record linkage, reference matching