Automatic wrapper induction from hidden-web sources with domain knowledge

ABSTRACT

We present an original approach to the automatic induction of wrappers for sources of the hidden Web that does not need any human supervision. Our approach only needs domain knowledge expressed as a set of concept names and concept instances. There are two parts in extracting valuable data from hidden-Web sources: understanding the structure of a given HTML form and relating its fields to concepts of the domain, and understanding how resulting records are represented in an HTML result page. For the former problem, we use a combination of heuristics and of probing with domain instances; for the latter, we use a supervised machine learning technique adapted to tree-like information on an automatic, imperfect, and imprecise, annotation using the domain knowledge. We show experiments that demonstrate the validity and potential of the approach.

REFERENCES

• 1: L. Barbosa and J. Freire. Siphoning hidden-Web data through keyword-based interfaces. In Proc. Simpósio Brasileiro de Bancos de Dados, Brasília, Brasil, Oct. 2004.
• 2: BrightPlanet. The deep Web: Surfacing hidden value. White Paper, July 2001.
• 3: Julien Carme , Rémi Gilleron , Aurélien Lemay , Joachim Niehren, Interactive learning of node selecting tree transducer, Machine Learning, v.66 n.1, p.33-67, January 2007 [doi>10.1007/s10994-006-9613-8]
• 4: James Caverlee , Ling Liu , David Buttler, Probe, Cluster, and Discover: Focused Extraction of QA-Pagelets from the Deep Web, Proceedings of the 20th International Conference on Data Engineering, p.103, March 30-April 02, 2004
• 5: Shui-Lung Chuang , Kevin Chen-Chuan Chang , ChengXiang Zhai, Context-aware wrapping: synchronized data extraction, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 6: Valter Crescenzi , Giansalvatore Mecca , Paolo Merialdo, RoadRunner: Towards Automatic Data Extraction from Large Web Sites, Proceedings of the 27th International Conference on Very Large Data Bases, p.109-118, September 11-14, 2001
• 7: Robert B. Doorenbos , Oren Etzioni , Daniel S. Weld, A scalable comparison-shopping agent for the World-Wide Web, Proceedings of the first international conference on Autonomous agents, p.39-48, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267666]
• 8: Dayne Freitag , Nicholas Kushmerick, Boosted Wrapper Induction, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.577-583, July 30-August 03, 2000
• 9: Junfei Geng , Jun Yang, AUTOBIB: Automatic Extraction of Bibliographic Information on the Web, Proceedings of the International Database Engineering and Applications Symposium, p.193-204, July 07-09, 2004 [doi>10.1109/IDEAS.2004.14]
• 10: Bin He , Mitesh Patel , Zhen Zhang , Kevin Chen-Chuan Chang, Accessing the deep web, Communications of the ACM, v.50 n.5, p.94-101, May 2007 [doi>10.1145/1230819.1241670]
• 11: Panagiotis G. Ipeirotis , Luis Gravano, Distributed search over the hidden web: hierarchical database sampling and selection, Proceedings of the 28th international conference on Very Large Data Bases, p.394-405, August 20-23, 2002, Hong Kong, China
• 12: F. Jousse, R. Gilleron, I. Tellier, and M. Tommasi. Conditional Random Fields for XML trees. In Proc. ECML Workshop on Mining and Learning in Graphs, Berlin, Germany, Sept. 2006.
• 13: John D. Lafferty , Andrew McCallum , Fernando C. N. Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.282-289, June 28-July 01, 2001
• 14: Imran R. Mansuri , Sunita Sarawagi, Integrating Unstructured Data into Relational Databases, Proceedings of the 22nd International Conference on Data Engineering, p.29, April 03-07, 2006 [doi>10.1109/ICDE.2006.83]
• 15: A. McCallum and W. Li. Early results for named entity recognition with conditional random fields. In Proc. CoNLL, Edmonton, Canada, May 2003.
• 16: A. Mittal. Probing the hidden Web. Research internship report. Technical Report 479, Gemo, INRIA Futurs, July 2007.
• 17: D. Muschick. Unsupervised learning of XML tree annotations. Master's thesis, Université de Technologie de Lille and Technischen Universität Graz, June 2007.
• 18: Ion Muslea , Steven Minton , Craig A. Knoblock, Hierarchical Wrapper Induction for Semistructured Information Sources, Autonomous Agents and Multi-Agent Systems, v.4 n.1-2, p.93-114, March-June 2001 [doi>10.1023/A:1010022931168]
• 19: David Pinto , Andrew McCallum , Xing Wei , W. Bruce Croft, Table extraction using conditional random fields, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860479]
• 20: M. F. Porter. An algorithm for suffix stripping. Program, 14(3):130--137, July 1980.
• 21: Princeton University Cognitive Science Laboratory. WordNet. http://wordnet.princeton.edu/.
• 22: Sriram Raghavan , Hector Garcia-Molina, Crawling the Hidden Web, Proceedings of the 27th International Conference on Very Large Data Bases, p.129-138, September 11-14, 2001
• 23: S. Sarawagi and W. W. Cohen. Semi-Markov conditional random fields for information extraction. In Proc. NIPS, Vancouver, Canada, Dec. 2004.
• 24: P. Senellart. Comprendre le Web caché. Understanding the Hidden Web. PhD thesis, Université Paris-Sud 11, Orsay, France, Dec. 2007.
• 25: Fei Sha , Fernando Pereira, Shallow parsing with conditional random fields, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.134-141, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073473]
• 26: A. Smith and M. Osborne. Using gazetteers in discriminative information extraction. In Proc. CoNLL, New York, USA, June 2006.
• 27: B. Thomas. Bottom-up learning of logic programs for information extraction from hypertext documents. In Proc. PKDD, Catvat-Dubrovnik, Croatia, Sept. 2003.
• 28: W3C. HTML 4.01 specification, Sept. 1999. http://www.w3.org/TR/REC-html40/.
• 29: W3C. Web Services Description Language (WSDL) 1.1, Mar. 2001. http://www.w3.org/TR/wsdl.
• 30: W. Wu, A. Doan, C. T. Yu, and W. Meng. Bootstrapping domain ontology for semantic Web services from source Web sites. In Proc. Technologies for E-Services, Trondheim, Norway, Sept. 2005.
• 31: Yanhong Zhai , Bing Liu, Web data extraction based on partial tree alignment, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060761]
• 32: Zhen Zhang , Bin He , Kevin Chen-Chuan Chang, Understanding Web query interfaces: best-effort parsing with hidden syntax, Proceedings of the 2004 ACM SIGMOD international conference on Management of data, June 13-18, 2004, Paris, France [doi>10.1145/1007568.1007583]
• 33: Zhen Zhang , Bin He , Kevin Chen-Chuan Chang, Light-weight domain-based form assistant: querying web databases on the fly, Proceedings of the 31st international conference on Very large data bases, August 30-September 02, 2005, Trondheim, Norway

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Design, Experimentation

Keywords:
deep web, form, hidden web, information extraction, invisible web, probing, web service, wrapper