Structure-driven crawler generation by example

ABSTRACT

Many Web IR and Digital Library applications require a crawling process to collect pages with the ultimate goal of taking advantage of useful information available on Web sites. For some of these applications the criteria to determine when a page is to be present in a collection are related to the page content. However, there are situations in which the inner structure of the pages provides a better criteria to guide the crawling process than their content. In this paper, we present a structure-driven approach for generating Web crawlers that requires a minimum effort from users. The idea is to take as input a sample page and an entry point to a Web site and generate a structure-driven crawler based on navigation patterns, sequences of patterns for the links a crawler has to follow to reach the pages structurally similar to the sample page. In the experiments we have carried out, structure-driven crawlers generated by our new approach were able to collect all pages that match the samples given, including those pages added after their generation.

REFERENCES

• 1: Charu C. Aggarwal , Fatima Al-Garawi , Philip S. Yu, On the design of a learning crawler for topical resource discovery, ACM Transactions on Information Systems (TOIS), v.19 n.3, p.286-309, July 2001 [doi>10.1145/502115.502119]
• 2: Ahnizeret, K., et al. Information retrieval aware web site modelling and generation. In Proceedings of the 23rd International Conference on Conceptual Modeling (Shanghai, China, 2004), pp. 402--419.
• 3: Arvind Arasu , Hector Garcia-Molina , Stanford University, Extracting structured data from Web pages, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872799]
• 4: Pável P. Calado , Altigran S. da Silva , Berthier Ribeiro-Neto , Alberto H. F. Laender , Juliano P. Lage , Davi C. Reis , Pablo A. Roberto , Monique V. Vieira , Marcos A. Gonçalves , Edward A. Fox, Web-DL: an experience in building digital libraries from the web, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA [doi>10.1145/584792.584916]
• 5: Soumen Chakrabarti , Kunal Punera , Mallela Subramanyam, Accelerated focused crawling through online relevance feedback, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511466]
• 6: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1623-1640, May 17, 1999
• 7: Sudarshan S. Chawathe, Comparing Hierarchical Data in External Memory, Proceedings of the 25th International Conference on Very Large Data Bases, p.90-101, September 07-10, 1999
• 8: Robert Cooley, The use of web structure and content to identify subjectively interesting web usage patterns, ACM Transactions on Internet Technology (TOIT), v.3 n.2, p.93-116, May 2003 [doi>10.1145/767193.767194]
• 9: Valter Crescenzi , Paolo Merialdo , Paolo Missier, Clustering web pages based on their structure, Data & Knowledge Engineering, v.54 n.3, p.279-299, September 2005 [doi>10.1016/j.datak.2004.11.004]
• 10: Hasan Davulcu , Juliana Freire , Michael Kifer , I. V. Ramakrishnan, A layered architecture for querying dynamic Web content, Proceedings of the 1999 ACM SIGMOD international conference on Management of data, p.491-502, May 31-June 03, 1999, Philadelphia, Pennsylvania, United States
• 11: Davi de Castro Reis , Robson Braga Araújo , Altigran Soares da Silva , Berthier A. Ribeiro-Neto, A Framework for Generating Attribute Extractors for Web Data Sources, Proceedings of the 9th International Symposium on String Processing and Information Retrieval, p.210-226, September 11-13, 2002
• 12: D. C. Reis , P. B. Golgher , A. S. Silva , A. F. Laender, Automatic web news extraction using tree edit distance, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988740]
• 13: Juliano Palmieri Lage , Altigran S. da Silva , Paulo B. Golgher , Alberto H. F. Laender, Automatic generation of agents for collecting hidden web pages for data extraction, Data & Knowledge Engineering, v.49 n.2, p.177-196, May 2004 [doi>10.1016/j.datak.2003.10.003]
• 14: Hongyu Liu , Evangelos Milios , Jeannette Janssen, Probabilistic models for focused web crawling, Proceedings of the 6th annual ACM international workshop on Web information and data management, November 12-13, 2004, Washington DC, USA [doi>10.1145/1031453.1031458]
• 15: Marc Najork , Janet L. Wiener, Breadth-first crawling yields high-quality pages, Proceedings of the 10th international conference on World Wide Web, p.114-118, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371965]
• 16: Jialun Qin , Yilu Zhou , Michael Chau, Building domain-specific web collections for scientific digital libraries: a meta-search enhanced focused crawling method, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996383]
• 17: Selkow, S. M. The tree-to-tree editing problem. Information Processing Letters 6 (Dec. 1977), 184--186.
• 18: Wuu Yang, Identifying syntactic differences between two programs, Software—Practice & Experience, v.21 n.7, p.739-755, July 1991 [doi>10.1002/spe.4380210706]
• 19: Yanhong Zhai , Bing Liu, Web data extraction based on partial tree alignment, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060761]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

General Terms:
Algorithms, Experimentation

Keywords:
digital libraries, tree edit distance, web crawlers