The impact of term selection in genre-aware focused crawling

ABSTRACT

The genre-aware approach to focused crawling aims at crawling pages related to specific topics that can be expressed in terms of both genre and content information. Such an approach requires an expert to specify a set of terms that describe the genre and the content of the pages of interest. In this paper, we analyze the impact of term selection on this approach. Thus, we have performed an experimental study in which we vary the number of genre and content terms used in focused crawling processes aimed at crawling pages related to syllabi (genre) of computer science courses (subject) and sale offers (genre) of computer equipments (subject). This experimental study showed that a small set of terms selected by an expert is usually enough to produce good results. In addition, we propose and experimentally evaluate a strategy for semi-automatic generation of terms to be used in such an approach. The results of these experiments showed that such a strategy is very effective and provides a means to assist an expert in the task of specifying the sets of required terms.

REFERENCES

• 1: G. T. Assis, A. H. F. Laender, M. A. Gonçalves and A. S. Silva. Exploiting Genre in Focused Crawling. In Proc. of the 14th Symposium on String Processing and Information Retrieval, Santiago, Chile, 2007, pp. 49--60.
• 2: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 3: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1623-1640, May 17, 1999
• 4: P. M. E. De Bra , R. D. J. Post, Information retrieval in the World-Wide Web: making client-based searching feasible, Computer Networks and ISDN Systems, v.27 n.2, p.183-192, Nov. 1994
• 5: Michelangelo Diligenti , Frans Coetzee , Steve Lawrence , C. Lee Giles , Marco Gori, Focused Crawling Using Context Graphs, Proceedings of the 26th International Conference on Very Large Data Bases, p.527-534, September 10-14, 2000
• 6: K. Lagus and S. Kaski. Keyword Selection Method for Characterizing Text Document Maps. In Proc. of the 9th International Conference on Artificial Neural Networks, Edinburgh, UK, 1999, pp. 371--376.
• 7: Andrew Kachites McCallum , Kamal Nigam , Jason Rennie , Kristie Seymore, Automating the Construction of Internet Portals with Machine Learning, Information Retrieval, v.3 n.2, p.127-163, July 2000 [doi>10.1023/A:1009953814988]
• 8: Filippo Menczer , Gautam Pant , Padmini Srinivasan, Topical web crawlers: Evaluating adaptive algorithms, ACM Transactions on Internet Technology (TOIT), v.4 n.4, p.378-419, November 2004 [doi>10.1145/1031114.1031117]
• 9: Filippo Menczer , Gautam Pant , Padmini Srinivasan , Miguel E. Ruiz, Evaluating topic-driven web crawlers, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.241-249, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383995]
• 10: G. Pant and F. Menczer. Topical Crawling for Business Intelligence. In Proc. of the 7th European Conference on Research and Advanced Technology for Digital Libraries, Trodheim, Norway, 2003, pp. 233--244.
• 11: Gautam Pant , Padmini Srinivasan, Link Contexts in Classifier-Guided Topical Crawlers, IEEE Transactions on Knowledge and Data Engineering, v.18 n.1, p.107-122, January 2006 [doi>10.1109/TKDE.2006.12]
• 12: Gautam Pant , Padmini Srinivasan, Learning to crawl: Comparing classification schemes, ACM Transactions on Information Systems (TOIS), v.23 n.4, p.430-462, October 2005 [doi>10.1145/1095872.1095875]
• 13: Gautam Pant , Kostas Tsioutsiouliklis , Judy Johnson , C. Lee Giles, Panorama: extending digital libraries with topical crawlers, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996384]
• 14: P. Srinivasan , F. Menczer , G. Pant, A General Evaluation Framework for Topical Crawlers, Information Retrieval, v.8 n.3, p.417-447, May 2005 [doi>10.1007/s10791-005-6993-5]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Standardization

Keywords:
focused crawling, terms selection, web crawling