Effective web crawling

ABSTRACT

The key factors for the success of the World Wide Web are its large size and the lack of a centralized control over its contents. Both issues are also the most important source of problems for locating information. The Web is a context in which traditional Information Retrieval methods are challenged, and given the volume of the Web and its speed of change, the coverage of modern search engines is relatively small. Moreover, the distribution of quality is very skewed, and interesting pages are scarce in comparison with the rest of the content.

REFERENCES

• 1: Ricardo Baeza-Yates and Carlos Castillo. Relating Web characteristics with link based Web page ranking. In Proceedings of String Processing and Information Retrieval, pages 21--32, Laguna San Rafael, Chile, November 2001. IEEE CS Press.
• 2: Ricardo Baeza-Yates and Carlos Castillo. Balancing volume, quality and freshness in web crawling. In Soft Computing Systems - Design, Management and Applications, pages 565--572, Santiago, Chile, 2002. IOS Press Amsterdam.
• 3: Ricardo Baeza-Yates and Carlos Castillo. Crawling the infinite Web: five levels are enough. In Proceedings of the third Workshop on Web Graphs (WAW), volume 3243 of Lecture Notes in Computer Science, pages 156--167, Rome, Italy. October 2004. Springer.
• 4: Ricardo Baeza-Yates, Carlos Castillo, and Felipe Saint-Jean. Web Dynamics, chapter Web Dynamics, Structure and Page Quality, pages 93--109. Springer, 2004.
• 5: Ricardo A. Baeza-Yates , Felipe Saint-Jean , Carlos Castillo, Web Structure, Dynamics and Page Quality, Proceedings of the 9th International Symposium on String Processing and Information Retrieval, p.117-130, September 11-13, 2002
• 6: Ricardo A. Baeza-Yates, Javier Ruiz del Solar, Rodrigo Verschae, Carlos Castillo, and Carlos A. Hurtado. Content-based image retrieval and characterization on specific Web collections. In Third international conference on image and video retrieval (CIVR), volume 3115 of Lecture Notes in Computer Science, pages 189--198, Dublin, Ireland, July 2004. Springer.
• 7: Carlos Castillo, Cooperation Schemes between a Web Server and a Web Search Engine, Proceedings of the First Conference on Latin American Web Congress, p.212, November 10-12, 2003
• 8: Carlos Castillo and Ricardo Baeza-Yates. A new crawling model. In Poster proceedings of the eleventh conference on World Wide Web, Honolulu, Hawaii, USA, May 2002. (Extended Poster).
• 9: Carlos Castillo , Mauricio Marin , Andrea Rodriguez , Ricardo Baeza-Yates, Scheduling Algorithms for Web Crawling, Proceedings of the WebMedia & LA-Web 2004 Joint Conference 10th Brazilian Symposium on Multimedia and the Web 2nd Latin American Web Congress, p.10-17, October 12-15, 2004
• 10: A. Jaimes, J. Ruiz del Solar, R. Verschae, R. Baeza-Yates, C. Castillo, D. Yaksic, and E. Davis. On the image content of a Web segment: Chile as a case study. Journal of Web Engineering, 3(2):153--168, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback
      H.3.4 Systems and Software

          Nouns: World Wide Web (WWW)

General Terms:
Algorithms, Performance