Sitemaps: above and beyond the crawl of duty

ABSTRACT

Comprehensive coverage of the public web is crucial to web search engines. Search engines use crawlers to retrieve pages and then discover new ones by extracting the pages' outgoing links. However, the set of pages reachable from the publicly linked web is estimated to be significantly smaller than the invisible web, the set of documents that have no incoming links and can only be retrieved through web applications and web forms. The Sitemaps protocol is a fast-growing web protocol supported jointly by major search engines to help content creators and search engines unlock this hidden data by making it available to search engines. In this paper, we perform a detailed study of how "classic" discovery crawling compares with Sitemaps, in key measures such as coverage and freshness over key representative websites as well as over billions of URLs seen at Google. We observe that Sitemaps and discovery crawling complement each other very well, and offer different tradeoffs.

REFERENCES

• 1: Hsin-Tsang Lee , Derek Leonard , Xiaoming Wang , Dmitri Loguinov, IRLbot: scaling to 6 billion pages and beyond, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367556]
• 2: Ricardo Baeza-Yates , Carlos Castillo , Mauricio Marin , Andrea Rodriguez, Crawling a country: better strategies than breadth-first for web page ordering, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062768]
• 3: Ziv Bar-Yossef , Maxim Gurevich, Random sampling from a search engine's index, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135833]
• 4: Ziv Bar-Yossef , Idit Keidar , Uri Schonfeld, Do not crawl in the dust: different urls with similar text, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242588]
• 5: M.K. Bergman. The Deep Web: Surfacing hidden value. Journal of Electronic Publishing, 7(1):07--01, 2001.
• 6: Monica Bianchini , Marco Gori , Franco Scarselli, Inside PageRank, ACM Transactions on Internet Technology (TOIT), v.5 n.1, p.92-128, February 2005 [doi>10.1145/1052934.1052938]
• 7: Onn Brandman , Junghoo Cho , Hector Garcia-Molina , Narayanan Shivakumar, Crawler-Friendly Web Servers, ACM SIGMETRICS Performance Evaluation Review, v.28 n.2, p.9-14, Sept. 2000 [doi>10.1145/362883.362894]
• 8: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 9: Junghoo Cho , Hector Garcia-Molina, Synchronizing a database to improve freshness, ACM SIGMOD Record, v.29 n.2, p.117-128, June 2000
• 10: Junghoo Cho , Hector Garcia-Molina, The Evolution of the Web and Implications for an Incremental Crawler, Proceedings of the 26th International Conference on Very Large Data Bases, p.200-209, September 10-14, 2000
• 11: Junghoo Cho , Hector Garcia-Molina, Effective page refresh policies for Web crawlers, ACM Transactions on Database Systems (TODS), v.28 n.4, p.390-426, December 2003 [doi>10.1145/958942.958945]
• 12: Junghoo Cho , Uri Schonfeld, RankMass crawler: a crawler with high personalized pagerank coverage guarantee, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 13: Dennis Fetterly , Mark Manasse , Marc Najork , Janet L. Wiener, A large-scale study of the evolution of web pages, Software—Practice & Experience, v.34 n.2, p.213-237, February 2004 [doi>10.1002/spe.577]
• 14: Dennis Fetterly , Mark Manasse , Marc Najork, Spam, damn spam, and statistics: using statistical analysis to locate spam web pages, Proceedings of the 7th International Workshop on the Web and Databases: colocated with ACM SIGMOD/PODS 2004, June 17-18, 2004, Paris, France [doi>10.1145/1017074.1017077]
• 15: Google. Joint support for the Sitemap protocol. Available online at: http://googlewebmastercentral.blogspot.com/2006/11/joint-support-for-sitemap-protocol.html, 2006.
• 16: Google. Retiring support for OAI. Available online at: http://googlewebmastercentral.blogspot.com/2008/04/retiring-support-for-oai-pmh-in.html, 2008.
• 17: Google. We knew the web was big... Available online at: http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html, 2008.
• 18: Microsoft Google, Yahoo. Sitemaps.org. Available online at: http://sitemaps.org, 2008.
• 19: J. Gray. A conversation with Werner Vogels. Available online at: http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=388, 2006.
• 20: Open Archive Initiative. Open archive. Available online at: http://www.openarchives.org, 2008.
• 21: Jayant Madhavan , David Ko , Łucja Kot , Vignesh Ganapathy , Alex Rasmussen , Alon Halevy, Google's Deep Web crawl, Proceedings of the VLDB Endowment, v.1 n.2, August 2008 [doi>10.1145/1454159.1454163]
• 22: Gurmeet Singh Manku , Arvind Jain , Anish Das Sarma, Detecting near-duplicates for web crawling, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242592]
• 23: Marc Najork , Janet L. Wiener, Breadth-first crawling yields high-quality pages, Proceedings of the 10th international conference on World Wide Web, p.114-118, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371965]
• 24: Alexandros Ntoulas , Junghoo Cho , Christopher Olston, What's new on the web?: the evolution of the web from a search engine perspective, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988674]
• 25: Christopher Olston , Sandeep Pandey, Recrawl scheduling based on information longevity, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367557]
• 26: Sandeep Pandey , Christopher Olston, User-centric Web crawling, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060805]
• 27: P. Ranjan and N. Shivakumar. Sitemaps: A content discovery protocol for the Web. In Proc. 17th WWW, 2008.
• 28: Reuters. Google, 4 states partner on government info search. Available online at: http://www.reuters.com/article/domesticNews/idUSN2946293620070430?sp=true, 2007.
• 29: J. L. Wolf , M. S. Squillante , P. S. Yu , J. Sethuraman , L. Ozsen, Optimal crawling strategies for web search engines, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511465]
• 30: Uri Schonfeld , Narayanan Shivakumar, Sitemaps: above and beyond the crawl of duty, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain [doi>10.1145/1526709.1526842]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation

Keywords:
crawling, metrics, quality, search engines, sitemaps