Lazy preservation: reconstructing websites by crawling the crawlers

ABSTRACT

Backup of websites is often not considered until after a catastrophic event has occurred to either the website or its webmaster. We introduce "lazy preservation" -- digital preservation performed as a result of the normal operation of web crawlers and caches. Lazy preservation is especially suitable for third parties; for example, a teacher reconstructing a missing website used in previous classes. We evaluate the effectiveness of lazy preservation by reconstructing 24 websites of varying sizes and composition using Warrick, a web-repository crawler. Because of varying levels of completeness in any one repository, our reconstructions sampled from four different web repositories: Google (44%), MSN (30%), Internet Archive (19%) and Yahoo (7%). We also measured the time required for web resources to be discovered and cached (10-103 days) as well as how long they remained in cache after deletion (7-61 days).

REFERENCES

• 1: Hal Berghel, Responsible web caching, Communications of the ACM, v.45 n.9, September 2002 [doi>10.1145/567498.567514]
• 2: Krishna Bharat , Andrei Broder, Mirror, mirror on the Web: a study of host pairs with replicated content, Proceeding of the eighth international conference on World Wide Web, p.1579-1590, May 1999, Toronto, Canada
• 3: Andrei Z. Broder , Steven C. Glassman , Mark S. Manasse , Geoffrey Zweig, Syntactic clustering of the Web, Computer Networks and ISDN Systems, v.29 n.8-13, p.1157-1166, Sep. 1997
• 4: M. Burner. Crawling towards eternity: Building an archive of the world wide web. Web Techniques Magazine, 2(5), 1997.
• 5: Fazli Can , Rabia Nuray , Ayisigi B. Sevdik, Automatic performance evaluation of web search engines, Information Processing and Management: an International Journal, v.40 n.3, p.495-514, January 2004 [doi>10.1016/S0306-4573(03)00040-2]
• 6: Junghoo Cho , Narayanan Shivakumar , Hector Garcia-Molina, Finding replicated Web collections, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.355-366, May 15-18, 2000, Dallas, Texas, United States
• 7: B. F. Cooper and H. Garcia-Molina. Infomonitor: Unobtrusively archiving a World Wide Web server. International Journal on Digital Libraries, 5(2):106--119, April 2005.
• 8: M. Day. Collecting and preserving the World Wide Web. 2003. http://library.wellcome.ac.uk/assets/WTL039229.pdf.
• 9: Curtis E. Dyreson , Hui-ling Lin , Yingxia Wang, Managing versions of web documents in a transaction-time web server, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988730]
• 10: Dennis Fetterly , Mark Manasse , Marc Najork, Spam, damn spam, and statistics: using statistical analysis to locate spam web pages, Proceedings of the 7th International Workshop on the Web and Databases: colocated with ACM SIGMOD/PODS 2004, June 17-18, 2004, Paris, France [doi>10.1145/1017074.1017077]
• 11: Dennis Fetterly , Mark Manasse , Marc Najork , Janet Wiener, A large-scale study of the evolution of web pages, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775246]
• 12: Google Sitemap Protocol, 2005. http://www.google.com/webmasters/sitemaps/docs/en/protocol.html.
• 13: Google webmaster help center: Webmaster guidelines, 2006. http://www.google.com/support/webmasters/bin/answer.py?answer=35769.
• 14: Michael Gordon , Praveen Pathak, Finding information on the World Wide Web: the retrieval effectiveness of search engines, Information Processing and Management: an International Journal, v.35 n.2, p.141-180, March 1999 [doi>10.1016/S0306-4573(98)00041-7]
• 15: A. Gulli , A. Signorini, The indexable web is more than 11.5 billion pages, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062789]
• 16: Internet Archive FAQ: How can I get my site included in the Archive?, 2006. http://www.archive.org/about/faqs.php.
• 17: Dirk Lewandowski , Henry Wahlig , Gunnar Meyer-Bautor, The freshness of web search engine databases, Journal of Information Science, v.32 n.2, p.131-148, April 2006 [doi>10.1177/0165551506062326]
• 18: Frank McCown , Xiaoming Liu , Michael L. Nelson , Mohammad Zubair, Search Engine Coverage of the OAI-PMH Corpus, IEEE Internet Computing, v.10 n.2, p.66-73, March 2006 [doi>10.1109/MIC.2006.41]
• 19: Frank McCown , Michael L. Nelson, Evaluation of crawling policies for a web-repository crawler, Proceedings of the seventeenth conference on Hypertext and hypermedia, August 22-25, 2006, Odense, Denmark [doi>10.1145/1149941.1149972]
• 20: F. McCown, J. A. Smith, M. L. Nelson, and J. Bollen. Reconstructing websites for the lazy webmaster. Technical report, Old Dominion University, 2005. http://arxiv.org/abs/cs.IR/0512069.
• 21: Alexandros Ntoulas , Junghoo Cho , Christopher Olston, What's new on the web?: the evolution of the web from a search engine perspective, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988674]
• 22: James S. Plank, A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems, Software—Practice & Experience, v.27 n.9, p.995-1012, Sept. 1997 [doi>10.1002/(SICI)1097-024X(199709)27:9<995::AID-SPE111>3.3.CO;2-Y]
• 23: Herman Chung-Hwa Rao , Yih-Farn Chen , Ming-Feng Chen, A proxy-based personal web archiving service, ACM SIGOPS Operating Systems Review, v.35 n.1, p.61-72, January 1, 2001 [doi>10.1145/371455.371462]
• 24: V. Reich and D. S. Rosenthal. LOCKSS: A permanent web publishing and access system. D-Lib Magazine, 7(6), 2001.
• 25: A. Ross. Internet Archive forums: Web forum posting. Oct 2004. http://www.archive.org/iathreads/post-view.php?id=23121.
• 26: J. A. Smith, F. McCown, and M. L. Nelson. Observed web robot behavior on decaying web subsites. D-Lib Magazine, 12(2), Feb 2006.
• 27: Melius Weideman , Mongezi Mgidana, Website navigation architectures and their effect on website visibility: a literature survey, Proceedings of the 2004 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, p.292-296, October 04-06, 2004, Stellenbosch, Western Cape, South Africa
• 28: Jin Zhang , Alexandra Dimitroff, The impact of webpage content characteristics on webpage visibility in search engine results (part I), Information Processing and Management: an International Journal, v.41 n.3, p.665-690, May 2005 [doi>10.1016/j.ipm.2003.12.001]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Collection

General Terms:
Design, Experimentation, Measurement

Keywords:
cached resources, digital preservation, recovery, search engine