EverLast: a distributed architecture for preserving the web

ABSTRACT

The World Wide Web has become a key source of knowledge pertaining to almost every walk of life. Unfortunately, much of data on the Web is highly ephemeral in nature, with more than 50-80% of content estimated to be changing within a short time. Continuing the pioneering efforts of many national (digital) libraries, organizations such as the International Internet Preservation Consortium (IIPC), the Internet Archive (IA) and the European Archive (EA) have been tirelessly working towards preserving the ever changing Web.

However, while these web archiving efforts have paid significant attention towards long term preservation of Web data, they have paid little attention to developing an global-scale infrastructure for collecting, archiving, and performing historical analyzes on the collected data. Based on insights from our recent work on building text analytics for Web Archives, we propose EverLast, a scalable distributed framework for next generation Web archival and temporal text analytics over the archive. Our system is built on a loosely-coupled distributed architecture that can be deployed over large-scale peer-to-peer networks. In this way, we allow the integration of many archival efforts taken mainly at a national level by national digital libraries. Key features of EverLast include support of time-based text search & analysis and the use of human-assisted archive gathering. In this paper, we outline the overall architecture of EverLast, and present some promising preliminary results.

REFERENCES

• 1: Internet archive. http://archive.org.
• 2: Swedish royal library: Kulturarw3 - long-term preservation of electronic documents. http://www.kb.se/kw3/ENG/.
• 3: Eytan Adar , Mira Dontcheva , James Fogarty , Daniel S. Weld, Zoetrope: interacting with the ephemeral web, Proceedings of the 21st annual ACM symposium on User interface software and technology, October 19-22, 2008, Monterey, CA, USA [doi>10.1145/1449715.1449756]
• 4: Avishek Anand. Indexing partitioning techniques for peer-to-peer web archival. Master's thesis, Universitaet des Saarlandes, FR Informatik, 2009.
• 5: M. Arlitt and T. Jin. 1998 World Cup Site Access Logs. http://www.acm.org/sigcomm/ITA/, 1998.
• 6: William Y. Arms , Selcuk Aya , Pavel Dmitriev , Blazej J. Kot , Ruth Mitchell , Lucia Walle, Building a research library for the history of the web, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141771]
• 7: R. A. Baeza-Yates, C. Castillo, F. Junqueira, V. Plachouras, and F. Silvestri. Challenges on distributed web retrieval. In Proc. of ICDE, 2007.
• 8: Bruno Becker , Stephan Gschwind , Thomas Ohler , Bernhard Seeger , Peter Widmayer, An asymptotically optimal multiversion B-tree, The VLDB Journal — The International Journal on Very Large Data Bases, v.5 n.4, p.264-275, December 1996 [doi>10.1007/s007780050028]
• 9: M. Bender, S. Michel, J. X. Parreira, and T. Crecelius. P2p web search: Make it light, make it y (demo). In Proc. of CIDR, 2007.
• 10: Klaus Berberich , Srikanta Bedathur , Thomas Neumann , Gerhard Weikum, A time machine for text search, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277831]
• 11: Klaus Berberich , Srikanta Bedathur , Thomas Neumann , Gerhard Weikum, FluxCapacitor: efficient time-travel text search, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 12: K. Berberich, S. Bedathur, and G. Weikum. Efficient Time-travel on Versioned Text Collections. In Proc. of GI-Fachtagung fur Datenbanksysteme in Business, Technologie und Web (BTW), 2007.
• 13: K. Berberich, S. Bedathur, and G. Weikum. Tunable Word--Level Index Compression for Versioned Corpora. In Proc. of Workshop EIIR, 2008.
• 14: Ranjita Bhagwan , Kiran Tati , Yu-Chung Cheng , Stefan Savage , Geoffrey M. Voelker, Total recall: system support for automated availability management, Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation, p.25-25, March 29-31, 2004, San Francisco, California
• 15: Byung-Gon Chun , Frank Dabek , Andreas Haeberlen , Emil Sit , Hakim Weatherspoon , M. Frans Kaashoek , John Kubiatowicz , Robert Morris, Efficient replica maintenance for distributed storage systems, Proceedings of the 3rd conference on Networked Systems Design & Implementation, p.4-4, May 08-10, 2006, San Jose, CA
• 16: James Clifford , Curtis Dyreson , Tomás Isakowitz , Christian S. Jensen , Richard Thomas Snodgrass, On the semantics of “now” in databases, ACM Transactions on Database Systems (TODS), v.22 n.2, p.171-214, June 1997 [doi>10.1145/249978.249980]
• 17: Philippe Cudre-Mauroux , Suchit Agarwal , Karl Aberer, GridVine: An Infrastructure for Peer Information Management, IEEE Internet Computing, v.11 n.5, p.36-44, September 2007 [doi>10.1109/MIC.2007.108]
• 18: Heritrix Archival Crawler. http://crawler.archive.org/.
• 19: E. Herder. Characterizations of User Web Revisit Behavior. In Proc. of Workshop on Adaptivity and User Modeling in Interactive Systems, 2005.
• 20: Panos Kalnis , Wee Siong Ng , Beng Chin Ooi , Kian-Lee Tan, Answering similarity queries in peer-to-peer networks, Information Systems, v.31 n.1, p.57-72, March 2006 [doi>10.1016/j.is.2004.09.003]
• 21: Ramakrishna Kotla , Lorenzo Alvisi , Mike Dahlin, SafeStore: a durable and practical storage system, 2007 USENIX Annual Technical Conference on Proceedings of the USENIX Annual Technical Conference, p.1-14, June 17-22, 2007, Santa Clara, CA
• 22: David Lomet , Mingsheng Hong , Rimma Nehme , Rui Zhang, Transaction time indexing with version compression, Proceedings of the VLDB Endowment, v.1 n.1, August 2008 [doi>10.1145/1453856.1453951]
• 23: David Lomet , Betty Salzberg, Access methods for multiversion data, Proceedings of the 1989 ACM SIGMOD international conference on Management of data, p.315-324, June 1989, Portland, Oregon, United States
• 24: Alexandros Ntoulas , Junghoo Cho , Christopher Olston, What's new on the web?: the evolution of the web from a search engine perspective, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988674]
• 25: Josiane Xavier Parreira , Carlos Castillo , Debora Donato , Sebastian Michel , Gerhard Weikum, The Juxtaposed approximate PageRank method for robust PageRank approximation in a peer-to-peer web search network, The VLDB Journal — The International Journal on Very Large Data Bases, v.17 n.2, p.291-313, March 2008 [doi>10.1007/s00778-007-0057-y]
• 26: I. Podnar, M. Rajman, T. Luu, F. Klemm, and K. Aberer. Scalable peer-to-peer web retrieval with highly discriminative keys. In Proc. of ICDE, 2007.
• 27: Antony I. T. Rowstron , Peter Druschel, Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems, Proceedings of the IFIP/ACM International Conference on Distributed Systems Platforms Heidelberg, p.329-350, November 12-16, 2001
• 28: Antony Rowstron , Peter Druschel, Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 29: Betty Salzberg , Vassilis J. Tsotras, Comparison of access methods for time-evolving data, ACM Computing Surveys (CSUR), v.31 n.2, p.158-221, June 1999 [doi>10.1145/319806.319816]
• 30: V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In Proc. of ICDE, 2001.
• 31: A. Singh, M. Srivatsa, L. Liu, and T. Miller. Apoidea: A decentralized peer-to-peer architecture for crawling the world wide web. In Proc. of ACM SIGIR, 2003.
• 32: Ion Stoica , Robert Morris , David Karger , M. Frans Kaashoek , Hari Balakrishnan, Chord: A scalable peer-to-peer lookup service for internet applications, Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications, p.149-160, August 2001, San Diego, California, United States
• 33: Stephan Strodl , Florian Motlik , Kevin Stadler , Andreas Rauber, Personal & soho archiving, Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, June 16-20, 2008, Pittsburgh PA, PA, USA [doi>10.1145/1378889.1378910]
• 34: Christos Tryfonopoulos , Christian Zimmer , Gerhard Weikum , Manolis Koubarakis, Architectural Alternatives for Information Filtering in Structured Overlays, IEEE Internet Computing, v.11 n.4, p.24-34, July 2007 [doi>10.1109/MIC.2007.79]
• 35: Hakim Weatherspoon , Chris Wells , Patrick R. Eaton , Ben Y. Zhao , John D. Kubiatowicz, Silverback: A Global-Scale Archival System, University of California at Berkeley, Berkeley, CA, 2001
• 36: The Size of the World Wide Web. http://www.worldwidewebsize.com/, March 2008.
• 37: C. Zimmer, S. Bedathur, and G. Weikum. Flood Little, Cache More: Effective Result-reuse in P2P IR Systems. In Proc. of DASFAA, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.6 Library Automation
          Subjects: Large text archives

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Design, Experimentation

Keywords:
crawling, indexing, time-travel search, web archives