The LOCKSS peer-to-peer digital preservation system

ABSTRACT

The LOCKSS project has developed and deployed in a world-wide test a peer-to-peer system for preserving access to journals and other archival information published on the Web. It consists of a large number of independent, low-cost, persistent Web caches that cooperate to detect and repair damage to their content by voting in “opinion polls.” Based on this experience, we present a design for and simulations of a novel protocol for voting in systems of this kind. It incorporates rate limitation and intrusion detection to ensure that even some very powerful adversaries attacking over many years have only a small probability of causing irrecoverable damage before being detected.

REFERENCES

• 1: Abadi, M., Burrows, M., Manasse, M., and Wobber, T. 2003. Moderately hard, memory-bound functions. In Proceedings of the 10th Annual Network and Distributed System Security Symposium (San Diego, CA).
• 2: Anderson, R. J. 1996. The Eternity Service. In Proceedings of the 1st International Conference on the Theory and Applications of Cryptology (PRAGOCRYPT 1996, Prague, Czech Republic).
• 3: ARL---Association of Research Libraries. 2001. ARL Statistics 2000-01. Available online at http://www.arl.org/stats/arlstat/01pub/intro.html.
• 4: Back, A. 2002. Hashcash---a denial of service counter measure. Available online at http://www.hashcash.org/hashcash.pdf.
• 5: Kenneth P. Birman , Mark Hayden , Oznur Ozkasap , Zhen Xiao , Mihai Budiu , Yaron Minsky, Bimodal multicast, ACM Transactions on Computer Systems (TOCS), v.17 n.2, p.41-88, May 1999 [doi>10.1145/312203.312207]
• 6: Miguel Castro , Barbara Liskov, Practical Byzantine fault tolerance, Proceedings of the third symposium on Operating systems design and implementation, p.173-186, February 1999, New Orleans, Louisiana, United States
• 7: Yuan Chen , Jan Edler , Andrew Goldberg , Allan Gottlieb , Sumeet Sobti , Peter Yianilos, A prototype implementation of archival Intermemory, Proceedings of the fourth ACM conference on Digital libraries, p.28-37, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313249]
• 8: Chor, B. and Dwork, C. 1989. Randomization in Byzantine agreement. Advan. Comput. Res. 5, 443--497.
• 9: Ian Clarke , Oskar Sandberg , Brandon Wiley , Theodore W. Hong, Freenet: a distributed anonymous information storage and retrieval system, International workshop on Designing privacy enhancing technologies: design issues in anonymity and unobservability, p.46-66, January 2001, Berkeley, California, United States
• 10: ConservationOnline. 2003. Electronic storage media. Available online at http://palimpsest.stanford.edu/bytopic/electronic-records/electronic-storage-media/.
• 11: Frank Dabek , M. Frans Kaashoek , David Karger , Robert Morris , Ion Stoica, Wide-area cooperative storage with CFS, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 12: Davis, D. 1996. Compliance defects in public-key cryptography. In Proceedings of the 6th USENIX Security Symposium (San Jose, CA). 171--178.
• 13: Dean, D. and Stubblefield, A. 2001. Using client puzzles to protect TLS. In Proceedings of the 10th USENIX Security Symposium (Washington, DC).
• 14: Diffie, W. 2003. Perspective: Decrypting the secret to strong security. Available online at http://news.com.com/2010-1071-980462.html.
• 15: Diffie, W. and Hellman, M. E. 1976. New directions in cryptography. IEEE Trans. Inform. Theor. IT-22, 6 (Nov.), 644--654.
• 16: Roger Dingledine , Michael J. Freedman , David Molnar, The free haven project: distributed anonymous storage service, International workshop on Designing privacy enhancing technologies: design issues in anonymity and unobservability, p.67-95, January 2001, Berkeley, California, United States
• 17: John R. Douceur, The Sybil Attack, Revised Papers from the First International Workshop on Peer-to-Peer Systems, p.251-260, March 07-08, 2002
• 18: Dwork, C., Goldberg, A., and Naor, M. 2003. On memory-bound functions for fighting spam. In Proceedings of the 23rd Annual International Cryptology Conference (Santa Barbara, CA).
• 19: Cynthia Dwork , Moni Naor, Pricing via Processing or Combatting Junk Mail, Proceedings of the 12th Annual International Cryptology Conference on Advances in Cryptology, p.139-147, August 16-20, 1992
• 20: Erdos, M. and Cantor, S. 2002. Shibboleth Architecture DRAFT v05. Available online at http://shibboleth.internet2.edu/docs/draft-internet2-shibboleth-arch-v05.pdf. Work in progress.
• 21: Friedman, E. J. and Resnick, P. 2001. The social costs of cheap pseudonyms. J. Econom. Managem. Strat. 10, 2 (Summer), 173--199.
• 22: Fullmer, C. 2002. Storage and multimedia: The facts and more. Available online at http://www.cse.ucsc.edu/classes/cmpe003/Fall02/L11_ch6.pps.
• 23: Giuli, T. and Baker, M. 2002. Narses: A scalable, flow-based network simulator. Tech. rep. arXiv:cs.PF/0211024. Computer Science Department, Stanford University, Stanford, CA.
• 24: Horlings, J. 2003. Cd-r's binnen twee jaar onleesbaar. Available online at http://www.pc-active.nl/toonArtikel.asp?artikelID=508. http://www.cdfreaks.com/news/7751.
• 25: Keller, M., Reich, V., and Herkovic, A. 2003. What is a library anymore, anyway? First Monday 8, 5 (May). Available online at http://www.firstmonday.org/issues/issue8_5/keller/index.html.
• 26: John Kubiatowicz , David Bindel , Yan Chen , Steven Czerwinski , Patrick Eaton , Dennis Geels , Ramakrishna Gummadi , Sean Rhea , Hakim Weatherspoon , Chris Wells , Ben Zhao, OceanStore: an architecture for global-scale persistent storage, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.190-201, November 2000, Cambridge, Massachusetts, United States
• 27: Dahlia Malkhi , Michael Reiter, Byzantine quorum systems, Distributed Computing, v.11 n.4, p.203-213, October 1998 [doi>10.1007/s004460050050]
• 28: Maniatis, P., Giuli, T., Roussopoulos, M., Rosenthal, D. S. H., and Baker, M. 2004. Impeding attrition attacks in P2P systems. In Proceedings of the 11th ACM SIGOPS European Workshop (Leuven, Belgium).
• 29: Petros Maniatis , David S. H. Rosenthal , Mema Roussopoulos , Mary Baker , TJ Giuli , Yanto Muliadi, Preserving peer replicas by rate-limited sampled voting, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 30: Manns, B. and Shahani, C. J. 2003. Longevity of CD media research at the Library of Congress. Available online at http://www.loc.gov/preserv/study\%20of\%20CD\%20longevity.pdf.
• 31: Michalakis, N., Chiu, D.-M., and Rosenthal, D. S. H. 2003. Long term data resilience using opinion polls. In Proceedings of the 22nd IEEE International Performance Computing and Communications Conference (Phoenix, AZ).
• 32: Mogge, D. 1999. Seven years of tracking electronic publishing: The ARL Directory of Electronic Journals, Newsletters and Academic Discussion Lists. Library Hi Tech 17, 1, 17--25.
• 33: Moore, D., Voelker, G. M., and Savage, S. 2001. Inferring Internet denial-of-service activity. In Proceedings of the 10th USENIX Security Symposium.
• 34: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 35: Quirke, C. 2002. Hard drive data corruption. Available online at http://users.iafrica.com/c/cq/cquirke/baddata.htm.
• 36: Michael K. Reiter, Secure agreement protocols: reliable and atomic group multicast in rampart, Proceedings of the 2nd ACM Conference on Computer and communications security, p.68-80, November 1994, Fairfax, Virginia, United States [doi>10.1145/191177.191194]
• 37: Rhea, S., Geels, D., Roscoe, T., and Kubiatowicz, J. 2004. Handling Churn in a DHT. In Proceedings of the Usenix Annual Technical Conference (Boston, MA).
• 38: Rosenthal, D. S. H. 2003. A digital preservation network appliance based on OpenBSD. In Proceedings of BSDcon 2003 (San Mateo, CA).
• 39: Rosenthal, D. S. H. and Reich, V. 2000. Permanent Web publishing. In Proceedings of the USENIX Annual Technical Conference, Freenix Track (San Diego, CA). 129--140.
• 40: Rosenthal, D. S. H., Roussopoulos, M., Maniatis, P., and Baker, M. 2003. Economic measures to resist attacks on a peer-to-peer network. In Proceedings of the Workshop on Economics of Peer-to-Peer Systems (Berkeley, CA).
• 41: Antony Rowstron , Peter Druschel, Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 42: Stuart Staniford , Vern Paxson , Nicholas Weaver, How to Own the Internet in Your Spare Time, Proceedings of the 11th USENIX Security Symposium, p.149-167, August 05-09, 2002
• 43: Tenopir, C. 2004. Online scholarly journals: How many? The Library Journal 2 (Feb.). Available online at http://www.libraryjournal.com/index.asp?layout=articlePrint&articleID=C%A374956.
• 44: U. S. Government. 2004. Web design and usability guidelines. Available online at http://usability.gov/guidelines/softhard.html.
• 45: Venema, W. 1996. Murphy's Law and computer security. In Proceedings of the 6th USENIX Security Symposium (San Jose, CA).
• 46: Marc Waldman , David Mazières, Tangler: a censorship-resistant publishing system based on document entanglements, Proceedings of the 8th ACM conference on Computer and Communications Security, November 05-08, 2001, Philadelphia, PA, USA [doi>10.1145/501983.502002]
• 47: Matthew M. Williamson, Throttling Viruses: Restricting propagation to defeat malicious mobile code, Proceedings of the 18th Annual Computer Security Applications Conference, p.61, December 09-13, 2002
• 48: Wingfield, N. 2002. EBay's figurine scandal: Auction site merchant disappears with the goods. Wall Street J., Feb 22.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability

General Terms:
Design, Economics, Reliability

Keywords:
Rate limiting, digital preservation, replicated storage