Creating trading networks of digital archives

ABSTRACT

Digital archives can best survive failures if they have made several c opies of their collections at remote sites. In this paper, we discuss how autonomous sites can cooperate to provide preservation by trading data. We examine the decisions that an archive must make when forming trading networks, such as the amount of storage space to provide and the best number of partner sites. We also deal with the fact that some sites may be more reliable than others. Experimental results from a data trading simulator illustrate which policies are most reliable. Our techniques focus on preserving the ``bits'' of digital collections; other services that focus on other archiving concerns (such as preserving meaningful metadata) can be built on top of the system we describe here.

REFERENCES

• 1: Farokh B. Bastani , I-Ling Yen, A Fault Tolerant Replicated Storage System, Proceedings of the Third International Conference on Data Engineering, p.449-454, February 03-05, 1987
• 2: Andrea Borr. Transaction monitoring in Encompass {TM}: Reliable distributed transaction processing. In Proc. 7th VLDB, September 1981.
• 3: W. W. Chu. Multiple file allocation in a multiple computer system. IEEE Transactions on Computing, C-18(10):885-889, October 1969.
• 4: Brian Cooper , Arturo Crespo , Hector Garcia-Molina, Implementing a Reliable Digital Object Archive, Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, p.128-143, September 18-20, 2000
• 5: Brian Cooper and Hector Garcia-Molina. Peer to peer data trading to preserve information. http://dbpubs.stanford.edu/pub/2001-7, 2001. Technical Report.
• 6: Arturo Crespo , Hector Garcia-Molina, Modeling Archival Repositories for Digital Libraries, Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, p.190-205, September 18-20, 2000
• 7: Xiaolin Du , Fred J. Maryanski, Data Allocation in a Dynamically Reconfigurable Environment, Proceedings of the Fourth International Conference on Data Engineering, p.74-81, February 01-05, 1988
• 8: Barbara Liskov , Sanjay Ghemawat , Robert Gruber , Paul Johnson , Liuba Shrira, Replication in the harp file system, Proceedings of the thirteenth ACM symposium on Operating systems principles, p.226-238, October 13-16, 1991, Pacific Grove, California, United States
• 9: James H. Morris , Mahadev Satyanarayanan , Michael H. Conner , John H. Howard , David S. Rosenthal , F. Donelson Smith, Andrew: a distributed personal computing environment, Communications of the ACM, v.29 n.3, p.184-201, March 1986 [doi>10.1145/5666.5671]
• 10: John Kubiatowicz , David Bindel , Yan Chen , Steven Czerwinski , Patrick Eaton , Dennis Geels , Ramakrishna Gummadi , Sean Rhea , Hakim Weatherspoon , Chris Wells , Ben Zhao, OceanStore: an architecture for global-scale persistent storage, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.190-201, November 2000, Cambridge, Massachusetts, United States
• 11: John Garrett and Donald Waters. Preserving digital information: Report of the Task Force on Archiving of Digital Information, May 1996. Accessible at http://www.rlg.org/ArchTF/.
• 12: A. V. Goldberg , P. N. Yianilos, Towards an Archival Intermemory, Proceedings of the Advances in Digital Libraries Conference, p.147, April 22-24, 1998
• 13: Jim Gray , Pat Helland , Patrick O'Neil , Dennis Shasha, The dangers of replication and a solution, Proceedings of the 1996 ACM SIGMOD international conference on Management of data, p.173-182, June 04-06, 1996, Montreal, Quebec, Canada
• 14: Alan Heminger and Steven Robertson. Digital Rosetta Stone: A conceptual model for maintaining long-term access to digital documents. In Proc. 6th DELOS Workshop on Preservation of Digital Information, June 1998.
• 15: Hui-I Hsiao , David J. DeWitt, Chained Declustering: A New Availability Strategy for Multiprocessor Database Machines, Proceedings of the Sixth International Conference on Data Engineering, p.456-465, February 05-09, 1990
• 16: James J. Kistler , M. Satyanarayanan, Disconnected operation in the Coda File System, ACM Transactions on Computer Systems (TOCS), v.10 n.1, p.3-25, Feb. 1992 [doi>10.1145/146941.146942]
• 17: Carl Lagoze , Jane Hunter , Dan Brickley, An Event-Aware Model for Metadata Interoperability, Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, p.103-116, September 18-20, 2000
• 18: Edward K. Lee , Chandramohan A. Thekkath, Petal: distributed virtual disks, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.84-92, October 01-04, 1996, Cambridge, Massachusetts, United States
• 19: Nuno Maria, Pedro Gaspar, Antonio Ferreira, and Mario Silva. Information preservation in ARIADNE. In Proc. 6th DELOS Workshop on Preservation of Digital Information, June 1998.
• 20: Silvano Martello , Paolo Toth, Knapsack problems: algorithms and computer implementations, John Wiley & Sons, Inc., New York, NY, 1990
• 21: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), ACM SIGMOD Record, v.17 n.3, p.109-116, June 1988
• 22: Sandra Payette , Carl Lagoze, Policy-Carrying, Policy-Enforcing Digital Objects, Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, p.144-157, September 18-20, 2000
• 23: Arcot Rajasekar, Richard Marciano, and Reagan Moore. Collection-based persistent archives. In Proceedings of the 16th IEEE Symposium on Mass Storage Systems, March 1999.
• 24: David S. H. Rosenthal and Vicky Reich. Permanent web publishing. In Proc. USENIX Annual Technical Conference, June 2000.
• 25: Jeff Rothenberg. Ensuring the longevity of digital documents. Scientific American, 272(1):24-29, January 1995.
• 26: Harjinder S. Sandhu , Songnian Zhou, Cluster-based file replication in large-scale distributed systems, Proceedings of the 1992 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, p.91-102, June 01-05, 1992, Newport, Rhode Island, United States
• 27: Ouri Wolfson , Sushil Jajodia , Yixiu Huang, An adaptive data replication algorithm, ACM Transactions on Database Systems (TODS), v.22 n.2, p.255-314, June 1997 [doi>10.1145/249978.249982]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Algorithms, Design, Documentation, Human Factors, Management, Measurement, Performance, Reliability, Theory

Keywords:
data trading, digital archiving, fault tolerance, preservation, replication