New techniques for ensuring the long term integrity of digital archives

ABSTRACT

A large portion of the government, business, cultural, and scientific digital data being created today needs to be archived and preserved for future use of periods ranging from a few years to decades and sometimes centuries. A fundamental requirement of a long term archive is to ensure the integrity of its holdings. In this paper, we develop a new methodology to address the integrity of long term archives using rigorous cryptographic techniques. Our approach involves the generation of a small-size integrity token for each digital object to be archived, and some cryptographic summary information based on all the objects handled within a dynamic time period. We present a framework that enables the continuous auditing of the holdings of the archive depending on the policy set by the archive. Moreover, an independent auditor will be able to verify the integrity of every version of an archived digital object as well as link the current version to the original form of the object when it was ingested into the archive. We built a prototype system that is completely independent of the archive's underlying architecture, and tested it on large scale data. We include in this paper some preliminary results on the validation and performance of our prototype.

REFERENCES

• 1: Chaitanya Baru , Reagan Moore , Arcot Rajasekar , Michael Wan, The SDSC storage resource broker, Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research, p.5, November 30-December 03, 1998, Toronto, Ontario, Canada
• 2: Johannes Buchmann , Alexander May , Ulrich Vollmer, Perspectives for cryptographic long-term security, Communications of the ACM, v.49 n.9, September 2006 [doi>10.1145/1151030.1151055]
• 3: W. Diffie and M. E. Hellman, "New directions in cryptography," IEEE Trans. Inform. Theory, IT-22, 6, 1976, pp.644--654.
• 4: Henry M. Gladney, Trustworthy 100-year digital objects: Evidence after every witness is dead, ACM Transactions on Information Systems (TOIS), v.22 n.3, p.406-436, July 2004 [doi>10.1145/1010614.1010617]
• 5: M. Gudgin, M. Hadley, N. Mendelsohn, J. Moreau and H. F. Nielsen, "SOAP Version 1.2 Part 1: Messaging Framework ", W3C Proposed Recommendation, 2003.
• 6: Stuart Haber and Pandurang Kamat. "Content Integrity Service for Long-Term Digital Archives." In Proceedings of Archiving 2006, May 2006, pp 159--164.
• 7: Stuart Haber and W. Scott Stornella, "How to time-stamp a digital document," Journal of Cryptology, 1991.
• 8: Ronald Jantz and Michael J. Giarlo. "Digital Preservation - Architecture and Technology for Trusted Digital Repositories Reich." D-Lib Magazine, 7(6), June 2005. http://www.dlib.org/dlib/june05/jantz/06jantz.html
• 9: Charlie Kaufman , Radia Perlman , Mike Speciner, Network security: private communication in a public world, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995
• 10: Lisa Kelly. "British Library secures integrity of digital archive." Computing. 25 Apr 25 2006. http://www.computing.co.uk/computing/news/2154704/britis h-li
• 11: Leslie Lamport , Robert Shostak , Marshall Pease, The Byzantine Generals Problem, ACM Transactions on Programming Languages and Systems (TOPLAS), v.4 n.3, p.382-401, July 1982 [doi>10.1145/357172.357176]
• 12: Clifford A. Lynch. "Authenticity and Integrity in the Digital Environment: An Exploratory Analysis of the Central Role of Trust." Authenticity in a Digital Environment (Washington, DC: Council on Library and Information Resources, 2000), pp 32--50. http://www.clir.org/pubs/reports/pub92/lynch.html
• 13: Petros Maniatis, T. J. Giuli, and Mary Baker, "Enabling the Long-Term Archival of Signed Documents through Time Stamping," May 2006.
• 14: Petros Maniatis , Mema Roussopoulos , T. J. Giuli , David S. H. Rosenthal , Mary Baker, The LOCKSS peer-to-peer digital preservation system, ACM Transactions on Computer Systems (TOCS), v.23 n.1, p.2-50, February 2005 [doi>10.1145/1047915.1047917]
• 15: Alfred J. Menezes , Scott A. Vanstone , Paul C. Van Oorschot, Handbook of Applied Cryptography, CRC Press, Inc., Boca Raton, FL, 1996
• 16: Ralph Merkle. "Protocols for public key cryptosystems." In Proceedings of the 1980 Symposium on Security and Privacy, IEEE Computer Society Press, 1980, pp 122--133.
• 17: MSN Groups, http://groups.msn.com/
• 18: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 19: Google Groups, http://groups.google.com/
• 20: Gopalan Sivathanu, Charles P. Wright, and Erez Zadok." Ensuring Data Integrity in Storage: Techniques and Applications." ACM TOS, 2005.
• 21: Surety, Inc. http://www.surety.com/
• 22: Xiaoyun Wang and Hongbo Yu. "How to break MD5 and other hash functions." In Ronald Cramer, editor, Advances in Cryptology - EUROCRYPT 2005, Volume 3494 of Lecture Notes in Computer Science, 2005.
• 23: Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu. "Finding collisions in the full SHA-1." In Victor Shoup, editor, Advances in Cryptology --- CRYPTO 2005, volume 3621 of Lecture Notes in Computer Science, 2005.
• 24: Yahoo Groups. http://groups.yahoo.com/

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
      H.3.6 Library Automation

General Terms:
Algorithms, Design, Reliability, Security

Keywords:
data integrity, digital archives, integrity audits, linked hashing