A fresh look at the reliability of long-term digital storage

ABSTRACT

Emerging Web services, such as email, photo sharing, and web site archives, must preserve large volumes of quickly accessible data indefinitely into the future. The costs of doing so often determine whether the service is economically viable. We make the case that these applications' demands on large scale storage systems over long time horizons require us to reevaluate traditional system designs. We examine threats to long-lived data from an end-to-end perspective, taking into account not just hardware and software faults but also faults due to humans and organizations. We present a simple model of long-term storage failures that helps us reason about various strategies for addressing some of these threats. Using this model we show that the most important strategies for increasing the reliability of long-term storage are detecting latent faults quickly, automating fault repair to make it cheaper and faster, and increasing the independence of data replicas.

REFERENCES

• 1: 104th Congress, United States of America. Public Law 104--191: Health Insurance Portability and Accountability Act (HIPAA), Aug. 1996.
• 2: 107th Congress, United States of America. Public Law 107--204: Sarbanes-Oxley Act of 2002, July 2002.
• 3: D. Akst. Postcard from Cyberspace. Los Angeles Times, Jan. 1995.
• 4: AMIA2003. Fact Sheet 5 - Estimating Tape Life. http://www.amianet.org/publication/resources/guidelines/videofacts/tapelife.html, 2003.
• 5: Dave Anderson , Jim Dykes , Erik Riedel, More Than an Interface---SCSI vs. ATA, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA
• 6: R. J. Anderson. The Eternity Service. In 1st Intl. Conf. on the Theory and Applications of Cryptology, 1996.
• 7: ARL - Association of Research Libraries. ARL Statistics 2000--01. http://www.arl.org/stats/arlstat/01pub/intro.html, 2001.
• 8: M. Baker, K. Keeton, and S. Martin. Why Traditional Storage Systems Don't Help Us Save Stuff Forever. In Proc. 1st IEEE Workshop on Hot Topics in System Dependability, 2005.
• 9: Wendy Bartlett , Lisa Spainhower, Commercial Fault Tolerance: A Tale of Two Systems, IEEE Transactions on Dependable and Secure Computing, v.1 n.1, p.87-96, January 2004 [doi>10.1109/TDSC.2004.4]
• 10: Peter M. Chen , Edward K. Lee , Garth A. Gibson , Randy H. Katz , David A. Patterson, RAID: high-performance, reliable secondary storage, ACM Computing Surveys (CSUR), v.26 n.2, p.145-185, June 1994 [doi>10.1145/176979.176981]
• 11: Yuan Chen , Jan Edler , Andrew Goldberg , Allan Gottlieb , Sumeet Sobti , Peter Yianilos, A prototype implementation of archival Intermemory, Proceedings of the fourth ACM conference on Digital libraries, p.28-37, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313249]
• 12: Peter Corbett , Bob English , Atul Goel , Tomislav Grcanac , Steven Kleiman , James Leong , Sunitha Sankar, Awarded Best Paper! -- Row-Diagonal Parity for Double Disk Failure Correction, Proceedings of the 3rd USENIX Conference on File and Storage Technologies, March 31-31, 2004, San Francisco, CA
• 13: Frank Dabek , M. Frans Kaashoek , David Karger , Robert Morris , Ion Stoica, Wide-area cooperative storage with CFS, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 14: T. Dawber, G. Meadors, and F. Moore. Epidemiological Approaches to Heart Disease: the Framingham Study. American Journal of Public Health, 41(3):279--81, Mar. 1951.
• 15: W. Diffie. Perspective: Decrypting The Secret to Strong Security. http://news.com.com/2010--1071--980462.html, Jan. 2003.
• 16: G. Goble. http://ghg.ecn.purdue.edu/~ghg/.
• 17: Google, Inc. About Gmail. http://gmail.google.com/gmail/help/about.html, June 2005.
• 18: J. Gray, A. Szalay, A. Thakar, C. Stoughton, and J. vandenBerg. Online Scientific Data Curation, Publication, and Archiving. Technical Report MSR-TR-2002--74, Microsoft Research, July 2002.
• 19: J. Gray and C. van Ingen. Emprical Measurements of Disk Failure Rates and Error Rates. Technical Report MSR-TR-2005-166, Microsoft Research, Dec. 2005.
• 20: E. Hansen. Hotmail Incinerates Customer Files. News.com, http://news.com.com/Hotmail+incinerates+customer+files/2100--1038_3--5226090.html, June 2004.
• 21: J. Horlings. CD-R's Binnen Twee Jaar Onleesbaar, 2003. PC Active, See http://www.cdfreaks.com/news/7751.
• 22: IT Committee Inst. of Chartered Accountants of India. Tape backup vis-à-vis online Backup. Harmony IT, http://isaicai.org/Harmony/2004--07/index_plain.htm, July 2004.
• 23: F. Junqueira, R. Bhagwan, A. Hevia, K. Marzullo, and G. M. Voelker. Surviving Internet Catastrophes. In Usenix Annual Technical Conference, 2005.
• 24: H. Kari. Latent Sector Faults and Reliability of Disk Arrays. PhD thesis, Computer Science Department, Helsinki University of Technology, Finaland, Espoo, Finland, 1997.
• 25: M. Keeney, E. Kowalski, D. Cappelli, A. Moore, T. Shimeall, and S. Rogers. Insider Threat Study: Computer System Sabotage in Critical Infrastructure Sectors. http://www.secretservice.gov/ntac/its_report_050516.pdf, May 2005.
• 26: K. Keeton, A Backup Appliance Composed of High-Capacity Disk Drives, Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, p.171, May 20-22, 2001
• 27: John Kubiatowicz , David Bindel , Yan Chen , Steven Czerwinski , Patrick Eaton , Dennis Geels , Ramakrishna Gummadi , Sean Rhea , Hakim Weatherspoon , Chris Wells , Ben Zhao, OceanStore: an architecture for global-scale persistent storage, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.190-201, November 2000, Cambridge, Massachusetts, United States
• 28: R. Lau. Personal Communication, Sept. 2004.
• 29: D. Lazarus. Prccious Photos Disappear. San Francisco Chronicle, http://www.sfgate.com/cgi-bin/article.cgi?file=/chronicle/archive/2005/02/02/BUG7QB3UOS1.DTL, Feb. 2005.
• 30: P. Luse and M. Schmisseur. Understanding Intelligent RAID 6. Technology@Intel Magazine, http://www.intel.com/technology/magazine/computing/RAID-6-0505.htm, 2006.
• 31: R. Malda. The Myth of the 100 Year CD-Rom. Slashdot, http://slashdot.org/article.pl?sid=04/04/22/1658251\&mode=flat\&tid=137\&ti, Apr. 2004.
• 32: Petros Maniatis , Mema Roussopoulos , T. J. Giuli , David S. H. Rosenthal , Mary Baker, The LOCKSS peer-to-peer digital preservation system, ACM Transactions on Computer Systems (TOCS), v.23 n.1, p.2-50, February 2005 [doi>10.1145/1047915.1047917]
• 33: D. Milbank. White House Web Scrubbing, Dec. 2003. The Washington Post, http://www.washingtonpost.com/ac2/wp-dyn?pagename=article&node=&contentId=A9821--2003Dec17¬Found=true.
• 34: C. Mohan, Disk Read-Write Optimizations and Data Integrity in Transaction Systems Using Write-Ahead Logging, Proceedings of the Eleventh International Conference on Data Engineering, p.324-331, March 06-10, 1995
• 35: NASA. Aviation Safety Reporting System. http://asrs.arc.nasa.gov/.
• 36: OCLC. Persistent Uniform Resource Locator. http://purl.oclc.org/.
• 37: K. Pang, K. Yau, and Hung-Hsiang Chou. The Earth's Palaeorotation, Postglacial Rebound and Lower Mantle Viscosity from Analysis of Ancient Chinese Eclipse Records. Pure and Applied Geophysics, 145(3--4):459--485, Sept. 1995.
• 38: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 39: Vijayan Prabhakaran , Lakshmi N. Bairavasundaram , Nitin Agrawal , Haryadi S. Gunawi , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau, IRON file systems, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom
• 40: J. Reason. Human Error. Cambridge University Press, 1990.
• 41: Reuters. Time Warner Says Employee Data Lost by Outside Storage Company. The New York Times, http://www.nytimes.com/2005/05/02/business/business-tech-timewarner.html?ex=1272686400&en=39cc177d5da055d2&ei=5090&partner=rssuserland&emc=rss, May 2005.
• 42: D. S. H. Rosenthal. A Digital Preservation Network Appliance Based on OpenBSD. In Proceedings of BSDcon 2003, San Mateo, CA, USA, Sept. 2003.
• 43: Antony Rowstron , Peter Druschel, Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 44: Jerome H. Saltzer, Fault-tolerance in very large archival systems, Proceedings of the 4th workshop on ACM SIGOPS European workshop, p.1-3, September 03-05, 1990, Bologna, Italy [doi>10.1145/504136.504144]
• 45: Thomas J. E. Schwarz , Qin Xin , Ethan L. Miller , Darrell D. E. Long , Andy Hospodor , Spencer Ng, Disk Scrubbing in Large Archival Storage Systems, Proceedings of the The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems (MASCOTS'04), p.409-418, October 04-08, 2004
• 46: Seagate. ST3200822A Configuration and Specifications. http://www.seagate.com/support/disc/specs/ata/st3200822a.html, Sept. 2003.
• 47: Seagate. Cheetah 15K.4. http://www.seagate.com/cda/products/discsales/enterprise/tech/0,1084,656,00.html, 2005.
• 48: R. F. Sproull and J. Eisenberg. Building an Electronic Records Archive at the National Archives and Records Administration: Recommendations for a Long-Term Strategy, http://www.nap.edu/catalog/11332.html, June 2005.
• 49: Nisha Darshi Talagala , David Patterson, Characterizing large storage systems: error behavior and performance benchmarks, 1999
• 50: The Memory Hole. Department of Education to Delete Years of Research From Its Website. http://www.thememoryhole.org/edu/ed-info.htm, 2002.
• 51: The OpenRAW Working Group. The RAW Problem. http://openraw.org, 2005.
• 52: J. Tom. When Mutilators Stalk the Stacks. http://gort.ucsd.edu/preseduc/bmlmutil.htm, 2000.
• 53: S. Towers. Personal Communication, July 2004.
• 54: Marc Waldman , David Mazières, Tangler: a censorship-resistant publishing system based on document entanglements, Proceedings of the 8th ACM conference on Computer and Communications Security, November 05-08, 2001, Philadelphia, PA, USA [doi>10.1145/501983.502002]
• 55: Hakim Weatherspoon , John Kubiatowicz, Erasure Coding Vs. Replication: A Quantitative Comparison, Revised Papers from the First International Workshop on Peer-to-Peer Systems, p.328-338, March 07-08, 2002
• 56: Hakim Weatherspoon , Tal Moscovitz , John Kubiatowicz, Introspective Failure Analysis: Avoiding Correlated Failures in Peer-to-Peer Systems, Proceedings of the 21st IEEE Symposium on Reliable Distributed Systems (SRDS'02), p.362, October 13-16, 2002
• 57: D. Whitehouse. Reworked Images Reveal Hot Venus. BBC News, Jan. 2004.
• 58: Lawrence L. You , Kristal T. Pollack , Darrell D. E. Long, Deep Store: An Archival Storage System Architecture, Proceedings of the 21st International Conference on Data Engineering (ICDE'05), p.804-8015, April 05-08, 2005 [doi>10.1109/ICDE.2005.47]

INDEX TERMS

Primary Classification:
  E. Data
  E.5 FILES
      Subjects: Backup/recovery

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Systems issues

General Terms:
Design, Management, Reliability

Keywords:
digital preservation, storage systems