Are disks the dominant contributor for storage failures?: A comprehensive study of storage subsystem failure characteristics

ABSTRACT

Building reliable storage systems becomes increasingly challenging as the complexity of modern storage systems continues to grow. Understanding storage failure characteristics is crucially important for designing and building a reliable storage system. While several recent studies have been conducted on understanding storage failures, almost all of them focus on the failure characteristics of one component—disks—and do not study other storage component failures.

This article analyzes the failure characteristics of storage subsystems. More specifically, we analyzed the storage logs collected from about 39,000 storage systems commercially deployed at various customer sites. The dataset covers a period of 44 months and includes about 1,800,000 disks hosted in about 155,000 storage-shelf enclosures. Our study reveals many interesting findings, providing useful guidelines for designing reliable storage systems. Some of our major findings include: (1) In addition to disk failures that contribute to 20--55% of storage subsystem failures, other components such as physical interconnects and protocol stacks also account for a significant percentage of storage subsystem failures. (2) Each individual storage subsystem failure type, and storage subsystem failure as a whole, exhibits strong self-correlations. In addition, these failures exhibit “bursty” patterns. (3) Storage subsystems configured with redundant interconnects experience 30--40% lower failure rates than those with a single interconnect. (4) Spanning disks of a RAID group across multiple shelves provides a more resilient solution for storage subsystems than within a single shelf.

REFERENCES

• 1: Bruce Allen, Monitoring hard disks with smart, Linux Journal, v.2004 n.117, p.9, January 2004
• 2: Lakshmi N. Bairavasundaram , Garth R. Goodson , Bianca Schroeder , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dussea, An analysis of data corruption in the storage stack, Proceedings of the 6th USENIX Conference on File and Storage Technologies, p.1-16, February 26-29, 2008, San Jose, California
• 3: Lakshmi N. Bairavasundaram , Garth R. Goodson , Shankar Pasupathy , Jiri Schindler, An analysis of latent sector errors in disk drives, ACM SIGMETRICS Performance Evaluation Review, v.35 n.1, June 2007
• 4: Cole, G. 2000. Estimating drive reliability in desktop computers and consumer electronics systems. Tech. Rep., Seagate Technology Paper TP-338.1.
• 5: Peter Corbett , Bob English , Atul Goel , Tomislav Grcanac , Steven Kleiman , James Leong , Sunitha Sankar, Awarded Best Paper! -- Row-Diagonal Parity for Double Disk Failure Correction, Proceedings of the 3rd USENIX Conference on File and Storage Technologies, March 31-31, 2004, San Francisco, CA
• 6: Elerath, J. G. and Shah, S. 2004. Server class disk drives: How reliable are they. In Proceedings of the IEEE Reliability and Maintainability Symposium, 151--156.
• 7: Elerath, J. G. and Shah, S. 2003. Disk drive reliability case study: Dependence upon head fly-height and quantity of heads. In Proceedings of the Reliability and Maintainability Symposium, 608--612.
• 8: EMC. 2007. EMC symmetrix DMX-4 specification sheet. http://www.emc.com/products/systems/symmetrix/symmetri_DMX1000/pdf/DMX3000.pdf.
• 9: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 10: Gray, J. 1990. A census of tandem system availability between 1985 and 1990. In Proceedings of the IEEE Transactions on Reliability.
• 11: Larry Lancaster , Alan Rowe, Measuring Real-World Data Availability, Proceedings of the 15th USENIX conference on System administration, December 02-07, 2001, San Diego, California
• 12: NetApp. 2008. FAS6000 series technical specifications. http://www.netapp.com/products/filer/fas6000_tech_specs.html.
• 13: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 14: Eduardo Pinheiro , Wolf-Dietrich Weber , Luiz André Barroso, Failure trends in a large disk drive population, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.2-2, February 13-16, 2007, San Jose, CA
• 15: Rosander, A. C. 1951. Elementary Principles of Statistics. D. Van Nostrand Company.
• 16: Bianca Schroeder , Garth A. Gibson, Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.1-es, February 13-16, 2007, San Jose, CA
• 17: Schulze, M., Gibson, G. A., Katz, R. H., and Patterson, D. A. 1989. How reliable is a RAID? In Proceedings of the COMPCON. 118--123.
• 18: Shah, S. and Elerath, J. G. 2005. Reliability analysis of disk drive failure mechanisms. In Proceedings of the IEEE Reliability and Maintainability Symposium, 226--231.
• 19: SNIA. 2008. Storage Networking Industry Association dictionary. http://www.snia.org/education/dictionary/.
• 20: Nisha Talagala , David Patterson, An Analysis of Error Behaviour in a Large Storage System, University of California at Berkeley, Berkeley, CA, 1999
• 21: Yang, J. and Sun, F.-B. 1999. A comprehensive review of hard-disk drive reliability. In Proceedings of the Reliability and Maintainability Symposium, 403--409.

INDEX TERMS

Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.4 System Management

General Terms:
Management, Measurement, Reliability

Keywords:
Storage system, disk failures, failure characteristics, storage subsystem