Understanding disk failure rates: What does an MTTF of 1,000,000 hours mean to you?

ABSTRACT

Component failure in large-scale IT installations is becoming an ever-larger problem as the number of components in a single cluster approaches a million.

This article is an extension of our previous study on disk failures [Schroeder and Gibson 2007] and presents and analyzes field-gathered disk replacement data from a number of large production systems, including high-performance computing sites and internet services sites. More than 110,000 disks are covered by this data, some for an entire lifetime of five years. The data includes drives with SCSI and FC, as well as SATA interfaces. The mean time-to-failure (MTTF) of those drives, as specified in their datasheets, ranges from 1,000,000 to 1,500,000 hours, suggesting a nominal annual failure rate of at most 0.88%.

We find that in the field, annual disk replacement rates typically exceed 1%, with 2--4% common and up to 13% observed on some systems. This suggests that field replacement is a fairly different process than one might predict based on datasheet MTTF.

We also find evidence, based on records of disk replacements in the field, that failure rate is not constant with age, and that rather than a significant infant mortality effect, we see a significant early onset of wear-out degradation. In other words, the replacement rates in our data grew constantly with age, an effect often assumed not to set in until after a nominal lifetime of 5 years.

Interestingly, we observe little difference in replacement rates between SCSI, FC, and SATA drives, potentially an indication that disk-independent factors such as operating conditions affect replacement rates more than component-specific ones. On the other hand, we see only one instance of a customer rejecting an entire population of disks as a bad batch, in this case because of media error rates, and this instance involved SATA disks.

Time between replacement, a proxy for time between failure, is not well modeled by an exponential distribution and exhibits significant levels of correlation, including autocorrelation and long-range dependence.

REFERENCES

• 1: Lakshmi N. Bairavasundaram , Garth R. Goodson , Shankar Pasupathy , Jiri Schindler, An analysis of latent sector errors in disk drives, Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 12-16, 2007, San Diego, California, USA
• 2: CFDR. 2007. The computer failure data repository. http://cfdr.usenix.org/.
• 3: Cole, G. 2000. Estimating drive reliability in desktop computers and consumer electronics systems. TP-338.1. Seagate Technology, November.
• 4: Peter Corbett , Bob English , Atul Goel , Tomislav Grcanac , Steven Kleiman , James Leong , Sunitha Sankar, Awarded Best Paper! -- Row-Diagonal Parity for Double Disk Failure Correction, Proceedings of the 3rd USENIX Conference on File and Storage Technologies, March 31-31, 2004, San Francisco, CA
• 5: Drummer, D., Khurshudov, A., Riedel, E., and Watts R. 2006. Personal communication.
• 6: Elerath, J. G. 2000a. AFR: Problems of definition, calculation and measurement in a commercial environment. In Proceedings of the Annual Reliability and Maintainability Symposium.
• 7: Elerath, J. G. 2000b. Specifying reliability in the disk drive industry: No more MTBFs. In Proceedings of the Annual Reliability and Maintainability Symposium.
• 8: Elerath, J. G. and Shah, S. 2004. Server class drives: How reliable are they? In Proceedings of the Annual Reliability and Maintainability Symposium.
• 9: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 10: Garth A. Gibson, Redundant disk arrays: reliable, parallel secondary storage, MIT Press, Cambridge, MA, 1992
• 11: Gray, J. 1990. A census of tandem system availability between 1985 and 1990. IEEE Trans. Reliabil. 39, 4.
• 12: Gray, J. 1986. Why do computers stop and what can be done about it. In Proceedings of the 5th Symposium on Reliability in Distributed Software and Database Systems.
• 13: Taliver Heath , Richard P. Martin , Thu D. Nguyen, Improving cluster availability using workstation validation, Proceedings of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 15-19, 2002, Marina Del Rey, California
• 14: R. K. Iyer , D. J. Rossetti , M. C. Hsueh, Measurement and modeling of computer reliability as affected by system activity, ACM Transactions on Computer Systems (TOCS), v.4 n.3, p.214-237, Aug. 1986 [doi>10.1145/6420.6422]
• 15: M. Kalyanakrishnam , Z. Kalbarczyk , R. Iyer, Failure Data Analysis of a LAN of Windows NT Based Computers, Proceedings of the 18th IEEE Symposium on Reliable Distributed Systems, p.178, October 18-21, 1999
• 16: Karagiannis, T. 2002. Selfis: A short tutorial. Tech. rep., University of California, Riverside.
• 17: Thomas Karagiannis , Mart Molle , Michalis Faloutsos, Long-Range Dependence: Ten Years of Internet Traffic Modeling, IEEE Internet Computing, v.8 n.5, p.57-64, September 2004 [doi>10.1109/MIC.2004.46]
• 18: LANL. http://www.lanl.gov/projects/computerscience/data/.
• 19: Will E. Leland , Murad S. Taqqu , Walter Willinger , Daniel V. Wilson, On the self-similar nature of Ethernet traffic (extended version), IEEE/ACM Transactions on Networking (TON), v.2 n.1, p.1-15, Feb. 1994 [doi>10.1109/90.282603]
• 20: Lin, T.-T. Y. and Siewiorek, D. P. 1990. Error log analysis: Statistical modeling and heuristic trend analysis. IEEE Trans. Reliabil. 39, 4.
• 21: Meyer, J. and Wei, L. 1988. Analysis of workload influence on dependability. In Proceedings of the International Symposium on Fault-Tolerant Computing.
• 22: Murphy, B. and Gent, T. 1995. Measuring system and software reliability using an automated data collection process. Qual. Reliabil. Eng. Int. 11, 5.
• 23: NERSC. 2007. Systems disk failure. http://pdsi.nersc.gov/all_diskfailure.php.
• 24: Nurmi, D., Brevik, J., and Wolski, R. 2005. Modeling machine availability in enterprise and wide-area distributed computing environments. In International Euro-Par Conference on Parallel Processing.
• 25: David Oppenheimer , Archana Ganapathi , David A. Patterson, Why do internet services fail, and what can be done about it?, Proceedings of the 4th conference on USENIX Symposium on Internet Technologies and Systems, p.1-1, March 26-28, 2003, Seattle, WA
• 26: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 27: Eduardo Pinheiro , Wolf-Dietrich Weber , Luiz André Barroso, Failure trends in a large disk drive population, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.2-2, February 13-16, 2007, San Jose, CA
• 28: Vijayan Prabhakaran , Lakshmi N. Bairavasundaram , Nitin Agrawal , Haryadi S. Gunawi , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau, IRON file systems, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom
• 29: Sheldon M. Ross, Introduction to Probability Models, Ninth Edition, Academic Press, Inc., Orlando, FL, 2006
• 30: Ramendra K. Sahoo , Anand Sivasubramaniam , Mark S. Squillante , Yanyong Zhang, Failure Data Analysis of a Large-Scale Heterogeneous Server Environment, Proceedings of the 2004 International Conference on Dependable Systems and Networks, p.772, June 28-July 01, 2004
• 31: Bianca Schroeder , Garth A. Gibson, Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.1-es, February 13-16, 2007, San Jose, CA
• 32: Bianca Schroeder , Garth A. Gibson, A large-scale study of failures in high-performance computing systems, Proceedings of the International Conference on Dependable Systems and Networks, p.249-258, June 25-28, 2006 [doi>10.1109/DSN.2006.5]
• 33: Schwarz, T., Baker, M., Bassi, S., Baumgart, B., Flagg, W., van Ingen, C., Joste, K., Manasse, M., and Shah, M. 2006. Disk failure investigations at the internet archive. In NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST) Work in Progress Session.
• 34: Talagala, N. and Patterson, D. 1999. An analysis of error behaviour in a large storage system. In The IEEE Workshop on Fault Tolerance in Parallel and Distributed Systems.
• 35: Tang, D., Iyer, R. K., and Subramani, S. S. 1990. Failure analysis and modelling of a VAX cluster system. In Proceedings of the International Symposium on Fault-tolerant Computing.
• 36: van Ingen, C. and Gray, J. 2005. Empirical measurements of disk failure rates and error rates. Tech. Rep. MSR-TR-2005-166, Microsoft Research, December.
• 37: Jun Xu , Zbigniew Kalbarczyk , Ravishankar K. Iyer, Networked Windows NT System Field Failure Data Analysis, Proceedings of the 1999 Pacific Rim International Symposium on Dependable Computing, p.178, December 16-17, 1999
• 38: Yang, J. and Sun, F.-B. 1999. A comprehensive review of hard-disk drive reliability. In Proceedings of the Annual Reliability and Maintainability Symposium.

INDEX TERMS

Primary Classification:
  B. Hardware
  B.8 Performance and Reliability
      B.8.0 General

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability

General Terms:
Measurement, Reliability

Keywords:
Hard drive replacements, MTTF, annual failure rates, annual replacement rates, datasheet MTTF, failure correlation, hard drive failure, infant mortality, storage reliability, time between failure, wear-out