DRAM errors in the wild: a large-scale field study

ABSTRACT

Errors in dynamic random access memory (DRAM) are a common form of hardware failure in modern compute clusters. Failures are costly both in terms of hardware replacement costs and service disruption. While a large body of work exists on DRAM in laboratory conditions, little has been reported on real DRAM failures in large production clusters. In this paper, we analyze measurements of memory errors in a large fleet of commodity servers over a period of 2.5 years. The collected data covers multiple vendors, DRAM capacities and technologies, and comprises many millions of DIMM days.

The goal of this paper is to answer questions such as the following: How common are memory errors in practice? What are their statistical properties? How are they affected by external factors, such as temperature and utilization, and by chip-specific factors, such as chip density, memory technology and DIMM age?

We find that DRAM error behavior in the field differs in many key aspects from commonly held assumptions. For example, we observe DRAM error rates that are orders of magnitude higher than previously reported, with 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs affected by errors per year. We provide strong evidence that memory errors are dominated by hard errors, rather than soft errors, which previous work suspects to be the dominant error mode. We find that temperature, known to strongly impact DIMM error rates in lab conditions, has a surprisingly small effect on error behavior in the field, when taking all other factors into account. Finally, unlike commonly feared, we don't observe any indication that newer generations of DIMMs have worse error behavior.

REFERENCES

• 1: Mosys adds soft-error protection, correction. Semiconductor Business News, 28 Jan. 2002.
• 2: Zaid Al-Ars , Ad. J. van de Goor , Jens Braun , Detlev Richter, Simulation based analysis of temperature effect on the faulty behavior of embedded DRAMs, Proceedings of the IEEE International Test Conference 2001, p.783-792, October 30-November 01, 2001
• 3: Robert Baumann, Soft Errors in Advanced Computer Systems, IEEE Design & Test, v.22 n.3, p.258-266, May 2005 [doi>10.1109/MDT.2005.69]
• 4: Fay Chang , Jeffrey Dean , Sanjay Ghemawat , Wilson C. Hsieh , Deborah A. Wallach , Mike Burrows , Tushar Chandra , Andrew Fikes , Robert E. Gruber, Bigtable: a distributed storage system for structured data, Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation, p.15-15, November 06-08, 2006, Seattle, WA
• 5: Fay Chang , Jeffrey Dean , Sanjay Ghemawat , Wilson C. Hsieh , Deborah A. Wallach , Mike Burrows , Tushar Chandra , Andrew Fikes , Robert E. Gruber, Bigtable: a distributed storage system for structured data, Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation, p.15-15, November 06-08, 2006, Seattle, WA
• 6: Fay Chang , Jeffrey Dean , Sanjay Ghemawat , Wilson C. Hsieh , Deborah A. Wallach , Mike Burrows , Tushar Chandra , Andrew Fikes , Robert E. Gruber, Bigtable: a distributed storage system for structured data, Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation, p.15-15, November 06-08, 2006, Seattle, WA
• 7: Sudhakar Govindavajhala , Andrew W. Appel, Using Memory Errors to Attack a Virtual Machine, Proceedings of the 2003 IEEE Symposium on Security and Privacy, p.154, May 11-14, 2003
• 8: T. Hamamoto, S. Sugiura, and S. Sawada. On the retention time distribution of dynamic random access memory (dram). IEEE Transactions on Electron Devices, 45(6):1300--1309, 1998.
• 9: A. H. Johnston. Scaling and technology issues for soft error rates. In Proc. of the 4th Annual Conf. on Reliability, 2000.
• 10: Xin Li , Kai Shen , Michael C. Huang , Lingkun Chu, A memory soft error measurement on production systems, 2007 USENIX Annual Technical Conference on Proceedings of the USENIX Annual Technical Conference, p.1-6, June 17-22, 2007, Santa Clara, CA
• 11: T. C. May and M. H. Woods. Alpha-particle-induced soft errors in dynamic memories. IEEE Transactions on Electron Devices, 26(1), 1979.
• 12: T. C. May and M. H. Woods. Alpha-particle-induced soft errors in dynamic memories. IEEE Transactions on Electron Devices, 26(1), 1979.
• 13: Dejan Milojicic , Alan Messer , James Shau , Guangrui Fu , Alberto Munoz, Increasing relevance of memory hardware errors: a case for recoverable programming models, Proceedings of the 9th workshop on ACM SIGOPS European workshop: beyond the PC: new challenges for the operating system, September 17-20, 2000, Kolding, Denmark [doi>10.1145/566726.566749]
• 14: Shubhendu S. Mukherjee , Joel Emer , Tryggve Fossum , Steven K. Reinhardt, Cache Scrubbing in Microprocessors: Myth or Necessity?, Proceedings of the 10th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC'04), p.37-42, March 03-05, 2004
• 15: Shubhendu S. Mukherjee , Joel Emer , Steven K. Reinhardt, The Soft Error Problem: An Architectural Perspective, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.243-247, February 12-16, 2005 [doi>10.1109/HPCA.2005.37]
• 16: E. Normand. Single event upset at ground level. IEEE Transaction on Nuclear Sciences, 6(43):2742--2750, 1996.
• 17: T. J. O'Gorman , J. M. Ross , A. H. Taber , J. F. Ziegler , H. P. Muhlfeld , C. J. Montrose , H. W. Curtis , J. L. Walsh, Field testing for cosmic ray soft errors in semiconductor memories, IBM Journal of Research and Development, v.40 n.1, p.41-50, Jan. 1996
• 18: Rob Pike , Sean Dorward , Robert Griesemer , Sean Quinlan, Interpreting the data: Parallel analysis with Sawzall, Scientific Programming, v.13 n.4, p.277-298, October 2005
• 19: Bianca Schroeder , Garth A. Gibson, A large-scale study of failures in high-performance computing systems, Proceedings of the International Conference on Dependable Systems and Networks, p.249-258, June 25-28, 2006 [doi>10.1109/DSN.2006.5]
• 20: Bianca Schroeder , Garth A. Gibson, Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.1-es, February 13-16, 2007, San Jose, CA
• 21: Bianca Schroeder , Garth A. Gibson, Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.1-es, February 13-16, 2007, San Jose, CA
• 22: Jun Xu , Shuo Chen , Zbigniew Kalbarczyk , Ravishankar K. Iyer, An Experimental Study of Security Vulnerabilities Caused by Errors, Proceedings of the 2001 International Conference on Dependable Systems and Networks (formerly: FTCS), p.421-432, July 01-04, 2001
• 23: J. F. Ziegler and W. A. Lanford. Effect of Cosmic Rays on Computer Memories. Science, 206:776--788, 1979.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Reliability, availability, and serviceability

General Terms:
Reliability

Keywords:
data corruption, dimm, dram, dram reliability, ecc, empirical study, hard error, large-scale systems, memory, soft error