Understanding the propagation of hard errors to software and implications for resilient system design

ABSTRACT

With continued CMOS scaling, future shipped hardware will be increasingly vulnerable to in-the-field faults. To be broadly deployable, the hardware reliability solution must incur low overheads, precluding use of expensive redundancy. We explore a cooperative hardware-software solution that watches for anomalous software behavior to indicate the presence of hardware faults. Fundamental to such a solution is a characterization of how hardware faults indifferent microarchitectural structures of a modern processor propagate through the application and OS.

This paper aims to provide such a characterization, resulting in identifying low-cost detection methods and providing guidelines for implementation of the recovery and diagnosis components of such a reliability solution. We focus on hard faults because they are increasingly important and have different system implications than the much studied transients. We achieve our goals through fault injection experiments with a microarchitecture-level full system timing simulator. Our main results are: (1) we are able to detect 95% of the unmasked faults in 7 out of 8 studied microarchitectural structures with simple detectors that incur zero to little hardware overhead; (2) over 86% of these detections are within latencies that existing hardware checkpointing schemes can handle, while others require software checkpointing; and (3) a surprisingly large fraction of the detected faults corrupt OS state, but almost all of these are detected with latencies short enough to use hardware checkpointing, thereby enabling OS recovery in virtually all such cases.

REFERENCES

• 1: J. Arlat , A. Costes , Y. Crouzet , J. C. Laprie , D. Powell, Fault Injection and Dependability Evaluation of Fault-Tolerant Systems, IEEE Transactions on Computers, v.42 n.8, p.913-923, August 1993 [doi>10.1109/12.238482]
• 2: Todd M. Austin, DIVA: a reliable substrate for deep submicron microarchitecture design, Proceedings of the 32nd annual ACM/IEEE international symposium on Microarchitecture, p.196-207, November 16-18, 1999, Haifa, Israel
• 3: David Bernick , Bill Bruckert , Paul Del Vigna , David Garcia , Robert Jardine , Jim Klecka , Jim Smullen, NonStop® Advanced Architecture, Proceedings of the 2005 International Conference on Dependable Systems and Networks, p.12-21, June 28-July 01, 2005 [doi>10.1109/DSN.2005.70]
• 4: Shekhar Borkar, Designing Reliable Systems from Unreliable Components: The Challenges of Transistor Variability and Degradation, IEEE Micro, v.25 n.6, p.10-16, November 2005 [doi>10.1109/MM.2005.110]
• 5: Shekhar Borkar, Microarchitecture and Design Challenges for Gigascale Integration, Proceedings of the 37th annual IEEE/ACM International Symposium on Microarchitecture, p.3-3, December 04-08, 2004, Portland, Oregon [doi>10.1109/MICRO.2004.24]
• 6: Fred A. Bower , Daniel J. Sorin , Sule Ozev, A Mechanism for Online Diagnosis of Hard Faults in Microprocessors, Proceedings of the 38th annual IEEE/ACM International Symposium on Microarchitecture, p.197-208, November 12-16, 2005, Barcelona, Spain [doi>10.1109/MICRO.2005.8]
• 7: Fred A. Bower , Paul G. Shealy , Sule Ozev , Daniel J. Sorin, Tolerating Hard Faults in Microprocessor Array Structures, Proceedings of the 2004 International Conference on Dependable Systems and Networks, p.51, June 28-July 01, 2004
• 8: Kypros Constantinides , Onur Mutlu , Todd Austin , Valeria Bertacco, Software-Based Online Detection of Hardware Defects Mechanisms, Architectural Support, and Evaluation, Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, p.97-108, December 01-05, 2007 [doi>10.1109/MICRO.2007.39]
• 9: Edward W. Czeck and Daniel P. Siewiorek. Effects of Transient Gate-Level Faults on Program Behavior. In International Symposium on Fault-Tolerant Computing (FTCS), 1990.
• 10: George W. Dunlap , Samuel T. King , Sukru Cinar , Murtaza A. Basrai , Peter M. Chen, ReVirt: enabling intrusion analysis through virtual-machine logging and replay, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060309]
• 11: Michael D. Ernst , Jeff H. Perkins , Philip J. Guo , Stephen McCamant , Carlos Pacheco , Matthew S. Tschantz , Chen Xiao, The Daikon system for dynamic detection of likely invariants, Science of Computer Programming, v.69 n.1-3, p.35-45, December, 2007 [doi>10.1016/j.scico.2007.01.015]
• 12: O. Goloubeva , M. Rebaudengo , M. Sonza Reorda , M. Violante, Soft-Error Detection Using Control Flow Assertions, Proceedings of the 18th IEEE International Symposium on Defect and Fault Tolerance in VLSI Systems, p.581, November 03-05, 2003
• 13: Mohamed Gomaa , Chad Scarbrough , T. N. Vijaykumar , Irith Pomeranz, Transient-fault recovery for chip multiprocessors, Proceedings of the 30th annual international symposium on Computer architecture, June 09-11, 2003, San Diego, California
• 14: Weining Gu , Zbigniew Kalbarczyk , Ravishankar K. Iyer, Error Sensitivity of the Linux Kernel Executing on PowerPC G4 and Pentium 4 Processors, Proceedings of the 2004 International Conference on Dependable Systems and Networks, p.887, June 28-July 01, 2004
• 15: Sudheendra Hangal , Monica S. Lam, Tracking down software bugs using automatic anomaly detection, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581377]
• 16: Mei-Chen Hsueh , Timothy K. Tsai , Ravishankar K. Iyer, Fault Injection Techniques and Tools, Computer, v.30 n.4, p.75-82, April 1997 [doi>10.1109/2.585157]
• 17: Ghani A. Kanawati , Nasser A. Kanawati , Jacob A. Abraham, FERRARI: A Flexible Software-Based Fault and Error Injection System, IEEE Transactions on Computers, v.44 n.2, p.248-260, February 1995 [doi>10.1109/12.364536]
• 18: Hue-Sung Kim , Arun K. Somani , Akhilesh Tyagi, A reconfigurable multi-function computing cache architecture, Proceedings of the 2000 ACM/SIGDA eighth international symposium on Field programmable gate arrays, p.85-94, February 10-11, 2000, Monterey, California, United States [doi>10.1145/329166.329185]
• 19: Chris Lattner , Vikram Adve, LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation, Proceedings of the international symposium on Code generation and optimization: feedback-directed and runtime optimization, p.75, March 20-24, 2004, Palo Alto, California
• 20: SoftArch: An Architecture Level Tool for Modeling and Analyzing Soft Errors, Proceedings of the 2005 International Conference on Dependable Systems and Networks, p.496-505, June 28-July 01, 2005 [doi>10.1109/DSN.2005.88]
• 21: Ben Liblit , Mayur Naik , Alice X. Zheng , Alex Aiken , Michael I. Jordan, Scalable statistical bug isolation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 22: Shan Lu , Joseph Tucek , Feng Qin , Yuanyuan Zhou, AVIO: detecting atomicity violations via access interleaving invariants, Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, October 21-25, 2006, San Jose, California, USA
• 23: Milo M. K. Martin , Daniel J. Sorin , Bradford M. Beckmann , Michael R. Marty , Min Xu , Alaa R. Alameldeen , Kevin E. Moore , Mark D. Hill , David A. Wood, Multifacet's general execution-driven multiprocessor simulator (GEMS) toolset, ACM SIGARCH Computer Architecture News, v.33 n.4, November 2005 [doi>10.1145/1105734.1105747]
• 24: Carl J. Mauer , Mark D. Hill , David A. Wood, Full-system timing-first simulation, ACM SIGMETRICS Performance Evaluation Review, v.30 n.1, June 2002
• 25: Albert Meixner , Michael E. Bauer , Daniel Sorin, Argus: Low-Cost, Comprehensive Error Detection in Simple Cores, Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, p.210-222, December 01-05, 2007 [doi>10.1109/MICRO.2007.8]
• 26: Albert Meixner , Daniel J. Sorin, Error Detection Using Dynamic Dataflow Verification, Proceedings of the 16th International Conference on Parallel Architecture and Compilation Techniques, p.104-118, September 15-19, 2007 [doi>10.1109/PACT.2007.30]
• 27: M. Mueller et al. RAS Strategy for IBM S/390 G5 and G6. IBM Journal on Research and Development, 43(5/6), Sept/Nov 1999.
• 28: Shubhendu S. Mukherjee , Christopher Weaver , Joel Emer , Steven K. Reinhardt , Todd Austin, A Systematic Methodology to Compute the Architectural Vulnerability Factors for a High-Performance Microprocessor, Proceedings of the 36th annual IEEE/ACM International Symposium on Microarchitecture, p.29, December 03-05, 2003
• 29: Jun Nakano et al. ReViveI/O: Efficient Handling of I/O in Highly-Available Rollback-Recovery Servers. In International Symposium on High Performance Computer Architecture (HPCA), 2006.
• 30: Nithin Nakka et al. An Architectural Framework for Detecting Process Hangs/Crashes. In European Dependable Computing Conference (EDCC), 2005.
• 31: K. Pattabiraman , G. P. Saggese , D. Chen , Z. Kalbarczyk , R. K. Iyer, Dynamic Derivation of Application-Specific Error Detectors and their Implementation in Hardware, Proceedings of the Sixth European Dependable Computing Conference, p.97-108, October 18-20, 2006 [doi>10.1109/EDCC.2006.9]
• 32: Milos Prvulovic , Zheng Zhang , Josep Torrellas, ReVive: cost-effective architectural support for rollback recovery in shared-memory multiprocessors, Proceedings of the 29th annual international symposium on Computer architecture, May 25-29, 2002, Anchorage, Alaska
• 33: Paul Racunas , Kypros Constantinides , Srilatha Manne , Shubhendu S. Mukherjee, Perturbation-based Fault Screening, Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, p.169-180, February 10-14, 2007 [doi>10.1109/HPCA.2007.346195]
• 34: V. Reddy et al. Assertion-Based Microarchitecture Design for Improved Fault Tolerance. In International Conference on Computer Design, 2006.
• 35: Steven K. Reinhardt , Shubhendu S. Mukherjee, Transient fault detection via simultaneous multithreading, Proceedings of the 27th annual international symposium on Computer architecture, p.25-36, June 2000, Vancouver, British Columbia, Canada
• 36: George A. Reis , Jonathan Chang , Neil Vachharajani , Ram Rangan , David I. August , Shubhendu S. Mukherjee, Software-controlled fault tolerance, ACM Transactions on Architecture and Code Optimization (TACO), v.2 n.4, p.366-396, December 2005 [doi>10.1145/1113841.1113843]
• 37: R. Rodriguez et al. Modeling and Experimental Verification of the Effect of Gate Oxide Breakdown on CMOS Inverters. In International Reliability Physics Symposium (IRPS), 2003.
• 38: Eric Rotenberg, AR-SMT: A Microarchitectural Approach to Fault Tolerance in Microprocessors, Proceedings of the Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing, p.84, June 15-18, 1999
• 39: Giacinto P. Saggese , Nicholas J. Wang , Zbigniew T. Kalbarczyk , Sanjay J. Patel , Ravishankar K. Iyer, An Experimental Study of Soft Errors in Microprocessors, IEEE Micro, v.25 n.6, p.30-39, November 2005 [doi>10.1109/MM.2005.104]
• 40: Design Panel, SELSE II -- Reverie, 2006. http://www.selse.org/selse2.org/recap.pdf.
• 41: Smitha Shyam , Kypros Constantinides , Sujay Phadke , Valeria Bertacco , Todd Austin, Ultra low-cost defect protection for microprocessor pipelines, Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, October 21-25, 2006, San Jose, California, USA
• 42: Daniel Sorin et al. Fast Checkpoint/Recovery to Support Kilo-Instruction Speculation and Hardware Fault Tolerance. Technical Report 1420, Computer Sciences Department, University of Wisconsin, Madison, 2000.
• 43: Daniel J. Sorin , Milo M. K. Martin , Mark D. Hill , David A. Wood, SafetyNet: improving the availability of shared memory multiprocessors with global checkpoint/recovery, Proceedings of the 29th annual international symposium on Computer architecture, May 25-29, 2002, Anchorage, Alaska
• 44: Jayanth Srinivasan , Sarita V. Adve , Pradip Bose , Jude A. Rivers, The Impact of Technology Scaling on Lifetime Reliability, Proceedings of the 2004 International Conference on Dependable Systems and Networks, p.177, June 28-July 01, 2004
• 45: Sudarshan M. Srinivasan , Srikanth Kandula , Christopher R. Andrews , Yuanyuan Zhou, Flashback: a lightweight extension for rollback and deterministic replay for software debugging, Proceedings of the annual conference on USENIX Annual Technical Conference, p.3-3, June 27-July 02, 2004, Boston, MA
• 46: Rajesh Venkatasubramanian et al. Low-Cost On-Line Fault Detection Using Control Flow Assertions. In International On-Line Test Symposium, 2003.
• 47: Virtutech. Simics Full System Simulator. Website, 2006. http://www.simics.net.
• 48: Nicholas J. Wang , Justin Quek , Todd M. Rafacz , Sanjay J. patel, Characterizing the Effects of Transient Faults on a High-Performance Processor Pipeline, Proceedings of the 2004 International Conference on Dependable Systems and Networks, p.61, June 28-July 01, 2004
• 49: Nicholas J. Wang , Sanjay J. Patel, ReStore: Symptom-Based Soft Error Detection in Microprocessors, IEEE Transactions on Dependable and Secure Computing, v.3 n.3, p.188-201, July 2006 [doi>10.1109/TDSC.2006.40]
• 50: David Yen. Chip Multithreading Processors Enable Reliable High Throughput Computing. In International Reliability Physics Symposium (IRPS), 2005. Keynote Address.
• 51: Pin Zhou , Wei Liu , Long Fei , Shan Lu , Feng Qin , Yuanyuan Zhou , Samuel Midkiff , Josep Torrellas, AccMon: Automatically Detecting Memory-Related Bugs via Program Counter-Based Invariants, Proceedings of the 37th annual IEEE/ACM International Symposium on Microarchitecture, p.269-280, December 04-08, 2004, Portland, Oregon [doi>10.1109/MICRO.2004.3]
• 52: Pin Zhou , Feng Qin , Wei Liu , Yuanyuan Zhou , Josep Torrellas, iWatcher: Efficient Architectural Support for Software Debugging, Proceedings of the 31st annual international symposium on Computer architecture, p.224, June 19-23, 2004, München, Germany
• 53: Pin Zhou , Radu Teodorescu , Yuanyuan Zhou, HARD: Hardware-Assisted Lockset-based Race Detection, Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, p.121-132, February 10-14, 2007 [doi>10.1109/HPCA.2007.346191]

INDEX TERMS

Primary Classification:
  B. Hardware
  B.8 Performance and Reliability
      B.8.1 Reliability, Testing, and Fault-Tolerance

General Terms:
Design, Experimentation, Reliability

Keywords:
architecture, error detection, fault injection, permanent fault