Fault injection framework for system resilience evaluation: fake faults for finding future failures

ABSTRACT

As high-performance computing (HPC) systems increase in size and complexity they become more difficult to manage. The enormous component counts associated with these large systems lead to significant challenges in system reliability and availability. This in turn is driving research into the resilience of large scale systems, which seeks to curb the effects of increased failures at large scales by masking the inevitable faults in these systems. The basic premise being that failure must be accepted as a reality of large scale system and coped with accordingly through system resilience.

A key component in the development and evaluation of system resilience techniques is having a means to conduct controlled experiments. A common method for performing such experiments is to generate synthetic faults and study the resulting effects. In this paper we discuss the motivation and our initial use of software fault injection to support the evaluation of resilience for HPC systems. We mention background and related work in the area and discuss the design of a tool to aid in fault injection experiments for both user-space (application-level) and system-level failures.

REFERENCES

• 1: Algirdas Avizienis, Jean-Claude Laprie, and Brian Randell. Fundamental concepts of computer system dependability. In IARP/IEEE-RAS Workshop on Robot Dependability: Technological Challenge of Dependable Robots in Human Environments, May 21-22, 2001.
• 2: Francis M. David , Roy H. Campbell, Building a Self-Healing Operating System, Proceedings of the Third IEEE International Symposium on Dependable, Autonomic and Secure Computing, p.3-10, September 25-26, 2007 [doi>10.1109/DASC.2007.5]
• 3: Dbench: Dependability benchmark project. Available at: http://www.laas.fr/DBench/. (Last accessed March 2009).
• 4: Fig: Library-level error injection for shared libraries in unix/linux. Available at: http://roc.cs.berkeley.edu/projects/fig/index.shtml (Last accessed: March 2009).
• 5: Mei-Chen Hsueh , Timothy K. Tsai , Ravishankar K. Iyer, Fault Injection Techniques and Tools, Computer, v.30 n.4, p.75-82, April 1997 [doi>10.1109/2.585157]
• 6: Tahar Jarboui , Jean Arlat , Yves Crouzet , Karama Kanoun, Experimental Analysis of the Errors Induced into Linux by Three Fault Injection Techniques, Proceedings of the 2002 International Conference on Dependable Systems and Networks, p.331-336, June 23-26, 2002
• 7: João Carreira , Henrique Madeira , João Gabriel Silva, Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers, IEEE Transactions on Software Engineering, v.24 n.2, p.125-136, February 1998 [doi>10.1109/32.666826]
• 8: Linux fault injection capabilities infrastructure. Documentation available at: http://lxr.linux.no/linux/Documentation/faultinjection/.
• 9: Adam Oliner , Jon Stearley, What Supercomputers Say: A Study of Five System Logs, Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, p.575-584, June 25-28, 2007 [doi>10.1109/DSN.2007.103]
• 10: S. Potyra , V. Sieh , M. Dal Cin, Evaluating fault-tolerant system designs using FAUmachine, Proceedings of the 2007 workshop on Engineering fault tolerant systems, p.9-es, September 04-04, 2007, Dubrovnik, Croatia [doi>10.1145/1316550.1316559]
• 11: Roc: Recovery-oriented computing. Available at: http://roc.cs.berkeley.edu/ (Last accessed: March 2009).
• 12: David T. Stott , Benjamin Floering , Zbigniew Kalbarczyk , Ravishankar K. Iyer, A Framework for Assessing Dependability in Distributed Systems with Lightweight Fault Injectors, Proceedings of the 4th International Computer Performance and Dependability Symposium, p.91, March 27-30, 2000

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Fault tolerance

General Terms:
Experimentation, Reliability

Keywords:
fault injection, resilience