Implementation and Evaluation of a Scalable Application-Level Checkpoint-Recovery Scheme for MPI Programs

ABSTRACT

The running times of many computational science applications are much longer than the mean-time-to-failure of current high-performance computing platforms. To run to completion, such applications must tolerate hardware failures. Checkpoint-and-restart (CPR) is the most commonly used scheme for accomplishing this - the state of the computation is saved periodically on stable storage, and when a hardware failure is detected, the computation is restarted from the most recently saved state. Most automatic CPR schemes in the literature can be classified as system-level checkpointing schemes because they take core-dump style snapshots of the computational state when all the processes are blocked at global barriers in the program. Unfortunately, a system that implements this style of checkpointing is tied to a particular platform; in addition, it cannot be used if there are no global barriers in the program. We are exploring an alternative called application-level, non-blocking checkpointing. In our approach, programs are transformed by a pre-processor so that they become self-checkpointing and self-restartable on any platform; there is also no assumption about the existence of global barriers in the code. In this paper, we describe our implementation of application-level, non-blocking checkpointing. We present experimental results on both a Windows cluster and a Compaq Alpha cluster, which show that the overheads introduced by our approach are small.

REFERENCES

• 1: Adnan Agbaria , Roy Friedman, Starfish: Fault-Tolerant Dynamic MPI Programs on Clusters of Workstations, Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing, p.31, August 03-06, 1999
• 2: Micah Beck , James S. Plank , Gerry Kingsley, Compiler-Assisted Checkpointing, University of Tennessee, Knoxville, TN, 1994
• 3: Adam Beguelin , Erik Seligman , Peter Stephan, Application level fault tolerance in heterogeneous networks of workstations, Journal of Parallel and Distributed Computing, v.43 n.2, p.147-155, June 15, 1997 [doi>10.1006/jpdc.1997.1338]
• 4: Bouteiller Bouteiller , Franck Cappello , Thomas Herault , Krawezik Krawezik , Pierre Lemarinier , Magniette Magniette, MPICH-V2: a Fault Tolerant MPI for Volatile Nodes based on Pessimistic Sender Based Message Logging, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.25, November 15-21, 2003
• 5: Greg Bronevetsky , Daniel Marques , Keshav Pingali , Paul Stodghill, Automated application-level checkpointing of MPI programs, Proceedings of the ninth ACM SIGPLAN symposium on Principles and practice of parallel programming, June 11-13, 2003, San Diego, California, USA
• 6: Greg Bronevetsky , Daniel Marques , Keshav Pingali , Paul Stodghill, Collective operations in application-level fault-tolerant MPI, Proceedings of the 17th annual international conference on Supercomputing, June 23-26, 2003, San Francisco, CA, USA [doi>10.1145/782814.782847]
• 7: [7] B. Carnes. The smg2000 benchmark code. Available at http://www.llnl.gov/asci/purple/ benchmarks/limited/smg/September 19 2001.
• 8: K. Mani Chandy , Leslie Lamport, Distributed snapshots: determining global states of distributed systems, ACM Transactions on Computer Systems (TOCS), v.3 n.1, p.63-75, Feb. 1985 [doi>10.1145/214451.214456]
• 9: [9] Condor. http://www.cs.wisc.edu/condor/manual.
• 10: Elmootazbellah N. Elnozahy , Willy Zwaenepoel, Manetho: Transparent Roll Back-Recovery with Low Overhead, Limited Rollback, and Fast Output Commit, IEEE Transactions on Computers, v.41 n.5, p.526-531, May 1992 [doi>10.1109/12.142678]
• 11: [11] M. Elnozahy, L. Alvisi, Y.M. Wang, and D.B. Johnson. A survey of rollback-recovery protocols in message passing systems. Technical Report CMU-CS-96-181, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA, Oct. 1996.
• 12: Graham E. Fagg , Jack Dongarra, FT-MPI: Fault Tolerant MPI, Supporting Dynamic Applications in a Dynamic World, Proceedings of the 7th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, p.346-353, January 2000
• 13: Adam J. Ferrari , Stephen J. Chapin , Andrew S. Grimshaw, Process Introspection: A Heterogeneous Checkpoint/Restart Mechanism Based on Automatic Code Modification, University of Virginia, Charlottesville, VA, 1997
• 14: Alexandros V. Gerbessiotis , Leslie G. Valiant, Direct bulk-synchronous parallel algorithms, Journal of Parallel and Distributed Computing, v.22 n.2, p.251-267, Aug. 1994 [doi>10.1006/jpdc.1994.1085]
• 15: [15] C.-C.J. Li and W.K. Fuchs. Catch - compiler-assisted techniques for checkpointing. In 20th International Symposium on Fault Tolerant Computing, pages 74-81, 1990.
• 16: Nancy A. Lynch, Distributed Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1996
• 17: Micah Beck , James S. Plank , Gerry Kingsley, Compiler-Assisted Checkpointing, University of Tennessee, Knoxville, TN, 1994
• 18: [18] J.B.M. Litzkow, T. Tannenbaum and M. Livny. Checkpoint and migration of UNIX processes in the condor distributed processing system. Technical Report 1346, University of Wisconsin-Madison, 1997.
• 19: [19] K. Perumalla and R. Fujimoto. Source-code transformations for efficient reversibility. Technical Report GIT-CC-99-21, College of Computing, Georgia Tech, September 1999.
• 20: [20] A. Petitet, R.C. Whaley, J. Dongarra, and A. Cleary. Hpl - a portable implementation of the high-performance linpack benchmark for distributed-memory computers. Available at http://www.netlib.org/benchmark/hpl/.
• 21: Balkrishna Ramkumar , Volker Strumpen, Portable Checkpointing for Heterogeneous Archtitectures, Proceedings of the 27th International Symposium on Fault-Tolerant Computing (FTCS '97), p.58, June 25-27, 1997
• 22: Sriram Rao , Lorenzo Alvisi , Harrick M. Vin, Egida: An Extensible Toolkit For Low-Overhead Fault-Tolerance, Proceedings of the Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing, p.48, June 15-18, 1999
• 23: Georg Stellner, CoCheck: Checkpointing and Process Migration for MPI, Proceedings of the 10th International Parallel Processing Symposium, p.526-531, April 15-19, 1996
• 24: [24] N. Stone, J. Kochmar, R. Reddy, J.R. Scott, J. Sommer field, and C. Vizino. A checkpoint and recovery system for the Pittsburgh Supercomputing Center Terascale Computing System. In Supercomputing, 2001. Available at http://www.psc.edu/publications/tech\ _reports/chkpt\_rcvry/ checkpoint-recovery-1.0.html
• 25: [25] S. Vadhiyar and J. Dongarra. Srs -a framework for developing malleable and migratable parallel software. Parallel Processing Letters, 13(2):291-312, June 2003.