CLIP: a checkpointing tool for message-passing parallel programs

ABSTRACT

Checkpointing is a useful technique for rollback recovery of parallel applications. While extensive research has been performed on checkpointing in parallel environments, there are few checkpointers available to application users on commercial parallel computers. This paper presents one such checkpointer: CLIP. CLIP is a user-level library that provides semi-transparent check-pointing for parallel programs on the Intel Paragon multicomputer. It is publicly available to Paragon users at no cost.Conceptually, checkpointing a multicomputer is quite straightforward. However, when creating an actual tool for checkpointing a complex machine like the Paragon, many more issues arise that require careful design decisions to be made. Sometimes ease-of-use must be sacrificed for efficiency and/or correctness. This paper details what these decisions are, and how they were made in CLIP.We also present performance data when checkpointing several long-running Paragon applications with CLIP. The bottom line is that a convenient, general-purpose checkpointing tool like CLIP can provide fault-tolerance on a massively parallel multicomputer like the Paragon with very good performance.

REFERENCES

• 1: A. Baratloo , P. Dasgupta , Z. M. Kedem, CALYPSO: a novel software system for fault-tolerant parallel processing on distributed platforms, Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing - HPDC '95, p.122, August 02-04, 1995
• 2: Adam Beguelin , Erik Seligman , Peter Stephan, Application level fault tolerance in heterogeneous networks of workstations, Journal of Parallel and Distributed Computing, v.43 n.2, p.147-155, June 15, 1997 [doi>10.1006/jpdc.1997.1338]
• 3: J. Casas, D. L. Clark, P. S. Galbiati, R. Konuru, S. W. Otto, R. M. Prouty, and J. Walpole. MIST: PVM with transparent migration and checkpointing. In 3rd Annual PVM Users' Group Meeting, Pittsburgh, PA, May 1995.
• 4: Tzi-Cker Chiueh , Peitao Deng, Evaluation of checkpoint mechanisms for massively parallel machines, Proceedings of the The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing (FTCS '96), p.370, June 25-27, 1996
• 5: E. N. Elnozahy, D. B. Johnson, and Y. M. Wang. A survey of rollback-recovery protocols in message-passing systems. Technical Report CMU-CS-96-181, Carnegie Mellon University, October 1996.
• 6: E. N. Elnozahy, D. B. Johnson, and W. Zwaenepoel. The performance of consistent checkpointing. In 11th Symposium on Reliable Distributed Systems, pages 39-47, October 1992.
• 7: E. N. Elnozahy and W. Zwaenepoel. On the use and implementation of message logging. In 24th International Symposium on Fault-Tolerant Computing, pages 298-307, Austin, TX, June 1994.
• 8: David Bailey et al. The nas parallel benchmarks 2.0. Technical Report NAS-95-020, NASA Ames Research Center, December 1995.
• 9: Stuart I. Feldman , Channing B. Brown, IGOR: a system for program debugging via reversible execution, ACM SIGPLAN Notices, v.24 n.1, p.112-123, Jan. 1989
• 10: Ronald D. Henderson , George E. Karniadakis, Unstructured spectral element methods for simulation of turbulent flows, Journal of Computational Physics, v.122 n.2, p.191-217, Dec. 1995 [doi>10.1006/jcph.1995.1208]
• 11: Y. Huang and C. Kintala. Software implemented fault tolerance: Technologies and experience. In 23rd International Symposium on Fault-Tolerant Computing, pages 2-9, July 1993.
• 12: B. A. Kingsbury and J. T. Kline. Job and process recovery in a UNIX-based operating system. In Usenix Winter 1989 Technical Conference, pages 355-364, San Diego, CA, January 1989.
• 13: C. R. Landau. The checkpoint mechanism in keykos. In Proceedings of the 2nd International Workshop on Object Orientation in Operating Systems, pages 86-91. IEEE, September 1992.
• 14: K. Li , J. F. Naughton , J. S. Plank, Low-Latency, Concurrent Checkpointing for Parallel Programs, IEEE Transactions on Parallel and Distributed Systems, v.5 n.8, p.874-879, August 1994 [doi>10.1109/71.298215]
• 15: Message Passing Interface Forum. MPI: A Message-Passing Interface Standard, 1.0 edition, May 1994.
• 16: Phyllis E. Crandall , Ruth A. Aydt , Andrew A. Chien , Daniel A. Reed, Input/output characteristics of scalable parallel applications, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.59-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224396]
• 17: Paul Pierce. The Paragon implementation of the NX message passing interface. In Proceedings of Scalable High-Performance Computing Conference (SHPCC) 94, 1994.
• 18: J. S. Plank, M. Beck, and G. Kingsley. Compiler-assisted memory exclusion for fast checkpointing. IEEE Technical Committee on Operating Systems and Application Environments, 7(4):10-14, Winter 1995.
• 19: J. S. Plank, M. Beck, G. Kingsley, and K. Li. Libckpt: Transparent checkpointing under unix. In Usenix Winter 1995 Technical Conference, pages 213-223, January 1995.
• 20: James S. Plank , Kai Li, ickp: A Consistent Checkpointer for Multicomputers, IEEE Parallel & Distributed Technology: Systems & Technology, v.2 n.2, p.62-67, June 1994 [doi>10.1109/88.311574]
• 21: M. Russinovich , Z. Segall, Fault Tolerance for Off-the-Shelf Applications and Hardware, Proceedings of the Twenty-Fifth International Symposium on Fault-Tolerant Computing, p.67, June 27-30, 1995
• 22: L. M. Silva, B. Veer, and J. G. Silva. Checkpointing SPMD applications on transputer networks. In Scalable High Performance Computing Conference, pages 694-701, Knoxville, TN, May 1994.
• 23: Georg Stellner, CoCheck: Checkpointing and Process Migration for MPI, Proceedings of the 10th International Parallel Processing Symposium, p.526-531, April 15-19, 1996
• 24: T. Tannenbaum and M. Litzkow. The Condor distributed processing system. Dr. Dobb's Journal, #227:40-48, February 1995.
• 25: P. H. Worley and I. T. Foster. Parallel spectral transform shallow water model: A runtime-tunable parallel benchmark code. In Proceedings of Scalable High-Performance Computing Conference (SHPCC) 94, 1994.